From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758134AbbIVOdT (ORCPT ); Tue, 22 Sep 2015 10:33:19 -0400 Received: from e06smtp12.uk.ibm.com ([195.75.94.108]:59788 "EHLO e06smtp12.uk.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757882AbbIVOdR (ORCPT ); Tue, 22 Sep 2015 10:33:17 -0400 X-Helo: d06dlp01.portsmouth.uk.ibm.com X-MailFrom: schwidefsky@de.ibm.com X-RcptTo: linux-kernel@vger.kernel.org Date: Tue, 22 Sep 2015 16:33:07 +0200 From: Martin Schwidefsky To: Boqun Feng Cc: paulmck@linux.vnet.ibm.com, Peter Zijlstra , Davidlohr Bueso , Ingo Molnar , Thomas Gleixner , linux-kernel@vger.kernel.org, Davidlohr Bueso , heiko.carstens@de.ibm.com Subject: Re: [PATCH -tip 2/3] sched/wake_q: Relax to acquire semantics Message-ID: <20150922163307.7eeff6b9@mschwide> In-Reply-To: <20150922132913.GA27867@fixme-laptop.cn.ibm.com> References: <20150915140922.GG4029@linux.vnet.ibm.com> <20150915141439.GE16853@twins.programming.kicks-ass.net> <20150915153448.GI4029@linux.vnet.ibm.com> <20150915163028.GG16853@twins.programming.kicks-ass.net> <20150915170941.GL4029@linux.vnet.ibm.com> <20150918214120.GA4405@linux.vnet.ibm.com> <20150921112252.3c2937e1@mschwide> <20150922122735.14f3c573@mschwide> <20150922122326.GA1032@fixme-laptop.cn.ibm.com> <20150922145136.761241da@mschwide> <20150922132913.GA27867@fixme-laptop.cn.ibm.com> X-Mailer: Claws Mail 3.9.3 (GTK+ 2.24.23; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/07j6Dk+T_Ds+sAaYRXvX2Bk"; protocol="application/pgp-signature" X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15092214-0009-0000-0000-000005A9FEC0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --Sig_/07j6Dk+T_Ds+sAaYRXvX2Bk Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Tue, 22 Sep 2015 21:29:14 +0800 Boqun Feng wrote: > On Tue, Sep 22, 2015 at 02:51:36PM +0200, Martin Schwidefsky wrote: > > On Tue, 22 Sep 2015 20:23:26 +0800 > > Boqun Feng wrote: > >=20 > > > Hi Martin, > > >=20 > > > On Tue, Sep 22, 2015 at 12:27:35PM +0200, Martin Schwidefsky wrote: > > > > On Mon, 21 Sep 2015 11:22:52 +0200 > > > > Martin Schwidefsky wrote: > > > >=20 > > > > > On Fri, 18 Sep 2015 14:41:20 -0700 > > > > > "Paul E. McKenney" wrote: > > > > >=20 > > > > > > On Tue, Sep 15, 2015 at 10:09:41AM -0700, Paul E. McKenney wrot= e: > > > > > > > On Tue, Sep 15, 2015 at 06:30:28PM +0200, Peter Zijlstra wrot= e: > > > > > > > > On Tue, Sep 15, 2015 at 08:34:48AM -0700, Paul E. McKenney = wrote: > > > > > > > > > On Tue, Sep 15, 2015 at 04:14:39PM +0200, Peter Zijlstra = wrote: > > > > > > > > > > On Tue, Sep 15, 2015 at 07:09:22AM -0700, Paul E. McKen= ney wrote: > > > > > > > > > > > On Tue, Sep 15, 2015 at 02:48:00PM +0200, Peter Zijls= tra wrote: > > > > > > > > > > > > On Tue, Sep 15, 2015 at 05:41:42AM -0700, Paul E. M= cKenney wrote: > > > > > > > > > > > > > > Never mind, the PPC people will implement this = with lwsync and that is > > > > > > > > > > > > > > very much not transitive IIRC. > > > > > > > > > > > > >=20 > > > > > > > > > > > > > I am probably lost on context, but... > > > > > > > > > > > > >=20 > > > > > > > > > > > > > It turns out that lwsync is transitive in special= cases. One of them > > > > > > > > > > > > > is a series of release-acquire pairs, which can e= xtend indefinitely. > > > > > > > > > > > > >=20 > > > > > > > > > > > > > Does that help in this case? > > > > > > > > > > > >=20 > > > > > > > > > > > > Probably not, but good to know. I still don't think= we want to rely on > > > > > > > > > > > > ACQUIRE/RELEASE being transitive in general though. > > > > > > > > > > >=20 > > > > > > > > > > > OK, I will bite... Why not? > > > > > > > > > >=20 > > > > > > > > > > It would mean us reviewing all archs (again) and docume= nting it I > > > > > > > > > > suppose. Which is of course entirely possible. > > > > > > > > > >=20 > > > > > > > > > > That said, I don't think the case at hand requires it, = so lets postpone > > > > > > > > > > this for now ;-) > > > > > > > > >=20 > > > > > > > > > True enough, but in my experience smp_store_release() and > > > > > > > > > smp_load_acquire() are a -lot- easier to use than other b= arriers, > > > > > > > > > and transitivity will help promote their use. So... > > > > > > > > >=20 > > > > > > > > > All the TSO architectures (x86, s390, SPARC, HPPA, ...) s= upport transitive > > > > > > > > > smp_store_release()/smp_load_acquire() via their native o= rdering in > > > > > > > > > combination with barrier() macros. x86 with CONFIG_X86_P= PRO_FENCE=3Dy, > > > > > > > > > which is not TSO, uses an mfence instruction. Power supp= orts this via > > > > > > > > > lwsync's partial cumulativity. ARM64 supports it in SMP = via the new ldar > > > > > > > > > and stlr instructions (in non-SMP, it uses barrier(), whi= ch suffices > > > > > > > > > in that case). IA64 supports this via total ordering of = all release > > > > > > > > > instructions in theory and by the actual full-barrier imp= lementation > > > > > > > > > in practice (and the fact that gcc emits st.rel and ld.ac= q instructions > > > > > > > > > for volatile stores and loads). All other architectures = use smp_mb(), > > > > > > > > > which is transitive. > > > > > > > > >=20 > > > > > > > > > Did I miss anything? > > > > > > > >=20 > > > > > > > > I think that about covers it.. the only odd duckling might = be s390 which > > > > > > > > is documented as TSO but recently grew smp_mb__{before,afte= r}_atomic(), > > > > > > > > which seems to confuse matters. > > > > > > >=20 > > > > > > > Fair point, adding Martin and Heiko on CC for their thoughts. > > > > >=20 > > > > > Well we always had the full memory barrier for the various versio= ns of > > > > > smp_mb__xxx, they just have moved around and renamed several time= s. > > > > >=20 > > > > > After discussing this with Heiko we came to the conclusion that w= e can use > > > > > a simple barrier() for smp_mb__before_atomic() and smp_mb__after_= atomic(). > > > > >=20 > > > > > > > It looks like this applies to recent mainframes that have new= atomic > > > > > > > instructions, which, yes, might need something to make them w= ork with > > > > > > > fully transitive smp_load_acquire() and smp_store_release(). > > > > > > >=20 > > > > > > > Martin, Heiko, the question is whether or not the current s390 > > > > > > > smp_store_release() and smp_load_acquire() can be transitive. > > > > > > > For example, if all the Xi variables below are initially zero, > > > > > > > is it possible for all the r0, r1, r2, ... rN variables to > > > > > > > have the value 1 at the end of the test. > > > > > >=20 > > > > > > Right... This time actually adding Martin and Heiko on CC... > > > > > >=20 > > > > > > Thanx, Paul > > > > > >=20 > > > > > > > CPU 0 > > > > > > > r0 =3D smp_load_acquire(&X0); > > > > > > > smp_store_release(&X1, 1); > > > > > > >=20 > > > > > > > CPU 1 > > > > > > > r1 =3D smp_load_acquire(&X1); > > > > > > > smp_store_release(&X2, 1); > > > > > > >=20 > > > > > > > CPU 2 > > > > > > > r2 =3D smp_load_acquire(&X2); > > > > > > > smp_store_release(&X3, 1); > > > > > > >=20 > > > > > > > ... > > > > > > >=20 > > > > > > > CPU N > > > > > > > rN =3D smp_load_acquire(&XN); > > > > > > > smp_store_release(&X0, 1); > > > > > > >=20 > > > > > > > If smp_store_release() and smp_load_acquire() are transitive,= the > > > > > > > answer would be "no". > > > > >=20 > > > > > The answer is "no". Christian recently summarized what the princi= ples of > > > > > operation has to say about the CPU read / write behavior. If you = consider > > > > > the sequential order of instructions then > > > > >=20 > > > > > 1) reads are in order > > > > > 2) writes are in order > > > > > 3) reads can happen earlier > > > > > 4) writes can happen later > > > >=20 > > > > Correction. The principles of operation states this: > > > >=20 > > > > "A storage-operand store specified by one instruction appears to pr= ecede > > > > all storage-operand stores specified by conceptually subsequent ins= tructions, > > > > but it does not necessarily precede storage-operand fetches specifi= ed by > > > > conceptually subsequent instructions. However, a storage-operand st= ore > > > > appears to precede a conceptually subsequent storage-operand fetch = from the > > > > same main-storage location." > > > >=20 >=20 > Confused... Yeah, seems like I'm confused as well. This stuff always make my head hurt.. =20 > IIUC, the previous paragraph actually means that a STORE-LOAD pair can be > reordered. But the below reasoning is saying that a LOAD-STORE pair can > be reordered. Do I miss something here? True, the above paragraph allows a store to move past a load and not the ot= her way around. > > > > As observed by other CPUs a write to one memory location can "overt= ake" a > > > > read of another memory location if there is no explicit memory-barr= ier > > > > between the load and the store instruction. > > > >=20 > > > > In the above example X0, X1, ... XN are different memory locations,= so > > > > architecturally the answer is "yes", all r0, r1, ... rN variables c= an have > > > > the value of 1 after the test. I doubt that any existing machine wi= ll > > > > show this behavior though. > > > >=20 > > >=20 > > > Just be curious, how about when N =3D=3D 1? The test then becomes: > > >=20 > > > CPU 0 > > > r0 =3D smp_load_acquire(&X0); > > > smp_store_release(&X1,1); > > >=20 > > > CPU 1 > > > r1 =3D smp_load_acquire(&X1); > > > smp_store_release(&X0,1); > > >=20 > > > Is it possible that r0 =3D=3D 1 and r1 =3D=3D 1 at the end, due to th= e same > > > reason? > >=20 > > Yes, that is possible for the same reason. To change that we would have > > to replace the barrier() in smp_load_acquire/smp_store_release with > > smp_mb(). > >=20 >=20 > I thought that s390 is TSO, so this is prohibitted. If that is possible, > I think, that means the current implementation of smp_load_acquire and > smp_store_release on s390 is incorrect... Ok, further reading of chapter 5 of the principles revealed this: "As observed by other CPUs and by channel programs, storage-operand fetches associated with one instruction execution appear to precede all storage operand references for conceptually subsequent instructions." So no writes before reads. Correction to the correction: all r0, r1, ...rN equal to one can not happen after all. Got me worried there ;-) --=20 blue skies, Martin. "Reality continues to ruin my life." - Calvin. --Sig_/07j6Dk+T_Ds+sAaYRXvX2Bk Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJWAWakAAoJELvpdr8mrl3SbQcQALIbNk8GadD+T3StBsst1BdC kNLftqC0HYgWrfrNKNPXvDJlw/oC2wnJYG0JKDiPjuL5UOV1s1ir2bLnT8Tk+Ldb WjKQM5IwRbU6OUBn67W7ClsWhNaFB7oOTJnpJpWqUPqPnGTbwihcbKmUigFaJT2k xQiH22pGbIRfzmScDChcoe7ivWirnxr4BLTNe66m32OBodractKVzbN9ZpgC06Mb Dx1SQ+gxKvS5spMhlHJE1PcA+jBGIGH7+NFTxAgrbziY35FRNxKa8Hy6/xkTzVsw Zpzdtav5bUMOhmf9QCoNqAB3CfGt5OkFieijkRmcSM9N1SZXaAzhNWFQR/EENuJE TZJk73OHog+me7GGOTD8/cCSBEfRDCLCVpfrahHY+Sx3kHVzaYMUBgy87H1f9cf9 k10jQJ5yeXZfb/a7jWoje94hSiO10FCg/B0sIGXpzNNrb8aisxm/MUT+Mj5dAxT1 AOhNjRNmnbTcmFTu/cSL24LJX0qvlY3m0jeQCo4TbSG0gFlFkZPcPYmEKin3uLWl cAUp+6umrGCxE5ZYBMHe6vDBtffu5v5yKN9o7fYSqwBs9MdLWiZRsCu65MFz9KsL 0Z97Do8+0X59Mg7HwwrVzpbViXKjXzr+npqINmDg444j6iOe/B2w5HoITb0sDijO I4B6SKmjF1KllzdeDST4 =YzUF -----END PGP SIGNATURE----- --Sig_/07j6Dk+T_Ds+sAaYRXvX2Bk--