From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757843AbbIVMXx (ORCPT ); Tue, 22 Sep 2015 08:23:53 -0400 Received: from mail-ob0-f182.google.com ([209.85.214.182]:34262 "EHLO mail-ob0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751591AbbIVMXv (ORCPT ); Tue, 22 Sep 2015 08:23:51 -0400 Date: Tue, 22 Sep 2015 20:23:26 +0800 From: Boqun Feng To: Martin Schwidefsky Cc: paulmck@linux.vnet.ibm.com, Peter Zijlstra , Davidlohr Bueso , Ingo Molnar , Thomas Gleixner , linux-kernel@vger.kernel.org, Davidlohr Bueso , heiko.carstens@de.ibm.com Subject: Re: [PATCH -tip 2/3] sched/wake_q: Relax to acquire semantics Message-ID: <20150922122326.GA1032@fixme-laptop.cn.ibm.com> References: <20150915124142.GF4029@linux.vnet.ibm.com> <20150915124800.GB16853@twins.programming.kicks-ass.net> <20150915140922.GG4029@linux.vnet.ibm.com> <20150915141439.GE16853@twins.programming.kicks-ass.net> <20150915153448.GI4029@linux.vnet.ibm.com> <20150915163028.GG16853@twins.programming.kicks-ass.net> <20150915170941.GL4029@linux.vnet.ibm.com> <20150918214120.GA4405@linux.vnet.ibm.com> <20150921112252.3c2937e1@mschwide> <20150922122735.14f3c573@mschwide> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="RnlQjJ0d97Da+TV1" Content-Disposition: inline In-Reply-To: <20150922122735.14f3c573@mschwide> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --RnlQjJ0d97Da+TV1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Martin, On Tue, Sep 22, 2015 at 12:27:35PM +0200, Martin Schwidefsky wrote: > On Mon, 21 Sep 2015 11:22:52 +0200 > Martin Schwidefsky wrote: >=20 > > On Fri, 18 Sep 2015 14:41:20 -0700 > > "Paul E. McKenney" wrote: > >=20 > > > On Tue, Sep 15, 2015 at 10:09:41AM -0700, Paul E. McKenney wrote: > > > > On Tue, Sep 15, 2015 at 06:30:28PM +0200, Peter Zijlstra wrote: > > > > > On Tue, Sep 15, 2015 at 08:34:48AM -0700, Paul E. McKenney wrote: > > > > > > On Tue, Sep 15, 2015 at 04:14:39PM +0200, Peter Zijlstra wrote: > > > > > > > On Tue, Sep 15, 2015 at 07:09:22AM -0700, Paul E. McKenney wr= ote: > > > > > > > > On Tue, Sep 15, 2015 at 02:48:00PM +0200, Peter Zijlstra wr= ote: > > > > > > > > > On Tue, Sep 15, 2015 at 05:41:42AM -0700, Paul E. McKenne= y wrote: > > > > > > > > > > > Never mind, the PPC people will implement this with l= wsync and that is > > > > > > > > > > > very much not transitive IIRC. > > > > > > > > > >=20 > > > > > > > > > > I am probably lost on context, but... > > > > > > > > > >=20 > > > > > > > > > > It turns out that lwsync is transitive in special cases= =2E One of them > > > > > > > > > > is a series of release-acquire pairs, which can extend = indefinitely. > > > > > > > > > >=20 > > > > > > > > > > Does that help in this case? > > > > > > > > >=20 > > > > > > > > > Probably not, but good to know. I still don't think we wa= nt to rely on > > > > > > > > > ACQUIRE/RELEASE being transitive in general though. > > > > > > > >=20 > > > > > > > > OK, I will bite... Why not? > > > > > > >=20 > > > > > > > It would mean us reviewing all archs (again) and documenting = it I > > > > > > > suppose. Which is of course entirely possible. > > > > > > >=20 > > > > > > > That said, I don't think the case at hand requires it, so let= s postpone > > > > > > > this for now ;-) > > > > > >=20 > > > > > > True enough, but in my experience smp_store_release() and > > > > > > smp_load_acquire() are a -lot- easier to use than other barrier= s, > > > > > > and transitivity will help promote their use. So... > > > > > >=20 > > > > > > All the TSO architectures (x86, s390, SPARC, HPPA, ...) support= transitive > > > > > > smp_store_release()/smp_load_acquire() via their native orderin= g in > > > > > > combination with barrier() macros. x86 with CONFIG_X86_PPRO_FE= NCE=3Dy, > > > > > > which is not TSO, uses an mfence instruction. Power supports t= his via > > > > > > lwsync's partial cumulativity. ARM64 supports it in SMP via th= e new ldar > > > > > > and stlr instructions (in non-SMP, it uses barrier(), which suf= fices > > > > > > in that case). IA64 supports this via total ordering of all re= lease > > > > > > instructions in theory and by the actual full-barrier implement= ation > > > > > > in practice (and the fact that gcc emits st.rel and ld.acq inst= ructions > > > > > > for volatile stores and loads). All other architectures use sm= p_mb(), > > > > > > which is transitive. > > > > > >=20 > > > > > > Did I miss anything? > > > > >=20 > > > > > I think that about covers it.. the only odd duckling might be s39= 0 which > > > > > is documented as TSO but recently grew smp_mb__{before,after}_ato= mic(), > > > > > which seems to confuse matters. > > > >=20 > > > > Fair point, adding Martin and Heiko on CC for their thoughts. > >=20 > > Well we always had the full memory barrier for the various versions of > > smp_mb__xxx, they just have moved around and renamed several times. > >=20 > > After discussing this with Heiko we came to the conclusion that we can = use > > a simple barrier() for smp_mb__before_atomic() and smp_mb__after_atomic= (). > >=20 > > > > It looks like this applies to recent mainframes that have new atomic > > > > instructions, which, yes, might need something to make them work wi= th > > > > fully transitive smp_load_acquire() and smp_store_release(). > > > >=20 > > > > Martin, Heiko, the question is whether or not the current s390 > > > > smp_store_release() and smp_load_acquire() can be transitive. > > > > For example, if all the Xi variables below are initially zero, > > > > is it possible for all the r0, r1, r2, ... rN variables to > > > > have the value 1 at the end of the test. > > >=20 > > > Right... This time actually adding Martin and Heiko on CC... > > >=20 > > > Thanx, Paul > > >=20 > > > > CPU 0 > > > > r0 =3D smp_load_acquire(&X0); > > > > smp_store_release(&X1, 1); > > > >=20 > > > > CPU 1 > > > > r1 =3D smp_load_acquire(&X1); > > > > smp_store_release(&X2, 1); > > > >=20 > > > > CPU 2 > > > > r2 =3D smp_load_acquire(&X2); > > > > smp_store_release(&X3, 1); > > > >=20 > > > > ... > > > >=20 > > > > CPU N > > > > rN =3D smp_load_acquire(&XN); > > > > smp_store_release(&X0, 1); > > > >=20 > > > > If smp_store_release() and smp_load_acquire() are transitive, the > > > > answer would be "no". > >=20 > > The answer is "no". Christian recently summarized what the principles of > > operation has to say about the CPU read / write behavior. If you consid= er > > the sequential order of instructions then > >=20 > > 1) reads are in order > > 2) writes are in order > > 3) reads can happen earlier > > 4) writes can happen later >=20 > Correction. The principles of operation states this: >=20 > "A storage-operand store specified by one instruction appears to precede > all storage-operand stores specified by conceptually subsequent instructi= ons, > but it does not necessarily precede storage-operand fetches specified by > conceptually subsequent instructions. However, a storage-operand store > appears to precede a conceptually subsequent storage-operand fetch from t= he > same main-storage location." >=20 > As observed by other CPUs a write to one memory location can "overtake" a > read of another memory location if there is no explicit memory-barrier > between the load and the store instruction. >=20 > In the above example X0, X1, ... XN are different memory locations, so > architecturally the answer is "yes", all r0, r1, ... rN variables can have > the value of 1 after the test. I doubt that any existing machine will > show this behavior though. >=20 Just be curious, how about when N =3D=3D 1? The test then becomes: CPU 0 r0 =3D smp_load_acquire(&X0); smp_store_release(&X1,1); CPU 1 r1 =3D smp_load_acquire(&X1); smp_store_release(&X0,1); Is it possible that r0 =3D=3D 1 and r1 =3D=3D 1 at the end, due to the same reason? Regards, Boqun > > > > A similar litmus test involving atomics would be as follows, again > > > > with all Xi initially zero: > > > >=20 > > > > CPU 0 > > > > atomic_inc(&X0); > > > > smp_store_release(&X1, 1); > > > >=20 > > > > CPU 1 > > > > r1 =3D smp_load_acquire(&X1); > > > > smp_store_release(&X2, 1); > > > >=20 > > > > CPU 2 > > > > r2 =3D smp_load_acquire(&X2); > > > > smp_store_release(&X3, 1); > > > >=20 > > > > ... > > > >=20 > > > > CPU N > > > > rN =3D smp_load_acquire(&XN); > > > > r0 =3D atomic_read(&X0); > > > >=20 > > > > Here, the question is whether r0 can be zero, but r1, r2, ... rN all > > > > being 1 at the end of the test. > >=20 > > r0 =3D 0 and all r1, r2, ... rN =3D 1 can not happen on s390. > =20 > This indeed can not happen as the atomic_inc is a store type reference > which precedes the smp_store_release store type reference. >=20 > --=20 > blue skies, > Martin. >=20 > "Reality continues to ruin my life." - Calvin. >=20 > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ --RnlQjJ0d97Da+TV1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAABCAAGBQJWAUg6AAoJEEl56MO1B/q4wGYH/idrB/2RYOElRolUbiGfsQp1 SnaYIMpxelsqOBiQpPQxkDauhSbF2k/AxoXES3ZD/C7yPzf1i3K9iVJN3X/PwbQS Ew0ugnfb9JP91qBEmHbCCTx+cE90fSMB3Nfp/INDouPej1uc2Xwpzj29kgFzDl+9 Afp+5D3FlCf2t7/G7UhrhkqVW9HA7P1mmdZyv4VUPJORzwq/lCAYlvDFMCKUhSu5 q674YuhQZHyp8ljnQf9oW/39swfMDEEL59ENZ96JrnWFrxeNs+FFfG3PArx0zQ80 73hx39eTihJ4Nw+BFEiYF70/4Udj+rVeo3Yp0yFbb61ofgyo5NGPHUahsW/Ywss= =isjf -----END PGP SIGNATURE----- --RnlQjJ0d97Da+TV1--