From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
To: "Toke Høiland-Jørgensen" <toke@redhat.com>
Cc: bpf <bpf@vger.kernel.org>
Subject: Re: Selftest failures related to kern_sync_rcu()
Date: Tue, 13 Apr 2021 14:43:24 -0700 [thread overview]
Message-ID: <CAEf4Bzahxw5-KTb2yOk8PHQmEyc6gDgTTR6znZjH2OhZ66wiUw@mail.gmail.com> (raw)
In-Reply-To: <87im4qo9ey.fsf@toke.dk>
On Tue, Apr 13, 2021 at 1:50 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:
>
> > On Thu, Apr 8, 2021 at 12:34 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> >>
> >> Hi Andrii
> >>
> >> I'm getting some selftest failures that all seem to have something to do
> >> with kern_sync_rcu() not being enough to trigger the kernel events that
> >> the selftest expects:
> >>
> >> $ ./test_progs | grep FAIL
> >> test_lookup_update:FAIL:map1_leak inner_map1 leaked!
> >> #15/1 lookup_update:FAIL
> >> #15 btf_map_in_map:FAIL
> >> test_exit_creds:FAIL:null_ptr_count unexpected null_ptr_count: actual 0 == expected 0
> >> #123/2 exit_creds:FAIL
> >> #123 task_local_storage:FAIL
> >> test_exit_creds:FAIL:null_ptr_count unexpected null_ptr_count: actual 0 == expected 0
> >> #123/2 exit_creds:FAIL
> >> #123 task_local_storage:FAIL
> >>
> >> They are all fixed by adding a sleep(1) after the call(s) to
> >> kern_sync_rcu(), so I'm guessing it's some kind of
> >> timing/synchronisation problem. Is there a particular kernel config
> >> that's needed for the membarrier syscall trick to work? I've tried with
> >> various settings of PREEMPT and that doesn't really seem to make any
> >> difference...
> >>
> >
> > If you check kern_sync_rcu(), it relies on membarrier() syscall
> > (passing cmd = MEMBARRIER_CMD_SHARED == MEMBARRIER_CMD_GLOBAL).
> > Now, looking at kernel sources:
> > - CONFIG_MEMBARRIER should be enabled for that syscall;
> > - it has some extra conditions:
> >
> > case MEMBARRIER_CMD_GLOBAL:
> > /* MEMBARRIER_CMD_GLOBAL is not compatible with nohz_full. */
> > if (tick_nohz_full_enabled())
> > return -EINVAL;
> > if (num_online_cpus() > 1)
> > synchronize_rcu();
> > return 0;
> >
> > Could it be that one of those conditions is not satisfied?
>
> Aha, bingo! Found the membarrier syscall stuff, but for some reason
> didn't think to actually read the code of it; and I was running this in
> a VM with a single CPU, adding another fixed this. Thanks! :)
>
> Do you think we could detect this in the tests? I suppose the
> tick_nohz_full_enabled() check should already result in a visible
> failure since that makes the syscall fail; but the CPU thing is silent,
> so it would be nice with a hint. Could kern_sync_rcu() check the CPU
> count and print a warning or fail if it is 1? Or maybe just straight up
> fall back to sleep()'ing?
If membarrier() is unreliable, I guess we can just go back to the
previous way of triggering synchronize_rcu() (create and update
map-in-map element)? See 635599bace25 ("selftests/bpf: Sync RCU before
unloading bpf_testmod") that removed that in favor of membarrier()
syscall.
>
> -Toke
>
next prev parent reply other threads:[~2021-04-13 21:43 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-08 19:34 Selftest failures related to kern_sync_rcu() Toke Høiland-Jørgensen
2021-04-13 3:38 ` Andrii Nakryiko
2021-04-13 8:50 ` Toke Høiland-Jørgensen
2021-04-13 21:43 ` Andrii Nakryiko [this message]
2021-04-14 15:54 ` Alexei Starovoitov
2021-04-14 17:52 ` Paul E. McKenney
2021-04-14 17:59 ` Alexei Starovoitov
2021-04-14 18:19 ` Paul E. McKenney
2021-04-14 18:39 ` Toke Høiland-Jørgensen
2021-04-14 18:41 ` Paul E. McKenney
2021-04-14 19:18 ` Toke Høiland-Jørgensen
2021-04-14 21:25 ` Paul E. McKenney
2021-04-14 22:13 ` Andrii Nakryiko
2021-04-14 22:27 ` Paul E. McKenney
2021-04-14 22:47 ` Toke Høiland-Jørgensen
2021-04-14 18:27 ` Toke Høiland-Jørgensen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAEf4Bzahxw5-KTb2yOk8PHQmEyc6gDgTTR6znZjH2OhZ66wiUw@mail.gmail.com \
--to=andrii.nakryiko@gmail.com \
--cc=bpf@vger.kernel.org \
--cc=toke@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.