* Still getting Deadlocks with condition variables
@ 2020-06-05 14:36 Lange Norbert
2020-06-05 15:40 ` Jan Kiszka
2020-06-07 20:15 ` Philippe Gerum
0 siblings, 2 replies; 13+ messages in thread
From: Lange Norbert @ 2020-06-05 14:36 UTC (permalink / raw
To: Xenomai (xenomai@xenomai.org)
Hello,
I brought this up once or twice at this ML [1], I am still getting some occasional lockups. Now the first time without running under an debugger,
Harwdare is a TQMxE39M (Goldmont Atom)
Kernel: 4.19.124-cip27-xeno12-static x86_64
I-pipe Version: 12
Xenomai Version: 3.1
Glibc Version 2.28
What happens (as far as I understand it):
The setup is an project with several cobalt threads (no "native" Linux thread as far as I can tell, apart maybe from the cobalt's printf thread).
They mostly sleep, and are triggered if work is available, the project also can load DSOs (specialized maths) during configuration stage - during this stages is when the exceptions occur
1. Linux Thread LWP 682 calls SYS_futex "wake"
Code immediately before syscall, file x86_64/lowlevellock.S:
movl$0, (%rdi)
LOAD_FUTEX_WAKE (%esi)
movl$1, %edx/* Wake one thread. */
movl$SYS_futex, %eax
syscall
2. Xenomai switches a cobalt thread to secondary, potentially because all threads are in primary:
Jun 05 12:35:19 buildroot kernel: [Xenomai] switching dispatcher to secondary mode after exception #14 from user-space at 0x7fd731299115 (pid 681)
Note that most threads are stuck waiting for a condvar in sc_cobalt_cond_wait_prologue (cond.c:313), LWP 681 is at the next instruction.
3. Xenomai gets XCPU signal -> coredump
gdb) thread apply all bt 3
Thread 9 (LWP 682):
#0 __lll_unlock_wake () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
#1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt (mutex=0x7fd7312f6968 <_rtld_global+2312>, decr=1) at pthread_mutex_unlock.c:54
#2 0x00007fd7312e0442 in ?? () from /home/lano/Downloads/bugcrash/lib64/ld-linux-x86-64.so.2
#3 0x00007fd7312c72ac in ?? () from /lib/libdl.so.2
#4 0x00007fd73104211f in _dl_catch_exception () from /lib/libc.so.6
#5 0x00007fd731042190 in _dl_catch_error () from /lib/libc.so.6
#6 0x00007fd7312c7975 in ?? () from /lib/libdl.so.2
#7 0x00007fd7312c7327 in dlopen () from /lib/libdl.so.2
(More stack frames follow...)
Thread 8 (LWP 686):
#0 0x00007fd731298d48 in __cobalt_clock_nanosleep (clock_id=0, flags=0, rqtp=0x7fd727e3ad10, rmtp=0x0) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:312
#1 0x00007fd731298d81 in __cobalt_nanosleep (rqtp=<optimized out>, rmtp=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:354
#2 0x0000000000434590 in operator() (__closure=0x7fd720006fb8) at ../../acpu.runner/asim/asim_com.cpp:685
(More stack frames follow...)
Thread 7 (LWP 677):
#0 0x00007fd73127b6c6 in __GI___nanosleep (requested_time=requested_time@entry=0x7fd7312b1fb0 <syncdelay>, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x00007fd73129b746 in printer_loop (arg=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/printf.c:635
#2 0x00007fd7312720f7 in start_thread (arg=<optimized out>) at pthread_create.c:486
(More stack frames follow...)
Thread 6 (LWP 685):
#0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f269660, mutex=0x7fd72f269630) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
#1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f269660, pMutex=0x7fd72f269630) at ../../alib/src/alib/posix/conditionvar.c:66
#2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f269660, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
(More stack frames follow...)
Thread 5 (LWP 684):
#0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f267790, mutex=0x7fd72f267760) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
#1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f267790, pMutex=0x7fd72f267760) at ../../alib/src/alib/posix/conditionvar.c:66
#2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f267790, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
(More stack frames follow...)
Thread 4 (LWP 680):
#0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, mutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
#1 0x000000000046377c in conditionvar_wait (pData=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, pMutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/conditionvar.c:66
#2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
(More stack frames follow...)
Thread 3 (LWP 683):
#0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f2658c0, mutex=0x7fd72f265890) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
#1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f2658c0, pMutex=0x7fd72f265890) at ../../alib/src/alib/posix/conditionvar.c:66
#2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f2658c0, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
(More stack frames follow...)
Thread 2 (LWP 675):
#0 0x00007fd73129aea4 in __cobalt_pthread_mutex_lock (mutex=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/mutex.c:375
#1 0x000000000046390a in mutex_lock (pData=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/mutex.c:33
#2 0x000000000040a530 in HIPASE::Posix::CAlib_Mutex::lock (this=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/include/alib/alib_mutex_posix.h:67
(More stack frames follow...)
Thread 1 (LWP 681):
#0 __cobalt_pthread_cond_wait (cond=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, mutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:316
#1 0x000000000046377c in conditionvar_wait (pData=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, pMutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/conditionvar.c:66
#2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
(More stack frames follow...)
[1] - https://xenomai.org/pipermail/xenomai/2020-January/042299.html
Mit besten Grüßen / Kind regards
NORBERT LANGE
AT-RD3
ANDRITZ HYDRO GmbH
Eibesbrunnergasse 20
1120 Vienna / AUSTRIA
p: +43 50805 56684
norbert.lange@andritz.com
andritz.com
________________________________
This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
ANDRITZ HYDRO GmbH
Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
Firmensitz/ Registered seat: Wien
Firmenbuchgericht/ Court of registry: Handelsgericht Wien
Firmenbuchnummer/ Company registration: FN 61833 g
DVR: 0605077
UID-Nr.: ATU14756806
Thank You
________________________________
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Still getting Deadlocks with condition variables
2020-06-05 14:36 Still getting Deadlocks with condition variables Lange Norbert
@ 2020-06-05 15:40 ` Jan Kiszka
2020-06-08 9:48 ` Lange Norbert
2020-06-07 20:15 ` Philippe Gerum
1 sibling, 1 reply; 13+ messages in thread
From: Jan Kiszka @ 2020-06-05 15:40 UTC (permalink / raw
To: Lange Norbert, Xenomai (xenomai@xenomai.org)
On 05.06.20 16:36, Lange Norbert via Xenomai wrote:
> Hello,
>
> I brought this up once or twice at this ML [1], I am still getting some occasional lockups. Now the first time without running under an debugger,
>
> Harwdare is a TQMxE39M (Goldmont Atom)
> Kernel: 4.19.124-cip27-xeno12-static x86_64
> I-pipe Version: 12
> Xenomai Version: 3.1
> Glibc Version 2.28
>
> What happens (as far as I understand it):
>
> The setup is an project with several cobalt threads (no "native" Linux thread as far as I can tell, apart maybe from the cobalt's printf thread).
> They mostly sleep, and are triggered if work is available, the project also can load DSOs (specialized maths) during configuration stage - during this stages is when the exceptions occur
>
>
> 1. Linux Thread LWP 682 calls SYS_futex "wake"
>
> Code immediately before syscall, file x86_64/lowlevellock.S:
> movl$0, (%rdi)
> LOAD_FUTEX_WAKE (%esi)
> movl$1, %edx/* Wake one thread. */
> movl$SYS_futex, %eax
> syscall
>
> 2. Xenomai switches a cobalt thread to secondary, potentially because all threads are in primary:
>
> Jun 05 12:35:19 buildroot kernel: [Xenomai] switching dispatcher to secondary mode after exception #14 from user-space at 0x7fd731299115 (pid 681)
#14 mean page fault, fixable or real. What is at that address? What
address was accessed by that instruction?
>
> Note that most threads are stuck waiting for a condvar in sc_cobalt_cond_wait_prologue (cond.c:313), LWP 681 is at the next instruction.
>
Stuck at what? Waiting for the condvar itsself or getting the enclosing
mutex again? What are the states of the involved synchonization objects?
Jan
> 3. Xenomai gets XCPU signal -> coredump
>
> gdb) thread apply all bt 3
>
> Thread 9 (LWP 682):
> #0 __lll_unlock_wake () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
> #1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt (mutex=0x7fd7312f6968 <_rtld_global+2312>, decr=1) at pthread_mutex_unlock.c:54
> #2 0x00007fd7312e0442 in ?? () from /home/lano/Downloads/bugcrash/lib64/ld-linux-x86-64.so.2
> #3 0x00007fd7312c72ac in ?? () from /lib/libdl.so.2
> #4 0x00007fd73104211f in _dl_catch_exception () from /lib/libc.so.6
> #5 0x00007fd731042190 in _dl_catch_error () from /lib/libc.so.6
> #6 0x00007fd7312c7975 in ?? () from /lib/libdl.so.2
> #7 0x00007fd7312c7327 in dlopen () from /lib/libdl.so.2
> (More stack frames follow...)
>
> Thread 8 (LWP 686):
> #0 0x00007fd731298d48 in __cobalt_clock_nanosleep (clock_id=0, flags=0, rqtp=0x7fd727e3ad10, rmtp=0x0) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:312
> #1 0x00007fd731298d81 in __cobalt_nanosleep (rqtp=<optimized out>, rmtp=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:354
> #2 0x0000000000434590 in operator() (__closure=0x7fd720006fb8) at ../../acpu.runner/asim/asim_com.cpp:685
> (More stack frames follow...)
>
> Thread 7 (LWP 677):
> #0 0x00007fd73127b6c6 in __GI___nanosleep (requested_time=requested_time@entry=0x7fd7312b1fb0 <syncdelay>, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
> #1 0x00007fd73129b746 in printer_loop (arg=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/printf.c:635
> #2 0x00007fd7312720f7 in start_thread (arg=<optimized out>) at pthread_create.c:486
> (More stack frames follow...)
>
> Thread 6 (LWP 685):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f269660, mutex=0x7fd72f269630) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f269660, pMutex=0x7fd72f269630) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f269660, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 5 (LWP 684):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f267790, mutex=0x7fd72f267760) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f267790, pMutex=0x7fd72f267760) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f267790, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 4 (LWP 680):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, mutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, pMutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 3 (LWP 683):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f2658c0, mutex=0x7fd72f265890) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f2658c0, pMutex=0x7fd72f265890) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f2658c0, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 2 (LWP 675):
> #0 0x00007fd73129aea4 in __cobalt_pthread_mutex_lock (mutex=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/mutex.c:375
> #1 0x000000000046390a in mutex_lock (pData=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/mutex.c:33
> #2 0x000000000040a530 in HIPASE::Posix::CAlib_Mutex::lock (this=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/include/alib/alib_mutex_posix.h:67
> (More stack frames follow...)
>
> Thread 1 (LWP 681):
> #0 __cobalt_pthread_cond_wait (cond=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, mutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:316
> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, pMutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
>
>
> [1] - https://xenomai.org/pipermail/xenomai/2020-January/042299.html
>
>
> Mit besten Grüßen / Kind regards
>
> NORBERT LANGE
>
> AT-RD3
>
> ANDRITZ HYDRO GmbH
> Eibesbrunnergasse 20
> 1120 Vienna / AUSTRIA
> p: +43 50805 56684
> norbert.lange@andritz.com
> andritz.com
>
> ________________________________
>
> This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
>
> ANDRITZ HYDRO GmbH
>
>
> Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
>
> Firmensitz/ Registered seat: Wien
>
> Firmenbuchgericht/ Court of registry: Handelsgericht Wien
>
> Firmenbuchnummer/ Company registration: FN 61833 g
>
> DVR: 0605077
>
> UID-Nr.: ATU14756806
>
>
> Thank You
> ________________________________
>
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Still getting Deadlocks with condition variables
2020-06-05 14:36 Still getting Deadlocks with condition variables Lange Norbert
2020-06-05 15:40 ` Jan Kiszka
@ 2020-06-07 20:15 ` Philippe Gerum
2020-06-08 10:08 ` Lange Norbert
1 sibling, 1 reply; 13+ messages in thread
From: Philippe Gerum @ 2020-06-07 20:15 UTC (permalink / raw
To: Lange Norbert, Xenomai (xenomai@xenomai.org)
On 6/5/20 4:36 PM, Lange Norbert wrote:
> Hello,
>
> I brought this up once or twice at this ML [1], I am still getting some occasional lockups. Now the first time without running under an debugger,
>
> Harwdare is a TQMxE39M (Goldmont Atom)
> Kernel: 4.19.124-cip27-xeno12-static x86_64
> I-pipe Version: 12
> Xenomai Version: 3.1
> Glibc Version 2.28
>
> What happens (as far as I understand it):
>
> The setup is an project with several cobalt threads (no "native" Linux thread as far as I can tell, apart maybe from the cobalt's printf thread).
> They mostly sleep, and are triggered if work is available, the project also can load DSOs (specialized maths) during configuration stage - during this stages is when the exceptions occur
>
>
> 1. Linux Thread LWP 682 calls SYS_futex "wake"
>
> Code immediately before syscall, file x86_64/lowlevellock.S:
> movl$0, (%rdi)
> LOAD_FUTEX_WAKE (%esi)
> movl$1, %edx/* Wake one thread. */
> movl$SYS_futex, %eax
> syscall
>
> 2. Xenomai switches a cobalt thread to secondary, potentially because all threads are in primary:
>
> Jun 05 12:35:19 buildroot kernel: [Xenomai] switching dispatcher to secondary mode after exception #14 from user-space at 0x7fd731299115 (pid 681)
>
This kernel message tells a different story, thread pid 681 received a #PF,
maybe due to accessing its own stack (cond.c, line 316). This may be a minor
fault though, nothing invalid. Such fault is not supposed to occur for Xenomai
threads on x86, but that would be another issue. Code-wise, I'm referring to
the current state of the master branch for lib/cobalt/cond.c, which seems to
match your description.
> Note that most threads are stuck waiting for a condvar in sc_cobalt_cond_wait_prologue (cond.c:313), LWP 681 is at the next instruction.
>
> 3. Xenomai gets XCPU signal -> coredump
>
More precisely, Xenomai is likely sending this signal to your application,
since it had to switch pid 681 to secondary mode for fixing up the #PF event.
You may have set PTHREAD_WARNSW with pthread_setmode_np() for that thread.
> gdb) thread apply all bt 3
>
> Thread 9 (LWP 682):
> #0 __lll_unlock_wake () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
> #1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt (mutex=0x7fd7312f6968 <_rtld_global+2312>, decr=1) at pthread_mutex_unlock.c:54
> #2 0x00007fd7312e0442 in ?? () from /home/lano/Downloads/bugcrash/lib64/ld-linux-x86-64.so.2
> #3 0x00007fd7312c72ac in ?? () from /lib/libdl.so.2
> #4 0x00007fd73104211f in _dl_catch_exception () from /lib/libc.so.6
> #5 0x00007fd731042190 in _dl_catch_error () from /lib/libc.so.6
> #6 0x00007fd7312c7975 in ?? () from /lib/libdl.so.2
> #7 0x00007fd7312c7327 in dlopen () from /lib/libdl.so.2
> (More stack frames follow...)
>
> Thread 8 (LWP 686):
> #0 0x00007fd731298d48 in __cobalt_clock_nanosleep (clock_id=0, flags=0, rqtp=0x7fd727e3ad10, rmtp=0x0) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:312
> #1 0x00007fd731298d81 in __cobalt_nanosleep (rqtp=<optimized out>, rmtp=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:354
> #2 0x0000000000434590 in operator() (__closure=0x7fd720006fb8) at ../../acpu.runner/asim/asim_com.cpp:685
> (More stack frames follow...)
>
> Thread 7 (LWP 677):
> #0 0x00007fd73127b6c6 in __GI___nanosleep (requested_time=requested_time@entry=0x7fd7312b1fb0 <syncdelay>, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
> #1 0x00007fd73129b746 in printer_loop (arg=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/printf.c:635
> #2 0x00007fd7312720f7 in start_thread (arg=<optimized out>) at pthread_create.c:486
> (More stack frames follow...)
>
> Thread 6 (LWP 685):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f269660, mutex=0x7fd72f269630) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f269660, pMutex=0x7fd72f269630) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f269660, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 5 (LWP 684):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f267790, mutex=0x7fd72f267760) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f267790, pMutex=0x7fd72f267760) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f267790, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 4 (LWP 680):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, mutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, pMutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 3 (LWP 683):
> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0x7fd72f2658c0, mutex=0x7fd72f265890) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f2658c0, pMutex=0x7fd72f265890) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0x7fd72f2658c0, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
> Thread 2 (LWP 675):
> #0 0x00007fd73129aea4 in __cobalt_pthread_mutex_lock (mutex=<optimized out>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/mutex.c:375
> #1 0x000000000046390a in mutex_lock (pData=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/mutex.c:33
> #2 0x000000000040a530 in HIPASE::Posix::CAlib_Mutex::lock (this=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/include/alib/alib_mutex_posix.h:67
> (More stack frames follow...)
>
> Thread 1 (LWP 681):
> #0 __cobalt_pthread_cond_wait (cond=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, mutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:316
> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, pMutex=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/conditionvar.c:66
> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait (this=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>, lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> (More stack frames follow...)
>
>
>
> [1] - https://xenomai.org/pipermail/xenomai/2020-January/042299.html
>
You refer to an older post describing a lockup, but this post describes an
application crashing with a core dump. What made you draw the conclusion that
the same bug would be at work? Also, could you give some details regarding the
following:
- what do you mean by 'lockup' in this case? Can you still access the board or
is there some runaway real-time code locking out everything else when this
happens? My understanding is that this is no hard lock up otherwise the
watchdog would have triggered. If this is a softer kind of lockup instead,
what does /proc/xenomai/sched/stat tell you about the thread states after the
problem occurred?
- did you determine that using the dynamic linker is required to trigger the
bug yet? Or could you observe it without such interaction with dl?
- what is the typical size of your Xenomai thread stack? It defaults to 64k
min with Xenomai 3.1.
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Still getting Deadlocks with condition variables
2020-06-05 15:40 ` Jan Kiszka
@ 2020-06-08 9:48 ` Lange Norbert
2020-06-08 10:09 ` Jan Kiszka
0 siblings, 1 reply; 13+ messages in thread
From: Lange Norbert @ 2020-06-08 9:48 UTC (permalink / raw
To: Jan Kiszka, Xenomai (xenomai@xenomai.org)
> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Freitag, 5. Juni 2020 17:40
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: Still getting Deadlocks with condition variables
>
> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
>
>
> On 05.06.20 16:36, Lange Norbert via Xenomai wrote:
> > Hello,
> >
> > I brought this up once or twice at this ML [1], I am still getting
> > some occasional lockups. Now the first time without running under an
> > debugger,
> >
> > Harwdare is a TQMxE39M (Goldmont Atom)
> > Kernel: 4.19.124-cip27-xeno12-static x86_64 I-pipe Version: 12 Xenomai
> > Version: 3.1 Glibc Version 2.28
> >
> > What happens (as far as I understand it):
> >
> > The setup is an project with several cobalt threads (no "native" Linux
> thread as far as I can tell, apart maybe from the cobalt's printf thread).
> > They mostly sleep, and are triggered if work is available, the project
> > also can load DSOs (specialized maths) during configuration stage -
> > during this stages is when the exceptions occur
> >
> >
> > 1. Linux Thread LWP 682 calls SYS_futex "wake"
> >
> > Code immediately before syscall, file x86_64/lowlevellock.S:
> > movl$0, (%rdi)
> > LOAD_FUTEX_WAKE (%esi)
> > movl$1, %edx/* Wake one thread. */
> > movl$SYS_futex, %eax
> > syscall
> >
> > 2. Xenomai switches a cobalt thread to secondary, potentially because all
> threads are in primary:
> >
> > Jun 05 12:35:19 buildroot kernel: [Xenomai] switching dispatcher to
> > secondary mode after exception #14 from user-space at 0x7fd731299115
> > (pid 681)
>
> #14 mean page fault, fixable or real. What is at that address? What address
> was accessed by that instruction?
>
> >
> > Note that most threads are stuck waiting for a condvar in
> sc_cobalt_cond_wait_prologue (cond.c:313), LWP 681 is at the next
> instruction.
> >
>
> Stuck at what? Waiting for the condvar itsself or getting the enclosing mutex
> again? What are the states of the involved synchonization objects?
All mutexes are free. There is one task (Thread 2) pulling the mutexes for the duration of signaling the condvars,
this task should never block outside of a sleep function giving it a 1ms cycle.
No deadlock is possible.
What happens is that for some weird reason, Thread 1 got a sporadic wakeup (handling a PF fault from another thread?),
Acquires the mutex and then either is getting demoted to Linux and cause a XCPU signal (if that check is enabled),
or stuck at sc_cobalt_cond_wait_epilogue infinitely.
Then Thread 2 will logically be stuck at re-aquiring the mutex.
I have an alternaivte implementation using Semaphores instead of condvars, I think I have never seen this issue crop up there.
>
> Jan
>
> > 3. Xenomai gets XCPU signal -> coredump
> >
> > gdb) thread apply all bt 3
> >
> > Thread 9 (LWP 682):
> > #0 __lll_unlock_wake () at
> > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
> > #1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt
> > (mutex=0x7fd7312f6968 <_rtld_global+2312>, decr=1) at
> > pthread_mutex_unlock.c:54
> > #2 0x00007fd7312e0442 in ?? () from
> > /home/lano/Downloads/bugcrash/lib64/ld-linux-x86-64.so.2
> > #3 0x00007fd7312c72ac in ?? () from /lib/libdl.so.2
> > #4 0x00007fd73104211f in _dl_catch_exception () from /lib/libc.so.6
> > #5 0x00007fd731042190 in _dl_catch_error () from /lib/libc.so.6
> > #6 0x00007fd7312c7975 in ?? () from /lib/libdl.so.2
> > #7 0x00007fd7312c7327 in dlopen () from /lib/libdl.so.2 (More stack
> > frames follow...)
> >
> > Thread 8 (LWP 686):
> > #0 0x00007fd731298d48 in __cobalt_clock_nanosleep (clock_id=0,
> > flags=0, rqtp=0x7fd727e3ad10, rmtp=0x0) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:312
> > #1 0x00007fd731298d81 in __cobalt_nanosleep (rqtp=<optimized out>,
> > rmtp=<optimized out>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:354
> > #2 0x0000000000434590 in operator() (__closure=0x7fd720006fb8) at
> > ../../acpu.runner/asim/asim_com.cpp:685
> > (More stack frames follow...)
> >
> > Thread 7 (LWP 677):
> > #0 0x00007fd73127b6c6 in __GI___nanosleep
> > (requested_time=requested_time@entry=0x7fd7312b1fb0 <syncdelay>,
> > remaining=remaining@entry=0x0) at
> > ../sysdeps/unix/sysv/linux/nanosleep.c:28
> > #1 0x00007fd73129b746 in printer_loop (arg=<optimized out>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/printf.c:635
> > #2 0x00007fd7312720f7 in start_thread (arg=<optimized out>) at
> > pthread_create.c:486 (More stack frames follow...)
> >
> > Thread 6 (LWP 685):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> > (cond=0x7fd72f269660, mutex=0x7fd72f269630) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f269660,
> > pMutex=0x7fd72f269630) at ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0x7fd72f269660, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 5 (LWP 684):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> > (cond=0x7fd72f267790, mutex=0x7fd72f267760) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f267790,
> > pMutex=0x7fd72f267760) at ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0x7fd72f267790, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 4 (LWP 680):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0xfeafa0
> > <(anonymous namespace)::m_MainTaskStart>, mutex=0xfeaf60
> <(anonymous
> > namespace)::m_TaskMutex>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0xfeafa0
> > <(anonymous namespace)::m_MainTaskStart>, pMutex=0xfeaf60
> <(anonymous
> > namespace)::m_TaskMutex>) at
> > ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 3 (LWP 683):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> > (cond=0x7fd72f2658c0, mutex=0x7fd72f265890) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f2658c0,
> > pMutex=0x7fd72f265890) at ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0x7fd72f2658c0, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 2 (LWP 675):
> > #0 0x00007fd73129aea4 in __cobalt_pthread_mutex_lock
> > (mutex=<optimized out>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/mutex.c:375
> > #1 0x000000000046390a in mutex_lock (pData=0xfeaf60 <(anonymous
> > namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/mutex.c:33
> > #2 0x000000000040a530 in HIPASE::Posix::CAlib_Mutex::lock
> > (this=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at
> > ../../alib/include/alib/alib_mutex_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 1 (LWP 681):
> > #0 __cobalt_pthread_cond_wait (cond=0xfeafe0 <(anonymous
> > namespace)::m_DispatcherTaskStart>, mutex=0xfeaf60 <(anonymous
> > namespace)::m_TaskMutex>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:316
> > #1 0x000000000046377c in conditionvar_wait (pData=0xfeafe0
> > <(anonymous namespace)::m_DispatcherTaskStart>, pMutex=0xfeaf60
> > <(anonymous namespace)::m_TaskMutex>) at
> > ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>,
> > lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> >
> >
> > [1] -
> > https://hes32-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%
> >
> 3a%2f%2fxenomai.org%2fpipermail%2fxenomai%2f2020%2dJanuary%2f0422
> 99.ht
> > ml&umid=b5b10a96-6e6b-4df0-80fd-
> ef11bb4216f1&auth=144056baf7302d777aca
> > d187aac74d4b9ba425e1-4c122b5928aeea18f0ff04626f26fdeeeb97cfa3
> >
> >
> > Mit besten Grüßen / Kind regards
> >
> > NORBERT LANGE
> >
> > AT-RD3
> >
> > ANDRITZ HYDRO GmbH
> > Eibesbrunnergasse 20
> > 1120 Vienna / AUSTRIA
> > p: +43 50805 56684
> > norbert.lange@andritz.com
> > andritz.com
> >
> > ________________________________
> >
> > This message and any attachments are solely for the use of the intended
> recipients. They may contain privileged and/or confidential information or
> other information protected from disclosure. If you are not an intended
> recipient, you are hereby notified that you received this email in error and
> that any review, dissemination, distribution or copying of this email and any
> attachment is strictly prohibited. If you have received this email in error,
> please contact the sender and delete the message and any attachment from
> your system.
> >
> > ANDRITZ HYDRO GmbH
> >
> >
> > Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung /
> > Corporation
> >
> > Firmensitz/ Registered seat: Wien
> >
> > Firmenbuchgericht/ Court of registry: Handelsgericht Wien
> >
> > Firmenbuchnummer/ Company registration: FN 61833 g
> >
> > DVR: 0605077
> >
> > UID-Nr.: ATU14756806
> >
> >
> > Thank You
> > ________________________________
> >
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate
> Competence Center Embedded Linux
Norbert Lange
________________________________
This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
ANDRITZ HYDRO GmbH
Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
Firmensitz/ Registered seat: Wien
Firmenbuchgericht/ Court of registry: Handelsgericht Wien
Firmenbuchnummer/ Company registration: FN 61833 g
DVR: 0605077
UID-Nr.: ATU14756806
Thank You
________________________________
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Still getting Deadlocks with condition variables
2020-06-07 20:15 ` Philippe Gerum
@ 2020-06-08 10:08 ` Lange Norbert
2020-06-08 14:17 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Lange Norbert @ 2020-06-08 10:08 UTC (permalink / raw
To: Philippe Gerum, Xenomai (xenomai@xenomai.org)
> -----Original Message-----
> From: Philippe Gerum <rpm@xenomai.org>
> Sent: Sonntag, 7. Juni 2020 22:16
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: Still getting Deadlocks with condition variables
>
> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
>
>
> On 6/5/20 4:36 PM, Lange Norbert wrote:
> > Hello,
> >
> > I brought this up once or twice at this ML [1], I am still getting
> > some occasional lockups. Now the first time without running under an
> > debugger,
> >
> > Harwdare is a TQMxE39M (Goldmont Atom)
> > Kernel: 4.19.124-cip27-xeno12-static x86_64 I-pipe Version: 12 Xenomai
> > Version: 3.1 Glibc Version 2.28
> >
> > What happens (as far as I understand it):
> >
> > The setup is an project with several cobalt threads (no "native" Linux
> thread as far as I can tell, apart maybe from the cobalt's printf thread).
> > They mostly sleep, and are triggered if work is available, the project
> > also can load DSOs (specialized maths) during configuration stage -
> > during this stages is when the exceptions occur
> >
> >
> > 1. Linux Thread LWP 682 calls SYS_futex "wake"
> >
> > Code immediately before syscall, file x86_64/lowlevellock.S:
> > movl$0, (%rdi)
> > LOAD_FUTEX_WAKE (%esi)
> > movl$1, %edx/* Wake one thread. */
> > movl$SYS_futex, %eax
> > syscall
> >
> > 2. Xenomai switches a cobalt thread to secondary, potentially because all
> threads are in primary:
> >
> > Jun 05 12:35:19 buildroot kernel: [Xenomai] switching dispatcher to
> > secondary mode after exception #14 from user-space at 0x7fd731299115
> > (pid 681)
> >
>
> This kernel message tells a different story, thread pid 681 received a #PF,
> maybe due to accessing its own stack (cond.c, line 316). This may be a minor
> fault though, nothing invalid. Such fault is not supposed to occur for Xenomai
> threads on x86, but that would be another issue. Code-wise, I'm referring to
> the current state of the master branch for lib/cobalt/cond.c, which seems to
> match your description.
I dont know what you mean with minor fault, from the perspective of Linux?
A RT thread getting demoted to Linux is rather serious to me.
Also, the thing is that I would not know how a PF in the long running thread, with locked memory,
With the call being close to the thread entry point in a wait-for-condvar-loop, never using more than an insignificant
amount of stack at this time should be possible.
On the other hand, the non-RT thread loads a DSO and is stuck somewhere after allocating memory.
My guess would be that the PF ends up at the wrong thread.
Note that both tasks are locked to the same CPU core.
>
> > Note that most threads are stuck waiting for a condvar in
> sc_cobalt_cond_wait_prologue (cond.c:313), LWP 681 is at the next
> instruction.
> >
> > 3. Xenomai gets XCPU signal -> coredump
> >
>
> More precisely, Xenomai is likely sending this signal to your application, since
> it had to switch pid 681 to secondary mode for fixing up the #PF event.
> You may have set PTHREAD_WARNSW with pthread_setmode_np() for that
> thread.
Yes, I use PTHREAD_WARNSW, if I did not, then chances are that the code would run
to the sc_cobalt_cond_wait_epilogue, never freeing the mutex and the other thread trying to send a
signal would never be able to acquire the mutex.
Ie. identical to my previous reports (where PTHREAD_WARNSW was disabled)
>
> > gdb) thread apply all bt 3
> >
> > Thread 9 (LWP 682):
> > #0 __lll_unlock_wake () at
> > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
> > #1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt
> > (mutex=0x7fd7312f6968 <_rtld_global+2312>, decr=1) at
> > pthread_mutex_unlock.c:54
> > #2 0x00007fd7312e0442 in ?? () from
> > /home/lano/Downloads/bugcrash/lib64/ld-linux-x86-64.so.2
> > #3 0x00007fd7312c72ac in ?? () from /lib/libdl.so.2
> > #4 0x00007fd73104211f in _dl_catch_exception () from /lib/libc.so.6
> > #5 0x00007fd731042190 in _dl_catch_error () from /lib/libc.so.6
> > #6 0x00007fd7312c7975 in ?? () from /lib/libdl.so.2
> > #7 0x00007fd7312c7327 in dlopen () from /lib/libdl.so.2 (More stack
> > frames follow...)
> >
> > Thread 8 (LWP 686):
> > #0 0x00007fd731298d48 in __cobalt_clock_nanosleep (clock_id=0,
> > flags=0, rqtp=0x7fd727e3ad10, rmtp=0x0) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:312
> > #1 0x00007fd731298d81 in __cobalt_nanosleep (rqtp=<optimized out>,
> > rmtp=<optimized out>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:354
> > #2 0x0000000000434590 in operator() (__closure=0x7fd720006fb8) at
> > ../../acpu.runner/asim/asim_com.cpp:685
> > (More stack frames follow...)
> >
> > Thread 7 (LWP 677):
> > #0 0x00007fd73127b6c6 in __GI___nanosleep
> > (requested_time=requested_time@entry=0x7fd7312b1fb0 <syncdelay>,
> > remaining=remaining@entry=0x0) at
> > ../sysdeps/unix/sysv/linux/nanosleep.c:28
> > #1 0x00007fd73129b746 in printer_loop (arg=<optimized out>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/printf.c:635
> > #2 0x00007fd7312720f7 in start_thread (arg=<optimized out>) at
> > pthread_create.c:486 (More stack frames follow...)
> >
> > Thread 6 (LWP 685):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> > (cond=0x7fd72f269660, mutex=0x7fd72f269630) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f269660,
> > pMutex=0x7fd72f269630) at ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0x7fd72f269660, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 5 (LWP 684):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> > (cond=0x7fd72f267790, mutex=0x7fd72f267760) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f267790,
> > pMutex=0x7fd72f267760) at ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0x7fd72f267790, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 4 (LWP 680):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0xfeafa0
> > <(anonymous namespace)::m_MainTaskStart>, mutex=0xfeaf60
> <(anonymous
> > namespace)::m_TaskMutex>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0xfeafa0
> > <(anonymous namespace)::m_MainTaskStart>, pMutex=0xfeaf60
> <(anonymous
> > namespace)::m_TaskMutex>) at
> > ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 3 (LWP 683):
> > #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> > (cond=0x7fd72f2658c0, mutex=0x7fd72f265890) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> > #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f2658c0,
> > pMutex=0x7fd72f265890) at ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0x7fd72f2658c0, lock=...) at
> > ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 2 (LWP 675):
> > #0 0x00007fd73129aea4 in __cobalt_pthread_mutex_lock
> > (mutex=<optimized out>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/mutex.c:375
> > #1 0x000000000046390a in mutex_lock (pData=0xfeaf60 <(anonymous
> > namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/mutex.c:33
> > #2 0x000000000040a530 in HIPASE::Posix::CAlib_Mutex::lock
> > (this=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at
> > ../../alib/include/alib/alib_mutex_posix.h:67
> > (More stack frames follow...)
> >
> > Thread 1 (LWP 681):
> > #0 __cobalt_pthread_cond_wait (cond=0xfeafe0 <(anonymous
> > namespace)::m_DispatcherTaskStart>, mutex=0xfeaf60 <(anonymous
> > namespace)::m_TaskMutex>) at
> > /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:316
> > #1 0x000000000046377c in conditionvar_wait (pData=0xfeafe0
> > <(anonymous namespace)::m_DispatcherTaskStart>, pMutex=0xfeaf60
> > <(anonymous namespace)::m_TaskMutex>) at
> > ../../alib/src/alib/posix/conditionvar.c:66
> > #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> > (this=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>,
> > lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> > (More stack frames follow...)
> >
> >
> >
> > [1] -
> > https://hes32-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%
> >
> 3a%2f%2fxenomai.org%2fpipermail%2fxenomai%2f2020%2dJanuary%2f0422
> 99.ht
> > ml&umid=2fd21686-c153-48d3-8e06-
> 2f4614e2b0be&auth=144056baf7302d777aca
> > d187aac74d4b9ba425e1-20e3be12930abc88ce2bec989959a3b0ae7b1889
> >
>
> You refer to an older post describing a lockup, but this post describes an
> application crashing with a core dump. What made you draw the conclusion
> that the same bug would be at work?
Same bug, different PTHREAD_WARNSW setting is my guess.
The underlying issue that a unrelated signal ends up to a RT thread.
> Also, could you give some details
> regarding the
> following:
>
> - what do you mean by 'lockup' in this case? Can you still access the board or
> is there some runaway real-time code locking out everything else when this
> happens? My understanding is that this is no hard lock up otherwise the
> watchdog would have triggered. If this is a softer kind of lockup instead, what
> does /proc/xenomai/sched/stat tell you about the thread states after the
> problem occurred?
This was a post-mortem, no access to /proc/xenomai/sched/stat anymore.
lockup means deadlock (the thread getting the signal holds a mutex, but is stuck),
Coredump happens if PTHREAD_WARNSW is enabled (means it asserts out before).
> - did you determine that using the dynamic linker is required to trigger the
> bug yet? Or could you observe it without such interaction with dl?
AFAIK, always occurred at the stage where we load a "configuration", and load DSOs.
>
> - what is the typical size of your Xenomai thread stack? It defaults to 64k min
> with Xenomai 3.1.
1MB
Norbert
________________________________
This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
ANDRITZ HYDRO GmbH
Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
Firmensitz/ Registered seat: Wien
Firmenbuchgericht/ Court of registry: Handelsgericht Wien
Firmenbuchnummer/ Company registration: FN 61833 g
DVR: 0605077
UID-Nr.: ATU14756806
Thank You
________________________________
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Still getting Deadlocks with condition variables
2020-06-08 9:48 ` Lange Norbert
@ 2020-06-08 10:09 ` Jan Kiszka
2020-06-08 13:28 ` Lange Norbert
0 siblings, 1 reply; 13+ messages in thread
From: Jan Kiszka @ 2020-06-08 10:09 UTC (permalink / raw
To: Lange Norbert, Xenomai (xenomai@xenomai.org)
On 08.06.20 11:48, Lange Norbert wrote:
>
>
>> -----Original Message-----
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>> Sent: Freitag, 5. Juni 2020 17:40
>> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
>> (xenomai@xenomai.org) <xenomai@xenomai.org>
>> Subject: Re: Still getting Deadlocks with condition variables
>>
>> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
>> ATTACHMENTS.
>>
>>
>> On 05.06.20 16:36, Lange Norbert via Xenomai wrote:
>>> Hello,
>>>
>>> I brought this up once or twice at this ML [1], I am still getting
>>> some occasional lockups. Now the first time without running under an
>>> debugger,
>>>
>>> Harwdare is a TQMxE39M (Goldmont Atom)
>>> Kernel: 4.19.124-cip27-xeno12-static x86_64 I-pipe Version: 12 Xenomai
>>> Version: 3.1 Glibc Version 2.28
>>>
>>> What happens (as far as I understand it):
>>>
>>> The setup is an project with several cobalt threads (no "native" Linux
>> thread as far as I can tell, apart maybe from the cobalt's printf thread).
>>> They mostly sleep, and are triggered if work is available, the project
>>> also can load DSOs (specialized maths) during configuration stage -
>>> during this stages is when the exceptions occur
>>>
>>>
>>> 1. Linux Thread LWP 682 calls SYS_futex "wake"
>>>
>>> Code immediately before syscall, file x86_64/lowlevellock.S:
>>> movl$0, (%rdi)
>>> LOAD_FUTEX_WAKE (%esi)
>>> movl$1, %edx/* Wake one thread. */
>>> movl$SYS_futex, %eax
>>> syscall
>>>
>>> 2. Xenomai switches a cobalt thread to secondary, potentially because all
>> threads are in primary:
>>>
>>> Jun 05 12:35:19 buildroot kernel: [Xenomai] switching dispatcher to
>>> secondary mode after exception #14 from user-space at 0x7fd731299115
>>> (pid 681)
>>
>> #14 mean page fault, fixable or real. What is at that address? What address
>> was accessed by that instruction?
>>
>>>
>>> Note that most threads are stuck waiting for a condvar in
>> sc_cobalt_cond_wait_prologue (cond.c:313), LWP 681 is at the next
>> instruction.
>>>
>>
>> Stuck at what? Waiting for the condvar itsself or getting the enclosing mutex
>> again? What are the states of the involved synchonization objects?
>
> All mutexes are free. There is one task (Thread 2) pulling the mutexes for the duration of signaling the condvars,
> this task should never block outside of a sleep function giving it a 1ms cycle.
> No deadlock is possible.
>
> What happens is that for some weird reason, Thread 1 got a sporadic wakeup (handling a PF fault from another thread?),
PFs are synchronous, not proxied.
As Philippe also pointed out, understanding that PF is the first step.
Afterwards, we may look into the secondary issue, if there is still one,
and that would be be behavior around the condvars after that PF.
Jan
> Acquires the mutex and then either is getting demoted to Linux and cause a XCPU signal (if that check is enabled),
> or stuck at sc_cobalt_cond_wait_epilogue infinitely.
>
> Then Thread 2 will logically be stuck at re-aquiring the mutex.
>
> I have an alternaivte implementation using Semaphores instead of condvars, I think I have never seen this issue crop up there.
>
>>
>> Jan
>>
>>> 3. Xenomai gets XCPU signal -> coredump
>>>
>>> gdb) thread apply all bt 3
>>>
>>> Thread 9 (LWP 682):
>>> #0 __lll_unlock_wake () at
>>> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
>>> #1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt
>>> (mutex=0x7fd7312f6968 <_rtld_global+2312>, decr=1) at
>>> pthread_mutex_unlock.c:54
>>> #2 0x00007fd7312e0442 in ?? () from
>>> /home/lano/Downloads/bugcrash/lib64/ld-linux-x86-64.so.2
>>> #3 0x00007fd7312c72ac in ?? () from /lib/libdl.so.2
>>> #4 0x00007fd73104211f in _dl_catch_exception () from /lib/libc.so.6
>>> #5 0x00007fd731042190 in _dl_catch_error () from /lib/libc.so.6
>>> #6 0x00007fd7312c7975 in ?? () from /lib/libdl.so.2
>>> #7 0x00007fd7312c7327 in dlopen () from /lib/libdl.so.2 (More stack
>>> frames follow...)
>>>
>>> Thread 8 (LWP 686):
>>> #0 0x00007fd731298d48 in __cobalt_clock_nanosleep (clock_id=0,
>>> flags=0, rqtp=0x7fd727e3ad10, rmtp=0x0) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:312
>>> #1 0x00007fd731298d81 in __cobalt_nanosleep (rqtp=<optimized out>,
>>> rmtp=<optimized out>) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:354
>>> #2 0x0000000000434590 in operator() (__closure=0x7fd720006fb8) at
>>> ../../acpu.runner/asim/asim_com.cpp:685
>>> (More stack frames follow...)
>>>
>>> Thread 7 (LWP 677):
>>> #0 0x00007fd73127b6c6 in __GI___nanosleep
>>> (requested_time=requested_time@entry=0x7fd7312b1fb0 <syncdelay>,
>>> remaining=remaining@entry=0x0) at
>>> ../sysdeps/unix/sysv/linux/nanosleep.c:28
>>> #1 0x00007fd73129b746 in printer_loop (arg=<optimized out>) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/printf.c:635
>>> #2 0x00007fd7312720f7 in start_thread (arg=<optimized out>) at
>>> pthread_create.c:486 (More stack frames follow...)
>>>
>>> Thread 6 (LWP 685):
>>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
>>> (cond=0x7fd72f269660, mutex=0x7fd72f269630) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
>>> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f269660,
>>> pMutex=0x7fd72f269630) at ../../alib/src/alib/posix/conditionvar.c:66
>>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
>>> (this=0x7fd72f269660, lock=...) at
>>> ../../alib/include/alib/alib_conditionvar_posix.h:67
>>> (More stack frames follow...)
>>>
>>> Thread 5 (LWP 684):
>>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
>>> (cond=0x7fd72f267790, mutex=0x7fd72f267760) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
>>> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f267790,
>>> pMutex=0x7fd72f267760) at ../../alib/src/alib/posix/conditionvar.c:66
>>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
>>> (this=0x7fd72f267790, lock=...) at
>>> ../../alib/include/alib/alib_conditionvar_posix.h:67
>>> (More stack frames follow...)
>>>
>>> Thread 4 (LWP 680):
>>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait (cond=0xfeafa0
>>> <(anonymous namespace)::m_MainTaskStart>, mutex=0xfeaf60
>> <(anonymous
>>> namespace)::m_TaskMutex>) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
>>> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafa0
>>> <(anonymous namespace)::m_MainTaskStart>, pMutex=0xfeaf60
>> <(anonymous
>>> namespace)::m_TaskMutex>) at
>>> ../../alib/src/alib/posix/conditionvar.c:66
>>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
>>> (this=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, lock=...) at
>>> ../../alib/include/alib/alib_conditionvar_posix.h:67
>>> (More stack frames follow...)
>>>
>>> Thread 3 (LWP 683):
>>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
>>> (cond=0x7fd72f2658c0, mutex=0x7fd72f265890) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
>>> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f2658c0,
>>> pMutex=0x7fd72f265890) at ../../alib/src/alib/posix/conditionvar.c:66
>>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
>>> (this=0x7fd72f2658c0, lock=...) at
>>> ../../alib/include/alib/alib_conditionvar_posix.h:67
>>> (More stack frames follow...)
>>>
>>> Thread 2 (LWP 675):
>>> #0 0x00007fd73129aea4 in __cobalt_pthread_mutex_lock
>>> (mutex=<optimized out>) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/mutex.c:375
>>> #1 0x000000000046390a in mutex_lock (pData=0xfeaf60 <(anonymous
>>> namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/mutex.c:33
>>> #2 0x000000000040a530 in HIPASE::Posix::CAlib_Mutex::lock
>>> (this=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at
>>> ../../alib/include/alib/alib_mutex_posix.h:67
>>> (More stack frames follow...)
>>>
>>> Thread 1 (LWP 681):
>>> #0 __cobalt_pthread_cond_wait (cond=0xfeafe0 <(anonymous
>>> namespace)::m_DispatcherTaskStart>, mutex=0xfeaf60 <(anonymous
>>> namespace)::m_TaskMutex>) at
>>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:316
>>> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafe0
>>> <(anonymous namespace)::m_DispatcherTaskStart>, pMutex=0xfeaf60
>>> <(anonymous namespace)::m_TaskMutex>) at
>>> ../../alib/src/alib/posix/conditionvar.c:66
>>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
>>> (this=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>,
>>> lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
>>> (More stack frames follow...)
>>>
>>>
>>>
>>> [1] -
>>> https://hes32-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%
>>>
>> 3a%2f%2fxenomai.org%2fpipermail%2fxenomai%2f2020%2dJanuary%2f0422
>> 99.ht
>>> ml&umid=b5b10a96-6e6b-4df0-80fd-
>> ef11bb4216f1&auth=144056baf7302d777aca
>>> d187aac74d4b9ba425e1-4c122b5928aeea18f0ff04626f26fdeeeb97cfa3
>>>
>>>
>>> Mit besten Grüßen / Kind regards
>>>
>>> NORBERT LANGE
>>>
>>> AT-RD3
>>>
>>> ANDRITZ HYDRO GmbH
>>> Eibesbrunnergasse 20
>>> 1120 Vienna / AUSTRIA
>>> p: +43 50805 56684
>>> norbert.lange@andritz.com
>>> andritz.com
>>>
>>> ________________________________
>>>
>>> This message and any attachments are solely for the use of the intended
>> recipients. They may contain privileged and/or confidential information or
>> other information protected from disclosure. If you are not an intended
>> recipient, you are hereby notified that you received this email in error and
>> that any review, dissemination, distribution or copying of this email and any
>> attachment is strictly prohibited. If you have received this email in error,
>> please contact the sender and delete the message and any attachment from
>> your system.
>>>
>>> ANDRITZ HYDRO GmbH
>>>
>>>
>>> Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung /
>>> Corporation
>>>
>>> Firmensitz/ Registered seat: Wien
>>>
>>> Firmenbuchgericht/ Court of registry: Handelsgericht Wien
>>>
>>> Firmenbuchnummer/ Company registration: FN 61833 g
>>>
>>> DVR: 0605077
>>>
>>> UID-Nr.: ATU14756806
>>>
>>>
>>> Thank You
>>> ________________________________
>>>
>>
>> --
>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate
>> Competence Center Embedded Linux
>
> Norbert Lange
>
> ________________________________
>
> This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
>
> ANDRITZ HYDRO GmbH
>
>
> Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
>
> Firmensitz/ Registered seat: Wien
>
> Firmenbuchgericht/ Court of registry: Handelsgericht Wien
>
> Firmenbuchnummer/ Company registration: FN 61833 g
>
> DVR: 0605077
>
> UID-Nr.: ATU14756806
>
>
> Thank You
> ________________________________
>
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Still getting Deadlocks with condition variables
2020-06-08 10:09 ` Jan Kiszka
@ 2020-06-08 13:28 ` Lange Norbert
0 siblings, 0 replies; 13+ messages in thread
From: Lange Norbert @ 2020-06-08 13:28 UTC (permalink / raw
To: Jan Kiszka, Xenomai (xenomai@xenomai.org)
> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Montag, 8. Juni 2020 12:09
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: Still getting Deadlocks with condition variables
>
> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
>
>
> On 08.06.20 11:48, Lange Norbert wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >> Sent: Freitag, 5. Juni 2020 17:40
> >> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> >> (xenomai@xenomai.org) <xenomai@xenomai.org>
> >> Subject: Re: Still getting Deadlocks with condition variables
> >>
> >> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
> >>
> >>
> >> On 05.06.20 16:36, Lange Norbert via Xenomai wrote:
> >>> Hello,
> >>>
> >>> I brought this up once or twice at this ML [1], I am still getting
> >>> some occasional lockups. Now the first time without running under an
> >>> debugger,
> >>>
> >>> Harwdare is a TQMxE39M (Goldmont Atom)
> >>> Kernel: 4.19.124-cip27-xeno12-static x86_64 I-pipe Version: 12 Xenomai
> >>> Version: 3.1 Glibc Version 2.28
> >>>
> >>> What happens (as far as I understand it):
> >>>
> >>> The setup is an project with several cobalt threads (no "native" Linux
> >> thread as far as I can tell, apart maybe from the cobalt's printf thread).
> >>> They mostly sleep, and are triggered if work is available, the project
> >>> also can load DSOs (specialized maths) during configuration stage -
> >>> during this stages is when the exceptions occur
> >>>
> >>>
> >>> 1. Linux Thread LWP 682 calls SYS_futex "wake"
> >>>
> >>> Code immediately before syscall, file x86_64/lowlevellock.S:
> >>> movl$0, (%rdi)
> >>> LOAD_FUTEX_WAKE (%esi)
> >>> movl$1, %edx/* Wake one thread. */
> >>> movl$SYS_futex, %eax
> >>> syscall
> >>>
> >>> 2. Xenomai switches a cobalt thread to secondary, potentially because all
> >> threads are in primary:
> >>>
> >>> Jun 05 12:35:19 buildroot kernel: [Xenomai] switching dispatcher to
> >>> secondary mode after exception #14 from user-space at 0x7fd731299115
> >>> (pid 681)
> >>
> >> #14 mean page fault, fixable or real. What is at that address? What
> address
> >> was accessed by that instruction?
> >>
> >>>
> >>> Note that most threads are stuck waiting for a condvar in
> >> sc_cobalt_cond_wait_prologue (cond.c:313), LWP 681 is at the next
> >> instruction.
> >>>
> >>
> >> Stuck at what? Waiting for the condvar itsself or getting the enclosing
> mutex
> >> again? What are the states of the involved synchonization objects?
> >
> > All mutexes are free. There is one task (Thread 2) pulling the mutexes for
> the duration of signaling the condvars,
> > this task should never block outside of a sleep function giving it a 1ms cycle.
> > No deadlock is possible.
> >
> > What happens is that for some weird reason, Thread 1 got a sporadic
> wakeup (handling a PF fault from another thread?),
>
> PFs are synchronous, not proxied.
>
> As Philippe also pointed out, understanding that PF is the first step.
> Afterwards, we may look into the secondary issue, if there is still one,
> and that would be be behavior around the condvars after that PF.
As I told Phillipe, there is no way I can image the thread holding the condvar running into a PF.
This code is run several thousand times before the issue happens, it's basically the outer loop that just waits for work,
Stack is plenty (1M).
The dlopen call triggers it, it's always stuck in the same position.
Thread 9 (LWP 682):
#0 __lll_unlock_wake () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
#1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt (mutex=0x7fd7312f6968 <_rtld_local+2312>, decr=1) at pthread_mutex_unlock.c:54
#2 0x00007fd7312e0442 in _dl_open (file=<optimized out>, mode=-2147483647, caller_dlopen=0x460864 <HIPASE::CAlib_SharedLibrary::loadImplRaw(char const*, HIPASE::CAlib_SharedLibrary::EFlags, bool)+124>, nsid=-2, argc=7, argv=0x7fd728680f90, env=0x7ffc1e972b88) at dl-open.c:627
#3 0x00007fd7312c72ac in dlopen_doit (a=a@entry=0x7fd7286811d0) at dlopen.c:66
#4 0x00007fd73104211f in __GI__dl_catch_exception (exception=exception@entry=0x7fd728681170, operate=operate@entry=0x7fd7312c7250 <dlopen_doit>, args=args@entry=0x7fd7286811d0) at dl-error-skeleton.c:196
#5 0x00007fd731042190 in __GI__dl_catch_error (objname=objname@entry=0x7fd72005c010, errstring=errstring@entry=0x7fd72005c018, mallocedp=mallocedp@entry=0x7fd72005c008, operate=operate@entry=0x7fd7312c7250 <dlopen_doit>, args=args@entry=0x7fd7286811d0) at dl-error-skeleton.c:215
#6 0x00007fd7312c7975 in _dlerror_run (operate=operate@entry=0x7fd7312c7250 <dlopen_doit>, args=args@entry=0x7fd7286811d0) at dlerror.c:163
#7 0x00007fd7312c7327 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#8 0x0000000000460864 in HIPASE::CAlib_SharedLibrary::loadImplRaw (szPath=0x7fd728681250 "/opt/acpu/poe/tof/v0100/tof.so", flags=HIPASE::CAlib_SharedLibrary::EFlags::Local, noload=false) at ../../alib/src/alib/alib_sharedlibrary.cpp:61
(glibc 2-28 if you want to make sense of the line numbers)
Norbert
>
> Jan
>
> > Acquires the mutex and then either is getting demoted to Linux and cause
> a XCPU signal (if that check is enabled),
> > or stuck at sc_cobalt_cond_wait_epilogue infinitely.
> >
> > Then Thread 2 will logically be stuck at re-aquiring the mutex.
> >
> > I have an alternaivte implementation using Semaphores instead of
> condvars, I think I have never seen this issue crop up there.
> >
> >>
> >> Jan
> >>
> >>> 3. Xenomai gets XCPU signal -> coredump
> >>>
> >>> gdb) thread apply all bt 3
> >>>
> >>> Thread 9 (LWP 682):
> >>> #0 __lll_unlock_wake () at
> >>> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:339
> >>> #1 0x00007fd731275d65 in __pthread_mutex_unlock_usercnt
> >>> (mutex=0x7fd7312f6968 <_rtld_global+2312>, decr=1) at
> >>> pthread_mutex_unlock.c:54
> >>> #2 0x00007fd7312e0442 in ?? () from
> >>> /home/lano/Downloads/bugcrash/lib64/ld-linux-x86-64.so.2
> >>> #3 0x00007fd7312c72ac in ?? () from /lib/libdl.so.2
> >>> #4 0x00007fd73104211f in _dl_catch_exception () from /lib/libc.so.6
> >>> #5 0x00007fd731042190 in _dl_catch_error () from /lib/libc.so.6
> >>> #6 0x00007fd7312c7975 in ?? () from /lib/libdl.so.2
> >>> #7 0x00007fd7312c7327 in dlopen () from /lib/libdl.so.2 (More stack
> >>> frames follow...)
> >>>
> >>> Thread 8 (LWP 686):
> >>> #0 0x00007fd731298d48 in __cobalt_clock_nanosleep (clock_id=0,
> >>> flags=0, rqtp=0x7fd727e3ad10, rmtp=0x0) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:312
> >>> #1 0x00007fd731298d81 in __cobalt_nanosleep (rqtp=<optimized out>,
> >>> rmtp=<optimized out>) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/clock.c:354
> >>> #2 0x0000000000434590 in operator() (__closure=0x7fd720006fb8) at
> >>> ../../acpu.runner/asim/asim_com.cpp:685
> >>> (More stack frames follow...)
> >>>
> >>> Thread 7 (LWP 677):
> >>> #0 0x00007fd73127b6c6 in __GI___nanosleep
> >>> (requested_time=requested_time@entry=0x7fd7312b1fb0
> <syncdelay>,
> >>> remaining=remaining@entry=0x0) at
> >>> ../sysdeps/unix/sysv/linux/nanosleep.c:28
> >>> #1 0x00007fd73129b746 in printer_loop (arg=<optimized out>) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/printf.c:635
> >>> #2 0x00007fd7312720f7 in start_thread (arg=<optimized out>) at
> >>> pthread_create.c:486 (More stack frames follow...)
> >>>
> >>> Thread 6 (LWP 685):
> >>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> >>> (cond=0x7fd72f269660, mutex=0x7fd72f269630) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> >>> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f269660,
> >>> pMutex=0x7fd72f269630) at ../../alib/src/alib/posix/conditionvar.c:66
> >>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> >>> (this=0x7fd72f269660, lock=...) at
> >>> ../../alib/include/alib/alib_conditionvar_posix.h:67
> >>> (More stack frames follow...)
> >>>
> >>> Thread 5 (LWP 684):
> >>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> >>> (cond=0x7fd72f267790, mutex=0x7fd72f267760) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> >>> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f267790,
> >>> pMutex=0x7fd72f267760) at ../../alib/src/alib/posix/conditionvar.c:66
> >>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> >>> (this=0x7fd72f267790, lock=...) at
> >>> ../../alib/include/alib/alib_conditionvar_posix.h:67
> >>> (More stack frames follow...)
> >>>
> >>> Thread 4 (LWP 680):
> >>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> (cond=0xfeafa0
> >>> <(anonymous namespace)::m_MainTaskStart>, mutex=0xfeaf60
> >> <(anonymous
> >>> namespace)::m_TaskMutex>) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> >>> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafa0
> >>> <(anonymous namespace)::m_MainTaskStart>, pMutex=0xfeaf60
> >> <(anonymous
> >>> namespace)::m_TaskMutex>) at
> >>> ../../alib/src/alib/posix/conditionvar.c:66
> >>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> >>> (this=0xfeafa0 <(anonymous namespace)::m_MainTaskStart>, lock=...)
> at
> >>> ../../alib/include/alib/alib_conditionvar_posix.h:67
> >>> (More stack frames follow...)
> >>>
> >>> Thread 3 (LWP 683):
> >>> #0 0x00007fd73129910a in __cobalt_pthread_cond_wait
> >>> (cond=0x7fd72f2658c0, mutex=0x7fd72f265890) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:313
> >>> #1 0x000000000046377c in conditionvar_wait (pData=0x7fd72f2658c0,
> >>> pMutex=0x7fd72f265890) at ../../alib/src/alib/posix/conditionvar.c:66
> >>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> >>> (this=0x7fd72f2658c0, lock=...) at
> >>> ../../alib/include/alib/alib_conditionvar_posix.h:67
> >>> (More stack frames follow...)
> >>>
> >>> Thread 2 (LWP 675):
> >>> #0 0x00007fd73129aea4 in __cobalt_pthread_mutex_lock
> >>> (mutex=<optimized out>) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/mutex.c:375
> >>> #1 0x000000000046390a in mutex_lock (pData=0xfeaf60 <(anonymous
> >>> namespace)::m_TaskMutex>) at ../../alib/src/alib/posix/mutex.c:33
> >>> #2 0x000000000040a530 in HIPASE::Posix::CAlib_Mutex::lock
> >>> (this=0xfeaf60 <(anonymous namespace)::m_TaskMutex>) at
> >>> ../../alib/include/alib/alib_mutex_posix.h:67
> >>> (More stack frames follow...)
> >>>
> >>> Thread 1 (LWP 681):
> >>> #0 __cobalt_pthread_cond_wait (cond=0xfeafe0 <(anonymous
> >>> namespace)::m_DispatcherTaskStart>, mutex=0xfeaf60 <(anonymous
> >>> namespace)::m_TaskMutex>) at
> >>> /opt/hipase2/src/xenomai-3.1.0/lib/cobalt/cond.c:316
> >>> #1 0x000000000046377c in conditionvar_wait (pData=0xfeafe0
> >>> <(anonymous namespace)::m_DispatcherTaskStart>, pMutex=0xfeaf60
> >>> <(anonymous namespace)::m_TaskMutex>) at
> >>> ../../alib/src/alib/posix/conditionvar.c:66
> >>> #2 0x000000000040a620 in HIPASE::Posix::CAlib_ConditionVariable::wait
> >>> (this=0xfeafe0 <(anonymous namespace)::m_DispatcherTaskStart>,
> >>> lock=...) at ../../alib/include/alib/alib_conditionvar_posix.h:67
> >>> (More stack frames follow...)
> >>>
> >>>
> >>>
> >>> [1] -
> >>> https://hes32-
> ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%
> >>>
> >>
> 3a%2f%2fxenomai.org%2fpipermail%2fxenomai%2f2020%2dJanuary%2f0422
> >> 99.ht
> >>> ml&umid=b5b10a96-6e6b-4df0-80fd-
> >> ef11bb4216f1&auth=144056baf7302d777aca
> >>> d187aac74d4b9ba425e1-4c122b5928aeea18f0ff04626f26fdeeeb97cfa3
> >>>
> >>>
> >>> Mit besten Grüßen / Kind regards
> >>>
> >>> NORBERT LANGE
> >>>
> >>> AT-RD3
> >>>
> >>> ANDRITZ HYDRO GmbH
> >>> Eibesbrunnergasse 20
> >>> 1120 Vienna / AUSTRIA
> >>> p: +43 50805 56684
> >>> norbert.lange@andritz.com
> >>> andritz.com
> >>>
> >>> ________________________________
> >>>
> >>> This message and any attachments are solely for the use of the intended
> >> recipients. They may contain privileged and/or confidential information or
> >> other information protected from disclosure. If you are not an intended
> >> recipient, you are hereby notified that you received this email in error and
> >> that any review, dissemination, distribution or copying of this email and
> any
> >> attachment is strictly prohibited. If you have received this email in error,
> >> please contact the sender and delete the message and any attachment
> from
> >> your system.
> >>>
> >>> ANDRITZ HYDRO GmbH
> >>>
> >>>
> >>> Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung /
> >>> Corporation
> >>>
> >>> Firmensitz/ Registered seat: Wien
> >>>
> >>> Firmenbuchgericht/ Court of registry: Handelsgericht Wien
> >>>
> >>> Firmenbuchnummer/ Company registration: FN 61833 g
> >>>
> >>> DVR: 0605077
> >>>
> >>> UID-Nr.: ATU14756806
> >>>
> >>>
> >>> Thank You
> >>> ________________________________
> >>>
> >>
> >> --
> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate
> >> Competence Center Embedded Linux
> >
> > Norbert Lange
> >
> > ________________________________
> >
> > This message and any attachments are solely for the use of the intended
> recipients. They may contain privileged and/or confidential information or
> other information protected from disclosure. If you are not an intended
> recipient, you are hereby notified that you received this email in error and
> that any review, dissemination, distribution or copying of this email and any
> attachment is strictly prohibited. If you have received this email in error,
> please contact the sender and delete the message and any attachment from
> your system.
> >
> > ANDRITZ HYDRO GmbH
> >
> >
> > Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung /
> Corporation
> >
> > Firmensitz/ Registered seat: Wien
> >
> > Firmenbuchgericht/ Court of registry: Handelsgericht Wien
> >
> > Firmenbuchnummer/ Company registration: FN 61833 g
> >
> > DVR: 0605077
> >
> > UID-Nr.: ATU14756806
> >
> >
> > Thank You
> > ________________________________
> >
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
________________________________
This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
ANDRITZ HYDRO GmbH
Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
Firmensitz/ Registered seat: Wien
Firmenbuchgericht/ Court of registry: Handelsgericht Wien
Firmenbuchnummer/ Company registration: FN 61833 g
DVR: 0605077
UID-Nr.: ATU14756806
Thank You
________________________________
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Still getting Deadlocks with condition variables
2020-06-08 10:08 ` Lange Norbert
@ 2020-06-08 14:17 ` Philippe Gerum
2020-06-09 17:10 ` Lange Norbert
0 siblings, 1 reply; 13+ messages in thread
From: Philippe Gerum @ 2020-06-08 14:17 UTC (permalink / raw
To: Lange Norbert, Xenomai (xenomai@xenomai.org)
On 6/8/20 12:08 PM, Lange Norbert wrote:
>
>> This kernel message tells a different story, thread pid 681 received a #PF,
>> maybe due to accessing its own stack (cond.c, line 316). This may be a minor
>> fault though, nothing invalid. Such fault is not supposed to occur for Xenomai
>> threads on x86, but that would be another issue. Code-wise, I'm referring to
>> the current state of the master branch for lib/cobalt/cond.c, which seems to
>> match your description.
>
> I dont know what you mean with minor fault, from the perspective of Linux?
> A RT thread getting demoted to Linux is rather serious to me.
>
Minor from a MMU standpoint: the memory the CPU dereferenced is valid but no
page table entry currently maps it. So, #PF in this case seems to be a 'minor'
fault in MMU lingo, but it is still not expected.
> Also, the thing is that I would not know how a PF in the long running thread, with locked memory,
> With the call being close to the thread entry point in a wait-for-condvar-loop, never using more than an insignificant> amount of stack at this time should be possible.>
Except if mapping an executable segment via dlopen() comes into play,
affecting the page table. Only an assumption at this stage.
> On the other hand, the non-RT thread loads a DSO and is stuck somewhere after allocating memory.
> My guess would be that the PF ends up at the wrong thread.
>
As Jan pointed out, #PF are synchronously taken, synchronously handled. I
really don't see how #PF handling could ever wander.
>>>
>>
>> You refer to an older post describing a lockup, but this post describes an
>> application crashing with a core dump. What made you draw the conclusion
>> that the same bug would be at work?
>
> Same bug, different PTHREAD_WARNSW setting is my guess.
> The underlying issue that a unrelated signal ends up to a RT thread.
>
>> Also, could you give some details
>> regarding the
>> following:
>>
>> - what do you mean by 'lockup' in this case? Can you still access the board or
>> is there some runaway real-time code locking out everything else when this
>> happens? My understanding is that this is no hard lock up otherwise the
>> watchdog would have triggered. If this is a softer kind of lockup instead, what
>> does /proc/xenomai/sched/stat tell you about the thread states after the
>> problem occurred?
>
> This was a post-mortem, no access to /proc/xenomai/sched/stat anymore.
> lockup means deadlock (the thread getting the signal holds a mutex, but is stuck),
> Coredump happens if PTHREAD_WARNSW is enabled (means it asserts out before).
>
>> - did you determine that using the dynamic linker is required to trigger the
>> bug yet? Or could you observe it without such interaction with dl?
>
> AFAIK, always occurred at the stage where we load a "configuration", and load DSOs.
>
>>
>> - what is the typical size of your Xenomai thread stack? It defaults to 64k min
>> with Xenomai 3.1.
>
> 1MB
I would dig the following distinct issues:
- why is #PF taken on an apparently innocuous instruction. dlopen(3)->mmap(2)
might be involved. With a simple test case, you could check the impact of
loading/unloading DSOs on memory management for real-time threads running in
parallel. Setting the WARNSW bit on for these threads would be required.
- whether dealing with a signal adversely affects the wait-side of a Xenomai
condvar. There is a specific trick to handle this in the Cobalt and libcobalt
code, which is the reason for the wait_prologue / wait_epilogue dance in the
implementation IIRC. Understanding why that thread receives a signal in the
first place would help too. According to your description, this may not be
directly due to taking #PF, but may be an indirect consequence of that event
on sibling threads (propagation of a debug condition of some sort, such as
those detected by CONFIG_XENO_OPT_DEBUG_MUTEX*).
At any rate, you may want to enable the function ftracer, enabling conditional
snapshots, e.g. when SIGXCPU is sent by the cobalt core. Guesswork with such
bug is unlikely to uncover every aspect of the issue, hard data would be
required to go to the bottom of it. With a bit of luck, that bug is not
time-sensitive in a way that the overhead due to ftracing would paper over it.
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Still getting Deadlocks with condition variables
2020-06-08 14:17 ` Philippe Gerum
@ 2020-06-09 17:10 ` Lange Norbert
2020-06-10 16:48 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Lange Norbert @ 2020-06-09 17:10 UTC (permalink / raw
To: Philippe Gerum, Xenomai (xenomai@xenomai.org),
'jan.kiszka@siemens.com'
> -----Original Message-----
> From: Philippe Gerum <rpm@xenomai.org>
> Sent: Montag, 8. Juni 2020 16:17
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: Still getting Deadlocks with condition variables
>
> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
>
>
> On 6/8/20 12:08 PM, Lange Norbert wrote:
> >
> >> This kernel message tells a different story, thread pid 681 received
> >> a #PF, maybe due to accessing its own stack (cond.c, line 316). This
> >> may be a minor fault though, nothing invalid. Such fault is not
> >> supposed to occur for Xenomai threads on x86, but that would be
> >> another issue. Code-wise, I'm referring to the current state of the
> >> master branch for lib/cobalt/cond.c, which seems to match your
> description.
> >
> > I dont know what you mean with minor fault, from the perspective of
> Linux?
> > A RT thread getting demoted to Linux is rather serious to me.
> >
>
> Minor from a MMU standpoint: the memory the CPU dereferenced is valid
> but no page table entry currently maps it. So, #PF in this case seems to be a
> 'minor'
> fault in MMU lingo, but it is still not expected.
>
> > Also, the thing is that I would not know how a PF in the long running
> > thread, with locked memory, With the call being close to the thread
> > entry point in a wait-for-condvar-loop, never using more than an
> > insignificant> amount of stack at this time should be possible.>
>
> Except if mapping an executable segment via dlopen() comes into play,
> affecting the page table. Only an assumption at this stage.
>
> > On the other hand, the non-RT thread loads a DSO and is stuck somewhere
> after allocating memory.
> > My guess would be that the PF ends up at the wrong thread.
> >
>
> As Jan pointed out, #PF are synchronously taken, synchronously handled. I
> really don't see how #PF handling could ever wander.
>
> >>>
> >>
> >> You refer to an older post describing a lockup, but this post
> >> describes an application crashing with a core dump. What made you
> >> draw the conclusion that the same bug would be at work?
> >
> > Same bug, different PTHREAD_WARNSW setting is my guess.
> > The underlying issue that a unrelated signal ends up to a RT thread.
> >
> >> Also, could you give some details
> >> regarding the
> >> following:
> >>
> >> - what do you mean by 'lockup' in this case? Can you still access the
> >> board or is there some runaway real-time code locking out everything
> >> else when this happens? My understanding is that this is no hard lock
> >> up otherwise the watchdog would have triggered. If this is a softer
> >> kind of lockup instead, what does /proc/xenomai/sched/stat tell you
> >> about the thread states after the problem occurred?
> >
> > This was a post-mortem, no access to /proc/xenomai/sched/stat anymore.
> > lockup means deadlock (the thread getting the signal holds a mutex,
> > but is stuck), Coredump happens if PTHREAD_WARNSW is enabled (means
> it asserts out before).
> >
> >> - did you determine that using the dynamic linker is required to
> >> trigger the bug yet? Or could you observe it without such interaction with
> dl?
> >
> > AFAIK, always occurred at the stage where we load a "configuration", and
> load DSOs.
> >
> >>
> >> - what is the typical size of your Xenomai thread stack? It defaults
> >> to 64k min with Xenomai 3.1.
> >
> > 1MB
>
> I would dig the following distinct issues:
>
> - why is #PF taken on an apparently innocuous instruction. dlopen(3)-
> >mmap(2) might be involved. With a simple test case, you could check the
> impact of loading/unloading DSOs on memory management for real-time
> threads running in parallel. Setting the WARNSW bit on for these threads
> would be required.
>
> - whether dealing with a signal adversely affects the wait-side of a Xenomai
> condvar. There is a specific trick to handle this in the Cobalt and libcobalt
> code, which is the reason for the wait_prologue / wait_epilogue dance in the
> implementation IIRC. Understanding why that thread receives a signal in the
> first place would help too. According to your description, this may not be
> directly due to taking #PF, but may be an indirect consequence of that event
> on sibling threads (propagation of a debug condition of some sort, such as
> those detected by CONFIG_XENO_OPT_DEBUG_MUTEX*).
>
> At any rate, you may want to enable the function ftracer, enabling
> conditional snapshots, e.g. when SIGXCPU is sent by the cobalt core.
> Guesswork with such bug is unlikely to uncover every aspect of the issue,
> hard data would be required to go to the bottom of it. With a bit of luck, that
> bug is not time-sensitive in a way that the overhead due to ftracing would
> paper over it.
>
> --
> Philippe.
This aint exactly easy to reproduce, managed to get something that often reproduces just now.
Tracing however hides the issue, as well as disabling PTHREAD_WARNSW (could be just that this changes timing enough to make a difference).
I got a few instances where the thread loading DSOS is stuck in an omnious
__make_stacks_executable, that seemingly iterates through *all* thread stacks,
And calls __mprotect on them.
If that’s the cause, and if the cobalt thread use that same stack,
and if the syscall does something funny like taking away write protection in-between,
then this could be the explanation (don’t know how this could ever be valid though).
int
__make_stacks_executable (void **stack_endp)
{
/* First the main thread's stack. */
int err = _dl_make_stack_executable (stack_endp);
if (err != 0)
return err;
#ifdef NEED_SEPARATE_REGISTER_STACK
const size_t pagemask = ~(__getpagesize () - 1);
#endif
lll_lock (stack_cache_lock, LLL_PRIVATE);
list_t *runp;
list_for_each (runp, &stack_used)
{
err = change_stack_perm (list_entry (runp, struct pthread, list)
#ifdef NEED_SEPARATE_REGISTER_STACK
, pagemask
#endif
);
if (err != 0)
break;
}
regards, Norbert
________________________________
This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
ANDRITZ HYDRO GmbH
Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
Firmensitz/ Registered seat: Wien
Firmenbuchgericht/ Court of registry: Handelsgericht Wien
Firmenbuchnummer/ Company registration: FN 61833 g
DVR: 0605077
UID-Nr.: ATU14756806
Thank You
________________________________
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Still getting Deadlocks with condition variables
2020-06-09 17:10 ` Lange Norbert
@ 2020-06-10 16:48 ` Philippe Gerum
2020-06-15 9:06 ` Lange Norbert
0 siblings, 1 reply; 13+ messages in thread
From: Philippe Gerum @ 2020-06-10 16:48 UTC (permalink / raw
To: Lange Norbert, Xenomai (xenomai@xenomai.org),
'jan.kiszka@siemens.com'
On 6/9/20 7:10 PM, Lange Norbert wrote:
>
>
>> -----Original Message-----
>> From: Philippe Gerum <rpm@xenomai.org>
>> Sent: Montag, 8. Juni 2020 16:17
>> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
>> (xenomai@xenomai.org) <xenomai@xenomai.org>
>> Subject: Re: Still getting Deadlocks with condition variables
>>
>> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
>> ATTACHMENTS.
>>
>>
>> On 6/8/20 12:08 PM, Lange Norbert wrote:
>>>
>>>> This kernel message tells a different story, thread pid 681 received
>>>> a #PF, maybe due to accessing its own stack (cond.c, line 316). This
>>>> may be a minor fault though, nothing invalid. Such fault is not
>>>> supposed to occur for Xenomai threads on x86, but that would be
>>>> another issue. Code-wise, I'm referring to the current state of the
>>>> master branch for lib/cobalt/cond.c, which seems to match your
>> description.
>>>
>>> I dont know what you mean with minor fault, from the perspective of
>> Linux?
>>> A RT thread getting demoted to Linux is rather serious to me.
>>>
>>
>> Minor from a MMU standpoint: the memory the CPU dereferenced is valid
>> but no page table entry currently maps it. So, #PF in this case seems to be a
>> 'minor'
>> fault in MMU lingo, but it is still not expected.
>>
>>> Also, the thing is that I would not know how a PF in the long running
>>> thread, with locked memory, With the call being close to the thread
>>> entry point in a wait-for-condvar-loop, never using more than an
>>> insignificant> amount of stack at this time should be possible.>
>>
>> Except if mapping an executable segment via dlopen() comes into play,
>> affecting the page table. Only an assumption at this stage.
>>
>>> On the other hand, the non-RT thread loads a DSO and is stuck somewhere
>> after allocating memory.
>>> My guess would be that the PF ends up at the wrong thread.
>>>
>>
>> As Jan pointed out, #PF are synchronously taken, synchronously handled. I
>> really don't see how #PF handling could ever wander.
>>
>>>>>
>>>>
>>>> You refer to an older post describing a lockup, but this post
>>>> describes an application crashing with a core dump. What made you
>>>> draw the conclusion that the same bug would be at work?
>>>
>>> Same bug, different PTHREAD_WARNSW setting is my guess.
>>> The underlying issue that a unrelated signal ends up to a RT thread.
>>>
>>>> Also, could you give some details
>>>> regarding the
>>>> following:
>>>>
>>>> - what do you mean by 'lockup' in this case? Can you still access the
>>>> board or is there some runaway real-time code locking out everything
>>>> else when this happens? My understanding is that this is no hard lock
>>>> up otherwise the watchdog would have triggered. If this is a softer
>>>> kind of lockup instead, what does /proc/xenomai/sched/stat tell you
>>>> about the thread states after the problem occurred?
>>>
>>> This was a post-mortem, no access to /proc/xenomai/sched/stat anymore.
>>> lockup means deadlock (the thread getting the signal holds a mutex,
>>> but is stuck), Coredump happens if PTHREAD_WARNSW is enabled (means
>> it asserts out before).
>>>
>>>> - did you determine that using the dynamic linker is required to
>>>> trigger the bug yet? Or could you observe it without such interaction with
>> dl?
>>>
>>> AFAIK, always occurred at the stage where we load a "configuration", and
>> load DSOs.
>>>
>>>>
>>>> - what is the typical size of your Xenomai thread stack? It defaults
>>>> to 64k min with Xenomai 3.1.
>>>
>>> 1MB
>>
>> I would dig the following distinct issues:
>>
>> - why is #PF taken on an apparently innocuous instruction. dlopen(3)-
>>> mmap(2) might be involved. With a simple test case, you could check the
>> impact of loading/unloading DSOs on memory management for real-time
>> threads running in parallel. Setting the WARNSW bit on for these threads
>> would be required.
>>
>> - whether dealing with a signal adversely affects the wait-side of a Xenomai
>> condvar. There is a specific trick to handle this in the Cobalt and libcobalt
>> code, which is the reason for the wait_prologue / wait_epilogue dance in the
>> implementation IIRC. Understanding why that thread receives a signal in the
>> first place would help too. According to your description, this may not be
>> directly due to taking #PF, but may be an indirect consequence of that event
>> on sibling threads (propagation of a debug condition of some sort, such as
>> those detected by CONFIG_XENO_OPT_DEBUG_MUTEX*).
>>
>> At any rate, you may want to enable the function ftracer, enabling
>> conditional snapshots, e.g. when SIGXCPU is sent by the cobalt core.
>> Guesswork with such bug is unlikely to uncover every aspect of the issue,
>> hard data would be required to go to the bottom of it. With a bit of luck, that
>> bug is not time-sensitive in a way that the overhead due to ftracing would
>> paper over it.
>>
>> --
>> Philippe.
>
> This aint exactly easy to reproduce, managed to get something that often reproduces just now.
> Tracing however hides the issue, as well as disabling PTHREAD_WARNSW (could be just that this changes timing enough to make a difference).
>
>
> I got a few instances where the thread loading DSOS is stuck in an omnious
> __make_stacks_executable, that seemingly iterates through *all* thread stacks,
> And calls __mprotect on them.
>
> If that’s the cause, and if the cobalt thread use that same stack,
> and if the syscall does something funny like taking away write protection in-between,
> then this could be the explanation (don’t know how this could ever be valid though).
>
> int
> __make_stacks_executable (void **stack_endp)
> {
> /* First the main thread's stack. */
> int err = _dl_make_stack_executable (stack_endp);
> if (err != 0)
> return err;
>
> #ifdef NEED_SEPARATE_REGISTER_STACK
> const size_t pagemask = ~(__getpagesize () - 1);
> #endif
>
> lll_lock (stack_cache_lock, LLL_PRIVATE);
>
> list_t *runp;
> list_for_each (runp, &stack_used)
> {
> err = change_stack_perm (list_entry (runp, struct pthread, list)
This code does not take away any protection, on the contrary this ensures that
PROT_EXEC is set for all stacks along with read and write access, which is
glibc's default for the x86_64 architecture.
The fault is likely due to mm code fixing up those protections for the
relevant page(s). It looks like such pages are force faulted-in, which would
explain the #PF, and the SIGXCPU notification as a consequence. These are
minor faults in the MMU management sense, so this is transparent for common
applications. Not for those of us who do not want the application code to run
into any page fault unfortunately.
Loading DSOs while the real-time system is running just proved to be a bad
idea it seems (did not check how other *libc implementations behave on
dlopen() though).
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Still getting Deadlocks with condition variables
2020-06-10 16:48 ` Philippe Gerum
@ 2020-06-15 9:06 ` Lange Norbert
2020-06-15 10:02 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Lange Norbert @ 2020-06-15 9:06 UTC (permalink / raw
To: Philippe Gerum, Xenomai (xenomai@xenomai.org),
'jan.kiszka@siemens.com'
> -----Original Message-----
> From: Philippe Gerum <rpm@xenomai.org>
> Sent: Mittwoch, 10. Juni 2020 18:48
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>;
> 'jan.kiszka@siemens.com' <jan.kiszka@siemens.com>
> Subject: Re: Still getting Deadlocks with condition variables
>
> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
>
>
> On 6/9/20 7:10 PM, Lange Norbert wrote:
> >
> >
> >> -----Original Message-----
> >> From: Philippe Gerum <rpm@xenomai.org>
> >> Sent: Montag, 8. Juni 2020 16:17
> >> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> >> (xenomai@xenomai.org) <xenomai@xenomai.org>
> >> Subject: Re: Still getting Deadlocks with condition variables
> >>
> >> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
> >>
> >>
> >> On 6/8/20 12:08 PM, Lange Norbert wrote:
> >>>
> >>>> This kernel message tells a different story, thread pid 681 received
> >>>> a #PF, maybe due to accessing its own stack (cond.c, line 316). This
> >>>> may be a minor fault though, nothing invalid. Such fault is not
> >>>> supposed to occur for Xenomai threads on x86, but that would be
> >>>> another issue. Code-wise, I'm referring to the current state of the
> >>>> master branch for lib/cobalt/cond.c, which seems to match your
> >> description.
> >>>
> >>> I dont know what you mean with minor fault, from the perspective of
> >> Linux?
> >>> A RT thread getting demoted to Linux is rather serious to me.
> >>>
> >>
> >> Minor from a MMU standpoint: the memory the CPU dereferenced is
> valid
> >> but no page table entry currently maps it. So, #PF in this case seems to be
> a
> >> 'minor'
> >> fault in MMU lingo, but it is still not expected.
> >>
> >>> Also, the thing is that I would not know how a PF in the long running
> >>> thread, with locked memory, With the call being close to the thread
> >>> entry point in a wait-for-condvar-loop, never using more than an
> >>> insignificant> amount of stack at this time should be possible.>
> >>
> >> Except if mapping an executable segment via dlopen() comes into play,
> >> affecting the page table. Only an assumption at this stage.
> >>
> >>> On the other hand, the non-RT thread loads a DSO and is stuck
> somewhere
> >> after allocating memory.
> >>> My guess would be that the PF ends up at the wrong thread.
> >>>
> >>
> >> As Jan pointed out, #PF are synchronously taken, synchronously handled.
> I
> >> really don't see how #PF handling could ever wander.
> >>
> >>>>>
> >>>>
> >>>> You refer to an older post describing a lockup, but this post
> >>>> describes an application crashing with a core dump. What made you
> >>>> draw the conclusion that the same bug would be at work?
> >>>
> >>> Same bug, different PTHREAD_WARNSW setting is my guess.
> >>> The underlying issue that a unrelated signal ends up to a RT thread.
> >>>
> >>>> Also, could you give some details
> >>>> regarding the
> >>>> following:
> >>>>
> >>>> - what do you mean by 'lockup' in this case? Can you still access the
> >>>> board or is there some runaway real-time code locking out everything
> >>>> else when this happens? My understanding is that this is no hard lock
> >>>> up otherwise the watchdog would have triggered. If this is a softer
> >>>> kind of lockup instead, what does /proc/xenomai/sched/stat tell you
> >>>> about the thread states after the problem occurred?
> >>>
> >>> This was a post-mortem, no access to /proc/xenomai/sched/stat
> anymore.
> >>> lockup means deadlock (the thread getting the signal holds a mutex,
> >>> but is stuck), Coredump happens if PTHREAD_WARNSW is enabled
> (means
> >> it asserts out before).
> >>>
> >>>> - did you determine that using the dynamic linker is required to
> >>>> trigger the bug yet? Or could you observe it without such interaction
> with
> >> dl?
> >>>
> >>> AFAIK, always occurred at the stage where we load a "configuration",
> and
> >> load DSOs.
> >>>
> >>>>
> >>>> - what is the typical size of your Xenomai thread stack? It defaults
> >>>> to 64k min with Xenomai 3.1.
> >>>
> >>> 1MB
> >>
> >> I would dig the following distinct issues:
> >>
> >> - why is #PF taken on an apparently innocuous instruction. dlopen(3)-
> >>> mmap(2) might be involved. With a simple test case, you could check the
> >> impact of loading/unloading DSOs on memory management for real-time
> >> threads running in parallel. Setting the WARNSW bit on for these threads
> >> would be required.
> >>
> >> - whether dealing with a signal adversely affects the wait-side of a
> Xenomai
> >> condvar. There is a specific trick to handle this in the Cobalt and libcobalt
> >> code, which is the reason for the wait_prologue / wait_epilogue dance in
> the
> >> implementation IIRC. Understanding why that thread receives a signal in
> the
> >> first place would help too. According to your description, this may not be
> >> directly due to taking #PF, but may be an indirect consequence of that
> event
> >> on sibling threads (propagation of a debug condition of some sort, such as
> >> those detected by CONFIG_XENO_OPT_DEBUG_MUTEX*).
> >>
> >> At any rate, you may want to enable the function ftracer, enabling
> >> conditional snapshots, e.g. when SIGXCPU is sent by the cobalt core.
> >> Guesswork with such bug is unlikely to uncover every aspect of the issue,
> >> hard data would be required to go to the bottom of it. With a bit of luck,
> that
> >> bug is not time-sensitive in a way that the overhead due to ftracing would
> >> paper over it.
> >>
> >> --
> >> Philippe.
> >
> > This aint exactly easy to reproduce, managed to get something that often
> reproduces just now.
> > Tracing however hides the issue, as well as disabling PTHREAD_WARNSW
> (could be just that this changes timing enough to make a difference).
> >
> >
> > I got a few instances where the thread loading DSOS is stuck in an omnious
> > __make_stacks_executable, that seemingly iterates through *all* thread
> stacks,
> > And calls __mprotect on them.
> >
> > If that’s the cause, and if the cobalt thread use that same stack,
> > and if the syscall does something funny like taking away write protection in-
> between,
> > then this could be the explanation (don’t know how this could ever be valid
> though).
> >
> > int
> > __make_stacks_executable (void **stack_endp)
> > {
> > /* First the main thread's stack. */
> > int err = _dl_make_stack_executable (stack_endp);
> > if (err != 0)
> > return err;
> >
> > #ifdef NEED_SEPARATE_REGISTER_STACK
> > const size_t pagemask = ~(__getpagesize () - 1);
> > #endif
> >
> > lll_lock (stack_cache_lock, LLL_PRIVATE);
> >
> > list_t *runp;
> > list_for_each (runp, &stack_used)
> > {
> > err = change_stack_perm (list_entry (runp, struct pthread, list)
>
> This code does not take away any protection, on the contrary this ensures
> that
> PROT_EXEC is set for all stacks along with read and write access, which is
> glibc's default for the x86_64 architecture.
I meant that it might have to do some non-atomic procedure, for example
when splitting up a continuous bigger mapping with the stack in the middle,
as the protection flags are now different.
> The fault is likely due to mm code fixing up those protections for the
> relevant page(s). It looks like such pages are force faulted-in, which would
> explain the #PF, and the SIGXCPU notification as a consequence. These are
> minor faults in the MMU management sense, so this is transparent for
> common applications.
I don’t know enough about the x86 (and don’t want to know), but this needs some
explanation. First, the DSOs don’t need executable stack
(build system did not care to add the .note.GNU-stack everywhere ), so this specific issue
can be worked around.
- I don’t understand why this is very timing sensitive, If page is marked to #PF (or removed)
Then this should fault predictable on the next access (I don’t share data on stack that Linux threads could run into an #PF instead)
- If that’s a non-atomic operation (perhaps only if the sparse tables need modification in a higher level), then I would expect
some sort of lazy locking (RCU?). Is this ending up in chaos as cores running Xenomai are "idle" for Linux, and pick up outdated data?
- Are/can such minor faults be handled in Xenomai? In other words is the WARNSW correct, or is
this actually just the check causing the issues?
Would it make sense to handle such minor faults in Xenomai (only demoting to Linux if necessary)?
> Not for those of us who do not want the application code to run
> into any page fault unfortunately.
>
> Loading DSOs while the real-time system is running just proved to be a bad
> idea it seems (did not check how other *libc implementations behave on
> dlopen() though).
glibc dlopens files on its own BTW, for nss plugins and encodings. Practically than means
you would need to check everything running (in non-rt threads), for dlopen and various
calls that could resolve names to uid/gid, do dns lookups, use iconv etc.
This is an issue of changing protection on existing mappings, or the mmap call in broader terms.
Knowing why and under what circumstances this causes trouble would be rather important
(would have been important some years ago when we *started* porting to Xenomai and picked solutions).
I could for ex. expect the kernel option CONFIG_COMPACTION causing similar issues (and pretty impossible to triage).
Norbert
________________________________
This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
ANDRITZ HYDRO GmbH
Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
Firmensitz/ Registered seat: Wien
Firmenbuchgericht/ Court of registry: Handelsgericht Wien
Firmenbuchnummer/ Company registration: FN 61833 g
DVR: 0605077
UID-Nr.: ATU14756806
Thank You
________________________________
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Still getting Deadlocks with condition variables
2020-06-15 9:06 ` Lange Norbert
@ 2020-06-15 10:02 ` Philippe Gerum
2020-06-15 11:52 ` Lange Norbert
0 siblings, 1 reply; 13+ messages in thread
From: Philippe Gerum @ 2020-06-15 10:02 UTC (permalink / raw
To: Lange Norbert, Xenomai (xenomai@xenomai.org),
'jan.kiszka@siemens.com'
On 6/15/20 11:06 AM, Lange Norbert wrote:
>>
>> This code does not take away any protection, on the contrary this ensures
>> that
>> PROT_EXEC is set for all stacks along with read and write access, which is
>> glibc's default for the x86_64 architecture.
>
> I meant that it might have to do some non-atomic procedure, for example
> when splitting up a continuous bigger mapping with the stack in the middle,
> as the protection flags are now different.
>
We are talking about mprotect(), not mmap().
>> The fault is likely due to mm code fixing up those protections for the
>> relevant page(s). It looks like such pages are force faulted-in, which would
>> explain the #PF, and the SIGXCPU notification as a consequence. These are
>> minor faults in the MMU management sense, so this is transparent for
>> common applications.
>
> I don’t know enough about the x86 (and don’t want to know), but this needs some
> explanation. First, the DSOs don’t need executable stack
> (build system did not care to add the .note.GNU-stack everywhere ), so this specific issue
> can be worked around.
>
> - I don’t understand why this is very timing sensitive, If page is marked to #PF (or removed)
> Then this should fault predictable on the next access (I don’t share data on stack that Linux threads could run into an #PF instead)
Faulting is what it does. It is predictable and synchronous, you seem to be
assuming that the fault is somehow async or proxied, it is not.
> - If that’s a non-atomic operation (perhaps only if the sparse tables need modification in a higher level), then I would expect
> some sort of lazy locking (RCU?). Is this ending up in chaos as cores running Xenomai are "idle" for Linux, and pick up outdated data?
I have no idea why you would bring up RCU in this picture, there is no
convoluted aspect in what happens. There is no chaos, only a plain simple #PF
event which unfortunately occurs as a result of running an apparently
innocuous regular operation which is loading a DSO. The reason for the #PF can
be explained, how it is dealt with is fine, the rt loop in your app just does
not like observing it for a legitimate reason.
> - Are/can such minor faults be handled in Xenomai? In other words is the WARNSW correct, or is
> this actually just the check causing the issues?
> Would it make sense to handle such minor faults in Xenomai (only demoting to Linux if necessary)?
>
>> Not for those of us who do not want the application code to run
>> into any page fault unfortunately.
>>
>> Loading DSOs while the real-time system is running just proved to be a bad
>> idea it seems (did not check how other *libc implementations behave on
>> dlopen() though).
>
> glibc dlopens files on its own BTW, for nss plugins and encodings. Practically than means
> you would need to check everything running (in non-rt threads), for dlopen and various
> calls that could resolve names to uid/gid, do dns lookups, use iconv etc.
>
The glibc is fortunately not dlopening DSOs at every corner. You mention very
specific features that would have to take place during the app init chores
instead, or at the very least in a way which is synchronized with a quiescent
state of the rt portion of the process.
> This is an issue of changing protection on existing mappings, or the mmap call in broader terms.
This is an issue with some of mprotect() side-effects.
> Knowing why and under what circumstances this causes trouble would be rather important
> (would have been important some years ago when we *started* porting to Xenomai and picked solutions).
>
I believe the whole thread has already framed the how and why fairly
precisely. With respect to knowing those things in advance, I can only
recommend that people who based their project on Xenomai over the past 15
years share their knowledge and experience by contributing documentation and
participating to the mailing list.
> I could for ex. expect the kernel option CONFIG_COMPACTION causing similar issues (and pretty impossible to triage).
>
CONFIG_COMPACTION is a known source of latency spots, just like transparent
huge pages are not going to be helpful, regardless of the underlying rt
infrastructure.
To sum up what we have all been saying, the problem is not about causing a PTE
miss due to mprotect() altering permissions on pages, but how we can handle
the minor fault from primary mode. On arm, arm64 and ppc, such fault can be
handled directly from primary mode. On x86, the inner MMU code which is in
charge of handling faults does not allow that, there is the need for switching
to secondary mode.
Regarding handling PTE misses directly from primary mode, this would be a mess
with x86: sharing the MMU management logic between Cobalt and the regular
linux mm sub-system which otherwise run totally asynchronously is something I
for one won't even try.
Without Xenomai, you would take a fault the same way, the difference is that
with Xenomai, the WARNSW notifier tells you so, issuing a signal, which may be
triggering a separate issue with condvars.
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: Still getting Deadlocks with condition variables
2020-06-15 10:02 ` Philippe Gerum
@ 2020-06-15 11:52 ` Lange Norbert
0 siblings, 0 replies; 13+ messages in thread
From: Lange Norbert @ 2020-06-15 11:52 UTC (permalink / raw
To: Philippe Gerum, Xenomai (xenomai@xenomai.org),
'jan.kiszka@siemens.com'
> -----Original Message-----
> From: Philippe Gerum <rpm@xenomai.org>
> Sent: Montag, 15. Juni 2020 12:03
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>;
> 'jan.kiszka@siemens.com' <jan.kiszka@siemens.com>
> Subject: Re: Still getting Deadlocks with condition variables
>
> NON-ANDRITZ SOURCE: BE CAUTIOUS WITH CONTENT, LINKS OR
> ATTACHMENTS.
>
>
> On 6/15/20 11:06 AM, Lange Norbert wrote:
> >>
> >> This code does not take away any protection, on the contrary this
> >> ensures that PROT_EXEC is set for all stacks along with read and
> >> write access, which is glibc's default for the x86_64 architecture.
> >
> > I meant that it might have to do some non-atomic procedure, for
> > example when splitting up a continuous bigger mapping with the stack
> > in the middle, as the protection flags are now different.
> >
>
> We are talking about mprotect(), not mmap().
My bad, yes.
>
> >> The fault is likely due to mm code fixing up those protections for
> >> the relevant page(s). It looks like such pages are force faulted-in,
> >> which would explain the #PF, and the SIGXCPU notification as a
> >> consequence. These are minor faults in the MMU management sense, so
> >> this is transparent for common applications.
> >
> > I don’t know enough about the x86 (and don’t want to know), but this
> > needs some explanation. First, the DSOs don’t need executable stack
> > (build system did not care to add the .note.GNU-stack everywhere ), so
> > this specific issue can be worked around.
> >
> > - I don’t understand why this is very timing sensitive, If page is marked to
> #PF (or removed)
> > Then this should fault predictable on the next access (I don’t
> > share data on stack that Linux threads could run into an #PF instead)
>
> Faulting is what it does. It is predictable and synchronous, you seem to be
> assuming that the fault is somehow async or proxied, it is not.
It does happen rather sparsely, affected by small changes in unrelated code,
Loading a DSO (which requires executable stack) might trigger a #PF or not.
Nothing but the respective RT-Thread is accessing it's *private* stack, the pagefault
Happens at a callq.
If that stack accesss is to cause a #PF, then the code would run into it *every* time.
Yet it does not, its rather really hard to get this to reproduce.
> > - If that’s a non-atomic operation (perhaps only if the sparse tables need
> modification in a higher level), then I would expect
> > some sort of lazy locking (RCU?). Is this ending up in chaos as cores
> running Xenomai are "idle" for Linux, and pick up outdated data?
>
> I have no idea why you would bring up RCU in this picture, there is no
> convoluted aspect in what happens. There is no chaos, only a plain simple
> #PF event which unfortunately occurs as a result of running an apparently
> innocuous regular operation which is loading a DSO. The reason for the #PF
> can be explained, how it is dealt with is fine, the rt loop in your app just does
> not like observing it for a legitimate reason.
I am asking a question, I assume pagetables need to be reallocated under
some circumstances. I understand a #PF *has to happen* according to your explanation,
and I don’t know *where it happens if I don’t observe the RT task switching*.
The faults are very timing sensitive.
So I could imagine the multilevel page-table map looking like this (dunt know how many levels x86 is using nowadays, but that’s beside the point):
[first level] -> [second level] -> [stack mapping]
if the mprotect syscall changes just the *private* stack mapping, then the RT thread will always fault - (not what I observe).
if the syscall modifies lower levels, then any thread can hit the PF, and if it's not a RT thread then no #PF will be observed (by WARNSW).
This is my conjecture, if that’s true, then the question is modified to:
- under what circumstances can this appear?
>
> > - Are/can such minor faults be handled in Xenomai? In other words is the
> WARNSW correct, or is
> > this actually just the check causing the issues?
> > Would it make sense to handle such minor faults in Xenomai (only
> demoting to Linux if necessary)?
> >
> >> Not for those of us who do not want the application code to run into
> >> any page fault unfortunately.
> >>
> >> Loading DSOs while the real-time system is running just proved to be
> >> a bad idea it seems (did not check how other *libc implementations
> >> behave on
> >> dlopen() though).
> >
> > glibc dlopens files on its own BTW, for nss plugins and encodings.
> > Practically than means you would need to check everything running (in
> > non-rt threads), for dlopen and various calls that could resolve names to
> uid/gid, do dns lookups, use iconv etc.
> >
>
> The glibc is fortunately not dlopening DSOs at every corner. You mention
> very specific features that would have to take place during the app init
> chores instead, or at the very least in a way which is synchronized with a
> quiescent state of the rt portion of the process.
Still important to know, so you could for ex. "prime" the library by doing those calls
up-front.
> > This is an issue of changing protection on existing mappings, or the mmap
> call in broader terms.
>
> This is an issue with some of mprotect() side-effects.
>
> > Knowing why and under what circumstances this causes trouble would be
> > rather important (would have been important some years ago when we
> *started* porting to Xenomai and picked solutions).
> >
>
> I believe the whole thread has already framed the how and why fairly
> precisely. With respect to knowing those things in advance, I can only
> recommend that people who based their project on Xenomai over the past
> 15 years share their knowledge and experience by contributing
> documentation and participating to the mailing list.
You can't expect "users" of Xenomai having your indepth knowledge,
I can't describe the inner workings and I hoped I could keep myself distanced
from the 24M LOC Linux kernel.
>
> > I could for ex. expect the kernel option CONFIG_COMPACTION causing
> similar issues (and pretty impossible to triage).
> >
>
> CONFIG_COMPACTION is a known source of latency spots, just like
> transparent huge pages are not going to be helpful, regardless of the
> underlying rt infrastructure.
COMPACTION could help for long running applications, you don’t want
latency, but you want heavy fragmentation even less.
(I'd guess that kind of latency is not really comparable with a RT thread
getting demoted)
>
> To sum up what we have all been saying, the problem is not about causing a
> PTE miss due to mprotect() altering permissions on pages, but how we can
> handle the minor fault from primary mode. On arm, arm64 and ppc, such
> fault can be handled directly from primary mode. On x86, the inner MMU
> code which is in charge of handling faults does not allow that, there is the
> need for switching to secondary mode.
>
> Regarding handling PTE misses directly from primary mode, this would be a
> mess with x86: sharing the MMU management logic between Cobalt and the
> regular linux mm sub-system which otherwise run totally asynchronously is
> something I for one won't even try.
Ok, sounds perfectly messy (the x86 architecture that is).
> Without Xenomai, you would take a fault the same way, the difference is
> that with Xenomai, the WARNSW notifier tells you so, issuing a signal, which
> may be triggering a separate issue with condvars.
So WARNSW works correctly, thanks for clearing that up.
Norbert
________________________________
This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.
ANDRITZ HYDRO GmbH
Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation
Firmensitz/ Registered seat: Wien
Firmenbuchgericht/ Court of registry: Handelsgericht Wien
Firmenbuchnummer/ Company registration: FN 61833 g
DVR: 0605077
UID-Nr.: ATU14756806
Thank You
________________________________
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2020-06-15 11:52 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-06-05 14:36 Still getting Deadlocks with condition variables Lange Norbert
2020-06-05 15:40 ` Jan Kiszka
2020-06-08 9:48 ` Lange Norbert
2020-06-08 10:09 ` Jan Kiszka
2020-06-08 13:28 ` Lange Norbert
2020-06-07 20:15 ` Philippe Gerum
2020-06-08 10:08 ` Lange Norbert
2020-06-08 14:17 ` Philippe Gerum
2020-06-09 17:10 ` Lange Norbert
2020-06-10 16:48 ` Philippe Gerum
2020-06-15 9:06 ` Lange Norbert
2020-06-15 10:02 ` Philippe Gerum
2020-06-15 11:52 ` Lange Norbert
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.