From: "François Legal" <devel@thom.fr.eu.org>
To: "임준수 수석 두산로보틱스" <junesoo.im@doosan.com>
Cc: "xenomai@lists.linux.dev" <xenomai@lists.linux.dev>,
"박경훈 팀장 두산로보틱스" <kyunghoon2.park@doosan.com>
Subject: RE: Linux system got freezing when "Xenomai App" started (or terminated).
Date: Mon, 15 Apr 2024 15:21:27 +0200 [thread overview]
Message-ID: <11eb-661d2a00-1d-4baabc00@149783243> (raw)
In-Reply-To: <SL2P216MB2274F54CFDA8A64A60C75671E0092@SL2P216MB2274.KORP216.PROD.OUTLOOK.COM>
Le Lundi, Avril 15, 2024 12:37 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:
> Okay, I answered your question below.
>
>
> So you mean xenomai watchdog is already active, but still the system hangs ?
> -> Yes. One thing is certain, when testing the device reboot, only system hang occurs.
> At this time, the watchdog did not occur.
>
> And the system hangs only when you run the RT application ?
> -> Yes. System hang does not occur unless you run the robot program (with Xenomai).
> For reference, the robot program is automatically executed by Linux rc.local. at start.
>
> Do your application rely on custom kernel drivers ?
> -> Yes. The robot program has EtherCAT functionality. And the EtherCAT function uses "atemsys.ko".
>
> Is JTAG debugging an option ?
> -> Unfortunately, there isn't any.
>
>
> I'll wait for your opinion.
> Thanks.
> Best Regards.
>
> -----Original Message-----
> From: François Legal <devel@thom.fr.eu.org>
> Sent: Monday, April 15, 2024 7:16 PM
> To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 <kyunghoon2.park@doosan.com>
> Subject: RE: Linux system got freezing when "Xenomai App" started (or terminated).
>
> 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
>
> CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
>
> Le Lundi, Avril 15, 2024 12:07 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:
>
> > Dear François Legal.
> >
> >
> > Thank you so much for your quick reply!!
> >
> >
> > I checked "CONFIG_XENO_OPT_WATCHDOG" in your message.
> >
> >
> > (This e-mail type is not HTML. So, I write down some informations.) I
> > search "CONFIG_XENO_OPT_WATCHDOG" option in kernel's ".config" file
> > -------------------------------------------------
> > $KERNEL_ROOT) grep CONFIG_XENO_OPT_WATCHDOG .config
> > Output> CONFIG_XENO_OPT_WATCHDOG=y
> > Output> CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4
> >
> > Also, Xenomai's watchdog works well.
> > Because Robot program is sometimes terminated by the watchdog.
> >
> > Xenomai watchdog is also a problem, but right now we are focusing on system freezing.
> >
> >
> > Is there anything else you can tell me?
> >
> > Thanks.
> > Best Regards.
> >
> >
> > -----Original Message-----
> > From: François Legal <devel@thom.fr.eu.org>
> > Sent: Monday, April 15, 2024 6:49 PM
> > To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> > Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스
> > <kyunghoon2.park@doosan.com>
> > Subject: Re: Linux system got freezing when "Xenomai App" started (or terminated).
> >
> > 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
> >
> > CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
> >
> > Le Lundi, Avril 15, 2024 11:23 CEST, ÀÓÁؼö(Junesoo Im) ¼ö¼® µÎ»ê·Îº¸Æ½½º <junesoo.im@doosan.com> a écrit:
> >
> > > Hello. I am a Linux Application Developer.
> > >
> > >
> > > Application uses Xenomai 3.1x.
> > > The machine environment is 64bit and kernel 5.4.255.
> > >
> > > Repeat machine reboot at 55 second intervals.
> > > Most of the time it works fine, but sometimes the system freezes.
> > > The mouse doesn't move and the keyboard doesn't work.
> > > The screen also freezes.
> > > There are no special messages in the kernel log.
> > > The same goes for syslog.
> > >
> > >
> > >
> > > So I'm running some tests.
> > > If you test without running the application, there is no problem with the system.
> > > Problems occur when the application runs.
> > > However, I don't see any special messages in the application log.
> > >
> > >
> > > I haven't found the cause of Freezing yet.
> > > I am currently viewing the ¡°https://secure-web.cisco.com/1XLvVz-8MEZDdYZfDgbDD6YEaNOjJeaQ6a47KjHffv0KQbTypxooAIDbLz8urSXEI6vRlttlL27KSW_mfF0ENueummSZ38ucNEvjty5-ZVBy60tljLbXzBNlB5cgnIzhT8moSYlzkUphFAcsDj-ghcCQ1QnKn1xIcEJho6xMExZ2wArvZaBvOJ0fcuxDro1q4C6PwSin_iyNBt03iXUJuhHltWbGv9K-oth9mI-QTIiNDu3wG7Xej8rdBFNijChtrgaOlUBzUBAxaipEmzDHdBpRA1HEHvIm523iNHg7XsdjgKftK0Od1fpZmEZRtSn1N/https%3A%2F%2Fv3.xenomai.org%2Ftroubleshooting%2F¡± site.
> > > I would like to test some of the things mentioned on this site.
> > > If possible, I would like you to look at this site together.
> > >
> > >
> > > This site is talking about Kernel configuration that needs to be removed.
> > > "CONFIG_KGDB"
> > > This option is currently enabled, so I removed it and tested it.
> > > But Freezing occurred.
> > > A problem has occurred, but should this option be removed?
> > >
> > >
> > > This site talks about "Freeze or machine lockup".
> > > 1. ¡°Stack space overflow issue now biting sim real-time kernel
> > > thread?¡± 2. ¡°Spurious delay/timeout values computed by the
> > > application (specifically: too short)¡± 3. ¡°A case of freeze is a
> > > system call called in a loop which fails without its return value
> > > being properly checked.¡±
> > >
> > >
> > > There are three explanations: I have a question about the first one.
> > > How many threads can the xenomai program create?
> > > What is the maximum stack size per thread in the xenomai program?
> > > Our program has at least 22 xenomai threads.
> > > The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
> > >
> > >
> > >
> > > The question is complicated and long, but I desperately need help.
> > > Please, reply.
> > > Thank you.
> >
> >
> > Hi
> >
> > have you tried enabling CONFIG_XENO_OPT_WATCHDOG to enable detection of runaway RT threads.
> > This might help you diagnosing the problem.
> >
> > François
> >
> >
>
> So you mean xenomai watchdog is already active, but still the system hangs ?
> And the system hangs only when you run the RT application ?
> Do your application rely on custom kernel drivers ?
> Is JTAG debugging an option ?
>
> François
>
Well, I can't say for sure, but IMHO, if xenomai watchdog does not trigger, I guess we can say it's not a userland runaway thread that causes the hang.
So this probably means problemsat kernel level. I don't know this kernel module your mentionning, but if it is proprietary, could you try to exercise it's functions with a simple test program (that would idealy reproduce the real application behaviour regarding driver access) and see if the hang can occur in that configuration.
Apart from that, and without JTAG debug possibility, I can not give any more advice on this. Sorry.
François
next prev parent reply other threads:[~2024-04-15 13:21 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-15 0:43 Linux system got freezing when "Xenomai App" started (or terminated) 임준수(Junesoo Im) 수석 두산로보틱스
2024-04-15 9:23 ` 임준수(Junesoo Im) 수석 두산로보틱스
2024-04-15 9:48 ` François Legal
2024-04-15 10:07 ` 임준수(Junesoo Im) 수석 두산로보틱스
2024-04-15 10:16 ` François Legal
2024-04-15 10:37 ` 임준수(Junesoo Im) 수석 두산로보틱스
2024-04-15 13:21 ` François Legal [this message]
2024-04-29 8:49 ` 임준수(Junesoo Im) 수석 두산로보틱스
2024-04-15 13:16 ` Jan Kiszka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=11eb-661d2a00-1d-4baabc00@149783243 \
--to=devel@thom.fr.eu.org \
--cc=junesoo.im@doosan.com \
--cc=kyunghoon2.park@doosan.com \
--cc=xenomai@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.