From: Daniel Lezcano <daniel.lezcano@linaro.org>
To: Sebastian Parschauer <sebastian.riemer@profitbricks.com>,
"Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: linux-pm@vger.kernel.org
Subject: Re: cpuidle: Kernel panics with AMD Opteron 6300 entering C2 - clock related
Date: Thu, 18 Jun 2015 15:23:56 +0200 [thread overview]
Message-ID: <5582C66C.1060301@linaro.org> (raw)
In-Reply-To: <5582A2DC.7060001@profitbricks.com>
On 06/18/2015 12:52 PM, Sebastian Parschauer wrote:
> On 18.06.2015 11:22, Daniel Lezcano wrote:
>> On 06/17/2015 12:06 PM, Sebastian Parschauer wrote:
>>> Hi cpuidle maintainers,
>>>
>>> we notice kernel panics with CPUs from the AMD Opteron 6300 series and
>>> kernel 3.12 when entering C2. In that C-state the clock is shut down but
>>> the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source
>>> for performance as our servers host KVM VMs. During the panics
>>> interrupts are enabled again and the timer interrupt corrupts the
>>> instruction pointer and/or the stack pointer.
>>>
>>> Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2?
>>> Or how to fix this?
>>
>> Did you try the flag ? Does it fix it ?
>
> Thanks for your reply. Unfortunately we can't roll out new kernels fast
> (VMs have to be migrated). But we've disabled the C2 state via sysfs for
> all CPU cores and all servers and had one more kernel panic with the
> same call trace although C2 was (or should have been) disabled. We use
> the menu governor and a v3.12.40 kernel.
>
> It's strange to me coming into the same code path with state index 2 as
> parameter again. I think I'll prepare a kernel with some debug messages
> when transitioning from one state to another and deploy it to a test system.
It is weird you disabled the state index 2 and the system enters with
this index again.
Are you sure you disabled effectively for all cores on the system this
state ?
Furthermore, if I am not wrong the C state on AMD differs a bit from the
C-state intel's semantic.
The firmware will put the cluster down if all core go to the C1 state, no ?
> Is there any better method to debug the cpuidle driver?
You can try by passing to the kernel command line:
processor.max_cstate=1
Note, that does not guarantee the firmware won't promote to a deeper
idle state.
If the kernel panics again, may be in the BIOS, there is an option to
set max idle states for the firmware.
> How do you guys test it?
On the x86 platform, most of the magic is in the firmware, so if there
is a bug there, hmm ... that will be hard to spot.
> Can we provide any missing additional information?
>
> Maybe something else corrupts the memory in an interrupt and the cpuidle
> driver is just the one noticing an unrelated problem.
>>> ==========
>>> Additional debug info:
>>>
>>> BUG: unable to handle kernel NULL pointer dereference at (null)
>>> IP: [< (null)>] (null)
>>> ...
>>> Call trace:
>>> [<ffffffff815af9b5>] cpuidle_idle_call+0xc5/0x150
>>> [<ffffffff8100b529>] arch_cpu_idle+0x9/0x20
>>> [<ffffffff81092e6f>] cpu_startup_entry+0xaf/0x240
>>> [<ffffffff8102df4b>] start_secondary+0x1db/0x240
>>>
>>> The CPUs provide three C-states:
>>> 0: POLL
>>> 1: C1
>>> 2: C2
>>>
>>> C2 information from the crash dump:
>>>
>>>> {
>>>> name = "C2\000\000\000\000\000\000\000\000\000\000\000\000\000",
>>>> desc = "ACPI IOPORT
>>>> 0x815\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
>>>> flags = 1,
>>>> exit_latency = 100,
>>>> power_usage = 0,
>>>> target_residency = 200,
>>>> disabled = false,
>>>> enter = 0xffffffffa00ab026 <acpi_idle_enter_simple>,
>>>> enter_dead = 0xffffffffa00aa39c <acpi_idle_play_dead>
>>>> }
>>>
>>> Assembly level analysis:
>>>
>>>> RDX: 0000000225c17d03
>>>
>>> So EDX is 00000002 and that's the entered state C2.
>>>
>>>> RDI: ffffffff81c15540
>>>> ..
>>>> crash> info symbol 0xffffffff81c15540
>>>> clocksource_tsc in section .data
>>>>
>>>> crash> disassemble cpuidle_enter_state
>>>> ...
>>>> 0xffffffff815af5fc <+60>: callq 0xffffffff8109b360 <ktime_get>
>>>> 0xffffffff815af601 <+65>: sti
>>>> 0xffffffff815af602 <+66>: sub %r13,%rax <- here rdi still
>>>> points to clocksource_tsc
>>>> 0xffffffff815af605 <+69>: mov %rax,%rdi <- rdi is
>>>> overwritten by the ktime_get return address
>>
>>
>
--
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
prev parent reply other threads:[~2015-06-18 13:23 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-06-17 10:06 cpuidle: Kernel panics with AMD Opteron 6300 entering C2 - clock related Sebastian Parschauer
2015-06-18 9:22 ` Daniel Lezcano
2015-06-18 10:52 ` Sebastian Parschauer
2015-06-18 11:21 ` Sebastian Parschauer
2015-06-18 13:28 ` Daniel Lezcano
2015-06-18 14:09 ` Sebastian Parschauer
2015-06-18 13:23 ` Daniel Lezcano [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5582C66C.1060301@linaro.org \
--to=daniel.lezcano@linaro.org \
--cc=linux-pm@vger.kernel.org \
--cc=rjw@rjwysocki.net \
--cc=sebastian.riemer@profitbricks.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.