* [PATCH] arm: Fix backtrace generation when IPI is masked
@ 2015-09-14 9:18 Daniel Thompson
2015-09-15 11:05 ` [PATCH v2] " Daniel Thompson
2015-09-15 14:40 ` [PATCH v3] " Daniel Thompson
0 siblings, 2 replies; 9+ messages in thread
From: Daniel Thompson @ 2015-09-14 9:18 UTC (permalink / raw
To: Russell King
Cc: Daniel Thompson, linux-arm-kernel, linux-kernel, patches,
linaro-kernel, Thomas Gleixner
Currently on ARM when <SysRq-L> is triggered from an interrupt handler
(e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
seconds with interrupts masked before issuing a backtrace for every CPU
except itself.
The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
basic support for on-demand backtrace of other CPUs") does not work
correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
is used to generate the backtrace on all CPUs but cannot preempt the
current calling context.
This can be fixed by detecting that the calling context cannot be
preempted and issuing the backtrace directly in this case. Some small
changes to the generic code are required to support this.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
arch/arm/kernel/smp.c | 7 +++++++
lib/nmi_backtrace.c | 5 ++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 48185a773852..4d8a80328c74 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -748,6 +748,13 @@ core_initcall(register_cpufreq_notifier);
static void raise_nmi(cpumask_t *mask)
{
+ /*
+ * Generate the backtrace directly if we are running in a
+ * calling context that is not preemptible by the backtrace IPI.
+ */
+ if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
+ nmi_cpu_backtrace(NULL);
+
smp_cross_call(mask, IPI_CPU_BACKTRACE);
}
diff --git a/lib/nmi_backtrace.c b/lib/nmi_backtrace.c
index 88d3d32e5923..be0466a80d0b 100644
--- a/lib/nmi_backtrace.c
+++ b/lib/nmi_backtrace.c
@@ -149,7 +149,10 @@ bool nmi_cpu_backtrace(struct pt_regs *regs)
/* Replace printk to write into the NMI seq */
this_cpu_write(printk_func, nmi_vprintk);
pr_warn("NMI backtrace for cpu %d\n", cpu);
- show_regs(regs);
+ if (regs)
+ show_regs(regs);
+ else
+ dump_stack();
this_cpu_write(printk_func, printk_func_save);
cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
--
2.4.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2] arm: Fix backtrace generation when IPI is masked
2015-09-14 9:18 [PATCH] arm: Fix backtrace generation when IPI is masked Daniel Thompson
@ 2015-09-15 11:05 ` Daniel Thompson
2015-09-15 11:30 ` Russell King - ARM Linux
2015-09-15 14:40 ` [PATCH v3] " Daniel Thompson
1 sibling, 1 reply; 9+ messages in thread
From: Daniel Thompson @ 2015-09-15 11:05 UTC (permalink / raw
To: Russell King
Cc: Daniel Thompson, linux-arm-kernel, linux-kernel, patches,
linaro-kernel, Thomas Gleixner, Hillf Danton
Currently on ARM when <SysRq-L> is triggered from an interrupt handler
(e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
seconds with interrupts masked before issuing a backtrace for every CPU
except itself.
The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
basic support for on-demand backtrace of other CPUs") does not work
correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
is used to generate the backtrace on all CPUs but cannot preempt the
current calling context.
This can be fixed by detecting that the calling context cannot be
preempted and issuing the backtrace directly in this case. Issuing
directly leaves us without any pt_regs to pass to nmi_cpu_backtrace()
so we also modify the generic code to call dump_stack() when its
argument is NULL.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
Notes:
Changes in v2:
* Improved commit message to better describe the changes to the generic
code (Hillf Danton).
arch/arm/kernel/smp.c | 7 +++++++
lib/nmi_backtrace.c | 5 ++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 48185a773852..4d8a80328c74 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -748,6 +748,13 @@ core_initcall(register_cpufreq_notifier);
static void raise_nmi(cpumask_t *mask)
{
+ /*
+ * Generate the backtrace directly if we are running in a
+ * calling context that is not preemptible by the backtrace IPI.
+ */
+ if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
+ nmi_cpu_backtrace(NULL);
+
smp_cross_call(mask, IPI_CPU_BACKTRACE);
}
diff --git a/lib/nmi_backtrace.c b/lib/nmi_backtrace.c
index 88d3d32e5923..be0466a80d0b 100644
--- a/lib/nmi_backtrace.c
+++ b/lib/nmi_backtrace.c
@@ -149,7 +149,10 @@ bool nmi_cpu_backtrace(struct pt_regs *regs)
/* Replace printk to write into the NMI seq */
this_cpu_write(printk_func, nmi_vprintk);
pr_warn("NMI backtrace for cpu %d\n", cpu);
- show_regs(regs);
+ if (regs)
+ show_regs(regs);
+ else
+ dump_stack();
this_cpu_write(printk_func, printk_func_save);
cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
--
2.4.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v2] arm: Fix backtrace generation when IPI is masked
2015-09-15 11:05 ` [PATCH v2] " Daniel Thompson
@ 2015-09-15 11:30 ` Russell King - ARM Linux
2015-09-15 13:15 ` Daniel Thompson
0 siblings, 1 reply; 9+ messages in thread
From: Russell King - ARM Linux @ 2015-09-15 11:30 UTC (permalink / raw
To: Daniel Thompson
Cc: linux-arm-kernel, linux-kernel, patches, linaro-kernel,
Thomas Gleixner, Hillf Danton
On Tue, Sep 15, 2015 at 12:05:12PM +0100, Daniel Thompson wrote:
> Currently on ARM when <SysRq-L> is triggered from an interrupt handler
> (e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
> seconds with interrupts masked before issuing a backtrace for every CPU
> except itself.
>
> The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
> basic support for on-demand backtrace of other CPUs") does not work
> correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
> is used to generate the backtrace on all CPUs but cannot preempt the
> current calling context.
This patch needs a little more work - what happens to the IPI_CPU_BACKTRACE
we've sent to ourselves? (It fires after the interrupt handler for the
UART/kbd has finished.) It ought to be masked out if we're going to
handle it a different way.
--
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2] arm: Fix backtrace generation when IPI is masked
2015-09-15 11:30 ` Russell King - ARM Linux
@ 2015-09-15 13:15 ` Daniel Thompson
2015-09-15 13:42 ` Russell King - ARM Linux
0 siblings, 1 reply; 9+ messages in thread
From: Daniel Thompson @ 2015-09-15 13:15 UTC (permalink / raw
To: Russell King - ARM Linux
Cc: linux-arm-kernel, linux-kernel, patches, linaro-kernel,
Thomas Gleixner, Hillf Danton
On 15/09/15 12:30, Russell King - ARM Linux wrote:
> On Tue, Sep 15, 2015 at 12:05:12PM +0100, Daniel Thompson wrote:
>> Currently on ARM when <SysRq-L> is triggered from an interrupt handler
>> (e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
>> seconds with interrupts masked before issuing a backtrace for every CPU
>> except itself.
>>
>> The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
>> basic support for on-demand backtrace of other CPUs") does not work
>> correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
>> is used to generate the backtrace on all CPUs but cannot preempt the
>> current calling context.
>
> This patch needs a little more work - what happens to the IPI_CPU_BACKTRACE
> we've sent to ourselves? (It fires after the interrupt handler for the
> UART/kbd has finished.) It ought to be masked out if we're going to
> handle it a different way.
Actually it already gets masked out. The argument to raise_nmi() points
to a data structure owned by the backtrace library functions and this
structure if altered during the execution of nmi_cpu_backtrace() to
clear the calling CPU.
I had originally planned to use cpumask_test_and_clear_cpu() for the
conditional branch but that would be broken because nmi_cpu_backtrace()
would become a nop if we clear anything from the mask before calling it!
I guess I should add a comment about this to save us from broken but
"obviously correct" cleanups in the future...
Daniel.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2] arm: Fix backtrace generation when IPI is masked
2015-09-15 13:15 ` Daniel Thompson
@ 2015-09-15 13:42 ` Russell King - ARM Linux
0 siblings, 0 replies; 9+ messages in thread
From: Russell King - ARM Linux @ 2015-09-15 13:42 UTC (permalink / raw
To: Daniel Thompson
Cc: linux-arm-kernel, linux-kernel, patches, linaro-kernel,
Thomas Gleixner, Hillf Danton
On Tue, Sep 15, 2015 at 02:15:10PM +0100, Daniel Thompson wrote:
> Actually it already gets masked out. The argument to raise_nmi() points to a
> data structure owned by the backtrace library functions and this structure
> if altered during the execution of nmi_cpu_backtrace() to clear the calling
> CPU.
>
> I had originally planned to use cpumask_test_and_clear_cpu() for the
> conditional branch but that would be broken because nmi_cpu_backtrace()
> would become a nop if we clear anything from the mask before calling it!
>
> I guess I should add a comment about this to save us from broken but
> "obviously correct" cleanups in the future...
Absolutely.
--
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3] arm: Fix backtrace generation when IPI is masked
2015-09-14 9:18 [PATCH] arm: Fix backtrace generation when IPI is masked Daniel Thompson
2015-09-15 11:05 ` [PATCH v2] " Daniel Thompson
@ 2015-09-15 14:40 ` Daniel Thompson
2015-09-16 2:43 ` Hillf Danton
` (2 more replies)
1 sibling, 3 replies; 9+ messages in thread
From: Daniel Thompson @ 2015-09-15 14:40 UTC (permalink / raw
To: Russell King
Cc: Daniel Thompson, linux-arm-kernel, linux-kernel, patches,
linaro-kernel, Thomas Gleixner, Hillf Danton
Currently on ARM when <SysRq-L> is triggered from an interrupt handler
(e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
seconds with interrupts masked before issuing a backtrace for every CPU
except itself.
The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
basic support for on-demand backtrace of other CPUs") does not work
correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
is used to generate the backtrace on all CPUs but cannot preempt the
current calling context.
This can be fixed by detecting that the calling context cannot be
preempted and issuing the backtrace directly in this case. Issuing
directly leaves us without any pt_regs to pass to nmi_cpu_backtrace()
so we also modify the generic code to call dump_stack() when its
argument is NULL.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
Notes:
Changes in v3:
* Added comments to describe how raise_nmi() and nmi_cpu_backtrace()
interact with backtrace_mask (Russell King).
Changes in v2:
* Improved commit message to better describe the changes to the generic
code (Hillf Danton).
arch/arm/kernel/smp.c | 9 +++++++++
lib/nmi_backtrace.c | 11 ++++++++++-
2 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 48185a773852..0c4e7fdb9636 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -748,6 +748,15 @@ core_initcall(register_cpufreq_notifier);
static void raise_nmi(cpumask_t *mask)
{
+ /*
+ * Generate the backtrace directly if we are running in a calling
+ * context that is not preemptible by the backtrace IPI. Note
+ * that nmi_cpu_backtrace() automatically removes the current cpu
+ * from mask.
+ */
+ if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
+ nmi_cpu_backtrace(NULL);
+
smp_cross_call(mask, IPI_CPU_BACKTRACE);
}
diff --git a/lib/nmi_backtrace.c b/lib/nmi_backtrace.c
index 88d3d32e5923..6019c53c669e 100644
--- a/lib/nmi_backtrace.c
+++ b/lib/nmi_backtrace.c
@@ -43,6 +43,12 @@ static void print_seq_line(struct nmi_seq_buf *s, int start, int end)
printk("%.*s", (end - start) + 1, buf);
}
+/*
+ * When raise() is called it will be is passed a pointer to the
+ * backtrace_mask. Architectures that call nmi_cpu_backtrace()
+ * directly from their raise() functions may rely on the mask
+ * they are passed being updated as a side effect of this call.
+ */
void nmi_trigger_all_cpu_backtrace(bool include_self,
void (*raise)(cpumask_t *mask))
{
@@ -149,7 +155,10 @@ bool nmi_cpu_backtrace(struct pt_regs *regs)
/* Replace printk to write into the NMI seq */
this_cpu_write(printk_func, nmi_vprintk);
pr_warn("NMI backtrace for cpu %d\n", cpu);
- show_regs(regs);
+ if (regs)
+ show_regs(regs);
+ else
+ dump_stack();
this_cpu_write(printk_func, printk_func_save);
cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
--
2.4.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v3] arm: Fix backtrace generation when IPI is masked
2015-09-15 14:40 ` [PATCH v3] " Daniel Thompson
@ 2015-09-16 2:43 ` Hillf Danton
2015-09-22 10:59 ` Thomas Gleixner
2015-10-03 15:40 ` Russell King - ARM Linux
2 siblings, 0 replies; 9+ messages in thread
From: Hillf Danton @ 2015-09-16 2:43 UTC (permalink / raw
To: 'Daniel Thompson', 'Russell King'
Cc: linux-arm-kernel, linux-kernel, patches, linaro-kernel,
'Thomas Gleixner'
>
> Currently on ARM when <SysRq-L> is triggered from an interrupt handler
> (e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
> seconds with interrupts masked before issuing a backtrace for every CPU
> except itself.
>
> The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
> basic support for on-demand backtrace of other CPUs") does not work
> correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
> is used to generate the backtrace on all CPUs but cannot preempt the
> current calling context.
>
> This can be fixed by detecting that the calling context cannot be
> preempted and issuing the backtrace directly in this case. Issuing
> directly leaves us without any pt_regs to pass to nmi_cpu_backtrace()
> so we also modify the generic code to call dump_stack() when its
> argument is NULL.
>
> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
> ---
Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
>
> Notes:
> Changes in v3:
>
> * Added comments to describe how raise_nmi() and nmi_cpu_backtrace()
> interact with backtrace_mask (Russell King).
>
> Changes in v2:
>
> * Improved commit message to better describe the changes to the generic
> code (Hillf Danton).
>
> arch/arm/kernel/smp.c | 9 +++++++++
> lib/nmi_backtrace.c | 11 ++++++++++-
> 2 files changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
> index 48185a773852..0c4e7fdb9636 100644
> --- a/arch/arm/kernel/smp.c
> +++ b/arch/arm/kernel/smp.c
> @@ -748,6 +748,15 @@ core_initcall(register_cpufreq_notifier);
>
> static void raise_nmi(cpumask_t *mask)
> {
> + /*
> + * Generate the backtrace directly if we are running in a calling
> + * context that is not preemptible by the backtrace IPI. Note
> + * that nmi_cpu_backtrace() automatically removes the current cpu
> + * from mask.
> + */
> + if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
> + nmi_cpu_backtrace(NULL);
> +
> smp_cross_call(mask, IPI_CPU_BACKTRACE);
> }
>
> diff --git a/lib/nmi_backtrace.c b/lib/nmi_backtrace.c
> index 88d3d32e5923..6019c53c669e 100644
> --- a/lib/nmi_backtrace.c
> +++ b/lib/nmi_backtrace.c
> @@ -43,6 +43,12 @@ static void print_seq_line(struct nmi_seq_buf *s, int start, int end)
> printk("%.*s", (end - start) + 1, buf);
> }
>
> +/*
> + * When raise() is called it will be is passed a pointer to the
> + * backtrace_mask. Architectures that call nmi_cpu_backtrace()
> + * directly from their raise() functions may rely on the mask
> + * they are passed being updated as a side effect of this call.
> + */
> void nmi_trigger_all_cpu_backtrace(bool include_self,
> void (*raise)(cpumask_t *mask))
> {
> @@ -149,7 +155,10 @@ bool nmi_cpu_backtrace(struct pt_regs *regs)
> /* Replace printk to write into the NMI seq */
> this_cpu_write(printk_func, nmi_vprintk);
> pr_warn("NMI backtrace for cpu %d\n", cpu);
> - show_regs(regs);
> + if (regs)
> + show_regs(regs);
> + else
> + dump_stack();
> this_cpu_write(printk_func, printk_func_save);
>
> cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
> --
> 2.4.3
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3] arm: Fix backtrace generation when IPI is masked
2015-09-15 14:40 ` [PATCH v3] " Daniel Thompson
2015-09-16 2:43 ` Hillf Danton
@ 2015-09-22 10:59 ` Thomas Gleixner
2015-10-03 15:40 ` Russell King - ARM Linux
2 siblings, 0 replies; 9+ messages in thread
From: Thomas Gleixner @ 2015-09-22 10:59 UTC (permalink / raw
To: Daniel Thompson
Cc: Russell King, linux-arm-kernel, linux-kernel, patches,
linaro-kernel, Hillf Danton
On Tue, 15 Sep 2015, Daniel Thompson wrote:
> Currently on ARM when <SysRq-L> is triggered from an interrupt handler
> (e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
> seconds with interrupts masked before issuing a backtrace for every CPU
> except itself.
>
> The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
> basic support for on-demand backtrace of other CPUs") does not work
> correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
> is used to generate the backtrace on all CPUs but cannot preempt the
> current calling context.
>
> This can be fixed by detecting that the calling context cannot be
> preempted and issuing the backtrace directly in this case. Issuing
> directly leaves us without any pt_regs to pass to nmi_cpu_backtrace()
> so we also modify the generic code to call dump_stack() when its
> argument is NULL.
>
> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
For the genric part.
Acked-by: Thomas Gleixner <tglx@linutronix.de>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3] arm: Fix backtrace generation when IPI is masked
2015-09-15 14:40 ` [PATCH v3] " Daniel Thompson
2015-09-16 2:43 ` Hillf Danton
2015-09-22 10:59 ` Thomas Gleixner
@ 2015-10-03 15:40 ` Russell King - ARM Linux
2 siblings, 0 replies; 9+ messages in thread
From: Russell King - ARM Linux @ 2015-10-03 15:40 UTC (permalink / raw
To: Daniel Thompson
Cc: linux-arm-kernel, linux-kernel, patches, linaro-kernel,
Thomas Gleixner, Hillf Danton
On Tue, Sep 15, 2015 at 03:40:05PM +0100, Daniel Thompson wrote:
> Currently on ARM when <SysRq-L> is triggered from an interrupt handler
> (e.g. a SysRq issued using UART or kbd) the main CPU will wedge for ten
> seconds with interrupts masked before issuing a backtrace for every CPU
> except itself.
>
> The new backtrace code introduced by commit 96f0e00378d4 ("ARM: add
> basic support for on-demand backtrace of other CPUs") does not work
> correctly when run from an interrupt handler because IPI_CPU_BACKTRACE
> is used to generate the backtrace on all CPUs but cannot preempt the
> current calling context.
>
> This can be fixed by detecting that the calling context cannot be
> preempted and issuing the backtrace directly in this case. Issuing
> directly leaves us without any pt_regs to pass to nmi_cpu_backtrace()
> so we also modify the generic code to call dump_stack() when its
> argument is NULL.
>
> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
When submitting a patch to the patch system, please ensure that you pick
up people's acks _before_ submitting it to there - don't expect me to
search the mailing list, identify which patch version is the one in the
patch system, and then read the entire thread finding all the acks, then
having to amend the commit to add them.
A patch in the patch system with no acks looks like a patch which hasn't
been sent to the mailing lists.
Thanks.
--
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-10-03 15:40 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-09-14 9:18 [PATCH] arm: Fix backtrace generation when IPI is masked Daniel Thompson
2015-09-15 11:05 ` [PATCH v2] " Daniel Thompson
2015-09-15 11:30 ` Russell King - ARM Linux
2015-09-15 13:15 ` Daniel Thompson
2015-09-15 13:42 ` Russell King - ARM Linux
2015-09-15 14:40 ` [PATCH v3] " Daniel Thompson
2015-09-16 2:43 ` Hillf Danton
2015-09-22 10:59 ` Thomas Gleixner
2015-10-03 15:40 ` Russell King - ARM Linux
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).