* [PATCH v1] perf test: Avoid hard coded metrics in stat std output test
@ 2024-04-17 18:32 Ian Rogers
2024-04-19 13:54 ` Liang, Kan
0 siblings, 1 reply; 6+ messages in thread
From: Ian Rogers @ 2024-04-17 18:32 UTC (permalink / raw
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Ian Rogers, Adrian Hunter, Kan Liang, Yicong Yang, Athira Rajeev,
linux-perf-users, linux-kernel
Hard coded metric names fail on ARM testing.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/shell/stat+std_output.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/perf/tests/shell/stat+std_output.sh b/tools/perf/tests/shell/stat+std_output.sh
index cbf2894b2c84..845f83213855 100755
--- a/tools/perf/tests/shell/stat+std_output.sh
+++ b/tools/perf/tests/shell/stat+std_output.sh
@@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
-skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
+skip_metric=($(perf list --raw Default 2> /dev/null))
cleanup() {
rm -f "${stat_output}"
--
2.44.0.683.g7961c838ac-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v1] perf test: Avoid hard coded metrics in stat std output test
2024-04-17 18:32 [PATCH v1] perf test: Avoid hard coded metrics in stat std output test Ian Rogers
@ 2024-04-19 13:54 ` Liang, Kan
2024-04-19 14:40 ` Ian Rogers
0 siblings, 1 reply; 6+ messages in thread
From: Liang, Kan @ 2024-04-19 13:54 UTC (permalink / raw
To: Ian Rogers, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, Yicong Yang, Athira Rajeev, linux-perf-users,
linux-kernel
On 2024-04-17 2:32 p.m., Ian Rogers wrote:
> Hard coded metric names fail on ARM testing.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> tools/perf/tests/shell/stat+std_output.sh | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/perf/tests/shell/stat+std_output.sh b/tools/perf/tests/shell/stat+std_output.sh
> index cbf2894b2c84..845f83213855 100755
> --- a/tools/perf/tests/shell/stat+std_output.sh
> +++ b/tools/perf/tests/shell/stat+std_output.sh
> @@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
>
> event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
> event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
> -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
> +skip_metric=($(perf list --raw Default 2> /dev/null))
The "perf list --raw Default" only gives the topdown metrics.
The "stalled cycles per insn" is not covered.
The check should skip the line of "stalled cycles per insn" as well.
3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle
1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle
2,603,501,247 instructions # 0.50 insns per cycle
# 1.48 stalled cycles
per insn
484,357,498 branches # 283.455 M/sec
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24
The newer Intel CPU doesn't have the stalled-cycles-* events. But it
seems power and older x86 CPU have the events.
Thanks,
Kan
>
> cleanup() {
> rm -f "${stat_output}"
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v1] perf test: Avoid hard coded metrics in stat std output test
2024-04-19 13:54 ` Liang, Kan
@ 2024-04-19 14:40 ` Ian Rogers
2024-04-19 15:09 ` Liang, Kan
0 siblings, 1 reply; 6+ messages in thread
From: Ian Rogers @ 2024-04-19 14:40 UTC (permalink / raw
To: Liang, Kan
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, Yicong Yang, Athira Rajeev, linux-perf-users,
linux-kernel
On Fri, Apr 19, 2024 at 6:54 AM Liang, Kan <kan.liang@linux.intel.com> wrote:
>
>
>
> On 2024-04-17 2:32 p.m., Ian Rogers wrote:
> > Hard coded metric names fail on ARM testing.
> >
> > Signed-off-by: Ian Rogers <irogers@google.com>
> > ---
> > tools/perf/tests/shell/stat+std_output.sh | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tools/perf/tests/shell/stat+std_output.sh b/tools/perf/tests/shell/stat+std_output.sh
> > index cbf2894b2c84..845f83213855 100755
> > --- a/tools/perf/tests/shell/stat+std_output.sh
> > +++ b/tools/perf/tests/shell/stat+std_output.sh
> > @@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
> >
> > event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
> > event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
> > -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
> > +skip_metric=($(perf list --raw Default 2> /dev/null))
>
>
> The "perf list --raw Default" only gives the topdown metrics.
> The "stalled cycles per insn" is not covered.
> The check should skip the line of "stalled cycles per insn" as well.
>
> 3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle
> 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle
> 2,603,501,247 instructions # 0.50 insns per cycle
> # 1.48 stalled cycles
> per insn
> 484,357,498 branches # 283.455 M/sec
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24
>
> The newer Intel CPU doesn't have the stalled-cycles-* events. But it
> seems power and older x86 CPU have the events.
Oh, sigh. This test should really ignore lines like that. How much do
we care about these metrics? The RISC-V event parsing change:
https://lore.kernel.org/lkml/20240416061533.921723-1-irogers@google.com/
means that legacy hardware events will be uncommon and we need to
adapt the hard coded metrics in stat-shadow.c to json ones. Once they
are json metrics they will be in Default.
Thanks,
Ian
> Thanks,
> Kan
>
> >
> > cleanup() {
> > rm -f "${stat_output}"
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v1] perf test: Avoid hard coded metrics in stat std output test
2024-04-19 14:40 ` Ian Rogers
@ 2024-04-19 15:09 ` Liang, Kan
2024-04-19 15:23 ` Ian Rogers
0 siblings, 1 reply; 6+ messages in thread
From: Liang, Kan @ 2024-04-19 15:09 UTC (permalink / raw
To: Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, Yicong Yang, Athira Rajeev, linux-perf-users,
linux-kernel
On 2024-04-19 10:40 a.m., Ian Rogers wrote:
> On Fri, Apr 19, 2024 at 6:54 AM Liang, Kan <kan.liang@linux.intel.com> wrote:
>>
>>
>>
>> On 2024-04-17 2:32 p.m., Ian Rogers wrote:
>>> Hard coded metric names fail on ARM testing.
>>>
>>> Signed-off-by: Ian Rogers <irogers@google.com>
>>> ---
>>> tools/perf/tests/shell/stat+std_output.sh | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/tools/perf/tests/shell/stat+std_output.sh b/tools/perf/tests/shell/stat+std_output.sh
>>> index cbf2894b2c84..845f83213855 100755
>>> --- a/tools/perf/tests/shell/stat+std_output.sh
>>> +++ b/tools/perf/tests/shell/stat+std_output.sh
>>> @@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
>>>
>>> event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
>>> event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
>>> -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
>>> +skip_metric=($(perf list --raw Default 2> /dev/null))
>>
>>
>> The "perf list --raw Default" only gives the topdown metrics.
>> The "stalled cycles per insn" is not covered.
>> The check should skip the line of "stalled cycles per insn" as well.
>>
>> 3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle
>> 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle
>> 2,603,501,247 instructions # 0.50 insns per cycle
>> # 1.48 stalled cycles
>> per insn
>> 484,357,498 branches # 283.455 M/sec
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24
>>
>> The newer Intel CPU doesn't have the stalled-cycles-* events. But it
>> seems power and older x86 CPU have the events.
>
> Oh, sigh. This test should really ignore lines like that. How much do
> we care about these metrics? The RISC-V event parsing change:
> https://lore.kernel.org/lkml/20240416061533.921723-1-irogers@google.com/
> means that legacy hardware events will be uncommon and we need to
> adapt the hard coded metrics in stat-shadow.c to json ones. Once they
> are json metrics they will be in Default.
It seems except the newer Intel CPU, all the other ARCHs support the two
stalled-cycles-* events and the metric. For Intel, there are Topdown
metrics. But it seems an important metrics for the other ARCHs.
RISC-V
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
Power
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
Arm
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53
So almost all json files have to be updated. I'm not sure if it's a
practical way to fix the issue.
Thanks,
Kan
>
> Thanks,
> Ian
>
>> Thanks,
>> Kan
>>
>>>
>>> cleanup() {
>>> rm -f "${stat_output}"
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v1] perf test: Avoid hard coded metrics in stat std output test
2024-04-19 15:09 ` Liang, Kan
@ 2024-04-19 15:23 ` Ian Rogers
2024-04-19 17:39 ` Liang, Kan
0 siblings, 1 reply; 6+ messages in thread
From: Ian Rogers @ 2024-04-19 15:23 UTC (permalink / raw
To: Liang, Kan
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, Yicong Yang, Athira Rajeev, linux-perf-users,
linux-kernel
On Fri, Apr 19, 2024 at 8:09 AM Liang, Kan <kan.liang@linux.intel.com> wrote:
>
>
>
> On 2024-04-19 10:40 a.m., Ian Rogers wrote:
> > On Fri, Apr 19, 2024 at 6:54 AM Liang, Kan <kan.liang@linux.intel.com> wrote:
> >>
> >>
> >>
> >> On 2024-04-17 2:32 p.m., Ian Rogers wrote:
> >>> Hard coded metric names fail on ARM testing.
> >>>
> >>> Signed-off-by: Ian Rogers <irogers@google.com>
> >>> ---
> >>> tools/perf/tests/shell/stat+std_output.sh | 2 +-
> >>> 1 file changed, 1 insertion(+), 1 deletion(-)
> >>>
> >>> diff --git a/tools/perf/tests/shell/stat+std_output.sh b/tools/perf/tests/shell/stat+std_output.sh
> >>> index cbf2894b2c84..845f83213855 100755
> >>> --- a/tools/perf/tests/shell/stat+std_output.sh
> >>> +++ b/tools/perf/tests/shell/stat+std_output.sh
> >>> @@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_output.std.XXXXX)
> >>>
> >>> event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
> >>> event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
> >>> -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
> >>> +skip_metric=($(perf list --raw Default 2> /dev/null))
> >>
> >>
> >> The "perf list --raw Default" only gives the topdown metrics.
> >> The "stalled cycles per insn" is not covered.
> >> The check should skip the line of "stalled cycles per insn" as well.
> >>
> >> 3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle
> >> 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle
> >> 2,603,501,247 instructions # 0.50 insns per cycle
> >> # 1.48 stalled cycles
> >> per insn
> >> 484,357,498 branches # 283.455 M/sec
> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24
> >>
> >> The newer Intel CPU doesn't have the stalled-cycles-* events. But it
> >> seems power and older x86 CPU have the events.
> >
> > Oh, sigh. This test should really ignore lines like that. How much do
> > we care about these metrics? The RISC-V event parsing change:
> > https://lore.kernel.org/lkml/20240416061533.921723-1-irogers@google.com/
> > means that legacy hardware events will be uncommon and we need to
> > adapt the hard coded metrics in stat-shadow.c to json ones. Once they
> > are json metrics they will be in Default.
>
> It seems except the newer Intel CPU, all the other ARCHs support the two
> stalled-cycles-* events and the metric. For Intel, there are Topdown
> metrics. But it seems an important metrics for the other ARCHs.
>
> RISC-V
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
> Power
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
> Arm
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53
>
> So almost all json files have to be updated. I'm not sure if it's a
> practical way to fix the issue.
So I'd very much like to get rid of the hard coded metrics:
- they don't use or respect event groups,
- their ad hoc printing can introduce extra metric results
unexpectedly in output,
- they fall outside of optimizations like Weilin's metric event grouping work.
I'm hoping the python json generation of metrics makes their removal practical:
https://lore.kernel.org/lkml/20240314055919.1979781-1-irogers@google.com/
That's a lot to get landed for this fix:
- 40+ patches for python based json generation.
- 10+ patches for parse events changes.
So I think a version that hard codes ignoring the hard coded metrics
is in order.
Thanks,
Ian
> Thanks,
> Kan
> >
> > Thanks,
> > Ian
> >
> >> Thanks,
> >> Kan
> >>
> >>>
> >>> cleanup() {
> >>> rm -f "${stat_output}"
> >
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v1] perf test: Avoid hard coded metrics in stat std output test
2024-04-19 15:23 ` Ian Rogers
@ 2024-04-19 17:39 ` Liang, Kan
0 siblings, 0 replies; 6+ messages in thread
From: Liang, Kan @ 2024-04-19 17:39 UTC (permalink / raw
To: Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Adrian Hunter, Yicong Yang, Athira Rajeev, linux-perf-users,
linux-kernel
On 2024-04-19 11:23 a.m., Ian Rogers wrote:
> On Fri, Apr 19, 2024 at 8:09 AM Liang, Kan <kan.liang@linux.intel.com> wrote:
>>
>>
>>
>> On 2024-04-19 10:40 a.m., Ian Rogers wrote:
>>> On Fri, Apr 19, 2024 at 6:54 AM Liang, Kan <kan.liang@linux.intel.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2024-04-17 2:32 p.m., Ian Rogers wrote:
>>>>> Hard coded metric names fail on ARM testing.
>>>>>
>>>>> Signed-off-by: Ian Rogers <irogers@google.com>
>>>>> ---
>>>>> tools/perf/tests/shell/stat+std_output.sh | 2 +-
>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/tools/perf/tests/shell/stat+std_output.sh b/tools/perf/tests/shell/stat+std_output.sh
>>>>> index cbf2894b2c84..845f83213855 100755
>>>>> --- a/tools/perf/tests/shell/stat+std_output.sh
>>>>> +++ b/tools/perf/tests/shell/stat+std_output.sh
>>>>> @@ -13,7 +13,7 @@ stat_output=$(mktemp /tmp/__perf_test.stat_outputstd.XXXXX)
>>>>>
>>>>> event_name=(cpu-clock task-clock context-switches cpu-migrations page-faults stalled-cycles-frontend stalled-cycles-backend cycles instructions branches branch-misses)
>>>>> event_metric=("CPUs utilized" "CPUs utilized" "/sec" "/sec" "/sec" "frontend cycles idle" "backend cycles idle" "GHz" "insn per cycle" "/sec" "of all branches")
>>>>> -skip_metric=("stalled cycles per insn" "tma_" "retiring" "frontend_bound" "bad_speculation" "backend_bound")
>>>>> +skip_metric=($(perf list --raw Default 2> /dev/null))
>>>>
>>>>
>>>> The "perf list --raw Default" only gives the topdown metrics.
>>>> The "stalled cycles per insn" is not covered.
>>>> The check should skip the line of "stalled cycles per insn" as well.
>>>>
>>>> 3,856,436,920 stalled-cycles-frontend # 74.09% frontend cycles idle
>>>> 1,600,790,871 stalled-cycles-backend # 30.75% backend cycles idle
>>>> 2,603,501,247 instructions # 0.50 insns per cycle
>>>> # 1.48 stalled cycles
>>>> per insn
>>>> 484,357,498 branches # 283.455 M/sec
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/builtin-stat.c#n24
>>>>
>>>> The newer Intel CPU doesn't have the stalled-cycles-* events. But it
>>>> seems power and older x86 CPU have the events.
>>>
>>> Oh, sigh. This test should really ignore lines like that. How much do
>>> we care about these metrics? The RISC-V event parsing change:
>>> https://lore.kernel.org/lkml/20240416061533.921723-1-irogers@google.com/
>>> means that legacy hardware events will be uncommon and we need to
>>> adapt the hard coded metrics in stat-shadow.c to json ones. Once they
>>> are json metrics they will be in Default.
>>
>> It seems except the newer Intel CPU, all the other ARCHs support the two
>> stalled-cycles-* events and the metric. For Intel, there are Topdown
>> metrics. But it seems an important metrics for the other ARCHs.
>>
>> RISC-V
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/riscv_pmu_sbi.c#n134
>> Power
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/perf/power9-pmu.c#n279
>> Arm
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmuv3.c#n53
>>
>> So almost all json files have to be updated. I'm not sure if it's a
>> practical way to fix the issue.
>
> So I'd very much like to get rid of the hard coded metrics:
> - they don't use or respect event groups,
> - their ad hoc printing can introduce extra metric results
> unexpectedly in output,
> - they fall outside of optimizations like Weilin's metric event grouping work.
> I'm hoping the python json generation of metrics makes their removal practical:
> https://lore.kernel.org/lkml/20240314055919.1979781-1-irogers@google.com/
>
It seems I wasn't listed in the Cc list, so it's missed in my main
inbox. But I was added to the same series for the Arm and AMD. :)
I will take a look at the patches.
> That's a lot to get landed for this fix:
> - 40+ patches for python based json generation.
> - 10+ patches for parse events changes.
> So I think a version that hard codes ignoring the hard coded metrics
> is in order.
Sounds good to me.
Thanks,
Kan
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-04-19 17:39 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-04-17 18:32 [PATCH v1] perf test: Avoid hard coded metrics in stat std output test Ian Rogers
2024-04-19 13:54 ` Liang, Kan
2024-04-19 14:40 ` Ian Rogers
2024-04-19 15:09 ` Liang, Kan
2024-04-19 15:23 ` Ian Rogers
2024-04-19 17:39 ` Liang, Kan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).