From: Jesper Dangaard Brouer <hawk@kernel.org>
To: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Waiman Long <longman@redhat.com>,
tj@kernel.org, hannes@cmpxchg.org, lizefan.x@bytedance.com,
cgroups@vger.kernel.org, yosryahmed@google.com,
netdev@vger.kernel.org, linux-mm@kvack.org,
kernel-team@cloudflare.com,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Subject: Re: [PATCH v1] cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints
Date: Fri, 3 May 2024 14:58:56 +0200 [thread overview]
Message-ID: <c5a79618-8c64-4e7b-aeed-69aeecb1590d@kernel.org> (raw)
In-Reply-To: <4m3x4rtztwxctwlq2pdorgbv2hblylnuc2haz7ni4ti52n57xi@utxkr5ripqp2>
On 02/05/2024 21.44, Shakeel Butt wrote:
> On Wed, May 01, 2024 at 07:22:26PM +0200, Jesper Dangaard Brouer wrote:
>>
> [...]
>>
>> More data, the histogram of time spend under the lock have some strange
>> variation issues with a group in 4ms to 65ms area. Investigating what
>> can be causeing this... which next step depend in these tracepoints.
>>
>> @lock_cnt: 759146
>>
>> @locked_ns:
>> [1K, 2K) 499 | |
>> [2K, 4K) 206928
>> |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
>> [4K, 8K) 147904 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
>> [8K, 16K) 64453 |@@@@@@@@@@@@@@@@ |
>> [16K, 32K) 135467 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
>> [32K, 64K) 75943 |@@@@@@@@@@@@@@@@@@@ |
>> [64K, 128K) 38359 |@@@@@@@@@ |
>> [128K, 256K) 46597 |@@@@@@@@@@@ |
>> [256K, 512K) 32466 |@@@@@@@@ |
>> [512K, 1M) 3945 | |
>> [1M, 2M) 642 | |
>> [2M, 4M) 750 | |
>> [4M, 8M) 1932 | |
>> [8M, 16M) 2114 | |
>> [16M, 32M) 1039 | |
>> [32M, 64M) 108 | |
>>
>
> Am I understanding correctly that 1K is 1 microsecond and 1M is 1
> millisecond?
Correct.
> Is it possible to further divide this table into update
> side and flush side?
>
This is *only* flush side.
You question indicate, that we are talking past each-other ;-)
Measurements above is with (recently) accepted tracepoints (e.g. not the
proposed tracepoints in this patch). I'm arguing with existing
tracepoint that I'm seeing this data, and arguing I need per-CPU
tracepoints to dig deeper into this (as proposed in this patch).
The "update side" can only be measured once we apply this patch.
This morning I got 6 prod machines booted with new kernels, that contain
this proposed per-CPU lock tracepoint patch. And 3 of these machines
have the Mutex lock change also. No data to share yet...
--Jesper
next prev parent reply other threads:[~2024-05-03 12:59 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-01 14:04 [PATCH v1] cgroup/rstat: add cgroup_rstat_cpu_lock helpers and tracepoints Jesper Dangaard Brouer
2024-05-01 14:24 ` Waiman Long
2024-05-01 17:22 ` Jesper Dangaard Brouer
2024-05-01 18:41 ` Waiman Long
2024-05-02 11:23 ` Jesper Dangaard Brouer
2024-05-02 18:19 ` Waiman Long
2024-05-03 14:00 ` Jesper Dangaard Brouer
2024-05-03 14:30 ` Waiman Long
2024-05-03 19:18 ` Shakeel Butt
2024-05-06 12:03 ` Jesper Dangaard Brouer
2024-05-06 16:22 ` Shakeel Butt
2024-05-06 16:28 ` Ivan Babrou
2024-05-06 19:45 ` Shakeel Butt
2024-05-06 19:54 ` Jesper Dangaard Brouer
2024-05-02 19:44 ` Shakeel Butt
2024-05-03 12:58 ` Jesper Dangaard Brouer [this message]
2024-05-03 18:11 ` Shakeel Butt
2024-05-14 5:18 ` Jesper Dangaard Brouer
2024-05-14 5:55 ` Tejun Heo
2024-05-14 16:59 ` Waiman Long
2024-05-15 16:58 ` Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c5a79618-8c64-4e7b-aeed-69aeecb1590d@kernel.org \
--to=hawk@kernel.org \
--cc=acme@kernel.org \
--cc=bigeasy@linutronix.de \
--cc=cgroups@vger.kernel.org \
--cc=hannes@cmpxchg.org \
--cc=kernel-team@cloudflare.com \
--cc=linux-mm@kvack.org \
--cc=lizefan.x@bytedance.com \
--cc=longman@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=tj@kernel.org \
--cc=yosryahmed@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).