From: K Prateek Nayak <kprateek.nayak@amd.com>
To: Mike Galbraith <efault@gmx.de>,
Peter Zijlstra <peterz@infradead.org>,
mingo@redhat.com, juri.lelli@redhat.com,
vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
bristot@redhat.com, vschneid@redhat.com,
linux-kernel@vger.kernel.org
Cc: wuyun.abel@bytedance.com, tglx@linutronix.de,
Chen Yu <yu.c.chen@intel.com>,
Oliver Sang <oliver.sang@intel.com>
Subject: Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Date: Tue, 16 Apr 2024 08:48:55 +0530 [thread overview]
Message-ID: <14330cf4-8d9e-1e55-7717-653b800e5cee@amd.com> (raw)
In-Reply-To: <ec6f811b9977eeef1b3f1b3fb951fda066fd95f5.camel@gmx.de>
(+ Chen Yu, Oliver Sang)
Hello Mike,
On 4/15/2024 4:26 PM, Mike Galbraith wrote:
> On Fri, 2024-04-12 at 16:12 +0530, K Prateek Nayak wrote:
>>
>> I ran into a few issues when testing the series on top of tip:sched/core
>> at commit 4475cd8bfd9b ("sched/balancing: Simplify the sg_status bitmask
>> and use separate ->overloaded and ->overutilized flags"). All of these
>> splats surfaced when running unixbench with Delayed Dequeue (echoing
>> NO_DELAY_DEQUEUE to /sys/kernel/debug/sched/features seems to make the
>> system stable when running Unixbench spawn)
>>
>> Unixbench (https://github.com/kdlucas/byte-unixbench.git) command:
>>
>> ./Run spawn -c 512
>
> That plus a hackbench loop works a treat.
>
>>
>> Splats appear soon into the run. Following are the splats and their
>> corresponding code blocks from my 3rd Generation EPYC system
>> (2 x 64C/128T):
>
> Seems a big box is not required. With a low fat sched config (no group
> sched), starting ./Run spawn -c 16 (cpus*2) along with a hackbench loop
> reliably blows my old i7-4790 box out of the water nearly instantly.
>
> DUMPFILE: vmcore
> CPUS: 8
> DATE: Mon Apr 15 07:20:29 CEST 2024
> UPTIME: 00:07:23
> LOAD AVERAGE: 1632.20, 684.99, 291.84
> TASKS: 1401
> NODENAME: homer
> RELEASE: 6.9.0.g0bbac3f-master
> VERSION: #7 SMP Mon Apr 15 06:40:05 CEST 2024
> MACHINE: x86_64 (3591 Mhz)
> MEMORY: 16 GB
> PANIC: "Oops: 0000 [#1] SMP NOPTI" (check log for details)
> PID: 22664
> COMMAND: "hackbench"
> TASK: ffff888100acbf00 [THREAD_INFO: ffff888100acbf00]
> CPU: 5
> STATE: TASK_WAKING (PANIC)
>
> crash> bt -sx
> PID: 22664 TASK: ffff888100acbf00 CPU: 5 COMMAND: "hackbench"
> #0 [ffff88817cc17920] machine_kexec+0x156 at ffffffff810642d6
> #1 [ffff88817cc17970] __crash_kexec+0xd7 at ffffffff81153147
> #2 [ffff88817cc17a28] crash_kexec+0x23 at ffffffff811535f3
> #3 [ffff88817cc17a38] oops_end+0xbe at ffffffff810329be
> #4 [ffff88817cc17a58] page_fault_oops+0x81 at ffffffff81071951
> #5 [ffff88817cc17ab8] exc_page_fault+0x62 at ffffffff8194f6f2
> #6 [ffff88817cc17ae0] asm_exc_page_fault+0x22 at ffffffff81a00ba2
> [exception RIP: pick_task_fair+71]
> RIP: ffffffff810d5b57 RSP: ffff88817cc17b90 RFLAGS: 00010046
> RAX: 0000000000000000 RBX: ffff88840ed70ec0 RCX: 00000001d7ec138c
> RDX: ffffffffe7a7f400 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: ffff88840ed70ec0 R8: 0000000000000c00 R9: 000000675402f79e
> R10: ffff88817cc17b30 R11: 00000000000000bb R12: ffff88840ed70f40
> R13: ffffffff81f64f16 R14: ffff888100acc560 R15: ffff888100acbf00
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> #7 [ffff88817cc17bb0] pick_next_task_fair+0x42 at ffffffff810d92c2
> #8 [ffff88817cc17be0] __schedule+0x10d at ffffffff8195936d
> #9 [ffff88817cc17c50] schedule+0x1c at ffffffff81959ddc
> #10 [ffff88817cc17c60] schedule_timeout+0x18c at ffffffff8195fc4c
> #11 [ffff88817cc17cc8] unix_stream_read_generic+0x2b7 at ffffffff81869917
> #12 [ffff88817cc17da8] unix_stream_recvmsg+0x68 at ffffffff8186a2d8
> #13 [ffff88817cc17de0] sock_read_iter+0x159 at ffffffff8170bd69
> #14 [ffff88817cc17e70] vfs_read+0x2ce at ffffffff812f195e
> #15 [ffff88817cc17ef8] ksys_read+0x40 at ffffffff812f21d0
> #16 [ffff88817cc17f30] do_syscall_64+0x57 at ffffffff8194b947
> #17 [ffff88817cc17f50] entry_SYSCALL_64_after_hwframe+0x76 at ffffffff81a0012b
> RIP: 00007f625660871e RSP: 00007ffc75d48188 RFLAGS: 00000246
> RAX: ffffffffffffffda RBX: 00007ffc75d48200 RCX: 00007f625660871e
> RDX: 0000000000000064 RSI: 00007ffc75d48190 RDI: 0000000000000007
> RBP: 00007ffc75d48260 R8: 00007ffc75d48140 R9: 00007f6256612010
> R10: 00007f62565f5070 R11: 0000000000000246 R12: 0000000000000064
> R13: 0000000000000000 R14: 0000000000000064 R15: 0000000000000000
> ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
> crash> dis pick_task_fair+71
> 0xffffffff810d5b57 <pick_task_fair+71>: cmpb $0x0,0x4c(%rax)
> crash> gdb list *pick_task_fair+71
> 0xffffffff810d5b57 is in pick_task_fair (kernel/sched/fair.c:5498).
> 5493 SCHED_WARN_ON(cfs_rq->next->sched_delayed);
> 5494 return cfs_rq->next;
> 5495 }
> 5496
> 5497 struct sched_entity *se = pick_eevdf(cfs_rq);
> 5498 if (se->sched_delayed) {
Wondering if you are running into the issue where pick_eevdf() returns
NULL despite there being runnable CFS task on the runqueue. Can you try
running with the following patch from Chenyu -
https://lore.kernel.org/lkml/20240226082349.302363-1-yu.c.chen@intel.com/
> 5499 dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> 5500 SCHED_WARN_ON(se->sched_delayed);
> 5501 SCHED_WARN_ON(se->on_rq);
> 5502 if (sched_feat(DELAY_ZERO) && se->vlag > 0)
> crash> struct -ox sched_entity
> struct sched_entity {
> [0x0] struct load_weight load;
> [0x10] struct rb_node run_node;
> [0x28] u64 deadline;
> [0x30] u64 min_vruntime;
> [0x38] struct list_head group_node;
> [0x48] unsigned int on_rq;
> [0x4c] unsigned char sched_delayed;
> [0x4d] unsigned char custom_slice;
> [0x50] u64 exec_start;
> [0x58] u64 sum_exec_runtime;
> [0x60] u64 prev_sum_exec_runtime;
> [0x68] u64 vruntime;
> [0x70] s64 vlag;
> [0x78] u64 slice;
> [0x80] u64 nr_migrations;
> [0xc0] struct sched_avg avg;
> }
> SIZE: 0x100
> crash>
>
--
Thanks and Regards,
Prateek
next prev parent reply other threads:[~2024-04-16 3:19 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-05 10:27 [RFC][PATCH 00/10] sched/fair: Complete EEVDF Peter Zijlstra
2024-04-05 10:27 ` [RFC][PATCH 01/10] sched/eevdf: Add feature comments Peter Zijlstra
2024-04-05 10:27 ` [RFC][PATCH 02/10] sched/eevdf: Remove min_vruntime_copy Peter Zijlstra
2024-04-05 10:27 ` [RFC][PATCH 03/10] sched/fair: Cleanup pick_task_fair() vs throttle Peter Zijlstra
2024-04-05 21:11 ` Benjamin Segall
2024-04-05 10:27 ` [RFC][PATCH 04/10] sched/fair: Cleanup pick_task_fair()s curr Peter Zijlstra
2024-04-05 10:27 ` [RFC][PATCH 05/10] sched/fair: Unify pick_{,next_}_task_fair() Peter Zijlstra
2024-04-06 2:20 ` Mike Galbraith
2024-04-05 10:28 ` [RFC][PATCH 06/10] sched: Allow sched_class::dequeue_task() to fail Peter Zijlstra
2024-04-05 10:28 ` [RFC][PATCH 07/10] sched/fair: Re-organize dequeue_task_fair() Peter Zijlstra
2024-04-05 10:28 ` [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue Peter Zijlstra
2024-04-06 9:23 ` Chen Yu
2024-04-08 9:06 ` Peter Zijlstra
2024-04-11 1:32 ` Yan-Jie Wang
2024-04-25 10:25 ` Peter Zijlstra
2024-04-12 10:42 ` K Prateek Nayak
2024-04-15 10:56 ` Mike Galbraith
2024-04-16 3:18 ` K Prateek Nayak [this message]
2024-04-16 5:36 ` Mike Galbraith
2024-04-18 16:24 ` Mike Galbraith
2024-04-18 17:08 ` K Prateek Nayak
2024-04-24 15:20 ` Peter Zijlstra
2024-04-25 11:28 ` Peter Zijlstra
2024-04-26 10:56 ` Peter Zijlstra
2024-04-26 11:16 ` Peter Zijlstra
2024-04-26 16:03 ` Mike Galbraith
2024-04-27 6:42 ` Mike Galbraith
2024-04-28 16:32 ` Mike Galbraith
2024-04-29 12:14 ` Peter Zijlstra
2024-04-15 17:07 ` Luis Machado
2024-04-24 15:15 ` Luis Machado
2024-04-25 10:42 ` Peter Zijlstra
2024-04-25 11:49 ` Peter Zijlstra
2024-04-26 9:32 ` Peter Zijlstra
2024-04-26 9:36 ` Peter Zijlstra
2024-04-26 10:16 ` Luis Machado
2024-04-29 14:33 ` Luis Machado
2024-05-02 10:26 ` Luis Machado
2024-05-10 14:49 ` Luis Machado
2024-05-15 9:36 ` Peter Zijlstra
2024-05-15 11:48 ` Peter Zijlstra
2024-05-15 18:03 ` Mike Galbraith
2024-05-20 15:20 ` Luis Machado
2024-05-29 22:50 ` Peter Zijlstra
2024-06-03 19:30 ` Luis Machado
2024-06-04 10:11 ` Peter Zijlstra
2024-06-04 13:59 ` Hongyan Xia
2024-06-04 14:23 ` Luis Machado
2024-06-04 14:49 ` Hongyan Xia
2024-06-04 19:12 ` Peter Zijlstra
2024-06-05 7:22 ` Peter Zijlstra
2024-06-05 9:14 ` Luis Machado
2024-06-05 9:42 ` Peter Zijlstra
2024-06-12 15:08 ` Luis Machado
2024-05-23 8:45 ` Peter Zijlstra
2024-05-23 9:06 ` Luis Machado
2024-05-23 9:33 ` Peter Zijlstra
2024-06-03 15:57 ` Hongyan Xia
2024-04-26 10:15 ` Luis Machado
2024-04-20 5:57 ` Mike Galbraith
2024-04-22 13:13 ` Tobias Huschle
2024-04-05 10:28 ` [RFC][PATCH 09/10] sched/eevdf: Allow shorter slices to wakeup-preempt Peter Zijlstra
2024-04-05 10:28 ` [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion Peter Zijlstra
2024-04-06 8:16 ` Hillf Danton
2024-05-07 5:34 ` Mike Galbraith
2024-05-15 10:13 ` Peter Zijlstra
2024-05-07 15:15 ` Chen Yu
2024-05-08 13:52 ` Mike Galbraith
2024-05-09 3:48 ` Chen Yu
2024-05-09 5:00 ` Mike Galbraith
2024-05-13 4:07 ` K Prateek Nayak
2024-05-14 9:18 ` Chen Yu
2024-05-14 15:23 ` K Prateek Nayak
2024-05-14 16:15 ` Chen Yu
2024-05-22 14:48 ` Chen Yu
2024-05-27 10:11 ` [RFC][PATCH 00/10] sched/fair: Complete EEVDF K Prateek Nayak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=14330cf4-8d9e-1e55-7717-653b800e5cee@amd.com \
--to=kprateek.nayak@amd.com \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=efault@gmx.de \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=oliver.sang@intel.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=wuyun.abel@bytedance.com \
--cc=yu.c.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).