From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7333C433DB for ; Tue, 12 Jan 2021 14:58:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 90F97221EC for ; Tue, 12 Jan 2021 14:58:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389223AbhALO6J (ORCPT ); Tue, 12 Jan 2021 09:58:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732558AbhALO6I (ORCPT ); Tue, 12 Jan 2021 09:58:08 -0500 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D699C061575 for ; Tue, 12 Jan 2021 06:57:28 -0800 (PST) Received: by mail-io1-xd2e.google.com with SMTP id w18so4661544iot.0 for ; Tue, 12 Jan 2021 06:57:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=a8cAC6W8EP8v2kz4Pqmyj0XUUN1a4NamLvxdd/3zGXE=; b=uPsGugOd/r/XLDFP7bKybOAeOBHVUUlGZD4F7spjrWcMsRPAXgLYNxbyb9lhieGNfL 8uQP+9A47ZGXl3xenI8QSTm54+B/JbJgk1GeXoNNbafCwtgMG1MtpWZ9+z2JZTRbNMW5 0c3/2iiPYddKPWhiuOxX3+dK5BnW604IPRd8nsrn6PURj3nD3a7u3U5xcgeGAmBmUcHq 3oqA47kB5YPvvnADcFTGXFGlhPBAFaGd1FOFnSQP27zSO/+EV1FFaLex8L+ry7+SkzDj PfB5bT2izfxNh68ovJTZNvGvzKCYKvxbBMR6ZOIJRY7szgDDnRV5bzAd0MdcDW0fqcU1 ZTRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=a8cAC6W8EP8v2kz4Pqmyj0XUUN1a4NamLvxdd/3zGXE=; b=S27/PFl60COXfK1ZXJqoX78IYsKraivMy0V5KWzvcfgvbhJEtIUGEzgaYisWx1/lFx phFP/LrlXbLDpFirHr33gLvaU2OgXaVcENnVPxjKIkK0jHzPWsmAZzcGCseVrO0uLIXx 47dZyiTcYdE9j4WlUwq6NaGAcjVOr44yPdAT/YF+sMibv4E1ihQHOFOMmCCQwXu8+AP6 njsFsjqec+g+4W4qvyXh1jY/76XyCfthM229vBqdXj2RRnydL+gRukEk2S0OB8Z+8V3w G2u88p7JWyTyQkljuEzH6awIHmp8MUuckVJhCA1g/l3DnXwb9px+3+EUwNz1zef6scjX Q+GA== X-Gm-Message-State: AOAM532AREaH5ablFaKpJ9JjeTDl0hwwWLa3NmCbCx6VUmOyRB5a4TRA SIwt7tHVMCd/aa8nowr9hsfo2Q== X-Google-Smtp-Source: ABdhPJwzQTc7uNYxdzeUSw4X1PHAwo+zkoyC9wVKHyV1W5qXznUwTwkO6rGoRGIcwtt7OijLQK+QFA== X-Received: by 2002:a5e:a614:: with SMTP id q20mr3587305ioi.198.1610463447562; Tue, 12 Jan 2021 06:57:27 -0800 (PST) Received: from [192.168.1.30] ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id n11sm2040414ioh.37.2021.01.12.06.57.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 12 Jan 2021 06:57:27 -0800 (PST) Subject: Re: [PATCH -tip V3 0/8] workqueue: break affinity initiatively To: Valentin Schneider , Peter Zijlstra , Thomas Gleixner Cc: Lai Jiangshan , linux-kernel@vger.kernel.org, Qian Cai , Vincent Donnefort , Dexuan Cui , Lai Jiangshan , Paul McKenney , Vincent Guittot , Steven Rostedt References: <20201226025117.2770-1-jiangshanlai@gmail.com> <87o8hv7pnd.fsf@nanos.tec.linutronix.de> From: Jens Axboe Message-ID: Date: Tue, 12 Jan 2021 07:57:26 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 1/11/21 12:21 PM, Valentin Schneider wrote: > On 11/01/21 18:16, Peter Zijlstra wrote: >> Sadly it appears like io_uring() uses kthread_create_on_cpu() without >> then having any hotplug crud on, so that needs additinoal frobbing. >> > > I noticed that as well sometime ago, and I believed then (still do) this > usage is broken. I don't think usage of kthread_create_on_cpu() outside > of smpboot makes sense, because without any hotplug step to park the > thread, its affinity can end up being reset after its dedicated CPU gets > offlined. > > I'm clueless about io_uring, but if it *actually* has a good reason to > use some pcpu kthreads (it seems it doesn't have to be on all CPUs?), > then it needs to register some hotplug step to park them / do something > sensible on hotplug. For io_uring, it's purely used by the SQPOLL mode, which sets aside a kernel thread for submissions so the application doesn't have to do anything but write new SQE entries to submit. The thread then notices these, and submits them. There's an option to affinitize that thread to a single CPU, which often makes sense for setups like that. Think of it like a strong hint. Things aren't going to break if this CPU goes away and we end up being affinitized to some other CPU, though it is suboptimal. So I guess we might need some notifiers to ensure that we reset the CPU back again if it's gone offline+online again? I can take a look at that. -- Jens Axboe