From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D81FD208A0 for ; Tue, 26 Mar 2024 01:39:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711417156; cv=none; b=R2Xf+mJ6JDxupsVcUsTyzdvHuXKYsbm6gT/Xzz8LzLiRHxo4tJRMifu8dAloxzyjrwudPt2mqwEuUakk4RJVspUXKzDxFDQW82DIH9QobASN+vKTO2pyFvh/yaNsC5G4wqHNkbo/6j9CLHCvgDkhjypyBKvKLuQfqBQh1p3SDNw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711417156; c=relaxed/simple; bh=ozN5T78X/8LPjCWi2AnyTnIYDZmkjh80Ju6ZW4H2//U=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=HoiIUPiz28WeOO1eZQH39wU0zD7HX9CDodETvKEGrXoC4aM3zKcUho/zM4g462SjXLvjZD3fU1uOJRQIKw3Rh/aXldKl9hMo6ASFKB3Nvfmhke9Af1Z/BcvloTsI6vxnEJhQa4ZnHQJIyepdXDSHpkYY16F0nXEJiQOXVL/v4/s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=eeLR3mjN; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="eeLR3mjN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711417153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WTaD6mudg7VhN+d5uv+mspRcpmC/ctCIgx4dGqOcYps=; b=eeLR3mjNt/8vZD/op0V2xrWfeyHNBoAdyc19DhJvqEMY9+YKNkGhYaWNwNTop33JzO/4zl JO5kr8w7yPr6lkSx8g2c83bBdUCdfsDNA8iyJ0zKUHyq6prQmy1oZ1cnVLBrnzP+SYO3sM 4wO27W2lgdk3FRKTEPdxaORubQ8aJdE= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-527-erYXpHexOpy9D2Rr6uPQoA-1; Mon, 25 Mar 2024 21:39:12 -0400 X-MC-Unique: erYXpHexOpy9D2Rr6uPQoA-1 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-33ec6c43a9cso4198825f8f.1 for ; Mon, 25 Mar 2024 18:39:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711417150; x=1712021950; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WTaD6mudg7VhN+d5uv+mspRcpmC/ctCIgx4dGqOcYps=; b=ndkXu95dhqjWMHNH9GR8+UwvmaZ2NRkXqz1XRtFhhMFoHXoDFsdnjj+r3XmhdGw3iS CAmdsp1S7eSs0YsDrL/WIyNccTlLpWCev8NR4V/NmpPMcH2cIMlcwkibj5Dka3sZc9Ew 5cuxqqkHA4/r64oO15aR6UwdjzQa0NEXRuGqgkACuaHip8d8nv5hsS1LYLMel4UAnCGA V+pgTgJWpgMLbHnkCa08GncwUUXE7QqflTxJxWfUtwQ5yFzk4WgAcpD+zT4H/dLLYjoB oUUcQKTFejYbwcs+7JTPn314HNJRNSOr5IQ+ZO4FFX/Cel+dXM9fWBwbi4UpqWxzL4iJ VGag== X-Forwarded-Encrypted: i=1; AJvYcCV+FytBppm2YAzvTtaC31qKZiucMj8NxnWNIoN/eqwvI0a9H0mg8yxuHmmWrD+N/xxgmTfvvqFhO8AwXelMVbclEFfsJOdrBB4pNA1R X-Gm-Message-State: AOJu0YywTbVnGCAiDwgo+iFGDZb9UeiLE8d9wPd2MoqlhRGYmDTJcDDA RBbFwIcQZaveeD1qJd8W99W18RM220ohbNVTUiw5Uk4gWu7bfQmjgmdXVgoX5bxyqLxRR818IVk AJS03hhg1goxfoHkld2WV+ykaNfLk5oH/a7sh6xt2JtMb0ZLGJ6QbObIfIgqwkY78TujdKwysvc q99+L2EFLlVmEHv1N80trXj2C1dKz0e3iGwC0JihddEWZxMQY= X-Received: by 2002:a5d:60c9:0:b0:341:865b:65c9 with SMTP id x9-20020a5d60c9000000b00341865b65c9mr7571759wrt.22.1711417150412; Mon, 25 Mar 2024 18:39:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHGAVRQvSolwKhN9lxaw7HUOTpaQxkO72OCZv+7YVENFG33vSWYVqDPTA5Pq/1JgP4Gbiom3nqQHbE/DxEgqA8= X-Received: by 2002:a5d:60c9:0:b0:341:865b:65c9 with SMTP id x9-20020a5d60c9000000b00341865b65c9mr7571746wrt.22.1711417150064; Mon, 25 Mar 2024 18:39:10 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <5d81d6d0-5afc-4d0e-8d2b-445d48921511@linux.dev> In-Reply-To: <5d81d6d0-5afc-4d0e-8d2b-445d48921511@linux.dev> From: Tao Liu Date: Tue, 26 Mar 2024 09:38:33 +0800 Message-ID: Subject: Re: Implementing .shutdown method for efa module To: Gal Pressman Cc: mrgolin@amazon.com, sleybo@amazon.com, jgg@ziepe.ca, leon@kernel.org, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Gal, On Mon, Mar 25, 2024 at 4:06=E2=80=AFPM Gal Pressman wrote: > > On 25/03/2024 4:10, Tao Liu wrote: > > Hi, > > > > Recently I experienced a kernel panic which is related to efa module > > when testing kexec -l && kexec -e to switch to a new kernel on AWS > > i4g.16xlarge instance. > > > > Here is the dmesg log: > > > > [ 6.379918] systemd[1]: Mounting FUSE Control File System... > > [ 6.381984] systemd[1]: Mounting Kernel Configuration File System... > > [ 6.383918] systemd[1]: Starting Apply Kernel Variables... > > [ 6.385430] systemd[1]: Started Journal Service. > > [ 6.394221] ACPI: bus type drm_connector registered > > [ 6.421408] systemd-journald[1263]: Received client request to > > flush runtime journal. > > [ 7.262543] efa 0000:00:1b.0: enabling device (0010 -> 0012) > > [ 7.432420] efa 0000:00:1b.0: Setup irq:191 name:efa-mgmnt@pci:0000:= 00:1b.0 > > [ 7.435581] efa 0000:00:1b.0 efa_0: IB device registered > > [ 7.885564] random: crng init done > > [ 8.139857] XFS (nvme0n1p2): Mounting V5 Filesystem > > d7003ecc-db6f-4bfb-bf92-60376b6a6563 > > [ 8.265233] XFS (nvme0n1p2): Ending clean mount > > [ 10.555612] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > > > > Red Hat Enterprise Linux 9.4 Beta (Plow) > > Kernel 5.14.0-425.el9.aarch64 on an aarch64 > > > > ip-10-0-27-226 login: [ 29.940381] kexec_core: Starting new kernel > > [ 30.079279] psci: CPU1 killed (polled 0 ms) > > [ 30.119222] psci: CPU2 killed (polled 0 ms) > > [ 30.199293] psci: CPU3 killed (polled 0 ms) > > [ 30.309214] psci: CPU4 killed (polled 0 ms) > > [ 30.379221] psci: CPU5 killed (polled 0 ms) > > [ 30.419210] psci: CPU6 killed (polled 0 ms) > > [ 30.489207] IRQ 191: no longer affine to CPU7 > > [ 30.489667] psci: CPU7 killed (polled 0 ms) > > ..snip... > > [ 33.849123] psci: CPU63 killed (polled 0 ms) > > [ 33.849943] Bye! > > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x413fd0c1] > > [ 0.000000] Linux version 5.14.0-417.el9.aarch64 > > (mockbuild@arm64-025.build.eng.bos.redhat.com) (gcc (GCC) 11.4.1 > > 20231218 (Red Hat 11.4.1-3), GNU ld version 2.35.2-42.el9) #1 SMP > > PREEMPT_DYNAMIC Thu Feb 1 21:23:03 EST 2024 > > ...snip... > > [ 1.012692] Freeing unused kernel memory: 6016K > > [ 2.370947] Checked W+X mappings: passed, no W+X pages found > > [ 2.370980] Run /init as init process > > [ 2.370982] with arguments: > > [ 2.370983] /init > > [ 2.370984] with environment: > > [ 2.370984] HOME=3D/ > > [ 2.370985] TERM=3Dlinux > > [ 2.373257] Kernel panic - not syncing: Attempted to kill init! > > exitcode=3D0x0000000b > > [ 2.373259] CPU: 1 PID: 1 Comm: init Not tainted 5.14.0-417.el9.aarc= h64 #1 > > [ 2.382240] Hardware name: Amazon EC2 i4g.16xlarge/, BIOS 1.0 11/1/2= 018 > > [ 2.383814] Call trace: > > [ 2.384410] dump_backtrace+0xa8/0x120 > > [ 2.385318] show_stack+0x1c/0x30 > > [ 2.386124] dump_stack_lvl+0x74/0x8c > > [ 2.387011] dump_stack+0x14/0x24 > > [ 2.387810] panic+0x158/0x368 > > [ 2.388553] do_exit+0x3a8/0x3b0 > > [ 2.389333] do_group_exit+0x38/0xa4 > > [ 2.390195] get_signal+0x7a4/0x810 > > [ 2.391044] do_signal+0x1bc/0x260 > > [ 2.391870] do_notify_resume+0x108/0x210 > > [ 2.392839] el0_da+0x154/0x160 > > [ 2.393603] el0t_64_sync_handler+0xdc/0x150 > > [ 2.394628] el0t_64_sync+0x17c/0x180 > > [ 2.395513] SMP: stopping secondary CPUs > > [ 2.396483] Kernel Offset: 0x586f04e00000 from 0xffff800008000000 > > [ 2.397934] PHYS_OFFSET: 0x40000000 > > [ 2.398774] CPU features: 0x0,00000101,70020143,10417a0b > > [ 2.400042] Memory Limit: none > > [ 2.400783] ---[ end Kernel panic - not syncing: Attempted to kill > > init! exitcode=3D0x0000000b ]--- > > > > In the dmesg log, I found "[ 30.489207] IRQ 191: no longer affine to > > CPU7" is suspicious, which is related to efa module. After blacklist > > efa module from automatic loading when bootup, the kernel panic issue > > doesn't appear again. > > > > It looks to me it is due to the efa being not properly shutdown during > > kexec, so the ongoing DMA/interrupts etc overwrite the memory range. > > > > Though the issue is reproduced on rhel's kernel, the upstream kernel > > [1] doesn't have the .shutdown method implemented either. Since I'm > > not very familiar with the efa driver, could you please implement the > > .shutdown method in drivers/infiniband/hw/efa/efa_main.c? Thanks in > > advance! > > Did you try to reproduce it on upstream kernel? > Thanks for your comments! No I haven't, I will give it a try. > > > > [1]: https://github.com/torvalds/linux/blob/master/drivers/infiniband/h= w/efa/efa_main.c#L674 > > > > Thanks, > > Tao Liu > > > > Try assigning efa_remove as the shutdown callback: > .shutdown =3D efa_remove, > > Does it fix it? Thanks, I will also try the code, and I will post the testing results. Thanks, Tao Liu >