From: Hans de Goede <hdegoede@redhat.com>
To: Jarkko Sakkinen <jarkko@kernel.org>
Cc: Jerry Snitselaar <jsnitsel@redhat.com>,
Matthew Garrett <mjg59@google.com>,
James Bottomley <James.Bottomley@hansenpartnership.com>,
linux-integrity <linux-integrity@vger.kernel.org>
Subject: Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
Date: Wed, 26 May 2021 21:03:26 +0200 [thread overview]
Message-ID: <81aa6340-f854-8f26-f599-03b116cc3c29@redhat.com> (raw)
In-Reply-To: <4f525bca-3836-7f5c-7913-e54d620473aa@redhat.com>
Hi,
On 5/11/21 10:37 AM, Hans de Goede wrote:
> Hi,
>
> On 5/10/21 7:25 PM, Jarkko Sakkinen wrote:
>> On Sat, May 08, 2021 at 11:07:43AM +0200, Hans de Goede wrote:
>>> Hi Jarko,
>>>
>>> On 3/16/21 8:18 PM, Jarkko Sakkinen wrote:
>>>> On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
>>>>> Hi,
>>>>>
>>>>> On 2/11/21 2:09 PM, Hans de Goede wrote:
>>>>>> Hi Jerry,
>>>>>>
>>>>>> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
>>>>>> handling. At least I think those are the cause I did not dive any deeper,
>>>>>> I just noticed that we (Fedora) have been receiving an aweful lot of
>>>>>> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
>>>>>>
>>>>>> See for example:
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
>>>>>>
>>>>>> Those are just the 3 which landed in my inbox today, for much more see:
>>>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>>>>> (this shows 18 bugs atm).
>>>>>>
>>>>>> These were reported through the Fedora ABRT tools which automatically
>>>>>> collects backtraces, the bugs have links to the ABRT reports, e.g. :
>>>>>> https://retrace.fedoraproject.org/faf/reports/28155/
>>>>>> https://retrace.fedoraproject.org/faf/reports/37107/
>>>>>>
>>>>>> The 28155 report says that so far there have been 308,412 (ouch) automatic
>>>>>> uploads of that particular variant of these backtraces
>>>>>>
>>>>>> Note the second (37107) retrace report is about this happening
>>>>>> on resume, rather then on probe/tpm_tis_probe_irq_... time.
>>>>>>
>>>>>> Did your work on this work land in 5.10 ? Or could it be that the
>>>>>> issue is an incomplete backport to the 5.10.y stable series ?
>>>>>
>>>>> Ping ?
>>>>>
>>>>> It is raining bug-reports about this:
>>>>>
>>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>>>>
>>>>> Currently lists 25 bugs and that is excluding bugs which have already
>>>>> been marked as a duplicate.
>>>>>
>>>>> Can someone involved in the patch-series which is causing this regression
>>>>> please take a look at these kernel backtraces ?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Hans
>>>>
>>>> I incorporated two fixes to this issue to my last PR, which were taken
>>>> to the mainline. What is the situation with the mainline?
>>>
>>> Thank you for your reply and sorry for being slow to respond.
>>>
>>> Is this expected to be fixed in 5.11, or when you say mainline you
>>> main Linus' master branch / so the fixes are in 5.12 only ?
>>>
>>> The reason I'm asking is because we just received another bugreport
>>> about this against 5.11.17. The bug is marked private (our tool to
>>> automatically file bugs for kernel backtraces does this) so let me
>>> just copy and paste the trace here:
>>>
>>> WARNING: CPU: 0 PID: 3060 at drivers/char/tpm/tpm_tis_core.c:205
>>> tpm_tis_status+0x66/0x70
>>>
>>> CPU: 0 PID: 3060 Comm: systemd-sleep Not tainted 5.11.17-200.fc33.x86_64 #1
>>> Hardware name: Hewlett-Packard HP ProBook 6460b/161D, BIOS 68SCE Ver. F.63
>>> 05/27/2016
>>> RIP: 0010:tpm_tis_status+0x66/0x70
>>> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d 38 02 56 01 00 75 f0 48 c7 c7 94 67
>>> 43 96 88 44 24 07 c6 05 24 02 56 01 01 e8 4a 53 3c 00 <0f> 0b 0f b6 44 24 07 eb
>>> d0 90 66 66 66 66 90 41 57 41 56 41 55 41
>>> RSP: 0018:ffffaac581427b10 EFLAGS: 00010282
>>> RAX: 000000000000001b RBX: ffff9dc800b93000 RCX: ffff9dc83b418ac8
>>> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9dc83b418ac0
>>> RBP: ffff9dc800b93000 R08: ffffffff96a64ec0 R09: ffffaac581427ab0
>>> R10: 0000000000000001 R11: 000000002d2d2d2d R12: ffff9dc80667c268
>>> R13: ffff9dc801fd1000 R14: 0000000000000000 R15: ffffaac581427bca
>>> FS: 00007f8f0f32c000(0000) GS:ffff9dc83b400000(0000) knlGS:0000000000000000
>>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> CR2: 0000557044ec6c86 CR3: 0000000062e92001 CR4: 00000000000606f0
>>> Call Trace:
>>> tpm_tis_send_data+0x2b/0x230
>>> tpm_tis_send_main+0x1e/0xe0
>>> tpm_transmit+0xd8/0x3d0
>>> tpm_transmit_cmd+0x25/0x90
>>> tpm1_do_selftest+0x88/0x130
>>> ? _cond_resched+0x16/0x40
>>> tpm_tis_resume+0x4d/0x120
>>> ? pnpacpi_resume+0x1b/0xa0
>>> ? pnp_bus_suspend+0x10/0x10
>>> pnp_bus_resume+0x63/0x90
>>> dpm_run_callback+0x4c/0x120
>>> device_resume+0xa7/0x200
>>> dpm_resume+0xce/0x2c0
>>> dpm_resume_end+0xd/0x20
>>> suspend_devices_and_enter+0x195/0x750
>>> pm_suspend.cold+0x329/0x374
>>> state_store+0x71/0xd0
>>> kernfs_fop_write_iter+0x124/0x1b0
>>> new_sync_write+0x108/0x180
>>> vfs_write+0x1bc/0x270
>>> ksys_write+0x4f/0xc0
>>> do_syscall_64+0x33/0x40
>>> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> RIP: 0033:0x7f8f102ec4e7
>>> Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64
>>> 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51
>>> c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
>>> RSP: 002b:00007ffe87216bf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>>> RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f8f102ec4e7
>>> RDX: 0000000000000004 RSI: 00007ffe87216ce0 RDI: 0000000000000004
>>> RBP: 00007ffe87216ce0 R08: 000055c485d835e0 R09: 00007f8f103830c0
>>> R10: 00007f8f10382fc0 R11: 0000000000000246 R12: 0000000000000004
>>> R13: 000055c485d7f650 R14: 0000000000000004 R15: 00007f8f103bf720
>>>
>>> Regards,
>>>
>>> Hans
>>
>> I sent a couple fixes (cc'd to you).
>
> I've seen the fixes, thank you.
>
> I'll probably add these as downstream patches to the Fedora 5.12 kernels for now
> and see if that helps.
I'm afraid that we are still getting tpm irq kernel-backtrace reports with
5.12.6 which has the 2 fixes (AFAICT), here are 2 of them:
WARNING: CPU: 5 PID: 142 at drivers/char/tpm/tpm_tis_core.c:205
tpm_tis_status+0x66/0x70
Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer xt_CHECKSUM
xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp
llc ccm michael_mic nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
nf_reject_ipv6 nft_reject nft_ct cmac nft_chain_nat ip6table_nat
ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security
ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter
snd_soc_skl_hda_dsp snd_soc_hdac_hdmi qrtr_mhi bnep snd_hda_codec_hdmi
snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl
snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation
soundwire_cadence sunrpc snd_sof_intel_hda snd_sof_pci iTCO_wdt snd_sof
intel_pmc_bxt iTCO_vendor_support snd_sof_xtensa_dsp snd_soc_hdac_hda
snd_hda_ext_core
snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core mei_hdcp
x86_pkg_temp_thermal qrtr snd_compress intel_pmt_telemetry intel_powerclamp
intel_rapl_msr ns snd_pcm_dmaengine intel_pmt_class ac97_bus dell_laptop
ath11k_pci coretemp ledtrig_audio ath11k dell_smm_hwmon kvm_intel snd_hda_intel
snd_intel_dspcfg snd_intel_sdw_acpi qmi_helpers kvm snd_hda_codec mac80211
snd_hda_core snd_hwdep snd_seq snd_seq_device irqbypass intel_cstate dell_wmi
intel_uncore snd_pcm dell_smbios dcdbas cfg80211 snd_timer pcspkr snd wmi_bmof
dell_wmi_sysman dell_wmi_descriptor i2c_i801 soundcore i2c_smbus mhi uvcvideo
libarc4 hci_uart videobuf2_vmalloc videobuf2_memops mei_me videobuf2_v4l2 vfat
mei videobuf2_common fat btqca joydev hid_sensor_als btrtl idma64 btbcm
hid_sensor_trigger videodev hid_sensor_iio_common processor_thermal_device
btintel industrialio_triggered_buffer processor_thermal_rfim kfifo_buf
processor_thermal_mbox mc industrialio processor_thermal_rapl bluetooth
thunderbolt
intel_rapl_common intel_pmt intel_soc_dts_iosf ecdh_generic ucsi_acpi rfkill
typec_ucsi typec ecc int3403_thermal intel_hid int340x_thermal_zone
sparse_keymap int3400_thermal acpi_pad acpi_tad acpi_thermal_rel zram ip_tables
dm_crypt trusted hid_sensor_hub intel_ishtp_loader intel_ishtp_hid
hid_multitouch i915 i2c_algo_bit rtsx_pci_sdmmc nvme drm_kms_helper mmc_core
nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel cec drm
ghash_clmulni_intel rtsx_pci serio_raw intel_ish_ipc intel_ishtp vmd
i2c_hid_acpi i2c_hid wmi video pinctrl_tigerlake fuse
CPU: 5 PID: 142 Comm: kworker/5:1 Not tainted 5.12.6-300.fc34.x86_64 #1
Hardware name: Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021
Workqueue: tpm_dev_wq tpm_dev_async_work
RIP: 0010:tpm_tis_status+0x66/0x70
Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c
44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb
d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
RSP: 0018:ffffafc80037bd40 EFLAGS: 00010286
RAX: 000000000000001b RBX: ffff9c8c47cff000 RCX: 0000000000000027
RDX: ffff9c93af7585c8 RSI: 0000000000000001 RDI: ffff9c93af7585c0
RBP: ffff9c8c47cff000 R08: 0000000000000000 R09: ffffafc80037bb70
R10: ffffafc80037bb68 R11: ffffffffa6b45f28 R12: ffff9c8c47df5aa8
R13: ffff9c8c4d14e0ba R14: 0000000000000000 R15: ffffafc80037bdf2
FS: 0000000000000000(0000) GS:ffff9c93af740000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fadaa0bd490 CR3: 0000000170c42005 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
tpm_tis_send_data+0x2b/0x230
? tpm_tcg_read_bytes+0x30/0x50
tpm_tis_send_main+0x1e/0xe0
tpm_transmit+0xd6/0x3d0WARNING: CPU: 2 PID: 1 at drivers/char/tpm/tpm_tis_core.c:205
tpm_tis_status+0x66/0x70
Modules linked in:
CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.12.6-300.fc34.x86_64 #1
Hardware name: Dell Inc. XPS 13 9310/0GG9PT, BIOS 2.2.0 04/06/2021
RIP: 0010:tpm_tis_status+0x66/0x70
Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c
44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb
d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
RSP: 0018:ffffad554006bae0 EFLAGS: 00010282
RAX: 000000000000001b RBX: ffff96bf471d5000 RCX: ffffffffa6b15ee8
RDX: c0000000ffffdfff RSI: 00000000ffffdfff RDI: ffffffffa752ec6c
RBP: ffff96bf471d5000 R08: 0000000000000000 R09: ffffad554006b910
R10: ffffad554006b908 R11: ffffffffa6b45f28 R12: ffff96bf472f61a8
R13: ffff96bf47d87000 R14: 0000000000000000 R15: ffffad554006bb92
FS: 0000000000000000(0000) GS:ffff96c2bf680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff201c82958 CR3: 0000000010a10001 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
tpm_tis_send_data+0x2b/0x230
tpm_tis_send_main+0x1e/0xe0
tpm_transmit+0xd6/0x3d0
tpm_transmit_cmd+0x25/0x90
tpm2_pcr_extend+0x1f9/0x240
tpm_pcr_extend+0xa1/0xb0
ima_add_template_entry+0x16e/0x220
? ima_store_template+0x3a/0xb0
? hash_setup+0xc5/0xc5
ima_add_boot_aggregate+0xd4/0x13e
ima_init+0x51/0x94
init_ima+0x23/0xb5
? hash_setup+0xc5/0xc5
do_one_initcall+0x44/0x1d0
kernel_init_freeable+0x1da/0x221
? rest_init+0xb4/0xb4
kernel_init+0xa/0x11c
ret_from_fork+0x1f/0x30
tpm_dev_transmit.constprop.0+0x47/0xa0
tpm_dev_async_work+0x62/0x90
process_one_work+0x1ec/0x380
worker_thread+0x53/0x3e0
? process_one_work+0x380/0x380
kthread+0x11b/0x140
? kthread_associate_blkcg+0xa0/0xa0
ret_from_fork+0x1f/0x30
Regards,
Hans
next prev parent reply other threads:[~2021-05-26 19:03 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-11 13:09 Recent tpm_tis IRQ handling changes are causing kernel backtraces Hans de Goede
2021-03-16 15:34 ` Hans de Goede
2021-03-16 19:18 ` Jarkko Sakkinen
2021-05-08 9:07 ` Hans de Goede
2021-05-10 17:25 ` Jarkko Sakkinen
2021-05-11 8:37 ` Hans de Goede
2021-05-11 23:48 ` Jarkko Sakkinen
2021-05-26 19:03 ` Hans de Goede [this message]
2021-05-27 14:00 ` Jarkko Sakkinen
2021-05-27 15:27 ` Hans de Goede
[not found] ` <20210531043616.u3v25qzkkrik5apq@kernel.org>
2021-05-31 8:24 ` Hans de Goede
2021-06-01 18:02 ` Jarkko Sakkinen
2021-06-01 16:04 ` Hans de Goede
2021-06-01 18:03 ` Jarkko Sakkinen
2021-06-14 13:33 ` Hans de Goede
2021-06-15 13:01 ` Jarkko Sakkinen
2021-06-15 13:59 ` Hans de Goede
2021-06-23 13:37 ` Jarkko Sakkinen
2021-06-21 12:04 ` Hans de Goede
2021-06-23 13:40 ` Jarkko Sakkinen
2021-06-23 13:54 ` Hans de Goede
2021-06-29 18:04 ` Recent tpm_tis IRQ handling changes are causing kernel backtraces] Jarkko Sakkinen
2021-06-29 19:14 ` Hans de Goede
2021-06-29 22:05 ` Jarkko Sakkinen
2021-06-30 12:47 ` Hans de Goede
2021-06-30 13:36 ` Hans de Goede
2021-07-09 18:44 ` Jarkko Sakkinen
2021-07-17 16:10 ` Hans de Goede
2021-07-27 2:50 ` Jarkko Sakkinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=81aa6340-f854-8f26-f599-03b116cc3c29@redhat.com \
--to=hdegoede@redhat.com \
--cc=James.Bottomley@hansenpartnership.com \
--cc=jarkko@kernel.org \
--cc=jsnitsel@redhat.com \
--cc=linux-integrity@vger.kernel.org \
--cc=mjg59@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.