From: Dag B <dag@bakke.com>
To: linux-pci@vger.kernel.org
Subject: PCIE BAR resizing blocked by another BAR on same device?
Date: Wed, 17 Apr 2024 15:16:06 +0200 [thread overview]
Message-ID: <45df4a61-580d-49f9-96a9-13b091b9e256@bakke.com> (raw)
Hi.
In short, I have a GPU for which lspci reveals:
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 128MB, supported: 64MB 128MB 256MB 512MB
1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
In dmesg I see:
[ 0.517191] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 0.517238] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
I take it the location of BAR 3 right after BAR 1 explains why I get:
p53 # echo 9 > resource1_resize
-bash: echo: write error: No space left on device
Shrinking it and increasing it to the orginal size works.
Is there anything I can do with current kernel functionality to reserve
memory address space for the full-fat BAR 1? Or relocate it?
If not, is this something which *can* be worked around in the kernel?
And if so, does it belong with the PCI subsystem? Or the devicedriver
for the device in question?
Is there a good ELI13 resource explaining how resizable BAR works in Linux?
My current kernel command-line contains: pci=assign-busses,realloc
My GPU is attached via TB3 to a system for which resizable BAR is and
will remain a foreign concept in the BIOS.
dmesg excerpt below reflects resizing the BAR to 128MB and then back to
256MB.
[ 1730.091789] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: releasing
[ 1730.091875] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: releasing
[ 1730.092072] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: releasing
[ 1730.092151] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: assigned
[ 1730.092223] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 1730.092335] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 1730.092444] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 1730.092510] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 1730.092595] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 1730.092680] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 1730.092778] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 1730.092850] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 1730.092933] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 1730.093018] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]
[ 1759.817306] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: releasing
[ 1759.817394] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: releasing
[ 1759.817591] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: releasing
[ 1759.817668] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: assigned
[ 1759.817740] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: assigned
[ 1759.817853] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: assigned
[ 1759.817964] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 1759.818035] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 1759.818120] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 1759.818204] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 1759.818303] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 1759.818374] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 1759.818459] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 1759.818544] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]
[ 1769.797178] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: releasing
[ 1769.797241] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: releasing
[ 1769.797417] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: releasing
[ 1769.797473] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: can't assign; no space
[ 1769.797515] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: failed to assign
[ 1769.797557] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
can't assign; no space
[ 1769.797594] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
failed to assign
[ 1769.797630] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
can't assign; no space
[ 1769.797666] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
failed to assign
[ 1769.797703] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 1769.797761] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 1769.797829] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 1769.797895] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 1769.797972] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 1769.798027] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 1769.798089] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 1769.798155] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]
[ 1769.798270] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: assigned
[ 1769.798358] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: assigned
[ 2669.324929] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: releasing
[ 2669.324992] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: releasing
[ 2669.325164] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: releasing
[ 2669.325219] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: can't assign; no space
[ 2669.327023] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: failed to assign
[ 2669.328798] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
can't assign; no space
[ 2669.330482] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
failed to assign
[ 2669.331104] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
can't assign; no space
[ 2669.331682] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
failed to assign
[ 2669.332258] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 2669.332855] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 2669.333444] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 2669.334130] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 2669.334821] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 2669.335460] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 2669.336063] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 2669.336657] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]
[ 2669.337442] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: assigned
[ 2669.338073] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: assigned
[ 2673.200263] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: releasing
[ 2673.201935] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: releasing
[ 2673.203801] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: releasing
[ 2673.205461] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: assigned
[ 2673.206197] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 2673.206800] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 2673.207534] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 2673.208143] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 2673.208734] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 2673.209323] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 2673.209916] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 2673.210526] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 2673.211170] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 2673.211755] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]
Unsure what else from dmesg or kernel config is relevant here. Can post
the full 100k here or somewhere else. Or excerpts.
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006f23bfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000006f23c000-0x000000006f23cfff]
reserved
[ 0.000000] BIOS-e820: [mem 0x000000006f23d000-0x000000007a792fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007a793000-0x000000007fa6cfff]
reserved
[ 0.000000] BIOS-e820: [mem 0x000000007fa6d000-0x000000007fca9fff]
ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x000000007fcaa000-0x000000007fd0efff]
ACPI data
[ 0.000000] BIOS-e820: [mem 0x000000007fd0f000-0x000000007fd0ffff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007fd10000-0x0000000087ffffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000887fffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000088800000-0x000000008f7fffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff]
reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000106c7fffff] usable
[ 0.000000] efi: Not removing mem55: MMIO
range=[0xfe010000-0xfe010fff] (4KB) from e820 map
[ 0.000003] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000006] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.503260] e820: reserve RAM buffer [mem 0x0009f000-0x0009ffff]
[ 0.503262] e820: reserve RAM buffer [mem 0x6f23c000-0x6fffffff]
[ 0.503263] e820: reserve RAM buffer [mem 0x7a793000-0x7bffffff]
[ 0.503264] e820: reserve RAM buffer [mem 0x7fd10000-0x7fffffff]
[ 0.503265] e820: reserve RAM buffer [mem 0x88800000-0x8bffffff]
[ 0.503266] e820: reserve RAM buffer [mem 0x106c800000-0x106fffffff]
[ 0.255546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.445755] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
ClockPM Segments MSI HPX-Type3]
[ 0.447023] acpi PNP0A08:00: _OSC: platform does not support [AER]
[ 0.449507] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME
PCIeCapability LTR]
[ 0.449511] acpi PNP0A08:00: FADT indicates ASPM is unsupported,
using BIOS configuration
[ 0.522380] caller snb_uncore_imc_init_box+0x73/0xd0 mapping multiple
BARs
Thanks,
Dag B
next reply other threads:[~2024-04-17 13:16 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-17 13:16 Dag B [this message]
2024-04-17 15:13 ` PCIE BAR resizing blocked by another BAR on same device? Bjorn Helgaas
2024-04-18 7:51 ` Christian König
2024-04-18 10:42 ` Dag B
2024-04-18 12:24 ` Christian König
2024-04-18 13:13 ` Dag B
2024-04-18 22:54 ` Dag B
2024-04-19 15:19 ` Ilpo Järvinen
2024-04-19 15:31 ` Christian König
2024-04-27 20:42 ` Dag B
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=45df4a61-580d-49f9-96a9-13b091b9e256@bakke.com \
--to=dag@bakke.com \
--cc=linux-pci@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).