From: Bjorn Helgaas <bhelgaas@google.com>
To: Gavin Shan <gwshan@linux.vnet.ibm.com>
Cc: Wei Yang <weiyang@linux.vnet.ibm.com>,
linuxppc-dev@lists.ozlabs.org, linux-pci@vger.kernel.org
Subject: Re: [PATCH V7 06/10] powerpc/eeh: Create PE for VFs
Date: Wed, 3 Jun 2015 10:46:38 -0500 [thread overview]
Message-ID: <20150603154638.GL3631@google.com> (raw)
In-Reply-To: <20150603051023.GA26652@gwshan>
On Wed, Jun 03, 2015 at 03:10:23PM +1000, Gavin Shan wrote:
> On Wed, Jun 03, 2015 at 11:31:42AM +0800, Wei Yang wrote:
> >On Mon, Jun 01, 2015 at 06:46:45PM -0500, Bjorn Helgaas wrote:
> >>On Tue, May 19, 2015 at 06:50:08PM +0800, Wei Yang wrote:
> >>> Current EEH recovery code works with the assumption: the PE has primary
> >>> bus. Unfortunately, that's not true to VF PEs, which generally contains
> >>> one or multiple VFs (for VF group case). The patch creates PEs for VFs
> >>> at PCI final fixup time. Those PEs for VFs are indentified with newly
> >>> introduced flag EEH_PE_VF so that we handle them differently during
> >>> EEH recovery.
> >>>
> >>> [gwshan: changelog and code refactoring]
> >>> Signed-off-by: Wei Yang <weiyang@linux.vnet.ibm.com>
> >>> Acked-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> >>> ---
> >>> arch/powerpc/include/asm/eeh.h | 1 +
> >>> arch/powerpc/kernel/eeh_pe.c | 10 ++++++++--
> >>> arch/powerpc/platforms/powernv/eeh-powernv.c | 17 +++++++++++++++++
> >>> 3 files changed, 26 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h
> >>> index 1b3614d..c1fde48 100644
> >>> --- a/arch/powerpc/include/asm/eeh.h
> >>> +++ b/arch/powerpc/include/asm/eeh.h
> >>> @@ -70,6 +70,7 @@ struct pci_dn;
> >>> #define EEH_PE_PHB (1 << 1) /* PHB PE */
> >>> #define EEH_PE_DEVICE (1 << 2) /* Device PE */
> >>> #define EEH_PE_BUS (1 << 3) /* Bus PE */
> >>> +#define EEH_PE_VF (1 << 4) /* VF PE */
> >>>
> >>> #define EEH_PE_ISOLATED (1 << 0) /* Isolated PE */
> >>> #define EEH_PE_RECOVERING (1 << 1) /* Recovering PE */
> >>> diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c
> >>> index 35f0b62..260a701 100644
> >>> --- a/arch/powerpc/kernel/eeh_pe.c
> >>> +++ b/arch/powerpc/kernel/eeh_pe.c
> >>> @@ -299,7 +299,10 @@ static struct eeh_pe *eeh_pe_get_parent(struct eeh_dev *edev)
> >>> * EEH device already having associated PE, but
> >>> * the direct parent EEH device doesn't have yet.
> >>> */
> >>> - pdn = pdn ? pdn->parent : NULL;
> >>> + if (edev->physfn)
> >>> + pdn = pci_get_pdn(edev->physfn);
> >>> + else
> >>> + pdn = pdn ? pdn->parent : NULL;
> >>> while (pdn) {
> >>> /* We're poking out of PCI territory */
> >>> parent = pdn_to_eeh_dev(pdn);
> >>> @@ -382,7 +385,10 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
> >>> }
> >>>
> >>> /* Create a new EEH PE */
> >>> - pe = eeh_pe_alloc(edev->phb, EEH_PE_DEVICE);
> >>> + if (edev->physfn)
> >>> + pe = eeh_pe_alloc(edev->phb, EEH_PE_VF);
> >>> + else
> >>> + pe = eeh_pe_alloc(edev->phb, EEH_PE_DEVICE);
> >>> if (!pe) {
> >>> pr_err("%s: out of memory!\n", __func__);
> >>> return -ENOMEM;
> >>> diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
> >>> index ce738ab..c505036 100644
> >>> --- a/arch/powerpc/platforms/powernv/eeh-powernv.c
> >>> +++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
> >>> @@ -1520,6 +1520,23 @@ static struct eeh_ops pnv_eeh_ops = {
> >>> .restore_config = pnv_eeh_restore_config
> >>> };
> >>>
> >>> +static void pnv_eeh_vf_final_fixup(struct pci_dev *pdev)
> >>> +{
> >>> + struct pci_dn *pdn = pci_get_pdn(pdev);
> >>> +
> >>> + if (!pdev->is_virtfn)
> >>> + return;
> >>> +
> >>> + /*
> >>> + * The following operations will fail if VF's sysfs files
> >>> + * aren't created or its resources aren't finalized.
> >>> + */
> >>
> >>I don't understand this comment. "The following operations" seems to refer
> >>to eeh_add_device_early() and eeh_add_device_late(), and
> >>"VF's sysfs files being created" seems to refer to eeh_sysfs_add_device().
> >>
> >>So the comment suggests that eeh_add_device_early() and
> >>eeh_add_device_late() will fail because they're called before
> >>eeh_sysfs_add_device(). So I think you must be talking about some other
> >>"following operations," not eeh_add_device_early() and
> >>eeh_add_device_late().
> >
> >Sorry for this confusion.
> >
> >The comment here wants to say the eeh_sysfs_add_device() will fail if the VF's
> >sysfs is not created well. Or it will fail if the VF's resources are not set
> >properly, since we would cache the VF's BAR in eeh_add_device_late().
> >
> >Gavin,
> >
> >If my understanding is not correct please let me know.
> >
>
> It's correct. "The following operations" refers to eeh_add_device_late()
> and eeh_sysfs_add_device(). The former one requires the resources for
> one particular PCI device (VF here) are finalized (assigned). eeh_sysfs_add_device()
> will fail if the sysfs entry for the PCI device isn't populated yet.
eeh_add_device_late() contains several things that read config space:
eeh_save_bars() caches the entire config header, and
eeh_addr_cache_insert_dev() looks at the device resources (which are
determined by BARs in config space). I think this is an error-prone
approach. I think it would be simpler and safer for you to capture what
you need in your PCI config accessors.
eeh_add_device_late() also contains code to deal with an EEH cache that
"might not be removed correctly because of unbalanced kref to the device
during unplug time." That's unrelated to this patch series, but it sounds
... like a hacky workaround for some bug in the unplug path.
> >>> + eeh_add_device_early(pdn);
> >>> + eeh_add_device_late(pdev);
> >>> + eeh_sysfs_add_device(pdev);
> >>> +}
> >>> +DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, pnv_eeh_vf_final_fixup);
> >>
> >>Ugh. This is powerpc code, but I don't like using fixups as a hook like
> >>this. There is a pcibios_add_device() -- could this be done there?
> >>
> >
> >I don't like it neither :-) But looks we can't put it in the
> >pcibios_add_device().
> >
> >>If not, what happens after pcibios_add_device() that is required for this
> >>code? Maybe we need a pcibios_bus_add_device() hook?
> >
> >The pnv_eeh_vf_final_fixup() will try to create sysfs for VFs. This requires
> >the VF sysfs(kobj) is initialized properly. If we put these into
> >pcibios_add_device(), the eeh_sysfs_add_device() would fail.
> >
> >Below is the call flow for your reference:
> >
> >pci_device_add()
> > pcibios_add_device()
> > device_add() <--- kobj initialized here
> >
>
> We can put it into pcibios_bus_add_device(), but we don't it currently. If
> Bjorn agree to add pcibios_bus_add_device(), I'm fine to move the block code
> there.
I think I'm OK with adding a pcibios_bus_add_device(). I think that would
be better than using the fixup mechanism for this.
Bjorn
next prev parent reply other threads:[~2015-06-03 15:46 UTC|newest]
Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-05-19 1:35 [PATCH V6 00/10] VF EEH on Power8 Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 01/10] PCI/IOV: Rename and export virtfn_add/virtfn_remove Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 5:24 ` Wei Yang
2015-05-19 5:24 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 02/10] powerpc/pci: Cache VF index in pci_dn Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 03/10] powerpc/pci: Remove VFs prior to PF Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 04/10] powerpc/eeh: Trace first 7 BARs in address cache Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 05/10] powerpc/powernv: EEH device for VF Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 06/10] powerpc/eeh: Create PE for VFs Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 07/10] powerpc/powernv: Support EEH reset for VF PE Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 08/10] powerpc/powernv: Support PCI config restore for VFs Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 09/10] powerpc/eeh: Support error recovery for VF PE Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 1:35 ` [PATCH V6 10/10] powerpc/powernv: compound PE for VFs Wei Yang
2015-05-19 1:35 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 00/10] VF EEH on Power8 Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 01/10] PCI/IOV: Rename and export virtfn_add/virtfn_remove Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-06-02 17:19 ` Bjorn Helgaas
2015-06-03 1:38 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 02/10] powerpc/pci: Cache VF index in pci_dn Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 03/10] powerpc/pci: Remove VFs prior to PF Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-06-01 23:20 ` Bjorn Helgaas
2015-06-02 3:44 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 04/10] powerpc/eeh: Trace first 7 BARs in address cache Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-06-01 23:32 ` Bjorn Helgaas
2015-06-02 3:51 ` Wei Yang
2015-06-02 4:11 ` Gavin Shan
2015-06-03 1:47 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 05/10] powerpc/powernv: EEH device for VF Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 06/10] powerpc/eeh: Create PE for VFs Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-06-01 23:46 ` Bjorn Helgaas
2015-06-03 3:31 ` Wei Yang
2015-06-03 5:10 ` Gavin Shan
2015-06-03 15:46 ` Bjorn Helgaas [this message]
2015-06-04 1:25 ` Gavin Shan
2015-06-04 5:46 ` Wei Yang
2015-06-04 7:10 ` Gavin Shan
2015-06-16 8:50 ` Wei Yang
2015-06-16 13:22 ` Bjorn Helgaas
2015-06-01 23:49 ` Bjorn Helgaas
2015-06-03 3:39 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 07/10] powerpc/powernv: Support EEH reset for VF PE Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 08/10] powerpc/powernv: Support PCI config restore for VFs Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-06-02 0:01 ` Bjorn Helgaas
2015-06-03 1:37 ` Wei Yang
2015-06-03 5:14 ` Gavin Shan
2015-05-19 10:50 ` [PATCH V7 09/10] powerpc/eeh: Support error recovery for VF PE Wei Yang
2015-05-19 10:50 ` Wei Yang
2015-05-19 10:50 ` [PATCH V7 10/10] powerpc/powernv: compound PE for VFs Wei Yang
2015-05-19 10:50 ` Wei Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150603154638.GL3631@google.com \
--to=bhelgaas@google.com \
--cc=gwshan@linux.vnet.ibm.com \
--cc=linux-pci@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=weiyang@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.