From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754206AbbKXOUg (ORCPT ); Tue, 24 Nov 2015 09:20:36 -0500 Received: from mail-ig0-f169.google.com ([209.85.213.169]:33118 "EHLO mail-ig0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752112AbbKXOUd (ORCPT ); Tue, 24 Nov 2015 09:20:33 -0500 Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC To: Lan Tianyu , a.motakis@virtualopensystems.com, alex.williamson@redhat.com, b.reynal@virtualopensystems.com, bhelgaas@google.com, carolyn.wyborny@intel.com, donald.c.skidmore@intel.com, eddie.dong@intel.com, nrupal.jani@intel.com, agraf@suse.de, kvm@vger.kernel.org, pbonzini@redhat.com, qemu-devel@nongnu.org, emil.s.tantilov@intel.com, gerlitz.or@gmail.com, mark.d.rustad@intel.com, mst@redhat.com, eric.auger@linaro.org, intel-wired-lan@lists.osuosl.org, jeffrey.t.kirsher@intel.com, jesse.brandeburg@intel.com, john.ronciak@intel.com, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org, matthew.vick@intel.com, mitch.a.williams@intel.com, netdev@vger.kernel.org, shannon.nelson@intel.com, weiyang@linux.vnet.ibm.com, zajec5@gmail.com References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> From: Alexander Duyck Message-ID: <5654722D.4010409@gmail.com> Date: Tue, 24 Nov 2015 06:20:29 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/24/2015 05:38 AM, Lan Tianyu wrote: > This patchset is to propose a solution of adding live migration > support for SRIOV NIC. > > During migration, Qemu needs to let VF driver in the VM to know > migration start and end. Qemu adds faked PCI migration capability > to help to sync status between two sides during migration. > > Qemu triggers VF's mailbox irq via sending MSIX msg when migration > status is changed. VF driver tells Qemu its mailbox vector index > via the new PCI capability. In some cases(NIC is suspended or closed), > VF mailbox irq is freed and VF driver can disable irq injecting via > new capability. > > VF driver will put down nic before migration and put up again on > the target machine. > > Lan Tianyu (3): > VFIO: Add new ioctl cmd VFIO_GET_PCI_CAP_INFO > PCI: Add macros for faked PCI migration capability > Ixgbevf: Add migration support for ixgbevf driver > > drivers/net/ethernet/intel/ixgbevf/ixgbevf.h | 5 ++ > drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 102 ++++++++++++++++++++++ > drivers/vfio/pci/vfio_pci.c | 21 +++++ > drivers/vfio/pci/vfio_pci_config.c | 38 ++++++-- > drivers/vfio/pci/vfio_pci_private.h | 5 ++ > include/uapi/linux/pci_regs.h | 18 +++- > include/uapi/linux/vfio.h | 12 +++ > 7 files changed, 194 insertions(+), 7 deletions(-) I'm still not a fan of this approach. I really feel like this is something that should be resolved by extending the existing PCI hot-plug rather than trying to instrument this per driver. Then you will get the goodness for multiple drivers and multiple OSes instead of just one. An added advantage to dealing with this in the PCI hot-plug environment would be that you could then still do a hot-plug even if the guest didn't load a driver for the VF since you would be working with the PCI slot instead of the device itself. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42856) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a1ESV-00086F-99 for qemu-devel@nongnu.org; Tue, 24 Nov 2015 09:20:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a1ESQ-0001hg-DN for qemu-devel@nongnu.org; Tue, 24 Nov 2015 09:20:39 -0500 Received: from mail-ig0-x22e.google.com ([2607:f8b0:4001:c05::22e]:36502) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a1ESQ-0001hb-0e for qemu-devel@nongnu.org; Tue, 24 Nov 2015 09:20:34 -0500 Received: by igcph11 with SMTP id ph11so73343039igc.1 for ; Tue, 24 Nov 2015 06:20:33 -0800 (PST) References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> From: Alexander Duyck Message-ID: <5654722D.4010409@gmail.com> Date: Tue, 24 Nov 2015 06:20:29 -0800 MIME-Version: 1.0 In-Reply-To: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Lan Tianyu , a.motakis@virtualopensystems.com, alex.williamson@redhat.com, b.reynal@virtualopensystems.com, bhelgaas@google.com, carolyn.wyborny@intel.com, donald.c.skidmore@intel.com, eddie.dong@intel.com, nrupal.jani@intel.com, agraf@suse.de, kvm@vger.kernel.org, pbonzini@redhat.com, qemu-devel@nongnu.org, emil.s.tantilov@intel.com, gerlitz.or@gmail.com, mark.d.rustad@intel.com, mst@redhat.com, eric.auger@linaro.org, intel-wired-lan@lists.osuosl.org, jeffrey.t.kirsher@intel.com, jesse.brandeburg@intel.com, john.ronciak@intel.com, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org, matthew.vick@intel.com, mitch.a.williams@intel.com, netdev@vger.kernel.org, shannon.nelson@intel.com, weiyang@linux.vnet.ibm.com, zajec5@gmail.com On 11/24/2015 05:38 AM, Lan Tianyu wrote: > This patchset is to propose a solution of adding live migration > support for SRIOV NIC. > > During migration, Qemu needs to let VF driver in the VM to know > migration start and end. Qemu adds faked PCI migration capability > to help to sync status between two sides during migration. > > Qemu triggers VF's mailbox irq via sending MSIX msg when migration > status is changed. VF driver tells Qemu its mailbox vector index > via the new PCI capability. In some cases(NIC is suspended or closed), > VF mailbox irq is freed and VF driver can disable irq injecting via > new capability. > > VF driver will put down nic before migration and put up again on > the target machine. > > Lan Tianyu (3): > VFIO: Add new ioctl cmd VFIO_GET_PCI_CAP_INFO > PCI: Add macros for faked PCI migration capability > Ixgbevf: Add migration support for ixgbevf driver > > drivers/net/ethernet/intel/ixgbevf/ixgbevf.h | 5 ++ > drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 102 ++++++++++++++++++++++ > drivers/vfio/pci/vfio_pci.c | 21 +++++ > drivers/vfio/pci/vfio_pci_config.c | 38 ++++++-- > drivers/vfio/pci/vfio_pci_private.h | 5 ++ > include/uapi/linux/pci_regs.h | 18 +++- > include/uapi/linux/vfio.h | 12 +++ > 7 files changed, 194 insertions(+), 7 deletions(-) I'm still not a fan of this approach. I really feel like this is something that should be resolved by extending the existing PCI hot-plug rather than trying to instrument this per driver. Then you will get the goodness for multiple drivers and multiple OSes instead of just one. An added advantage to dealing with this in the PCI hot-plug environment would be that you could then still do a hot-plug even if the guest didn't load a driver for the VF since you would be working with the PCI slot instead of the device itself. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Date: Tue, 24 Nov 2015 06:20:29 -0800 Subject: [Intel-wired-lan] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC In-Reply-To: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> Message-ID: <5654722D.4010409@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On 11/24/2015 05:38 AM, Lan Tianyu wrote: > This patchset is to propose a solution of adding live migration > support for SRIOV NIC. > > During migration, Qemu needs to let VF driver in the VM to know > migration start and end. Qemu adds faked PCI migration capability > to help to sync status between two sides during migration. > > Qemu triggers VF's mailbox irq via sending MSIX msg when migration > status is changed. VF driver tells Qemu its mailbox vector index > via the new PCI capability. In some cases(NIC is suspended or closed), > VF mailbox irq is freed and VF driver can disable irq injecting via > new capability. > > VF driver will put down nic before migration and put up again on > the target machine. > > Lan Tianyu (3): > VFIO: Add new ioctl cmd VFIO_GET_PCI_CAP_INFO > PCI: Add macros for faked PCI migration capability > Ixgbevf: Add migration support for ixgbevf driver > > drivers/net/ethernet/intel/ixgbevf/ixgbevf.h | 5 ++ > drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 102 ++++++++++++++++++++++ > drivers/vfio/pci/vfio_pci.c | 21 +++++ > drivers/vfio/pci/vfio_pci_config.c | 38 ++++++-- > drivers/vfio/pci/vfio_pci_private.h | 5 ++ > include/uapi/linux/pci_regs.h | 18 +++- > include/uapi/linux/vfio.h | 12 +++ > 7 files changed, 194 insertions(+), 7 deletions(-) I'm still not a fan of this approach. I really feel like this is something that should be resolved by extending the existing PCI hot-plug rather than trying to instrument this per driver. Then you will get the goodness for multiple drivers and multiple OSes instead of just one. An added advantage to dealing with this in the PCI hot-plug environment would be that you could then still do a hot-plug even if the guest didn't load a driver for the VF since you would be working with the PCI slot instead of the device itself. - Alex