From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754357AbbKYFa3 (ORCPT ); Wed, 25 Nov 2015 00:30:29 -0500 Received: from mail-ig0-f174.google.com ([209.85.213.174]:36621 "EHLO mail-ig0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751291AbbKYFa1 convert rfc822-to-8bit (ORCPT ); Wed, 25 Nov 2015 00:30:27 -0500 MIME-Version: 1.0 In-Reply-To: <56552888.90108@intel.com> References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> Date: Tue, 24 Nov 2015 21:30:26 -0800 Message-ID: Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC From: Alexander Duyck To: Lan Tianyu Cc: a.motakis@virtualopensystems.com, Alex Williamson , b.reynal@virtualopensystems.com, Bjorn Helgaas , Carolyn Wyborny , "Skidmore, Donald C" , eddie.dong@intel.com, nrupal.jani@intel.com, Alexander Graf , kvm@vger.kernel.org, Paolo Bonzini , qemu-devel@nongnu.org, "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , "Michael S. Tsirkin" , Eric Auger , intel-wired-lan , Jeff Kirsher , "Brandeburg, Jesse" , "Ronciak, John" , linux-api@vger.kernel.org, "linux-kernel@vger.kernel.org" , "Vick, Matthew" , Mitch Williams , Netdev , "Nelson, Shannon" , Wei Yang , zajec5@gmail.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Nov 24, 2015 at 7:18 PM, Lan Tianyu wrote: > On 2015年11月24日 22:20, Alexander Duyck wrote: >> I'm still not a fan of this approach. I really feel like this is >> something that should be resolved by extending the existing PCI hot-plug >> rather than trying to instrument this per driver. Then you will get the >> goodness for multiple drivers and multiple OSes instead of just one. An >> added advantage to dealing with this in the PCI hot-plug environment >> would be that you could then still do a hot-plug even if the guest >> didn't load a driver for the VF since you would be working with the PCI >> slot instead of the device itself. >> >> - Alex > > Hi Alex: > What's you mentioned seems the bonding driver solution. > Paper "Live Migration with Pass-through Device for Linux VM" describes > it. It does VF hotplug during migration. In order to maintain Network > connection when VF is out, it takes advantage of Linux bonding driver to > switch between VF NIC and emulated NIC. But the side affects, that > requires VM to do additional configure and the performance during > switching two NIC is not good. No, what I am getting at is that you can't go around and modify the configuration space for every possible device out there. This solution won't scale. If you instead moved the logic for notifying the device into a separate mechanism such as making it a part of the hot-plug logic then you only have to write the code once per OS in order to get the hot-plug capability to pause/resume the device. What I am talking about is not full hot-plug, but rather to extend the existing hot-plug in Qemu and the Linux kernel to support a "pause/resume" functionality. The PCI hot-plug specification calls out the option of implementing something like this, but we don't currently have support for it. I just feel doing it through PCI hot-plug messages will scale much better as you could likely make use of the power management suspend/resume calls to take care of most of the needed implementation details. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59359) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a1Sex-0006dS-Lx for qemu-devel@nongnu.org; Wed, 25 Nov 2015 00:30:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a1Sew-0004uc-Kq for qemu-devel@nongnu.org; Wed, 25 Nov 2015 00:30:27 -0500 Received: from mail-ig0-x22d.google.com ([2607:f8b0:4001:c05::22d]:38808) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a1Sew-0004uY-G9 for qemu-devel@nongnu.org; Wed, 25 Nov 2015 00:30:26 -0500 Received: by igbxm8 with SMTP id xm8so30711153igb.1 for ; Tue, 24 Nov 2015 21:30:26 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <56552888.90108@intel.com> References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> Date: Tue, 24 Nov 2015 21:30:26 -0800 Message-ID: From: Alexander Duyck Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Lan Tianyu Cc: Wei Yang , "Tantilov, Emil S" , kvm@vger.kernel.org, "Michael S. Tsirkin" , qemu-devel@nongnu.org, "Brandeburg, Jesse" , "Rustad, Mark D" , Carolyn Wyborny , Eric Auger , "Skidmore, Donald C" , zajec5@gmail.com, Alexander Graf , "Vick, Matthew" , intel-wired-lan , Jeff Kirsher , Or Gerlitz , Mitch Williams , nrupal.jani@intel.com, Bjorn Helgaas , a.motakis@virtualopensystems.com, b.reynal@virtualopensystems.com, linux-api@vger.kernel.org, "Nelson, Shannon" , eddie.dong@intel.com, Alex Williamson , "linux-kernel@vger.kernel.org" , "Ronciak, John" , Netdev , Paolo Bonzini On Tue, Nov 24, 2015 at 7:18 PM, Lan Tianyu wrote: > On 2015=E5=B9=B411=E6=9C=8824=E6=97=A5 22:20, Alexander Duyck wrote: >> I'm still not a fan of this approach. I really feel like this is >> something that should be resolved by extending the existing PCI hot-plug >> rather than trying to instrument this per driver. Then you will get the >> goodness for multiple drivers and multiple OSes instead of just one. An >> added advantage to dealing with this in the PCI hot-plug environment >> would be that you could then still do a hot-plug even if the guest >> didn't load a driver for the VF since you would be working with the PCI >> slot instead of the device itself. >> >> - Alex > > Hi Alex: > What's you mentioned seems the bonding driver solution. > Paper "Live Migration with Pass-through Device for Linux VM" describes > it. It does VF hotplug during migration. In order to maintain Network > connection when VF is out, it takes advantage of Linux bonding driver to > switch between VF NIC and emulated NIC. But the side affects, that > requires VM to do additional configure and the performance during > switching two NIC is not good. No, what I am getting at is that you can't go around and modify the configuration space for every possible device out there. This solution won't scale. If you instead moved the logic for notifying the device into a separate mechanism such as making it a part of the hot-plug logic then you only have to write the code once per OS in order to get the hot-plug capability to pause/resume the device. What I am talking about is not full hot-plug, but rather to extend the existing hot-plug in Qemu and the Linux kernel to support a "pause/resume" functionality. The PCI hot-plug specification calls out the option of implementing something like this, but we don't currently have support for it. I just feel doing it through PCI hot-plug messages will scale much better as you could likely make use of the power management suspend/resume calls to take care of most of the needed implementation details. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Tue, 24 Nov 2015 21:30:26 -0800 Message-ID: References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: a.motakis@virtualopensystems.com, Alex Williamson , b.reynal@virtualopensystems.com, Bjorn Helgaas , Carolyn Wyborny , "Skidmore, Donald C" , eddie.dong@intel.com, nrupal.jani@intel.com, Alexander Graf , kvm@vger.kernel.org, Paolo Bonzini , qemu-devel@nongnu.org, "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , "Michael S. Tsirkin" , Eric Auger , intel-wired-lan , Jeff Kirsher , "Brandeburg, Jesse" , "Ronciak, John" , linux-api@vger.kernel.org, "linux-kernel@vger.kernel.org" Return-path: Received: from mail-ig0-f174.google.com ([209.85.213.174]:36621 "EHLO mail-ig0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751291AbbKYFa1 convert rfc822-to-8bit (ORCPT ); Wed, 25 Nov 2015 00:30:27 -0500 In-Reply-To: <56552888.90108@intel.com> Sender: netdev-owner@vger.kernel.org List-ID: On Tue, Nov 24, 2015 at 7:18 PM, Lan Tianyu wrot= e: > On 2015=E5=B9=B411=E6=9C=8824=E6=97=A5 22:20, Alexander Duyck wrote: >> I'm still not a fan of this approach. I really feel like this is >> something that should be resolved by extending the existing PCI hot-= plug >> rather than trying to instrument this per driver. Then you will get= the >> goodness for multiple drivers and multiple OSes instead of just one.= An >> added advantage to dealing with this in the PCI hot-plug environment >> would be that you could then still do a hot-plug even if the guest >> didn't load a driver for the VF since you would be working with the = PCI >> slot instead of the device itself. >> >> - Alex > > Hi Alex: > What's you mentioned seems the bonding driver solution. > Paper "Live Migration with Pass-through Device for Linux VM" describe= s > it. It does VF hotplug during migration. In order to maintain Network > connection when VF is out, it takes advantage of Linux bonding driver= to > switch between VF NIC and emulated NIC. But the side affects, that > requires VM to do additional configure and the performance during > switching two NIC is not good. No, what I am getting at is that you can't go around and modify the configuration space for every possible device out there. This solution won't scale. If you instead moved the logic for notifying the device into a separate mechanism such as making it a part of the hot-plug logic then you only have to write the code once per OS in order to get the hot-plug capability to pause/resume the device. What I am talking about is not full hot-plug, but rather to extend the existing hot-plug in Qemu and the Linux kernel to support a "pause/resume" functionality. The PCI hot-plug specification calls out the option of implementing something like this, but we don't currently have support for it. I just feel doing it through PCI hot-plug messages will scale much better as you could likely make use of the power management suspend/resume calls to take care of most of the needed implementation details. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Tue, 24 Nov 2015 21:30:26 -0800 Message-ID: References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <56552888.90108@intel.com> Sender: netdev-owner@vger.kernel.org To: Lan Tianyu Cc: a.motakis@virtualopensystems.com, Alex Williamson , b.reynal@virtualopensystems.com, Bjorn Helgaas , Carolyn Wyborny , "Skidmore, Donald C" , eddie.dong@intel.com, nrupal.jani@intel.com, Alexander Graf , kvm@vger.kernel.org, Paolo Bonzini , qemu-devel@nongnu.org, "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , "Michael S. Tsirkin" , Eric Auger , intel-wired-lan , Jeff Kirsher , "Brandeburg, Jesse" , "Ronciak, John" , linux-api@vger.kernel.org, "linux-kernel@vger.kernel.org" List-Id: linux-api@vger.kernel.org On Tue, Nov 24, 2015 at 7:18 PM, Lan Tianyu wrot= e: > On 2015=E5=B9=B411=E6=9C=8824=E6=97=A5 22:20, Alexander Duyck wrote: >> I'm still not a fan of this approach. I really feel like this is >> something that should be resolved by extending the existing PCI hot-= plug >> rather than trying to instrument this per driver. Then you will get= the >> goodness for multiple drivers and multiple OSes instead of just one.= An >> added advantage to dealing with this in the PCI hot-plug environment >> would be that you could then still do a hot-plug even if the guest >> didn't load a driver for the VF since you would be working with the = PCI >> slot instead of the device itself. >> >> - Alex > > Hi Alex: > What's you mentioned seems the bonding driver solution. > Paper "Live Migration with Pass-through Device for Linux VM" describe= s > it. It does VF hotplug during migration. In order to maintain Network > connection when VF is out, it takes advantage of Linux bonding driver= to > switch between VF NIC and emulated NIC. But the side affects, that > requires VM to do additional configure and the performance during > switching two NIC is not good. No, what I am getting at is that you can't go around and modify the configuration space for every possible device out there. This solution won't scale. If you instead moved the logic for notifying the device into a separate mechanism such as making it a part of the hot-plug logic then you only have to write the code once per OS in order to get the hot-plug capability to pause/resume the device. What I am talking about is not full hot-plug, but rather to extend the existing hot-plug in Qemu and the Linux kernel to support a "pause/resume" functionality. The PCI hot-plug specification calls out the option of implementing something like this, but we don't currently have support for it. I just feel doing it through PCI hot-plug messages will scale much better as you could likely make use of the power management suspend/resume calls to take care of most of the needed implementation details. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Date: Tue, 24 Nov 2015 21:30:26 -0800 Subject: [Intel-wired-lan] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC In-Reply-To: <56552888.90108@intel.com> References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Tue, Nov 24, 2015 at 7:18 PM, Lan Tianyu wrote: > On 2015?11?24? 22:20, Alexander Duyck wrote: >> I'm still not a fan of this approach. I really feel like this is >> something that should be resolved by extending the existing PCI hot-plug >> rather than trying to instrument this per driver. Then you will get the >> goodness for multiple drivers and multiple OSes instead of just one. An >> added advantage to dealing with this in the PCI hot-plug environment >> would be that you could then still do a hot-plug even if the guest >> didn't load a driver for the VF since you would be working with the PCI >> slot instead of the device itself. >> >> - Alex > > Hi Alex: > What's you mentioned seems the bonding driver solution. > Paper "Live Migration with Pass-through Device for Linux VM" describes > it. It does VF hotplug during migration. In order to maintain Network > connection when VF is out, it takes advantage of Linux bonding driver to > switch between VF NIC and emulated NIC. But the side affects, that > requires VM to do additional configure and the performance during > switching two NIC is not good. No, what I am getting at is that you can't go around and modify the configuration space for every possible device out there. This solution won't scale. If you instead moved the logic for notifying the device into a separate mechanism such as making it a part of the hot-plug logic then you only have to write the code once per OS in order to get the hot-plug capability to pause/resume the device. What I am talking about is not full hot-plug, but rather to extend the existing hot-plug in Qemu and the Linux kernel to support a "pause/resume" functionality. The PCI hot-plug specification calls out the option of implementing something like this, but we don't currently have support for it. I just feel doing it through PCI hot-plug messages will scale much better as you could likely make use of the power management suspend/resume calls to take care of most of the needed implementation details. - Alex