From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752830AbbKYPcI (ORCPT ); Wed, 25 Nov 2015 10:32:08 -0500 Received: from mail-io0-f194.google.com ([209.85.223.194]:35810 "EHLO mail-io0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751194AbbKYPcF convert rfc822-to-8bit (ORCPT ); Wed, 25 Nov 2015 10:32:05 -0500 MIME-Version: 1.0 In-Reply-To: <56556F98.5060507@intel.com> References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> Date: Wed, 25 Nov 2015 07:32:03 -0800 Message-ID: Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC From: Alexander Duyck To: Lan Tianyu Cc: a.motakis@virtualopensystems.com, Alex Williamson , b.reynal@virtualopensystems.com, Bjorn Helgaas , Carolyn Wyborny , "Skidmore, Donald C" , eddie.dong@intel.com, nrupal.jani@intel.com, Alexander Graf , kvm@vger.kernel.org, Paolo Bonzini , qemu-devel@nongnu.org, "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , "Michael S. Tsirkin" , Eric Auger , intel-wired-lan , Jeff Kirsher , "Brandeburg, Jesse" , "Ronciak, John" , linux-api@vger.kernel.org, "linux-kernel@vger.kernel.org" , "Vick, Matthew" , Mitch Williams , Netdev , "Nelson, Shannon" , Wei Yang , zajec5@gmail.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Nov 25, 2015 at 12:21 AM, Lan Tianyu wrote: > On 2015年11月25日 13:30, Alexander Duyck wrote: >> No, what I am getting at is that you can't go around and modify the >> configuration space for every possible device out there. This >> solution won't scale. > > > PCI config space regs are emulation by Qemu and so We can find the free > PCI config space regs for the faked PCI capability. Its position can be > not permanent. Yes, but do you really want to edit every driver on every OS that you plan to support this on. What about things like direct assignment of regular Ethernet ports? What you really need is a solution that will work generically on any existing piece of hardware out there. >> If you instead moved the logic for notifying >> the device into a separate mechanism such as making it a part of the >> hot-plug logic then you only have to write the code once per OS in >> order to get the hot-plug capability to pause/resume the device. What >> I am talking about is not full hot-plug, but rather to extend the >> existing hot-plug in Qemu and the Linux kernel to support a >> "pause/resume" functionality. The PCI hot-plug specification calls >> out the option of implementing something like this, but we don't >> currently have support for it. >> > > Could you elaborate the part of PCI hot-plug specification you mentioned? > > My concern is whether it needs to change PCI spec or not. In the PCI Hot-Plug Specification 1.1, in section 4.1.2 it states: In addition to quiescing add-in card activity, an operating-system vendor may optionally implement a less drastic “pause” capability, in anticipation of the same or a similar add-in card being reinserted. The idea I had was basically if we were to implement something like that in Linux then we could pause/resume the device instead of outright removing it. The pause functionality could make use of the suspend/resume functionality most drivers already have for PCI power management. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39778) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a1c3B-000216-6N for qemu-devel@nongnu.org; Wed, 25 Nov 2015 10:32:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a1c3A-0001tS-6M for qemu-devel@nongnu.org; Wed, 25 Nov 2015 10:32:05 -0500 Received: from mail-io0-x244.google.com ([2607:f8b0:4001:c06::244]:33186) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a1c3A-0001tN-1r for qemu-devel@nongnu.org; Wed, 25 Nov 2015 10:32:04 -0500 Received: by ioef137 with SMTP id f137so5113535ioe.0 for ; Wed, 25 Nov 2015 07:32:03 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <56556F98.5060507@intel.com> References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> Date: Wed, 25 Nov 2015 07:32:03 -0800 Message-ID: From: Alexander Duyck Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Lan Tianyu Cc: Wei Yang , "Tantilov, Emil S" , kvm@vger.kernel.org, "Michael S. Tsirkin" , qemu-devel@nongnu.org, "Brandeburg, Jesse" , "Rustad, Mark D" , Carolyn Wyborny , Eric Auger , "Skidmore, Donald C" , zajec5@gmail.com, Alexander Graf , "Vick, Matthew" , intel-wired-lan , Jeff Kirsher , Or Gerlitz , Mitch Williams , nrupal.jani@intel.com, Bjorn Helgaas , a.motakis@virtualopensystems.com, b.reynal@virtualopensystems.com, linux-api@vger.kernel.org, "Nelson, Shannon" , eddie.dong@intel.com, Alex Williamson , "linux-kernel@vger.kernel.org" , "Ronciak, John" , Netdev , Paolo Bonzini On Wed, Nov 25, 2015 at 12:21 AM, Lan Tianyu wrote: > On 2015=E5=B9=B411=E6=9C=8825=E6=97=A5 13:30, Alexander Duyck wrote: >> No, what I am getting at is that you can't go around and modify the >> configuration space for every possible device out there. This >> solution won't scale. > > > PCI config space regs are emulation by Qemu and so We can find the free > PCI config space regs for the faked PCI capability. Its position can be > not permanent. Yes, but do you really want to edit every driver on every OS that you plan to support this on. What about things like direct assignment of regular Ethernet ports? What you really need is a solution that will work generically on any existing piece of hardware out there. >> If you instead moved the logic for notifying >> the device into a separate mechanism such as making it a part of the >> hot-plug logic then you only have to write the code once per OS in >> order to get the hot-plug capability to pause/resume the device. What >> I am talking about is not full hot-plug, but rather to extend the >> existing hot-plug in Qemu and the Linux kernel to support a >> "pause/resume" functionality. The PCI hot-plug specification calls >> out the option of implementing something like this, but we don't >> currently have support for it. >> > > Could you elaborate the part of PCI hot-plug specification you mentioned? > > My concern is whether it needs to change PCI spec or not. In the PCI Hot-Plug Specification 1.1, in section 4.1.2 it states: In addition to quiescing add-in card activity, an operating-system vendor may optionally implement a less drastic =E2=80=9Cpause=E2=80=9D capa= bility, in anticipation of the same or a similar add-in card being reinserted. The idea I had was basically if we were to implement something like that in Linux then we could pause/resume the device instead of outright removing it. The pause functionality could make use of the suspend/resume functionality most drivers already have for PCI power management. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 25 Nov 2015 07:32:03 -0800 Message-ID: References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org, Alex Williamson , b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org, Bjorn Helgaas , Carolyn Wyborny , "Skidmore, Donald C" , eddie.dong-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, nrupal.jani-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, Alexander Graf , kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Paolo Bonzini , qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , "Michael S. Tsirkin" , Eric Auger , intel-wired-lan , Jeff Kirsher , "Brandeburg, Jesse" , "Ronciak, John" , linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" Return-path: In-Reply-To: <56556F98.5060507-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: netdev.vger.kernel.org On Wed, Nov 25, 2015 at 12:21 AM, Lan Tianyu wro= te: > On 2015=E5=B9=B411=E6=9C=8825=E6=97=A5 13:30, Alexander Duyck wrote: >> No, what I am getting at is that you can't go around and modify the >> configuration space for every possible device out there. This >> solution won't scale. > > > PCI config space regs are emulation by Qemu and so We can find the fr= ee > PCI config space regs for the faked PCI capability. Its position can = be > not permanent. Yes, but do you really want to edit every driver on every OS that you plan to support this on. What about things like direct assignment of regular Ethernet ports? What you really need is a solution that will work generically on any existing piece of hardware out there. >> If you instead moved the logic for notifying >> the device into a separate mechanism such as making it a part of the >> hot-plug logic then you only have to write the code once per OS in >> order to get the hot-plug capability to pause/resume the device. Wh= at >> I am talking about is not full hot-plug, but rather to extend the >> existing hot-plug in Qemu and the Linux kernel to support a >> "pause/resume" functionality. The PCI hot-plug specification calls >> out the option of implementing something like this, but we don't >> currently have support for it. >> > > Could you elaborate the part of PCI hot-plug specification you mentio= ned? > > My concern is whether it needs to change PCI spec or not. In the PCI Hot-Plug Specification 1.1, in section 4.1.2 it states: In addition to quiescing add-in card activity, an operating-system vendor may optionally implement a less drastic =E2=80=9Cpause=E2=80=9D = capability, in anticipation of the same or a similar add-in card being reinserted. The idea I had was basically if we were to implement something like that in Linux then we could pause/resume the device instead of outright removing it. The pause functionality could make use of the suspend/resume functionality most drivers already have for PCI power management. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 25 Nov 2015 07:32:03 -0800 Message-ID: References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <56556F98.5060507-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Lan Tianyu Cc: a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org, Alex Williamson , b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org, Bjorn Helgaas , Carolyn Wyborny , "Skidmore, Donald C" , eddie.dong-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, nrupal.jani-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org, Alexander Graf , kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Paolo Bonzini , qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , "Michael S. Tsirkin" , Eric Auger , intel-wired-lan , Jeff Kirsher , "Brandeburg, Jesse" , "Ronciak, John" , linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" List-Id: linux-api@vger.kernel.org On Wed, Nov 25, 2015 at 12:21 AM, Lan Tianyu wro= te: > On 2015=E5=B9=B411=E6=9C=8825=E6=97=A5 13:30, Alexander Duyck wrote: >> No, what I am getting at is that you can't go around and modify the >> configuration space for every possible device out there. This >> solution won't scale. > > > PCI config space regs are emulation by Qemu and so We can find the fr= ee > PCI config space regs for the faked PCI capability. Its position can = be > not permanent. Yes, but do you really want to edit every driver on every OS that you plan to support this on. What about things like direct assignment of regular Ethernet ports? What you really need is a solution that will work generically on any existing piece of hardware out there. >> If you instead moved the logic for notifying >> the device into a separate mechanism such as making it a part of the >> hot-plug logic then you only have to write the code once per OS in >> order to get the hot-plug capability to pause/resume the device. Wh= at >> I am talking about is not full hot-plug, but rather to extend the >> existing hot-plug in Qemu and the Linux kernel to support a >> "pause/resume" functionality. The PCI hot-plug specification calls >> out the option of implementing something like this, but we don't >> currently have support for it. >> > > Could you elaborate the part of PCI hot-plug specification you mentio= ned? > > My concern is whether it needs to change PCI spec or not. In the PCI Hot-Plug Specification 1.1, in section 4.1.2 it states: In addition to quiescing add-in card activity, an operating-system vendor may optionally implement a less drastic =E2=80=9Cpause=E2=80=9D = capability, in anticipation of the same or a similar add-in card being reinserted. The idea I had was basically if we were to implement something like that in Linux then we could pause/resume the device instead of outright removing it. The pause functionality could make use of the suspend/resume functionality most drivers already have for PCI power management. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Date: Wed, 25 Nov 2015 07:32:03 -0800 Subject: [Intel-wired-lan] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC In-Reply-To: <56556F98.5060507@intel.com> References: <1448372298-28386-1-git-send-email-tianyu.lan@intel.com> <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Wed, Nov 25, 2015 at 12:21 AM, Lan Tianyu wrote: > On 2015?11?25? 13:30, Alexander Duyck wrote: >> No, what I am getting at is that you can't go around and modify the >> configuration space for every possible device out there. This >> solution won't scale. > > > PCI config space regs are emulation by Qemu and so We can find the free > PCI config space regs for the faked PCI capability. Its position can be > not permanent. Yes, but do you really want to edit every driver on every OS that you plan to support this on. What about things like direct assignment of regular Ethernet ports? What you really need is a solution that will work generically on any existing piece of hardware out there. >> If you instead moved the logic for notifying >> the device into a separate mechanism such as making it a part of the >> hot-plug logic then you only have to write the code once per OS in >> order to get the hot-plug capability to pause/resume the device. What >> I am talking about is not full hot-plug, but rather to extend the >> existing hot-plug in Qemu and the Linux kernel to support a >> "pause/resume" functionality. The PCI hot-plug specification calls >> out the option of implementing something like this, but we don't >> currently have support for it. >> > > Could you elaborate the part of PCI hot-plug specification you mentioned? > > My concern is whether it needs to change PCI spec or not. In the PCI Hot-Plug Specification 1.1, in section 4.1.2 it states: In addition to quiescing add-in card activity, an operating-system vendor may optionally implement a less drastic ?pause? capability, in anticipation of the same or a similar add-in card being reinserted. The idea I had was basically if we were to implement something like that in Linux then we could pause/resume the device instead of outright removing it. The pause functionality could make use of the suspend/resume functionality most drivers already have for PCI power management. - Alex