From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753680AbbLIL3S (ORCPT ); Wed, 9 Dec 2015 06:29:18 -0500 Received: from mx1.redhat.com ([209.132.183.28]:43898 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750980AbbLIL3P (ORCPT ); Wed, 9 Dec 2015 06:29:15 -0500 Date: Wed, 9 Dec 2015 13:28:36 +0200 From: "Michael S. Tsirkin" To: "Lan, Tianyu" Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis@virtualopensystems.com" , Alex Williamson , "b.reynal@virtualopensystems.com" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm@vger.kernel.org" , Paolo Bonzini , "qemu-devel@nongnu.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , "Brandeburg, Jesse" , "Ronciak, John" , "linux-api@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Williams, Mitch A" , Netdev , "Nelson, Shannon" , Wei Yang , "zajec5@gmail.com" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Message-ID: <20151209132233-mutt-send-email-mst@redhat.com> References: <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> <56680E33.6040204@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56680E33.6040204@intel.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Dec 09, 2015 at 07:19:15PM +0800, Lan, Tianyu wrote: > On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > >On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > >>Hi Michael & Alexander: > >>Thanks a lot for your comments and suggestions. > > > >It's nice that it's appreciated, but you then go on and ignore > >all that I have written here: > >https://www.mail-archive.com/kvm@vger.kernel.org/msg123826.html > > > > No, I will reply it separately and according your suggestion to snip it into > 3 thread. > > >>We still need to support Windows guest for migration and this is why our > >>patches keep all changes in the driver since it's impossible to change > >>Windows kernel. > > > >This is not a reasonable argument. It makes no sense to duplicate code > >on Linux because you must duplicate code on Windows. Let's assume you > >must do it in the driver on windows because windows has closed source > >drivers. What does it matter? Linux can still do it as part of DMA API > >and have it apply to all drivers. > > > > Sure. Duplicated code should be encapsulated and make it able to reuse > by other drivers. Just like you said the dummy write part. > > I meant the framework should not require to change Windows kernel code > (such as PM core or PCI bus driver)and this will block implementation on > the Windows. I remember reading that it's possible to implement a bus driver on windows if required. But basically I don't see how windows can be relevant to discussing guest driver patches. That discussion probably belongs on the qemu maling list, not on lkml. > I think it's not problem to duplicate code in the Windows drivers. > > >>Following is my idea to do DMA tracking. > >> > >>Inject event to VF driver after memory iterate stage > >>and before stop VCPU and then VF driver marks dirty all > >>using DMA memory. The new allocated pages also need to > >>be marked dirty before stopping VCPU. All dirty memory > >>in this time slot will be migrated until stop-and-copy > >>stage. We also need to make sure to disable VF via clearing the > >>bus master enable bit for VF before migrating these memory. > >> > >>The dma page allocated by VF driver also needs to reserve space > >>to do dummy write. > > > >I suggested ways to do it all in the hypervisor without driver hacks, or > >hide it within DMA API without need to reserve extra space. Both > >approaches seem much cleaner. > > > > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40643) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a6cvv-0001BG-Rk for qemu-devel@nongnu.org; Wed, 09 Dec 2015 06:29:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a6cvs-0001Uq-Ec for qemu-devel@nongnu.org; Wed, 09 Dec 2015 06:29:19 -0500 Received: from mx1.redhat.com ([209.132.183.28]:43507) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a6cvs-0001Uh-88 for qemu-devel@nongnu.org; Wed, 09 Dec 2015 06:29:16 -0500 Date: Wed, 9 Dec 2015 13:28:36 +0200 From: "Michael S. Tsirkin" Message-ID: <20151209132233-mutt-send-email-mst@redhat.com> References: <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> <56680E33.6040204@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56680E33.6040204@intel.com> Subject: Re: [Qemu-devel] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Lan, Tianyu" Cc: Wei Yang , "Tantilov, Emil S" , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" , Alexander Duyck , "Brandeburg, Jesse" , "Rustad, Mark D" , "Wyborny, Carolyn" , Eric Auger , "Skidmore, Donald C" , "zajec5@gmail.com" , Alexander Graf , intel-wired-lan , "Kirsher, Jeffrey T" , Or Gerlitz , "Williams, Mitch A" , "Jani, Nrupal" , Bjorn Helgaas , "a.motakis@virtualopensystems.com" , "b.reynal@virtualopensystems.com" , "linux-api@vger.kernel.org" , "Nelson, Shannon" , "Dong, Eddie" , Alex Williamson , "linux-kernel@vger.kernel.org" , "Ronciak, John" , Netdev , Paolo Bonzini On Wed, Dec 09, 2015 at 07:19:15PM +0800, Lan, Tianyu wrote: > On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > >On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > >>Hi Michael & Alexander: > >>Thanks a lot for your comments and suggestions. > > > >It's nice that it's appreciated, but you then go on and ignore > >all that I have written here: > >https://www.mail-archive.com/kvm@vger.kernel.org/msg123826.html > > > > No, I will reply it separately and according your suggestion to snip it into > 3 thread. > > >>We still need to support Windows guest for migration and this is why our > >>patches keep all changes in the driver since it's impossible to change > >>Windows kernel. > > > >This is not a reasonable argument. It makes no sense to duplicate code > >on Linux because you must duplicate code on Windows. Let's assume you > >must do it in the driver on windows because windows has closed source > >drivers. What does it matter? Linux can still do it as part of DMA API > >and have it apply to all drivers. > > > > Sure. Duplicated code should be encapsulated and make it able to reuse > by other drivers. Just like you said the dummy write part. > > I meant the framework should not require to change Windows kernel code > (such as PM core or PCI bus driver)and this will block implementation on > the Windows. I remember reading that it's possible to implement a bus driver on windows if required. But basically I don't see how windows can be relevant to discussing guest driver patches. That discussion probably belongs on the qemu maling list, not on lkml. > I think it's not problem to duplicate code in the Windows drivers. > > >>Following is my idea to do DMA tracking. > >> > >>Inject event to VF driver after memory iterate stage > >>and before stop VCPU and then VF driver marks dirty all > >>using DMA memory. The new allocated pages also need to > >>be marked dirty before stopping VCPU. All dirty memory > >>in this time slot will be migrated until stop-and-copy > >>stage. We also need to make sure to disable VF via clearing the > >>bus master enable bit for VF before migrating these memory. > >> > >>The dma page allocated by VF driver also needs to reserve space > >>to do dummy write. > > > >I suggested ways to do it all in the hypervisor without driver hacks, or > >hide it within DMA API without need to reserve extra space. Both > >approaches seem much cleaner. > > > > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 9 Dec 2015 13:28:36 +0200 Message-ID: <20151209132233-mutt-send-email-mst@redhat.com> References: <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> <56680E33.6040204@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , " To: "Lan, Tianyu" Return-path: Content-Disposition: inline In-Reply-To: <56680E33.6040204-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: netdev.vger.kernel.org On Wed, Dec 09, 2015 at 07:19:15PM +0800, Lan, Tianyu wrote: > On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > >On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > >>Hi Michael & Alexander: > >>Thanks a lot for your comments and suggestions. > > > >It's nice that it's appreciated, but you then go on and ignore > >all that I have written here: > >https://www.mail-archive.com/kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg123826.html > > > > No, I will reply it separately and according your suggestion to snip it into > 3 thread. > > >>We still need to support Windows guest for migration and this is why our > >>patches keep all changes in the driver since it's impossible to change > >>Windows kernel. > > > >This is not a reasonable argument. It makes no sense to duplicate code > >on Linux because you must duplicate code on Windows. Let's assume you > >must do it in the driver on windows because windows has closed source > >drivers. What does it matter? Linux can still do it as part of DMA API > >and have it apply to all drivers. > > > > Sure. Duplicated code should be encapsulated and make it able to reuse > by other drivers. Just like you said the dummy write part. > > I meant the framework should not require to change Windows kernel code > (such as PM core or PCI bus driver)and this will block implementation on > the Windows. I remember reading that it's possible to implement a bus driver on windows if required. But basically I don't see how windows can be relevant to discussing guest driver patches. That discussion probably belongs on the qemu maling list, not on lkml. > I think it's not problem to duplicate code in the Windows drivers. > > >>Following is my idea to do DMA tracking. > >> > >>Inject event to VF driver after memory iterate stage > >>and before stop VCPU and then VF driver marks dirty all > >>using DMA memory. The new allocated pages also need to > >>be marked dirty before stopping VCPU. All dirty memory > >>in this time slot will be migrated until stop-and-copy > >>stage. We also need to make sure to disable VF via clearing the > >>bus master enable bit for VF before migrating these memory. > >> > >>The dma page allocated by VF driver also needs to reserve space > >>to do dummy write. > > > >I suggested ways to do it all in the hypervisor without driver hacks, or > >hide it within DMA API without need to reserve extra space. Both > >approaches seem much cleaner. > > > > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 9 Dec 2015 13:28:36 +0200 Message-ID: <20151209132233-mutt-send-email-mst@redhat.com> References: <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> <56680E33.6040204@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <56680E33.6040204-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: "Lan, Tianyu" Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" List-Id: linux-api@vger.kernel.org On Wed, Dec 09, 2015 at 07:19:15PM +0800, Lan, Tianyu wrote: > On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > >On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > >>Hi Michael & Alexander: > >>Thanks a lot for your comments and suggestions. > > > >It's nice that it's appreciated, but you then go on and ignore > >all that I have written here: > >https://www.mail-archive.com/kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg123826.html > > > > No, I will reply it separately and according your suggestion to snip it into > 3 thread. > > >>We still need to support Windows guest for migration and this is why our > >>patches keep all changes in the driver since it's impossible to change > >>Windows kernel. > > > >This is not a reasonable argument. It makes no sense to duplicate code > >on Linux because you must duplicate code on Windows. Let's assume you > >must do it in the driver on windows because windows has closed source > >drivers. What does it matter? Linux can still do it as part of DMA API > >and have it apply to all drivers. > > > > Sure. Duplicated code should be encapsulated and make it able to reuse > by other drivers. Just like you said the dummy write part. > > I meant the framework should not require to change Windows kernel code > (such as PM core or PCI bus driver)and this will block implementation on > the Windows. I remember reading that it's possible to implement a bus driver on windows if required. But basically I don't see how windows can be relevant to discussing guest driver patches. That discussion probably belongs on the qemu maling list, not on lkml. > I think it's not problem to duplicate code in the Windows drivers. > > >>Following is my idea to do DMA tracking. > >> > >>Inject event to VF driver after memory iterate stage > >>and before stop VCPU and then VF driver marks dirty all > >>using DMA memory. The new allocated pages also need to > >>be marked dirty before stopping VCPU. All dirty memory > >>in this time slot will be migrated until stop-and-copy > >>stage. We also need to make sure to disable VF via clearing the > >>bus master enable bit for VF before migrating these memory. > >> > >>The dma page allocated by VF driver also needs to reserve space > >>to do dummy write. > > > >I suggested ways to do it all in the hypervisor without driver hacks, or > >hide it within DMA API without need to reserve extra space. Both > >approaches seem much cleaner. > > > > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael S. Tsirkin Date: Wed, 9 Dec 2015 13:28:36 +0200 Subject: [Intel-wired-lan] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC In-Reply-To: <56680E33.6040204@intel.com> References: <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> <56680E33.6040204@intel.com> Message-ID: <20151209132233-mutt-send-email-mst@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Wed, Dec 09, 2015 at 07:19:15PM +0800, Lan, Tianyu wrote: > On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > >On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > >>Hi Michael & Alexander: > >>Thanks a lot for your comments and suggestions. > > > >It's nice that it's appreciated, but you then go on and ignore > >all that I have written here: > >https://www.mail-archive.com/kvm at vger.kernel.org/msg123826.html > > > > No, I will reply it separately and according your suggestion to snip it into > 3 thread. > > >>We still need to support Windows guest for migration and this is why our > >>patches keep all changes in the driver since it's impossible to change > >>Windows kernel. > > > >This is not a reasonable argument. It makes no sense to duplicate code > >on Linux because you must duplicate code on Windows. Let's assume you > >must do it in the driver on windows because windows has closed source > >drivers. What does it matter? Linux can still do it as part of DMA API > >and have it apply to all drivers. > > > > Sure. Duplicated code should be encapsulated and make it able to reuse > by other drivers. Just like you said the dummy write part. > > I meant the framework should not require to change Windows kernel code > (such as PM core or PCI bus driver)and this will block implementation on > the Windows. I remember reading that it's possible to implement a bus driver on windows if required. But basically I don't see how windows can be relevant to discussing guest driver patches. That discussion probably belongs on the qemu maling list, not on lkml. > I think it's not problem to duplicate code in the Windows drivers. > > >>Following is my idea to do DMA tracking. > >> > >>Inject event to VF driver after memory iterate stage > >>and before stop VCPU and then VF driver marks dirty all > >>using DMA memory. The new allocated pages also need to > >>be marked dirty before stopping VCPU. All dirty memory > >>in this time slot will be migrated until stop-and-copy > >>stage. We also need to make sure to disable VF via clearing the > >>bus master enable bit for VF before migrating these memory. > >> > >>The dma page allocated by VF driver also needs to reserve space > >>to do dummy write. > > > >I suggested ways to do it all in the hypervisor without driver hacks, or > >hide it within DMA API without need to reserve extra space. Both > >approaches seem much cleaner. > > > > This sounds reasonable. We may discuss it detail in the separate thread.