From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754576AbbLIKiB (ORCPT ); Wed, 9 Dec 2015 05:38:01 -0500 Received: from mx1.redhat.com ([209.132.183.28]:34069 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754066AbbLIKh5 (ORCPT ); Wed, 9 Dec 2015 05:37:57 -0500 Date: Wed, 9 Dec 2015 12:37:46 +0200 From: "Michael S. Tsirkin" To: "Lan, Tianyu" Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis@virtualopensystems.com" , Alex Williamson , "b.reynal@virtualopensystems.com" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm@vger.kernel.org" , Paolo Bonzini , "qemu-devel@nongnu.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , "Brandeburg, Jesse" , "Ronciak, John" , "linux-api@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Williams, Mitch A" , Netdev , "Nelson, Shannon" , Wei Yang , "zajec5@gmail.com" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Message-ID: <20151209122831-mutt-send-email-mst@redhat.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5661C000.8070201@intel.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > Hi Michael & Alexander: > Thanks a lot for your comments and suggestions. It's nice that it's appreciated, but you then go on and ignore all that I have written here: https://www.mail-archive.com/kvm@vger.kernel.org/msg123826.html > We still need to support Windows guest for migration and this is why our > patches keep all changes in the driver since it's impossible to change > Windows kernel. This is not a reasonable argument. It makes no sense to duplicate code on Linux because you must duplicate code on Windows. Let's assume you must do it in the driver on windows because windows has closed source drivers. What does it matter? Linux can still do it as part of DMA API and have it apply to all drivers. > Following is my idea to do DMA tracking. > > Inject event to VF driver after memory iterate stage > and before stop VCPU and then VF driver marks dirty all > using DMA memory. The new allocated pages also need to > be marked dirty before stopping VCPU. All dirty memory > in this time slot will be migrated until stop-and-copy > stage. We also need to make sure to disable VF via clearing the > bus master enable bit for VF before migrating these memory. > > The dma page allocated by VF driver also needs to reserve space > to do dummy write. I suggested ways to do it all in the hypervisor without driver hacks, or hide it within DMA API without need to reserve extra space. Both approaches seem much cleaner. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57973) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a6c8I-0007JG-2W for qemu-devel@nongnu.org; Wed, 09 Dec 2015 05:38:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a6c8E-0006HE-Pc for qemu-devel@nongnu.org; Wed, 09 Dec 2015 05:38:02 -0500 Received: from mx1.redhat.com ([209.132.183.28]:55063) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a6c8E-0006H0-Il for qemu-devel@nongnu.org; Wed, 09 Dec 2015 05:37:58 -0500 Date: Wed, 9 Dec 2015 12:37:46 +0200 From: "Michael S. Tsirkin" Message-ID: <20151209122831-mutt-send-email-mst@redhat.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5661C000.8070201@intel.com> Subject: Re: [Qemu-devel] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Lan, Tianyu" Cc: Wei Yang , "Tantilov, Emil S" , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" , Alexander Duyck , "Brandeburg, Jesse" , "Rustad, Mark D" , "Wyborny, Carolyn" , Eric Auger , "Skidmore, Donald C" , "zajec5@gmail.com" , Alexander Graf , intel-wired-lan , "Kirsher, Jeffrey T" , Or Gerlitz , "Williams, Mitch A" , "Jani, Nrupal" , Bjorn Helgaas , "a.motakis@virtualopensystems.com" , "b.reynal@virtualopensystems.com" , "linux-api@vger.kernel.org" , "Nelson, Shannon" , "Dong, Eddie" , Alex Williamson , "linux-kernel@vger.kernel.org" , "Ronciak, John" , Netdev , Paolo Bonzini On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > Hi Michael & Alexander: > Thanks a lot for your comments and suggestions. It's nice that it's appreciated, but you then go on and ignore all that I have written here: https://www.mail-archive.com/kvm@vger.kernel.org/msg123826.html > We still need to support Windows guest for migration and this is why our > patches keep all changes in the driver since it's impossible to change > Windows kernel. This is not a reasonable argument. It makes no sense to duplicate code on Linux because you must duplicate code on Windows. Let's assume you must do it in the driver on windows because windows has closed source drivers. What does it matter? Linux can still do it as part of DMA API and have it apply to all drivers. > Following is my idea to do DMA tracking. > > Inject event to VF driver after memory iterate stage > and before stop VCPU and then VF driver marks dirty all > using DMA memory. The new allocated pages also need to > be marked dirty before stopping VCPU. All dirty memory > in this time slot will be migrated until stop-and-copy > stage. We also need to make sure to disable VF via clearing the > bus master enable bit for VF before migrating these memory. > > The dma page allocated by VF driver also needs to reserve space > to do dummy write. I suggested ways to do it all in the hypervisor without driver hacks, or hide it within DMA API without need to reserve extra space. Both approaches seem much cleaner. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 9 Dec 2015 12:37:46 +0200 Message-ID: <20151209122831-mutt-send-email-mst@redhat.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , " To: "Lan, Tianyu" Return-path: Content-Disposition: inline In-Reply-To: <5661C000.8070201-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: netdev.vger.kernel.org On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > Hi Michael & Alexander: > Thanks a lot for your comments and suggestions. It's nice that it's appreciated, but you then go on and ignore all that I have written here: https://www.mail-archive.com/kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg123826.html > We still need to support Windows guest for migration and this is why our > patches keep all changes in the driver since it's impossible to change > Windows kernel. This is not a reasonable argument. It makes no sense to duplicate code on Linux because you must duplicate code on Windows. Let's assume you must do it in the driver on windows because windows has closed source drivers. What does it matter? Linux can still do it as part of DMA API and have it apply to all drivers. > Following is my idea to do DMA tracking. > > Inject event to VF driver after memory iterate stage > and before stop VCPU and then VF driver marks dirty all > using DMA memory. The new allocated pages also need to > be marked dirty before stopping VCPU. All dirty memory > in this time slot will be migrated until stop-and-copy > stage. We also need to make sure to disable VF via clearing the > bus master enable bit for VF before migrating these memory. > > The dma page allocated by VF driver also needs to reserve space > to do dummy write. I suggested ways to do it all in the hypervisor without driver hacks, or hide it within DMA API without need to reserve extra space. Both approaches seem much cleaner. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 9 Dec 2015 12:37:46 +0200 Message-ID: <20151209122831-mutt-send-email-mst@redhat.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <5661C000.8070201-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: "Lan, Tianyu" Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" List-Id: linux-api@vger.kernel.org On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > Hi Michael & Alexander: > Thanks a lot for your comments and suggestions. It's nice that it's appreciated, but you then go on and ignore all that I have written here: https://www.mail-archive.com/kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg123826.html > We still need to support Windows guest for migration and this is why our > patches keep all changes in the driver since it's impossible to change > Windows kernel. This is not a reasonable argument. It makes no sense to duplicate code on Linux because you must duplicate code on Windows. Let's assume you must do it in the driver on windows because windows has closed source drivers. What does it matter? Linux can still do it as part of DMA API and have it apply to all drivers. > Following is my idea to do DMA tracking. > > Inject event to VF driver after memory iterate stage > and before stop VCPU and then VF driver marks dirty all > using DMA memory. The new allocated pages also need to > be marked dirty before stopping VCPU. All dirty memory > in this time slot will be migrated until stop-and-copy > stage. We also need to make sure to disable VF via clearing the > bus master enable bit for VF before migrating these memory. > > The dma page allocated by VF driver also needs to reserve space > to do dummy write. I suggested ways to do it all in the hypervisor without driver hacks, or hide it within DMA API without need to reserve extra space. Both approaches seem much cleaner. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael S. Tsirkin Date: Wed, 9 Dec 2015 12:37:46 +0200 Subject: [Intel-wired-lan] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC In-Reply-To: <5661C000.8070201@intel.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> Message-ID: <20151209122831-mutt-send-email-mst@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: > Hi Michael & Alexander: > Thanks a lot for your comments and suggestions. It's nice that it's appreciated, but you then go on and ignore all that I have written here: https://www.mail-archive.com/kvm at vger.kernel.org/msg123826.html > We still need to support Windows guest for migration and this is why our > patches keep all changes in the driver since it's impossible to change > Windows kernel. This is not a reasonable argument. It makes no sense to duplicate code on Linux because you must duplicate code on Windows. Let's assume you must do it in the driver on windows because windows has closed source drivers. What does it matter? Linux can still do it as part of DMA API and have it apply to all drivers. > Following is my idea to do DMA tracking. > > Inject event to VF driver after memory iterate stage > and before stop VCPU and then VF driver marks dirty all > using DMA memory. The new allocated pages also need to > be marked dirty before stopping VCPU. All dirty memory > in this time slot will be migrated until stop-and-copy > stage. We also need to make sure to disable VF via clearing the > bus master enable bit for VF before migrating these memory. > > The dma page allocated by VF driver also needs to reserve space > to do dummy write. I suggested ways to do it all in the hypervisor without driver hacks, or hide it within DMA API without need to reserve extra space. Both approaches seem much cleaner. -- MST