From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754327AbbLAP3B (ORCPT ); Tue, 1 Dec 2015 10:29:01 -0500 Received: from mx1.redhat.com ([209.132.183.28]:53618 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750790AbbLAP26 (ORCPT ); Tue, 1 Dec 2015 10:28:58 -0500 Date: Tue, 1 Dec 2015 17:28:48 +0200 From: "Michael S. Tsirkin" To: "Lan, Tianyu" Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis@virtualopensystems.com" , Alex Williamson , "b.reynal@virtualopensystems.com" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm@vger.kernel.org" , Paolo Bonzini , "qemu-devel@nongnu.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , "Brandeburg, Jesse" , "Ronciak, John" , "linux-api@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Williams, Mitch A" , Netdev , "Nelson, Shannon" , Wei Yang , "zajec5@gmail.com" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Message-ID: <20151201171140-mutt-send-email-mst@redhat.com> References: <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <565DB6FF.1050602@intel.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Dec 01, 2015 at 11:04:31PM +0800, Lan, Tianyu wrote: > > > On 12/1/2015 12:07 AM, Alexander Duyck wrote: > >They can only be corrected if the underlying assumptions are correct > >and they aren't. Your solution would have never worked correctly. > >The problem is you assume you can keep the device running when you are > >migrating and you simply cannot. At some point you will always have > >to stop the device in order to complete the migration, and you cannot > >stop it before you have stopped your page tracking mechanism. So > >unless the platform has an IOMMU that is somehow taking part in the > >dirty page tracking you will not be able to stop the guest and then > >the device, it will have to be the device and then the guest. > > > >>>Doing suspend and resume() may help to do migration easily but some > >>>devices requires low service down time. Especially network and I got > >>>that some cloud company promised less than 500ms network service downtime. > >Honestly focusing on the downtime is getting the cart ahead of the > >horse. First you need to be able to do this without corrupting system > >memory and regardless of the state of the device. You haven't even > >gotten to that state yet. Last I knew the device had to be up in > >order for your migration to even work. > > I think the issue is that the content of rx package delivered to stack maybe > changed during migration because the piece of memory won't be migrated to > new machine. This may confuse applications or stack. Current dummy write > solution can ensure the content of package won't change after doing dummy > write while the content maybe not received data if migration happens before > that point. We can recheck the content via checksum or crc in the protocol > after dummy write to ensure the content is what VF received. I think stack > has already done such checks and the package will be abandoned if failed to > pass through the check. Most people nowdays rely on hardware checksums so I don't think this can fly. > Another way is to tell all memory driver are using to Qemu and let Qemu to > migrate these memory after stopping VCPU and the device. This seems safe but > implementation maybe complex. Not really 100% safe. See below. I think hiding these details behind dma_* API does have some appeal. In any case, it gives us a good terminology as it covers what most drivers do. There are several components to this: - dma_map_* needs to prevent page from being migrated while device is running. For example, expose some kind of bitmap from guest to host, set bit there while page is mapped. What happens if we stop the guest and some bits are still set? See dma_alloc_coherent below for some ideas. - dma_unmap_* needs to mark page as dirty This can be done by writing into a page. - dma_sync_* needs to mark page as dirty This is trickier as we can not change the data. One solution is using atomics. For example: int x = ACCESS_ONCE(*p); cmpxchg(p, x, x); Seems to do a write without changing page contents. - dma_alloc_coherent memory (e.g. device rings) must be migrated after device stopped modifying it. Just stopping the VCPU is not enough: you must make sure device is not changing it. Or maybe the device has some kind of ring flush operation, if there was a reasonably portable way to do this (e.g. a flush capability could maybe be added to SRIOV) then hypervisor could do this. With existing devices, either do it after device reset, or disable memory access in the IOMMU. Maybe both. In case you need to resume on source, you really need to follow the same path as on destination, preferably detecting device reset and restoring the device state. A similar approach could work for dma_map_ above. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33605) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a3mrW-0006tf-Jf for qemu-devel@nongnu.org; Tue, 01 Dec 2015 10:29:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a3mrT-0006NQ-4Q for qemu-devel@nongnu.org; Tue, 01 Dec 2015 10:29:02 -0500 Received: from mx1.redhat.com ([209.132.183.28]:36462) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a3mrS-0006NA-Q3 for qemu-devel@nongnu.org; Tue, 01 Dec 2015 10:28:59 -0500 Date: Tue, 1 Dec 2015 17:28:48 +0200 From: "Michael S. Tsirkin" Message-ID: <20151201171140-mutt-send-email-mst@redhat.com> References: <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <565DB6FF.1050602@intel.com> Subject: Re: [Qemu-devel] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Lan, Tianyu" Cc: Wei Yang , "Tantilov, Emil S" , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" , Alexander Duyck , "Brandeburg, Jesse" , "Rustad, Mark D" , "Wyborny, Carolyn" , Eric Auger , "Skidmore, Donald C" , "zajec5@gmail.com" , Alexander Graf , intel-wired-lan , "Kirsher, Jeffrey T" , Or Gerlitz , "Williams, Mitch A" , "Jani, Nrupal" , Bjorn Helgaas , "a.motakis@virtualopensystems.com" , "b.reynal@virtualopensystems.com" , "linux-api@vger.kernel.org" , "Nelson, Shannon" , "Dong, Eddie" , Alex Williamson , "linux-kernel@vger.kernel.org" , "Ronciak, John" , Netdev , Paolo Bonzini On Tue, Dec 01, 2015 at 11:04:31PM +0800, Lan, Tianyu wrote: > > > On 12/1/2015 12:07 AM, Alexander Duyck wrote: > >They can only be corrected if the underlying assumptions are correct > >and they aren't. Your solution would have never worked correctly. > >The problem is you assume you can keep the device running when you are > >migrating and you simply cannot. At some point you will always have > >to stop the device in order to complete the migration, and you cannot > >stop it before you have stopped your page tracking mechanism. So > >unless the platform has an IOMMU that is somehow taking part in the > >dirty page tracking you will not be able to stop the guest and then > >the device, it will have to be the device and then the guest. > > > >>>Doing suspend and resume() may help to do migration easily but some > >>>devices requires low service down time. Especially network and I got > >>>that some cloud company promised less than 500ms network service downtime. > >Honestly focusing on the downtime is getting the cart ahead of the > >horse. First you need to be able to do this without corrupting system > >memory and regardless of the state of the device. You haven't even > >gotten to that state yet. Last I knew the device had to be up in > >order for your migration to even work. > > I think the issue is that the content of rx package delivered to stack maybe > changed during migration because the piece of memory won't be migrated to > new machine. This may confuse applications or stack. Current dummy write > solution can ensure the content of package won't change after doing dummy > write while the content maybe not received data if migration happens before > that point. We can recheck the content via checksum or crc in the protocol > after dummy write to ensure the content is what VF received. I think stack > has already done such checks and the package will be abandoned if failed to > pass through the check. Most people nowdays rely on hardware checksums so I don't think this can fly. > Another way is to tell all memory driver are using to Qemu and let Qemu to > migrate these memory after stopping VCPU and the device. This seems safe but > implementation maybe complex. Not really 100% safe. See below. I think hiding these details behind dma_* API does have some appeal. In any case, it gives us a good terminology as it covers what most drivers do. There are several components to this: - dma_map_* needs to prevent page from being migrated while device is running. For example, expose some kind of bitmap from guest to host, set bit there while page is mapped. What happens if we stop the guest and some bits are still set? See dma_alloc_coherent below for some ideas. - dma_unmap_* needs to mark page as dirty This can be done by writing into a page. - dma_sync_* needs to mark page as dirty This is trickier as we can not change the data. One solution is using atomics. For example: int x = ACCESS_ONCE(*p); cmpxchg(p, x, x); Seems to do a write without changing page contents. - dma_alloc_coherent memory (e.g. device rings) must be migrated after device stopped modifying it. Just stopping the VCPU is not enough: you must make sure device is not changing it. Or maybe the device has some kind of ring flush operation, if there was a reasonably portable way to do this (e.g. a flush capability could maybe be added to SRIOV) then hypervisor could do this. With existing devices, either do it after device reset, or disable memory access in the IOMMU. Maybe both. In case you need to resume on source, you really need to follow the same path as on destination, preferably detecting device reset and restoring the device state. A similar approach could work for dma_map_ above. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Tue, 1 Dec 2015 17:28:48 +0200 Message-ID: <20151201171140-mutt-send-email-mst@redhat.com> References: <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , " To: "Lan, Tianyu" Return-path: Content-Disposition: inline In-Reply-To: <565DB6FF.1050602-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: netdev.vger.kernel.org On Tue, Dec 01, 2015 at 11:04:31PM +0800, Lan, Tianyu wrote: > > > On 12/1/2015 12:07 AM, Alexander Duyck wrote: > >They can only be corrected if the underlying assumptions are correct > >and they aren't. Your solution would have never worked correctly. > >The problem is you assume you can keep the device running when you are > >migrating and you simply cannot. At some point you will always have > >to stop the device in order to complete the migration, and you cannot > >stop it before you have stopped your page tracking mechanism. So > >unless the platform has an IOMMU that is somehow taking part in the > >dirty page tracking you will not be able to stop the guest and then > >the device, it will have to be the device and then the guest. > > > >>>Doing suspend and resume() may help to do migration easily but some > >>>devices requires low service down time. Especially network and I got > >>>that some cloud company promised less than 500ms network service downtime. > >Honestly focusing on the downtime is getting the cart ahead of the > >horse. First you need to be able to do this without corrupting system > >memory and regardless of the state of the device. You haven't even > >gotten to that state yet. Last I knew the device had to be up in > >order for your migration to even work. > > I think the issue is that the content of rx package delivered to stack maybe > changed during migration because the piece of memory won't be migrated to > new machine. This may confuse applications or stack. Current dummy write > solution can ensure the content of package won't change after doing dummy > write while the content maybe not received data if migration happens before > that point. We can recheck the content via checksum or crc in the protocol > after dummy write to ensure the content is what VF received. I think stack > has already done such checks and the package will be abandoned if failed to > pass through the check. Most people nowdays rely on hardware checksums so I don't think this can fly. > Another way is to tell all memory driver are using to Qemu and let Qemu to > migrate these memory after stopping VCPU and the device. This seems safe but > implementation maybe complex. Not really 100% safe. See below. I think hiding these details behind dma_* API does have some appeal. In any case, it gives us a good terminology as it covers what most drivers do. There are several components to this: - dma_map_* needs to prevent page from being migrated while device is running. For example, expose some kind of bitmap from guest to host, set bit there while page is mapped. What happens if we stop the guest and some bits are still set? See dma_alloc_coherent below for some ideas. - dma_unmap_* needs to mark page as dirty This can be done by writing into a page. - dma_sync_* needs to mark page as dirty This is trickier as we can not change the data. One solution is using atomics. For example: int x = ACCESS_ONCE(*p); cmpxchg(p, x, x); Seems to do a write without changing page contents. - dma_alloc_coherent memory (e.g. device rings) must be migrated after device stopped modifying it. Just stopping the VCPU is not enough: you must make sure device is not changing it. Or maybe the device has some kind of ring flush operation, if there was a reasonably portable way to do this (e.g. a flush capability could maybe be added to SRIOV) then hypervisor could do this. With existing devices, either do it after device reset, or disable memory access in the IOMMU. Maybe both. In case you need to resume on source, you really need to follow the same path as on destination, preferably detecting device reset and restoring the device state. A similar approach could work for dma_map_ above. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Tue, 1 Dec 2015 17:28:48 +0200 Message-ID: <20151201171140-mutt-send-email-mst@redhat.com> References: <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <565DB6FF.1050602-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: "Lan, Tianyu" Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" List-Id: linux-api@vger.kernel.org On Tue, Dec 01, 2015 at 11:04:31PM +0800, Lan, Tianyu wrote: > > > On 12/1/2015 12:07 AM, Alexander Duyck wrote: > >They can only be corrected if the underlying assumptions are correct > >and they aren't. Your solution would have never worked correctly. > >The problem is you assume you can keep the device running when you are > >migrating and you simply cannot. At some point you will always have > >to stop the device in order to complete the migration, and you cannot > >stop it before you have stopped your page tracking mechanism. So > >unless the platform has an IOMMU that is somehow taking part in the > >dirty page tracking you will not be able to stop the guest and then > >the device, it will have to be the device and then the guest. > > > >>>Doing suspend and resume() may help to do migration easily but some > >>>devices requires low service down time. Especially network and I got > >>>that some cloud company promised less than 500ms network service downtime. > >Honestly focusing on the downtime is getting the cart ahead of the > >horse. First you need to be able to do this without corrupting system > >memory and regardless of the state of the device. You haven't even > >gotten to that state yet. Last I knew the device had to be up in > >order for your migration to even work. > > I think the issue is that the content of rx package delivered to stack maybe > changed during migration because the piece of memory won't be migrated to > new machine. This may confuse applications or stack. Current dummy write > solution can ensure the content of package won't change after doing dummy > write while the content maybe not received data if migration happens before > that point. We can recheck the content via checksum or crc in the protocol > after dummy write to ensure the content is what VF received. I think stack > has already done such checks and the package will be abandoned if failed to > pass through the check. Most people nowdays rely on hardware checksums so I don't think this can fly. > Another way is to tell all memory driver are using to Qemu and let Qemu to > migrate these memory after stopping VCPU and the device. This seems safe but > implementation maybe complex. Not really 100% safe. See below. I think hiding these details behind dma_* API does have some appeal. In any case, it gives us a good terminology as it covers what most drivers do. There are several components to this: - dma_map_* needs to prevent page from being migrated while device is running. For example, expose some kind of bitmap from guest to host, set bit there while page is mapped. What happens if we stop the guest and some bits are still set? See dma_alloc_coherent below for some ideas. - dma_unmap_* needs to mark page as dirty This can be done by writing into a page. - dma_sync_* needs to mark page as dirty This is trickier as we can not change the data. One solution is using atomics. For example: int x = ACCESS_ONCE(*p); cmpxchg(p, x, x); Seems to do a write without changing page contents. - dma_alloc_coherent memory (e.g. device rings) must be migrated after device stopped modifying it. Just stopping the VCPU is not enough: you must make sure device is not changing it. Or maybe the device has some kind of ring flush operation, if there was a reasonably portable way to do this (e.g. a flush capability could maybe be added to SRIOV) then hypervisor could do this. With existing devices, either do it after device reset, or disable memory access in the IOMMU. Maybe both. In case you need to resume on source, you really need to follow the same path as on destination, preferably detecting device reset and restoring the device state. A similar approach could work for dma_map_ above. -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael S. Tsirkin Date: Tue, 1 Dec 2015 17:28:48 +0200 Subject: [Intel-wired-lan] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC In-Reply-To: <565DB6FF.1050602@intel.com> References: <5654722D.4010409@gmail.com> <56552888.90108@intel.com> <56556F98.5060507@intel.com> <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> Message-ID: <20151201171140-mutt-send-email-mst@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Tue, Dec 01, 2015 at 11:04:31PM +0800, Lan, Tianyu wrote: > > > On 12/1/2015 12:07 AM, Alexander Duyck wrote: > >They can only be corrected if the underlying assumptions are correct > >and they aren't. Your solution would have never worked correctly. > >The problem is you assume you can keep the device running when you are > >migrating and you simply cannot. At some point you will always have > >to stop the device in order to complete the migration, and you cannot > >stop it before you have stopped your page tracking mechanism. So > >unless the platform has an IOMMU that is somehow taking part in the > >dirty page tracking you will not be able to stop the guest and then > >the device, it will have to be the device and then the guest. > > > >>>Doing suspend and resume() may help to do migration easily but some > >>>devices requires low service down time. Especially network and I got > >>>that some cloud company promised less than 500ms network service downtime. > >Honestly focusing on the downtime is getting the cart ahead of the > >horse. First you need to be able to do this without corrupting system > >memory and regardless of the state of the device. You haven't even > >gotten to that state yet. Last I knew the device had to be up in > >order for your migration to even work. > > I think the issue is that the content of rx package delivered to stack maybe > changed during migration because the piece of memory won't be migrated to > new machine. This may confuse applications or stack. Current dummy write > solution can ensure the content of package won't change after doing dummy > write while the content maybe not received data if migration happens before > that point. We can recheck the content via checksum or crc in the protocol > after dummy write to ensure the content is what VF received. I think stack > has already done such checks and the package will be abandoned if failed to > pass through the check. Most people nowdays rely on hardware checksums so I don't think this can fly. > Another way is to tell all memory driver are using to Qemu and let Qemu to > migrate these memory after stopping VCPU and the device. This seems safe but > implementation maybe complex. Not really 100% safe. See below. I think hiding these details behind dma_* API does have some appeal. In any case, it gives us a good terminology as it covers what most drivers do. There are several components to this: - dma_map_* needs to prevent page from being migrated while device is running. For example, expose some kind of bitmap from guest to host, set bit there while page is mapped. What happens if we stop the guest and some bits are still set? See dma_alloc_coherent below for some ideas. - dma_unmap_* needs to mark page as dirty This can be done by writing into a page. - dma_sync_* needs to mark page as dirty This is trickier as we can not change the data. One solution is using atomics. For example: int x = ACCESS_ONCE(*p); cmpxchg(p, x, x); Seems to do a write without changing page contents. - dma_alloc_coherent memory (e.g. device rings) must be migrated after device stopped modifying it. Just stopping the VCPU is not enough: you must make sure device is not changing it. Or maybe the device has some kind of ring flush operation, if there was a reasonably portable way to do this (e.g. a flush capability could maybe be added to SRIOV) then hypervisor could do this. With existing devices, either do it after device reset, or disable memory access in the IOMMU. Maybe both. In case you need to resume on source, you really need to follow the same path as on destination, preferably detecting device reset and restoring the device state. A similar approach could work for dma_map_ above. -- MST