From: Maxim Levitsky <mlevitsk@redhat.com>
To: Stefan Hajnoczi <stefanha@gmail.com>
Cc: linux-nvme@lists.infradead.org, Fam Zheng <fam@euphon.net>,
Keith Busch <keith.busch@intel.com>,
Sagi Grimberg <sagi@grimberg.me>,
kvm@vger.kernel.org, Wolfram Sang <wsa@the-dreams.de>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Liang Cunming <cunming.liang@intel.com>,
Nicolas Ferre <nicolas.ferre@microchip.com>,
linux-kernel@vger.kernel.org,
Kirti Wankhede <kwankhede@nvidia.com>,
"David S . Miller" <davem@davemloft.net>,
Jens Axboe <axboe@fb.com>,
Alex Williamson <alex.williamson@redhat.com>,
John Ferlan <jferlan@redhat.com>,
Mauro Carvalho Chehab <mchehab+samsung@kernel.org>,
Paolo Bonzini <pbonzini@redhat.com>,
Liu Changpeng <changpeng.liu@intel.com>,
"Paul E . McKenney" <paulmck@linux.ibm.com>,
Amnon Ilan <ailan@redhat.com>, Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH 0/9] RFC: NVME VFIO mediated device [BENCHMARKS]
Date: Tue, 26 Mar 2019 11:50:47 +0200 [thread overview]
Message-ID: <ad88c611e6ce0df3d7e0137d7eaa0eff4c862a89.camel@redhat.com> (raw)
In-Reply-To: <20190326093858.GI21018@stefanha-x1.localdomain>
On Tue, 2019-03-26 at 09:38 +0000, Stefan Hajnoczi wrote:
> On Mon, Mar 25, 2019 at 08:52:32PM +0200, Maxim Levitsky wrote:
> > Hi
> >
> > This is first round of benchmarks.
> >
> > The system is Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
> >
> > The system has 2 numa nodes, but only cpus and memory from node 0 were used to
> > avoid noise from numa.
> >
> > The SSD is Intel® Optane™ SSD 900P Series, 280 GB version
> >
> >
> > https://ark.intel.com/content/www/us/en/ark/products/123628/intel-optane-ssd-900p-series-280gb-1-2-height-pcie-x4-20nm-3d-xpoint.html
> >
> >
> > ** Latency benchmark with no interrupts at all **
> >
> > spdk was complited with fio plugin in the host and in the guest.
> > spdk was first run in the host
> > then vm was started with one of spdk,pci passthrough, mdev and inside the
> > vm spdk was run with fio plugin.
> >
> > spdk was taken from my branch on gitlab, and fio was complied from source for
> > 3.4 branch as needed by the spdk fio plugin.
> >
> > The following spdk command line was used:
> >
> > $WORK/fio/fio \
> > --name=job --runtime=40 --ramp_time=0 --time_based \
> > --filename="trtype=PCIe traddr=$DEVICE_FOR_FIO ns=1" --ioengine=spdk \
> > --direct=1 --rw=randread --bs=4K --cpus_allowed=0 \
> > --iodepth=1 --thread
> >
> > The average values for slat (submission latency), clat (completion latency) and
> > its sum (slat+clat) were noted.
> >
> > The results:
> >
> > spdk fio host:
> > 573 Mib/s - slat 112.00ns, clat 6.400us, lat 6.52ms
> > 573 Mib/s - slat 111.50ns, clat 6.406us, lat 6.52ms
> >
> >
> > pci passthough host/
> > spdk fio guest
> > 571 Mib/s - slat 124.56ns, clat 6.422us lat 6.55ms
> > 571 Mib/s - slat 122.86ns, clat 6.410us lat 6.53ms
> > 570 Mib/s - slat 124.95ns, clat 6.425us lat 6.55ms
> >
> > spdk host/
> > spdk fio guest:
> > 535 Mib/s - slat 125.00ns, clat 6.895us lat 7.02ms
> > 534 Mib/s - slat 125.36ns, clat 6.896us lat 7.02ms
> > 534 Mib/s - slat 125.82ns, clat 6.892us lat 7.02ms
> >
> > mdev host/
> > spdk fio guest:
> > 534 Mib/s - slat 128.04ns, clat 6.902us lat 7.03ms
> > 535 Mib/s - slat 126.97ns, clat 6.900us lat 7.03ms
> > 535 Mib/s - slat 127.00ns, clat 6.898us lat 7.03ms
> >
> >
> > As you see, native latency is 6.52ms, pci passthrough barely adds any latency,
> > while both mdev/spdk added about (7.03/2 - 6.52) - 0.51ms/0.50ms of latency.
>
> Milliseconds is surprising. The SSD's spec says 10us read/write
> latency. Did you mean microseconds?
Yea, this is typo - all of this is microseconds.
>
> >
> > In addtion to that I added few 'rdtsc' into my mdev driver to strategically
> > capture the cycle count it takes it to do 3 things:
> >
> > 1. translate a just received command (till it is copied to the hardware
> > submission queue)
> >
> > 2. receive a completion (divided by the number of completion received in one
> > round of polling)
> >
> > 3. deliver an interupt to the guest (call to eventfd_signal)
> >
> > This is not the whole latency as there is also a latency between the point the
> > submission entry is written and till it is visible on the polling cpu, plus
> > latency till polling cpu gets to the code which reads the submission entry,
> > and of course latency of interrupt delivery, but the above measurements mostly
> > capture the latency I can control.
> >
> > The results are:
> >
> > commands translated : avg cycles: 459.844 avg time(usec): 0.135
> > commands completed : avg cycles: 354.61 avg time(usec): 0.104
> > interrupts sent : avg cycles: 590.227 avg time(usec): 0.174
> >
> > avg time total: 0.413 usec
> >
> > All measurmenets done in the host kernel. the time calculated using tsc_khz
> > kernel variable.
> >
> > The biggest take from this is that both spdk and my driver are very fast and
> > overhead is just a thousand of cpu cycles give it or take.
>
> Nice!
>
> Stefan
Best regards,
Maxim Levitsky
WARNING: multiple messages have this Message-ID (diff)
From: mlevitsk@redhat.com (Maxim Levitsky)
Subject: [PATCH 0/9] RFC: NVME VFIO mediated device [BENCHMARKS]
Date: Tue, 26 Mar 2019 11:50:47 +0200 [thread overview]
Message-ID: <ad88c611e6ce0df3d7e0137d7eaa0eff4c862a89.camel@redhat.com> (raw)
In-Reply-To: <20190326093858.GI21018@stefanha-x1.localdomain>
On Tue, 2019-03-26@09:38 +0000, Stefan Hajnoczi wrote:
> On Mon, Mar 25, 2019@08:52:32PM +0200, Maxim Levitsky wrote:
> > Hi
> >
> > This is first round of benchmarks.
> >
> > The system is Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
> >
> > The system has 2 numa nodes, but only cpus and memory from node 0 were used to
> > avoid noise from numa.
> >
> > The SSD is Intel? Optane? SSD 900P Series, 280 GB version
> >
> >
> > https://ark.intel.com/content/www/us/en/ark/products/123628/intel-optane-ssd-900p-series-280gb-1-2-height-pcie-x4-20nm-3d-xpoint.html
> >
> >
> > ** Latency benchmark with no interrupts at all **
> >
> > spdk was complited with fio plugin in the host and in the guest.
> > spdk was first run in the host
> > then vm was started with one of spdk,pci passthrough, mdev and inside the
> > vm spdk was run with fio plugin.
> >
> > spdk was taken from my branch on gitlab, and fio was complied from source for
> > 3.4 branch as needed by the spdk fio plugin.
> >
> > The following spdk command line was used:
> >
> > $WORK/fio/fio \
> > --name=job --runtime=40 --ramp_time=0 --time_based \
> > --filename="trtype=PCIe traddr=$DEVICE_FOR_FIO ns=1" --ioengine=spdk \
> > --direct=1 --rw=randread --bs=4K --cpus_allowed=0 \
> > --iodepth=1 --thread
> >
> > The average values for slat (submission latency), clat (completion latency) and
> > its sum (slat+clat) were noted.
> >
> > The results:
> >
> > spdk fio host:
> > 573 Mib/s - slat 112.00ns, clat 6.400us, lat 6.52ms
> > 573 Mib/s - slat 111.50ns, clat 6.406us, lat 6.52ms
> >
> >
> > pci passthough host/
> > spdk fio guest
> > 571 Mib/s - slat 124.56ns, clat 6.422us lat 6.55ms
> > 571 Mib/s - slat 122.86ns, clat 6.410us lat 6.53ms
> > 570 Mib/s - slat 124.95ns, clat 6.425us lat 6.55ms
> >
> > spdk host/
> > spdk fio guest:
> > 535 Mib/s - slat 125.00ns, clat 6.895us lat 7.02ms
> > 534 Mib/s - slat 125.36ns, clat 6.896us lat 7.02ms
> > 534 Mib/s - slat 125.82ns, clat 6.892us lat 7.02ms
> >
> > mdev host/
> > spdk fio guest:
> > 534 Mib/s - slat 128.04ns, clat 6.902us lat 7.03ms
> > 535 Mib/s - slat 126.97ns, clat 6.900us lat 7.03ms
> > 535 Mib/s - slat 127.00ns, clat 6.898us lat 7.03ms
> >
> >
> > As you see, native latency is 6.52ms, pci passthrough barely adds any latency,
> > while both mdev/spdk added about (7.03/2 - 6.52) - 0.51ms/0.50ms of latency.
>
> Milliseconds is surprising. The SSD's spec says 10us read/write
> latency. Did you mean microseconds?
Yea, this is typo - all of this is microseconds.
>
> >
> > In addtion to that I added few 'rdtsc' into my mdev driver to strategically
> > capture the cycle count it takes it to do 3 things:
> >
> > 1. translate a just received command (till it is copied to the hardware
> > submission queue)
> >
> > 2. receive a completion (divided by the number of completion received in one
> > round of polling)
> >
> > 3. deliver an interupt to the guest (call to eventfd_signal)
> >
> > This is not the whole latency as there is also a latency between the point the
> > submission entry is written and till it is visible on the polling cpu, plus
> > latency till polling cpu gets to the code which reads the submission entry,
> > and of course latency of interrupt delivery, but the above measurements mostly
> > capture the latency I can control.
> >
> > The results are:
> >
> > commands translated : avg cycles: 459.844 avg time(usec): 0.135
> > commands completed : avg cycles: 354.61 avg time(usec): 0.104
> > interrupts sent : avg cycles: 590.227 avg time(usec): 0.174
> >
> > avg time total: 0.413 usec
> >
> > All measurmenets done in the host kernel. the time calculated using tsc_khz
> > kernel variable.
> >
> > The biggest take from this is that both spdk and my driver are very fast and
> > overhead is just a thousand of cpu cycles give it or take.
>
> Nice!
>
> Stefan
Best regards,
Maxim Levitsky
WARNING: multiple messages have this Message-ID (diff)
From: Maxim Levitsky <mlevitsk@redhat.com>
To: Stefan Hajnoczi <stefanha@gmail.com>
Cc: linux-nvme@lists.infradead.org, Fam Zheng <fam@euphon.net>,
Keith Busch <keith.busch@intel.com>,
Sagi Grimberg <sagi@grimberg.me>,
kvm@vger.kernel.org, Wolfram Sang <wsa@the-dreams.de>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Liang Cunming <cunming.liang@intel.com>,
Nicolas Ferre <nicolas.ferre@microchip.com>,
linux-kernel@vger.kernel.org,
Kirti Wankhede <kwankhede@nvidia.com>,
"David S . Miller" <davem@davemloft.net>,
Jens Axboe <axboe@fb.com>,
Alex Williamson <alex.williamson@redhat.com>,
John Ferlan <jferlan@redhat.com>,
Mauro Carvalho Chehab <mchehab+samsung@kernel.org>,
Paolo Bonzini <pbonzini@redhat.com>,
Liu Changpeng <changpeng.liu@intel.com>,
"Paul E . McKenney" <paulmck@linux.ibm.com>,
Amnon Ilan <ailan@redhat.com>,
Christoph Hellw
Subject: Re: [PATCH 0/9] RFC: NVME VFIO mediated device [BENCHMARKS]
Date: Tue, 26 Mar 2019 11:50:47 +0200 [thread overview]
Message-ID: <ad88c611e6ce0df3d7e0137d7eaa0eff4c862a89.camel@redhat.com> (raw)
In-Reply-To: <20190326093858.GI21018@stefanha-x1.localdomain>
On Tue, 2019-03-26 at 09:38 +0000, Stefan Hajnoczi wrote:
> On Mon, Mar 25, 2019 at 08:52:32PM +0200, Maxim Levitsky wrote:
> > Hi
> >
> > This is first round of benchmarks.
> >
> > The system is Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
> >
> > The system has 2 numa nodes, but only cpus and memory from node 0 were used to
> > avoid noise from numa.
> >
> > The SSD is Intel® Optane™ SSD 900P Series, 280 GB version
> >
> >
> > https://ark.intel.com/content/www/us/en/ark/products/123628/intel-optane-ssd-900p-series-280gb-1-2-height-pcie-x4-20nm-3d-xpoint.html
> >
> >
> > ** Latency benchmark with no interrupts at all **
> >
> > spdk was complited with fio plugin in the host and in the guest.
> > spdk was first run in the host
> > then vm was started with one of spdk,pci passthrough, mdev and inside the
> > vm spdk was run with fio plugin.
> >
> > spdk was taken from my branch on gitlab, and fio was complied from source for
> > 3.4 branch as needed by the spdk fio plugin.
> >
> > The following spdk command line was used:
> >
> > $WORK/fio/fio \
> > --name=job --runtime=40 --ramp_time=0 --time_based \
> > --filename="trtype=PCIe traddr=$DEVICE_FOR_FIO ns=1" --ioengine=spdk \
> > --direct=1 --rw=randread --bs=4K --cpus_allowed=0 \
> > --iodepth=1 --thread
> >
> > The average values for slat (submission latency), clat (completion latency) and
> > its sum (slat+clat) were noted.
> >
> > The results:
> >
> > spdk fio host:
> > 573 Mib/s - slat 112.00ns, clat 6.400us, lat 6.52ms
> > 573 Mib/s - slat 111.50ns, clat 6.406us, lat 6.52ms
> >
> >
> > pci passthough host/
> > spdk fio guest
> > 571 Mib/s - slat 124.56ns, clat 6.422us lat 6.55ms
> > 571 Mib/s - slat 122.86ns, clat 6.410us lat 6.53ms
> > 570 Mib/s - slat 124.95ns, clat 6.425us lat 6.55ms
> >
> > spdk host/
> > spdk fio guest:
> > 535 Mib/s - slat 125.00ns, clat 6.895us lat 7.02ms
> > 534 Mib/s - slat 125.36ns, clat 6.896us lat 7.02ms
> > 534 Mib/s - slat 125.82ns, clat 6.892us lat 7.02ms
> >
> > mdev host/
> > spdk fio guest:
> > 534 Mib/s - slat 128.04ns, clat 6.902us lat 7.03ms
> > 535 Mib/s - slat 126.97ns, clat 6.900us lat 7.03ms
> > 535 Mib/s - slat 127.00ns, clat 6.898us lat 7.03ms
> >
> >
> > As you see, native latency is 6.52ms, pci passthrough barely adds any latency,
> > while both mdev/spdk added about (7.03/2 - 6.52) - 0.51ms/0.50ms of latency.
>
> Milliseconds is surprising. The SSD's spec says 10us read/write
> latency. Did you mean microseconds?
Yea, this is typo - all of this is microseconds.
>
> >
> > In addtion to that I added few 'rdtsc' into my mdev driver to strategically
> > capture the cycle count it takes it to do 3 things:
> >
> > 1. translate a just received command (till it is copied to the hardware
> > submission queue)
> >
> > 2. receive a completion (divided by the number of completion received in one
> > round of polling)
> >
> > 3. deliver an interupt to the guest (call to eventfd_signal)
> >
> > This is not the whole latency as there is also a latency between the point the
> > submission entry is written and till it is visible on the polling cpu, plus
> > latency till polling cpu gets to the code which reads the submission entry,
> > and of course latency of interrupt delivery, but the above measurements mostly
> > capture the latency I can control.
> >
> > The results are:
> >
> > commands translated : avg cycles: 459.844 avg time(usec): 0.135
> > commands completed : avg cycles: 354.61 avg time(usec): 0.104
> > interrupts sent : avg cycles: 590.227 avg time(usec): 0.174
> >
> > avg time total: 0.413 usec
> >
> > All measurmenets done in the host kernel. the time calculated using tsc_khz
> > kernel variable.
> >
> > The biggest take from this is that both spdk and my driver are very fast and
> > overhead is just a thousand of cpu cycles give it or take.
>
> Nice!
>
> Stefan
Best regards,
Maxim Levitsky
next prev parent reply other threads:[~2019-03-26 9:50 UTC|newest]
Thread overview: 117+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-19 14:41 No subject Maxim Levitsky
2019-03-19 14:41 ` (unknown) Maxim Levitsky
2019-03-19 14:41 ` [PATCH 1/9] vfio/mdev: add .request callback Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` [PATCH 2/9] nvme/core: add some more values from the spec Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` [PATCH 3/9] nvme/core: add NVME_CTRL_SUSPENDED controller state Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` [PATCH 4/9] nvme/pci: use the NVME_CTRL_SUSPENDED state Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-20 2:54 ` Fam Zheng
2019-03-20 2:54 ` Fam Zheng
2019-03-19 14:41 ` [PATCH 5/9] nvme/pci: add known admin effects to augument admin effects log page Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` [PATCH 6/9] nvme/pci: init shadow doorbell after each reset Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` [PATCH 7/9] nvme/core: add mdev interfaces Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-20 11:46 ` Stefan Hajnoczi
2019-03-20 11:46 ` Stefan Hajnoczi
2019-03-20 11:46 ` Stefan Hajnoczi
2019-03-20 12:50 ` Maxim Levitsky
2019-03-20 12:50 ` Maxim Levitsky
2019-03-20 12:50 ` Maxim Levitsky
2019-03-19 14:41 ` [PATCH 8/9] nvme/core: add nvme-mdev core driver Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` [PATCH 9/9] nvme/pci: implement the mdev external queue allocation interface Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:41 ` Maxim Levitsky
2019-03-19 14:58 ` [PATCH 0/9] RFC: NVME VFIO mediated device Maxim Levitsky
2019-03-19 14:58 ` Maxim Levitsky
2019-03-25 18:52 ` [PATCH 0/9] RFC: NVME VFIO mediated device [BENCHMARKS] Maxim Levitsky
2019-03-25 18:52 ` Maxim Levitsky
2019-03-26 9:38 ` Stefan Hajnoczi
2019-03-26 9:38 ` Stefan Hajnoczi
2019-03-26 9:38 ` Stefan Hajnoczi
2019-03-26 9:50 ` Maxim Levitsky [this message]
2019-03-26 9:50 ` Maxim Levitsky
2019-03-26 9:50 ` Maxim Levitsky
2019-03-19 15:22 ` your mail Keith Busch
2019-03-19 15:22 ` Keith Busch
2019-03-19 15:22 ` Keith Busch
2019-03-19 23:49 ` Chaitanya Kulkarni
2019-03-19 23:49 ` Chaitanya Kulkarni
2019-03-19 23:49 ` Chaitanya Kulkarni
2019-03-20 16:44 ` Maxim Levitsky
2019-03-20 16:44 ` Maxim Levitsky
2019-03-20 16:44 ` Maxim Levitsky
2019-03-20 16:30 ` Maxim Levitsky
2019-03-20 16:30 ` Maxim Levitsky
2019-03-20 16:30 ` Maxim Levitsky
2019-03-20 17:03 ` Keith Busch
2019-03-20 17:03 ` Keith Busch
2019-03-20 17:03 ` Keith Busch
2019-03-20 17:33 ` Maxim Levitsky
2019-03-20 17:33 ` Maxim Levitsky
2019-03-20 17:33 ` Maxim Levitsky
2019-04-08 10:04 ` Maxim Levitsky
2019-04-08 10:04 ` Maxim Levitsky
2019-03-20 11:03 ` Felipe Franciosi
2019-03-20 11:03 ` Re: Felipe Franciosi
2019-03-20 11:03 ` No subject Felipe Franciosi
2019-03-20 19:08 ` Maxim Levitsky
2019-03-20 19:08 ` Re: Maxim Levitsky
2019-03-20 19:08 ` No subject Maxim Levitsky
2019-03-21 16:12 ` Stefan Hajnoczi
2019-03-21 16:12 ` Re: Stefan Hajnoczi
2019-03-21 16:12 ` No subject Stefan Hajnoczi
2019-03-21 16:21 ` Keith Busch
2019-03-21 16:21 ` Re: Keith Busch
2019-03-21 16:21 ` No subject Keith Busch
2019-03-21 16:41 ` Felipe Franciosi
2019-03-21 16:41 ` Re: Felipe Franciosi
2019-03-21 16:41 ` No subject Felipe Franciosi
2019-03-21 17:04 ` Maxim Levitsky
2019-03-21 17:04 ` Re: Maxim Levitsky
2019-03-21 17:04 ` No subject Maxim Levitsky
2019-03-22 7:54 ` Felipe Franciosi
2019-03-22 7:54 ` Re: Felipe Franciosi
2019-03-22 7:54 ` No subject Felipe Franciosi
2019-03-22 10:32 ` Maxim Levitsky
2019-03-22 10:32 ` Re: Maxim Levitsky
2019-03-22 10:32 ` No subject Maxim Levitsky
2019-03-22 15:30 ` Keith Busch
2019-03-22 15:30 ` Re: Keith Busch
2019-03-22 15:30 ` No subject Keith Busch
2019-03-25 15:44 ` Felipe Franciosi
2019-03-25 15:44 ` Re: Felipe Franciosi
2019-03-25 15:44 ` No subject Felipe Franciosi
2019-03-20 15:08 ` [PATCH 0/9] RFC: NVME VFIO mediated device Bart Van Assche
2019-03-20 15:08 ` Bart Van Assche
2019-03-20 16:48 ` Maxim Levitsky
2019-03-20 16:48 ` Maxim Levitsky
2019-03-20 15:28 ` Bart Van Assche
2019-03-20 15:28 ` Bart Van Assche
2019-03-20 16:42 ` Maxim Levitsky
2019-03-20 16:42 ` Maxim Levitsky
2019-03-20 17:03 ` Alex Williamson
2019-03-20 17:03 ` Alex Williamson
2019-03-20 17:03 ` Alex Williamson
2019-03-21 16:13 ` your mail Stefan Hajnoczi
2019-03-21 16:13 ` Stefan Hajnoczi
2019-03-21 16:13 ` Stefan Hajnoczi
2019-03-21 17:07 ` Maxim Levitsky
2019-03-21 17:07 ` Maxim Levitsky
2019-03-21 17:07 ` Maxim Levitsky
2019-03-25 16:46 ` Stefan Hajnoczi
2019-03-25 16:46 ` Stefan Hajnoczi
2019-03-25 16:46 ` Stefan Hajnoczi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ad88c611e6ce0df3d7e0137d7eaa0eff4c862a89.camel@redhat.com \
--to=mlevitsk@redhat.com \
--cc=ailan@redhat.com \
--cc=alex.williamson@redhat.com \
--cc=axboe@fb.com \
--cc=changpeng.liu@intel.com \
--cc=cunming.liang@intel.com \
--cc=davem@davemloft.net \
--cc=fam@euphon.net \
--cc=gregkh@linuxfoundation.org \
--cc=hch@lst.de \
--cc=jferlan@redhat.com \
--cc=keith.busch@intel.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=mchehab+samsung@kernel.org \
--cc=nicolas.ferre@microchip.com \
--cc=paulmck@linux.ibm.com \
--cc=pbonzini@redhat.com \
--cc=sagi@grimberg.me \
--cc=stefanha@gmail.com \
--cc=wsa@the-dreams.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.