From: Jinpu Wang <jinpu.wang@ionos.com>
To: Peter Xu <peterx@redhat.com>
Cc: "Zheng Chuan" <zhengchuan@huawei.com>,
"Gonglei (Arei)" <arei.gonglei@huawei.com>,
"Daniel P. Berrangé" <berrange@redhat.com>,
"Markus Armbruster" <armbru@redhat.com>,
"Michael Galaxy" <mgalaxy@akamai.com>,
"Yu Zhang" <yu.zhang@ionos.com>,
"Zhijian Li (Fujitsu)" <lizhijian@fujitsu.com>,
"Elmar Gerdes" <elmar.gerdes@ionos.com>,
"qemu-devel@nongnu.org" <qemu-devel@nongnu.org>,
"Yuval Shaia" <yuval.shaia.ml@gmail.com>,
"Kevin Wolf" <kwolf@redhat.com>,
"Prasanna Kumar Kalever" <prasanna.kalever@redhat.com>,
"Cornelia Huck" <cohuck@redhat.com>,
"Michael Roth" <michael.roth@amd.com>,
"Prasanna Kumar Kalever" <prasanna4324@gmail.com>,
"integration@gluster.org" <integration@gluster.org>,
"Paolo Bonzini" <pbonzini@redhat.com>,
"qemu-block@nongnu.org" <qemu-block@nongnu.org>,
"devel@lists.libvirt.org" <devel@lists.libvirt.org>,
"Hanna Reitz" <hreitz@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
"Thomas Huth" <thuth@redhat.com>,
"Eric Blake" <eblake@redhat.com>,
"Song Gao" <gaosong@loongson.cn>,
"Marc-André Lureau" <marcandre.lureau@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>,
"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
"Beraldo Leal" <bleal@redhat.com>,
Pannengyuan <pannengyuan@huawei.com>,
Xiexiangyou <xiexiangyou@huawei.com>
Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling
Date: Mon, 13 May 2024 09:30:49 +0200 [thread overview]
Message-ID: <CAMGffEn-DKpMZ4tA71MJYdyemg0Zda15wVAqk81vXtKzx-LfJQ@mail.gmail.com> (raw)
In-Reply-To: <ZjzaIAMgUHV8zdNz@x1n>
Hi Peter, Hi Chuan,
On Thu, May 9, 2024 at 4:14 PM Peter Xu <peterx@redhat.com> wrote:
>
> On Thu, May 09, 2024 at 04:58:34PM +0800, Zheng Chuan via wrote:
> > That's a good news to see the socket abstraction for RDMA!
> > When I was developed the series above, the most pain is the RDMA migration has no QIOChannel abstraction and i need to take a 'fake channel'
> > for it which is awkward in code implementation.
> > So, as far as I know, we can do this by
> > i. the first thing is that we need to evaluate the rsocket is good enough to satisfy our QIOChannel fundamental abstraction
> > ii. if it works right, then we will continue to see if it can give us opportunity to hide the detail of rdma protocol
> > into rsocket by remove most of code in rdma.c and also some hack in migration main process.
> > iii. implement the advanced features like multi-fd and multi-uri for rdma migration.
> >
> > Since I am not familiar with rsocket, I need some times to look at it and do some quick verify with rdma migration based on rsocket.
> > But, yes, I am willing to involved in this refactor work and to see if we can make this migration feature more better:）
>
> Based on what we have now, it looks like we'd better halt the deprecation
> process a bit, so I think we shouldn't need to rush it at least in 9.1
> then, and we'll need to see how it goes on the refactoring.
>
> It'll be perfect if rsocket works, otherwise supporting multifd with little
> overhead / exported APIs would also be a good thing in general with
> whatever approach. And obviously all based on the facts that we can get
> resources from companies to support this feature first.
>
> Note that so far nobody yet compared with rdma v.s. nic perf, so I hope if
> any of us can provide some test results please do so. Many people are
> saying RDMA is better, but I yet didn't see any numbers comparing it with
> modern TCP networks. I don't want to have old impressions floating around
> even if things might have changed.. When we have consolidated results, we
> should share them out and also reflect that in QEMU's migration docs when a
> rdma document page is ready.
I also did a tests with Mellanox ConnectX-6 100 G RoCE nic, the
results are mixed, for less than 3 streams native ethernet is faster,
and when more than 3 streams rsocket performs better.
root@x4-right:~# iperf -c 1.1.1.16 -P 1
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 3] local 1.1.1.15 port 44214 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-10.0000 sec 52.9 GBytes 45.4 Gbits/sec
root@x4-right:~# iperf -c 1.1.1.16 -P 2
[ 3] local 1.1.1.15 port 33118 connected with 1.1.1.16 port 5001
[ 4] local 1.1.1.15 port 33130 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-10.0001 sec 45.0 GBytes 38.7 Gbits/sec
[ 4] 0.0000-10.0000 sec 43.9 GBytes 37.7 Gbits/sec
[SUM] 0.0000-10.0000 sec 88.9 GBytes 76.4 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
0.172/0.189/0.205/0.172 ms (tot/err) = 2/0
root@x4-right:~# iperf -c 1.1.1.16 -P 4
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 5] local 1.1.1.15 port 50748 connected with 1.1.1.16 port 5001
[ 4] local 1.1.1.15 port 50734 connected with 1.1.1.16 port 5001
[ 6] local 1.1.1.15 port 50764 connected with 1.1.1.16 port 5001
[ 3] local 1.1.1.15 port 50730 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0000-10.0000 sec 24.7 GBytes 21.2 Gbits/sec
[ 3] 0.0000-10.0004 sec 23.6 GBytes 20.3 Gbits/sec
[ 4] 0.0000-10.0000 sec 27.8 GBytes 23.9 Gbits/sec
[ 5] 0.0000-10.0000 sec 28.0 GBytes 24.0 Gbits/sec
[SUM] 0.0000-10.0000 sec 104 GBytes 89.4 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
0.104/0.156/0.204/0.124 ms (tot/err) = 4/0
root@x4-right:~# iperf -c 1.1.1.16 -P 8
[ 4] local 1.1.1.15 port 55588 connected with 1.1.1.16 port 5001
[ 5] local 1.1.1.15 port 55600 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 10] local 1.1.1.15 port 55628 connected with 1.1.1.16 port 5001
[ 15] local 1.1.1.15 port 55648 connected with 1.1.1.16 port 5001
[ 7] local 1.1.1.15 port 55620 connected with 1.1.1.16 port 5001
[ 3] local 1.1.1.15 port 55584 connected with 1.1.1.16 port 5001
[ 14] local 1.1.1.15 port 55644 connected with 1.1.1.16 port 5001
[ 6] local 1.1.1.15 port 55610 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0000-10.0015 sec 8.47 GBytes 7.27 Gbits/sec
[ 4] 0.0000-10.0011 sec 8.62 GBytes 7.40 Gbits/sec
[ 7] 0.0000-10.0000 sec 18.1 GBytes 15.5 Gbits/sec
[ 14] 0.0000-10.0000 sec 8.69 GBytes 7.46 Gbits/sec
[ 5] 0.0000-10.0006 sec 18.5 GBytes 15.9 Gbits/sec
[ 10] 0.0000-10.0006 sec 16.1 GBytes 13.9 Gbits/sec
[ 3] 0.0000-10.0000 sec 17.1 GBytes 14.6 Gbits/sec
[ 15] 0.0000-10.0016 sec 8.54 GBytes 7.34 Gbits/sec
[SUM] 0.0000-10.0017 sec 104 GBytes 89.4 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
0.049/0.095/0.213/0.062 ms (tot/err) = 8/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so iperf -c
1.1.1.16 -P 1
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 1.1.1.15 port 45596 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-10.0000 sec 37.8 GBytes 32.5 Gbits/sec
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so iperf -c
1.1.1.16 -P 2
[ 4] local 1.1.1.15 port 46782 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 1.1.1.15 port 43237 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0000-10.0000 sec 37.5 GBytes 32.2 Gbits/sec
[ 3] 0.0000-10.0000 sec 40.7 GBytes 34.9 Gbits/sec
[SUM] 0.0000-10.0000 sec 78.2 GBytes 67.2 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.819/6.579/7.340/7.340 ms (tot/err) = 2/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so iperf -c
1.1.1.16 -P 4
[ 4] local 1.1.1.15 port 60385 connected with 1.1.1.16 port 5001
[ 7] local 1.1.1.15 port 55203 connected with 1.1.1.16 port 5001
[ 6] local 1.1.1.15 port 35084 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 1.1.1.15 port 37253 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0000-10.0000 sec 28.4 GBytes 24.4 Gbits/sec
[ 4] 0.0000-10.0000 sec 28.3 GBytes 24.3 Gbits/sec
[ 7] 0.0000-10.0000 sec 28.4 GBytes 24.4 Gbits/sec
[ 3] 0.0000-10.0001 sec 28.2 GBytes 24.3 Gbits/sec
[SUM] 0.0000-10.0001 sec 113 GBytes 97.3 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.311/7.579/10.019/4.165 ms (tot/err) = 4/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so iperf -c
1.1.1.16 -P 8
[ 8] local 1.1.1.15 port 33684 connected with 1.1.1.16 port 5001
[ 10] local 1.1.1.15 port 40620 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 1.1.1.15 port 56988 connected with 1.1.1.16 port 5001
[ 4] local 1.1.1.15 port 51139 connected with 1.1.1.16 port 5001
[ 12] local 1.1.1.15 port 44712 connected with 1.1.1.16 port 5001
[ 5] local 1.1.1.15 port 50838 connected with 1.1.1.16 port 5001
[ 6] local 1.1.1.15 port 51334 connected with 1.1.1.16 port 5001
[ 9] local 1.1.1.15 port 40611 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-10.0000 sec 13.8 GBytes 11.9 Gbits/sec
[ 5] 0.0000-10.0001 sec 13.9 GBytes 11.9 Gbits/sec
[ 12] 0.0000-10.0001 sec 13.8 GBytes 11.9 Gbits/sec
[ 10] 0.0000-10.0001 sec 13.9 GBytes 11.9 Gbits/sec
[ 9] 0.0000-10.0000 sec 13.8 GBytes 11.9 Gbits/sec
[ 6] 0.0000-10.0000 sec 13.9 GBytes 11.9 Gbits/sec
[ 8] 0.0000-10.0000 sec 13.8 GBytes 11.9 Gbits/sec
[ 4] 0.0000-10.0001 sec 13.8 GBytes 11.9 Gbits/sec
[SUM] 0.0000-10.0001 sec 111 GBytes 95.1 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.973/10.699/15.943/4.251 ms (tot/err) = 8/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so iperf -c
1.1.1.16 -P 1
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 1.1.1.15 port 36960 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-10.0000 sec 41.1 GBytes 35.3 Gbits/sec
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so iperf -c
1.1.1.16 -P 2
[ 3] local 1.1.1.15 port 32799 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 1.1.1.15 port 35912 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0000-10.0000 sec 36.6 GBytes 31.4 Gbits/sec
[ 3] 0.0000-10.0000 sec 36.6 GBytes 31.4 Gbits/sec
[SUM] 0.0000-10.0000 sec 73.2 GBytes 62.9 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.172/5.842/6.512/6.512 ms (tot/err) = 2/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so iperf -c
1.1.1.16 -P 4
[ 4] local 1.1.1.15 port 53311 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 1.1.1.15 port 37243 connected with 1.1.1.16 port 5001
[ 7] local 1.1.1.15 port 60801 connected with 1.1.1.16 port 5001
[ 6] local 1.1.1.15 port 49694 connected with 1.1.1.16 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0000-10.0000 sec 28.2 GBytes 24.2 Gbits/sec
[ 7] 0.0000-10.0000 sec 28.2 GBytes 24.3 Gbits/sec
[ 3] 0.0000-10.0000 sec 28.2 GBytes 24.2 Gbits/sec
[ 4] 0.0000-10.0000 sec 28.2 GBytes 24.2 Gbits/sec
[SUM] 0.0000-10.0000 sec 113 GBytes 96.9 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.570/7.762/10.045/4.265 ms (tot/err) = 4/0
root@x4-right:~#
>
> Chuan, please check the whole thread discussion, it may help to understand
> what we are looking for on rdma migrations [1]. Meanwhile please feel free
> to sync with Jinpu's team and see how to move forward with such a project.
We are happy to work with community to improve rdma migration.
>
> [1] https://lore.kernel.org/qemu-devel/87frwatp7n.fsf@suse.de/
>
> Thanks,
Regards!
>
> --
> Peter Xu
>
next prev parent reply other threads:[~2024-05-13 7:32 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-28 13:02 [PATCH-for-9.1 v2 0/3] rdma: Remove RDMA subsystem and pvrdma device Philippe Mathieu-Daudé
2024-03-28 13:02 ` [PATCH-for-9.1 v2 1/3] hw/rdma: Remove pvrdma device and rdmacm-mux helper Philippe Mathieu-Daudé
2024-03-28 17:51 ` Thomas Huth
2024-03-28 13:02 ` [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling Philippe Mathieu-Daudé
2024-03-28 14:18 ` Fabiano Rosas
2024-03-28 15:01 ` Peter Xu
2024-03-28 15:22 ` Thomas Huth
2024-03-28 19:04 ` Peter Xu
2024-03-29 1:53 ` Zhijian Li (Fujitsu) via
2024-03-29 10:28 ` Philippe Mathieu-Daudé
2024-03-29 19:44 ` Daniel P. Berrangé
2024-04-01 7:55 ` Zhijian Li (Fujitsu) via
2024-04-01 21:26 ` Yu Zhang
2024-04-02 21:23 ` Peter Xu
2024-04-08 14:07 ` Jinpu Wang
2024-04-08 16:18 ` Peter Xu
2024-04-09 7:32 ` Jinpu Wang
2024-04-09 19:46 ` Peter Xu
2024-04-10 2:28 ` Zhijian Li (Fujitsu) via
2024-04-10 13:49 ` Peter Xu
2024-04-11 14:20 ` Peter Xu
2024-04-11 16:36 ` Yu Zhang
2024-04-12 14:04 ` Peter Xu
2024-04-29 13:08 ` Michael Galaxy
2024-04-29 14:56 ` Peter Xu
2024-04-29 20:45 ` Yu Zhang
2024-04-29 20:56 ` Michael Galaxy
2024-04-30 7:15 ` Markus Armbruster
2024-04-30 8:00 ` Daniel P. Berrangé
2024-05-01 15:31 ` Peter Xu
2024-05-01 15:59 ` Daniel P. Berrangé
2024-05-01 16:16 ` Peter Xu
2024-05-02 13:22 ` Michael Galaxy
2024-05-02 13:30 ` Jinpu Wang
2024-05-02 16:19 ` Peter Xu
2024-05-02 17:10 ` Jinpu Wang
2024-05-03 6:40 ` Jinpu Wang
2024-05-03 14:33 ` Peter Xu
2024-05-06 10:08 ` Jinpu Wang
2024-05-06 15:28 ` Peter Xu
2024-05-07 4:52 ` Jinpu Wang
2024-05-08 10:06 ` Daniel P. Berrangé
2024-05-06 2:06 ` Gonglei (Arei) via
2024-05-06 15:18 ` Peter Xu
2024-05-07 1:50 ` Gonglei (Arei) via
2024-05-07 16:28 ` Peter Xu
2024-05-09 8:58 ` Zheng Chuan via
2024-05-09 14:13 ` Peter Xu
2024-05-13 7:30 ` Jinpu Wang [this message]
2024-05-14 15:19 ` Yu Zhang
2024-05-16 17:29 ` Michael Galaxy
2024-05-17 13:01 ` Yu Zhang
2024-05-21 22:15 ` Peter Xu
2024-05-28 9:06 ` Gonglei (Arei) via
2024-05-28 9:11 ` Jinpu Wang
2024-05-28 15:54 ` Peter Xu
2024-05-29 2:43 ` Gonglei (Arei) via
2024-05-29 4:33 ` Jinpu Wang
2024-05-29 6:05 ` Greg Sword
2024-05-29 7:04 ` Jinpu Wang
2024-05-29 8:30 ` Gonglei (Arei) via
2024-05-29 9:17 ` Jinpu Wang
2024-05-29 9:34 ` Gonglei (Arei) via
2024-05-29 9:44 ` Jinpu Wang
2024-05-29 9:47 ` Gonglei (Arei) via
2024-05-29 11:13 ` Haris Iqbal
2024-05-30 18:23 ` Sean Hefty
2024-05-29 16:33 ` Peter Xu
2024-05-13 18:52 ` Michael Galaxy
2024-06-05 0:31 ` Dr. David Alan Gilbert
2024-06-05 14:10 ` Peter Xu
2024-06-05 14:59 ` Peter Xu
2024-06-05 20:48 ` Dr. David Alan Gilbert
2024-06-05 21:18 ` Peter Xu
2024-06-07 8:57 ` Gonglei (Arei) via
2024-04-11 14:42 ` Jinpu Wang
2024-04-09 9:00 ` Markus Armbruster
2024-03-28 13:02 ` [PATCH-for-9.1 v2 3/3] block/gluster: " Philippe Mathieu-Daudé
2024-03-28 17:54 ` Thomas Huth
2024-03-29 9:17 ` [PATCH-for-9.1 v2 0/3] rdma: Remove RDMA subsystem and pvrdma device Michael S. Tsirkin
2024-04-03 9:37 ` Philippe Mathieu-Daudé
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAMGffEn-DKpMZ4tA71MJYdyemg0Zda15wVAqk81vXtKzx-LfJQ@mail.gmail.com \
--to=jinpu.wang@ionos.com \
--cc=alex.bennee@linaro.org \
--cc=arei.gonglei@huawei.com \
--cc=armbru@redhat.com \
--cc=berrange@redhat.com \
--cc=bleal@redhat.com \
--cc=cohuck@redhat.com \
--cc=devel@lists.libvirt.org \
--cc=eblake@redhat.com \
--cc=elmar.gerdes@ionos.com \
--cc=gaosong@loongson.cn \
--cc=hreitz@redhat.com \
--cc=integration@gluster.org \
--cc=kwolf@redhat.com \
--cc=lizhijian@fujitsu.com \
--cc=marcandre.lureau@redhat.com \
--cc=mgalaxy@akamai.com \
--cc=michael.roth@amd.com \
--cc=mst@redhat.com \
--cc=pannengyuan@huawei.com \
--cc=pbonzini@redhat.com \
--cc=peterx@redhat.com \
--cc=prasanna.kalever@redhat.com \
--cc=prasanna4324@gmail.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=thuth@redhat.com \
--cc=wainersm@redhat.com \
--cc=xiexiangyou@huawei.com \
--cc=yu.zhang@ionos.com \
--cc=yuval.shaia.ml@gmail.com \
--cc=zhengchuan@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).