From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
To: Jason Wang <jasowang@redhat.com>
Cc: "Michael S . Tsirkin" <mst@redhat.com>,
Cornelia Huck <cohuck@redhat.com>,
parav@nvidia.com, virtio-dev@lists.oasis-open.org,
"virtio-comment@lists.oasis-open.org"
<virtio-comment@lists.oasis-open.org>,
"helei.sig11@bytedance.com" <helei.sig11@bytedance.com>,
houp@yusur.tech, zhenwei pi <pizhenwei@bytedance.com>
Subject: Re: [virtio-dev] Re: Re: [virtio-comment] [PROPOSAL] Virtio Over Fabrics(TCP/RDMA)
Date: Wed, 26 Apr 2023 17:29:33 +0800 [thread overview]
Message-ID: <1682501373.3154237-2-xuanzhuo@linux.alibaba.com> (raw)
In-Reply-To: <CACGkMEu9Tzrj+9CUzWOeGhRK-qONHk6cf+2RTTJtxtcKFB8cqw@mail.gmail.com>
On Tue, 25 Apr 2023 14:36:04 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Mon, Apr 24, 2023 at 9:38 PM zhenwei pi <pizhenwei@bytedance.com> wrote:
> >
> >
> >
> > On 4/24/23 11:40, Jason Wang wrote:
> > > On Sun, Apr 23, 2023 at 7:31 PM zhenwei pi <pizhenwei@bytedance.com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> In the past years, virtio supports lots of device specifications by
> > >> PCI/MMIO/CCW. These devices work fine in the virtualization environment,
> > >> and we have a chance to support virtio device family for the
> > >> container/host scenario.
> > >
> > > PCI can work for containers for sure (or does it meet any issue like
> > > scalability?). It's better to describe what problems you met and why
> > > you choose this way to solve it.
> > >
> > > It's better to compare this with
> > >
> > > 1) hiding the fabrics details via DPU
> > > 2) vDPA
> > >
> > Hi,
> >
> > Sorry, I missed this part. "Network defined peripheral devices of virtio
> > family" is the main purpose of this proposal,
>
> This can be achieved by either DPU or vDPA.
I agree this.
So I didn't understand what the meaning of this realization. Although I am also
very excited to this idea, this broaden the possibility of virtio. But, I still
really want to know what the meaning of this idea is, better performance? Or
can achieve some situations that we cannot achieved now.
> I think the advantages is,
> if we standardize this in the spec, it avoids vendor specific
> protocol.
Sorry, I dont got this.
Thanks.
>
> > this allows us to use many
> > types of remote resources which are provided by virtio target.
> >
> > From the point of my view, there are 3 cases:
> > 1, Host/container scenario. For example, host kernel connects a virtio
> > target block service, maps it as a vdx(virtio-blk) device(used by
> > Map-Reduce service which needs a fast/large size disk). The host kernel
> > also connects a virtio target crypto service, maps it as virtio crypto
> > device(used by nginx to accelarate HTTPS). And so on.
> >
> > +----------+ +----------+ +----------+
> > |Map-Reduce| | nginx | ... | processes|
> > +----------+ +----------+ +----------+
> > ------------------------------------------------------------
> > Host | | |
> > Kernel +-------+ +-------+ +-------+
> > | ext4 | | LKCF | | HWRNG |
> > +-------+ +-------+ +-------+
> > | | |
> > +-------+ +-------+ +-------+
> > | vdx | |vCrypto| | vRNG |
> > +-------+ +-------+ +-------+
> > | | |
> > | +--------+ |
> > +---------->|TCP/RDMA|<------------+
> > +--------+
> > |
> > +------+
> > |NIC/IB|
> > +------+
> > | +-------------+
> > +--------------------->|virtio target|
> > +-------------+
> >
> > 2, Typical virtualization environment. The workloads run in a guest, and
> > QEMU handles virtio-pci(or MMIO), and forwards requests to target.
> > +----------+ +----------+ +----------+
> > |Map-Reduce| | nginx | ... | processes|
> > +----------+ +----------+ +----------+
> > ------------------------------------------------------------
> > Guest | | |
> > Kernel +-------+ +-------+ +-------+
> > | ext4 | | LKCF | | HWRNG |
> > +-------+ +-------+ +-------+
> > | | |
> > +-------+ +-------+ +-------+
> > | vdx | |vCrypto| | vRNG |
> > +-------+ +-------+ +-------+
> > | | |
> > PCI --------------------------------------------------------
> > |
> > QEMU +--------------+
> > |virtio backend|
> > +--------------+
> > |
> > +------+
> > |NIC/IB|
> > +------+
> > | +-------------+
> > +--------------------->|virtio target|
> > +-------------+
> >
>
> So it's the job of the Qemu to do the translation from virtqueue to packet here?
>
> > 3, SmartNIC/DPU/vDPA environment. It's possible to convert virtio-pci
> > request to virtio-of request by hardware, and forward request to virtio
> > target directly.
> > +----------+ +----------+ +----------+
> > |Map-Reduce| | nginx | ... | processes|
> > +----------+ +----------+ +----------+
> > ------------------------------------------------------------
> > Host | | |
> > Kernel +-------+ +-------+ +-------+
> > | ext4 | | LKCF | | HWRNG |
> > +-------+ +-------+ +-------+
> > | | |
> > +-------+ +-------+ +-------+
> > | vdx | |vCrypto| | vRNG |
> > +-------+ +-------+ +-------+
> > | | |
> > PCI --------------------------------------------------------
> > |
> > SmartNIC +---------------+
> > |virtio HW queue|
> > +---------------+
> > |
> > +------+
> > |NIC/IB|
> > +------+
> > | +-------------+
> > +--------------------->|virtio target|
> > +-------------+
> >
> > >>
> > >> - Theory
> > >> "Virtio Over Fabrics" aims at "reuse virtio device specifications", and
> > >> provides network defined peripheral devices.
> > >> And this protocol also could be used in virtualization environment,
> > >> typically hypervisor(or vhost-user process) handles request from virtio
> > >> PCI/MMIO/CCW, remaps request and forwards to target by fabrics.
> > >
> > > This requires meditation in the datapath, isn't it?
> > >
> > >>
> > >> - Protocol
> > >> The detail protocol definition see:
> > >> https://github.com/pizhenwei/linux/blob/virtio-of-github/include/uapi/linux/virtio_of.h
> > >
> > > I'd say a RFC patch for virtio spec is more suitable than the codes.
> > >
> >
> > OK. I'll send a RFC patch for virtio spec later if this proposal is
> > acceptable.
>
> Well, I think we need to have an RFC first to know if it is acceptable or not.
>
> >
> > [...]
> >
> > >
> > > A quick glance at the code told me it's a mediation layer that convert
> > > descriptors in the vring to the fabric specific packet. This is the
> > > vDPA way.
> > >
> > > If we agree virtio of fabic is useful, we need invent facilities to
> > > allow building packet directly without bothering the virtqueue (the
> > > API is layout independent anyhow).
> > >
> > > Thanks
> > >
> >
> > This code describes the case 1[Host/container scenario], also proves
> > this case works.
> > Create a virtqueue in the virtio fabric module, also emulate a
> > "virtqueue backend" here, when uplayer kicks vring, the "backend" gets
> > notified and builds packet to TCP/RDMA.
>
> In this case, it won't perform good. Since it still use virtqueue
> which is unnecessary in the datapath for fabric.
>
> Thanks
>
> >
> > [...]
> >
> > --
> > zhenwei pi
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
next prev parent reply other threads:[~2023-04-26 9:33 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1ab0beff-8b18-7a94-1a68-6bf36bcd0394@bytedance.com>
2023-04-24 3:40 ` [virtio-dev] Re: [virtio-comment] [PROPOSAL] Virtio Over Fabrics(TCP/RDMA) Jason Wang
[not found] ` <8f65c9aa-c867-0929-151c-21bbe25a0693@bytedance.com>
2023-04-25 5:03 ` Parav Pandit
2023-04-25 6:31 ` Jason Wang
2023-04-25 13:27 ` [virtio-dev] " Parav Pandit
2023-04-27 8:20 ` [virtio-dev] Re: " zhenwei pi
2023-04-27 20:31 ` [virtio-dev] " Parav Pandit
2023-04-25 6:36 ` [virtio-dev] " Jason Wang
2023-04-26 9:29 ` Xuan Zhuo [this message]
2023-04-25 13:55 ` [virtio-dev] " Stefan Hajnoczi
2023-04-26 1:08 ` [virtio-dev] " zhenwei pi
2023-04-25 14:09 ` [virtio-dev] " Stefan Hajnoczi
2023-04-26 3:03 ` Jason Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1682501373.3154237-2-xuanzhuo@linux.alibaba.com \
--to=xuanzhuo@linux.alibaba.com \
--cc=cohuck@redhat.com \
--cc=helei.sig11@bytedance.com \
--cc=houp@yusur.tech \
--cc=jasowang@redhat.com \
--cc=mst@redhat.com \
--cc=parav@nvidia.com \
--cc=pizhenwei@bytedance.com \
--cc=virtio-comment@lists.oasis-open.org \
--cc=virtio-dev@lists.oasis-open.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).