Linux-RDMA Archive mirror
 help / color / mirror / Atom feed

From: Honggang LI <honggangli@163.com>
To: linux-rdma@vger.kernel.org
Subject: Question about extra 40 bytes needed for UD receive buffer
Date: Mon, 15 Apr 2024 10:55:12 +0800	[thread overview]
Message-ID: <ZhyXEOC9SMIxjXP1@fc39> (raw)

hi,

According to volume 1 of IBAT specification of release 1.4,

1) 11.4.1.2 POST RECEIVE REQUEST
Note that for UD QPs, the first 40 bytes of the buffer(s) referred
to by the Scatter/Gather list will contain the GRH of the incoming message.
If no GRH is present, the contents of first 40 bytes of the buffer(s) will
be undefined. The presence of the GRH will be indicated by a bit in the
Work Completion.

2) 11.4.2.1 POLL FOR COMPLETION
GRH Present indicator, for UD RQs only. If this indicator is
set, the first 40 bytes of the buffer(s) referred to by the Scatter/Gather
list will contain the GRH of the incoming message. If it is not set, the
contents of first 40 bytes of the buffer(s) will be undefined. Contents 
of the payload of the message will begin after the first 40 bytes

3) A17.4.5.2 SCATTERING OF THE L3 HEADER IN UD
The first 40 bytes of user posted UD Receive Buffers are reserved for the
L3 header of the incoming packet (as per the InfiniBand Spec Section
11.4.1.2). In RoCEv2, this area is filled up with the IP header. IPv6 header
uses the entire 40 bytes. IPv4 headers use the 20 bytes in the second half
of the reserved 40 bytes area (i.e. offset 20 from the beginning of the
receive buffer). In this case, the content of the first 20 bytes is undefined.



After function `ibv_poll_cq` return 1, the dump of receive buffer of
`ibv_ud_pingpong` shows there is 40 bytes GRH in the head of receive
buffer. The first 20 bytes are zeros and the second 20 bytes is a valid
IP header. At this point, everything works as required by the
specification rules 1), 2) and 3).

However, I'm confused by the dump of `mckey` receive buffer. The flag
`IBV_WC_GRH` is set in `ibv_wc.wc_flags`, but there is no GRH in
the receive buffer. Received data starts from the *first* byte of
receive buffer. As multicast over UD QP only, can someone please explian
why there is no GRH in receive buffer and the data starts from the first
bytes of receive buffer with `IBV_WC_GRH` was set?

Thanks

next             reply	other threads:[~2024-04-15  2:55 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-04-15  2:55 Honggang LI [this message]
2024-04-15 10:10 ` Question about extra 40 bytes needed for UD receive buffer Honggang LI

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ZhyXEOC9SMIxjXP1@fc39 \
    --to=honggangli@163.com \
    --cc=linux-rdma@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).