From: Juan Quintela <quintela@redhat.com>
To: "Leonardo Brás" <leobras@redhat.com>
Cc: qemu-devel@nongnu.org,
"Marcel Apfelbaum" <marcel.apfelbaum@gmail.com>,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
"Peter Xu" <peterx@redhat.com>, "Eric Blake" <eblake@redhat.com>,
"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
"Yanan Wang" <wangyanan55@huawei.com>,
"Markus Armbruster" <armbru@redhat.com>,
"Eduardo Habkost" <eduardo@habkost.net>
Subject: Re: [PATCH v7 06/12] multifd: Make flags field thread local
Date: Fri, 19 Aug 2022 12:03:06 +0200 [thread overview]
Message-ID: <87czcw1rfp.fsf@secure.mitica> (raw)
In-Reply-To: <ef7bff6220e3759c7acb5382ae211de1623cdf5e.camel@redhat.com> ("Leonardo Brás"'s message of "Thu, 11 Aug 2022 06:04:11 -0300")
Leonardo Brás <leobras@redhat.com> wrote:
> On Tue, 2022-08-02 at 08:39 +0200, Juan Quintela wrote:
>> Use of flags with respect to locking was incensistant. For the
>> sending side:
>> - it was set to 0 with mutex held on the multifd channel.
>> - MULTIFD_FLAG_SYNC was set with mutex held on the migration thread.
>> - Everything else was done without the mutex held on the multifd channel.
>>
>> On the reception side, it is not used on the migration thread, only on
>> the multifd channels threads.
>>
>> So we move it to the multifd channels thread only variables, and we
>> introduce a new bool sync_needed on the send side to pass that information.
>>
>> Signed-off-by: Juan Quintela <quintela@redhat.com>
>> ---
>> migration/multifd.h | 10 ++++++----
>> migration/multifd.c | 23 +++++++++++++----------
>> 2 files changed, 19 insertions(+), 14 deletions(-)
>>
>> diff --git a/migration/multifd.h b/migration/multifd.h
>> index 36f899c56f..a67cefc0a2 100644
>> --- a/migration/multifd.h
>> +++ b/migration/multifd.h
>> @@ -98,12 +98,12 @@ typedef struct {
>
> Just noticed having no name in 'typedef struct' line makes it harder to
> understand what is going on.
It is common idiom in QEMU. The principal reason is that if you don't
want anyone to use "struct MultiFDSendParams" but MultiFDSendParams, the
best way to achieve that is to do it this way.
>> @@ -172,6 +172,8 @@ typedef struct {
>>
>> /* pointer to the packet */
>> MultiFDPacket_t *packet;
>> + /* multifd flags for each packet */
>> + uint32_t flags;
>> /* size of the next packet that contains pages */
>> uint32_t next_packet_size;
>> /* packets sent through this channel */
>
> So, IIUC, the struct member flags got moved down (same struct) to an area
> described as thread-local, meaning it does not need locking.
>
> Interesting, I haven't noticed this different areas in the same struct.
It has changed in the last two weeks or so in upstream (it has been on
this patchset for several months.)
>
>> diff --git a/migration/multifd.c b/migration/multifd.c
>> index e25b529235..09a40a9135 100644
>> --- a/migration/multifd.c
>> +++ b/migration/multifd.c
>> @@ -602,7 +602,7 @@ int multifd_send_sync_main(QEMUFile *f)
>> }
>>
>> p->packet_num = multifd_send_state->packet_num++;
>> - p->flags |= MULTIFD_FLAG_SYNC;
>> + p->sync_needed = true;
>> p->pending_job++;
>> qemu_mutex_unlock(&p->mutex);
>> qemu_sem_post(&p->sem);
>> @@ -658,7 +658,11 @@ static void *multifd_send_thread(void *opaque)
>>
>> if (p->pending_job) {
>> uint64_t packet_num = p->packet_num;
>> - uint32_t flags = p->flags;
>> + p->flags = 0;
>> + if (p->sync_needed) {
>> + p->flags |= MULTIFD_FLAG_SYNC;
>> + p->sync_needed = false;
>> + }
>
> Any particular reason why doing p->flags = 0, then p->flags |= MULTIFD_FLAG_SYNC
> ?
It is a bitmap field, and if there is anything on the future, we need to
set it. I agree that when there is only one flag, it seems "weird".
> [1] Couldn't it be done without the |= , since it's already being set to zero
> before? (becoming "p->flags = MULTIFD_FLAG_SYNC" )
As said, easier to modify later, and also easier if we want to setup a
flag by default.
I agree that it is a matter of style/taste.
>> p->normal_num = 0;
>>
>> if (use_zero_copy_send) {
>> @@ -680,14 +684,13 @@ static void *multifd_send_thread(void *opaque)
>> }
>> }
>> multifd_send_fill_packet(p);
>> - p->flags = 0;
>> p->num_packets++;
>> p->total_normal_pages += p->normal_num;
>> p->pages->num = 0;
>> p->pages->block = NULL;
>> qemu_mutex_unlock(&p->mutex);
>>
>> - trace_multifd_send(p->id, packet_num, p->normal_num, flags,
>> + trace_multifd_send(p->id, packet_num, p->normal_num, p->flags,
>> p->next_packet_size);
>>
>> if (use_zero_copy_send) {
>> @@ -715,7 +718,7 @@ static void *multifd_send_thread(void *opaque)
>> p->pending_job--;
>> qemu_mutex_unlock(&p->mutex);
>>
>> - if (flags & MULTIFD_FLAG_SYNC) {
>> + if (p->flags & MULTIFD_FLAG_SYNC) {
>> qemu_sem_post(&p->sem_sync);
>> }
>> qemu_sem_post(&multifd_send_state->channels_ready);
>
> IIUC it uses p->sync_needed to keep the sync info, instead of the previous flags
> local var, and thus it can set p->flags = 0 earlier. Seems to not change any
> behavior AFAICS.
The protection of the global flags was being wrong. That is the reason
that I decided to change it to the sync_needed.
The problem was that at some point we were still sending a packet (that
shouldn't have the SYNC flag enabled), but we received a
multifd_main_sync() and it got enabled anyways. The easier way that I
found te fix it was this way.
Problem was difficult to detect, that is the reason that I change it
this way.
>> - if (flags & MULTIFD_FLAG_SYNC) {
>> + if (sync_needed) {
>> qemu_sem_post(&multifd_recv_state->sem_sync);
>> qemu_sem_wait(&p->sem_sync);
>> }
>
> Ok, IIUC this part should have the same behavior as before, but using a bool
> instead of an u32.
I changed it to make sure that we only checked the flags at the
beggining of the function, with the lock taken.
>
> FWIW:
> Reviewed-by: Leonardo Bras <leobras@redhat.com>
Thanks, Juan.
next prev parent reply other threads:[~2022-08-19 10:07 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-02 6:38 [PATCH v7 00/12] Migration: Transmit and detect zero pages in the multifd threads Juan Quintela
2022-08-02 6:38 ` [PATCH v7 01/12] multifd: Create page_size fields into both MultiFD{Recv, Send}Params Juan Quintela
2022-08-11 8:10 ` [PATCH v7 01/12] multifd: Create page_size fields into both MultiFD{Recv,Send}Params Leonardo Brás
2022-08-13 15:41 ` Juan Quintela
2022-08-02 6:38 ` [PATCH v7 02/12] multifd: Create page_count fields into both MultiFD{Recv, Send}Params Juan Quintela
2022-08-11 8:10 ` [PATCH v7 02/12] multifd: Create page_count fields into both MultiFD{Recv,Send}Params Leonardo Brás
2022-08-02 6:38 ` [PATCH v7 03/12] migration: Export ram_transferred_ram() Juan Quintela
2022-08-11 8:11 ` Leonardo Brás
2022-08-13 15:36 ` Juan Quintela
2022-08-02 6:38 ` [PATCH v7 04/12] multifd: Count the number of bytes sent correctly Juan Quintela
2022-08-11 8:11 ` Leonardo Brás
2022-08-19 9:35 ` Juan Quintela
2022-08-02 6:39 ` [PATCH v7 05/12] migration: Make ram_save_target_page() a pointer Juan Quintela
2022-08-11 8:11 ` Leonardo Brás
2022-08-19 9:51 ` Juan Quintela
2022-08-20 7:14 ` Leonardo Bras Soares Passos
2022-08-22 21:35 ` Juan Quintela
2022-08-02 6:39 ` [PATCH v7 06/12] multifd: Make flags field thread local Juan Quintela
2022-08-11 9:04 ` Leonardo Brás
2022-08-19 10:03 ` Juan Quintela [this message]
2022-08-20 7:24 ` Leonardo Bras Soares Passos
2022-08-23 13:00 ` Juan Quintela
2022-08-02 6:39 ` [PATCH v7 07/12] multifd: Prepare to send a packet without the mutex held Juan Quintela
2022-08-11 9:16 ` Leonardo Brás
2022-08-19 11:32 ` Juan Quintela
2022-08-20 7:27 ` Leonardo Bras Soares Passos
2022-08-02 6:39 ` [PATCH v7 08/12] multifd: Add capability to enable/disable zero_page Juan Quintela
2022-08-11 9:29 ` Leonardo Brás
2022-08-19 11:36 ` Juan Quintela
2022-08-02 6:39 ` [PATCH v7 09/12] migration: Export ram_release_page() Juan Quintela
2022-08-11 9:31 ` Leonardo Brás
2022-08-02 6:39 ` [PATCH v7 10/12] multifd: Support for zero pages transmission Juan Quintela
2022-09-02 13:27 ` Leonardo Brás
2022-11-14 12:09 ` Juan Quintela
2022-10-25 9:10 ` chuang xu
2022-11-14 12:10 ` Juan Quintela
2022-08-02 6:39 ` [PATCH v7 11/12] multifd: Zero " Juan Quintela
2022-09-02 13:27 ` Leonardo Brás
2022-11-14 12:20 ` Juan Quintela
2022-11-14 12:27 ` Juan Quintela
2022-08-02 6:39 ` [PATCH v7 12/12] So we use multifd to transmit zero pages Juan Quintela
2022-09-02 13:27 ` Leonardo Brás
2022-11-14 12:30 ` Juan Quintela
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87czcw1rfp.fsf@secure.mitica \
--to=quintela@redhat.com \
--cc=armbru@redhat.com \
--cc=dgilbert@redhat.com \
--cc=eblake@redhat.com \
--cc=eduardo@habkost.net \
--cc=f4bug@amsat.org \
--cc=leobras@redhat.com \
--cc=marcel.apfelbaum@gmail.com \
--cc=peterx@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=wangyanan55@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.