From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
To: Ian Campbell <ian.campbell@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>,
Stefano Stabellini <stefano.stabellini@eu.citrix.com>,
George Dunlap <george.dunlap@eu.citrix.com>,
Andrew Cooper <andrew.cooper3@citrix.com>,
Ian Jackson <Ian.Jackson@eu.citrix.com>,
Don Slutz <dslutz@verizon.com>,
"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: QEMU bumping memory bug analysis
Date: Wed, 17 Jun 2015 14:35:25 +0100 [thread overview]
Message-ID: <alpine.DEB.2.02.1506161750470.21829@kaball.uk.xensource.com> (raw)
In-Reply-To: <1433853905.7108.550.camel@citrix.com>
On Tue, 9 Jun 2015, Ian Campbell wrote:
> On Tue, 2015-06-09 at 11:14 +0100, Stefano Stabellini wrote:
> > I don't think that the current solution is inherently racy. If we are
> > really interested in an honest evaluation of the current solution in
> > terms of races, I would be happy to do so.
>
> Consider a domain with 1M of RAM (==target and maxmem for the sake of
> argument) and two simultaneous calls to libxl_set_memory_target, both
> with relative=0 and enforce=1, one with target=3 and the other with
> target=5.
>
> target=3 call target=5 call
>
> get ptr.max_memkb, gets 1
>
> get ptr.max_memkb, gets 1
> start transaction
> do xenstore stuff, seeing target=1, setting target=3
> memorykb = ptr.max_memkb - current_target_memkb + new_target_memkb;
> memorykb = 1 - 1 + 3
> xc_setmaxmem(3)
> transaction commit (success)
>
> Now target=maxmem=3
>
> start transaction
> do xenstore stuff, seeing target=3, setting target=5
> memorykb = ptr.max_memkb - current_target_memkb + new_target_memkb;
> memorykb = 1 - 3 + 5
> xc_setmaxmem(3)
> transaction commit (success)
>
> Now target=5, maxmem=3.
Yes, I see the issue. Pretty nasty!
> The obvious fix of moving the get ptr.max_memkb inside the transaction
> fails in a different way in the case where the first transaction commit
> fails and is retried after the second one, I think.
The underlying problem is that xc_domain_setmaxmem only takes absolute
values. Retrieving the old maxmem value and setting the new one cannot
be done atomically, so they need to be protected by a lock or a
transaction. You are right that moving get ptr.max_memkb inside the
transaction would only fix the problem if we also properly rolled back
the old maxmem value in case of failures. But I don't think it is
actually possible because it could race against other
libxl_set_memory_target calls. Am I wrong? This would also be a
problem before 0c029c4da21.
> Prior to 0c029c4da21 "libxl_set_memory_target: retain the same maxmem
> offset on top of the current target" this issue didn't exist because
> memorykb was just memorykb = new_target_memkb + videoram.
It is true that reverting 0c029c4da21 would improve the situation.
However I think that you found a problem here that goes beyond
0c029c4da21 :-(
> BTW, I noticed some other (unrelated) dubious stuff in there while
> looking at this, in particular the setmaxmem is not rolled back if
> set_pod_target fails.
Yes, you are right! It is also not rolled back in case the xenstore
transaction fails with errno != EAGAIN, even before 0c029c4da21.
> Also the transiently "wrong" maxmem between the
> setmaxmem and a failed transaction getting redone might possibly give
> rise to some interesting cases, especially if anything else fails the
> second time around the loop.
I am thinking that the while idea of an "enforce" option was a bad to
begin with. Not only we should revert 0c029c4da21, but actually we
should just get rid of "enforce" altogether? I am suggesting it because
I don't think that reverting 0c029c4da21 would fix all the issues.
In a pre-0c029c4da21 world:
target=3 enforce=1
start transaction
do xenstore stuff, seeing target=1, setting target=3
memorykb = new_target_memkb + videoram;
xc_setmaxmem(3+videoram)
transaction commit (fail errno=ESOMETHING)
Now target=1, maxmem=3+videoram
Unless we say that we don't care about leaving the system in a
consistent state in case of failures != EAGAIN.
next prev parent reply other threads:[~2015-06-17 13:35 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-06-05 16:43 QEMU bumping memory bug analysis Wei Liu
2015-06-05 16:58 ` Ian Campbell
2015-06-05 17:13 ` Stefano Stabellini
2015-06-05 19:06 ` Wei Liu
2015-06-05 17:17 ` Andrew Cooper
2015-06-05 17:39 ` Wei Liu
2015-06-05 17:10 ` Stefano Stabellini
2015-06-05 18:10 ` Wei Liu
2015-06-08 11:39 ` Stefano Stabellini
2015-06-08 12:14 ` Andrew Cooper
2015-06-08 13:01 ` Stefano Stabellini
2015-06-08 13:33 ` Jan Beulich
2015-06-08 13:10 ` Wei Liu
2015-06-08 13:27 ` Stefano Stabellini
2015-06-08 13:32 ` Wei Liu
2015-06-08 13:38 ` Stefano Stabellini
2015-06-08 13:44 ` Andrew Cooper
2015-06-08 13:45 ` Stefano Stabellini
2015-06-05 18:49 ` Ian Campbell
2015-06-08 11:40 ` Stefano Stabellini
2015-06-08 12:11 ` Ian Campbell
2015-06-08 13:22 ` Stefano Stabellini
2015-06-08 13:52 ` Ian Campbell
2015-06-08 14:20 ` George Dunlap
2015-06-08 15:01 ` Don Slutz
2015-06-08 15:37 ` George Dunlap
2015-06-08 16:06 ` Don Slutz
2015-06-09 10:00 ` George Dunlap
2015-06-09 10:17 ` Wei Liu
2015-06-09 10:14 ` Stefano Stabellini
2015-06-09 11:20 ` George Dunlap
2015-06-16 16:44 ` Stefano Stabellini
2015-06-09 12:45 ` Ian Campbell
2015-06-17 13:35 ` Stefano Stabellini [this message]
2015-06-08 14:53 ` Konrad Rzeszutek Wilk
2015-06-08 15:20 ` George Dunlap
2015-06-08 15:42 ` Konrad Rzeszutek Wilk
2015-06-08 14:14 ` George Dunlap
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.02.1506161750470.21829@kaball.uk.xensource.com \
--to=stefano.stabellini@eu.citrix.com \
--cc=Ian.Jackson@eu.citrix.com \
--cc=andrew.cooper3@citrix.com \
--cc=dslutz@verizon.com \
--cc=george.dunlap@eu.citrix.com \
--cc=ian.campbell@citrix.com \
--cc=wei.liu2@citrix.com \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.