* [PATCH v3] tools/libxc: Batch memory allocations for PV guests
@ 2015-06-15 10:12 Ross Lagerwall
2015-06-15 16:08 ` Wei Liu
0 siblings, 1 reply; 6+ messages in thread
From: Ross Lagerwall @ 2015-06-15 10:12 UTC (permalink / raw
To: xen-devel
Cc: Ian Jackson, Ross Lagerwall, Wei Liu, Ian Campbell,
Stefano Stabellini
The current code for allocating memory for PV guests batches the
hypercalls to allocate memory by allocating 1024*1024 extents of order 0
at a time. To make this faster, first try allocating extents of order 9
(2 MiB) before falling back to the order 0 allocating if the order 9
allocation fails.
On my test machine this reduced the time to start a 128 GiB PV guest by
about 60 seconds.
Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
---
Changed in v3: Only batch up to 512 superpage allocations per hypercall.
tools/libxc/xc_dom_x86.c | 35 +++++++++++++++++++++++++++++++++--
1 file changed, 33 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/xc_dom_x86.c b/tools/libxc/xc_dom_x86.c
index 783f749..9c7e9eb 100644
--- a/tools/libxc/xc_dom_x86.c
+++ b/tools/libxc/xc_dom_x86.c
@@ -42,6 +42,7 @@
#define SUPERPAGE_PFN_SHIFT 9
#define SUPERPAGE_NR_PFNS (1UL << SUPERPAGE_PFN_SHIFT)
+#define SUPERPAGE_BATCH_SIZE 512
#define bits_to_mask(bits) (((xen_vaddr_t)1 << (bits))-1)
#define round_down(addr, mask) ((addr) & ~(mask))
@@ -761,7 +762,7 @@ int arch_setup_meminit(struct xc_dom_image *dom)
{
int rc;
xen_pfn_t pfn, allocsz, mfn, total, pfn_base;
- int i, j;
+ int i, j, k;
rc = x86_compat(dom->xch, dom->guest_domid, dom->guest_type);
if ( rc )
@@ -869,6 +870,9 @@ int arch_setup_meminit(struct xc_dom_image *dom)
unsigned int memflags;
uint64_t pages;
unsigned int pnode = dom->vnode_to_pnode[dom->vmemranges[i].nid];
+ int nr_spages = dom->total_pages >> SUPERPAGE_PFN_SHIFT;
+ xen_pfn_t extents[SUPERPAGE_BATCH_SIZE];
+ xen_pfn_t pfn_base_idx;
memflags = 0;
if ( pnode != XC_NUMA_NO_NODE )
@@ -881,7 +885,33 @@ int arch_setup_meminit(struct xc_dom_image *dom)
for ( pfn = pfn_base; pfn < pfn_base+pages; pfn++ )
dom->p2m_host[pfn] = pfn;
- for ( j = 0; j < pages; j += allocsz )
+ pfn_base_idx = pfn_base;
+ while (nr_spages) {
+ int count = min(nr_spages, SUPERPAGE_BATCH_SIZE);
+ nr_spages -= count;
+
+ for ( pfn = pfn_base_idx, j = 0;
+ pfn < pfn_base_idx + (count << SUPERPAGE_PFN_SHIFT);
+ pfn += SUPERPAGE_NR_PFNS, j++ )
+ extents[j] = dom->p2m_host[pfn];
+ rc = xc_domain_populate_physmap(dom->xch, dom->guest_domid, count,
+ SUPERPAGE_PFN_SHIFT, memflags,
+ extents);
+ if ( rc < 0 )
+ return rc;
+
+ /* Expand the returned mfns into the p2m array. */
+ pfn = pfn_base_idx;
+ for ( j = 0; j < rc; j++ )
+ {
+ mfn = extents[j];
+ for ( k = 0; k < SUPERPAGE_NR_PFNS; k++, pfn++ )
+ dom->p2m_host[pfn] = mfn + k;
+ }
+ pfn_base_idx = pfn;
+ }
+
+ for ( j = pfn_base_idx - pfn_base; j < pages; j += allocsz )
{
allocsz = pages - j;
if ( allocsz > 1024*1024 )
@@ -904,6 +934,7 @@ int arch_setup_meminit(struct xc_dom_image *dom)
return rc;
}
}
+ rc = 0;
}
/* Ensure no unclaimed pages are left unused.
--
2.1.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v3] tools/libxc: Batch memory allocations for PV guests
2015-06-15 10:12 [PATCH v3] tools/libxc: Batch memory allocations for PV guests Ross Lagerwall
@ 2015-06-15 16:08 ` Wei Liu
2015-06-17 11:14 ` Ian Campbell
0 siblings, 1 reply; 6+ messages in thread
From: Wei Liu @ 2015-06-15 16:08 UTC (permalink / raw
To: Ross Lagerwall
Cc: Ian Jackson, Stefano Stabellini, Wei Liu, Ian Campbell, xen-devel
On Mon, Jun 15, 2015 at 11:12:07AM +0100, Ross Lagerwall wrote:
> The current code for allocating memory for PV guests batches the
> hypercalls to allocate memory by allocating 1024*1024 extents of order 0
> at a time. To make this faster, first try allocating extents of order 9
> (2 MiB) before falling back to the order 0 allocating if the order 9
> allocation fails.
>
> On my test machine this reduced the time to start a 128 GiB PV guest by
> about 60 seconds.
>
> Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] tools/libxc: Batch memory allocations for PV guests
2015-06-15 16:08 ` Wei Liu
@ 2015-06-17 11:14 ` Ian Campbell
2015-06-17 11:22 ` Ross Lagerwall
2015-06-17 11:24 ` Wei Liu
0 siblings, 2 replies; 6+ messages in thread
From: Ian Campbell @ 2015-06-17 11:14 UTC (permalink / raw
To: Wei Liu; +Cc: Ross Lagerwall, Stefano Stabellini, Ian Jackson, xen-devel
On Mon, 2015-06-15 at 17:08 +0100, Wei Liu wrote:
> On Mon, Jun 15, 2015 at 11:12:07AM +0100, Ross Lagerwall wrote:
> > The current code for allocating memory for PV guests batches the
> > hypercalls to allocate memory by allocating 1024*1024 extents of order 0
> > at a time. To make this faster, first try allocating extents of order 9
> > (2 MiB) before falling back to the order 0 allocating if the order 9
> > allocation fails.
> >
> > On my test machine this reduced the time to start a 128 GiB PV guest by
> > about 60 seconds.
>From 61s or from 600s?
> >
> > Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
>
> Acked-by: Wei Liu <wei.liu2@citrix.com>
Applied. There were some rejects due to "libxc: unify handling of vNUMA
layout" having gone in first, I think they were just contextual and they
seemed trivial to resolve, but please do (both) check.
Ian.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] tools/libxc: Batch memory allocations for PV guests
2015-06-17 11:14 ` Ian Campbell
@ 2015-06-17 11:22 ` Ross Lagerwall
2015-06-17 11:40 ` Ian Campbell
2015-06-17 11:24 ` Wei Liu
1 sibling, 1 reply; 6+ messages in thread
From: Ross Lagerwall @ 2015-06-17 11:22 UTC (permalink / raw
To: Ian Campbell, Wei Liu; +Cc: Stefano Stabellini, Ian Jackson, xen-devel
On 06/17/2015 12:14 PM, Ian Campbell wrote:
> On Mon, 2015-06-15 at 17:08 +0100, Wei Liu wrote:
>> On Mon, Jun 15, 2015 at 11:12:07AM +0100, Ross Lagerwall wrote:
>>> The current code for allocating memory for PV guests batches the
>>> hypercalls to allocate memory by allocating 1024*1024 extents of order 0
>>> at a time. To make this faster, first try allocating extents of order 9
>>> (2 MiB) before falling back to the order 0 allocating if the order 9
>>> allocation fails.
>>>
>>> On my test machine this reduced the time to start a 128 GiB PV guest by
>>> about 60 seconds.
>
> From 61s or from 600s?
It reduced the total time _by_ 60 seconds. I think it takes about 10
seconds to start the guest now, but it is not related to the size of
guest memory.
Regards
--
Ross Lagerwall
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] tools/libxc: Batch memory allocations for PV guests
2015-06-17 11:14 ` Ian Campbell
2015-06-17 11:22 ` Ross Lagerwall
@ 2015-06-17 11:24 ` Wei Liu
1 sibling, 0 replies; 6+ messages in thread
From: Wei Liu @ 2015-06-17 11:24 UTC (permalink / raw
To: Ian Campbell
Cc: Ross Lagerwall, Stefano Stabellini, Wei Liu, Ian Jackson,
xen-devel
On Wed, Jun 17, 2015 at 12:14:03PM +0100, Ian Campbell wrote:
> On Mon, 2015-06-15 at 17:08 +0100, Wei Liu wrote:
> > On Mon, Jun 15, 2015 at 11:12:07AM +0100, Ross Lagerwall wrote:
> > > The current code for allocating memory for PV guests batches the
> > > hypercalls to allocate memory by allocating 1024*1024 extents of order 0
> > > at a time. To make this faster, first try allocating extents of order 9
> > > (2 MiB) before falling back to the order 0 allocating if the order 9
> > > allocation fails.
> > >
> > > On my test machine this reduced the time to start a 128 GiB PV guest by
> > > about 60 seconds.
>
> >From 61s or from 600s?
>
> > >
> > > Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
> >
> > Acked-by: Wei Liu <wei.liu2@citrix.com>
>
> Applied. There were some rejects due to "libxc: unify handling of vNUMA
> layout" having gone in first, I think they were just contextual and they
> seemed trivial to resolve, but please do (both) check.
>
The patch looks good to me.
Wei.
> Ian.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] tools/libxc: Batch memory allocations for PV guests
2015-06-17 11:22 ` Ross Lagerwall
@ 2015-06-17 11:40 ` Ian Campbell
0 siblings, 0 replies; 6+ messages in thread
From: Ian Campbell @ 2015-06-17 11:40 UTC (permalink / raw
To: Ross Lagerwall; +Cc: Ian Jackson, Stefano Stabellini, Wei Liu, xen-devel
On Wed, 2015-06-17 at 12:22 +0100, Ross Lagerwall wrote:
> On 06/17/2015 12:14 PM, Ian Campbell wrote:
> > On Mon, 2015-06-15 at 17:08 +0100, Wei Liu wrote:
> >> On Mon, Jun 15, 2015 at 11:12:07AM +0100, Ross Lagerwall wrote:
> >>> The current code for allocating memory for PV guests batches the
> >>> hypercalls to allocate memory by allocating 1024*1024 extents of order 0
> >>> at a time. To make this faster, first try allocating extents of order 9
> >>> (2 MiB) before falling back to the order 0 allocating if the order 9
> >>> allocation fails.
> >>>
> >>> On my test machine this reduced the time to start a 128 GiB PV guest by
> >>> about 60 seconds.
> >
> > From 61s or from 600s?
>
> It reduced the total time _by_ 60 seconds.
I know, my question was whether 60s was a significant or insignificant
fraction of the time take to start a guest. Seems like it was pretty
significant.
> I think it takes about 10
> seconds to start the guest now, but it is not related to the size of
> guest memory.
>
> Regards
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-06-17 11:40 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-06-15 10:12 [PATCH v3] tools/libxc: Batch memory allocations for PV guests Ross Lagerwall
2015-06-15 16:08 ` Wei Liu
2015-06-17 11:14 ` Ian Campbell
2015-06-17 11:22 ` Ross Lagerwall
2015-06-17 11:40 ` Ian Campbell
2015-06-17 11:24 ` Wei Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).