* [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
@ 2015-07-09 11:47 Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 1/7] memory: get rid of memory_region_destructor_ram_from_ptr() Igor Mammedov
` (7 more replies)
0 siblings, 8 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
Changelog:
v3->v4:
* drop patch extending memory_region_subregion_add()
with error argument
* and add memory_region_add_subregion_to_hva() API instead
* add madvise(DONTNEED) when returning range to HVA container
v2->v3:
* fixed(work-arouned) unmapping issues,
now memory subsytem keeps track of HVA mapped
regions and doesn't allow to map a new region
at address where previos has benn mapped until
previous region is gone
* fixed offset calculations in memory_region_find_hva_range()
in 2/8
* redone MemorySection folding into HVA range for VHOST,
now compacted memory map is temporary and passed only to vhost
backend and doesn't touch original memory map used by QEMU
v1->v2:
* take into account Paolo's review comments
* do not overload ram_addr
* ifdef linux specific code
* reseve HVA using API from exec.c instead of calling
mmap() dircely from memory.c
* support unmapping of HVA remapped region
When more than ~50 pc-dimm devices are hotplugged with
vhost enabled, QEMU will assert in vhost vhost_commit()
due to backend refusing to accept too many memory ranges.
Series introduces Reserved HVA MemoryRegion container
where to all hotplugged memory is remapped and passes
the single container range to vhost instead of multiple
memory ranges for each hotlugged pc-dimm device.
It's an alternative approach to increasing backend supported
memory regions limit.
Tested it a bit more, so now
- migration from current master to patched version seems to work
- memory is returned to host after device_del+object_del sequence,
but I can't bet if cgroups won't charge it.
git branch for testing:
https://github.com/imammedo/qemu/commits/vhost_one_hp_range_v4
Igor Mammedov (7):
memory: get rid of memory_region_destructor_ram_from_ptr()
memory: introduce MemoryRegion container with reserved HVA range
pc: reserve hotpluggable memory range with
memory_region_init_hva_range()
pc: fix QEMU crashing when more than ~50 memory hotplugged
exec: make sure that RAMBlock descriptor won't be leaked
exec: add qemu_ram_unmap_hva() API for unmapping memory from HVA area
memory: add support for deleting HVA mapped MemoryRegion
exec.c | 71 +++++++++++++++++++----------
hw/i386/pc.c | 4 +-
hw/mem/pc-dimm.c | 6 ++-
hw/virtio/vhost.c | 47 ++++++++++++++++++--
include/exec/cpu-common.h | 3 ++
include/exec/memory.h | 67 +++++++++++++++++++++++++++-
include/exec/ram_addr.h | 1 -
include/hw/virtio/vhost.h | 1 +
memory.c | 111 +++++++++++++++++++++++++++++++++++++++++++---
9 files changed, 272 insertions(+), 39 deletions(-)
--
1.8.3.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Qemu-devel] [PATCH v4 1/7] memory: get rid of memory_region_destructor_ram_from_ptr()
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
@ 2015-07-09 11:47 ` Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 2/7] memory: introduce MemoryRegion container with reserved HVA range Igor Mammedov
` (6 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
memory_region_destructor_ram() -> qemu_ram_free()
calls reclaim_ramblock() which does:
if (!(block->flags & RAM_PREALLOC))
free_host_memory()
g_free(block)
while
memory_region_destructor_ram_from_ptr() -> qemu_ram_free_from_ptr()
calls g_free_rcu(block, rcu) results -> g_free(block)
and for memory_region_init_ram_ptr() we set RAM_PREALLOC
so qemu_ram_free() degrades to just g_free(block) so we could
just use qemu_ram_free() instead of qemu_ram_free_from_ptr().
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Conflicts:
memory.c
---
exec.c | 19 -------------------
include/exec/ram_addr.h | 1 -
memory.c | 7 +------
3 files changed, 1 insertion(+), 26 deletions(-)
diff --git a/exec.c b/exec.c
index b7f7f98..ca53537 100644
--- a/exec.c
+++ b/exec.c
@@ -1576,25 +1576,6 @@ ram_addr_t qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true, mr, errp);
}
-void qemu_ram_free_from_ptr(ram_addr_t addr)
-{
- RAMBlock *block;
-
- qemu_mutex_lock_ramlist();
- QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
- if (addr == block->offset) {
- QLIST_REMOVE_RCU(block, next);
- ram_list.mru_block = NULL;
- /* Write list before version */
- smp_wmb();
- ram_list.version++;
- g_free_rcu(block, rcu);
- break;
- }
- }
- qemu_mutex_unlock_ramlist();
-}
-
static void reclaim_ramblock(RAMBlock *block)
{
if (block->flags & RAM_PREALLOC) {
diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
index c113f21..9c10462 100644
--- a/include/exec/ram_addr.h
+++ b/include/exec/ram_addr.h
@@ -37,7 +37,6 @@ int qemu_get_ram_fd(ram_addr_t addr);
void *qemu_get_ram_block_host_ptr(ram_addr_t addr);
void *qemu_get_ram_ptr(ram_addr_t addr);
void qemu_ram_free(ram_addr_t addr);
-void qemu_ram_free_from_ptr(ram_addr_t addr);
int qemu_ram_resize(ram_addr_t base, ram_addr_t newsize, Error **errp);
diff --git a/memory.c b/memory.c
index 5a0cc66..ec07ae8 100644
--- a/memory.c
+++ b/memory.c
@@ -864,11 +864,6 @@ static void memory_region_destructor_alias(MemoryRegion *mr)
memory_region_unref(mr->alias);
}
-static void memory_region_destructor_ram_from_ptr(MemoryRegion *mr)
-{
- qemu_ram_free_from_ptr(mr->ram_addr);
-}
-
static void memory_region_destructor_rom_device(MemoryRegion *mr)
{
qemu_ram_free(mr->ram_addr & TARGET_PAGE_MASK);
@@ -1251,7 +1246,7 @@ void memory_region_init_ram_ptr(MemoryRegion *mr,
memory_region_init(mr, owner, name, size);
mr->ram = true;
mr->terminates = true;
- mr->destructor = memory_region_destructor_ram_from_ptr;
+ mr->destructor = memory_region_destructor_ram;
mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
/* qemu_ram_alloc_from_ptr cannot fail with ptr != NULL. */
--
1.8.3.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Qemu-devel] [PATCH v4 2/7] memory: introduce MemoryRegion container with reserved HVA range
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 1/7] memory: get rid of memory_region_destructor_ram_from_ptr() Igor Mammedov
@ 2015-07-09 11:47 ` Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 3/7] pc: reserve hotpluggable memory range with memory_region_init_hva_range() Igor Mammedov
` (5 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
Patch adds
- memory_region_init_hva_range()
- memory_region_add_subregion_to_hva()
- memory_region_find_hva_range()
API to allocate, map into and lookup reserved HVA MemoryRegion.
MemoryRegion with reserved HVA range will be used for
providing linear 1:1 HVA->GVA mapping for RAM MemoryRegion-s
that are added as subregions inside it.
It will be used for memory hotplug and vhost integration,
reducing all hotplugged MemoryRegions down to a single
memory range descriptor, which allows to overcome
vhost's limitation on number of allowed memory ranges.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v1->v4:
- fix offset calculation in memory_region_find_hva_range()
- add memory_region_add_subregion_to_hva()
RFC->v1:
- rename:
memory_region_init_rsvd_hva -> memory_region_init_hva_range
memory_region_find_rsvd_hva -> memory_region_find_hva_range
- replace using ram_addr with "void *rsvd_hva"
- guard linux specific calls with ifdef
- split memory reservation into qemu_ram_reserve_hva()
---
exec.c | 30 ++++++++++++++++++++++
include/exec/cpu-common.h | 2 ++
include/exec/memory.h | 63 +++++++++++++++++++++++++++++++++++++++++++++--
memory.c | 50 +++++++++++++++++++++++++++++++++++++
4 files changed, 143 insertions(+), 2 deletions(-)
diff --git a/exec.c b/exec.c
index ca53537..562dae5 100644
--- a/exec.c
+++ b/exec.c
@@ -1339,6 +1339,36 @@ static int memory_try_enable_merging(void *addr, size_t len)
return qemu_madvise(addr, len, QEMU_MADV_MERGEABLE);
}
+#ifdef __linux__
+void *qemu_ram_reserve_hva(ram_addr_t length)
+{
+ return mmap(0, length, PROT_NONE,
+ MAP_NORESERVE | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+}
+
+void qemu_ram_remap_hva(ram_addr_t addr, void *new_hva)
+{
+ RAMBlock *block = find_ram_block(addr);
+
+ assert(block);
+ block->host = mremap(block->host, block->used_length,
+ block->used_length,
+ MREMAP_MAYMOVE | MREMAP_FIXED, new_hva);
+ memory_try_enable_merging(block->host, block->used_length);
+ qemu_ram_setup_dump(block->host, block->used_length);
+}
+#else
+void *qemu_ram_reserve_hva(ram_addr_t length)
+{
+ return NULL;
+}
+
+void qemu_ram_remap_hva(ram_addr_t addr, void *new_hva)
+{
+ assert(0);
+}
+#endif
+
/* Only legal before guest might have detected the memory size: e.g. on
* incoming migration, or right after reset.
*
diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index 9fb1d54..301f50b 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -62,6 +62,8 @@ typedef void CPUWriteMemoryFunc(void *opaque, hwaddr addr, uint32_t value);
typedef uint32_t CPUReadMemoryFunc(void *opaque, hwaddr addr);
void qemu_ram_remap(ram_addr_t addr, ram_addr_t length);
+void *qemu_ram_reserve_hva(ram_addr_t length);
+void qemu_ram_remap_hva(ram_addr_t addr, void *new_hva);
/* This should not be used by devices. */
MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr);
void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev);
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 1394715..1f2cbd1 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -173,6 +173,7 @@ struct MemoryRegion {
bool terminates;
bool romd_mode;
bool ram;
+ void *rsvd_hva;
bool skip_dump;
bool readonly; /* For RAM regions */
bool enabled;
@@ -285,6 +286,26 @@ void memory_region_init(MemoryRegion *mr,
uint64_t size);
/**
+ * memory_region_init_hva_range: Initialize a reserved HVA memory region
+ *
+ * The container for RAM memory regions.
+ * When adding subregion with memory_region_add_subregion(), subregion's
+ * backing host memory will be remapped inside of the reserved by this
+ * region HVA.
+ * Supported only on Linux. If memory reservation and remapping is not
+ * implemented for platform, this call degrades to regular memory_region_init().
+ *
+ * @mr: the #MemoryRegion to be initialized
+ * @owner: the object that tracks the region's reference count
+ * @name: used for debugging; not visible to the user or ABI
+ * @size: size of the region; any subregions beyond this size will be clipped
+ */
+void memory_region_init_hva_range(MemoryRegion *mr,
+ struct Object *owner,
+ const char *name,
+ uint64_t size);
+
+/**
* memory_region_ref: Add 1 to a memory region's reference count
*
* Whenever memory regions are accessed outside the BQL, they need to be
@@ -634,8 +655,8 @@ int memory_region_get_fd(MemoryRegion *mr);
* memory_region_get_ram_ptr: Get a pointer into a RAM memory region.
*
* Returns a host pointer to a RAM memory region (created with
- * memory_region_init_ram() or memory_region_init_ram_ptr()). Use with
- * care.
+ * memory_region_init_ram() or memory_region_init_ram_ptr()) or
+ * memory_region_init_hva_range(). Use with care.
*
* @mr: the memory region being queried.
*/
@@ -909,6 +930,24 @@ void memory_region_del_eventfd(MemoryRegion *mr,
void memory_region_add_subregion(MemoryRegion *mr,
hwaddr offset,
MemoryRegion *subregion);
+
+/**
+ * memory_region_add_subregion_to_hva: Add a subregion to a HVA container.
+ *
+ * the same as memory_region_add_subregion() with only difference that
+ * it remaps RAM subregion's backing memory into HVA range of @mr.
+ * If HVA region is not supported by host the call degrades to and behaves as
+ * memory_region_add_subregion().
+ *
+ * @mr: the region to contain the new subregion; must be a container
+ * initialized with memory_region_init().
+ * @offset: the offset relative to @mr where @subregion is added.
+ * @subregion: the subregion to be added.
+ */
+void memory_region_add_subregion_to_hva(MemoryRegion *mr,
+ hwaddr offset,
+ MemoryRegion *subregion);
+
/**
* memory_region_add_subregion_overlap: Add a subregion to a container
* with overlap.
@@ -1052,6 +1091,26 @@ MemoryRegionSection memory_region_find(MemoryRegion *mr,
hwaddr addr, uint64_t size);
/**
+ * memory_region_find_hva_range: finds a parent MemoryRegion with
+ * reserved HVA and translates it into a #MemoryRegionSection.
+ *
+ * Locates the first parent #MemoryRegion of @mr that is
+ * of reserved HVA type.
+ *
+ * Returns a #MemoryRegionSection that describes a reserved HVA
+ * memory region.
+ * .@offset_within_address_space is offset of found
+ * (in the .@mr field) memory region relative to the address
+ * space that contains it.
+ * .@offset_within_region is offset of @mr relative
+ * to the returned region (in the .@mr field).
+ * .@size is size of found memory region
+ *
+ * @mr: a MemoryRegion whose HVA parent is looked up
+ */
+MemoryRegionSection memory_region_find_hva_range(MemoryRegion *mr);
+
+/**
* address_space_sync_dirty_bitmap: synchronize the dirty log for all memory
*
* Synchronizes the dirty page log for an entire address space.
diff --git a/memory.c b/memory.c
index ec07ae8..bf6aa4e 100644
--- a/memory.c
+++ b/memory.c
@@ -929,6 +929,15 @@ void memory_region_init(MemoryRegion *mr,
}
}
+void memory_region_init_hva_range(MemoryRegion *mr,
+ Object *owner,
+ const char *name,
+ uint64_t size)
+{
+ memory_region_init(mr, owner, name, size);
+ mr->rsvd_hva = qemu_ram_reserve_hva(memory_region_size(mr));
+}
+
static void memory_region_get_addr(Object *obj, Visitor *v, void *opaque,
const char *name, Error **errp)
{
@@ -1517,6 +1526,10 @@ int memory_region_get_fd(MemoryRegion *mr)
void *memory_region_get_ram_ptr(MemoryRegion *mr)
{
+ if (mr->rsvd_hva) {
+ return mr->rsvd_hva;
+ }
+
if (mr->alias) {
return memory_region_get_ram_ptr(mr->alias) + mr->alias_offset;
}
@@ -1777,6 +1790,17 @@ void memory_region_add_subregion_overlap(MemoryRegion *mr,
memory_region_add_subregion_common(mr, offset, subregion);
}
+void memory_region_add_subregion_to_hva(MemoryRegion *mr,
+ hwaddr offset,
+ MemoryRegion *subregion)
+{
+ if (mr->rsvd_hva && subregion->ram) {
+ qemu_ram_remap_hva(subregion->ram_addr,
+ memory_region_get_ram_ptr(mr) + offset);
+ }
+ memory_region_add_subregion(mr, offset, subregion);
+}
+
void memory_region_del_subregion(MemoryRegion *mr,
MemoryRegion *subregion)
{
@@ -1897,6 +1921,32 @@ bool memory_region_is_mapped(MemoryRegion *mr)
return mr->container ? true : false;
}
+MemoryRegionSection memory_region_find_hva_range(MemoryRegion *mr)
+{
+ MemoryRegionSection ret = { .mr = NULL };
+ MemoryRegion *hva_container = NULL;
+ hwaddr addr = 0;
+ MemoryRegion *root;
+
+ for (root = mr; root->container; root = root->container) {
+ if (!hva_container && root->rsvd_hva) {
+ hva_container = root;
+ ret.offset_within_region = addr;
+ }
+ addr += root->addr;
+ }
+
+ ret.address_space = memory_region_to_address_space(root);
+ if (!ret.address_space || !hva_container) {
+ return ret;
+ }
+
+ ret.mr = hva_container;
+ ret.offset_within_address_space = addr;
+ ret.size = int128_make64(memory_region_size(ret.mr));
+ return ret;
+}
+
MemoryRegionSection memory_region_find(MemoryRegion *mr,
hwaddr addr, uint64_t size)
{
--
1.8.3.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Qemu-devel] [PATCH v4 3/7] pc: reserve hotpluggable memory range with memory_region_init_hva_range()
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 1/7] memory: get rid of memory_region_destructor_ram_from_ptr() Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 2/7] memory: introduce MemoryRegion container with reserved HVA range Igor Mammedov
@ 2015-07-09 11:47 ` Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged Igor Mammedov
` (4 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
It makes sure that all hotplugged memory will be put in
continuos HVA range allowing to use 1:1 GVA<->HVA mapping.
1:1 mapping will be used by vhost to reduce number of memory
ranges for hotplugged memory to a single range that covers
all hotpluggable memory address space.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
hw/i386/pc.c | 4 ++--
hw/mem/pc-dimm.c | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 7661ea9..75f73ec 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1373,8 +1373,8 @@ FWCfgState *pc_memory_init(MachineState *machine,
exit(EXIT_FAILURE);
}
- memory_region_init(&pcms->hotplug_memory.mr, OBJECT(pcms),
- "hotplug-memory", hotplug_mem_size);
+ memory_region_init_hva_range(&pcms->hotplug_memory.mr, OBJECT(pcms),
+ "hotplug-memory", hotplug_mem_size);
memory_region_add_subregion(system_memory, pcms->hotplug_memory.base,
&pcms->hotplug_memory.mr);
}
diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
index bb04862..cb98926 100644
--- a/hw/mem/pc-dimm.c
+++ b/hw/mem/pc-dimm.c
@@ -95,7 +95,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
goto out;
}
- memory_region_add_subregion(&hpms->mr, addr - hpms->base, mr);
+ memory_region_add_subregion_to_hva(&hpms->mr, addr - hpms->base, mr);
vmstate_register_ram(mr, dev);
numa_set_mem_node_id(addr, memory_region_size(mr), dimm->node);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
` (2 preceding siblings ...)
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 3/7] pc: reserve hotpluggable memory range with memory_region_init_hva_range() Igor Mammedov
@ 2015-07-09 11:47 ` Igor Mammedov
2015-07-09 13:06 ` Michael S. Tsirkin
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 5/7] exec: make sure that RAMBlock descriptor won't be leaked Igor Mammedov
` (3 subsequent siblings)
7 siblings, 1 reply; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
QEMU asserts in vhost due to hitting vhost backend limit
on number of supported memory regions.
Describe all hotplugged memory as one continuos range
to vhost with linear 1:1 HVA->GPA mapping in backend.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
hw/virtio/vhost.c | 47 ++++++++++++++++++++++++++++++++++++++++++++---
include/hw/virtio/vhost.h | 1 +
2 files changed, 45 insertions(+), 3 deletions(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 2712c6f..7bc27f0 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -432,6 +432,10 @@ static void vhost_set_memory(MemoryListener *listener,
assert(size);
+ if (!dev->rsvd_hva.mr) {
+ dev->rsvd_hva = memory_region_find_hva_range(section->mr);
+ }
+
/* Optimize no-change case. At least cirrus_vga does this a lot at this time. */
ram = memory_region_get_ram_ptr(section->mr) + section->offset_within_region;
if (add) {
@@ -472,6 +476,42 @@ static void vhost_begin(MemoryListener *listener)
dev->mem_changed_start_addr = -1;
}
+static int vhost_set_mem_table(struct vhost_dev *dev)
+{
+ hwaddr start_addr = 0;
+ ram_addr_t size = 0;
+ struct vhost_memory *mem;
+ int r, i;
+
+ /* drop memory ranges from continuos HVA */
+ mem = g_memdup(dev->mem, offsetof(struct vhost_memory, regions) +
+ dev->mem->nregions * sizeof dev->mem->regions[0]);
+ start_addr = dev->rsvd_hva.offset_within_address_space;
+ size = int128_get64(dev->rsvd_hva.size);
+ for (i = 0; i < mem->nregions; i++) {
+ if (mem->regions[i].guest_phys_addr >= start_addr &&
+ mem->regions[i].guest_phys_addr < start_addr + size) {
+ mem->nregions--;
+ memmove(&mem->regions[i], &mem->regions[i + 1],
+ (mem->nregions - i) * sizeof mem->regions[0]);
+ }
+ }
+ /* add one continuos HVA entry if memory ranges from it is present */
+ if (dev->mem->nregions > mem->nregions) {
+ struct vhost_memory_region *reg = &mem->regions[mem->nregions];
+
+ reg->guest_phys_addr = start_addr;
+ reg->memory_size = size;
+ reg->userspace_addr =
+ (__u64)memory_region_get_ram_ptr(dev->rsvd_hva.mr);
+ mem->nregions++;
+ }
+
+ r = dev->vhost_ops->vhost_call(dev, VHOST_SET_MEM_TABLE, mem);
+ g_free(mem);
+ return r;
+}
+
static void vhost_commit(MemoryListener *listener)
{
struct vhost_dev *dev = container_of(listener, struct vhost_dev,
@@ -500,7 +540,7 @@ static void vhost_commit(MemoryListener *listener)
}
if (!dev->log_enabled) {
- r = dev->vhost_ops->vhost_call(dev, VHOST_SET_MEM_TABLE, dev->mem);
+ r = vhost_set_mem_table(dev);
assert(r >= 0);
dev->memory_changed = false;
return;
@@ -513,7 +553,7 @@ static void vhost_commit(MemoryListener *listener)
if (dev->log_size < log_size) {
vhost_dev_log_resize(dev, log_size + VHOST_LOG_BUFFER);
}
- r = dev->vhost_ops->vhost_call(dev, VHOST_SET_MEM_TABLE, dev->mem);
+ r = vhost_set_mem_table(dev);
assert(r >= 0);
/* To log less, can only decrease log size after table update. */
if (dev->log_size > log_size + VHOST_LOG_BUFFER) {
@@ -956,6 +996,7 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
migrate_add_blocker(hdev->migration_blocker);
}
hdev->mem = g_malloc0(offsetof(struct vhost_memory, regions));
+ memset(&hdev->rsvd_hva, 0, sizeof hdev->rsvd_hva);
hdev->n_mem_sections = 0;
hdev->mem_sections = NULL;
hdev->log = NULL;
@@ -1119,7 +1160,7 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
if (r < 0) {
goto fail_features;
}
- r = hdev->vhost_ops->vhost_call(hdev, VHOST_SET_MEM_TABLE, hdev->mem);
+ r = vhost_set_mem_table(hdev);
if (r < 0) {
r = -errno;
goto fail_mem;
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index dd51050..d41bf2f 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -40,6 +40,7 @@ struct vhost_dev {
struct vhost_memory *mem;
int n_mem_sections;
MemoryRegionSection *mem_sections;
+ MemoryRegionSection rsvd_hva;
struct vhost_virtqueue *vqs;
int nvqs;
/* the first virtqueue which would be used by this vhost dev */
--
1.8.3.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Qemu-devel] [PATCH v4 5/7] exec: make sure that RAMBlock descriptor won't be leaked
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
` (3 preceding siblings ...)
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged Igor Mammedov
@ 2015-07-09 11:47 ` Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 6/7] exec: add qemu_ram_unmap_hva() API for unmapping memory from HVA area Igor Mammedov
` (2 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
HVA remapped file backed RAMBlock shouldn't be freed
with munmap() as it will create a hole in HVA area
but file descriptor should be freed so it won't leak.
Rearrange code so that file descriptor is freed always
if it's been used and drop unnecessary munmap()
call/branch because qemu_anon_ram_free() is doing the same.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
exec.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/exec.c b/exec.c
index 562dae5..08767ab 100644
--- a/exec.c
+++ b/exec.c
@@ -1612,14 +1612,14 @@ static void reclaim_ramblock(RAMBlock *block)
;
} else if (xen_enabled()) {
xen_invalidate_map_cache_entry(block->host);
-#ifndef _WIN32
- } else if (block->fd >= 0) {
- munmap(block->host, block->max_length);
- close(block->fd);
-#endif
} else {
qemu_anon_ram_free(block->host, block->max_length);
}
+#ifndef _WIN32
+ if (block->fd >= 0) {
+ close(block->fd);
+ }
+#endif
g_free(block);
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Qemu-devel] [PATCH v4 6/7] exec: add qemu_ram_unmap_hva() API for unmapping memory from HVA area
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
` (4 preceding siblings ...)
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 5/7] exec: make sure that RAMBlock descriptor won't be leaked Igor Mammedov
@ 2015-07-09 11:47 ` Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 7/7] memory: add support for deleting HVA mapped MemoryRegion Igor Mammedov
2015-07-15 15:12 ` [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
7 siblings, 0 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
it will allow to return atomically return RAMBlock's host
address range into continuos HVA area so that no hole
would appear in there.
also mark RAMBlock with RAM_PREALLOC flag so it won't be
umapped as conventional memory by
reclaim_ramblock()->qemu_anon_ram_free()
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
* add madvise(DONTNEED) on unmapping
---
exec.c | 12 ++++++++++++
include/exec/cpu-common.h | 1 +
2 files changed, 13 insertions(+)
diff --git a/exec.c b/exec.c
index 08767ab..70ced86 100644
--- a/exec.c
+++ b/exec.c
@@ -1346,11 +1346,23 @@ void *qemu_ram_reserve_hva(ram_addr_t length)
MAP_NORESERVE | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
}
+void qemu_ram_unmap_hva(ram_addr_t addr)
+{
+ RAMBlock *block = find_ram_block(addr);
+
+ assert(block);
+ mmap(block->host, block->used_length, PROT_NONE,
+ MAP_FIXED | MAP_NORESERVE | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ qemu_madvise(block->host, block->used_length, QEMU_MADV_DONTNEED);
+}
+
void qemu_ram_remap_hva(ram_addr_t addr, void *new_hva)
{
RAMBlock *block = find_ram_block(addr);
assert(block);
+ assert(!(block->flags & RAM_PREALLOC));
+ block->flags |= RAM_PREALLOC;
block->host = mremap(block->host, block->used_length,
block->used_length,
MREMAP_MAYMOVE | MREMAP_FIXED, new_hva);
diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index 301f50b..4da5cd7 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -64,6 +64,7 @@ typedef uint32_t CPUReadMemoryFunc(void *opaque, hwaddr addr);
void qemu_ram_remap(ram_addr_t addr, ram_addr_t length);
void *qemu_ram_reserve_hva(ram_addr_t length);
void qemu_ram_remap_hva(ram_addr_t addr, void *new_hva);
+void qemu_ram_unmap_hva(ram_addr_t addr);
/* This should not be used by devices. */
MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr);
void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Qemu-devel] [PATCH v4 7/7] memory: add support for deleting HVA mapped MemoryRegion
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
` (5 preceding siblings ...)
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 6/7] exec: add qemu_ram_unmap_hva() API for unmapping memory from HVA area Igor Mammedov
@ 2015-07-09 11:47 ` Igor Mammedov
2015-07-15 15:12 ` [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
7 siblings, 0 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-09 11:47 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, mst
Although memory_region_del_subregion() removes MemoryRegion
from current address space, it's possible that it's still
in use/referenced until old address space view is destroyed.
That doesn't allow to unmap it from HVA region at the time
of memory_region_del_subregion().
As a solution track HVA mapped MemoryRegions in a list and
don't allow to map another MemoryRegion at the same address
until respective MemoryRegion is destroyed, delaying unmapping
from HVA range to the time MemoryRegion destructor is called.
Also add checks to memory_region_add_subregion_to_hva() to make
sure that HVA range is available for mapping a new region
or return error from it if address range is not available.
In memory hotplug terms it would mean that user should delete
corresponding backend along with pc-dimm device:
device_del dimm1
object_del dimm1_backend_memdev
after that dimm1_backend_memdev's MemoryRegion will be destroyed
once all accesses to it are gone and old flatview is destroyed as
well.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
hw/mem/pc-dimm.c | 6 +++++-
include/exec/memory.h | 6 +++++-
memory.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 65 insertions(+), 3 deletions(-)
diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
index cb98926..8e5f388 100644
--- a/hw/mem/pc-dimm.c
+++ b/hw/mem/pc-dimm.c
@@ -95,7 +95,11 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
goto out;
}
- memory_region_add_subregion_to_hva(&hpms->mr, addr - hpms->base, mr);
+ memory_region_add_subregion_to_hva(&hpms->mr, addr - hpms->base, mr,
+ &local_err);
+ if (local_err) {
+ goto out;
+ }
vmstate_register_ram(mr, dev);
numa_set_mem_node_id(addr, memory_region_size(mr), dimm->node);
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 1f2cbd1..0af272d 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -174,6 +174,7 @@ struct MemoryRegion {
bool romd_mode;
bool ram;
void *rsvd_hva;
+ bool hva_mapped;
bool skip_dump;
bool readonly; /* For RAM regions */
bool enabled;
@@ -188,6 +189,7 @@ struct MemoryRegion {
QTAILQ_HEAD(subregions, MemoryRegion) subregions;
QTAILQ_ENTRY(MemoryRegion) subregions_link;
QTAILQ_HEAD(coalesced_ranges, CoalescedMemoryRange) coalesced;
+ QTAILQ_ENTRY(MemoryRegion) hva_link;
const char *name;
uint8_t dirty_log_mask;
unsigned ioeventfd_nb;
@@ -943,10 +945,12 @@ void memory_region_add_subregion(MemoryRegion *mr,
* initialized with memory_region_init().
* @offset: the offset relative to @mr where @subregion is added.
* @subregion: the subregion to be added.
+ * @errp: contains error if remapping to HVA fails/not possible
*/
void memory_region_add_subregion_to_hva(MemoryRegion *mr,
hwaddr offset,
- MemoryRegion *subregion);
+ MemoryRegion *subregion,
+ Error **errp);
/**
* memory_region_add_subregion_overlap: Add a subregion to a container
diff --git a/memory.c b/memory.c
index bf6aa4e..fad0b8f 100644
--- a/memory.c
+++ b/memory.c
@@ -34,6 +34,7 @@ static unsigned memory_region_transaction_depth;
static bool memory_region_update_pending;
static bool ioeventfd_update_pending;
static bool global_dirty_log = false;
+static QemuMutex hva_lock;
static QTAILQ_HEAD(memory_listeners, MemoryListener) memory_listeners
= QTAILQ_HEAD_INITIALIZER(memory_listeners);
@@ -1761,6 +1762,24 @@ done:
memory_region_transaction_commit();
}
+static QTAILQ_HEAD(, MemoryRegion) hva_mapped_head =
+ QTAILQ_HEAD_INITIALIZER(hva_mapped_head);
+
+static void memory_region_destructor_hva_ram(MemoryRegion *mr)
+{
+ MemoryRegion *h, *tmp;
+
+ qemu_mutex_lock(&hva_lock);
+ qemu_ram_unmap_hva(mr->ram_addr);
+ memory_region_destructor_ram(mr);
+ QTAILQ_FOREACH_SAFE(h, &hva_mapped_head, hva_link, tmp) {
+ if (mr == h) {
+ QTAILQ_REMOVE(&hva_mapped_head, h, hva_link);
+ }
+ }
+ qemu_mutex_unlock(&hva_lock);
+}
+
static void memory_region_add_subregion_common(MemoryRegion *mr,
hwaddr offset,
MemoryRegion *subregion)
@@ -1792,11 +1811,45 @@ void memory_region_add_subregion_overlap(MemoryRegion *mr,
void memory_region_add_subregion_to_hva(MemoryRegion *mr,
hwaddr offset,
- MemoryRegion *subregion)
+ MemoryRegion *subregion,
+ Error **errp)
{
if (mr->rsvd_hva && subregion->ram) {
+ MemoryRegion *h, *tmp;
+ Int128 e, oe;
+
+ qemu_mutex_lock(&hva_lock);
+ QTAILQ_FOREACH_SAFE(h, &hva_mapped_head, hva_link, tmp) {
+ if (subregion->hva_mapped) {
+ error_setg(errp, "HVA mapped memory region '%s' is not "
+ "reusable, use a new one instead",
+ subregion->name);
+ qemu_mutex_unlock(&hva_lock);
+ return;
+ }
+
+ e = int128_add(int128_make64(h->addr),
+ int128_make64(memory_region_size(h)));
+ oe = int128_add(int128_make64(offset),
+ int128_make64(memory_region_size(subregion)));
+ if (offset >= h->addr && int128_le(oe, e)) {
+ MemoryRegionSection rsvd_hva;
+ rsvd_hva = memory_region_find_hva_range(mr);
+ error_setg(errp, "memory at 0x%" PRIx64 " is still in use"
+ "by HVA mapped region: %s",
+ rsvd_hva.offset_within_address_space + offset,
+ h->name);
+ qemu_mutex_unlock(&hva_lock);
+ return;
+ }
+ }
+
+ QTAILQ_INSERT_TAIL(&hva_mapped_head, subregion, hva_link);
+ subregion->destructor = memory_region_destructor_hva_ram;
+ subregion->hva_mapped = true;
qemu_ram_remap_hva(subregion->ram_addr,
memory_region_get_ram_ptr(mr) + offset);
+ qemu_mutex_unlock(&hva_lock);
}
memory_region_add_subregion(mr, offset, subregion);
}
@@ -2290,6 +2343,7 @@ static const TypeInfo memory_region_info = {
static void memory_register_types(void)
{
type_register_static(&memory_region_info);
+ qemu_mutex_init(&hva_lock);
}
type_init(memory_register_types)
--
1.8.3.1
^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged Igor Mammedov
@ 2015-07-09 13:06 ` Michael S. Tsirkin
2015-07-09 13:43 ` Paolo Bonzini
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-09 13:06 UTC (permalink / raw)
To: Igor Mammedov; +Cc: pbonzini, qemu-devel
On Thu, Jul 09, 2015 at 01:47:21PM +0200, Igor Mammedov wrote:
> QEMU asserts in vhost due to hitting vhost backend limit
> on number of supported memory regions.
>
> Describe all hotplugged memory as one continuos range
> to vhost with linear 1:1 HVA->GPA mapping in backend.
>
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Hmm - a bunch of work here to recombine MRs that memory listener
interface breaks up. In particular KVM could benefit from this too (on
workloads that change the table a lot). Can't we teach memory core to
pass hva range as a single continuous range to memory listeners?
> ---
> hw/virtio/vhost.c | 47 ++++++++++++++++++++++++++++++++++++++++++++---
> include/hw/virtio/vhost.h | 1 +
> 2 files changed, 45 insertions(+), 3 deletions(-)
>
> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> index 2712c6f..7bc27f0 100644
> --- a/hw/virtio/vhost.c
> +++ b/hw/virtio/vhost.c
> @@ -432,6 +432,10 @@ static void vhost_set_memory(MemoryListener *listener,
>
> assert(size);
>
> + if (!dev->rsvd_hva.mr) {
> + dev->rsvd_hva = memory_region_find_hva_range(section->mr);
> + }
> +
> /* Optimize no-change case. At least cirrus_vga does this a lot at this time. */
> ram = memory_region_get_ram_ptr(section->mr) + section->offset_within_region;
> if (add) {
> @@ -472,6 +476,42 @@ static void vhost_begin(MemoryListener *listener)
> dev->mem_changed_start_addr = -1;
> }
>
> +static int vhost_set_mem_table(struct vhost_dev *dev)
> +{
> + hwaddr start_addr = 0;
> + ram_addr_t size = 0;
> + struct vhost_memory *mem;
> + int r, i;
> +
> + /* drop memory ranges from continuos HVA */
> + mem = g_memdup(dev->mem, offsetof(struct vhost_memory, regions) +
> + dev->mem->nregions * sizeof dev->mem->regions[0]);
> + start_addr = dev->rsvd_hva.offset_within_address_space;
> + size = int128_get64(dev->rsvd_hva.size);
> + for (i = 0; i < mem->nregions; i++) {
> + if (mem->regions[i].guest_phys_addr >= start_addr &&
> + mem->regions[i].guest_phys_addr < start_addr + size) {
> + mem->nregions--;
> + memmove(&mem->regions[i], &mem->regions[i + 1],
> + (mem->nregions - i) * sizeof mem->regions[0]);
> + }
> + }
> + /* add one continuos HVA entry if memory ranges from it is present */
> + if (dev->mem->nregions > mem->nregions) {
> + struct vhost_memory_region *reg = &mem->regions[mem->nregions];
> +
> + reg->guest_phys_addr = start_addr;
> + reg->memory_size = size;
> + reg->userspace_addr =
> + (__u64)memory_region_get_ram_ptr(dev->rsvd_hva.mr);
> + mem->nregions++;
> + }
> +
> + r = dev->vhost_ops->vhost_call(dev, VHOST_SET_MEM_TABLE, mem);
> + g_free(mem);
> + return r;
> +}
> +
> static void vhost_commit(MemoryListener *listener)
> {
> struct vhost_dev *dev = container_of(listener, struct vhost_dev,
> @@ -500,7 +540,7 @@ static void vhost_commit(MemoryListener *listener)
> }
>
> if (!dev->log_enabled) {
> - r = dev->vhost_ops->vhost_call(dev, VHOST_SET_MEM_TABLE, dev->mem);
> + r = vhost_set_mem_table(dev);
> assert(r >= 0);
> dev->memory_changed = false;
> return;
> @@ -513,7 +553,7 @@ static void vhost_commit(MemoryListener *listener)
> if (dev->log_size < log_size) {
> vhost_dev_log_resize(dev, log_size + VHOST_LOG_BUFFER);
> }
> - r = dev->vhost_ops->vhost_call(dev, VHOST_SET_MEM_TABLE, dev->mem);
> + r = vhost_set_mem_table(dev);
> assert(r >= 0);
> /* To log less, can only decrease log size after table update. */
> if (dev->log_size > log_size + VHOST_LOG_BUFFER) {
> @@ -956,6 +996,7 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
> migrate_add_blocker(hdev->migration_blocker);
> }
> hdev->mem = g_malloc0(offsetof(struct vhost_memory, regions));
> + memset(&hdev->rsvd_hva, 0, sizeof hdev->rsvd_hva);
> hdev->n_mem_sections = 0;
> hdev->mem_sections = NULL;
> hdev->log = NULL;
> @@ -1119,7 +1160,7 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
> if (r < 0) {
> goto fail_features;
> }
> - r = hdev->vhost_ops->vhost_call(hdev, VHOST_SET_MEM_TABLE, hdev->mem);
> + r = vhost_set_mem_table(hdev);
> if (r < 0) {
> r = -errno;
> goto fail_mem;
> diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
> index dd51050..d41bf2f 100644
> --- a/include/hw/virtio/vhost.h
> +++ b/include/hw/virtio/vhost.h
> @@ -40,6 +40,7 @@ struct vhost_dev {
> struct vhost_memory *mem;
> int n_mem_sections;
> MemoryRegionSection *mem_sections;
> + MemoryRegionSection rsvd_hva;
> struct vhost_virtqueue *vqs;
> int nvqs;
> /* the first virtqueue which would be used by this vhost dev */
> --
> 1.8.3.1
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-09 13:06 ` Michael S. Tsirkin
@ 2015-07-09 13:43 ` Paolo Bonzini
2015-07-09 13:46 ` Michael S. Tsirkin
0 siblings, 1 reply; 24+ messages in thread
From: Paolo Bonzini @ 2015-07-09 13:43 UTC (permalink / raw)
To: Michael S. Tsirkin, Igor Mammedov; +Cc: qemu-devel
On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > QEMU asserts in vhost due to hitting vhost backend limit
> > on number of supported memory regions.
> >
> > Describe all hotplugged memory as one continuos range
> > to vhost with linear 1:1 HVA->GPA mapping in backend.
> >
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>
> Hmm - a bunch of work here to recombine MRs that memory listener
> interface breaks up. In particular KVM could benefit from this too (on
> workloads that change the table a lot). Can't we teach memory core to
> pass hva range as a single continuous range to memory listeners?
Memory listeners are based on memory regions, not HVA ranges.
Paolo
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-09 13:43 ` Paolo Bonzini
@ 2015-07-09 13:46 ` Michael S. Tsirkin
2015-07-10 10:12 ` Igor Mammedov
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-09 13:46 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Igor Mammedov, qemu-devel
On Thu, Jul 09, 2015 at 03:43:01PM +0200, Paolo Bonzini wrote:
>
>
> On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > > QEMU asserts in vhost due to hitting vhost backend limit
> > > on number of supported memory regions.
> > >
> > > Describe all hotplugged memory as one continuos range
> > > to vhost with linear 1:1 HVA->GPA mapping in backend.
> > >
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> >
> > Hmm - a bunch of work here to recombine MRs that memory listener
> > interface breaks up. In particular KVM could benefit from this too (on
> > workloads that change the table a lot). Can't we teach memory core to
> > pass hva range as a single continuous range to memory listeners?
>
> Memory listeners are based on memory regions, not HVA ranges.
>
> Paolo
Many listeners care about HVA ranges. I know KVM and vhost do.
I guess we could create dummy MRs to fill in the holes left by
memory hotplug? vhost already has logic to recombine
consequitive chunks created by memory core.
--
MST
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-09 13:46 ` Michael S. Tsirkin
@ 2015-07-10 10:12 ` Igor Mammedov
2015-07-13 6:55 ` Michael S. Tsirkin
0 siblings, 1 reply; 24+ messages in thread
From: Igor Mammedov @ 2015-07-10 10:12 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Paolo Bonzini, qemu-devel
On Thu, 9 Jul 2015 16:46:43 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Jul 09, 2015 at 03:43:01PM +0200, Paolo Bonzini wrote:
> >
> >
> > On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > > > QEMU asserts in vhost due to hitting vhost backend limit
> > > > on number of supported memory regions.
> > > >
> > > > Describe all hotplugged memory as one continuos range
> > > > to vhost with linear 1:1 HVA->GPA mapping in backend.
> > > >
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > >
> > > Hmm - a bunch of work here to recombine MRs that memory listener
> > > interface breaks up. In particular KVM could benefit from this too (on
> > > workloads that change the table a lot). Can't we teach memory core to
> > > pass hva range as a single continuous range to memory listeners?
> >
> > Memory listeners are based on memory regions, not HVA ranges.
> >
> > Paolo
>
> Many listeners care about HVA ranges. I know KVM and vhost do.
I'm not sure about KVM, it works just fine with fragmented memory regions,
the same will apply to vhost once module parameter to increase limit
is merged.
but changing generic memory listener interface to replace HVA mapped
regions with HVA container would lead to a case when listeners
won't see exact layout that they might need.
In addition vhost itself will suffer from working with big HVA
since it allocates log depending on size of memory => bigger log.
That's one of the reasons that in this patch HVA ranges in
memory map are compacted only for backend consumption,
QEMU's side of vhost uses exact map for internal purposes.
And the other reason is I don't know vhost enough to rewrite it
to use big HVA for everything.
> I guess we could create dummy MRs to fill in the holes left by
> memory hotplug?
it looks like nice thing from vhost pov but complicates other side,
hence I dislike an idea inventing dummy MRs for vhost's convenience.
> vhost already has logic to recombine
> consequitive chunks created by memory core.
which looks a bit complicated and I was thinking about simplifying
it some time in the future.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-10 10:12 ` Igor Mammedov
@ 2015-07-13 6:55 ` Michael S. Tsirkin
2015-07-13 18:55 ` Igor Mammedov
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-13 6:55 UTC (permalink / raw)
To: Igor Mammedov; +Cc: Paolo Bonzini, qemu-devel
On Fri, Jul 10, 2015 at 12:12:36PM +0200, Igor Mammedov wrote:
> On Thu, 9 Jul 2015 16:46:43 +0300
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Thu, Jul 09, 2015 at 03:43:01PM +0200, Paolo Bonzini wrote:
> > >
> > >
> > > On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > > > > QEMU asserts in vhost due to hitting vhost backend limit
> > > > > on number of supported memory regions.
> > > > >
> > > > > Describe all hotplugged memory as one continuos range
> > > > > to vhost with linear 1:1 HVA->GPA mapping in backend.
> > > > >
> > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > >
> > > > Hmm - a bunch of work here to recombine MRs that memory listener
> > > > interface breaks up. In particular KVM could benefit from this too (on
> > > > workloads that change the table a lot). Can't we teach memory core to
> > > > pass hva range as a single continuous range to memory listeners?
> > >
> > > Memory listeners are based on memory regions, not HVA ranges.
> > >
> > > Paolo
> >
> > Many listeners care about HVA ranges. I know KVM and vhost do.
> I'm not sure about KVM, it works just fine with fragmented memory regions,
> the same will apply to vhost once module parameter to increase limit
> is merged.
>
> but changing generic memory listener interface to replace HVA mapped
> regions with HVA container would lead to a case when listeners
> won't see exact layout that they might need.
I don't think they care, really.
> In addition vhost itself will suffer from working with big HVA
> since it allocates log depending on size of memory => bigger log.
Not really - it allocates the log depending on the PA range.
Leaving unused holes doesn't reduce it's size.
> That's one of the reasons that in this patch HVA ranges in
> memory map are compacted only for backend consumption,
> QEMU's side of vhost uses exact map for internal purposes.
> And the other reason is I don't know vhost enough to rewrite it
> to use big HVA for everything.
>
> > I guess we could create dummy MRs to fill in the holes left by
> > memory hotplug?
> it looks like nice thing from vhost pov but complicates other side,
What other side do you have in mind?
> hence I dislike an idea inventing dummy MRs for vhost's convenience.
>
>
> > vhost already has logic to recombine
> > consequitive chunks created by memory core.
> which looks a bit complicated and I was thinking about simplifying
> it some time in the future.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-13 6:55 ` Michael S. Tsirkin
@ 2015-07-13 18:55 ` Igor Mammedov
2015-07-13 20:14 ` Michael S. Tsirkin
0 siblings, 1 reply; 24+ messages in thread
From: Igor Mammedov @ 2015-07-13 18:55 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Paolo Bonzini, qemu-devel
On Mon, 13 Jul 2015 09:55:18 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Fri, Jul 10, 2015 at 12:12:36PM +0200, Igor Mammedov wrote:
> > On Thu, 9 Jul 2015 16:46:43 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Thu, Jul 09, 2015 at 03:43:01PM +0200, Paolo Bonzini wrote:
> > > >
> > > >
> > > > On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > > > > > QEMU asserts in vhost due to hitting vhost backend limit
> > > > > > on number of supported memory regions.
> > > > > >
> > > > > > Describe all hotplugged memory as one continuos range
> > > > > > to vhost with linear 1:1 HVA->GPA mapping in backend.
> > > > > >
> > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > >
> > > > > Hmm - a bunch of work here to recombine MRs that memory
> > > > > listener interface breaks up. In particular KVM could
> > > > > benefit from this too (on workloads that change the table a
> > > > > lot). Can't we teach memory core to pass hva range as a
> > > > > single continuous range to memory listeners?
> > > >
> > > > Memory listeners are based on memory regions, not HVA ranges.
> > > >
> > > > Paolo
> > >
> > > Many listeners care about HVA ranges. I know KVM and vhost do.
> > I'm not sure about KVM, it works just fine with fragmented memory
> > regions, the same will apply to vhost once module parameter to
> > increase limit is merged.
> >
> > but changing generic memory listener interface to replace HVA mapped
> > regions with HVA container would lead to a case when listeners
> > won't see exact layout that they might need.
>
> I don't think they care, really.
>
> > In addition vhost itself will suffer from working with big HVA
> > since it allocates log depending on size of memory => bigger log.
>
> Not really - it allocates the log depending on the PA range.
> Leaving unused holes doesn't reduce it's size.
if it would use HVA container instead then it will always allocate
log for max possible GPA, meaning that -m 1024,maxmem=1T will waste
a lot of memory and more so for bigger maxmem.
It's still possible to induce worst case by plugging pc-dimm at the end
of hotplug-memory area by specifying address for it explicitly.
That problem exists since memory hot-add was introduced, I've just
haven't noticed it back then.
It's perfectly fine to allocate log by last GPA as far as
memory is nearly continuous but memory hot-add makes it possible to
have sparse layout with a huge gaps between guest mapped RAM
which makes current log handling inefficient.
I wonder how hard it would be to make log_size depend on present RAM
size rather than max present GPA so it wouldn't allocate excess
memory for log.
>
>
> > That's one of the reasons that in this patch HVA ranges in
> > memory map are compacted only for backend consumption,
> > QEMU's side of vhost uses exact map for internal purposes.
> > And the other reason is I don't know vhost enough to rewrite it
> > to use big HVA for everything.
> >
> > > I guess we could create dummy MRs to fill in the holes left by
> > > memory hotplug?
> > it looks like nice thing from vhost pov but complicates other side,
>
> What other side do you have in mind?
>
> > hence I dislike an idea inventing dummy MRs for vhost's convenience.
memory core, but lets see what Paolo thinks about it.
> >
> >
> > > vhost already has logic to recombine
> > > consequitive chunks created by memory core.
> > which looks a bit complicated and I was thinking about simplifying
> > it some time in the future.
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-13 18:55 ` Igor Mammedov
@ 2015-07-13 20:14 ` Michael S. Tsirkin
2015-07-14 13:02 ` Igor Mammedov
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-13 20:14 UTC (permalink / raw)
To: Igor Mammedov; +Cc: Paolo Bonzini, qemu-devel
On Mon, Jul 13, 2015 at 08:55:13PM +0200, Igor Mammedov wrote:
> On Mon, 13 Jul 2015 09:55:18 +0300
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Fri, Jul 10, 2015 at 12:12:36PM +0200, Igor Mammedov wrote:
> > > On Thu, 9 Jul 2015 16:46:43 +0300
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >
> > > > On Thu, Jul 09, 2015 at 03:43:01PM +0200, Paolo Bonzini wrote:
> > > > >
> > > > >
> > > > > On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > > > > > > QEMU asserts in vhost due to hitting vhost backend limit
> > > > > > > on number of supported memory regions.
> > > > > > >
> > > > > > > Describe all hotplugged memory as one continuos range
> > > > > > > to vhost with linear 1:1 HVA->GPA mapping in backend.
> > > > > > >
> > > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > >
> > > > > > Hmm - a bunch of work here to recombine MRs that memory
> > > > > > listener interface breaks up. In particular KVM could
> > > > > > benefit from this too (on workloads that change the table a
> > > > > > lot). Can't we teach memory core to pass hva range as a
> > > > > > single continuous range to memory listeners?
> > > > >
> > > > > Memory listeners are based on memory regions, not HVA ranges.
> > > > >
> > > > > Paolo
> > > >
> > > > Many listeners care about HVA ranges. I know KVM and vhost do.
> > > I'm not sure about KVM, it works just fine with fragmented memory
> > > regions, the same will apply to vhost once module parameter to
> > > increase limit is merged.
> > >
> > > but changing generic memory listener interface to replace HVA mapped
> > > regions with HVA container would lead to a case when listeners
> > > won't see exact layout that they might need.
> >
> > I don't think they care, really.
> >
> > > In addition vhost itself will suffer from working with big HVA
> > > since it allocates log depending on size of memory => bigger log.
> >
> > Not really - it allocates the log depending on the PA range.
> > Leaving unused holes doesn't reduce it's size.
> if it would use HVA container instead then it will always allocate
> log for max possible GPA, meaning that -m 1024,maxmem=1T will waste
> a lot of memory and more so for bigger maxmem.
> It's still possible to induce worst case by plugging pc-dimm at the end
> of hotplug-memory area by specifying address for it explicitly.
> That problem exists since memory hot-add was introduced, I've just
> haven't noticed it back then.
There you are then. Depending on maxmem seems cleaner as it's more
predictable.
> It's perfectly fine to allocate log by last GPA as far as
> memory is nearly continuous but memory hot-add makes it possible to
> have sparse layout with a huge gaps between guest mapped RAM
> which makes current log handling inefficient.
>
> I wonder how hard it would be to make log_size depend on present RAM
> size rather than max present GPA so it wouldn't allocate excess
> memory for log.
We can simply map the unused parts of the log RESERVED.
That can be a natural continuation of these series, but
I don't think it needs to block it.
>
> >
> >
> > > That's one of the reasons that in this patch HVA ranges in
> > > memory map are compacted only for backend consumption,
> > > QEMU's side of vhost uses exact map for internal purposes.
> > > And the other reason is I don't know vhost enough to rewrite it
> > > to use big HVA for everything.
> > >
> > > > I guess we could create dummy MRs to fill in the holes left by
> > > > memory hotplug?
> > > it looks like nice thing from vhost pov but complicates other side,
> >
> > What other side do you have in mind?
> >
> > > hence I dislike an idea inventing dummy MRs for vhost's convenience.
> memory core, but lets see what Paolo thinks about it.
>
> > >
> > >
> > > > vhost already has logic to recombine
> > > > consequitive chunks created by memory core.
> > > which looks a bit complicated and I was thinking about simplifying
> > > it some time in the future.
> >
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-13 20:14 ` Michael S. Tsirkin
@ 2015-07-14 13:02 ` Igor Mammedov
2015-07-14 13:14 ` Michael S. Tsirkin
0 siblings, 1 reply; 24+ messages in thread
From: Igor Mammedov @ 2015-07-14 13:02 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Paolo Bonzini, qemu-devel
On Mon, 13 Jul 2015 23:14:37 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Jul 13, 2015 at 08:55:13PM +0200, Igor Mammedov wrote:
> > On Mon, 13 Jul 2015 09:55:18 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Fri, Jul 10, 2015 at 12:12:36PM +0200, Igor Mammedov wrote:
> > > > On Thu, 9 Jul 2015 16:46:43 +0300
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > >
> > > > > On Thu, Jul 09, 2015 at 03:43:01PM +0200, Paolo Bonzini wrote:
> > > > > >
> > > > > >
> > > > > > On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > > > > > > > QEMU asserts in vhost due to hitting vhost backend limit
> > > > > > > > on number of supported memory regions.
> > > > > > > >
> > > > > > > > Describe all hotplugged memory as one continuos range
> > > > > > > > to vhost with linear 1:1 HVA->GPA mapping in backend.
> > > > > > > >
> > > > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > > >
> > > > > > > Hmm - a bunch of work here to recombine MRs that memory
> > > > > > > listener interface breaks up. In particular KVM could
> > > > > > > benefit from this too (on workloads that change the table a
> > > > > > > lot). Can't we teach memory core to pass hva range as a
> > > > > > > single continuous range to memory listeners?
> > > > > >
> > > > > > Memory listeners are based on memory regions, not HVA ranges.
> > > > > >
> > > > > > Paolo
> > > > >
> > > > > Many listeners care about HVA ranges. I know KVM and vhost do.
> > > > I'm not sure about KVM, it works just fine with fragmented memory
> > > > regions, the same will apply to vhost once module parameter to
> > > > increase limit is merged.
> > > >
> > > > but changing generic memory listener interface to replace HVA mapped
> > > > regions with HVA container would lead to a case when listeners
> > > > won't see exact layout that they might need.
> > >
> > > I don't think they care, really.
> > >
> > > > In addition vhost itself will suffer from working with big HVA
> > > > since it allocates log depending on size of memory => bigger log.
> > >
> > > Not really - it allocates the log depending on the PA range.
> > > Leaving unused holes doesn't reduce it's size.
> > if it would use HVA container instead then it will always allocate
> > log for max possible GPA, meaning that -m 1024,maxmem=1T will waste
> > a lot of memory and more so for bigger maxmem.
> > It's still possible to induce worst case by plugging pc-dimm at the end
> > of hotplug-memory area by specifying address for it explicitly.
> > That problem exists since memory hot-add was introduced, I've just
> > haven't noticed it back then.
>
> There you are then. Depending on maxmem seems cleaner as it's more
> predictable.
>
> > It's perfectly fine to allocate log by last GPA as far as
> > memory is nearly continuous but memory hot-add makes it possible to
> > have sparse layout with a huge gaps between guest mapped RAM
> > which makes current log handling inefficient.
> >
> > I wonder how hard it would be to make log_size depend on present RAM
> > size rather than max present GPA so it wouldn't allocate excess
> > memory for log.
>
> We can simply map the unused parts of the log RESERVED.
meaning that vhost listener should get RAM regions so it would know
which parts of log it has to mmap(NORESERVE|DONTNEED)
it would also require custom allocator for log, that could manage
punching/unpunching holes in log depending on RAM layout.
btw is it possible for guest to force vhost module access
NORESERVE area and what would happen it that case?
>
> That can be a natural continuation of these series, but
> I don't think it needs to block it.
>
> >
> > >
> > >
> > > > That's one of the reasons that in this patch HVA ranges in
> > > > memory map are compacted only for backend consumption,
> > > > QEMU's side of vhost uses exact map for internal purposes.
> > > > And the other reason is I don't know vhost enough to rewrite it
> > > > to use big HVA for everything.
> > > >
> > > > > I guess we could create dummy MRs to fill in the holes left by
> > > > > memory hotplug?
> > > > it looks like nice thing from vhost pov but complicates other side,
> > >
> > > What other side do you have in mind?
> > >
> > > > hence I dislike an idea inventing dummy MRs for vhost's convenience.
> > memory core, but lets see what Paolo thinks about it.
> >
> > > >
> > > >
> > > > > vhost already has logic to recombine
> > > > > consequitive chunks created by memory core.
> > > > which looks a bit complicated and I was thinking about simplifying
> > > > it some time in the future.
> > >
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged
2015-07-14 13:02 ` Igor Mammedov
@ 2015-07-14 13:14 ` Michael S. Tsirkin
0 siblings, 0 replies; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-14 13:14 UTC (permalink / raw)
To: Igor Mammedov; +Cc: Paolo Bonzini, qemu-devel
On Tue, Jul 14, 2015 at 03:02:44PM +0200, Igor Mammedov wrote:
> On Mon, 13 Jul 2015 23:14:37 +0300
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Mon, Jul 13, 2015 at 08:55:13PM +0200, Igor Mammedov wrote:
> > > On Mon, 13 Jul 2015 09:55:18 +0300
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >
> > > > On Fri, Jul 10, 2015 at 12:12:36PM +0200, Igor Mammedov wrote:
> > > > > On Thu, 9 Jul 2015 16:46:43 +0300
> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > >
> > > > > > On Thu, Jul 09, 2015 at 03:43:01PM +0200, Paolo Bonzini wrote:
> > > > > > >
> > > > > > >
> > > > > > > On 09/07/2015 15:06, Michael S. Tsirkin wrote:
> > > > > > > > > QEMU asserts in vhost due to hitting vhost backend limit
> > > > > > > > > on number of supported memory regions.
> > > > > > > > >
> > > > > > > > > Describe all hotplugged memory as one continuos range
> > > > > > > > > to vhost with linear 1:1 HVA->GPA mapping in backend.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > > > >
> > > > > > > > Hmm - a bunch of work here to recombine MRs that memory
> > > > > > > > listener interface breaks up. In particular KVM could
> > > > > > > > benefit from this too (on workloads that change the table a
> > > > > > > > lot). Can't we teach memory core to pass hva range as a
> > > > > > > > single continuous range to memory listeners?
> > > > > > >
> > > > > > > Memory listeners are based on memory regions, not HVA ranges.
> > > > > > >
> > > > > > > Paolo
> > > > > >
> > > > > > Many listeners care about HVA ranges. I know KVM and vhost do.
> > > > > I'm not sure about KVM, it works just fine with fragmented memory
> > > > > regions, the same will apply to vhost once module parameter to
> > > > > increase limit is merged.
> > > > >
> > > > > but changing generic memory listener interface to replace HVA mapped
> > > > > regions with HVA container would lead to a case when listeners
> > > > > won't see exact layout that they might need.
> > > >
> > > > I don't think they care, really.
> > > >
> > > > > In addition vhost itself will suffer from working with big HVA
> > > > > since it allocates log depending on size of memory => bigger log.
> > > >
> > > > Not really - it allocates the log depending on the PA range.
> > > > Leaving unused holes doesn't reduce it's size.
> > > if it would use HVA container instead then it will always allocate
> > > log for max possible GPA, meaning that -m 1024,maxmem=1T will waste
> > > a lot of memory and more so for bigger maxmem.
> > > It's still possible to induce worst case by plugging pc-dimm at the end
> > > of hotplug-memory area by specifying address for it explicitly.
> > > That problem exists since memory hot-add was introduced, I've just
> > > haven't noticed it back then.
> >
> > There you are then. Depending on maxmem seems cleaner as it's more
> > predictable.
> >
> > > It's perfectly fine to allocate log by last GPA as far as
> > > memory is nearly continuous but memory hot-add makes it possible to
> > > have sparse layout with a huge gaps between guest mapped RAM
> > > which makes current log handling inefficient.
> > >
> > > I wonder how hard it would be to make log_size depend on present RAM
> > > size rather than max present GPA so it wouldn't allocate excess
> > > memory for log.
> >
> > We can simply map the unused parts of the log RESERVED.
> meaning that vhost listener should get RAM regions so it would know
> which parts of log it has to mmap(NORESERVE|DONTNEED)
>
> it would also require custom allocator for log, that could manage
> punching/unpunching holes in log depending on RAM layout.
Yea. Anyway, this isn't urgent I think.
> btw is it possible for guest to force vhost module access
> NORESERVE area and what would happen it that case?
Sure. I think you'll get EFAULT, vhost will stop processing the ring then.
>
> >
> > That can be a natural continuation of these series, but
> > I don't think it needs to block it.
> >
> > >
> > > >
> > > >
> > > > > That's one of the reasons that in this patch HVA ranges in
> > > > > memory map are compacted only for backend consumption,
> > > > > QEMU's side of vhost uses exact map for internal purposes.
> > > > > And the other reason is I don't know vhost enough to rewrite it
> > > > > to use big HVA for everything.
> > > > >
> > > > > > I guess we could create dummy MRs to fill in the holes left by
> > > > > > memory hotplug?
> > > > > it looks like nice thing from vhost pov but complicates other side,
> > > >
> > > > What other side do you have in mind?
> > > >
> > > > > hence I dislike an idea inventing dummy MRs for vhost's convenience.
> > > memory core, but lets see what Paolo thinks about it.
> > >
> > > > >
> > > > >
> > > > > > vhost already has logic to recombine
> > > > > > consequitive chunks created by memory core.
> > > > > which looks a bit complicated and I was thinking about simplifying
> > > > > it some time in the future.
> > > >
> >
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
` (6 preceding siblings ...)
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 7/7] memory: add support for deleting HVA mapped MemoryRegion Igor Mammedov
@ 2015-07-15 15:12 ` Igor Mammedov
2015-07-15 16:32 ` Michael S. Tsirkin
7 siblings, 1 reply; 24+ messages in thread
From: Igor Mammedov @ 2015-07-15 15:12 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, pbonzini, mst
On Thu, 9 Jul 2015 13:47:17 +0200
Igor Mammedov <imammedo@redhat.com> wrote:
there also is yet another issue with vhost-user. It also has
very low limit on amount of memory regions (if I recall correctly 8)
and it's possible to trigger even without memory hotplug.
one just need to start QEMU with a several -numa memdev= options
to create a necessary amount of memory regions to trigger it.
lowrisk option to fix it would be increasing limit in vhost-user
backend.
another option is disabling vhost and fall-back to virtio,
but I don't know much about vhost if it's possible to
to switch it off without loosing packets guest was sending
at the moment and if it will work at all with vhost.
> Changelog:
> v3->v4:
> * drop patch extending memory_region_subregion_add()
> with error argument
> * and add memory_region_add_subregion_to_hva() API instead
> * add madvise(DONTNEED) when returning range to HVA container
> v2->v3:
> * fixed(work-arouned) unmapping issues,
> now memory subsytem keeps track of HVA mapped
> regions and doesn't allow to map a new region
> at address where previos has benn mapped until
> previous region is gone
> * fixed offset calculations in memory_region_find_hva_range()
> in 2/8
> * redone MemorySection folding into HVA range for VHOST,
> now compacted memory map is temporary and passed only to vhost
> backend and doesn't touch original memory map used by QEMU
> v1->v2:
> * take into account Paolo's review comments
> * do not overload ram_addr
> * ifdef linux specific code
> * reseve HVA using API from exec.c instead of calling
> mmap() dircely from memory.c
> * support unmapping of HVA remapped region
>
> When more than ~50 pc-dimm devices are hotplugged with
> vhost enabled, QEMU will assert in vhost vhost_commit()
> due to backend refusing to accept too many memory ranges.
>
> Series introduces Reserved HVA MemoryRegion container
> where to all hotplugged memory is remapped and passes
> the single container range to vhost instead of multiple
> memory ranges for each hotlugged pc-dimm device.
>
> It's an alternative approach to increasing backend supported
> memory regions limit.
>
> Tested it a bit more, so now
> - migration from current master to patched version seems to work
> - memory is returned to host after device_del+object_del sequence,
> but I can't bet if cgroups won't charge it.
>
> git branch for testing:
> https://github.com/imammedo/qemu/commits/vhost_one_hp_range_v4
>
>
> Igor Mammedov (7):
> memory: get rid of memory_region_destructor_ram_from_ptr()
> memory: introduce MemoryRegion container with reserved HVA range
> pc: reserve hotpluggable memory range with
> memory_region_init_hva_range()
> pc: fix QEMU crashing when more than ~50 memory hotplugged
> exec: make sure that RAMBlock descriptor won't be leaked
> exec: add qemu_ram_unmap_hva() API for unmapping memory from HVA area
> memory: add support for deleting HVA mapped MemoryRegion
>
> exec.c | 71 +++++++++++++++++++----------
> hw/i386/pc.c | 4 +-
> hw/mem/pc-dimm.c | 6 ++-
> hw/virtio/vhost.c | 47 ++++++++++++++++++--
> include/exec/cpu-common.h | 3 ++
> include/exec/memory.h | 67 +++++++++++++++++++++++++++-
> include/exec/ram_addr.h | 1 -
> include/hw/virtio/vhost.h | 1 +
> memory.c | 111 +++++++++++++++++++++++++++++++++++++++++++---
> 9 files changed, 272 insertions(+), 39 deletions(-)
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
2015-07-15 15:12 ` [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
@ 2015-07-15 16:32 ` Michael S. Tsirkin
2015-07-16 7:26 ` Igor Mammedov
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-15 16:32 UTC (permalink / raw)
To: Igor Mammedov; +Cc: pbonzini, qemu-devel, peter.maydell
On Wed, Jul 15, 2015 at 05:12:01PM +0200, Igor Mammedov wrote:
> On Thu, 9 Jul 2015 13:47:17 +0200
> Igor Mammedov <imammedo@redhat.com> wrote:
>
> there also is yet another issue with vhost-user. It also has
> very low limit on amount of memory regions (if I recall correctly 8)
> and it's possible to trigger even without memory hotplug.
> one just need to start QEMU with a several -numa memdev= options
> to create a necessary amount of memory regions to trigger it.
>
> lowrisk option to fix it would be increasing limit in vhost-user
> backend.
>
> another option is disabling vhost and fall-back to virtio,
> but I don't know much about vhost if it's possible to
> to switch it off without loosing packets guest was sending
> at the moment and if it will work at all with vhost.
With vhost-user you can't fall back to virtio: it's
not an accelerator, it's the backend.
Updating the protocol to support a bigger table
is possible but old remotes won't be able to support it.
--
MST
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
2015-07-15 16:32 ` Michael S. Tsirkin
@ 2015-07-16 7:26 ` Igor Mammedov
2015-07-16 7:35 ` Michael S. Tsirkin
0 siblings, 1 reply; 24+ messages in thread
From: Igor Mammedov @ 2015-07-16 7:26 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: pbonzini, qemu-devel, peter.maydell
On Wed, 15 Jul 2015 19:32:31 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Wed, Jul 15, 2015 at 05:12:01PM +0200, Igor Mammedov wrote:
> > On Thu, 9 Jul 2015 13:47:17 +0200
> > Igor Mammedov <imammedo@redhat.com> wrote:
> >
> > there also is yet another issue with vhost-user. It also has
> > very low limit on amount of memory regions (if I recall correctly 8)
> > and it's possible to trigger even without memory hotplug.
> > one just need to start QEMU with a several -numa memdev= options
> > to create a necessary amount of memory regions to trigger it.
> >
> > lowrisk option to fix it would be increasing limit in vhost-user
> > backend.
> >
> > another option is disabling vhost and fall-back to virtio,
> > but I don't know much about vhost if it's possible to
> > to switch it off without loosing packets guest was sending
> > at the moment and if it will work at all with vhost.
>
> With vhost-user you can't fall back to virtio: it's
> not an accelerator, it's the backend.
>
> Updating the protocol to support a bigger table
> is possible but old remotes won't be able to support it.
>
it looks like increasing limit is the only option left.
it's not ideal that old remotes /with hardcoded limit/
might not be able to handle bigger table but at least
new ones and ones that handle VhostUserMsg payload
dynamically would be able to work without crashing.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
2015-07-16 7:26 ` Igor Mammedov
@ 2015-07-16 7:35 ` Michael S. Tsirkin
2015-07-16 9:42 ` Igor Mammedov
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-16 7:35 UTC (permalink / raw)
To: Igor Mammedov; +Cc: pbonzini, qemu-devel, peter.maydell
On Thu, Jul 16, 2015 at 09:26:21AM +0200, Igor Mammedov wrote:
> On Wed, 15 Jul 2015 19:32:31 +0300
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Wed, Jul 15, 2015 at 05:12:01PM +0200, Igor Mammedov wrote:
> > > On Thu, 9 Jul 2015 13:47:17 +0200
> > > Igor Mammedov <imammedo@redhat.com> wrote:
> > >
> > > there also is yet another issue with vhost-user. It also has
> > > very low limit on amount of memory regions (if I recall correctly 8)
> > > and it's possible to trigger even without memory hotplug.
> > > one just need to start QEMU with a several -numa memdev= options
> > > to create a necessary amount of memory regions to trigger it.
> > >
> > > lowrisk option to fix it would be increasing limit in vhost-user
> > > backend.
> > >
> > > another option is disabling vhost and fall-back to virtio,
> > > but I don't know much about vhost if it's possible to
> > > to switch it off without loosing packets guest was sending
> > > at the moment and if it will work at all with vhost.
> >
> > With vhost-user you can't fall back to virtio: it's
> > not an accelerator, it's the backend.
> >
> > Updating the protocol to support a bigger table
> > is possible but old remotes won't be able to support it.
> >
> it looks like increasing limit is the only option left.
>
> it's not ideal that old remotes /with hardcoded limit/
> might not be able to handle bigger table but at least
> new ones and ones that handle VhostUserMsg payload
> dynamically would be able to work without crashing.
I think we need a way for hotplug to fail gracefully. As long as we
don't implement the hva trick, it's needed for old kernels with vhost in
kernel, too.
--
MST
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
2015-07-16 7:35 ` Michael S. Tsirkin
@ 2015-07-16 9:42 ` Igor Mammedov
2015-07-16 10:24 ` Michael S. Tsirkin
0 siblings, 1 reply; 24+ messages in thread
From: Igor Mammedov @ 2015-07-16 9:42 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: pbonzini, qemu-devel, peter.maydell
On Thu, 16 Jul 2015 10:35:33 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Jul 16, 2015 at 09:26:21AM +0200, Igor Mammedov wrote:
> > On Wed, 15 Jul 2015 19:32:31 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Wed, Jul 15, 2015 at 05:12:01PM +0200, Igor Mammedov wrote:
> > > > On Thu, 9 Jul 2015 13:47:17 +0200
> > > > Igor Mammedov <imammedo@redhat.com> wrote:
> > > >
> > > > there also is yet another issue with vhost-user. It also has
> > > > very low limit on amount of memory regions (if I recall correctly 8)
> > > > and it's possible to trigger even without memory hotplug.
> > > > one just need to start QEMU with a several -numa memdev= options
> > > > to create a necessary amount of memory regions to trigger it.
> > > >
> > > > lowrisk option to fix it would be increasing limit in vhost-user
> > > > backend.
> > > >
> > > > another option is disabling vhost and fall-back to virtio,
> > > > but I don't know much about vhost if it's possible to
> > > > to switch it off without loosing packets guest was sending
> > > > at the moment and if it will work at all with vhost.
> > >
> > > With vhost-user you can't fall back to virtio: it's
> > > not an accelerator, it's the backend.
> > >
> > > Updating the protocol to support a bigger table
> > > is possible but old remotes won't be able to support it.
> > >
> > it looks like increasing limit is the only option left.
> >
> > it's not ideal that old remotes /with hardcoded limit/
> > might not be able to handle bigger table but at least
> > new ones and ones that handle VhostUserMsg payload
> > dynamically would be able to work without crashing.
>
> I think we need a way for hotplug to fail gracefully. As long as we
> don't implement the hva trick, it's needed for old kernels with vhost in
> kernel, too.
I don't see a reliable way to fail hotplug though.
In case of hotplug failure path comes from memory listener
which can't fail by design but it fails in vhost case, i.e.
vhost side doesn't follow protocol.
We already have considered idea of querying vhost, for limit
from memory hotplug handler before mapping memory region
but it has drawbacks:
1. amount of memory ranges changes during guest lifecycle
as it initializes different devices.
which leads to a case when we can hotplug more pc-dimms
than cold-plug.
Which leads to inability to migrate guest with hotplugged
pc-dimms since target QEMU won't start with that amount
of dimms from source due to hitting limit.
2. it's ugly hack to query random 'vhost' entity when plugging
dimm device from modeling pov, but we can live with it
if it helps QEMU not to crash.
If it's acceptable to break/ignore #1 issue, I can post related
QEMU patches that I have, at least qemu won't crash with old
vhost backends.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
2015-07-16 9:42 ` Igor Mammedov
@ 2015-07-16 10:24 ` Michael S. Tsirkin
2015-07-16 11:11 ` Igor Mammedov
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2015-07-16 10:24 UTC (permalink / raw)
To: Igor Mammedov; +Cc: pbonzini, qemu-devel, peter.maydell
On Thu, Jul 16, 2015 at 11:42:36AM +0200, Igor Mammedov wrote:
> On Thu, 16 Jul 2015 10:35:33 +0300
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Thu, Jul 16, 2015 at 09:26:21AM +0200, Igor Mammedov wrote:
> > > On Wed, 15 Jul 2015 19:32:31 +0300
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >
> > > > On Wed, Jul 15, 2015 at 05:12:01PM +0200, Igor Mammedov wrote:
> > > > > On Thu, 9 Jul 2015 13:47:17 +0200
> > > > > Igor Mammedov <imammedo@redhat.com> wrote:
> > > > >
> > > > > there also is yet another issue with vhost-user. It also has
> > > > > very low limit on amount of memory regions (if I recall correctly 8)
> > > > > and it's possible to trigger even without memory hotplug.
> > > > > one just need to start QEMU with a several -numa memdev= options
> > > > > to create a necessary amount of memory regions to trigger it.
> > > > >
> > > > > lowrisk option to fix it would be increasing limit in vhost-user
> > > > > backend.
> > > > >
> > > > > another option is disabling vhost and fall-back to virtio,
> > > > > but I don't know much about vhost if it's possible to
> > > > > to switch it off without loosing packets guest was sending
> > > > > at the moment and if it will work at all with vhost.
> > > >
> > > > With vhost-user you can't fall back to virtio: it's
> > > > not an accelerator, it's the backend.
> > > >
> > > > Updating the protocol to support a bigger table
> > > > is possible but old remotes won't be able to support it.
> > > >
> > > it looks like increasing limit is the only option left.
> > >
> > > it's not ideal that old remotes /with hardcoded limit/
> > > might not be able to handle bigger table but at least
> > > new ones and ones that handle VhostUserMsg payload
> > > dynamically would be able to work without crashing.
> >
> > I think we need a way for hotplug to fail gracefully. As long as we
> > don't implement the hva trick, it's needed for old kernels with vhost in
> > kernel, too.
> I don't see a reliable way to fail hotplug though.
>
> In case of hotplug failure path comes from memory listener
> which can't fail by design but it fails in vhost case, i.e.
> vhost side doesn't follow protocol.
>
> We already have considered idea of querying vhost, for limit
> from memory hotplug handler before mapping memory region
> but it has drawbacks:
> 1. amount of memory ranges changes during guest lifecycle
> as it initializes different devices.
> which leads to a case when we can hotplug more pc-dimms
> than cold-plug.
> Which leads to inability to migrate guest with hotplugged
> pc-dimms since target QEMU won't start with that amount
> of dimms from source due to hitting limit.
> 2. it's ugly hack to query random 'vhost' entity when plugging
> dimm device from modeling pov, but we can live with it
> if it helps QEMU not to crash.
>
> If it's acceptable to break/ignore #1 issue, I can post related
> QEMU patches that I have, at least qemu won't crash with old
> vhost backends.
Old kvm has lower limits on 3 of slots as well. How is this handled?
--
MST
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on
2015-07-16 10:24 ` Michael S. Tsirkin
@ 2015-07-16 11:11 ` Igor Mammedov
0 siblings, 0 replies; 24+ messages in thread
From: Igor Mammedov @ 2015-07-16 11:11 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: pbonzini, qemu-devel, peter.maydell
On Thu, 16 Jul 2015 13:24:35 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Jul 16, 2015 at 11:42:36AM +0200, Igor Mammedov wrote:
> > On Thu, 16 Jul 2015 10:35:33 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Thu, Jul 16, 2015 at 09:26:21AM +0200, Igor Mammedov wrote:
> > > > On Wed, 15 Jul 2015 19:32:31 +0300
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > >
> > > > > On Wed, Jul 15, 2015 at 05:12:01PM +0200, Igor Mammedov wrote:
> > > > > > On Thu, 9 Jul 2015 13:47:17 +0200
> > > > > > Igor Mammedov <imammedo@redhat.com> wrote:
> > > > > >
> > > > > > there also is yet another issue with vhost-user. It also has
> > > > > > very low limit on amount of memory regions (if I recall correctly 8)
> > > > > > and it's possible to trigger even without memory hotplug.
> > > > > > one just need to start QEMU with a several -numa memdev= options
> > > > > > to create a necessary amount of memory regions to trigger it.
> > > > > >
> > > > > > lowrisk option to fix it would be increasing limit in vhost-user
> > > > > > backend.
> > > > > >
> > > > > > another option is disabling vhost and fall-back to virtio,
> > > > > > but I don't know much about vhost if it's possible to
> > > > > > to switch it off without loosing packets guest was sending
> > > > > > at the moment and if it will work at all with vhost.
> > > > >
> > > > > With vhost-user you can't fall back to virtio: it's
> > > > > not an accelerator, it's the backend.
> > > > >
> > > > > Updating the protocol to support a bigger table
> > > > > is possible but old remotes won't be able to support it.
> > > > >
> > > > it looks like increasing limit is the only option left.
> > > >
> > > > it's not ideal that old remotes /with hardcoded limit/
> > > > might not be able to handle bigger table but at least
> > > > new ones and ones that handle VhostUserMsg payload
> > > > dynamically would be able to work without crashing.
> > >
> > > I think we need a way for hotplug to fail gracefully. As long as we
> > > don't implement the hva trick, it's needed for old kernels with vhost in
> > > kernel, too.
> > I don't see a reliable way to fail hotplug though.
> >
> > In case of hotplug failure path comes from memory listener
> > which can't fail by design but it fails in vhost case, i.e.
> > vhost side doesn't follow protocol.
> >
> > We already have considered idea of querying vhost, for limit
> > from memory hotplug handler before mapping memory region
> > but it has drawbacks:
> > 1. amount of memory ranges changes during guest lifecycle
> > as it initializes different devices.
> > which leads to a case when we can hotplug more pc-dimms
> > than cold-plug.
> > Which leads to inability to migrate guest with hotplugged
> > pc-dimms since target QEMU won't start with that amount
> > of dimms from source due to hitting limit.
> > 2. it's ugly hack to query random 'vhost' entity when plugging
> > dimm device from modeling pov, but we can live with it
> > if it helps QEMU not to crash.
> >
> > If it's acceptable to break/ignore #1 issue, I can post related
> > QEMU patches that I have, at least qemu won't crash with old
> > vhost backends.
>
> Old kvm has lower limits on 3 of slots as well. How is this handled?
the same ugly/non perfect way,
pc_dimm_plug() - > pc_dimm_memory_plug() {
...
if (kvm_enabled() && !kvm_has_free_slot(machine)) {
error_setg(&local_err, "hypervisor has no free memory slots left");
...
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2015-07-16 11:11 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-07-09 11:47 [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 1/7] memory: get rid of memory_region_destructor_ram_from_ptr() Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 2/7] memory: introduce MemoryRegion container with reserved HVA range Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 3/7] pc: reserve hotpluggable memory range with memory_region_init_hva_range() Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 4/7] pc: fix QEMU crashing when more than ~50 memory hotplugged Igor Mammedov
2015-07-09 13:06 ` Michael S. Tsirkin
2015-07-09 13:43 ` Paolo Bonzini
2015-07-09 13:46 ` Michael S. Tsirkin
2015-07-10 10:12 ` Igor Mammedov
2015-07-13 6:55 ` Michael S. Tsirkin
2015-07-13 18:55 ` Igor Mammedov
2015-07-13 20:14 ` Michael S. Tsirkin
2015-07-14 13:02 ` Igor Mammedov
2015-07-14 13:14 ` Michael S. Tsirkin
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 5/7] exec: make sure that RAMBlock descriptor won't be leaked Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 6/7] exec: add qemu_ram_unmap_hva() API for unmapping memory from HVA area Igor Mammedov
2015-07-09 11:47 ` [Qemu-devel] [PATCH v4 7/7] memory: add support for deleting HVA mapped MemoryRegion Igor Mammedov
2015-07-15 15:12 ` [Qemu-devel] [PATCH v4 0/7] Fix QEMU crash during memory hotplug with vhost=on Igor Mammedov
2015-07-15 16:32 ` Michael S. Tsirkin
2015-07-16 7:26 ` Igor Mammedov
2015-07-16 7:35 ` Michael S. Tsirkin
2015-07-16 9:42 ` Igor Mammedov
2015-07-16 10:24 ` Michael S. Tsirkin
2015-07-16 11:11 ` Igor Mammedov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.