* [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend
@ 2017-06-21 14:02 Marc-André Lureau
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 1/3] memfd: split qemu_memfd_alloc() Marc-André Lureau
` (3 more replies)
0 siblings, 4 replies; 11+ messages in thread
From: Marc-André Lureau @ 2017-06-21 14:02 UTC (permalink / raw
To: qemu-devel; +Cc: imammedo, ehabkost, Marc-André Lureau
Add a new Linux-specific memory backend, similar to hostmem-file,
except that it doesn't need file path. It also try to enforce memory
sealing if available. It is thus slightly easier and secure, and is
compatible with transparent huge-pages since Linux 4.8.
v4:
- rebased, now that preliminary patches are merged
v3:
- make vhost-user-test use memfd only if possible
- rebased
v1->v2:
- make it linux-specific
- minor changes and commit message tweaks
Marc-André Lureau (3):
memfd: split qemu_memfd_alloc()
Add memfd based hostmem
tests: use memfd in vhost-user-test
include/qemu/memfd.h | 2 ++
backends/hostmem-memfd.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++
tests/vhost-user-test.c | 68 +++++++++++++++++++++++++++++++-----------------
util/memfd.c | 42 ++++++++++++++++++------------
backends/Makefile.objs | 2 ++
qemu-options.hx | 11 ++++++++
6 files changed, 151 insertions(+), 41 deletions(-)
create mode 100644 backends/hostmem-memfd.c
--
2.13.1.395.gf7b71de06
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Qemu-devel] [PATCH v4 1/3] memfd: split qemu_memfd_alloc()
2017-06-21 14:02 [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend Marc-André Lureau
@ 2017-06-21 14:02 ` Marc-André Lureau
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem Marc-André Lureau
` (2 subsequent siblings)
3 siblings, 0 replies; 11+ messages in thread
From: Marc-André Lureau @ 2017-06-21 14:02 UTC (permalink / raw
To: qemu-devel; +Cc: imammedo, ehabkost, Marc-André Lureau
Add a function to only create a memfd, without mmap. The function is
used in the following memory backend.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
include/qemu/memfd.h | 2 ++
util/memfd.c | 42 +++++++++++++++++++++++++-----------------
2 files changed, 27 insertions(+), 17 deletions(-)
diff --git a/include/qemu/memfd.h b/include/qemu/memfd.h
index 745a8c501e..30c1ab1d91 100644
--- a/include/qemu/memfd.h
+++ b/include/qemu/memfd.h
@@ -16,6 +16,8 @@
#define F_SEAL_WRITE 0x0008 /* prevent writes */
#endif
+int qemu_memfd_create(const char *name, size_t size, unsigned int seals,
+ bool must_seal);
void *qemu_memfd_alloc(const char *name, size_t size, unsigned int seals,
int *fd);
void qemu_memfd_free(void *ptr, size_t size, int fd);
diff --git a/util/memfd.c b/util/memfd.c
index 4571d1aba8..e6476df083 100644
--- a/util/memfd.c
+++ b/util/memfd.c
@@ -55,45 +55,53 @@ static int memfd_create(const char *name, unsigned int flags)
#define MFD_ALLOW_SEALING 0x0002U
#endif
-/*
- * This is a best-effort helper for shared memory allocation, with
- * optional sealing. The helper will do his best to allocate using
- * memfd with sealing, but may fallback on other methods without
- * sealing.
- */
-void *qemu_memfd_alloc(const char *name, size_t size, unsigned int seals,
- int *fd)
+int qemu_memfd_create(const char *name, size_t size, unsigned int seals,
+ bool must_seal)
{
- void *ptr;
int mfd = -1;
- *fd = -1;
-
#ifdef CONFIG_LINUX
if (seals) {
mfd = memfd_create(name, MFD_ALLOW_SEALING | MFD_CLOEXEC);
}
- if (mfd == -1) {
+ if (mfd == -1 && !must_seal) {
/* some systems have memfd without sealing */
mfd = memfd_create(name, MFD_CLOEXEC);
seals = 0;
}
-#endif
- if (mfd != -1) {
+ if (mfd >= 0) {
if (ftruncate(mfd, size) == -1) {
perror("ftruncate");
close(mfd);
- return NULL;
+ return -1;
}
if (seals && fcntl(mfd, F_ADD_SEALS, seals) == -1) {
perror("fcntl");
close(mfd);
- return NULL;
+ return -1;
}
- } else {
+ }
+#endif
+
+ return mfd;
+}
+
+/*
+ * This is a best-effort helper for shared memory allocation, with
+ * optional sealing. The helper will do his best to allocate using
+ * memfd with sealing, but may fallback on other methods without
+ * sealing.
+ */
+void *qemu_memfd_alloc(const char *name, size_t size, unsigned int seals,
+ int *fd)
+{
+ void *ptr;
+ int mfd = qemu_memfd_create(name, size, seals, false);
+
+ if (mfd == -1) {
const char *tmpdir = g_get_tmp_dir();
gchar *fname;
--
2.13.1.395.gf7b71de06
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem
2017-06-21 14:02 [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend Marc-André Lureau
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 1/3] memfd: split qemu_memfd_alloc() Marc-André Lureau
@ 2017-06-21 14:02 ` Marc-André Lureau
2017-06-23 21:08 ` Eduardo Habkost
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 3/3] tests: use memfd in vhost-user-test Marc-André Lureau
2017-06-22 9:58 ` [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend David Hildenbrand
3 siblings, 1 reply; 11+ messages in thread
From: Marc-André Lureau @ 2017-06-21 14:02 UTC (permalink / raw
To: qemu-devel; +Cc: imammedo, ehabkost, Marc-André Lureau
Add a new memory backend, similar to hostmem-file, except that it
doesn't need to create files. It also enforces memory sealing.
This backend is mainly useful for sharing the memory with other
processes.
Note that Linux supports transparent huge-pages of shmem/memfd memory
since 4.8. It is relatively easier to set up THP than a dedicate
hugepage mount point by using "madvise" in
/sys/kernel/mm/transparent_hugepage/shmem_enabled.
Usage:
-object memory-backend-memfd,id=mem1,size=1G
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
backends/hostmem-memfd.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++
backends/Makefile.objs | 2 ++
qemu-options.hx | 11 ++++++++
3 files changed, 80 insertions(+)
create mode 100644 backends/hostmem-memfd.c
diff --git a/backends/hostmem-memfd.c b/backends/hostmem-memfd.c
new file mode 100644
index 0000000000..13d300d9ad
--- /dev/null
+++ b/backends/hostmem-memfd.c
@@ -0,0 +1,67 @@
+/*
+ * QEMU host memfd memory backend
+ *
+ * Copyright (C) 2016 Red Hat Inc
+ *
+ * Authors:
+ * Marc-André Lureau <marcandre.lureau@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "sysemu/hostmem.h"
+#include "sysemu/sysemu.h"
+#include "qom/object_interfaces.h"
+#include "qemu/memfd.h"
+#include "qapi/error.h"
+
+#define TYPE_MEMORY_BACKEND_MEMFD "memory-backend-memfd"
+
+static void
+memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
+{
+ int fd;
+
+ if (!backend->size) {
+ error_setg(errp, "can't create backend with size 0");
+ return;
+ }
+
+ if (!memory_region_size(&backend->mr)) {
+ backend->force_prealloc = mem_prealloc;
+ fd = qemu_memfd_create(TYPE_MEMORY_BACKEND_MEMFD,
+ backend->size,
+ F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL,
+ true);
+ if (fd == -1) {
+ error_setg(errp, "can't allocate memfd backend");
+ return;
+ }
+ memory_region_init_ram_from_fd(&backend->mr, OBJECT(backend),
+ object_get_canonical_path(OBJECT(backend)),
+ backend->size, true, fd, errp);
+ }
+}
+
+static void
+memfd_backend_class_init(ObjectClass *oc, void *data)
+{
+ HostMemoryBackendClass *bc = MEMORY_BACKEND_CLASS(oc);
+
+ bc->alloc = memfd_backend_memory_alloc;
+}
+
+static const TypeInfo memfd_backend_info = {
+ .name = TYPE_MEMORY_BACKEND_MEMFD,
+ .parent = TYPE_MEMORY_BACKEND,
+ .class_init = memfd_backend_class_init,
+};
+
+static void register_types(void)
+{
+ type_register_static(&memfd_backend_info);
+}
+
+type_init(register_types);
diff --git a/backends/Makefile.objs b/backends/Makefile.objs
index 0400799efd..67eeeba5fc 100644
--- a/backends/Makefile.objs
+++ b/backends/Makefile.objs
@@ -8,3 +8,5 @@ common-obj-$(CONFIG_LINUX) += hostmem-file.o
common-obj-y += cryptodev.o
common-obj-y += cryptodev-builtin.o
+
+common-obj-$(CONFIG_LINUX) += hostmem-memfd.o
diff --git a/qemu-options.hx b/qemu-options.hx
index 30c4f9850f..6db76cf54f 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3974,6 +3974,17 @@ The @option{share} boolean option determines whether the memory
region is marked as private to QEMU, or shared. The latter allows
a co-operating external process to access the QEMU memory region.
+@item -object memory-backend-memfd,id=@var{id},size=@var{size}
+
+Creates an anonymous memory file backend object, which can be used to
+share the memory with a co-operating external process. The memory is
+allocated with memfd and sealing. (Linux only)
+
+The @option{id} parameter is a unique ID that will be used to
+reference this memory region when configuring the @option{-numa}
+argument. The @option{size} option provides the size of the memory
+region, and accepts common suffixes, eg @option{500M}.
+
@item -object rng-random,id=@var{id},filename=@var{/dev/random}
Creates a random number generator backend which obtains entropy from
--
2.13.1.395.gf7b71de06
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [Qemu-devel] [PATCH v4 3/3] tests: use memfd in vhost-user-test
2017-06-21 14:02 [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend Marc-André Lureau
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 1/3] memfd: split qemu_memfd_alloc() Marc-André Lureau
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem Marc-André Lureau
@ 2017-06-21 14:02 ` Marc-André Lureau
2017-06-22 9:58 ` [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend David Hildenbrand
3 siblings, 0 replies; 11+ messages in thread
From: Marc-André Lureau @ 2017-06-21 14:02 UTC (permalink / raw
To: qemu-devel; +Cc: imammedo, ehabkost, Marc-André Lureau
This will exercise the memfd memory backend and should generally be
better for testing than memory-backend-file (thanks to anonymous files
and sealing).
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
tests/vhost-user-test.c | 68 ++++++++++++++++++++++++++++++++-----------------
1 file changed, 44 insertions(+), 24 deletions(-)
diff --git a/tests/vhost-user-test.c b/tests/vhost-user-test.c
index b3cc045765..2b5ca1c1b0 100644
--- a/tests/vhost-user-test.c
+++ b/tests/vhost-user-test.c
@@ -17,6 +17,7 @@
#include "qemu/range.h"
#include "qemu/sockets.h"
#include "chardev/char-fe.h"
+#include "qemu/memfd.h"
#include "sysemu/sysemu.h"
#include "libqos/libqos.h"
#include "libqos/pci-pc.h"
@@ -40,15 +41,14 @@
#define HAVE_MONOTONIC_TIME
#endif
-#define QEMU_CMD_MEM " -m %d -object memory-backend-file,id=mem,size=%dM,"\
+#define QEMU_CMD_MEM " -m %d -object memory-backend-file,id=mem,size=%dM," \
"mem-path=%s,share=on -numa node,memdev=mem"
+#define QEMU_CMD_MEMFD " -m %d -object memory-backend-memfd,id=mem,size=%dM," \
+ " -numa node,memdev=mem"
#define QEMU_CMD_CHR " -chardev socket,id=%s,path=%s%s"
#define QEMU_CMD_NETDEV " -netdev vhost-user,id=net0,chardev=%s,vhostforce"
#define QEMU_CMD_NET " -device virtio-net-pci,netdev=net0"
-#define QEMU_CMD QEMU_CMD_MEM QEMU_CMD_CHR \
- QEMU_CMD_NETDEV QEMU_CMD_NET
-
#define HUGETLBFS_MAGIC 0x958458f6
/*********** FROM hw/virtio/vhost-user.c *************************************/
@@ -159,6 +159,22 @@ typedef struct TestServer {
static const char *tmpfs;
static const char *root;
+static char *get_qemu_cmd(TestServer *s, int mem, const char *mem_path,
+ const char *chr_opts, const char *extra)
+{
+ if (qemu_memfd_check()) {
+ return g_strdup_printf(QEMU_CMD_MEMFD QEMU_CMD_CHR
+ QEMU_CMD_NETDEV QEMU_CMD_NET "%s", mem, mem,
+ s->chr_name, s->socket_path,
+ chr_opts, s->chr_name, extra);
+ } else {
+ return g_strdup_printf(QEMU_CMD_MEM QEMU_CMD_CHR
+ QEMU_CMD_NETDEV QEMU_CMD_NET "%s", mem, mem,
+ mem_path, s->chr_name, s->socket_path,
+ chr_opts, s->chr_name, extra);
+ }
+}
+
static void init_virtio_dev(TestServer *s)
{
QVirtioPCIDevice *dev;
@@ -477,14 +493,6 @@ static inline void test_server_connect(TestServer *server)
test_server_create_chr(server, ",reconnect=1");
}
-#define GET_QEMU_CMD(s) \
- g_strdup_printf(QEMU_CMD, 512, 512, (root), (s)->chr_name, \
- (s)->socket_path, "", (s)->chr_name)
-
-#define GET_QEMU_CMDE(s, mem, chr_opts, extra, ...) \
- g_strdup_printf(QEMU_CMD extra, (mem), (mem), (root), (s)->chr_name, \
- (s)->socket_path, (chr_opts), (s)->chr_name, ##__VA_ARGS__)
-
static gboolean _test_server_free(TestServer *server)
{
int i;
@@ -628,7 +636,7 @@ static void test_migrate(void)
char *uri = g_strdup_printf("%s%s", "unix:", dest->mig_path);
QTestState *global = global_qtest, *from, *to;
GSource *source;
- gchar *cmd;
+ gchar *cmd, *tmp;
QDict *rsp;
guint8 *log;
guint64 size;
@@ -636,7 +644,7 @@ static void test_migrate(void)
test_server_listen(s);
test_server_listen(dest);
- cmd = GET_QEMU_CMDE(s, 2, "", "");
+ cmd = get_qemu_cmd(s, 2, root, "", "");
from = qtest_start(cmd);
g_free(cmd);
@@ -645,7 +653,9 @@ static void test_migrate(void)
size = get_log_size(s);
g_assert_cmpint(size, ==, (2 * 1024 * 1024) / (VHOST_LOG_PAGE * 8));
- cmd = GET_QEMU_CMDE(dest, 2, "", " -incoming %s", uri);
+ tmp = g_strdup_printf(" -incoming %s", uri);
+ cmd = get_qemu_cmd(dest, 2, root, "", tmp);
+ g_free(tmp);
to = qtest_init(cmd);
g_free(cmd);
@@ -751,7 +761,7 @@ static void test_reconnect_subprocess(void)
char *cmd;
g_thread_new("connect", connect_thread, s);
- cmd = GET_QEMU_CMDE(s, 2, ",server", "");
+ cmd = get_qemu_cmd(s, 2, root, ",server", "");
qtest_start(cmd);
g_free(cmd);
@@ -787,7 +797,7 @@ static void test_connect_fail_subprocess(void)
s->test_fail = true;
g_thread_new("connect", connect_thread, s);
- cmd = GET_QEMU_CMDE(s, 2, ",server", "");
+ cmd = get_qemu_cmd(s, 2, root, ",server", "");
qtest_start(cmd);
g_free(cmd);
@@ -815,7 +825,7 @@ static void test_flags_mismatch_subprocess(void)
s->test_flags = TEST_FLAGS_DISCONNECT;
g_thread_new("connect", connect_thread, s);
- cmd = GET_QEMU_CMDE(s, 2, ",server", "");
+ cmd = get_qemu_cmd(s, 2, root, ",server", "");
qtest_start(cmd);
g_free(cmd);
@@ -883,11 +893,21 @@ static void test_multiqueue(void)
s->queues = queues;
test_server_listen(s);
- cmd = g_strdup_printf(QEMU_CMD_MEM QEMU_CMD_CHR QEMU_CMD_NETDEV ",queues=%d "
- "-device virtio-net-pci,netdev=net0,mq=on,vectors=%d",
- 512, 512, root, s->chr_name,
- s->socket_path, "", s->chr_name,
- queues, queues * 2 + 2);
+ if (qemu_memfd_check()) {
+ cmd = g_strdup_printf(
+ QEMU_CMD_MEMFD QEMU_CMD_CHR QEMU_CMD_NETDEV ",queues=%d "
+ "-device virtio-net-pci,netdev=net0,mq=on,vectors=%d",
+ 512, 512, s->chr_name,
+ s->socket_path, "", s->chr_name,
+ queues, queues * 2 + 2);
+ } else {
+ cmd = g_strdup_printf(
+ QEMU_CMD_MEM QEMU_CMD_CHR QEMU_CMD_NETDEV ",queues=%d "
+ "-device virtio-net-pci,netdev=net0,mq=on,vectors=%d",
+ 512, 512, root, s->chr_name,
+ s->socket_path, "", s->chr_name,
+ queues, queues * 2 + 2);
+ }
qtest_start(cmd);
g_free(cmd);
@@ -953,7 +973,7 @@ int main(int argc, char **argv)
/* run the main loop thread so the chardev may operate */
thread = g_thread_new(NULL, thread_function, loop);
- qemu_cmd = GET_QEMU_CMD(server);
+ qemu_cmd = get_qemu_cmd(server, 512, root, "", "");
s = qtest_start(qemu_cmd);
g_free(qemu_cmd);
--
2.13.1.395.gf7b71de06
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend
2017-06-21 14:02 [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend Marc-André Lureau
` (2 preceding siblings ...)
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 3/3] tests: use memfd in vhost-user-test Marc-André Lureau
@ 2017-06-22 9:58 ` David Hildenbrand
2017-06-23 11:29 ` Eduardo Habkost
3 siblings, 1 reply; 11+ messages in thread
From: David Hildenbrand @ 2017-06-22 9:58 UTC (permalink / raw
To: Marc-André Lureau, qemu-devel; +Cc: imammedo, ehabkost
On 21.06.2017 16:02, Marc-André Lureau wrote:
> Add a new Linux-specific memory backend, similar to hostmem-file,
> except that it doesn't need file path. It also try to enforce memory
> sealing if available. It is thus slightly easier and secure, and is
> compatible with transparent huge-pages since Linux 4.8.
>
> v4:
> - rebased, now that preliminary patches are merged
>
> v3:
> - make vhost-user-test use memfd only if possible
> - rebased
>
> v1->v2:
> - make it linux-specific
> - minor changes and commit message tweaks
>
> Marc-André Lureau (3):
> memfd: split qemu_memfd_alloc()
> Add memfd based hostmem
> tests: use memfd in vhost-user-test
>
> include/qemu/memfd.h | 2 ++
> backends/hostmem-memfd.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++
> tests/vhost-user-test.c | 68 +++++++++++++++++++++++++++++++-----------------
> util/memfd.c | 42 ++++++++++++++++++------------
> backends/Makefile.objs | 2 ++
> qemu-options.hx | 11 ++++++++
> 6 files changed, 151 insertions(+), 41 deletions(-)
> create mode 100644 backends/hostmem-memfd.c
>
Just wondering if it would make more sense to add a new parameter to the
ram backend. Sorry if this has already been discussed.
--
Thanks,
David
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend
2017-06-22 9:58 ` [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend David Hildenbrand
@ 2017-06-23 11:29 ` Eduardo Habkost
0 siblings, 0 replies; 11+ messages in thread
From: Eduardo Habkost @ 2017-06-23 11:29 UTC (permalink / raw
To: David Hildenbrand; +Cc: Marc-André Lureau, qemu-devel, imammedo
On Thu, Jun 22, 2017 at 11:58:14AM +0200, David Hildenbrand wrote:
> On 21.06.2017 16:02, Marc-André Lureau wrote:
> > Add a new Linux-specific memory backend, similar to hostmem-file,
> > except that it doesn't need file path. It also try to enforce memory
> > sealing if available. It is thus slightly easier and secure, and is
> > compatible with transparent huge-pages since Linux 4.8.
> >
> > v4:
> > - rebased, now that preliminary patches are merged
> >
> > v3:
> > - make vhost-user-test use memfd only if possible
> > - rebased
> >
> > v1->v2:
> > - make it linux-specific
> > - minor changes and commit message tweaks
> >
> > Marc-André Lureau (3):
> > memfd: split qemu_memfd_alloc()
> > Add memfd based hostmem
> > tests: use memfd in vhost-user-test
> >
> > include/qemu/memfd.h | 2 ++
> > backends/hostmem-memfd.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++
> > tests/vhost-user-test.c | 68 +++++++++++++++++++++++++++++++-----------------
> > util/memfd.c | 42 ++++++++++++++++++------------
> > backends/Makefile.objs | 2 ++
> > qemu-options.hx | 11 ++++++++
> > 6 files changed, 151 insertions(+), 41 deletions(-)
> > create mode 100644 backends/hostmem-memfd.c
> >
>
> Just wondering if it would make more sense to add a new parameter to the
> ram backend. Sorry if this has already been discussed.
That was my first thought, too. But although it requires more
boilerplate code, a separate class gives us better input
specification/validation for free: e.g. if we add memfd-specific options
in the future, we don't even need to document them as "valid only in
memfd mode", because they will exist only in the memfd class.
--
Eduardo
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem Marc-André Lureau
@ 2017-06-23 21:08 ` Eduardo Habkost
2017-06-27 8:23 ` Marc-André Lureau
0 siblings, 1 reply; 11+ messages in thread
From: Eduardo Habkost @ 2017-06-23 21:08 UTC (permalink / raw
To: Marc-André Lureau; +Cc: qemu-devel, imammedo
On Wed, Jun 21, 2017 at 04:02:18PM +0200, Marc-André Lureau wrote:
> Add a new memory backend, similar to hostmem-file, except that it
> doesn't need to create files. It also enforces memory sealing.
>
> This backend is mainly useful for sharing the memory with other
> processes.
How exactly can the memfd be used to share memory? Is there an existing
mechanism for sharing the memfd file descriptor with another process?
>
> Note that Linux supports transparent huge-pages of shmem/memfd memory
> since 4.8. It is relatively easier to set up THP than a dedicate
> hugepage mount point by using "madvise" in
> /sys/kernel/mm/transparent_hugepage/shmem_enabled.
>
> Usage:
> -object memory-backend-memfd,id=mem1,size=1G
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
> backends/hostmem-memfd.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++
> backends/Makefile.objs | 2 ++
> qemu-options.hx | 11 ++++++++
> 3 files changed, 80 insertions(+)
> create mode 100644 backends/hostmem-memfd.c
>
[...]
--
Eduardo
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem
2017-06-23 21:08 ` Eduardo Habkost
@ 2017-06-27 8:23 ` Marc-André Lureau
2017-06-28 18:29 ` Eduardo Habkost
0 siblings, 1 reply; 11+ messages in thread
From: Marc-André Lureau @ 2017-06-27 8:23 UTC (permalink / raw
To: Eduardo Habkost; +Cc: imammedo, qemu-devel
Hi Eduardo
On Fri, Jun 23, 2017 at 11:09 PM Eduardo Habkost <ehabkost@redhat.com>
wrote:
> On Wed, Jun 21, 2017 at 04:02:18PM +0200, Marc-André Lureau wrote:
> > Add a new memory backend, similar to hostmem-file, except that it
> > doesn't need to create files. It also enforces memory sealing.
> >
> > This backend is mainly useful for sharing the memory with other
> > processes.
>
> How exactly can the memfd be used to share memory? Is there an existing
> mechanism for sharing the memfd file descriptor with another process
Since there is no backing file, the traditional mechanism is by passing fd,
via socket ancillary data or forking etc.. Both ivshmem and vhost-user have
such messages, with eventually details for the memory map usage.
> >
> > Note that Linux supports transparent huge-pages of shmem/memfd memory
> > since 4.8. It is relatively easier to set up THP than a dedicate
> > hugepage mount point by using "madvise" in
> > /sys/kernel/mm/transparent_hugepage/shmem_enabled.
> >
> > Usage:
> > -object memory-backend-memfd,id=mem1,size=1G
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> > backends/hostmem-memfd.c | 67
> ++++++++++++++++++++++++++++++++++++++++++++++++
> > backends/Makefile.objs | 2 ++
> > qemu-options.hx | 11 ++++++++
> > 3 files changed, 80 insertions(+)
> > create mode 100644 backends/hostmem-memfd.c
> >
> [...]
>
> --
> Eduardo
>
> --
Marc-André Lureau
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem
2017-06-27 8:23 ` Marc-André Lureau
@ 2017-06-28 18:29 ` Eduardo Habkost
2017-06-29 13:34 ` Marc-André Lureau
0 siblings, 1 reply; 11+ messages in thread
From: Eduardo Habkost @ 2017-06-28 18:29 UTC (permalink / raw
To: Marc-André Lureau; +Cc: imammedo, qemu-devel
On Tue, Jun 27, 2017 at 08:23:03AM +0000, Marc-André Lureau wrote:
> Hi Eduardo
>
> On Fri, Jun 23, 2017 at 11:09 PM Eduardo Habkost <ehabkost@redhat.com>
> wrote:
>
> > On Wed, Jun 21, 2017 at 04:02:18PM +0200, Marc-André Lureau wrote:
> > > Add a new memory backend, similar to hostmem-file, except that it
> > > doesn't need to create files. It also enforces memory sealing.
> > >
> > > This backend is mainly useful for sharing the memory with other
> > > processes.
> >
> > How exactly can the memfd be used to share memory? Is there an existing
> > mechanism for sharing the memfd file descriptor with another process
>
>
>
> Since there is no backing file, the traditional mechanism is by passing fd,
> via socket ancillary data or forking etc.. Both ivshmem and vhost-user have
> such messages, with eventually details for the memory map usage.
The documentation is very similar to memory-backend-file, so it sounded
like there was a generic mechanism to ask QEMU to share the backend FD.
Maybe it would be interesting to mention on which cases the FD can
actually be shared. Are ivshmem and vhost-user the only existing cases?
>
>
> > >
> > > Note that Linux supports transparent huge-pages of shmem/memfd memory
> > > since 4.8. It is relatively easier to set up THP than a dedicate
> > > hugepage mount point by using "madvise" in
> > > /sys/kernel/mm/transparent_hugepage/shmem_enabled.
> > >
> > > Usage:
> > > -object memory-backend-memfd,id=mem1,size=1G
> > >
> > > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > ---
> > > backends/hostmem-memfd.c | 67
> > ++++++++++++++++++++++++++++++++++++++++++++++++
> > > backends/Makefile.objs | 2 ++
> > > qemu-options.hx | 11 ++++++++
> > > 3 files changed, 80 insertions(+)
> > > create mode 100644 backends/hostmem-memfd.c
> > >
> > [...]
> >
> > --
> > Eduardo
> >
> > --
> Marc-André Lureau
--
Eduardo
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem
2017-06-28 18:29 ` Eduardo Habkost
@ 2017-06-29 13:34 ` Marc-André Lureau
2017-07-03 14:34 ` Eduardo Habkost
0 siblings, 1 reply; 11+ messages in thread
From: Marc-André Lureau @ 2017-06-29 13:34 UTC (permalink / raw
To: Eduardo Habkost; +Cc: imammedo, qemu-devel
Hi
On Wed, Jun 28, 2017 at 8:29 PM Eduardo Habkost <ehabkost@redhat.com> wrote:
> On Tue, Jun 27, 2017 at 08:23:03AM +0000, Marc-André Lureau wrote:
> > Hi Eduardo
> >
> > On Fri, Jun 23, 2017 at 11:09 PM Eduardo Habkost <ehabkost@redhat.com>
> > wrote:
> >
> > > On Wed, Jun 21, 2017 at 04:02:18PM +0200, Marc-André Lureau wrote:
> > > > Add a new memory backend, similar to hostmem-file, except that it
> > > > doesn't need to create files. It also enforces memory sealing.
> > > >
> > > > This backend is mainly useful for sharing the memory with other
> > > > processes.
> > >
> > > How exactly can the memfd be used to share memory? Is there an
> existing
> > > mechanism for sharing the memfd file descriptor with another process
> >
> >
> >
> > Since there is no backing file, the traditional mechanism is by passing
> fd,
> > via socket ancillary data or forking etc.. Both ivshmem and vhost-user
> have
> > such messages, with eventually details for the memory map usage.
>
> The documentation is very similar to memory-backend-file, so it sounded
> like there was a generic mechanism to ask QEMU to share the backend FD.
> Maybe it would be interesting to mention on which cases the FD can
> actually be shared. Are ivshmem and vhost-user the only existing cases?
>
>
Actually, vhost-user may be the only way today to get the fd outside of
qemu process.
(ivshmem needs the server to provide the fd)
We could quite easily add or extend QMP messages for that in the future.
Do you want this detail to be written in the commit message or elsewhere?
> >
> >
> > > >
> > > > Note that Linux supports transparent huge-pages of shmem/memfd memory
> > > > since 4.8. It is relatively easier to set up THP than a dedicate
> > > > hugepage mount point by using "madvise" in
> > > > /sys/kernel/mm/transparent_hugepage/shmem_enabled.
> > > >
> > > > Usage:
> > > > -object memory-backend-memfd,id=mem1,size=1G
> > > >
> > > > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > > ---
> > > > backends/hostmem-memfd.c | 67
> > > ++++++++++++++++++++++++++++++++++++++++++++++++
> > > > backends/Makefile.objs | 2 ++
> > > > qemu-options.hx | 11 ++++++++
> > > > 3 files changed, 80 insertions(+)
> > > > create mode 100644 backends/hostmem-memfd.c
> > > >
> > > [...]
> > >
> > > --
> > > Eduardo
> > >
> > > --
> > Marc-André Lureau
>
> --
> Eduardo
>
--
Marc-André Lureau
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem
2017-06-29 13:34 ` Marc-André Lureau
@ 2017-07-03 14:34 ` Eduardo Habkost
0 siblings, 0 replies; 11+ messages in thread
From: Eduardo Habkost @ 2017-07-03 14:34 UTC (permalink / raw
To: Marc-André Lureau; +Cc: imammedo, qemu-devel
On Thu, Jun 29, 2017 at 01:34:22PM +0000, Marc-André Lureau wrote:
> Hi
>
> On Wed, Jun 28, 2017 at 8:29 PM Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> > On Tue, Jun 27, 2017 at 08:23:03AM +0000, Marc-André Lureau wrote:
> > > Hi Eduardo
> > >
> > > On Fri, Jun 23, 2017 at 11:09 PM Eduardo Habkost <ehabkost@redhat.com>
> > > wrote:
> > >
> > > > On Wed, Jun 21, 2017 at 04:02:18PM +0200, Marc-André Lureau wrote:
> > > > > Add a new memory backend, similar to hostmem-file, except that it
> > > > > doesn't need to create files. It also enforces memory sealing.
> > > > >
> > > > > This backend is mainly useful for sharing the memory with other
> > > > > processes.
> > > >
> > > > How exactly can the memfd be used to share memory? Is there an
> > existing
> > > > mechanism for sharing the memfd file descriptor with another process
> > >
> > >
> > >
> > > Since there is no backing file, the traditional mechanism is by passing
> > fd,
> > > via socket ancillary data or forking etc.. Both ivshmem and vhost-user
> > have
> > > such messages, with eventually details for the memory map usage.
> >
> > The documentation is very similar to memory-backend-file, so it sounded
> > like there was a generic mechanism to ask QEMU to share the backend FD.
> > Maybe it would be interesting to mention on which cases the FD can
> > actually be shared. Are ivshmem and vhost-user the only existing cases?
> >
> >
> Actually, vhost-user may be the only way today to get the fd outside of
> qemu process.
>
> (ivshmem needs the server to provide the fd)
>
> We could quite easily add or extend QMP messages for that in the future.
>
> Do you want this detail to be written in the commit message or elsewhere?
(Sorry for taking so long to reply.)
I think this should be in the documentation of the new option (in
qemu-options.hx).
I don't think it needs lots of extra detail, maybe just change "which
can be used to share the memory with a co-operating external process" to
"which allows QEMU to share the memory with an external process in some
cases (e.g. when using vhost-user)".
--
Eduardo
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2017-07-03 14:34 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-06-21 14:02 [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend Marc-André Lureau
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 1/3] memfd: split qemu_memfd_alloc() Marc-André Lureau
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 2/3] Add memfd based hostmem Marc-André Lureau
2017-06-23 21:08 ` Eduardo Habkost
2017-06-27 8:23 ` Marc-André Lureau
2017-06-28 18:29 ` Eduardo Habkost
2017-06-29 13:34 ` Marc-André Lureau
2017-07-03 14:34 ` Eduardo Habkost
2017-06-21 14:02 ` [Qemu-devel] [PATCH v4 3/3] tests: use memfd in vhost-user-test Marc-André Lureau
2017-06-22 9:58 ` [Qemu-devel] [PATCH v4 0/3] Add memfd memory backend David Hildenbrand
2017-06-23 11:29 ` Eduardo Habkost
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.