From: Wen Congyang <wency@cn.fujitsu.com>
To: qemu devel <qemu-devel@nongnu.org>, Fam Zheng <famz@redhat.com>,
Max Reitz <mreitz@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>, qemu block <qemu-block@nongnu.org>,
Lai Jiangshan <laijs@cn.fujitsu.com>,
Jiang Yunhong <yunhong.jiang@intel.com>,
Dong Eddie <eddie.dong@intel.com>,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
Gonglei <arei.gonglei@huawei.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
Yang Hongyang <yanghy@cn.fujitsu.com>,
zhanghailiang <zhang.zhanghailiang@huawei.com>
Subject: [Qemu-devel] [PATCH COLO-Block v6 01/16] docs: block replication's description
Date: Thu, 18 Jun 2015 16:49:06 +0800 [thread overview]
Message-ID: <1434617361-17778-2-git-send-email-wency@cn.fujitsu.com> (raw)
In-Reply-To: <1434617361-17778-1-git-send-email-wency@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
docs/block-replication.txt | 179 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 179 insertions(+)
create mode 100644 docs/block-replication.txt
diff --git a/docs/block-replication.txt b/docs/block-replication.txt
new file mode 100644
index 0000000..a29f51a
--- /dev/null
+++ b/docs/block-replication.txt
@@ -0,0 +1,179 @@
+Block replication
+----------------------------------------
+Copyright Fujitsu, Corp. 2015
+Copyright (c) 2015 Intel Corporation
+Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+
+This work is licensed under the terms of the GNU GPL, version 2 or later.
+See the COPYING file in the top-level directory.
+
+Block replication is used for continuous checkpoints. It is designed
+for COLO (COurse-grain LOck-stepping) where the Secondary VM is running.
+It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario,
+where the Secondary VM is not running.
+
+This document gives an overview of block replication's design.
+
+== Background ==
+High availability solutions such as micro checkpoint and COLO will do
+consecutive checkpoints. The VM state of Primary VM and Secondary VM is
+identical right after a VM checkpoint, but becomes different as the VM
+executes till the next checkpoint. To support disk contents checkpoint,
+the modified disk contents in the Secondary VM must be buffered, and are
+only dropped at next checkpoint time. To reduce the network transportation
+effort at the time of checkpoint, the disk modification operations of
+Primary disk are asynchronously forwarded to the Secondary node.
+
+== Workflow ==
+The following is the image of block replication workflow:
+
+ +----------------------+ +------------------------+
+ |Primary Write Requests| |Secondary Write Requests|
+ +----------------------+ +------------------------+
+ | |
+ | (4)
+ | V
+ | /-------------\
+ | Copy and Forward | |
+ |---------(1)----------+ | Disk Buffer |
+ | | | |
+ | (3) \-------------/
+ | speculative ^
+ | write through (2)
+ | | |
+ V V |
+ +--------------+ +----------------+
+ | Primary Disk | | Secondary Disk |
+ +--------------+ +----------------+
+
+ 1) Primary write requests will be copied and forwarded to Secondary
+ QEMU.
+ 2) Before Primary write requests are written to Secondary disk, the
+ original sector content will be read from Secondary disk and
+ buffered in the Disk buffer, but it will not overwrite the existing
+ sector content(it could be from either "Secondary Write Requests" or
+ previous COW of "Primary Write Requests") in the Disk buffer.
+ 3) Primary write requests will be written to Secondary disk.
+ 4) Secondary write requests will be buffered in the Disk buffer and it
+ will overwrite the existing sector content in the buffer.
+
+== Architecture ==
+We are going to implement block replication from many basic
+blocks that are already in QEMU.
+
+ virtio-blk ||
+ ^ || .----------
+ | || | Secondary
+ 1 Quorum || '----------
+ / \ ||
+ / \ ||
+ Primary 2 filter
+ disk ^ virtio-blk
+ | ^
+ 3 NBD -------> 3 NBD |
+ client || server 2 filter
+ || ^ ^
+--------. || | |
+Primary | || Secondary disk <--------- hidden-disk 5 <--------- active-disk 4
+--------' || | backing ^ backing
+ || | |
+ || | |
+ || '-------------------------'
+ || drive-backup sync=none
+
+1) The disk on the primary is represented by a block device with two
+children, providing replication between a primary disk and the host that
+runs the secondary VM. The read pattern for quorum can be extended to
+make the primary always read from the local disk instead of going through
+NBD.
+
+2) The new block filter(the name is replication) will control the block
+replication.
+
+3) The secondary disk receives writes from the primary VM through QEMU's
+embedded NBD server (speculative write-through).
+
+4) The disk on the secondary is represented by a custom block device
+(called active-disk). It should be an empty disk, and the format should
+support bdrv_make_empty() and backing file.
+
+5) The hidden-disk is created automatically. It buffers the original content
+that is modified by the primary VM. It should also be an empty disk, and
+the driver supports bdrv_make_empty() and backing file.
+
+== Failure Handling ==
+There are 6 internal errors when block replication is running:
+1. I/O error on primary disk
+2. Forwarding primay write requests failed
+3. Bacup failed or writing to secondary disk failed
+4. I/O error on secondary disk
+5. I/O error on active disk
+6. Making active disk or hidden disk empty failed
+In case 1 and 5, we just report the error to the disk layer. In case 2, 3,
+4 and 6, we just report block replication's error to FT/HA manager(which
+decides when to do a new checkpoint, when to do failover).
+There is one internal error when doing failover:
+1. Commiting the data in active disk/hidden disk to secondary disk failed
+We just to report this error to FT/HA manager.
+
+== New block driver interface ==
+We add three block driver interfaces to control block replication:
+a. bdrv_start_replication()
+ Start block replication, called in migration/checkpoint thread.
+ We must call bdrv_start_replication() in secondary QEMU before
+ calling bdrv_start_replication() in primary QEMU. The caller
+ must hold the I/O mutex lock if it is in migration/checkpoint
+ thread.
+b. bdrv_do_checkpoint()
+ This interface is called after all VM state is transferred to
+ Secondary QEMU. The Disk buffer will be dropped in this interface.
+ The caller must hold the I/O mutex lock if it is in migration/checkpoint
+ thread.
+c. bdrv_stop_replication()
+ It is called on failover. We will flush the Disk buffer into
+ Secondary Disk and stop block replication. The vm should be stopped
+ before calling it. The caller must hold the I/O mutex lock if it is
+ in migration/checkpoint thread.
+
+== Usage ==
+Primary:
+ -drive if=xxx,driver=quorum,read-pattern=fifo,no-connect=on,\
+ children.0.file.filename=1.raw,\
+ children.0.driver=raw,\
+ children.1.file.driver=nbd,\
+ children.1.file.host=xxx,\
+ children.1.file.port=xxx,\
+ children.1.file.export=xxx,\
+ children.1.driver=replication,\
+ children.1.mode=primary,\
+ children.1.ignore-errors=on
+ Note:
+ 1. NBD Client should not be the first child of quorum.
+ 2. There should be only one NBD Client.
+ 3. host is the secondary physical machine's hostname or IP
+ 4. Each disk must have its own export name.
+ 5. It is all a single argument to -drive, and you should ignore
+ the leading whitespace.
+
+Secondary:
+ -drive if=none,driver=raw,file=1.raw,id=nbd_target1 \
+ -drive if=xxx,driver=replication,mode=secondary,export=xxx,\
+ file.file.filename=active_disk.qcow2,\
+ file.driver=qcow2,\
+ file.backing_reference.drive_id=nbd_target1,\
+ file.backing_reference.hidden-disk.file.filename=hidden_disk.qcow2,\
+ file.backing_reference.hidden-disk.driver=qcow2,\
+ file.backing_reference.hidden-disk.allow-write-backing-file=on
+ Then run qmp command:
+ nbd-server-start host:port
+ Note:
+ 1. The export name for the same disk must be the same in primary
+ and secondary QEMU command line
+ 2. The qmp command nbd-server-start must be run before running the
+ qmp command migrate on primary QEMU
+ 3. Don't use nbd-server-start's other options
+ 4. Active disk, hidden disk and nbd target's length should be the
+ same.
+ 5. It is better to put active disk and hidden disk in ramdisk.
+ 6. It is all a single argument to -drive, and you should ignore
+ the leading whitespace.
--
2.4.3
next prev parent reply other threads:[~2015-06-18 8:45 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-06-18 8:49 [Qemu-devel] [PATCH COLO-Block v6 00/16] Block replication for continuous checkpoints Wen Congyang
2015-06-18 8:49 ` Wen Congyang [this message]
2015-06-18 10:34 ` [Qemu-devel] [PATCH COLO-Block v6 01/16] docs: block replication's description Stefan Hajnoczi
2015-06-18 10:51 ` Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 02/16] allow writing to the backing file Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 03/16] Allow creating backup jobs when opening BDS Wen Congyang
2015-06-18 10:40 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 04/16] block: Parse "backing_reference" option to reference existing BDS Wen Congyang
2015-06-18 10:50 ` Stefan Hajnoczi
2015-06-18 11:40 ` Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 05/16] Backup: clear all bitmap when doing block checkpoint Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 06/16] Don't allow a disk use backing reference target Wen Congyang
2015-06-18 12:47 ` Stefan Hajnoczi
2015-06-18 15:17 ` Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 07/16] Add new block driver interface to connect/disconnect the remote target Wen Congyang
2015-06-18 12:55 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2015-06-18 14:36 ` Wen Congyang
2015-06-18 16:06 ` Stefan Hajnoczi
2015-06-19 0:54 ` Wen Congyang
2015-06-19 10:49 ` Stefan Hajnoczi
2015-06-20 3:31 ` Wen Congyang
2015-06-22 12:39 ` Stefan Hajnoczi
2015-06-22 13:56 ` Wen Congyang
2015-06-23 13:42 ` Stefan Hajnoczi
2015-06-23 13:54 ` Wen Congyang
2015-06-24 14:07 ` Dr. David Alan Gilbert
2015-06-25 1:01 ` Wen Congyang
2015-06-26 19:03 ` Dr. David Alan Gilbert
2015-06-29 1:05 ` Wen Congyang
2015-06-30 19:01 ` Dr. David Alan Gilbert
2015-07-01 1:01 ` Wen Congyang
2015-07-01 8:11 ` Dr. David Alan Gilbert
2015-07-01 8:18 ` Wen Congyang
2015-07-01 18:42 ` Dr. David Alan Gilbert
2015-07-02 0:55 ` Wen Congyang
2015-07-02 7:54 ` Dr. David Alan Gilbert
2015-07-02 8:05 ` Wen Congyang
2015-06-24 1:11 ` Wen Congyang
2015-06-23 6:44 ` Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 08/16] NBD client: implement block driver interfaces to connect/disconnect NBD server Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 09/16] Introduce a new -drive option to control whether to connect to remote target Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 10/16] NBD client: connect to nbd server later Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 11/16] Add new block driver interfaces to control block replication Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 12/16] skip nbd_target when starting " Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 13/16] quorum: implement block driver interfaces for " Wen Congyang
2015-06-18 13:06 ` Stefan Hajnoczi
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 14/16] introduce a new API qemu_opts_absorb_qdict_by_index() Wen Congyang
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 15/16] quorum: allow ignoring child errors Wen Congyang
2015-06-18 13:06 ` Stefan Hajnoczi
2015-06-18 8:49 ` [Qemu-devel] [PATCH COLO-Block v6 16/16] Implement new driver for block replication Wen Congyang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1434617361-17778-2-git-send-email-wency@cn.fujitsu.com \
--to=wency@cn.fujitsu.com \
--cc=arei.gonglei@huawei.com \
--cc=dgilbert@redhat.com \
--cc=eddie.dong@intel.com \
--cc=famz@redhat.com \
--cc=kwolf@redhat.com \
--cc=laijs@cn.fujitsu.com \
--cc=mreitz@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=yanghy@cn.fujitsu.com \
--cc=yunhong.jiang@intel.com \
--cc=zhang.zhanghailiang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.