From: Fam Zheng <famz@redhat.com>
To: qemu-devel@nongnu.org
Cc: Kevin Wolf <kwolf@redhat.com>,
famz@redhat.com, John Snow <jsnow@redhat.com>,
Jeff Cody <jcody@redhat.com>, Max Reitz <mreitz@redhat.com>,
vsementsov@parallels.com, stefanha@redhat.com
Subject: [Qemu-devel] [PATCH v3 00/15] block: incremental backup transactions using BlockJobTxn
Date: Fri, 10 Jul 2015 11:46:37 +0800 [thread overview]
Message-ID: <1436500012-32593-1-git-send-email-famz@redhat.com> (raw)
v3: Simplify the code a bit to implement the idea as discussed in v2 thread:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg02130.html
Unchanged:
[01/15] qapi: Add transaction support to block-dirty-bitmap operations
[02/15] iotests: add transactional incremental backup test
[03/15] block: rename BlkTransactionState and BdrvActionOps
[04/15] block: keep bitmap if incremental backup job is cancelled
New patch to refactor out dirty bitmap reclaimation code in backup:
[05/15] backup: Extract dirty bitmap handling as a separate function
Two new transaction specific operations for block job driver:
[06/15] blockjob: Add .txn_commit and .txn_abort transaction actions
Two new fields that will make managing state of block job easier:
[07/15] blockjob: Add "completed" and "ret" in BlockJob
New. Necessary to avoid race conditions between jobs:
[08/15] blockjob: Simplify block_job_finish_sync
[09/15] blockjob: Move BlockJobDeferToMainLoopData into BlockJob
Partly rewrote the implementation:
[10/15] block: add block job transactions
Slightly adjust to the new API (no block_job_txn_begin):
[11/15] blockdev: make BlockJobTxn available to qmp 'transaction'
Slightly adjust to the new API (no block_job_txn_job_done).
[12/15] block/backup: support block job transactions
Unchanged:
[13/15] iotests: 124 - transactional failure test
[14/15] qmp-commands.hx: Update the supported 'transaction' operations
[15/15] tests: add BlockJobTxn unit test
---
Original cover letter from Stefan's v2, note that block_job_txn_begin and
block_job_txn_job_done are not necessary now:
v2:
Patch 5:
* Set txn pointer to NULL in block_job_txn_begin() [jsnow]
* Rename block_job_txn_job_done to block_job_txn_job_done [jsnow]
* Rename block_job_txn_complete to block_job_txn_kick [jsnow]
* Add BLOCK_JOB_TXN_CANCEL_PENDING to solve race condition on cancel [jsnow]
* Document when txn may be NULL
Patch 7:
* Convert blockdev-backup in addition to drive-backup
* Add 'transactional-cancel' argument so users can enable/disable
transactional behavior. This preserves semantics for existing users
and allows fine-grained control over when to use transaction
semantics. [jsnow]
Patch 10:
* Test fail/cancel race condition [jsnow]
* Use new 'transactional-cancel' QMP argument
This series uses patches from John Snow's "[PATCH v6 00/10] block: incremental
backup transactions" series but implements the feature with a new transaction
mechanism for blockjobs called BlockJobTxn.
Recap: motivation for block job transactions
--------------------------------------------
If an incremental backup block job fails then we reclaim the bitmap so
the job can be retried. The problem comes when multiple jobs are started as
part of a qmp 'transaction' command. We need to group these jobs in a
transaction so that either all jobs complete successfully or all bitmaps are
reclaimed.
Without transactions, there is a case where some jobs complete successfully and
throw away their bitmaps, making it impossible to retry the backup by rerunning
the command if one of the jobs fails.
How does this implementation work?
----------------------------------
These patches add a BlockJobTxn object with the following API:
txn = block_job_txn_new();
block_job_txn_add_job(txn, job1);
block_job_txn_add_job(txn, job2);
block_job_txn_begin();
The jobs either both complete successfully or they both fail/cancel. If the
user cancels job1 then job2 will also be cancelled and vice versa.
Jobs stay alive waiting for other jobs to complete. They can be cancelled by
the user during this time. Job blockers are still in effect and no other block
job can run on this device in the meantime (since QEMU currently only allows 1
job per device). This is the main drawback to this approach but reasonable
since you probably don't want to run other jobs/operations until you're sure
the backup was successful (you won't be able to retry a failed backup if
there's a new job running).
Adding transaction support to the backup job is very easy. It just needs to
make a call before throwing away the bitmap and returning from its coroutine:
block_job_txn_job_done(job->txn, job, ret);
if (job->sync_bitmap) {
BdrvDirtyBitmap *bm;
if (ret < 0 || block_job_is_cancelled(&job->common)) {
...
Fam Zheng (5):
backup: Extract dirty bitmap handling as a separate function
blockjob: Add .txn_commit and .txn_abort transaction actions
blockjob: Add "completed" and "ret" in BlockJob
blockjob: Simplify block_job_finish_sync
blockjob: Move BlockJobDeferToMainLoopData into BlockJob
John Snow (4):
qapi: Add transaction support to block-dirty-bitmap operations
iotests: add transactional incremental backup test
block: rename BlkTransactionState and BdrvActionOps
iotests: 124 - transactional failure test
Kashyap Chamarthy (1):
qmp-commands.hx: Update the supported 'transaction' operations
Stefan Hajnoczi (5):
block: keep bitmap if incremental backup job is cancelled
block: add block job transactions
blockdev: make BlockJobTxn available to qmp 'transaction'
block/backup: support block job transactions
tests: add BlockJobTxn unit test
block.c | 19 ++-
block/backup.c | 49 ++++--
blockdev.c | 360 ++++++++++++++++++++++++++++++++++++---------
blockjob.c | 146 +++++++++++++-----
docs/bitmaps.md | 6 +-
hmp.c | 2 +-
include/block/block.h | 2 +-
include/block/block_int.h | 6 +-
include/block/blockjob.h | 63 +++++++-
qapi-schema.json | 6 +-
qapi/block-core.json | 14 +-
qmp-commands.hx | 21 ++-
tests/Makefile | 3 +
tests/qemu-iotests/124 | 182 ++++++++++++++++++++++-
tests/qemu-iotests/124.out | 4 +-
tests/test-blockjob-txn.c | 228 ++++++++++++++++++++++++++++
16 files changed, 963 insertions(+), 148 deletions(-)
create mode 100644 tests/test-blockjob-txn.c
--
2.4.3
next reply other threads:[~2015-07-10 3:47 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-07-10 3:46 Fam Zheng [this message]
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 01/15] qapi: Add transaction support to block-dirty-bitmap operations Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 02/15] iotests: add transactional incremental backup test Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 03/15] block: rename BlkTransactionState and BdrvActionOps Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 04/15] block: keep bitmap if incremental backup job is cancelled Fam Zheng
2015-07-13 19:48 ` John Snow
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 05/15] backup: Extract dirty bitmap handling as a separate function Fam Zheng
2015-07-13 23:06 ` John Snow
2015-07-14 2:46 ` Fam Zheng
2015-07-14 8:26 ` Stefan Hajnoczi
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 06/15] blockjob: Add .txn_commit and .txn_abort transaction actions Fam Zheng
2015-07-13 23:06 ` John Snow
2015-07-14 8:35 ` Stefan Hajnoczi
2015-07-14 9:26 ` Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 07/15] blockjob: Add "completed" and "ret" in BlockJob Fam Zheng
2015-07-13 23:08 ` John Snow
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 08/15] blockjob: Simplify block_job_finish_sync Fam Zheng
2015-07-13 23:08 ` John Snow
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 09/15] blockjob: Move BlockJobDeferToMainLoopData into BlockJob Fam Zheng
2015-07-14 10:03 ` Stefan Hajnoczi
2015-07-14 10:36 ` Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 10/15] block: add block job transactions Fam Zheng
2015-07-13 23:12 ` John Snow
2015-07-14 3:04 ` Fam Zheng
2015-07-14 15:05 ` John Snow
2015-07-14 10:27 ` Stefan Hajnoczi
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 11/15] blockdev: make BlockJobTxn available to qmp 'transaction' Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 12/15] block/backup: support block job transactions Fam Zheng
2015-07-13 23:14 ` John Snow
2015-07-14 3:12 ` Fam Zheng
2015-07-14 10:32 ` Stefan Hajnoczi
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 13/15] iotests: 124 - transactional failure test Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 14/15] qmp-commands.hx: Update the supported 'transaction' operations Fam Zheng
2015-07-10 3:46 ` [Qemu-devel] [PATCH v3 15/15] tests: add BlockJobTxn unit test Fam Zheng
2015-07-13 23:14 ` John Snow
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1436500012-32593-1-git-send-email-famz@redhat.com \
--to=famz@redhat.com \
--cc=jcody@redhat.com \
--cc=jsnow@redhat.com \
--cc=kwolf@redhat.com \
--cc=mreitz@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=vsementsov@parallels.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.