From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50300) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZDPHD-0006tM-Or for qemu-devel@nongnu.org; Thu, 09 Jul 2015 23:47:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZDPHA-0006py-Gw for qemu-devel@nongnu.org; Thu, 09 Jul 2015 23:47:03 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36162) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZDPHA-0006pp-9I for qemu-devel@nongnu.org; Thu, 09 Jul 2015 23:47:00 -0400 From: Fam Zheng Date: Fri, 10 Jul 2015 11:46:37 +0800 Message-Id: <1436500012-32593-1-git-send-email-famz@redhat.com> Subject: [Qemu-devel] [PATCH v3 00/15] block: incremental backup transactions using BlockJobTxn List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: Kevin Wolf , famz@redhat.com, John Snow , Jeff Cody , Max Reitz , vsementsov@parallels.com, stefanha@redhat.com v3: Simplify the code a bit to implement the idea as discussed in v2 thread: https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg02130.html Unchanged: [01/15] qapi: Add transaction support to block-dirty-bitmap operations [02/15] iotests: add transactional incremental backup test [03/15] block: rename BlkTransactionState and BdrvActionOps [04/15] block: keep bitmap if incremental backup job is cancelled New patch to refactor out dirty bitmap reclaimation code in backup: [05/15] backup: Extract dirty bitmap handling as a separate function Two new transaction specific operations for block job driver: [06/15] blockjob: Add .txn_commit and .txn_abort transaction actions Two new fields that will make managing state of block job easier: [07/15] blockjob: Add "completed" and "ret" in BlockJob New. Necessary to avoid race conditions between jobs: [08/15] blockjob: Simplify block_job_finish_sync [09/15] blockjob: Move BlockJobDeferToMainLoopData into BlockJob Partly rewrote the implementation: [10/15] block: add block job transactions Slightly adjust to the new API (no block_job_txn_begin): [11/15] blockdev: make BlockJobTxn available to qmp 'transaction' Slightly adjust to the new API (no block_job_txn_job_done). [12/15] block/backup: support block job transactions Unchanged: [13/15] iotests: 124 - transactional failure test [14/15] qmp-commands.hx: Update the supported 'transaction' operations [15/15] tests: add BlockJobTxn unit test --- Original cover letter from Stefan's v2, note that block_job_txn_begin and block_job_txn_job_done are not necessary now: v2: Patch 5: * Set txn pointer to NULL in block_job_txn_begin() [jsnow] * Rename block_job_txn_job_done to block_job_txn_job_done [jsnow] * Rename block_job_txn_complete to block_job_txn_kick [jsnow] * Add BLOCK_JOB_TXN_CANCEL_PENDING to solve race condition on cancel [jsnow] * Document when txn may be NULL Patch 7: * Convert blockdev-backup in addition to drive-backup * Add 'transactional-cancel' argument so users can enable/disable transactional behavior. This preserves semantics for existing users and allows fine-grained control over when to use transaction semantics. [jsnow] Patch 10: * Test fail/cancel race condition [jsnow] * Use new 'transactional-cancel' QMP argument This series uses patches from John Snow's "[PATCH v6 00/10] block: incremental backup transactions" series but implements the feature with a new transaction mechanism for blockjobs called BlockJobTxn. Recap: motivation for block job transactions -------------------------------------------- If an incremental backup block job fails then we reclaim the bitmap so the job can be retried. The problem comes when multiple jobs are started as part of a qmp 'transaction' command. We need to group these jobs in a transaction so that either all jobs complete successfully or all bitmaps are reclaimed. Without transactions, there is a case where some jobs complete successfully and throw away their bitmaps, making it impossible to retry the backup by rerunning the command if one of the jobs fails. How does this implementation work? ---------------------------------- These patches add a BlockJobTxn object with the following API: txn = block_job_txn_new(); block_job_txn_add_job(txn, job1); block_job_txn_add_job(txn, job2); block_job_txn_begin(); The jobs either both complete successfully or they both fail/cancel. If the user cancels job1 then job2 will also be cancelled and vice versa. Jobs stay alive waiting for other jobs to complete. They can be cancelled by the user during this time. Job blockers are still in effect and no other block job can run on this device in the meantime (since QEMU currently only allows 1 job per device). This is the main drawback to this approach but reasonable since you probably don't want to run other jobs/operations until you're sure the backup was successful (you won't be able to retry a failed backup if there's a new job running). Adding transaction support to the backup job is very easy. It just needs to make a call before throwing away the bitmap and returning from its coroutine: block_job_txn_job_done(job->txn, job, ret); if (job->sync_bitmap) { BdrvDirtyBitmap *bm; if (ret < 0 || block_job_is_cancelled(&job->common)) { ... Fam Zheng (5): backup: Extract dirty bitmap handling as a separate function blockjob: Add .txn_commit and .txn_abort transaction actions blockjob: Add "completed" and "ret" in BlockJob blockjob: Simplify block_job_finish_sync blockjob: Move BlockJobDeferToMainLoopData into BlockJob John Snow (4): qapi: Add transaction support to block-dirty-bitmap operations iotests: add transactional incremental backup test block: rename BlkTransactionState and BdrvActionOps iotests: 124 - transactional failure test Kashyap Chamarthy (1): qmp-commands.hx: Update the supported 'transaction' operations Stefan Hajnoczi (5): block: keep bitmap if incremental backup job is cancelled block: add block job transactions blockdev: make BlockJobTxn available to qmp 'transaction' block/backup: support block job transactions tests: add BlockJobTxn unit test block.c | 19 ++- block/backup.c | 49 ++++-- blockdev.c | 360 ++++++++++++++++++++++++++++++++++++--------- blockjob.c | 146 +++++++++++++----- docs/bitmaps.md | 6 +- hmp.c | 2 +- include/block/block.h | 2 +- include/block/block_int.h | 6 +- include/block/blockjob.h | 63 +++++++- qapi-schema.json | 6 +- qapi/block-core.json | 14 +- qmp-commands.hx | 21 ++- tests/Makefile | 3 + tests/qemu-iotests/124 | 182 ++++++++++++++++++++++- tests/qemu-iotests/124.out | 4 +- tests/test-blockjob-txn.c | 228 ++++++++++++++++++++++++++++ 16 files changed, 963 insertions(+), 148 deletions(-) create mode 100644 tests/test-blockjob-txn.c -- 2.4.3