From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33883) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z2zgR-0000Mg-Fa for qemu-devel@nongnu.org; Thu, 11 Jun 2015 06:26:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z2zgL-0003Dc-RT for qemu-devel@nongnu.org; Thu, 11 Jun 2015 06:26:03 -0400 Received: from mx2.parallels.com ([199.115.105.18]:34403) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z2zgL-0003DP-DP for qemu-devel@nongnu.org; Thu, 11 Jun 2015 06:25:57 -0400 Message-ID: <55796228.9070708@virtuozzo.com> Date: Thu, 11 Jun 2015 13:25:44 +0300 From: Vladimir Sementsov-Ogievskiy MIME-Version: 1.0 References: <1433776886-27239-1-git-send-email-vsementsov@virtuozzo.com> <1433776886-27239-2-git-send-email-vsementsov@virtuozzo.com> <20150610153400.GG4899@noname.str.redhat.com> In-Reply-To: <20150610153400.GG4899@noname.str.redhat.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 1/8] spec: add qcow2-dirty-bitmaps specification List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kevin Wolf Cc: qemu-devel@nongnu.org, Vladimir Sementsov-Ogievskiy , stefanha@redhat.com, pbonzini@redhat.com, den@openvz.org, jsnow@redhat.com On 10.06.2015 18:34, Kevin Wolf wrote: > Am 08.06.2015 um 17:21 hat Vladimir Sementsov-Ogievskiy geschrieben: >> From: Vladimir Sementsov-Ogievskiy >> >> Persistent dirty bitmaps will be saved into qcow2 files. It may be used >> as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for >> other drives (there may be qcow2 file with zero disk size but with >> several dirty bitmaps for other drives). >> >> Signed-off-by: Vladimir Sementsov-Ogievskiy >> --- >> docs/specs/qcow2.txt | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 66 insertions(+) >> >> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt >> index 121dfc8..0fffba2 100644 >> --- a/docs/specs/qcow2.txt >> +++ b/docs/specs/qcow2.txt >> @@ -123,6 +123,7 @@ be stored. Each extension has a structure like the following: >> 0x00000000 - End of the header extension area >> 0xE2792ACA - Backing file format name >> 0x6803f857 - Feature name table >> + 0x23852875 - Dirty bitmaps >> other - Unknown header extension, can be safely >> ignored >> >> @@ -166,6 +167,19 @@ the header extension data. Each entry look like this: >> terminated if it has full length) >> >> >> +== Dirty bitmaps == >> + >> +Dirty bitmaps is an optional header extension. It provides a possibility of >> +storing dirty bitmaps in qcow2 image. The fields are: >> + >> + 0 - 3: nb_dirty_bitmaps >> + Number of dirty bitmaps contained in the image >> + >> + 4 - 11: dirty_bitmaps_offset >> + Offset into the image file at which the dirty bitmaps table >> + starts. Must be aligned to a cluster boundary. >> + >> + >> == Host cluster management == > You need to use a compatibility flag because for old qemu versions, the > dirty bitmaps (and associated metadata) are leaked clusters and qemu-img > check would "repair" them by resetting the refcount to 0. > > At second sight, I see that your patches add an autoclear flag. > Presumably the contents of the dirty bitmaps is outdated when you > accessed the image with an older version, so this seems right. We just > need to document it. > >> qcow2 manages the allocation of host clusters by maintaining a reference count >> @@ -360,3 +374,55 @@ Snapshot table entry: >> >> variable: Padding to round up the snapshot table entry size to the >> next multiple of 8. >> + >> + >> +== Dirty bitmaps == >> + >> +The feature supports storing several dirty bitmaps in the qcow2 file. >> + >> +=== Cluster mapping === >> + >> +Dirty bitmaps are stored using a ONE-level structure for the mapping of >> +bitmaps to host clusters. There is only an L1 table. >> + >> +The L1 table has a variable size (stored in the Bitmap table entry) and may >> +use multiple clusters, however it must be contiguous in the image file. >> + >> +Given an offset into the bitmap, the offset into the image file can be >> +obtained as follows: >> + >> + offset = l1_table[offset / cluster_size] + (offset % cluster_size) >> + >> +L1 table entry: >> + >> + Bit 0 - 61: Standard cluster descriptor >> + >> + 62 - 63: Reserved > Stefan already mentioned that we don't have a "L1" when there is only > one level, and that you shouldn't reuse the cluster descriptors from L2 > tables. > >> +=== Bitmap table === >> + >> +A directory of all bitmaps is stored in the bitmap table, a contiguous area in >> +the image file, whose starting offset and length are given by the header fields >> +dirty_bitmaps_offset and nb_dirty_bitmaps. The entries of the bitmap table have >> +variable length, depending on the length of name and extra data. >> + >> +Bitmap table entry: >> + >> + Byte 0 - 7: Offset into the image file at which the L1 table for the >> + bitmap starts. Must be aligned to a cluster boundary. >> + >> + 8 - 11: Number of entries in the L1 table of the bitmap > Worth using 64 bits here? This can only cover 4 * 512 GB = 2 TB for the > smallest possible cluster size. Though it's 65536 * 512 = 32 PB for the > default, which might be enough for a while. > >> + 12 - 15: Bitmap granularity in bytes >> + >> + 16 - 23: Bitmap size in sectors > Please don't use sectors, that's a meaningless unit. Bytes is better. Just bad description. Actually it is ~ (number of bits in bitmap * granularity), and it is corresponding to number of sectors in the image. > >> + 24 - 25: Size of the bitmap name > We should use a smaller limit than the possible 64k to avoid too large > memory allocations. Nobody needs really long bitmap names. > >> + >> + variable: The name of the bitmap (not null terminated) >> + >> + variable: Padding to round up the bitmap table entry size to the >> + next multiple of 8. > Kevin -- Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.