From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39612) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZDXER-0006l0-6d for qemu-devel@nongnu.org; Fri, 10 Jul 2015 08:16:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZDXEP-00021E-Jj for qemu-devel@nongnu.org; Fri, 10 Jul 2015 08:16:43 -0400 Date: Fri, 10 Jul 2015 14:16:33 +0200 (CEST) From: Alexandre DERUMIER Message-ID: <41945133.9627108.1436530593237.JavaMail.zimbra@oxygem.tv> In-Reply-To: <534630199.9583032.1436524600797.JavaMail.zimbra@oxygem.tv> References: <1436413678-7114-1-git-send-email-famz@redhat.com> <20150709130208.GD11166@stefanha-thinkpad.redhat.com> <20150710064350.GG31230@ad.nay.redhat.com> <278695903.9362838.1436511286250.JavaMail.zimbra@oxygem.tv> <20150710071333.GA16297@ad.nay.redhat.com> <534630199.9583032.1436524600797.JavaMail.zimbra@oxygem.tv> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest responsiveness during bitmap scan List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: Kevin Wolf , Stefan Hajnoczi , qemu-devel , qemu-block@nongnu.org Hi again, I have redone a lot of tests, with raw on nfs --------------- Patch 3/3, fix my problem (not apply patch 1/3 and patch 2/3). without patch 3/3, I'm seeing a lot of lseek, can take some minutes with gu= est hang. with patch 3/3, it almost take no time to generate the bitmap, no guest han= g . (I have tested with differents raw files, different sizes) with raw on ext3 ---------------- Here, this hurt a lot. Not with every disk, but with some disks is totaly h= anging the guest ----------------------- PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND= =20 4387 root 20 0 8985788 288900 12532 R 99.3 1.8 2:50.82 qemu = =20 10.27% [kernel] [k] ext4_es_find_delayed_extent_range 9.64% [kernel] [k] _raw_read_lock 9.17% [kernel] [k] ext4_es_lookup_extent 6.63% [kernel] [k] down_read 5.49% [kernel] [k] ext4_map_blocks 4.82% [kernel] [k] up_read 3.09% [kernel] [k] ext4_ind_map_blocks 2.58% [kernel] [k] ext4_block_to_path.isra.9 2.18% [kernel] [k] kvm_arch_vcpu_ioctl_run 1.99% [kernel] [k] vmx_get_segment 1.99% [kernel] [k] ext4_llseek 1.62% [kernel] [k] ext4_get_branch 1.43% [kernel] [k] vmx_segment_access_rights.isra.28.p= art.29 1.23% [kernel] [k] x86_decode_insn 1.00% [kernel] [k] copy_user_generic_string 0.83% [kernel] [k] x86_emulate_instruction 0.81% [kernel] [k] rmode_segment_valid 0.78% [kernel] [k] gfn_to_hva_prot 0.74% [kernel] [k] __es_tree_search 0.73% [kernel] [k] do_insn_fetch 0.70% [kernel] [k] vmx_vcpu_run 0.69% [kernel] [k] vmx_read_guest_seg_selector 0.67% [kernel] [k] vmcs_writel 0.64% [kernel] [k] init_emulate_ctxt 0.62% [kernel] [k] native_write_msr_safe 0.61% [kernel] [k] kvm_apic_has_interrupt 0.57% [kernel] [k] vmx_handle_external_intr 0.57% [kernel] [k] x86_emulate_insn 0.56% [kernel] [k] vmx_handle_exit 0.50% [kernel] [k] native_read_tsc 0.47% [kernel] [k] _cond_resched 0.46% [kernel] [k] decode_operand 0.44% [kernel] [k] __srcu_read_lock 0.41% [kernel] [k] __linearize.isra.25 0.41% [kernel] [k] vmx_read_l1_tsc 0.39% [kernel] [k] clear_page_c 0.39% [kernel] [k] vmx_get_interrupt_shadow 0.36% [kernel] [k] vmx_set_interrupt_shadow Seem to be a known kernel bug: http://qemu.11.n7.nabble.com/Re-2-Strange-problems-with-lseek-in-qemu-img-m= ap-td334414.html http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id= =3D14516bb7bb6ffbd49f35389f9ece3b2045ba5815 So, I think patch 3/3 is enough. (could be great to have it for qemu 2.4) Thanks again Fam for your hard work. Regards, Alexandre ----- Mail original ----- De: "aderumier" =C3=80: "Fam Zheng" Cc: "Kevin Wolf" , "Stefan Hajnoczi" = , "qemu-devel" , qemu-block@nongnu.org Envoy=C3=A9: Vendredi 10 Juillet 2015 12:36:40 Objet: Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest responsi= veness during bitmap scan >>Does it completely hang?=20 yes, can't ping network, and vnc console is frozen.=20 >>What does "perf top" show?=20 I'll do test today . (I'm going to vacation this night,I'll try to send res= ults before that)=20 ----- Mail original -----=20 De: "Fam Zheng" =20 =C3=80: "aderumier" =20 Cc: "Stefan Hajnoczi" , "Kevin Wolf" = , "qemu-devel" , qemu-block@nongnu.org=20 Envoy=C3=A9: Vendredi 10 Juillet 2015 09:13:33=20 Objet: Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest responsi= veness during bitmap scan=20 On Fri, 07/10 08:54, Alexandre DERUMIER wrote:=20 > >>Thinking about this again, I doubt=20 > >>that lengthening the duration with a hardcoded value benifits everyone;= and=20 > >>before Alexandre's reply on old server/slow disks=20 >=20 > With 1ms sleep, I can reproduce the hang 100% with a fast cpu (xeon e5 v3= 3,1ghz) and source raw file on nfs.=20 Does it completely hang? I can't reproduce this with my machine mirroring f= rom=20 nfs to local: the guest runs smoothly. What does "perf top" show?=20 Fam=20 >=20 >=20 > ----- Mail original -----=20 > De: "Fam Zheng" =20 > =C3=80: "Stefan Hajnoczi" =20 > Cc: "Kevin Wolf" , "qemu-devel" = , qemu-block@nongnu.org=20 > Envoy=C3=A9: Vendredi 10 Juillet 2015 08:43:50=20 > Objet: Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest respon= siveness during bitmap scan=20 >=20 > On Thu, 07/09 14:02, Stefan Hajnoczi wrote:=20 > > This patch only converts the mirror block job to use the new relax=20 > > function. The other block jobs (stream, backup, commit) are still using= =20 > > a 0 ns delay and are therefore broken. They should probably be=20 > > converted in the same series.=20 >=20 > That's because they all can be perfectly mitigated by setting a reasonabl= e=20 > "speed" that matchs the host horsepower. Thinking about this again, I dou= bt=20 > that lengthening the duration with a hardcoded value benifits everyone; a= nd=20 > before Alexandre's reply on old server/slow disks, I don't recall any rep= ort,=20 > because the coroutines would already yield often enough, when waiting for= IO to=20 > complete. So I am not sure whether they're broken in practice.=20 >=20 > Fam=20