From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
To: Christoffer Dall <christofferdall@gmail.com>
Cc: David Vrabel <david.vrabel@citrix.com>,
Wei Liu <wei.liu2@citrix.com>,
Ian Campbell <ian.campbell@citrix.com>,
xen-devel@lists.xen.org
Subject: Re: Dom0 crash with apache bench (ab)
Date: Fri, 31 Jul 2015 11:24:59 +0100 [thread overview]
Message-ID: <alpine.DEB.2.02.1507311122240.11337@kaball.uk.xensource.com> (raw)
In-Reply-To: <CAEDV+gL_e9Ak0_nzaH+k6LQn2+EAOHHoviQ8zKj-xFnC84SwJQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4066 bytes --]
This is a Linux Dom0 crash on x86 (Dell PowerEdge R320, Xeon E5-2450),
CC'ing relevant people. As you can see from the links below the crash
is:
[ 253.619326] Call Trace:
[ 253.619330] <IRQ>
[ 253.619332] [<ffffffff815d7c25>] ? skb_copy_ubufs+0xa5/0x230
[ 253.619347] [<ffffffff815e8525>] __netif_receive_skb_core+0x6f5/0x940
[ 253.619353] [<ffffffff815e8788>] __netif_receive_skb+0x18/0x60
[ 253.619360] [<ffffffff815e87f8>] netif_receive_skb_internal+0x28/0x90
[ 253.619366] [<ffffffff815e91f5>] napi_gro_frags+0x125/0x1a0
[ 253.619378] [<ffffffffa01b1173>] mlx4_en_process_rx_cq+0x753/0xb50 [mlx4_en]
[ 253.619387] [<ffffffffa01b1657>] mlx4_en_poll_rx_cq+0x97/0x160 [mlx4_en]
[ 253.619393] [<ffffffff815e8bcd>] net_rx_action+0x13d/0x2f0
[ 253.619400] [<ffffffff8109fdea>] __do_softirq+0xda/0x1f0
[ 253.619406] [<ffffffff810a013d>] irq_exit+0x9d/0xb0
[ 253.619412] [<ffffffff813e3825>] xen_evtchn_do_upcall+0x35/0x50
[ 253.619420] [<ffffffff816c7bce>] xen_do_hypervisor_callback+0x1e/0x40
[ 253.619423] <EOI>
[ 253.619426] [<ffffffff811a7870>] ? shrink_dcache_for_umount+0x90/0x90
[ 253.619437] [<ffffffff811a7ad9>] ? d_alloc_pseudo+0x9/0x10
[ 253.619443] [<ffffffff815cbbed>] ? sock_alloc_file+0x4d/0x120
[ 253.619448] [<ffffffff815cdf78>] ? SYSC_accept4+0xb8/0x200
[ 253.619454] [<ffffffff811d0377>] ? SyS_epoll_wait+0x87/0xe0
[ 253.619459] [<ffffffff815cf5c9>] ? SyS_accept4+0x9/0x10
[ 253.619465] [<ffffffff816c630d>] ? system_call_fastpath+0x16/0x1b
[ 253.619469] Code: 4e 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 e8 6b fc
ff ff eb e1 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83
e2 07 <f3
> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c
[ 253.619513] RIP [<ffffffff81318b0d>] __memcpy+0xd/0x110
[ 253.619520] RSP <ffff88006b823c60>
[ 253.619524] ---[ end trace ba5d35a466b03856 ]---
On Tue, 28 Jul 2015, Christoffer Dall wrote:
> On Tue, Jul 28, 2015 at 4:55 PM, Ian Campbell <ian.campbell@citrix.com> wrote:
> On Tue, 2015-07-28 at 10:50 -0400, Konrad Rzeszutek Wilk wrote:
> > On Tue, Jul 28, 2015 at 03:09:31PM +0200, Christoffer Dall wrote:
> > > Hi,
> > >
> > > I've been doing some performance comparisons lately, and wanted to
> > > compare
> > > the performance overhead of using Xen with apache bench, but
> > > unfortunately
> > > the Dom0 kernel crashes when hitting it with ab from a remote machine.
> > > Most other workloads seem to be stable, however, I do see similar
> > > crashes
> > > if hitting Dom0 mysql with a mysql benchmark with a high level of
> > > parallelism.
> > >
> > > I use a 10G Mellanox MX354A Dual port FDR CX3 adapter for networking on
> > > a
> > > Dell PowerEdge R320 system with a Xeon E5-2450 and 16 GB of RAM.
> > >
> > > Interestingly, we had a similarly looking issue on arm64 recently, but
> > > that
> > > was fixed with an APM-soecific fix to the hypervisor, so I am guessing
> > > this
> > > is unrelated, see:
> > > http://lists.xenproject.org/archives/html/xen-devel/2015
> > > -03/msg02731.html
> > > and the fix:
> > > http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=50dcb3de603927db2fd
> > > 87ba09e29c817415aaa44
> > >
> > > I have tried with several Linux versions, v3.13, v3.18, v4.0-rc4, and
> > > v4.1,
> > > same issue. I have tried with Xen 4.5-0 release, and the Ubuntu
> > > packaged
> > > Xen 4.4 release, same issue.
> > >
> > > Examples of crash:
> > > http://pastebin.ubuntu.com/11953498/
> > > http://pastebin.ubuntu.com/11953443/
> >
> > 4.0-rc4?
> >
> > Have you tried 4.1?
>
> According to the previous paragraph, yes he has.
>
> yes, I have. Just for clarify, I used 4.0-rc4 because that's a branch which contained arm64 PCI support and has
> been used for other measurements, so this was simply my 'working tree'.
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2015-07-31 10:24 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-07-28 13:09 Dom0 crash with apache bench (ab) Christoffer Dall
2015-07-28 14:50 ` Konrad Rzeszutek Wilk
2015-07-28 14:55 ` Ian Campbell
2015-07-28 15:00 ` Christoffer Dall
2015-07-31 10:24 ` Stefano Stabellini [this message]
2015-07-31 10:28 ` David Vrabel
2015-07-31 13:17 ` Christoffer Dall
2015-09-14 12:40 ` Christoffer Dall
2015-09-14 15:11 ` Konrad Rzeszutek Wilk
2015-09-14 15:20 ` Ian Campbell
2015-09-14 16:16 ` Christoffer Dall
2015-09-28 20:53 ` Christoffer Dall
2015-09-30 15:12 ` Konrad Rzeszutek Wilk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.02.1507311122240.11337@kaball.uk.xensource.com \
--to=stefano.stabellini@eu.citrix.com \
--cc=christofferdall@gmail.com \
--cc=david.vrabel@citrix.com \
--cc=ian.campbell@citrix.com \
--cc=wei.liu2@citrix.com \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.