* Debugging SR-IOV related packet flow problem with DPDK
@ 2015-06-16 12:33 Anjali Kulkarni
2015-06-16 16:09 ` Anjali Kulkarni
0 siblings, 1 reply; 7+ messages in thread
From: Anjali Kulkarni @ 2015-06-16 12:33 UTC (permalink / raw)
To: dev@dpdk.org
Hi,
I am observing that the SR-IOV enabled NIC does not increment any VF packet counters despite incoming packet flow. I have enabled some debug logs like CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
Can someone point me to where these logs are being printed so I can try to figure out where packets are being dropped? Or how can I debug DPDK packet flow related issues (are there any stats I can look at)?
Thanks
Anjali
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Debugging SR-IOV related packet flow problem with DPDK
2015-06-16 12:33 Debugging SR-IOV related packet flow problem with DPDK Anjali Kulkarni
@ 2015-06-16 16:09 ` Anjali Kulkarni
2015-06-16 20:22 ` Anjali Kulkarni
0 siblings, 1 reply; 7+ messages in thread
From: Anjali Kulkarni @ 2015-06-16 16:09 UTC (permalink / raw)
To: dev@dpdk.org, Anjali Kulkarni; +Cc: Anjali Kulkarni
Pls keep me on cc for response. Thanks!!
Hi,
I am observing that the SR-IOV enabled NIC does not increment any VF packet counters despite incoming packet flow. I have enabled some debug logs like CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
Can someone point me to where these logs are being printed so I can try to figure out where packets are being dropped? Or how can I debug DPDK packet flow related issues (are there any stats I can look at)?
Thanks
Anjali
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Debugging SR-IOV related packet flow problem with DPDK
2015-06-16 16:09 ` Anjali Kulkarni
@ 2015-06-16 20:22 ` Anjali Kulkarni
2015-06-16 20:52 ` Thomas Monjalon
0 siblings, 1 reply; 7+ messages in thread
From: Anjali Kulkarni @ 2015-06-16 20:22 UTC (permalink / raw)
To: dev@dpdk.org
Hi,
Can someone please help with this? How can packet flow be traced on DPDK? Where are all the logs stored if we do enable the _DEBUG configs in config/ directory?
Anjali
From: Microsoft Office User <sumerj@juniper.net<mailto:sumerj@juniper.net>>
Date: Tuesday, June 16, 2015 at 9:10 AM
To: "dev@dpdk.org<mailto:dev@dpdk.org>" <dev@dpdk.org<mailto:dev@dpdk.org>>, Microsoft Office User <sumerj@juniper.net<mailto:sumerj@juniper.net>>
Cc: Microsoft Office User <sumerj@juniper.net<mailto:sumerj@juniper.net>>
Subject: Re: Debugging SR-IOV related packet flow problem with DPDK
Pls keep me on cc for response. Thanks!!
Hi,
I am observing that the SR-IOV enabled NIC does not increment any VF packet counters despite incoming packet flow. I have enabled some debug logs like CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
Can someone point me to where these logs are being printed so I can try to figure out where packets are being dropped? Or how can I debug DPDK packet flow related issues (are there any stats I can look at)?
Thanks
Anjali
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Debugging SR-IOV related packet flow problem with DPDK
2015-06-16 20:22 ` Anjali Kulkarni
@ 2015-06-16 20:52 ` Thomas Monjalon
2015-06-16 21:08 ` Anjali Kulkarni
0 siblings, 1 reply; 7+ messages in thread
From: Thomas Monjalon @ 2015-06-16 20:52 UTC (permalink / raw)
To: Anjali Kulkarni; +Cc: dev
Hi,
That's third email in 8 hours for this question.
Do you plan to send an email every 4 hours?
It's really hard to help you without knowing which application you run?
in which VM? which VLAN configuration? etc
2015-06-16 20:22, Anjali Kulkarni:
> Hi,
>
> Can someone please help with this? How can packet flow be traced on DPDK?
> Where are all the logs stored if we do enable the _DEBUG configs in
> config/ directory?
stdout or somewhere else if you configured logs in your app.
> Anjali
[...]
> I am observing that the SR-IOV enabled NIC does not increment any VF packet
> counters despite incoming packet flow. I have enabled some debug logs like
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
> Can someone point me to where these logs are being printed so I can try to
> figure out where packets are being dropped? Or how can I debug DPDK packet
> flow related issues (are there any stats I can look at)?
Statistics can be requested through a dedicated ethdev API.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Debugging SR-IOV related packet flow problem with DPDK
2015-06-16 20:52 ` Thomas Monjalon
@ 2015-06-16 21:08 ` Anjali Kulkarni
2015-06-17 10:52 ` Anjali Kulkarni
0 siblings, 1 reply; 7+ messages in thread
From: Anjali Kulkarni @ 2015-06-16 21:08 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev@dpdk.org
Thanks for your response.
I am running my own application, and it is actually running inside a
docker, not a VM.
No VLAN configuration. Just SR-IOV is enabled on 2 NICs, and both devices
(/dev/uio0 and /dev/uio1) are assigned to the docker.
This worked a while ago perfectly.
What I see is that an ixia connected to the 2 10G NICs is pumping traffic,
but the VF statistics are 0. It is as if the VF or NIC did not see the
packets.
The source dst MAC is set correctly to the VF¹s MAC. I don¹t see any
prints on stdout after enabling the _DEBUG logs in config.
Anjali
On 6/16/15, 1:52 PM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:
>Hi,
>
>That's third email in 8 hours for this question.
>Do you plan to send an email every 4 hours?
>
>It's really hard to help you without knowing which application you run?
>in which VM? which VLAN configuration? etc
>
>2015-06-16 20:22, Anjali Kulkarni:
>> Hi,
>>
>> Can someone please help with this? How can packet flow be traced on
>>DPDK?
>> Where are all the logs stored if we do enable the _DEBUG configs in
>> config/ directory?
>
>stdout or somewhere else if you configured logs in your app.
>
>> Anjali
>[...]
>> I am observing that the SR-IOV enabled NIC does not increment any VF
>>packet
>> counters despite incoming packet flow. I have enabled some debug logs
>>like
>> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
>> Can someone point me to where these logs are being printed so I can try
>>to
>> figure out where packets are being dropped? Or how can I debug DPDK
>>packet
>> flow related issues (are there any stats I can look at)?
>
>Statistics can be requested through a dedicated ethdev API.
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Debugging SR-IOV related packet flow problem with DPDK
2015-06-16 21:08 ` Anjali Kulkarni
@ 2015-06-17 10:52 ` Anjali Kulkarni
2015-06-17 15:16 ` Andrew Harvey (agh)
0 siblings, 1 reply; 7+ messages in thread
From: Anjali Kulkarni @ 2015-06-17 10:52 UTC (permalink / raw)
To: dev@dpdk.org
Folks,
I am seeing this in the dmesg (VF 0 Hung DMA)
[586307.789098] VF 0 setting Multicast Promiscuos Enable TRUE
[586307.789099] Register before write : 0x9000000
[586307.789100] Register after write : 0x19000000
[586307.802585] dmar: DRHD: handling fault status reg 502
[586307.802597] dmar: DMAR:[DMA Read] Request device [06:10.0] fault addr
7c820000
[563607.533814] DMAR:[fault reason 02] Present bit in context entry is
clear
[563607.544246] ixgbe 0000:06:00.1 int3: VF 0 Hung DMA
[563608.344639] docker0: port 1(veth261ed02) entered forwarding state
[563711.695571] VF 0 setting Multicast Promiscuos Enable FALSE
[563711.695578] Register before write : 0x19000000
[563711.695581] Register after write : 0x9000000
[563711.695752] VF 0 setting Multicast Promiscuos Enable FALSE
[563711.695758] Register before write : 0x19000000
[563711.695761] Register after write : 0x9000000
I also see some bootp and ipv6 traffic when I do tcpdump on the PF NIC.
I am thinking perhaps DPDK has not completely taken over the nIC, and
perhaps the linux driver is also somehow interacting?
For intents, my DPDK app is running on the host (actually inside docker)
and NOT inside a VM.
Anjali
On 6/16/15, 2:08 PM, "Anjali Kulkarni" <anjali@juniper.net> wrote:
>Thanks for your response.
>I am running my own application, and it is actually running inside a
>docker, not a VM.
>No VLAN configuration. Just SR-IOV is enabled on 2 NICs, and both devices
>(/dev/uio0 and /dev/uio1) are assigned to the docker.
>This worked a while ago perfectly.
>What I see is that an ixia connected to the 2 10G NICs is pumping traffic,
>but the VF statistics are 0. It is as if the VF or NIC did not see the
>packets.
>The source dst MAC is set correctly to the VF¹s MAC. I don¹t see any
>prints on stdout after enabling the _DEBUG logs in config.
>
>Anjali
>
>On 6/16/15, 1:52 PM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:
>
>>Hi,
>>
>>That's third email in 8 hours for this question.
>>Do you plan to send an email every 4 hours?
>>
>>It's really hard to help you without knowing which application you run?
>>in which VM? which VLAN configuration? etc
>>
>>2015-06-16 20:22, Anjali Kulkarni:
>>> Hi,
>>>
>>> Can someone please help with this? How can packet flow be traced on
>>>DPDK?
>>> Where are all the logs stored if we do enable the _DEBUG configs in
>>> config/ directory?
>>
>>stdout or somewhere else if you configured logs in your app.
>>
>>> Anjali
>>[...]
>>> I am observing that the SR-IOV enabled NIC does not increment any VF
>>>packet
>>> counters despite incoming packet flow. I have enabled some debug logs
>>>like
>>> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
>>> Can someone point me to where these logs are being printed so I can try
>>>to
>>> figure out where packets are being dropped? Or how can I debug DPDK
>>>packet
>>> flow related issues (are there any stats I can look at)?
>>
>>Statistics can be requested through a dedicated ethdev API.
>>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Debugging SR-IOV related packet flow problem with DPDK
2015-06-17 10:52 ` Anjali Kulkarni
@ 2015-06-17 15:16 ` Andrew Harvey (agh)
0 siblings, 0 replies; 7+ messages in thread
From: Andrew Harvey (agh) @ 2015-06-17 15:16 UTC (permalink / raw)
To: Anjali Kulkarni, dev@dpdk.org
Please type the following:
dmesg | grep DMAR. I think you will find that rebinding the uio driver
has caused issues, uio know nothing about the iommu. If you can you may
have better luck with the vfio driver.
HTH ― Andy
On 6/17/15, 3:52 AM, "dev on behalf of Anjali Kulkarni"
<dev-bounces@dpdk.org on behalf of anjali@juniper.net> wrote:
>Folks,
>
>I am seeing this in the dmesg (VF 0 Hung DMA)
>
>[586307.789098] VF 0 setting Multicast Promiscuos Enable TRUE
>[586307.789099] Register before write : 0x9000000
>[586307.789100] Register after write : 0x19000000
>[586307.802585] dmar: DRHD: handling fault status reg 502
>[586307.802597] dmar: DMAR:[DMA Read] Request device [06:10.0] fault addr
>7c820000
>
>
>[563607.533814] DMAR:[fault reason 02] Present bit in context entry is
>clear
>[563607.544246] ixgbe 0000:06:00.1 int3: VF 0 Hung DMA
>[563608.344639] docker0: port 1(veth261ed02) entered forwarding state
>[563711.695571] VF 0 setting Multicast Promiscuos Enable FALSE
>[563711.695578] Register before write : 0x19000000
>[563711.695581] Register after write : 0x9000000
>[563711.695752] VF 0 setting Multicast Promiscuos Enable FALSE
>[563711.695758] Register before write : 0x19000000
>[563711.695761] Register after write : 0x9000000
>
>I also see some bootp and ipv6 traffic when I do tcpdump on the PF NIC.
>
>I am thinking perhaps DPDK has not completely taken over the nIC, and
>perhaps the linux driver is also somehow interacting?
>
>For intents, my DPDK app is running on the host (actually inside docker)
>and NOT inside a VM.
>
>Anjali
>
>On 6/16/15, 2:08 PM, "Anjali Kulkarni" <anjali@juniper.net> wrote:
>
>>Thanks for your response.
>>I am running my own application, and it is actually running inside a
>>docker, not a VM.
>>No VLAN configuration. Just SR-IOV is enabled on 2 NICs, and both devices
>>(/dev/uio0 and /dev/uio1) are assigned to the docker.
>>This worked a while ago perfectly.
>>What I see is that an ixia connected to the 2 10G NICs is pumping
>>traffic,
>>but the VF statistics are 0. It is as if the VF or NIC did not see the
>>packets.
>>The source dst MAC is set correctly to the VF¹s MAC. I don¹t see any
>>prints on stdout after enabling the _DEBUG logs in config.
>>
>>Anjali
>>
>>On 6/16/15, 1:52 PM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:
>>
>>>Hi,
>>>
>>>That's third email in 8 hours for this question.
>>>Do you plan to send an email every 4 hours?
>>>
>>>It's really hard to help you without knowing which application you run?
>>>in which VM? which VLAN configuration? etc
>>>
>>>2015-06-16 20:22, Anjali Kulkarni:
>>>> Hi,
>>>>
>>>> Can someone please help with this? How can packet flow be traced on
>>>>DPDK?
>>>> Where are all the logs stored if we do enable the _DEBUG configs in
>>>> config/ directory?
>>>
>>>stdout or somewhere else if you configured logs in your app.
>>>
>>>> Anjali
>>>[...]
>>>> I am observing that the SR-IOV enabled NIC does not increment any VF
>>>>packet
>>>> counters despite incoming packet flow. I have enabled some debug logs
>>>>like
>>>> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
>>>> Can someone point me to where these logs are being printed so I can
>>>>try
>>>>to
>>>> figure out where packets are being dropped? Or how can I debug DPDK
>>>>packet
>>>> flow related issues (are there any stats I can look at)?
>>>
>>>Statistics can be requested through a dedicated ethdev API.
>>>
>>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-06-17 15:16 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-16 12:33 Debugging SR-IOV related packet flow problem with DPDK Anjali Kulkarni
2015-06-16 16:09 ` Anjali Kulkarni
2015-06-16 20:22 ` Anjali Kulkarni
2015-06-16 20:52 ` Thomas Monjalon
2015-06-16 21:08 ` Anjali Kulkarni
2015-06-17 10:52 ` Anjali Kulkarni
2015-06-17 15:16 ` Andrew Harvey (agh)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.