* Packet duplication in kernels 2.6 onwards
@ 2015-05-04 18:30 Richard Stearn
2015-05-05 4:05 ` David Ranch
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Richard Stearn @ 2015-05-04 18:30 UTC (permalink / raw
To: linux-hams
Just put togther a packet system to test wireshark on and
discovered that packets are being duplicated at the ax0 interface.
My tests so far show that the issue is not present in kernels
2.4.29 & 2.4.37.11 and is present in kernel 2.6.33.3.
I also have reports of the issue being in kernel 3.x.x.x
Is this a known issue?
If so is there any idea as to when it appeared? My guess is during
the 2.5 kernel series.
Has anybody on the list attempted to trace the cause?
--
Regards
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
2015-05-04 18:30 Packet duplication in kernels 2.6 onwards Richard Stearn
@ 2015-05-05 4:05 ` David Ranch
2015-05-06 15:01 ` Patrick Ouellette
[not found] ` <55483EB3.6070600@trinnet.net>
2015-05-26 7:29 ` Ralf Baechle DL5RB
2 siblings, 1 reply; 9+ messages in thread
From: David Ranch @ 2015-05-05 4:05 UTC (permalink / raw
To: Richard Stearn, linux-hams
Hello Richard,
Can you give us some more details? Are the duplicates on every packet?
Is this RX or TX duplicates Only some? Is this unproto or connected?
Are you using a HW TNC or a SW TNC?
I'm using the 3.17.5 kernel w/o issues.
--David
KI6ZHD
On 05/04/2015 11:30 AM, Richard Stearn wrote:
> Just put togther a packet system to test wireshark on and
> discovered that packets are being duplicated at the ax0 interface.
>
> My tests so far show that the issue is not present in kernels
> 2.4.29 & 2.4.37.11 and is present in kernel 2.6.33.3.
> I also have reports of the issue being in kernel 3.x.x.x
>
> Is this a known issue?
>
> If so is there any idea as to when it appeared? My guess is during
> the 2.5 kernel series.
>
> Has anybody on the list attempted to trace the cause?
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
2015-05-05 4:05 ` David Ranch
@ 2015-05-06 15:01 ` Patrick Ouellette
0 siblings, 0 replies; 9+ messages in thread
From: Patrick Ouellette @ 2015-05-06 15:01 UTC (permalink / raw
To: David Ranch; +Cc: Richard Stearn, linux-hams
This has just recently come up in Debian's bug system at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783160https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783160
Pat
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
[not found] ` <55483EB3.6070600@trinnet.net>
@ 2015-05-17 20:28 ` Richard Stearn
2015-05-18 6:15 ` Ugo Poddine
2015-05-20 12:33 ` Richard Stearn
0 siblings, 2 replies; 9+ messages in thread
From: Richard Stearn @ 2015-05-17 20:28 UTC (permalink / raw
To: linux-hams
David Ranch wrote:
> Hello Richard,
>
> Can you give us some more details? Are the duplicates on every packet? Is this
> RX or TX duplicates Only some? Is this unproto or connected? Are you using a
> HW TNC or a SW TNC?
> I'm using the 3.17.5 kernel w/o issues.
Refined the conditions a little.
1) Only occurs on ax25 packets containing IP payloads
2) Occurs in both UI and VC packets
3) Only occurs on packets being sent
4) Does not appear to be sent "to air" by mkiss module,
i.e. the duplicate packet does not appear on the serial line.
5) Is visible in "listen" output
I discovered this as I am documenting the amateur radio protocols in
wireshark. It was also reported to Ubuntu (thank you Patrick) by
Ugo Poddine. In the discussion on the Ubuntu bug report it would appear
that this is known about, hence my posting here to see what was known.
Apparently nothing.
Other things raised by this, in the process of rebuilding my test system
I discovered that the ROSE kernel module did not unload correctly (actually
not at all). I traced this to the device usage counts not being maintained
correctly. I have a patch for this if anybody wants it.
--
Regards
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
2015-05-17 20:28 ` Richard Stearn
@ 2015-05-18 6:15 ` Ugo Poddine
2015-05-20 12:33 ` Richard Stearn
1 sibling, 0 replies; 9+ messages in thread
From: Ugo Poddine @ 2015-05-18 6:15 UTC (permalink / raw
To: linux-hams
Hello everyone, hi Richard,
a couple of remarks, regarding this issue.
a) "duplication" on the air
Indeed I discovered the problem exactly "on the air", at least using
software "TNCs".
I have some logs taken from Direwolf or MIXw2 showing duplication being
sent on the radio channel (I can share, if somebody needs), and also
some reports from Canada (thanks to VE1JOT) reporing practical
impossibility to transmit TCP over AX25 on HF channels due to the
duplication.
b) 6PACK problem
I was curious to understand if the problem affects also 6PACK, but I was
unable to test it : on Debian Wheezy, switching kissattack in 6pack mode
always crashes the kernel, on Jessie it doesn’t, but no data exchange is
possible and - worst - sniffing the interface always show "kiss" frames
instead of 6PACK frames (at least in the ARP dialogue). This happens
also on Fedora21. I opened a ticket on the Jessie side on Saturday.
Thank-you
Regards
73, Ugo
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
2015-05-17 20:28 ` Richard Stearn
2015-05-18 6:15 ` Ugo Poddine
@ 2015-05-20 12:33 ` Richard Stearn
2015-05-26 7:54 ` Ralf Baechle DL5RB
1 sibling, 1 reply; 9+ messages in thread
From: Richard Stearn @ 2015-05-20 12:33 UTC (permalink / raw
To: linux-hams
I think I have got to the bottom of this issue. The good news is that it is
not a bug in the AX25 protocol or driver code, the bad news is that it is
the result of an architectural change in the kernel.
One correction to my original post, the duplication is on receive not
transmit.
In kernels upto and including 2.4 where an AX25 payload was IP or ARP the
packet was handed off by calling the ip or arp protocol driver directly.
In kernels 2.6 onwards after adjusting the packet details the payload is
handed back to the network interface for that interface to reroute the
packet into the IP or ARP protocol drivers.
A couple of code fragments:
from 2.4.37.11:
net/ax25/ax25_in.c
287 case AX25_P_IP:
288 skb_pull(skb,2); /* drop PID/CTRL */
289 skb->h.raw = skb->data;
290 skb->nh.raw = skb->data;
291 skb->dev = dev;
292 skb->pkt_type = PACKET_HOST;
293 skb->protocol = htons(ETH_P_IP);
294 ip_rcv(skb, dev, ptype); /* Note ptype here is
the wrong one, fix me later */
295 break;
296
from 2.6.32.65:
net/ax25/ax25_in.c
244 case AX25_P_IP:
245 skb_pull(skb,2); /* drop PID/CTRL */
246 skb_reset_transport_header(skb);
247 skb_reset_network_header(skb);
248 skb->dev = dev;
249 skb->pkt_type = PACKET_HOST;
250 skb->protocol = htons(ETH_P_IP);
251 netif_rx(skb);
252 break;
I was able to revert kernel 2.6.32.65 to the 2.4 method as ip_rcv & arp_rcv still
exist and it was a simple matter to expose the functions (exporting the symbols and
adding to the header files). Once reverted the duplicated packets in the 2.6 kernel
disappeared.
All the testing has been done using virtual machines linked by a virtual serial link.
Testing was done to establish connections using AX25-UI, AX25-VC, NETROM, ROSE IP/ICMP,
IP/UDP & IP/TCP.
The first setup was of two 2.4.37.11 kernels.
The second setup was of one 2.4.37.11 kernel and one 2.6.32.65 kernel.
The third setup was of one 2.4.37.11 kernel and one 3.2.69 kernel.
For the first setup no duplicates were seen at any point.
For the second setup, duplicates of received packets were seen on the 2.6 kernel
wherever IP was involved. No duplicates were seen on the 2.4 kernel.
For the third setup, duplicates of received packets were seen on the 3.2 kernel
wherever IP was involved. No duplicates were seen on the 2.4 kernel.
These duplicate packets are seen by both "listen" and libpcap (and thus tcpdump or
wireshark). From the protocol analysis point of view any kernel 2.6 onwards is
suspect where encapsulated protocols, such as IP-over-AX25, are concerned.
Notably NETROM and ROSE are not affected, this may be related to the manner in
which those protocols "register" their handlers with AX25. The AX25 protocol
driver considers AX25_P_TEXT, AX25_P_SEGMENT, AX25_P_IP or AX25_P_ARP as
"internal" protocols (ax25/ax25_iface.c). The solution may be to "register"
IP & ARP with AX25 although how to achieve this I can not immediate see.
The above explains why the amateur radio protocols work however when one starts
to look at the packets analytically what is seen at the network interface
for IP & ARP is false.
Of course I could have it totally wrong.
--
Regards
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
2015-05-04 18:30 Packet duplication in kernels 2.6 onwards Richard Stearn
2015-05-05 4:05 ` David Ranch
[not found] ` <55483EB3.6070600@trinnet.net>
@ 2015-05-26 7:29 ` Ralf Baechle DL5RB
2 siblings, 0 replies; 9+ messages in thread
From: Ralf Baechle DL5RB @ 2015-05-26 7:29 UTC (permalink / raw
To: Richard Stearn; +Cc: linux-hams
On Mon, May 04, 2015 at 07:30:36PM +0100, Richard Stearn wrote:
> Just put togther a packet system to test wireshark on and
> discovered that packets are being duplicated at the ax0 interface.
>
> My tests so far show that the issue is not present in kernels
> 2.4.29 & 2.4.37.11 and is present in kernel 2.6.33.3.
> I also have reports of the issue being in kernel 3.x.x.x
>
> Is this a known issue?
>
> If so is there any idea as to when it appeared? My guess is during
> the 2.5 kernel series.
>
> Has anybody on the list attempted to trace the cause?
There has been a number of pretty vague bug reports on the issue that
have been insufficient for me to replicate the issue on my system.
Ralf
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
2015-05-20 12:33 ` Richard Stearn
@ 2015-05-26 7:54 ` Ralf Baechle DL5RB
2015-05-26 15:09 ` Richard Stearn
0 siblings, 1 reply; 9+ messages in thread
From: Ralf Baechle DL5RB @ 2015-05-26 7:54 UTC (permalink / raw
To: Richard Stearn; +Cc: linux-hams
On Wed, May 20, 2015 at 01:33:42PM +0100, Richard Stearn wrote:
> I think I have got to the bottom of this issue. The good news is that it is
> not a bug in the AX25 protocol or driver code, the bad news is that it is
> the result of an architectural change in the kernel.
>
> One correction to my original post, the duplication is on receive not
> transmit.
>
> In kernels upto and including 2.4 where an AX25 payload was IP or ARP the
> packet was handed off by calling the ip or arp protocol driver directly.
> In kernels 2.6 onwards after adjusting the packet details the payload is
> handed back to the network interface for that interface to reroute the
> packet into the IP or ARP protocol drivers.
>
> A couple of code fragments:
>
> from 2.4.37.11:
> net/ax25/ax25_in.c
> 287 case AX25_P_IP:
> 288 skb_pull(skb,2); /* drop PID/CTRL */
> 289 skb->h.raw = skb->data;
> 290 skb->nh.raw = skb->data;
> 291 skb->dev = dev;
> 292 skb->pkt_type = PACKET_HOST;
> 293 skb->protocol = htons(ETH_P_IP);
> 294 ip_rcv(skb, dev, ptype); /* Note
> ptype here is the wrong one, fix me later */
> 295 break;
> 296
>
> from 2.6.32.65:
> net/ax25/ax25_in.c
> 244 case AX25_P_IP:
> 245 skb_pull(skb,2); /* drop PID/CTRL */
> 246 skb_reset_transport_header(skb);
> 247 skb_reset_network_header(skb);
> 248 skb->dev = dev;
> 249 skb->pkt_type = PACKET_HOST;
> 250 skb->protocol = htons(ETH_P_IP);
> 251 netif_rx(skb);
> 252 break;
>
> I was able to revert kernel 2.6.32.65 to the 2.4 method as ip_rcv & arp_rcv still
> exist and it was a simple matter to expose the functions (exporting the symbols and
> adding to the header files). Once reverted the duplicated packets in the 2.6 kernel
> disappeared.
>
> All the testing has been done using virtual machines linked by a virtual serial link.
> Testing was done to establish connections using AX25-UI, AX25-VC, NETROM, ROSE IP/ICMP,
> IP/UDP & IP/TCP.
> The first setup was of two 2.4.37.11 kernels.
> The second setup was of one 2.4.37.11 kernel and one 2.6.32.65 kernel.
> The third setup was of one 2.4.37.11 kernel and one 3.2.69 kernel.
>
> For the first setup no duplicates were seen at any point.
>
> For the second setup, duplicates of received packets were seen on the 2.6 kernel
> wherever IP was involved. No duplicates were seen on the 2.4 kernel.
>
> For the third setup, duplicates of received packets were seen on the 3.2 kernel
> wherever IP was involved. No duplicates were seen on the 2.4 kernel.
>
> These duplicate packets are seen by both "listen" and libpcap (and thus tcpdump or
> wireshark). From the protocol analysis point of view any kernel 2.6 onwards is
> suspect where encapsulated protocols, such as IP-over-AX25, are concerned.
>
> Notably NETROM and ROSE are not affected, this may be related to the manner in
> which those protocols "register" their handlers with AX25. The AX25 protocol
> driver considers AX25_P_TEXT, AX25_P_SEGMENT, AX25_P_IP or AX25_P_ARP as
> "internal" protocols (ax25/ax25_iface.c). The solution may be to "register"
> IP & ARP with AX25 although how to achieve this I can not immediate see.
>
> The above explains why the amateur radio protocols work however when one starts
> to look at the packets analytically what is seen at the network interface
> for IP & ARP is false.
>
> Of course I could have it totally wrong.
I think you're if not spot on at least very close.
In fact when somebody else earlier in this thread said it did only affect
IP traffic it was quite clear where the issue must be. I've only ever
tried to replicate this with non-IP traffic so never got to see the
duplication.
Most of the IP-over-{AX25.NETROM,ROSE} is quite forward. Except ARP which
qualifies as an atrocity to the network stack.
I have hopes that Eric Biederman's patch series may have resolved the issue
but that. It's currently sitting in linux-next and should probably make
4.2.
This might also fix the hard to properly fix ARP-over-ROSE issue that
resulted in a kernel crash.
73 de DL5RB op Ralf
--
Loc. JN47BS / CQ 14 / ITU 28 / DOK A21
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Packet duplication in kernels 2.6 onwards
2015-05-26 7:54 ` Ralf Baechle DL5RB
@ 2015-05-26 15:09 ` Richard Stearn
0 siblings, 0 replies; 9+ messages in thread
From: Richard Stearn @ 2015-05-26 15:09 UTC (permalink / raw
To: Ralf Baechle; +Cc: linux-hams
Ralf Baechle DL5RB wrote:
> In fact when somebody else earlier in this thread said it did only affect
> IP traffic it was quite clear where the issue must be. I've only ever
> tried to replicate this with non-IP traffic so never got to see the
> duplication.
I was working from reports, in the form of capture files, coming from a
different direction, Wireshark.
> This might also fix the hard to properly fix ARP-over-ROSE issue that
> resulted in a kernel crash.
Post 2.5 kernels there is nothing in the ROSE module that is IP or ARP
related. Could this be related to the lack of socket locking and device
use count tracking in the ROSE module? I have published a patch for that.
This weeks job is to raise a kernel bug report and attach the patch to
that.
The packet "duplication" affects more than IP-over-AX25/NETROM/ROSE, I
think it affects IP-over-IP, AX25-over-IP and other packet encapsulating
protocols. Probably anywhere where "netif_rx()" is used because that call
puts the packet back onto the network interface for further processing.
Note: although this is being called "packet duplication" it appears to be
a packet being seen twice by promiscuous tools (listen & libpcap) as it
is sequentially processed, firstly on it's way to the AX25 handler and
then on it's way to the IP handler. It is probably necessary to setup
filtering for "listen" and libpcap to select only AX25 packets rather
than all packets on an AX25 interface.
--
Regards
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-05-26 15:09 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-05-04 18:30 Packet duplication in kernels 2.6 onwards Richard Stearn
2015-05-05 4:05 ` David Ranch
2015-05-06 15:01 ` Patrick Ouellette
[not found] ` <55483EB3.6070600@trinnet.net>
2015-05-17 20:28 ` Richard Stearn
2015-05-18 6:15 ` Ugo Poddine
2015-05-20 12:33 ` Richard Stearn
2015-05-26 7:54 ` Ralf Baechle DL5RB
2015-05-26 15:09 ` Richard Stearn
2015-05-26 7:29 ` Ralf Baechle DL5RB
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.