buglwIP - A Lightweight TCP/IP stack - Bugs: bug #24493, Dropped incoming packets cause a...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #24493: Dropped incoming packets cause a burst of duplicate ACKs

Submitter:  Bill Auerbach <billauerbach>
Submitted:  Wed 08 Oct 2008 10:09:48 PM UTC
   
 
Category:  TCP Severity:  3 - Normal
Item Group:  Faulty Behaviour Status:  Invalid
Privacy:  Public Assigned to:  None
Open/Closed:  Closed Planned Release:  None
lwIP version:  CVS Head

Jump to the original submission

Wed 06 May 2009 05:26:47 PM UTC, comment #12: 


> I'm not duplicating the not recovering problem - you can close it.


I hope it stays like that. Thanks for checking.

Simon Goldschmidt <goldsimon>
Group administrator
Wed 06 May 2009 02:09:59 PM UTC, comment #11: 

I'm not duplicating the not recovering problem - you can close it.

Bill Auerbach <billauerbach>
Wed 22 Apr 2009 10:35:49 AM UTC, comment #10: 


> I'd really like to make sure it's at least understood before that release


I understand that, and I also think a complete stall is a serious bug, but without more info, there is no way to understand what's happening here.

Simon Goldschmidt <goldsimon>
Group administrator
Wed 22 Apr 2009 08:06:12 AM UTC, comment #9: 

I marked this for 1.3.1 because I'd really like to make sure it's at least understood before that release.  If there's a situation where connections can completely stall it's a serious bug, but at the moment we're not really sure.

Lets leave it for 1.4.0 for now, but if Bill can get some more data on this bug I'd be pleased to see it understood and fixed sooner.

Kieran Mansley <kieranm>
Group Member
Wed 22 Apr 2009 07:20:59 AM UTC, comment #8: 

Ok, awaiting further investigations, I've changed the planned release to 1.4.0.

Simon Goldschmidt <goldsimon>
Group administrator
Tue 21 Apr 2009 09:26:42 PM UTC, comment #7: 

Simon, I didn't capture at the right times to see the stall because I was after the source of dropped packets, which were real, as they caused us to have real-time issues.  The stall happened only a few times over dozens of runs to find the pattern and cause of the dropped packets.  Once I figured out how to stop the dropped packets, everything was (and has been) error-free.

Not recovering meant all transmissions stopped.  My guess was some queue in lwIP was full and either could no longer accept packets or lwIP could no longer transmit packets.  It had to be timing related because often it would recover leaving us with our real-time issue (it took too much time for Windows to recover from the packets it dropped and we would run out of data on the lwIP receiving end).  This cause of dropped packets was different than normal errors because Windows would just go away for a longer time than it would for normal errors because it was a multiple packet drop. One or two packet drops we can handle and maintain the real-time aspect of our application.

The duplicate ACKs are OK as Kieren mentioned.  My comment about the not recovering was made because I thought TCP would always recover.  But because the drops were on the Windows side, it very well could be the loss of transmissions are also on the Windows side.

You could leave this open for 1.4.0 and I'll squirrel away some time to try to duplicate and capture this.  It will take ~9:30S to run to duplicate the packet drops as were occurring originally and who knows how long for the loss of recovery.  Or I can try to simulate it and see if it fails the same way.  If I can't duplicate it then it's a non-issue I guess.

Bill Auerbach <billauerbach>
Tue 21 Apr 2009 07:22:33 PM UTC, comment #6: 

Bill, did you have time to capture a transfer where it does not recover?

Kieran, I'm curious why this is marked as planned for 1.3.1? We don't even know the source of this, it might be nothing or anything.

BTW, Bill, what does "the transfer does not recover" mean? Does it mean that instead of correct packets, only duplicate ACKs are sent? If so, that sounds like a real bug (in contrast to the burst of duplicate ACKs, which I also encountered multiple times, on slow hardware).

Simon Goldschmidt <goldsimon>
Group administrator
Thu 09 Oct 2008 02:19:12 PM UTC, comment #5: 

A transfer where it does not recover is indeed interesting.

Kieran Mansley <kieranm>
Group Member
Thu 09 Oct 2008 02:01:58 PM UTC, comment #4: 

Thank you for the explanation.  I understand.  We can close this unless you'd like to see a capture where the transfer does not recover.  I may have to simulate this so I don't have to wait 9+ minutes for the dropped packets to occur.  I believe in my case it's happening because of the burst of data and a single reply packet, which starts yet another burst and so on.  It's dropping right after the one packet reply and the next burst.

Bill Auerbach <billauerbach>
Thu 09 Oct 2008 01:41:48 PM UTC, comment #3: 

Perhaps I had relative sequence numbers turned on, and you didn't, which would explain the difference in sequence numbers seen in the trace, sorry.

The reason it sends 14 identical packets (the duplicate ACKs) is that it's just processed 14 out of order packets.  It correctly sends one duplicate ACK for each out of order packet it sees.  These packets are out of order because it's expecting to see a packet that was lost.

I.e.

1) Frames 13-25 are sent by the PC, and recorded in the capture

2) A short time later these are received by lwIP and are processed one by one.

3) Each is judged to be out of order because it's not the next packet expected in the stream: it's still looking for the first lost one.

4) lwIP sends a duplicate ACK for each packet it receives to tell the host that it's still waiting for the lost packet (frames 32-45)

Because steps 1) and 4) don't overlap in time, you don't see a single packet sent by the PC followed by a single ACK in response, but the many duplicate ACKs are each in response to an earlier out of order packet sent by the PC.

Kieran Mansley <kieranm>
Group Member
Thu 09 Oct 2008 01:23:00 PM UTC, comment #2: 

I don't understand - maybe our display options are different?  I don't see 8892.  I can attach the whole 9+ minute capture if you want me to.  Up to what I attached (41000+ packets) all was error-free and after the section with errors it also continued error-free.  It's the burst of identical packets that doesn't look right to me.  Why is it sending 14 identical packets (except for the identification and checksum) in 100mS without any intervening packet receptions?

Bill Auerbach <billauerbach>
Thu 09 Oct 2008 08:17:30 AM UTC, comment #1: 

From the trace you've shown the behaviour of lwIP looks, at first glace, to be correct.

The problem in the stream is that there seems to be a lot of missing packets between sequence 8892 and 39552.  Unfortunately this part of the connection appears to be before the start of the capture, or possibly they got lost in the second of inactivity (and an ARP request, which is a little suspicious in the middle of a connection) near frames 9 - 10.

As a result of these missing packets lwIP sees a large number of packets that are out of order (frames 13 - 25 for example) - it's expecting 8892 and sees 39552 and so on.  For each packet it receives out of order it sends a duplicate ACK, hence the large number of duplicate ACKs in a short space of time.  These duplicate ACKs should trigger fast retransmissions by the sender, which we see in the following frames (frame 30, and a long sequence from frame 48 onwards).

The series of duplicate ACKs from 97-107 are due to the unnecessary retransmission of some of the frames in the earlier sequence - the sender doesn't know that we've buffered some of the frames after the original loss, and so sends them all again.  The stack has now received these twice.  I'd need to read up to check on whether sending duplicate ACKs in this case is absolutely correct, but I've a feeling it is.

Kieran Mansley <kieranm>
Group Member
Wed 08 Oct 2008 10:09:48 PM UTC, original submission:  

When incoming packets are dropped, lwIP can transmit several duplicate ACKs very quickly in succession.  The attached PCAP file shows 14 duplicates ACKs transmitted in 100.1mS.

Bill Auerbach <billauerbach>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #16632:  BurstAcks.pcap added by billauerbach (84KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by kieranm (Posted a comment)
  • -email is unavailable- added by billauerbach (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-05-06 goldsimon StatusNeed Info Invalid
        Open/ClosedOpen Closed
    2009-04-22 goldsimon Planned Release1.3.1 1.4.0
    2009-04-21 goldsimon StatusNone Need Info
    2009-04-03 kieranm Planned Release 1.3.1
    2008-10-08 billauerbach Attached File- Added BurstAcks.pcap, #16632

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code