buglwIP - A Lightweight TCP/IP stack - Bugs: bug #42633, Write many frames at high speed to...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #42633: Write many frames at high speed to a PC stops communication random.

Submitter:  Bart Houkes <mensfort>
Submitted:  Fri 27 Jun 2014 08:45:48 AM UTC
   
 
Category:  TCP Severity:  3 - Normal
Item Group:  Faulty Behaviour Status:  Invalid
Privacy:  Public Assigned to:  goldsimon
Open/Closed:  Closed Planned Release:  None
lwIP version:  1.4.1

Jump to the original submission

Tue 02 Sep 2014 06:32:45 PM UTC, comment #8: 

I take it this one is invalid (i.e. no lwIP bug) -> closed


As a side remark: I don't see how this can be a wrong checksum as the packet capture (a picture??? WTF???) clearly shows lwIP is not sending 2 frames. But I leave that analysis to the OP...

Simon Goldschmidt <goldsimon>
Group administrator
Wed 02 Jul 2014 01:10:49 PM UTC, comment #7: 

It is not "rare" to have a 16-bit word (not "byte"), particularly on 16-bit DSPs, and many of them (since they are DSPs not general processors and so they are designed to handle DSP tasks not general software) don't support byte operations.
So:

  • If your problem lies on your DSP code, I'm afraid you'll have to fix it yourself, or ask TI for support.
  • In lwIP code, you have many different checksum algorithms to try, you might want to test them and see if someone does the trick.
  • You also have the option to offload the checksum calculation to your chip, check you are configuring lwIP to behave accordingly.
  • There is also a thread in one of the lists regarding the checksum algorithms and one particular bug, you might want to search for it
  • You might want to check the comments for bug #33672 and bug #36153


Sergio R. Caprile <scaprile>
Wed 02 Jul 2014 06:27:19 AM UTC, comment #6: 

Dear friends,

BUG analysed. Thanks for helping out!
ISSUE: Checksum is wrong calculated sometimes. This part of code was also changed, because TI DSP's have a strange architecture with 16 bits per byte.

Reentrancy was not the issue, since I lock the core during writing. DMA is no issue, because I read data without DMA.

              LOCK_TCPIP_CORE();
err = tcp_write( tpcb, ptr, length, flags);
if ( err ==ERR_OK)
{
tcp_output( tpcb);
}
UNLOCK_TCPIP_CORE();

Today I have to solve the checksum issue!!

I didn't find this earlier, because Wireshark disables the checksum validation. ONLY after I switch it on, I found the issue. I would always check the checksum in Wireshark, since it is only for 20 bytes. :-)

Bart Houkes <mensfort>
Mon 30 Jun 2014 01:14:25 PM UTC, comment #5: 

Bart,
the capture snapshot clearly shows that you are missing a segment somewhere, there is a big sign in wireshark telling you that, and the PC does not ACK what you sent because it is missing what you did NOT send, and THAT is the missing segment.
frame 13888: "TCP previous segment not captured"
frame 13887: your seq is 2969 and you send 1360 bytes, then the next segment (frame 13888) has seq 7049
7049 - 1360 - 2969 = 2720 = 2*1360 bytes missing "somewhere" in between your function call to send and the PC + wireshark. This looks like a memory problem, however, I'm no expert here.
frame 13892, and all the others in black saying "dup ACK": The PC is sending ACK to 4329 telling the sender to send that, what you didn't.
This is the magic of networking.
I don't know why there is a segment missing, if frames got lost in DMA, TCP should still have them, or is there a no copy option that causes TCP to lose what it has sent before it gets an ACK ? Is there any way of "bad calling" the stack to cause it to skip a segment ?
I think this is memory corruption caused by reentrancy.

Sergio R. Caprile <scaprile>
Sat 28 Jun 2014 03:58:17 PM UTC, comment #4: 

I'm not using SYS_LIGHTWEIGHT_PROT, but instead I use a semaphore in following functions:
sys_prot_t sys_arch_protect(void);
void sys_arch_unprotect(sys_prot_t pval);

I use pbuf_alloc only when I use RAW TCP api, with netconn this is hidden. However, both RAW API and NETCONN have the problem for fast transmissions.

Also I investigated the LAN9311 with extra statistics. It seems I can never get this chip full. Transmission of Ethernet is faster than my program.

When this problem happens, it is always the PC didn't ACK several frames and LWIP has the full window used and is resending the first message in the waiting list.. and that is never acknowledged by the PC anymore.

I wonder if the ACK from the receiver(PC) has a collision with the next frame from LWIP. Because sometimes at this hang, there is also a frame missing sent by LWIP.

I'll post when I know more.

Bart Houkes <mensfort>
Fri 27 Jun 2014 01:27:05 PM UTC, comment #3: 


> There is a thread switch in between which I can investigate more.
> At lower speed with delays in communication, it seems to work better. Below 8192 bytes there is never a communication issue. (Window size)


This definitely looks like a low level driver not following lwIP threading model.

If you are using threads, are you using the NETCONN or SOCKET API only ? RAW API is not thread safe.

If you are using pbuf_alloc() outside of the lwIP main thread, is SYS_LIGHTWEIGHT_PROT set in your lwipopts.h ?

Sylvain




Sylvain Rochet <gradator>
Group Member
Fri 27 Jun 2014 01:15:04 PM UTC, comment #2: 

Thanks for your comment in helping me!

I am not using DMA. It is a LAN9311 driver, which I read directly from C-code. There is a thread switch in between which I can investigate more. At lower speed with delays in communication, it seems to work better. Below 8192 bytes there is never a communication issue. (Window size)

I strongly suspect LWIP, what else can this be?

Bart Houkes <mensfort>
Fri 27 Jun 2014 12:56:32 PM UTC, comment #1: 

As usual, it looks like a DMA bug in the low level driver you are using.


> so it must be a problem in LWIP.


it must ?, really ?

Sylvain Rochet <gradator>
Group Member
Fri 27 Jun 2014 08:45:48 AM UTC, original submission:  

I have an embedded device with LWIP that sends 400KByte text to a PC at high speed.

During monitoring on Wireshark I see following:

Usually, every block of data is acknowledged immediately after sending the data. After some time, LWIP sends 3 frames to a PC and receives an acknowledge for the first frame. So, 2 frames are not acknowledged yet. Then LWIP sends more frames until TCP_WND is reached. Then, the first outstanding frame that needs an acknowledge is resent several times... the PC does not react anymore, although I think it should send an ACK.

This issue happens when I call the TCP-layer directly, but also when I use netconn to send data.

I have tried this in Windows, Linux, Telnet and others... so it must be a problem in LWIP. What can be wrong?

Bart Houkes <mensfort>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #31626:  evidence.png added by mensfort (144KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by mensfort (Voted in favor of this item)
  • -email is unavailable- added by gradator (Posted a comment)
  • -email is unavailable- added by mensfort (Submitted the item)
  • -email is unavailable- added by mensfort (LWIP issue)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-09-02 goldsimon StatusNeed Info Invalid
        Assigned toNone goldsimon
        Open/ClosedOpen Closed
    2014-07-02 mensfort Carbon-Copy- Added mensfort
    2014-06-27 gradator StatusNone Need Info
    2014-06-27 mensfort Attached File- Added evidence.png, #31626
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code