buglwIP - A Lightweight TCP/IP stack - Bugs: bug #51033, LWIP_ASSERT while doing repetitive...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #51033: LWIP_ASSERT while doing repetitive transmission

Submitter:  preet <preetpal>
Submitted:  Tue 16 May 2017 01:34:08 AM UTC
   
 
Category:  IPv4 Severity:  3 - Normal
Item Group:  Crash Error Status:  Invalid
Privacy:  Public Assigned to:  None
Open/Closed:  Closed Planned Release:  None
lwIP version:  2.0.2

Jump to the original submission

Thu 18 May 2017 07:32:55 AM UTC, comment #10: 

I already thought it's something like this... :-)

Simon Goldschmidt <goldsimon>
Group administrator
Wed 17 May 2017 11:24:26 PM UTC, comment #9: 

Found the REAL root cause.  When the linkoutput function called our Ethernet output, and all of the ethernet transmit descriptors were consumed, we returned with an error:


return ERR_BUF;


The problem is that we did not take away the offset of the ETH_PAD_SIZE which was added upon entry to the function.

This issue can be closed.

preet <preetpal>
Wed 17 May 2017 10:53:13 PM UTC, comment #8: 

Simon,

I'm working with Preet on this issue. We were able to identify the root cause of the LWIP_ASSERT. The problem occurs when we have fully consumed all available descriptors inside our Ethernet tx function and return ERR_BUF to the calling function inside LWIP. Our solution was to place a system wait inside the transmit function until the tx output was no longer full rather than exiting the tx function and returning an error.

Not sure why LWIP is handling ERR_BUF returns with an assert.

Fayek <fwahhab>
Tue 16 May 2017 08:39:20 PM UTC, comment #7: 

Yes, tcpip_input is passed to the input function:


netif_add(&enet_if[i], &enet_ip[i].ip, &enet_ip[i].nm,
    &enet_ip[i].gw,
    NULL, enet_hw_init,
    tcpip_input))


preet <preetpal>
Tue 16 May 2017 07:33:16 PM UTC, comment #6: 


>  and pass it to the LWIP using the input function pointer of
> their own interface:
> e0_interface.input(pbuf, e0_interface)


... and you pass 'tcpip_input' as input function when setting up the netifs via 'netif_add()', I guess?

In that case, it looks like a bug in TCP_OVERSIZE. I'll see if I can reproduce that...

Simon Goldschmidt <goldsimon>
Group administrator
Tue 16 May 2017 07:26:27 PM UTC, comment #5: 

1)  Reproduced it with Nagle algorithm off and a simple loop to calls send() function inside the loop.

  /*
   * Phase 2: concat_p can be concatenated onto last_unsent->p, unless we
   * determined that the last ROM pbuf can be extended to include the new data.
   */
  if (concat_p != NULL) {
    LWIP_ASSERT("tcp_write: cannot concatenate when pcb->unsent is empty",
      (last_unsent != NULL));
    pbuf_cat(last_unsent->p, concat_p);
    last_unsent->len += concat_p->tot_len;
  }


2)  The application uses two Ethernets and I do not believe there are any threading issues
-  E0_rx interrupt just posts a counting semahore
-  Likewise, E1_rx posts a different E1_cnt_sem
-  The two sleeping tasks for each ethernet, namely E0_rx_task and E1_rx_task, take the semaphore
-  Sleeping tasks read their own Ethernet descriptor, pull a pbuf_alloc() and pass it to the LWIP using the input function pointer of their own interface:
e0_interface.input(pbuf, e0_interface)


preet <preetpal>
Tue 16 May 2017 07:09:56 PM UTC, comment #4: 

1) pbuf_cat() is called multiple times from tcp_write(). Could you tell us the line please?

2) Are you absolutely sure you don't have threading issues? With so many people on the lists, I keep forgetting who I already asked this, so make sure e.g. you don't pass RX packets into lwIP from interrupt (i.g. call ethernet_input() from interrupt or pass anything else than 'tcpip_input' to netif_add()) while using sockets (that uses tcpip_thread()).

Simon Goldschmidt <goldsimon>
Group administrator
Tue 16 May 2017 02:23:24 AM UTC, comment #3: 

One more important update:
This behavior only occurs when the Nagle algorithm is disabled

preet <preetpal>
Tue 16 May 2017 02:06:19 AM UTC, comment #2: 

Disregard the previous comment... it is still hitting the assertion.

p->total_len was 67
p->len was 65

The assertion seems to want both values to be equal to each other.

preet <preetpal>
Tue 16 May 2017 01:59:02 AM UTC, comment #1: 

As I as fiddling with the LWIP options, it looks like I introduced this problem.  When I use the default settings, then it works:


#define TCP_SND_BUF                     (5 * TCP_MSS)

//#define TCP_SND_QUEUELEN                (10)            ///< TCP sender buffer space (pbufs).  Must be at least (2 * TCP_SND_BUF/TCP_MSS)
//#define TCP_SNDLOWAT                    (1 * TCP_MSS)   ///< TCP writable space (bytes) bofore a socket becomes writable.  This must be less than TCP_SND_BUF
//#define TCP_SNDQUEUELOWAT               (1)             ///< TCP writable bufs (pbuf count), when select becomes writable.  This must be less than TCP_SND_QUEUELEN



Uncommenting the options above makes it hit the assertion but commenting them out to revert to the defaults at opts.h makes it work.

preet <preetpal>
Tue 16 May 2017 01:34:08 AM UTC, original submission:  

During some rigorous LWIP tests, I hit an LWIP exception while using the following code:


for (int i = 0; i < 20; i++) {
        len = snprintf(buffer, buffer_len, "%2u) %s\n", i + 1, "Some command");
        send(socket, buffer, len, 0);
    }


Stack trace (from highest to lowest)
- send()
- lwip_send()
- netconn_write_partly()
- tcpip_send_msg_wait_sem()
- lwip_netconn_do_write()
- lwip_netconn_do_writemore()
- tcp_write()
- pbuf_cat()
- Assert at: LWIP_ASSERT("p->tot_len == p->len (of last pbuf in chain)", p->tot_len == p->len);

I get about 4-5 lines from the for loop before the assertion hits.  I am using memory pools with these options:


LWIP_MALLOC_MEMPOOL_START
LWIP_MALLOC_MEMPOOL( 8, 64)
LWIP_MALLOC_MEMPOOL(1, 128)  // On purpose
LWIP_MALLOC_MEMPOOL(16, 256)
LWIP_MALLOC_MEMPOOL(32, 1564)
LWIP_MALLOC_MEMPOOL_END


More LWIP options:

#define MEM_USE_POOLS                   1
#define MEMP_USE_CUSTOM_POOLS           1
#define MEM_USE_POOLS_TRY_BIGGER_POOL   1


preet <preetpal>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by fwahhab (Posted a comment)
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by preetpal (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-05-18 goldsimon StatusNone Invalid
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-caa5.
    Corresponding source code