buglwIP - A Lightweight TCP/IP stack - Bugs: bug #52719, sys_timeouts synchronization issue...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #52719: sys_timeouts synchronization issue with core locking (NO_SYS == 0)

Submitter:  Joel Cunningham <jcunningham>
Submitted:  Thu 21 Dec 2017 09:50:59 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  Faulty Behaviour Status:  Fixed
Privacy:  Public Assigned to:  jcunningham
Open/Closed:  Closed Planned Release:  2.1.0
lwIP version:  git head

Jump to the original submission

Tue 02 Jan 2018 06:19:53 PM UTC, comment #6: 

Patches applied in 53499f5e9faecb1593f9a3de54385ac4ebd9aa83 and 1c7a02429739e4d1b2858ad0fe5114f47a1d2ffa

I ended up dropping the assert checks in cyclic_timer and tcpip_tcp_timer because they are:

1) static functions with which can only be called as a timer callback
2) Redundant with the check in sys_check_timeouts, which ensures timer handlers (callbacks) are called with the core lock acquired

Joel Cunningham <jcunningham>
Group Member
Tue 02 Jan 2018 05:41:06 PM UTC, comment #5: 

Cool, thanks!!!

Dirk Ziegelmeier <dziegel>
Group administrator
Tue 02 Jan 2018 05:39:26 PM UTC, comment #4: 

Dirk,

I got LWIP_ASSERT_CORE_LOCKED working in my test environments and indeed we are failing the assert check in sys_timeouts_sleeptime and sys_check_timeouts

I applied Douglas' patch and no more core locking asserts are failing.  I can do the minor cleanups I mentioned below and submit the patch if it looks good to everyone else

Joel Cunningham <jcunningham>
Group Member
Tue 02 Jan 2018 05:28:36 PM UTC, comment #3: 

Joel, can you please check whether the new LWIP_ASSERT_CORE_LOCKED() stuff would reveal your issue?

Dirk Ziegelmeier <dziegel>
Group administrator
Tue 02 Jan 2018 04:03:07 PM UTC, comment #2: 

Douglas,

Thanks for the attached patch, it looks to correctly solve the core issue.  A couple minor things:

1) The addition of LWIP_ASSERT_CORE_LOCKED in tcpip_tcp_timer, tcp_timer_needed, and cyclic_timer is not related to this bug, so it would be better to separate that into another change.

2) In

-  if (sleeptime == 0 || sys_arch_mbox_fetch(mbox, msg, sleeptime) == SYS_ARCH_TIMEOUT) {
+  if (sleeptime == 0) {
+    sys_check_timeouts();
+    /* We try again to fetch a message from the mbox. */
+    goto again;
+  }
+
+  UNLOCK_TCPIP_CORE();
+  u32_t res = sys_arch_mbox_fetch(mbox, msg, sleeptime);
+  LOCK_TCPIP_CORE();
+  if (res == SYS_ARCH_TIMEOUT) {


You have introduced a declaration (u32_t res) after the first executable statement in the current scope, which won't compile with C89 (which LwIP targets)

Joel Cunningham <jcunningham>
Group Member
Tue 02 Jan 2018 03:09:05 PM UTC, comment #1: 

Attached is a suggested patch. This pushes down the release of the core lock to the actual mbox fetch, keeping the lock held while checking the timer data structures.

(file #42788)

Douglas <ourairquality>
Thu 21 Dec 2017 09:50:59 PM UTC, original submission:  

I've found a synchronization issue with sys_timeouts and core locking for NO_SYS == 0. This was found during code inspection.

The TCP/IP thread calls sys_timeouts_mbox_fetch() with the core lock explicitly released.  Within this function, next_timeout is read/modified in sys_timeouts_sleeptime and sys_check_timeouts.

Since sys_timeout/untimeout also read/modify next_timeout, this can only be safely called from the TCP/IP thread.  I believe I've found instances with core locking where sys_timeout/untimeout will be called on another thread.

The macro TCP_REG will be executed in a number of cases from other threads and it calls tcp_timer_needed().  If tcpip_tcp_timer_active is 0 and tcp_active_pcbs or tcp_tw_pcbs are non-NULL, sys_timeout will be called to start the TCP timer.

In one of my test environments, I put a breakpoint on when this particular sys_timeout is called and I found an application thread inside lwip_connect() which triggered it

I also see a lot of calls to sys_timeout in PPP code, which may also be executed on another thread with core locking (I'm not that familiar with the PPP architecture though)




Joel Cunningham <jcunningham>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dziegel (Posted a comment)
  • -email is unavailable- added by ourairquality (Updated the item)
  • -email is unavailable- added by goldsimon (Updated the item)
  • -email is unavailable- added by jcunningham (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-01-02 jcunningham StatusNone Fixed
        Open/ClosedOpen Closed
    2018-01-02 dziegel Assigned toNone jcunningham
    2018-01-02 ourairquality Attached File- Added 0001-timers-rework-the-core-locking-around-timers.patch, #42788
    2018-01-02 goldsimon Planned ReleaseNone 2.1.0

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code