tasklwIP - A Lightweight TCP/IP stack - Tasks: task #6683, Document lwIPs thread safety...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #6683: Document lwIPs thread safety requirements

Submitter:  Kieran Mansley <kieranm>
Submitted:  Mon 26 Mar 2007 10:50:58 AM UTC
   
 
Category:  Documentation Should Start On:  Sun 25 Mar 2007 11:00:00 PM UTC
Should be Finished on:  Sun 25 Mar 2007 11:00:00 PM UTC Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  None Percent Complete:  0%
Open/Closed:  Closed Planned Release:  None
Effort:  0.00

Jump to the original submission

Tue 05 May 2009 07:34:47 PM UTC, comment #24: 

I've added 2 paragraphs on other APIs and threading to rawapi.txt, which should be sufficient to close this task.

Simon Goldschmidt <goldsimon>
Group administrator
Wed 09 Jul 2008 04:23:14 PM UTC, comment #23: 

Wow, this is so great!

Kieran, can you delete a post? Or even delete this user?? ;-)

Simon Goldschmidt <goldsimon>
Group administrator
Fri 09 Nov 2007 01:13:02 PM UTC, comment #22: 

Downgrading the priority of this as thanks to the excellent documentation work by others it's now in a pretty good state.

Kieran Mansley <kieranm>
Group Member
Mon 05 Nov 2007 12:49:58 PM UTC, comment #21: 

I guess Frederic is right that we can't keep tasks open just because the documentation wants to be improved - otherwise they would be open forever because they can always be improved :-).

As for merging those other bugs/tasks, I think not, personally - although related, they are not identical, and discussion for each should be kept in its own place, otherwise it will get very hard to follow. 21049 and 6735 are close, but present two different solutions to the same problem. If Kieran is against the bug 21049 approach, maybe it should be closed.

Jonathan Larmour <jifl>
Group Member
Sun 04 Nov 2007 01:01:29 PM UTC, comment #20: 

I think that the aim of this task is reached: "document" lwIP thread safety requirements is done on wiki (and if it miss some elements, I/we can add them).

Some coments from this record are not really about the documentation, but more about driver design. I think this is lot of task or bugs record about "pbufs & driver":

#21433 Calling mem_free/pbuf_free from interrupt context isn't safe
#21049 Allow reuse of pbufs after return from APIs
#6735 Provide new pbuf type: PBUF_RAM_NOCOPY
#7013 Create option to have all packets delivered to netif->output in one piece

Since, most of them are planned for post-1.3.0, I'm in favor to got only one record for all these tasks/bugs.

Frédéric Bernon <fbernon>
Group Member
Sun 04 Nov 2007 11:42:14 AM UTC, comment #19: 

On the one hand, I think we can close this to avoid duplicates. But on the other hand, if this task is still open, it reminds us of what's to do for 1.3.0, so I would leave it open until it's really done.

Simon Goldschmidt <goldsimon>
Group administrator
Fri 02 Nov 2007 07:43:09 PM UTC, comment #18: 

Since now, http://lwip.scribblewiki.com seems to be accessible to most of users, and since most of active developers have an account on it, I propose to close this task to avoid to have multiple place to centralize the documentation. No objections?

You could find the information (and change it) at :

http://lwip.scribblewiki.com/LwIP_and_multithreading


Frédéric Bernon <fbernon>
Group Member
Fri 20 Jul 2007 03:33:29 PM UTC, comment #17: 

This may not be as helpful as it initially sounds, but I've written some documentation for lwIP for the commercial version of eCos I am involved with (called eCosPro). Much of it is derived (with acknowledgements, fear not) from one of Adam Dunkel's documents; but updated to reflect the current status.

Unfortunately it's not so helpful because:
- The Powers That Be restrict distribution on this ("Distribution of the work or derivative of the work in any form is prohibited unless prior permission obtained from the copyright holder.") as per http://www.ecoscentric.com/ecospro/doc/html/ecospro-ref/ecospro-ref.html

- it's slanted specifically towards the eCosPro port, which is even different from the eCos port, and has several patches (some of them contributed back though) relative to the lwIP it was originally branched from (1.1.1).

Nevertheless, there would be no problem linking to it. However I won't be offended if you don't want to, given the above drawbacks.

The direct link is: http://www.ecoscentric.com/ecospro/doc/html/ecospro-ref/lwip.html

(In case you're wondering, the reason for the restrictions is so that we can recoup the investment made in the first place - once that is done, we would hopefully be able to relax things).

Jonathan Larmour <jifl>
Group Member
Fri 13 Jul 2007 12:07:45 PM UTC, comment #16: 

Note to developers - I've now made this the highest priority task we have!  All help with this, and other documentation, greatly appreciated.

Kieran Mansley <kieranm>
Group Member
Thu 19 Apr 2007 12:44:59 PM UTC, comment #15: 


>Just my opinion, but that would be rather gratuitous API breakage


Not just your opinion, I thought about that also. Maybe we could modify api.h in a way that the raw-API files are no automatically included (ip.h/raw.h/udp.h/tcp.h), and document that application thread files only have to include api.h or sockets.h to work. That would also make it more clear, I think.

My idea is that, for example we could remove the line #include "lwip/raw.h" from api.h and instead forward-declarate 'struct raw_pcb;'. That way, we can make the PCBs (which the user should not manipulate when using netconn API) kind of 'private'... (I'm not really happy with the forward-declaration of structs, though...)

Same for sockets.h: by moving #include "tcph.h" to sockets.c and including "inet.h" we could again hide lwip-internals from sockets.h-users.

Simon Goldschmidt <goldsimon>
Group administrator
Thu 19 Apr 2007 12:22:37 PM UTC, comment #14: 

Just my opinion, but that would be rather gratuitous API breakage, and there are no doubt many corner cases. Being divided per-file is good enough - what's missing is documentation to let people know about it.

Jonathan Larmour <jifl>
Group Member
Thu 19 Apr 2007 10:54:44 AM UTC, comment #13: 

To clarify which functions application threads may use, would it help to create an extra include directory to separate the header files that include functions that may be used from application threads from the core stack include files?

i.e. create new directory src/include/lwip/api and move api.h, sockets.h, netifapi.h and some more there?

Simon Goldschmidt <goldsimon>
Group administrator
Mon 16 Apr 2007 07:32:56 PM UTC, comment #12: 


>An interrupt routine could put pbufs in some form of queue (not necessarily an lwip pbuf queue) and the main loop could pass them into lwip.


That's right. And that queue could be a pbuf queue (or somehow else a queue containing pbufs), so protecting pbuf.c is needed in that case, I think (if the driver wants to allocate pbufs from interrupt context).

Simon Goldschmidt <goldsimon>
Group administrator
Mon 16 Apr 2007 04:58:21 PM UTC, comment #11: 

I wrote:

> An interrupt routine could put pbufs in some form of queue


Or more likely just set some sort of flag to make the main loop go and fetch the packet or process the driver event etc.

Jonathan Larmour <jifl>
Group Member
Mon 16 Apr 2007 04:54:33 PM UTC, comment #10: 


>> Raw-API only (NO_SYS=1), network interfaces work with interrupts
>
> Can this really work? netif->input may not be called from interrupt context,
> since this might re.enter functions which are not re-entrant (tcp!). So the
> only thing we can do would be freeing pbufs after sending (e.g. with a
> DMA-ethernetif that calls pbuf_ref() before sending).


I think it can work. Every raw API program needs some sort of loop or similar to sit there waiting for events such as incoming packets to happen. An interrupt routine could put pbufs in some form of queue (not necessarily an lwip pbuf queue) and the main loop could pass them into lwip. Well, I think anyway,

Jifl

Jonathan Larmour <jifl>
Group Member
Mon 16 Apr 2007 03:32:48 PM UTC, comment #9: 

Forgot to say in the last comment:
I think it should be well documented that lwIP is designed to work when receiving packets from interrupt context but that the OS layer must support this!

re my own comment #6:

>First, there are 3 different scenarios:


regarding scenario 2):

>Raw-API only (NO_SYS=1), network interfaces work with interrupts


Can this really work? netif->input may not be called from interrupt context, since this might re.enter functions which are not re-entrant (tcp!). So the only thing we can do would be freeing pbufs after sending (e.g. with a DMA-ethernetif that calls pbuf_ref() before sending).

Simon Goldschmidt <goldsimon>
Group administrator
Mon 16 Apr 2007 02:08:55 PM UTC, comment #8: 


>but sys_mbox_post can be a problem: OS-provided mailboxes may not be interrupt-safe - they may only be intended to be task-safe.


>because posting to a mailbox can cause a task switch


Good point... If posting to a mailbox should always succeed, other tasks might have to run until there is free space in the mailbox. But eCos is designed to have DSRs, while e.g. µC/OS is not. In µC/OS (as in other OSes), there is a function like mbox_trypost().

I think whether mbox posting can be used from interrupt context or not is up to the user to decide. After all, the 'user' is still a developer who should know it's system...

Involving mem_free() (and thus, waiting for a semaphore) waits (and context-switches) on every OS, so at least TX-interrupt must not call pbuf_free() with the current code...

Simon Goldschmidt <goldsimon>
Group administrator
Mon 16 Apr 2007 01:40:33 PM UTC, comment #7: 

First of all, with respect to interpretation of NO_SYS etc., IMHO we can lay down new policy, irrespective of what was originally intended. Clearly things are a bit muddy at present anyway.

Secondly, as well as interaction between apps and lwIP, I think more attention is needed with the intended interface between ethernet (or serial?) drivers and lwIP. This affects SYS_LIGHTWEIGHT_PROT for example: should it disable tasking or interrupts? Must drivers call into the stack in an interrupt context or a task context? Or can ports do either, as long as they code appropriately? I suspect at the moment ports do either, but I for one am not sure that this can work reliably (at least in the current code - maybe that can be changed). A key issue is whether drivers (or code called from drivers) can wait on synchronisation objects - possible from task or high level interrupt context (aka bottom half aka deferred service routine, etc.), but not a low level interrupt context.

As an example, if a driver called tcpip_ethinput() in current CVS from a low level interrupt context, then that in turn calls memp_malloc and sys_mbox_post. Okay, memp_malloc can be protected with SYS_LIGHTWEIGHT_PROT and interrupts disabled, but sys_mbox_post can be a problem: OS-provided mailboxes may not be interrupt-safe - they may only be intended to be task-safe. This is true for eCos and no doubt others. You also can't just disable interrupts around the call either because posting to a mailbox can cause a task switch, which, if interrupts are off, would cause all sorts of bad things. If lwIP requires people to write their own mailbox implementation as part of the "OS abstraction" then I don't think the OS abstraction can be quite right.

The "solution" is to ensure that tcpip_ethinput is not called from an interrupt context - it must either be called from a thread or from a high level interrupt handler aka bottom half aka deferred service routine aka other names on other systems. Not all systems have such facets to their interrupt handling though.

Anyway, this is was only meant to be one example where thread interactions are unclear, and why we need more clarity.

Jonathan Larmour <jifl>
Group Member
Mon 16 Apr 2007 11:09:55 AM UTC, comment #6: 

Kieran,

as a summary for lwIP's protection against concurrent access / multithreading protection, I would like to summarize it like this:

First, there are 3 different scenarios:
1) Raw-API only (NO_SYS=1), no access to lwIP functions from interrupt context, network interfaces are polled --> no protection needed!
2) Raw-API only (NO_SYS=1), network interfaces work with interrupts (-> interrupt context accesses pbuf.c...) --> LIGHTWEIGHT_PROT needed to protect pbuf pool / memp / mem.c heap **
3) complete multithreaded environment (NO_SYS=0)

** mem.c protection: what if TX-done-interrupt frees a PBUF_RAM?

- The difference between 1) and 2) is only the mode of the network-driver.
- With 2) & 3), required protection is the same.
- lwIP core is not thread-safe. In multithreading environment, threads other than the main lwIP thread (tcpip_thread()) only may use functions defined in sys.h, api.h, sockets.h,  mem.h, memp.h and pbuf.h.


I think that's about it, any comments are welcome! We still need to clarify if my interpretations of NO_SYS are correct and if we need a SYS_HEAVYWEIGHT_PROT or something (I think we don't and thus, should rename SYS_LIGHTWEIGHT_PROT to SYS_PROT or something).

Then, we could inlcude those three forms in a contrib module somewhere (unix port should be best as it seems the most used...).

Simon Goldschmidt <goldsimon>
Group administrator
Thu 12 Apr 2007 04:06:41 PM UTC, comment #5: 

Re comment #4:
One semaphore per thread would be good. One sem would be enough since a thread can't execute more than one socket (or netconn) operation at a time. The way it is now, the conn->mbox is used like a semaphore.

Simon Goldschmidt <goldsimon>
Group administrator
Thu 12 Apr 2007 02:07:46 PM UTC, comment #4: 

Agree with you Simon, but to do that, we have to :

- use a critical section in api_msg_post
or
- use a semaphore (and not a mbox) to do the synchronize between application-thread and tcpip_thread. This semaphore could be a local variable (but it's not a good solution), or a semaphore based on a per thread base (like sys_timeouts).

Frédéric Bernon <fbernon>
Group Member
Thu 12 Apr 2007 01:50:22 PM UTC, comment #3: 

Sockets generally can't be used in from more than one thread, for now: a socket has a netconn nad that netconn's mbox is used to tell the application its request to tcpip_thread has finished. Thus, if you call socket functions from 2 or more threads at the same time, it won't be clear which request has finished when tcpip_thread posts to sock->conn->mbox.

This is something I'd like to see changed, though :-)

Simon Goldschmidt <goldsimon>
Group administrator
Mon 26 Mar 2007 04:45:12 PM UTC, comment #2: 

Some others informations to remember: some special functions can also cause problem with multithread: netif_xxx and dhcp_xxx. These functions used same variables than tcpip_thread. Some cases:

- if an "application" thread remove an netif from list, during tcpip_thread is running, so, you can get a crash. Adding two netif in the same time can cause to lost one of interface (https://savannah.nongnu.org/bugs/?19347).

- if an "application" thread stop dhcp on an interface, because the netif's dhcp field can be used by dhcp's timers (runnning in tcpip_thread context), you can get a crash (https://savannah.nongnu.org/patch/?5798).



Frédéric Bernon <fbernon>
Group Member
Mon 26 Mar 2007 12:48:17 PM UTC, comment #1: 

I'm ok and have start to study several points. First, I think we have to define what in lwIP have to be thread safe. I think that we can tell that :

- lwIP core isn't thread safe. If we have to use lwIP in a multithread environment, we should (or HAVE TO) use "upper" layers (netconn or sockets). raw api's need to protect themselft the core.

- some upper operations have to be handled directly by application: socket, bind, connect, setsockopt and close. I don't think it's the lwIP role to handle that (by example, it will be difficult to know if a "int socket" is always same between close/socket calls:

T1
   int s1=socket() (tell s1=5)
T1,T2
   use s1 with send,recv (ok, it could be possible).
T2
   closesocket(s1)
T3
   int s2=socket() (and... it could be s2=5)
T1
   send(s1) !!!! Problem, s1 would not the same proto/port than the first we open, but only application can know that !!!!

- some operations can be transform to thread-safe : recv, recvfrom, send, sendto... To enable fullduplex protocols. We have to move some processing from netconn/sockets to api_msg, but it's possible, and even can be faster...

- I don't use select, and I think it is a special case (lot of code is only use for this function, and could/should be disable)...





Frédéric Bernon <fbernon>
Group Member
Mon 26 Mar 2007 10:50:58 AM UTC, original submission:  

This is a common question and point of difficulty, so we should provide a up to date document to describe what is required.


Kieran Mansley <kieranm>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jifl (Posted a comment)
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by fbernon (Posted a comment)
  • -email is unavailable- added by kieranm (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-05-05 goldsimon StatusNone Done
        Open/ClosedOpen Closed
    2007-11-09 kieranm Planned Release1.3.0 None
    2007-11-09 kieranm Priority8 5 - Normal
    2007-07-13 kieranm Priority5 - Normal 8
        Assigned tokieranm None
        Planned ReleaseNone 1.3.0

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code