patchlwIP - A Lightweight TCP/IP stack - Patches: patch #6865, SO_REUSEADDR for TCP

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #6865: SO_REUSEADDR for TCP

Submitter:  Jeff Barber <jeffbabar>
Submitted:  Tue 14 Jul 2009 03:22:04 PM UTC
   
 
Category:  None Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  goldsimon Open/Closed:  Closed
Planned Release:  1.4.0

Jump to the original submission

Wed 12 May 2010 10:33:26 PM UTC, comment #18: 

Done. tcp_bind() now virtually allows any combination of double-used ports/addresses since there is no way to check it at that point (since the remote IP/port is unknown at that point).

Instead, tcp_connect() checks for a unique 5-tuple for active connections, passive connections rely on the remote IP/port being unique.

I did it this way after checking what linux allows and what it doesn't, so it's a bit different to other OSes (especially windows, I guess), but since the (opengroup-)spec doesn't say much about the details, that's OK with me.

Please reopen if there are still problems with SO_REUSEADDR.

Simon Goldschmidt <goldsimon>
Group administrator
Tue 16 Feb 2010 12:22:13 PM UTC, comment #17: 

Sorry, when syn packet is not sent, local ip address is not defined,
so active pcb list can have ANY address.
But pcb in time wait list always have defined local ip address.

Oleg Tyshev <olegreen>
Tue 16 Feb 2010 12:15:45 PM UTC, comment #16: 

If we miss out that check, let's replace it with an assertion that the local port is not ANY.


Kieran Mansley <kieranm>
Group Member
Tue 16 Feb 2010 12:09:37 PM UTC, comment #15: 

I see tcp_bind() and it always used
 ip_addr_isany(&(cpcb->local_ip))
check.

For bind pcb list and listen pcb list it is correct.
But for active pcb list and time_wait pcb list local port can't be ANY, so this check is superfluous.

 /* Check the connected pcbs. */
  for(cpcb = tcp_active_pcbs;
      cpcb != NULL; cpcb = cpcb->next) {
    if (cpcb->local_port == port) {
-      if (ip_addr_isany(&(cpcb->local_ip)) ||
-          ip_addr_isany(ipaddr) ||
+      if (ip_addr_isany(ipaddr) ||
          ip_addr_cmp(&(cpcb->local_ip), ipaddr)) {
        return ERR_USE;
      }
    }
  }


  /* Unless the REUSEADDR flag is set,
   we have to check the pcbs in TIME-WAIT state, also: /
  if ((pcb->so_options & SOF_REUSEADDR) == 0) {
    for(cpcb = tcp_tw_pcbs; cpcb != NULL; cpcb = cpcb->next) {
      if (cpcb->local_port == port) {
-        if (ip_addr_isany(&(cpcb->local_ip)) ||
-            ip_addr_isany(ipaddr) ||
+        if (ip_addr_isany(ipaddr) ||
            ip_addr_cmp(&(cpcb->local_ip), ipaddr)) {
          return ERR_USE;
        }
      }
    }
  }




Oleg Tyshev <olegreen>
Sat 13 Feb 2010 11:07:57 PM UTC, comment #14: 


> Not sure I followed your point about "binding listen pcbs to
> ports used by non-listen pcbs". I didn't mean to say that and I
> don't think linux allows it.


I don't know about linux and it wasn't about what you said but about something I read somewhere on the web about SO_REUSERADDR - after all information on the web got me very confused about what's supported on which platform...

The case was this
- listen server started
- accepts a connection
- server stops (i.e. listen socket closed)
- server restarted -> connection from point 2 still active

at this point, you might need SO_REUSEADDR on some systems while you wouldn't on others. I'd have to check what lwIP does, but given the tcp_bind() function I have in my mind, it won't allow it currently.

Simon Goldschmidt <goldsimon>
Group administrator
Sat 13 Feb 2010 10:10:49 PM UTC, comment #13: 

I agree about SO_REUSEPORT.  It seems pretty much the same thing with differences too tiny to bother with.

Not sure I followed your point about "binding listen pcbs to ports used by non-listen pcbs".  I didn't mean to say that and I don't think linux allows it.  Both the existing endpoints and new ones with the same port number must not be in LISTEN state and must have the REUSEADDR flag set in order for bind to succeed.

The current patch certainly doesn't handle all the cases that linux does.  It really handles only the most common case: a single remote FTP client is attached to a local server and is making consecutive one-at-a-time data transfers where each uses the same lwip-side port number (hence the previous just-closed connection is often still in TIME_WAIT).  It doesn't handle the case of multiple remote FTP clients.  But it was plenty for my purposes since I only expect a single remote ftp client to be updating my local storage at a time anyway, but it is less general.

If you implement it as flexibly as linux, there are some possibilities to get multiple 5-tuples with the same endpoint addresses.  However, the remote system would need to collude in that as well which means practically speaking it won't happen.  Even when the local FTP server, for example, is re-using the same local port over and over, the remote system is typically creating a unique port number for each endpoint.

Jeff Barber <jeffbabar>
Sat 13 Feb 2010 11:25:57 AM UTC, comment #12: 

Agree about TIME_WAIT, especially as it doesn't hurt: when allocating a new pcb and running out of resources, TIME_WAIT pcbs are killed anyway.

Reading about SO_REUSERADDR vs. SO_REUSEPORT, am I correct that SO_REUSEPORT is not necessary since it's not standardised and mostly does the same?

And it seems that our SO_REUSEADDR for TCP is still missing the 2 points you wrote (binding to different local IPs, binding listen pcbs to ports used by non-listen pcbs)?

About the latter case: how can bind() detect that, it would allow 2 connection pcb (i.e. non-listen) to bind to the same local address, wouldn't it? If the 5-tuple must be unique, then connect() (and maybe accept(), too?) would have to check the remote address (only if SO_REUSEADDR is set) to prevent a non-unique 5-tuple!

Simon Goldschmidt <goldsimon>
Group administrator
Fri 12 Feb 2010 07:52:49 PM UTC, comment #11: 

On linux, the rule is that the same port can be re-used if the endpoints are on different interfaces (different IP addresses) OR if all existing endpoints are not in LISTEN state and all have the REUSEADDR flag set (and the to-be-created endpoint has to match the same criteria).


Re comment #9: we should not dump TIME_WAIT pcb's; they can still be matched by incoming packets using both local and remote IP addresses and ports to distinguish. 

The new locally bound endpoint doesn't have a remote IP address and port (yet).  If it were later to be connected to the same remote address and port, one could dump the TIME_WAIT pcb, I suppose, but that's probably not likely and probably not worth checking for.

Jeff Barber <jeffbabar>
Fri 12 Feb 2010 04:37:09 PM UTC, comment #10: 

I slightly change the patch to use pcb.so_options and SOF_REUSEADDR instead of pcb.flags and TF_REUSEADDR (which I removed). This way it will be easier to integrate with socket options in the future.

Currently, this is still disabled at socket level since it's not fully implemented: The code and the define to enable it is there but init.c checks that it isn't used...

To come back on my question in comment #4: what happens if there are 2 FTP clients running at the same time on one PC connecting the server (e.g. port 1023 and port 1024)? I think the server would try to open 2 connections (20:1023 and 20:1024), which will fail since binding to port 20 fails (same local address and port, remote not checked).

How do other stacks handle this?

Simon Goldschmidt <goldsimon>
Group administrator
Fri 12 Feb 2010 03:35:08 PM UTC, comment #9: 

Checked in.

I do have one question left on this: wouldn't we want to dump the pcb in TIME_WAIT when we reuse its endpoint? It will hang around and never get packets because they are passed to non-TIME_WAIT pcbs first.

We can leave this to tcp_tmr, of course, it will be freed eventually...

Simon Goldschmidt <goldsimon>
Group administrator
Wed 21 Oct 2009 03:06:17 PM UTC, comment #8: 

That's of course OK for me.

Simon Goldschmidt <goldsimon>
Group administrator
Wed 21 Oct 2009 11:42:41 AM UTC, comment #7: 

Unless there's a strong case for this I'd rather leave it for 1.4.0 - it's a feature rather than a bug fix.

Kieran Mansley <kieranm>
Group Member
Fri 16 Oct 2009 09:17:05 PM UTC, comment #6: 

Anything against this going in to 1.3.2? It's still not a full support for SO_REUSEADDR (on the socket level), but nevertheless seems like a good start for it...

Simon Goldschmidt <goldsimon>
Group administrator
Thu 27 Aug 2009 02:22:47 AM UTC, comment #5: 

The only way I think that could happen (concurrent transfers with different endpoints bound to the same local port) is when a listening endpoint accepts two different connection requests.  In that case, both derive from the same listening endpoint but the two are distinguishable because the remote address/port is unique in each.

I don't think any platform allows multiple callers to bind different endpoints to the same address/port combination.

The patch only allows the reuse of the port if the existing PCB using it is already "dead".  (It's hanging around in TIME_WAIT only for the unusual cases where the last ACK got lost or there is a duplicate stale packet wandering around the net.)

Jeff Barber <jeffbabar>
Wed 26 Aug 2009 09:27:15 PM UTC, comment #4: 

What about 2 concurrent transfers where the first hasn't reached TIME_WAIT yet? Does that work already or do we need to change something for that, too?

Simon Goldschmidt <goldsimon>
Group administrator
Tue 14 Jul 2009 03:55:44 PM UTC, comment #3: 

This happens in tcp_bind prior to the tcp_listen or tcp_connect so there isn't yet a remote address or port to check.


Jeff Barber <jeffbabar>
Tue 14 Jul 2009 03:38:14 PM UTC, comment #2: 

Rather than just bypassing the checking of the PCBs in TIME_WAIT, shouldn't we test that if we're going to try and re-use one from the TIME_WAIT state that the remote address/port/protocol differs from the earlier value?

Kieran Mansley <kieranm>
Group Member
Tue 14 Jul 2009 03:37:44 PM UTC, comment #1: 

I should have mentioned that I also included a fairly trivial bug fix in the patch:  I duplicated the traversal code used in checking the other PCB lists for the time-wait list.  The previous code checking the time-wait list did not explicitly check for ipaddr_is_any() which originally caused my code to crash because I was using a NULL pointer rather than &ip_addr_any. 

(I don't whether passing a null pointer to mean "any" is really recommended or not but it seems to be handled in most places in the stack correctly.)

Jeff Barber <jeffbabar>
Tue 14 Jul 2009 03:22:04 PM UTC, original submission:  

In developing an FTP server (for example) it is necessary to bind each outgoing PORT connection to the well-known FTP-data port (20).  Back-to-back transfers usually mean that the second and subsequent connections need to be bound to port 20 while the previous PCB is still in TIME_WAIT.  Hence, the equivalent of SO_REUSEADDR required in order to implement the server.

This patch provides a new TCP flag "TF_REUSEADDR" that, when set during the processing of tcp_bind, bypasses the checking of the PCBs still in TIME_WAIT.

Applies cleanly to top-of-tree and 1.3.1-RC1.  It does not apply cleanly to 1.3.0 but the sole failure is the one line definition of the new flag and is trivial to fix.

Usage:
    pcb = tcp_new();
    pcb->flags |= TF_REUSEADDR;
    status = tcp_bind(pcb, &in_addr_any, FTP_DATA_PORT);
    ...

Sorry, I only use the raw API with NO_SYS=1 so I can't provide a patch that works for other APIs.

Jeff Barber <jeffbabar>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #18422:  tcp_reuseaddr.patch added by jeffbabar (2KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by olegreen (Posted a comment)
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by kieranm (Posted a comment)
  • -email is unavailable- added by jeffbabar (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-05-12 goldsimon StatusNeed Info Done
        Open/ClosedOpen Closed
    2010-02-12 goldsimon StatusDone Need Info
    2010-02-12 goldsimon StatusNone Done
        Assigned toNone goldsimon
    2009-08-20 kieranm Planned ReleaseNone 1.4.0
    2009-07-14 jeffbabar Attached File- Added tcp_reuseaddr.patch, #18422

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code