buglwIP - A Lightweight TCP/IP stack - Bugs: bug #27049, DHCP IP address assignment with...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #27049: DHCP IP address assignment with multiple lwIP devices fails

Submitter:  Bill Auerbach <billauerbach>
Submitted:  Fri 17 Jul 2009 05:04:50 PM UTC
   
 
Category:  UDP Severity:  3 - Normal
Item Group:  Faulty Behaviour Status:  Fixed
Privacy:  Public Assigned to:  stoklund
Open/Closed:  Closed Planned Release:  None
lwIP version:  CVS Head

Jump to the original submission

Mon 27 Jul 2009 03:13:00 PM UTC, comment #29: 

Thanks

Kieran Mansley <kieranm>
Group Member
Mon 27 Jul 2009 02:47:30 PM UTC, comment #28: 

Patches were committed Thursday, but I think the messages to lwip-commits were blocked because it was my first commit to lwip. I should have notified you, my bad.

Jakob Stoklund Olesen <stoklund>
Group Member
Mon 27 Jul 2009 02:04:45 PM UTC, comment #27: 

Please do so, and then close this when done so I can prepare the next release candidate

Kieran Mansley <kieranm>
Group Member
Wed 22 Jul 2009 04:38:39 PM UTC, comment #26: 

I will commmit the patches iof there are no objections

Jakob Stoklund Olesen <stoklund>
Group Member
Wed 22 Jul 2009 03:03:42 PM UTC, comment #25: 

Great!  I'm happy with this.  Something about it is better.  Although the 5th device retries a discover and gets the retried offer, the router processes the earlier request and ACKs it.  I don't recall seeing this earlier.

5 devices come up in under 6 seconds even if the 5th has to retry its discover (which occurs less than half the time).  This is a significant improvement over 22 seconds for 4 devices.

Thank you for the effort.

Bill Auerbach <billauerbach>
Wed 22 Jul 2009 12:55:58 PM UTC, comment #24: 

The patch in msg #23 doesn't have:

 msecs = (dhcp->tries < 6 ? 1 << dhcp->tries : 60) * 1000;

Bill Auerbach <billauerbach>
Wed 22 Jul 2009 09:16:00 AM UTC, comment #23: 

Patch attached. Unfortunately I am not currently set up to test it.

The patch reuses XIDs for retransmissions as discussed. In order to do this, I have slightly changed the dhcp state machine behaviour when dhcp_create_request() returns a memory error. I actually think the new behaviour is better - the state machine now treats out-of-memory errors the same as a lost packet.

Bill, could I ask you to test this patch with 5 devices, please?

(file #18453)

Jakob Stoklund Olesen <stoklund>
Group Member
Wed 22 Jul 2009 08:03:49 AM UTC, comment #22: 

Happy to go with Jakob's solution as I too was concerned that by increasing the timeouts a little we'd just postpone the problem to a larger number of devices.

If someone can get this in quickly I reckon it could go into 1.3.1 as I think we'll need a second release candidate.

Kieran Mansley <kieranm>
Group Member
Tue 21 Jul 2009 09:40:56 PM UTC, comment #21: 

I should say, I don't have a problem with increasing the dhcp retransmission timeouts. In fact, RFC2131 ยง4.1 requires a randomized, exponential backoff with an initial value suited to the network speed.

I suggest:

  msecs = (dhcp->tries < 6 ? 1 << dhcp->tries : 60) * 1000;

That produces timeouts of 2, 4, 8, 16, 32, 60, 60, ... seconds.

Exponential backoff is better in the situation where many devices are turned on simultaneously.

Jakob Stoklund Olesen <stoklund>
Group Member
Tue 21 Jul 2009 08:41:14 PM UTC, comment #20: 

The problem with increasing the timeout is that it doesn't solve your problem. It looks like your router can make about one offer per second. This caused problems with an initial timeout of one second. If you increase the timeout to 4 seconds, you will get into trouble when you have 5 devices or more; 10 seconds, 11 devices. And so on.

On the other hand, if lwip reuses XIDs in retransmissions, it can accept an offer that has been arbitrarily delayed. You will be able to boot 100 devices in two minutes.

Of course your fallback to a static IP after a fixed timeout will limit the number of devices that can be turned on at once. Note that AutoIP fallback keeps the DHCP client process running. Yes, it will choose an AutoIP address after N tries, but as soon as it receives a DHCP offer+ACK, it goes back to the DHCP address. That means it can handle very long DHCP delays and still be quick to boot with AutoIP.

Sure, retransmissions is not a good thing, but you have to admit that booting 5 devices simultaneously off a consumer grade router is not a normal situation.

Thanks for explaining the 5-device pcap. Notice that all requests have been acked after 7 seconds, and in fact the last 4 devices are handled (discover to ack) in 4 seconds. That limit is imposed by the router, not by lwip retransmissions.

Jakob Stoklund Olesen <stoklund>
Group Member
Tue 21 Jul 2009 08:10:05 PM UTC, comment #19: 

Yes, it is slow.  I meant to say that the 5 device capture was a test - its the original DHCP with accepting responses to XID and XID-1.  This way the behavior is as-if the ID is the same, but I could see which packets the DHCP server was replying to.

You're right, it's slow.  Each device makes it even slow, as if it has a queue and processes them in order a little at a time.

I have also been corrected on something.  We have a 24-port switch and the router running DHCP is the cable modem.  This helps explain the slowness I think.

IMO, retries indicate a problem.  The retries should be eliminated if they reasonably can be.  The initial timeouts seem to have been chosen because they were reasonable when the code was developed.  I have a case that shows they should be longer.  If it can take an off-the-shelf router to 4-5 seconds to reply, then a 3 second timeout is going to be too short.

I guess I'm done arguing for longer timeouts.  No one has countered why it's a bad idea or what could possible break or be effected by doing so.

I see no harm in repeating the IDs being used as suggested, but IMO it's only helping the problem, not solving the problem.

I can test 5 devices coming up on our slow router and am happy to test anything want to see.

Bill Auerbach <billauerbach>
Tue 21 Jul 2009 07:40:19 PM UTC, comment #18: 

Bill, the last pcaps you attached look perfectly healthy to me. Why do you think the DHCP server is starting over? The XID you mention goes like this:


  4   0.780724      0.0.0.0 -> 255.255.255.255 DHCP DHCP Discover - Transaction ID 0x1485a4c
 13   3.823652      0.0.0.0 -> 255.255.255.255 DHCP DHCP Discover - Transaction ID 0x1485a4c
 16   4.366731 192.168.123.254 -> 255.255.255.255 DHCP DHCP Offer    - Transaction ID 0x1485a4c
 17   4.367109      0.0.0.0 -> 255.255.255.255 DHCP DHCP Request  - Transaction ID 0x1485a4d
 23   5.182788 192.168.123.254 -> 255.255.255.255 DHCP DHCP Offer    - Transaction ID 0x1485a4c
 26   5.187731 192.168.123.254 -> 255.255.255.255 DHCP DHCP ACK      - Transaction ID 0x1485a4d


lwip ignores the second offer because it already has one - it has sent REQUEST and moved on to a new state. The server sends two OFFER messages because there were two DISCOVER messages. It is just really slow.

In your dhcp_5_devices.pcap file, it looks like you have not updated the software to reuse XIDs.

Jakob Stoklund Olesen <stoklund>
Group Member
Tue 21 Jul 2009 06:55:44 PM UTC, comment #17: 

The question is whether the server must start over (according to the RFC) or if it is allowed to process every packet as it comes in. If it starts over, reusing XIDs won't change anything. However, if it doesn't (e.g. embedded device that takes some time for answering), things get easier when reusing XIDs.

The point is I don't have a clue why cisco routers take so long to answer: I cannot imagine them being poorly programmed, so it must be some kind of load balancing.

To summarize, we should both increase the timeouts and think about reusing XIDs, but as the RFCs don't seem too chatty about this, I think it would be a good solution to look at other OSes/stacks and see what they do (since most routers/DHCP servers will be compatible to windows or linux).

Anyway, is this planned for 1.3.1 or later?

Simon Goldschmidt <goldsimon>
Group administrator
Tue 21 Jul 2009 06:23:52 PM UTC, comment #16: 

If the router starts over on a retry then it doesn't matter what the ID is does it?  I now have 5 devices powering up together and I have a question in the PCAP.  Why are there 2 back-to-back DHCP Requests from the same MAC? (packets 59 & 65)  Oh, maybe it timed out and is re-requesting instead of ACKing.  Again this points to timeouts being too short.

(file #18447)

Bill Auerbach <billauerbach>
Tue 21 Jul 2009 06:12:45 PM UTC, comment #15: 


> Therefore, I think there is no point in processing old DHCP
> offers because the router treats the ACK as invalid.


Still, reusing XIDs, like Jakob suggested in comment #11, should be OK. However, I didn't double check if that's OK with the RFC.

Simon Goldschmidt <goldsimon>
Group administrator
Tue 21 Jul 2009 05:32:08 PM UTC, comment #14: 

I have verified that with my router, if a DHCP discovery is started, any DHCP processing from the same MAC address is aborted.  Therefore, I think there is no point in processing old DHCP offers because the router treats the ACK as invalid.

Bill Auerbach <billauerbach>
Tue 21 Jul 2009 03:52:33 PM UTC, comment #13: 

No, sorry.  It looks like the repeated request has canceled the open offer and there is no DHCP ACK.  The retried requests's offer is ACKed.  Maybe you'll disagree.  PCAP attached. The ID of interest ends in 5a4c.

Nonetheless, I don't see the harm in increasing the timeout 2 or 3 seconds.  This prevents the retries.



(file #18446)

Bill Auerbach <billauerbach>
Tue 21 Jul 2009 03:35:38 PM UTC, comment #12: 

There must be another dhcp_set_state because on the timed out and retried DHCP Request, the ID has been incremented.  The DHCP Offer was .4mS before the repeated request and wasn't processed.  A request after a time out must be setting the state?

Bill Auerbach <billauerbach>
Tue 21 Jul 2009 08:15:58 AM UTC, comment #11: 

I think the proper solution would be to reuse XID for retransmissions. This is perfectly OK according to RFC2131.

In dhcp_create_request() only increment xid when dhcp->tries==0:


  /* give unique transaction identifier to this request */
  if (dhcp->tries==0) xid++;
  dhcp->xid = xid;


In dhcp_select() and dhcp_discover() call dhcp_set_state() before calling dhcp_create_request().

Bill, could you try this, please?

Jakob Stoklund Olesen <stoklund>
Group Member
Mon 20 Jul 2009 09:51:14 PM UTC, comment #10: 

Right now, the timeouts in DHCP_SELECTING state are:

2, 3, 4, 10, 10 ... seconds. The change I made makes it:

4, 5, 6, 10, 10 ... seconds.  The only effect that will be noticed is if the first DHCP offer is missed.  It will be 2 seconds longer to make another request.  If there is no DHCP server, it's a moot point because it never exists the DHCP_SELECTING state (AFAICT).  If AUTOIP is used, it checks retries, so it will be 2 seconds later before falling back.  But without this change and AUTOIP reties set to 1, it might fall back when it could have received an address.

This only solves a problem, from what I see, when multiple units come up together and possibly only with some routers.

The effect may only be present for a timed fallback such as I implemented.  Although you could argue I should give it more time, the converse argument is I don't think 20+ seconds for 4 units to acquire IP addresses is reasonable when single devices do it in a couple of seconds.

Note that this timeout algorithm is repeated in a couple of places, but this is for the case of waiting for a DHCP request.

I found nothing in DHCP RFCs that call out an explicit time out.  You could conclude that a router manufacturer has no spec to follow as well.  Considering the speed of a router and it's priority to deliver packets quickly, I would expect its DHCP task to be a lower priority task.

Bill Auerbach <billauerbach>
Mon 20 Jul 2009 08:05:29 AM UTC, comment #9: 

I agree that adjusting the timeouts seems the most sensible approach.  The question is to what?  comment #7 alludes to something about 4 per 60 seconds from the RFC.  Is there anything more concrete in the RFC (or other implementations) that we could follow?

Kieran Mansley <kieranm>
Group Member
Sat 18 Jul 2009 08:46:42 AM UTC, comment #8: 

Unfortunately, I don't know the DHCP RFC very well, but it seems like our timeouts are at least not taken from there.

Accepting responses to earlier packets would do the trick as well, but we would also need to ignore responses for later packets (which will come, or won't they?) when a response for an earlier packet has been received. And we would need a list of IDs used for requests to compare to (or a range).

Since that would lead us away from being lightweight a bit, I'd favor adjusting the timeouts. After all, in a normal situation, these timeouts don't occur (or only when using fallback to AutoIP).

Simon Goldschmidt <goldsimon>
Group administrator
Fri 17 Jul 2009 07:35:36 PM UTC, comment #7: 

It's a Cisco (LinkSys) 24 port GigE router.  It could be part of the problem but being an industry standard small networking device, I would expect it to be fairly common but I suspect used mostly for PCs.  PCs never come up together.  Embedded devices can (and do).

We discussed adding a random power-up delay to the device, but that won't guarantee that 2 or more devices still won't come up at the same time.

RFC 2131 makes more than 1 reference to sending 4 DHCPREQUESTs over a 60 second time period.  This would suggest our DHCP timeouts are way too short.

The way it is now, it's possible for many (12+) lwIP devices to come up and have it take a minute or longer to finish with a router like ours here.

Another solution might be to accept DHCP responses with IDs less than we expect (since they are older but still valid responses).

Bill Auerbach <billauerbach>
Fri 17 Jul 2009 07:18:28 PM UTC, comment #6: 

All 4 of my units now come up consistently in under 3.5 seconds with the following change in dhcp_discover:

  msecs = dhcp->tries < 4 ? (dhcp->tries + 3) 1000 : 10 1000;

This also stops the thrashing that I was seeing.

Bill Auerbach <billauerbach>
Fri 17 Jul 2009 07:13:40 PM UTC, comment #5: 

Looking at the pcap it seems the DHCP server is wrong, not the client. Could it somehow be that it is somehow overloaded and cannot respond fast enough?

I wouldn't set the timeout higher as default in lwIP, but making it configurable would be OK, I guess... That is, if it really helps..

Simon Goldschmidt <goldsimon>
Group administrator
Fri 17 Jul 2009 06:32:06 PM UTC, comment #4: 

No, the IDs are unique.

Looks like a timing issue.  The DHCP server seems to take longer to handle multiple requests and so lwIP starts asking for repeated "Requests".  It doesn't complete in my 10 second timeout before I fall back to a static IP.  I'm going to see if 20 seconds helps.  It appears if lwIP were a little more patient, it would get the response before it re-requests again and it would all complete faster.  Attached is a pcap of 4 devices coming up together which takes more than 10 seconds to complete.

(file #18430)

Bill Auerbach <billauerbach>
Fri 17 Jul 2009 05:52:28 PM UTC, comment #3: 

Just to be clear: It didn't help the repeating IDs, or it didn't help the DHCP server confusion?

Robert Sprowson <sprow>
Fri 17 Jul 2009 05:41:47 PM UTC, comment #2: 

Yes, I did add this DHCP_GLOBAL_XID define after seeing the repeating IDs hoping that it was the solution.  It didn't help.

Bill Auerbach <billauerbach>
Fri 17 Jul 2009 05:25:41 PM UTC, comment #1: 

Did you define something (a rand() like function) for DHCP_GLOBAL_XID, per my patch #6763?

If not defined the default debug value of 0xABCD0000 will be used for all lwIP devices and cause much DHCP server confusion.

Robert Sprowson <sprow>
Fri 17 Jul 2009 05:04:50 PM UTC, original submission:  

Powering up multiple (3+) lwIP-based devices at the same time appears more often than not to confuse DHCP on all of the devices.  When this occurs, none of the devices complete DHCP and none receive an IP address.  By using WireShark, I can see the devices come up about 10mS to 15mS apart.  The DHCP exchanges are all present and appear OK, but they are intermingled which is why I suspect they are confusing each other.

If I power up each device manually seconds apart they all come up fine.

[We need a category for DHCP :-) ]

Bill Auerbach <billauerbach>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #18454:  dhcp-backoff.patch added by stoklund (1KiB - text/x-patch - Implement exponential backoff in dhcp)
file #18453:  dhcp-reuse-xid.patch added by stoklund (3KiB - text/x-patch - Reuse XIDs in dhcp.c)
file #18447:  dhcp_5_devices.pcap added by billauerbach (12KiB - application/octet-stream)
file #18446:  dhcp_4_devices.pcap added by billauerbach (12KiB - application/octet-stream)
file #18430:  dhcp_4_devices.pcap added by billauerbach (17KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by stoklund (Posted a comment)
  • -email is unavailable- added by kieranm (Posted a comment)
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by sprow (Posted a comment)
  • -email is unavailable- added by billauerbach (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-07-27 kieranm StatusReady For Test Fixed
    2009-07-27 stoklund Open/ClosedOpen Closed
    2009-07-23 stoklund StatusNone Ready For Test
        Assigned toNone stoklund
        Planned Release 1.3.1
    2009-07-22 stoklund Attached File- Added dhcp-backoff.patch, #18454
    2009-07-22 stoklund Attached File- Added dhcp-reuse-xid.patch, #18453
    2009-07-21 billauerbach Attached File- Added dhcp_5_devices.pcap, #18447
    2009-07-21 billauerbach Attached File- Added dhcp_4_devices.pcap, #18446
    2009-07-17 billauerbach Attached File- Added dhcp_4_devices.pcap, #18430

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code