tasklwIP - A Lightweight TCP/IP stack - Tasks: task #7896, Support zero-copy drivers

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #7896: Support zero-copy drivers

Submitter:  Jonathan Larmour <jifl>
Submitted:  Thu 27 Mar 2008 10:28:14 PM UTC
   
 
Category:  Network drivers Should Start On:  Thu 27 Mar 2008 12:00:00 AM UTC
Should be Finished on:  Thu 26 Jun 2008 11:00:00 PM UTC Priority:  5 - Normal
Status:  None Privacy:  Public
Assigned to:  None Percent Complete:  50%
Open/Closed:  Open Planned Release:  None
Effort:  0.00

Jump to the original submission

Wed 24 May 2017 05:15:54 PM UTC, comment #16: 

I'd be happy to help testing any changes. For my AVR32-based project I wrote my own MAC driver with both zero-copy RX and TX, without any problems so far. Luckily on this machine the DMA for TX does not require any special alignment, and the 2-byte RX padding to align the Ethernet header is implemented in hardware.

Daniel Elstner <danielk>
Mon 22 May 2017 08:10:57 PM UTC, comment #15: 

ZeroCopyRx.c seems unfinished as it doesn't cover dma ring allocation. I think we should describe this. And the fact that using preallocated PBUF_POOL is the best way to try this...

Simon Goldschmidt <goldsimon>
Group administrator
Sat 07 Jan 2017 07:53:33 PM UTC, comment #14: 

See also discussion in bug #49914

Dirk Ziegelmeier <dziegel>
Group administrator
Thu 01 Oct 2015 10:42:05 PM UTC, comment #13: 

I created task #13757 to investigate the resetting the RTO during a fast retransmit

Joel Cunningham <jcunningham>
Group Member
Thu 01 Oct 2015 06:48:35 PM UTC, comment #12: 


> I've actually encountered the TCP retransmission issue


Me too with lwIP in situations where the MAC driver was slow in sending out. In this case I just ignored pbufs with ref > 1. Good enough for me, but not a good option in general, I guess.

>  pcb->rtime is only reset during an actual RTO via the slow
> timer and not during tcp_rexmit_fast().


That's somewhat off-topic here, but I'd appreaciate if you had more info on this (maybe in its own bug tracker entry).

Simon Goldschmidt <goldsimon>
Group administrator
Thu 01 Oct 2015 03:10:24 PM UTC, comment #11: 

I've actually encountered the TCP retransmission issue with another embedded stack that supported asynchronous zero-copy TX.  That stack had a behavior where if a fast retransmit happened, it didn't restart the original RTO timer. This made it possible that if a fast retransmit happened within close succession to the original RTO timing out, we'd end up performing two retransmits of the same packet

Taking a quick look at LwIP, I think the same thing could happen. pcb->rtime is only reset during an actual RTO via the slow timer and not during tcp_rexmit_fast().

Joel Cunningham <jcunningham>
Group Member
Thu 01 Oct 2015 02:18:53 PM UTC, comment #10: 

It would be asynchronous zero-copy TX. And it already works if the assumption holds that TCP retransmission is only done when TX is long over. At least it works for me ;-)

Your last paragraph is correct though. And maybe letting TCP duplicate the pbuf on retransmission could be a good idea as well to help zero-copy drivers.

Simon Goldschmidt <goldsimon>
Group administrator
Thu 01 Oct 2015 02:01:39 PM UTC, comment #9: 

For zero-copy TX into the driver, is the goal here to support a synchronous zero-copy output through the netif_output_fn()? 

Meaning that once the output function returns, the transmit has been complete and the pbuf memory is no longer referenced by the driver

I ask because in my products, the underlying driver operates in an asynchronous manner where the packet is enqueued during the netif_output_fn and then is transmitted at a later time.

Asynchronous zero-copy TX is more complicated.  I'm thinking it would work by the driver queuing a reference to the pbuf (incrementing pbuf->ref) and then calling pbuf_free() once it has completed the transmit.  Any modifications to the pbuf when pbuf->ref > 1 (by LwIP on TCP rexmit or driver) would be required making a copy of the packet.  Also, any PBUF_REF would need to be copied since that memory may not be in scope after LwIP is done with the netif_output_fn (UDP fits this case)

Joel Cunningham <jcunningham>
Group Member
Wed 30 Sep 2015 02:23:03 PM UTC, comment #8: 

Using PBUF_REF for RX should now work.

DMA-alignment of TX buffers remains to be done...

Simon Goldschmidt <goldsimon>
Group administrator
Sun 22 Feb 2015 09:14:22 PM UTC, comment #7: 

We're actually working on this. It showed up that we still need to
- fully support a REF type pbuf for RX (fails at least on IPv6 reassembly, I think)
- need to DMA-align TX buffers (the resulting payload pointers that get sent on the link). This might be more tricky as it would mean the 'header' of a pbuf is a different one for UDP/TCP and IPv4/IPv6...

Simon Goldschmidt <goldsimon>
Group administrator
Fri 07 Feb 2014 06:50:32 PM UTC, comment #6: 

And another user mail requesting support for this:
https://lists.gnu.org/archive/html/lwip-users/2014-02/msg00003.html

Simon Goldschmidt <goldsimon>
Group administrator
Tue 13 Mar 2012 04:39:26 PM UTC, comment #5: 

This isn't needed for 1.4.1

Kieran Mansley <kieranm>
Group Member
Thu 29 Sep 2011 06:21:02 AM UTC, comment #4: 

Just a link to 2 mails on the mailing list from users using DMA:

http://lists.gnu.org/archive/html/lwip-users/2003-03/msg00085.html

http://lists.gnu.org/archive/html/lwip-devel/2011-09/msg00057.html (lower half of the mail)

Simon Goldschmidt <goldsimon>
Group administrator
Thu 24 Jun 2010 08:30:28 PM UTC, comment #3: 

I'm setting this to 1.4.1 (if not later): we need people with bug reports who can tell us in which way they need this fixed. For me, it just works (after the changes to IP_FRAG and to the docs that say applications may not reuse pbufs).

Simon Goldschmidt <goldsimon>
Group administrator
Sat 22 May 2010 08:59:29 AM UTC, comment #2: 

I am wondering how zero-copy transmission could be implemented with Coldfire v2 Ethernet controller. The limitations of its DMA is that transmit buffers must be 32-bit aligned and each buffer's size must be a multiple of 16 bytes. The latter limitation is strange since in the very same microcontroller manual they suggest to allocate one buffer for IP header, another for TCP header and so on, but those headers' sizes are not multiples of 16 bytes...
Anyway, with those limitations I cannot see how zero-copy transmission could work without imposing severe restrictions on how the application supplies data to lwip. Basically, it would not be practical.
Am I right in this?

Mike Kleshov <kleshov>
Group Member
Tue 18 May 2010 07:32:43 PM UTC, comment #1: 

Ok, seems like this is one of the last tasks left before releasing 1.4.0 (together with bug #29361, which is kind of related to this one).

I think there is not much coding involved here but we rather need to know what we need to support zero copy. For that, we would need input from many developers/users telling us what they need. Personally, the only requirements for my implementation are:

- delayed freeing of TX pbufs (i.e. transmit might take some time / pbuf is not transmitted when netif->linkoutput returns - this is already taken care of by adjusting the docs to say "applications may not reuse pbufs after passing them to send/output functions")
- RX pbufs should be properly aligned (depending on the cache line size - this currently doesn't work as there is only one MEM_ALIGNMENT)

As Jonathan pointed out some more requirements below, I think the most portable solution would be to allow the driver to allocate pbufs in its own memory and to let pbuf_free() call a port-defined function to free those pbufs - we have plenty numbers left in the 'enum pbuf_type'.

This would also help to solve bug #29361. However, the downside is that the whole stack would have to be reviewed in order to remove assumptions made on different pbuf types (i.e. pbuf_header fails on PBUF_REF in some cases, etc.).

Simon Goldschmidt <goldsimon>
Group administrator
Thu 27 Mar 2008 10:28:14 PM UTC, original submission:  

This is an off-shoot from task #6735 which is now invalid. Here's what I wrote there:


There's a lot to do to support zero-copy transmits and receives. For receives, there are usually requirements for things such as using specific memory regions (PCI windows, or dedicated DMA regions, or on-chip RAM, etc.), as well as alignment and buffer size constraints. One port I've already done with my own zero-copy rx mods is to AT91SAM7X which uses fixed 128-byte buffers, thus requiring fixed 128-byte pool pbufs. Fortunately I could still keep the existing pool structure, but that won't be true on other systems - it will need to be possible to separate the struct pbuf, from the following pool memory itself. The simplest way I can think of is to keep the struct pbuf and the pbuf memory itself in parallel arrays. So if you have the pbuf start, then since the pbufs are of constant size and you know the base, you can easily calculate the index of this pbuf, and therefore the index of the corresponding struct pbuf.

Another requirement is that drivers need to be informed when more of that pool memory becomes available, as often the hardware has its own circular buffer lists, and these will need to be marked as available again, once a pool pbuf has been freed.

So far I only have a solution for one ethernet driver at a time. Implementing this for multiple ones (efficiently) is more tricky since of course it's possible for drivers to have different requirements. The easiest solution is probably to say they can't do it then.


Jonathan Larmour <jifl>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by danielk (Posted a comment)
  • -email is unavailable- added by dziegel (Posted a comment)
  • -email is unavailable- added by jcunningham (Posted a comment)
  • -email is unavailable- added by kieranm (Posted a comment)
  • -email is unavailable- added by kleshov (Posted a comment)
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by jifl (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-09-30 goldsimon Percent Complete0% 50%
    2013-01-15 goldsimon CategoryNone Network drivers
    2012-03-13 kieranm Planned Release1.4.1 None
    2010-06-24 goldsimon Planned Release1.4.0 1.4.1

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code