tasklwIP - A Lightweight TCP/IP stack - Tasks: task #7896, Support zero-copy drivers


You are not allowed to post comments on this tracker with your current authentication level.

task #7896: Support zero-copy drivers

Submitted by:  Jonathan Larmour <jifl>
Submitted on:  Thu 27 Mar 2008 10:28:14 PM UTC  
Category: Network driversShould Start On: Thu 27 Mar 2008 12:00:00 AM UTC
Should be Finished on: Thu 26 Jun 2008 11:00:00 PM UTCPriority: 5 - Normal
Status: NonePrivacy: Public
Percent Complete: 50%Assigned to: None
Open/Closed: OpenPlanned Release: None
Effort: 0.00

(Jump to the original submission Jump to the original submission)

Wed 24 May 2017 05:15:54 PM UTC, comment #16:

I'd be happy to help testing any changes. For my AVR32-based project I wrote my own MAC driver with both zero-copy RX and TX, without any problems so far. Luckily on this machine the DMA for TX does not require any special alignment, and the 2-byte RX padding to align the Ethernet header is implemented in hardware.

Daniel Elstner <danielk>
Mon 22 May 2017 08:10:57 PM UTC, comment #15:

ZeroCopyRx.c seems unfinished as it doesn't cover dma ring allocation. I think we should describe this. And the fact that using preallocated PBUF_POOL is the best way to try this...

Simon Goldschmidt <goldsimon>
Project Administrator
Sat 07 Jan 2017 07:53:33 PM UTC, comment #14:

See also discussion in bug #49914

Dirk Ziegelmeier <dziegel>
Project Member
Thu 01 Oct 2015 10:42:05 PM UTC, comment #13:

I created task #13757 to investigate the resetting the RTO during a fast retransmit

Joel Cunningham <jcunningham>
Project Member
Thu 01 Oct 2015 06:48:35 PM UTC, comment #12:

> I've actually encountered the TCP retransmission issue

Me too with lwIP in situations where the MAC driver was slow in sending out. In this case I just ignored pbufs with ref > 1. Good enough for me, but not a good option in general, I guess.

> pcb->rtime is only reset during an actual RTO via the slow
> timer and not during tcp_rexmit_fast().

That's somewhat off-topic here, but I'd appreaciate if you had more info on this (maybe in its own bug tracker entry).

Simon Goldschmidt <goldsimon>
Project Administrator
Thu 01 Oct 2015 03:10:24 PM UTC, comment #11:

I've actually encountered the TCP retransmission issue with another embedded stack that supported asynchronous zero-copy TX. That stack had a behavior where if a fast retransmit happened, it didn't restart the original RTO timer. This made it possible that if a fast retransmit happened within close succession to the original RTO timing out, we'd end up performing two retransmits of the same packet

Taking a quick look at LwIP, I think the same thing could happen. pcb->rtime is only reset during an actual RTO via the slow timer and not during tcp_rexmit_fast().

Joel Cunningham <jcunningham>
Project Member
Thu 01 Oct 2015 02:18:53 PM UTC, comment #10:

It would be asynchronous zero-copy TX. And it already works if the assumption holds that TCP retransmission is only done when TX is long over. At least it works for me ;-)

Your last paragraph is correct though. And maybe letting TCP duplicate the pbuf on retransmission could be a good idea as well to help zero-copy drivers.

Simon Goldschmidt <goldsimon>
Project Administrator
Thu 01 Oct 2015 02:01:39 PM UTC, comment #9:

For zero-copy TX into the driver, is the goal here to support a synchronous zero-copy output through the netif_output_fn()?

Meaning that once the output function returns, the transmit has been complete and the pbuf memory is no longer referenced by the driver

I ask because in my products, the underlying driver operates in an asynchronous manner where the packet is enqueued during the netif_output_fn and then is transmitted at a later time.

Asynchronous zero-copy TX is more complicated. I'm thinking it would work by the driver queuing a reference to the pbuf (incrementing pbuf->ref) and then calling pbuf_free() once it has completed the transmit. Any modifications to the pbuf when pbuf->ref > 1 (by LwIP on TCP rexmit or driver) would be required making a copy of the packet. Also, any PBUF_REF would need to be copied since that memory may not be in scope after LwIP is done with the netif_output_fn (UDP fits this case)

Joel Cunningham <jcunningham>
Project Member
Wed 30 Sep 2015 02:23:03 PM UTC, comment #8:

Using PBUF_REF for RX should now work.

DMA-alignment of TX buffers remains to be done...

Simon Goldschmidt <goldsimon>
Project Administrator
Sun 22 Feb 2015 09:14:22 PM UTC, comment #7:

We're actually working on this. It showed up that we still need to
- fully support a REF type pbuf for RX (fails at least on IPv6 reassembly, I think)
- need to DMA-align TX buffers (the resulting payload pointers that get sent on the link). This might be more tricky as it would mean the 'header' of a pbuf is a different one for UDP/TCP and IPv4/IPv6...

Simon Goldschmidt <goldsimon>
Project Administrator
Fri 07 Feb 2014 06:50:32 PM UTC, comment #6:

And another user mail requesting support for this:

Simon Goldschmidt <goldsimon>
Project Administrator
Tue 13 Mar 2012 04:39:26 PM UTC, comment #5:

This isn't needed for 1.4.1

Kieran Mansley <kieranm>
Project Administrator
Thu 29 Sep 2011 06:21:02 AM UTC, comment #4:

Just a link to 2 mails on the mailing list from users using DMA:


http://lists.gnu.org/archive/html/lwip-devel/2011-09/msg00057.html (lower half of the mail)

Simon Goldschmidt <goldsimon>
Project Administrator
Thu 24 Jun 2010 08:30:28 PM UTC, comment #3:

I'm setting this to 1.4.1 (if not later): we need people with bug reports who can tell us in which way they need this fixed. For me, it just works (after the changes to IP_FRAG and to the docs that say applications may not reuse pbufs).

Simon Goldschmidt <goldsimon>
Project Administrator
Sat 22 May 2010 08:59:29 AM UTC, comment #2:

I am wondering how zero-copy transmission could be implemented with Coldfire v2 Ethernet controller. The limitations of its DMA is that transmit buffers must be 32-bit aligned and each buffer's size must be a multiple of 16 bytes. The latter limitation is strange since in the very same microcontroller manual they suggest to allocate one buffer for IP header, another for TCP header and so on, but those headers' sizes are not multiples of 16 bytes...
Anyway, with those limitations I cannot see how zero-copy transmission could work without imposing severe restrictions on how the application supplies data to lwip. Basically, it would not be practical.
Am I right in this?

Mike Kleshov <kleshov>
Project Member
Tue 18 May 2010 07:32:43 PM UTC, comment #1:

Ok, seems like this is one of the last tasks left before releasing 1.4.0 (together with bug #29361, which is kind of related to this one).

I think there is not much coding involved here but we rather need to know what we need to support zero copy. For that, we would need input from many developers/users telling us what they need. Personally, the only requirements for my implementation are:

- delayed freeing of TX pbufs (i.e. transmit might take some time / pbuf is not transmitted when netif->linkoutput returns - this is already taken care of by adjusting the docs to say "applications may not reuse pbufs after passing them to send/output functions")
- RX pbufs should be properly aligned (depending on the cache line size - this currently doesn't work as there is only one MEM_ALIGNMENT)

As Jonathan pointed out some more requirements below, I think the most portable solution would be to allow the driver to allocate pbufs in its own memory and to let pbuf_free() call a port-defined function to free those pbufs - we have plenty numbers left in the 'enum pbuf_type'.

This would also help to solve bug #29361. However, the downside is that the whole stack would have to be reviewed in order to remove assumptions made on different pbuf types (i.e. pbuf_header fails on PBUF_REF in some cases, etc.).

Simon Goldschmidt <goldsimon>
Project Administrator
Thu 27 Mar 2008 10:28:14 PM UTC, original submission:

This is an off-shoot from task #6735 which is now invalid. Here's what I wrote there:

There's a lot to do to support zero-copy transmits and receives. For receives, there are usually requirements for things such as using specific memory regions (PCI windows, or dedicated DMA regions, or on-chip RAM, etc.), as well as alignment and buffer size constraints. One port I've already done with my own zero-copy rx mods is to AT91SAM7X which uses fixed 128-byte buffers, thus requiring fixed 128-byte pool pbufs. Fortunately I could still keep the existing pool structure, but that won't be true on other systems - it will need to be possible to separate the struct pbuf, from the following pool memory itself. The simplest way I can think of is to keep the struct pbuf and the pbuf memory itself in parallel arrays. So if you have the pbuf start, then since the pbufs are of constant size and you know the base, you can easily calculate the index of this pbuf, and therefore the index of the corresponding struct pbuf.

Another requirement is that drivers need to be informed when more of that pool memory becomes available, as often the hardware has its own circular buffer lists, and these will need to be marked as available again, once a pool pbuf has been freed.

So far I only have a solution for one ethernet driver at a time. Implementing this for multiple ones (efficiently) is more tricky since of course it's possible for drivers to have different requirements. The easiest solution is probably to say they can't do it then.

Jonathan Larmour <jifl>
Project Member


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by danielk (Posted a comment)
  • -unavailable- added by dziegel (Posted a comment)
  • -unavailable- added by jcunningham (Posted a comment)
  • -unavailable- added by kieranm (Posted a comment)
  • -unavailable- added by kleshov (Posted a comment)
  • -unavailable- added by goldsimon (Posted a comment)
  • -unavailable- added by jifl (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 30 Sep 2015 02:23:09 PM UTCgoldsimonPercent Complete0%=>50%
    Tue 15 Jan 2013 08:21:57 AM UTCgoldsimonCategoryNone=>Network drivers
    Tue 13 Mar 2012 04:39:26 PM UTCkieranmPlanned Release1.4.1=>None
    Thu 24 Jun 2010 08:30:28 PM UTCgoldsimonPlanned Release1.4.0=>1.4.1

    Back to the top

    Powered by Savane 3.1-cleanup1