patchlwIP - A Lightweight TCP/IP stack - Patches: patch #10111, [RFC] Hardware checksum offloading

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #10111: [RFC] Hardware checksum offloading

Submitter:  Simon Kuenzer <skuenzer>
Submitted:  Sat 25 Sep 2021 10:42:04 PM UTC
   
 
Category:  TCP Priority:  5 - Normal
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Planned Release:  None

Fri 12 Nov 2021 12:07:19 AM UTC, comment #5: 

comment #4:

> As outlook (not part of these patches yet), I plan to extend this work with another pbuf flag (for instance called DATA_VALID) that can tell the stack to skip checksum checks for packets. We need this on a per packet base because we need to be able to handle mixed traffic: (1) checksums might be already checked (but not necessarily) by a physical network card attached to the hypervisor, (2) the checksum field might be incomplete - actually incorrect - because traffic is received from another virtual machine that did offloading. Because this is an in-memory communication, no corruption is assumed.


I attached a completed version (see v2 patches, previous patches can be now ignored). The patches implement the partial checksum offloading capabilities for TCP and UDP for transmission and reception.
For this purpose, we introduce two pbuf flags in order to be able to handle mixed traffic:

  • PBUF_FLAG_CSUM_PARTIAL marks a packet as containing only a partially computed TCP or UDP checksum. The csum_start and csum_offset fields point to the checksum field of the header and to where the computation needs to be completed. As soon as the stack receives such a packet, we skip checksum validation but check csum_start and csum_offset. Similar to Linux, we assume that traffic with a partial checksum is only resulting from in-memory communication where corruption is unlikely, e.g., guest-to-guest, host-to-guest. Unfortunately, some virtual devices (e.g., netfront) do not tell csum_start and csum_offset which is why those fields are optional for reception and only checked when they are available.
  • PBUF_FLAG_DATA_VALID marks a packet whose UDP or TCP checksum has already been validated (e.g., physical NIC on the host). On reception, lwIP will skip checksum validation. For improving loopback traffic, this flag is set for transmitted TCP and UDP packets.


Through lwipopts.h, the new features can be compiled-in or even compiled-out. For each network device, the handling of partially checksummed traffic can be set through netif checksum control flags.
A current limitation of this version of implementation is the lack of support for forwarding and L2-bridging. Basically, the missing piece of the puzzle is a function that can complete the checksum computation in the software for partially checksummed packets that are forwarded to devices that do not support partially checksummed packets.

Simon Kuenzer <skuenzer>
Mon 27 Sep 2021 10:32:13 AM UTC, comment #4: 


comment #3:

> OK, thanks for clarifiying. Reading the code, I'd rather have this somehow in line with the 'IF__NETIF_CHECKSUM_ENABLED' macro: in other words, when changing code here, it would be best if there was the option to make this netif specific. Because if it's not netif specific, you'll have to disable it when mixing interfaces.


Right, this is a good point and I totally agree. In fact, I have considered and implemented this (patch 0003): Overall, I tried to introduce my changes so that you can still build a minimal lwIP configuration without extra pbuf fields. So the feature is added with two configurations: lwipopts.h and the netif feature flags. In lwipopt.h you need to enable:

- Enable checksum generation for TCP
- Additionally enable partial checksums for TCP

This enables all the required functions, extra pbuf fields and the pbuf flag.

The computation depends on netif feature flags:
- Without netif feature flags activated, lwip should always compute only a partial checksum.
- In case, netif feature flags are enabled, a partial checksum is only computed if the netif interface indicates this with a similar flag: Both, NETIF_CHECKSUM_GEN_TCP and NETIF_CHECKSUM_PARTIAL_TCP, need to be set. If only GEN_TCP is set, lwip computes the whole checksum in software. If none is set, the checksum field is kept zero.

Simon Kuenzer <skuenzer>
Mon 27 Sep 2021 09:42:45 AM UTC, comment #3: 

OK, thanks for clarifiying. Reading the code, I'd rather have this somehow in line with the 'IF__NETIF_CHECKSUM_ENABLED' macro: in other words, when changing code here, it would be best if there was the option to make this netif specific. Because if it's not netif specific, you'll have to disable it when mixing interfaces.

Simon Goldschmidt <goldsimon>
Group administrator
Mon 27 Sep 2021 09:32:10 AM UTC, comment #2: 

comment #1:

> > These patches implement the capability to offload checksum computation
>
> Without having the time to look into the patches right now: You do know that we have the stack prepared for checksum offloading? From reading the text, it doesn't seem so. Please describe what the differences are to the existing version.


Yes, I read the documentation and also I checked the code. Sorry for my long formulation. In short: We need the following changes to make offloading work with virtio-net and netfront:
- A software computed partial checksum of just the pseudo header in the TCP/UDP header. The hardware unit will take this one to complete computation from there. I could have done this within our netif driver and switch TCP/UDP checksum generation completely off, but this is not very efficient: I would need to parse protocol headers while we setup the transmission. If I really missed that you already compute such a partial checksum for TCP and UDP, then I am sorry.
- A per-packet flag and offsets that indicate where the checksum field is and where to continue computation in hardware. Our netif driver needs to tell this virtio and we should not parse headers in the driver.

As outlook (not part of these patches yet), I plan to extend this work with another pbuf flag (for instance called DATA_VALID) that can tell the stack to skip checksum checks for packets. We need this on a per packet base because we need to be able to handle mixed traffic: (1) checksums might be already checked (but not necessarily) by a physical network card attached to the hypervisor, (2) the checksum field might be incomplete - actually incorrect - because traffic is received from another virtual machine that did offloading. Because this is an in-memory communication, no corruption is assumed. We have an equivalent flag for each received packet that is set by the virtual network cards for such cases.

>
> Also, development takes place on the master branch. The STABLE-2_1_x branch only gets cherry-picked patches applied for a fast release.


I already assumed so, but I got confused by your contributing guide. The following two points would have helped me as newcomer: (1) Tell that you need a Savannah account for submitting. For me, Savannah looked in-active because the patch submission button was crossed out. (2) Shortly describe the branch layout for contributors as you just did to me. I also picked STABLE-2_1_x because it is the one I could test the work. We have currently a number of other dependencies in Unikraft to lwip that do not apply with the master branch that we obviously need to fix.
I attached you a rebased version that applies to master.

(file #51977, file #51978, file #51979)

Simon Kuenzer <skuenzer>
Mon 27 Sep 2021 06:06:27 AM UTC, comment #1: 


> These patches implement the capability to offload checksum computation


Without having the time to look into the patches right now: You do know that we have the stack prepared for checksum offloading? From reading the text, it doesn't seem so. Please describe what the differences are to the existing version.

Also, development takes place on the master branch. The STABLE-2_1_x branch only gets cherry-picked patches applied for a fast release.

Simon Goldschmidt <goldsimon>
Group administrator
Sat 25 Sep 2021 10:42:04 PM UTC, original submission:  

These patches implement the capability to offload checksum computation to modern network cards. For each pbuf the stack can point to a checksum field that should be computed and filled by the device. Expensive checksumming of payload bytes can be offloaded which means that this feature is intended for checksums of transport protocols only (like TCP, UDP). This is inline with todays standards, like you find with virtio-net.

As initial implementation, I have it done for TCP, specifically for segments and control packets that are sent. For this purpose, lwip will compute a partial checksum of only the pseudo header. Starting with this partial checksum, the hardware will continue computing the final 16bit ones’ complement over the payload. In principle, UDP can be added, too, but I think it makes sense to have discussed the basic infrastructure first… I am open for comments. Maybe you have a better name suggestion for the checksum functions introduced with patch 2.

I wasn't actually sure to which branch I should send you this contribution. I selected the STABLE-2_1_x branch because we tested the feature with this branch with our Unikraft project. We run lwIP directly on top of virtual network cards, like netfront (Xen) and virtio-net (KVM/QEMU).

As outlook: The introduced pbuf fields are even interesting for us for the receiving side: In virtual environments you can receive traffic from physical networks and from other virtual machines running on the same host. In the latter case, the checksum computation might be still incomplete and we should not drop the packet. The virtio standard (but also Xen netfront does this) introduces another flag that indicates if the checksum for a packet was already checked. This could been done by a physical network card that originally received the packet. In such a scenario you would also be able to skip checking in lwIP.

Simon Kuenzer <skuenzer>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #51965:  0002-inet_chksum-Partial-checksums.patch added by skuenzer (6KiB - application/octet-stream)
file #51966:  0003-tcp_out-Partial-checksumming.patch added by skuenzer (7KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by goldsimon (Posted a comment)
  • -email is unavailable- added by skuenzer (Submitted the item)
  • -email is unavailable- added by skuenzer
  • -email is unavailable- added by skuenzer
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 18 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-11-12 skuenzer Attached File- Added master-v2-0009-tcp_in-Skip-checksum-validation-when-PBUF_FLAG_DA.patch, #52252
        Attached File- Added master-v2-0010-udp-Skip-checksum-validation-when-PBUF_FLAG_DATA_.patch, #52253
    2021-11-12 skuenzer Attached File- Added master-v2-0005-tcp_in-Check-partial-checksummed-packets-pseudo-h.patch, #52248
        Attached File- Added master-v2-0006-udp-Compute-partial-checksum-pseudo-header.patch, #52249
        Attached File- Added master-v2-0007-udp-Check-partial-checksummed-packets-pseudo-head.patch, #52250
        Attached File- Added master-v2-0008-pbuf-Introduce-PBUF_FLAG_DATA_VALID.patch, #52251
    2021-11-12 skuenzer Attached File- Added master-v2-0001-netif-Avoid-compiling-mistakes-IF__NETIF_CHECKSUM.patch, #52244
        Attached File- Added master-v2-0002-inet_chksum-TCP-UDP-pseudo-header-only-checksums.patch, #52245
        Attached File- Added master-v2-0003-pbuf-Introduce-PBUF_FLAG_CSUM_PARTIAL.patch, #52246
        Attached File- Added master-v2-0004-tcp_out-Compute-partial-checksum-pseudo-header.patch, #52247
    2021-09-27 skuenzer Attached File- Added master-0001-pbuf-Introduce-PBUF_FLAG_CSUM_PARTIAL.patch, #51977
        Attached File- Added master-0002-inet_chksum-Partial-checksums.patch, #51978
        Attached File- Added master-0003-tcp_out-Partial-checksumming.patch, #51979
    2021-09-25 skuenzer Attached File- Added 0001-pbuf-Introduce-PBUF_FLAG_CSUM_PARTIAL.patch, #51964
        Attached File- Added 0002-inet_chksum-Partial-checksums.patch, #51965
        Attached File- Added 0003-tcp_out-Partial-checksumming.patch, #51966
        Carbon-Copy- Added -email is unavailable-
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code