tasklwIP - A Lightweight TCP/IP stack - Tasks: task #6849, Test how checksum on copy could be...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #6849: Test how checksum on copy could be integrated into the stack

Submitted by:  Simon Goldschmidt <goldsimon>
Submitted on:  Fri 04 May 2007 06:40:48 AM UTC  
 
Category:  None Should Start On:  Fri 04 May 2007 12:00:00 AM UTC
Should be Finished on:  Fri 04 May 2007 12:00:00 AM UTC Priority:  5 - Normal
Status:  Done Privacy:  Public
Percent Complete:  100% Assigned to:  Simon Goldschmidt <goldsimon>
Open/Closed:  Closed Planned Release:  1.4.0
Effort:  0.00

( Jump to the original submission)

Sun 16 May 2010 01:17:15 PM UTC, comment #35: 

> Oh, and we might want to use the new checksum-on-copy from the
> socket API when sending UDP data...


Done.

> What's now missing is only a tcp_write_pbuf() that also takes a
> checksum.


We have task #10273 for that.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Fri 30 Apr 2010 08:41:02 PM UTC, comment #34: 

Oh, and we might want to use the new checksum-on-copy from the socket API when sending UDP data...

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Fri 30 Apr 2010 08:40:08 PM UTC, comment #33: 

Added udp_send(_to/_if) functions that take a precalculated checksum, added pbuf_fill_chksum() to copy data into a pbuf and at the same time calculating the checksum for that data.

What's now missing is only a tcp_write_pbuf() that also takes a checksum.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Fri 26 Mar 2010 09:20:00 AM UTC, comment #32: 

> I think on RX, should we assume that there is full checksum offloading?


I always thought so. Either, the MAC checks the checksum and drops packets with wrong checksums (not even generating an interrupt for them), or (in a partly-handwritten environment as I plan it), the custom DMA-engine creates a checksum when copying, which can then easily be checked against the checksum in the header.

Being like that, checksum-on-copy would only apply to TX, not to RX.

I hoped the current checksum-on-copy implementation I started in tcp_write would work on all platforms, since it uses a memcpy-like function that creates the checksum while copying (as the name says :)

What's yet missing is a way to pass in precalculated checksums (either calculated at compile time or calculated in the application while creating packet data). However, I think that would also work as an argument to a tcp_write-/ udp_send-like function, and we wouldn't need to save the checksum in the pbuf for that?

Can you think of any platform here were this wouldn't work? How I test it is a) using a (slow) memcpy-checksum function and b) creating checksums at compile-time using the httpd-makefsdata tool letting httpd pass these checksums to tcp_write (which I already did and it works).

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Thu 25 Mar 2010 09:20:22 PM UTC, comment #31: 

I think on RX, should we assume that there is full checksum offloading? If the MAC is going to do this, I would expect it to do everything (as it does on the PC, or seems to).  If the Ethernet driver is going to receive the packet and then do the hardware checksum, it will have to do some parsing to decide what to do and fill in the checksum in the correct places.  If it's to be done any higher up than that, then inet_chksum can use the hardware to checksum a pbuf (or pbuf chain) just as easily.  I guess the question is, where is the copy we're checksumming on copy?  The callbacks provide a pointer to the payload and I used to hardware copy to concatenate the packets or I've never had to copy incoming data - just look at it and do something locally with it.  It's been outgoing where I could do the checksum as the data was being streamed.

It is a tough thing.  The data is coming through the MAC and the driver first.  On TX you originate with the data so it's easier to do it first or do it as the data is formed into packets to send.

Do you implement this so it works for some platforms or methods and not others?  Or insist it work for all cases or none at all?  And how do you test?  Just fill in the checksum using a software call to simulate having the checksum already?

Bill Auerbach <billauerbach>
Thu 25 Mar 2010 09:05:14 PM UTC, comment #30: 

Well, OK, but how is that going to work for RX? As far as I can remember, pbuf_header works on all pbuf types when hiding headers (not when unhiding them, i.e. expanding to the front, which is what you took advantage of).

Anyway, that solution doesn't work for PBUF_RAM pbufs passed to udp_send and doesn't work for TCP, too (where PBUF_RAM pbufs are used when copying data), so it doesn't work for checksum-on-copy, only for "precalculated-checksum-no-copy".

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Thu 25 Mar 2010 06:47:38 PM UTC, comment #29: 

I meant to reply but it got away from me.

I see the problem you describe.  If the checksum is set for the pbuf->payload, don't call pbuf_header but create a new pbuf for the header and chain to the checksummed pbuf.  It's more work, but the checksum savings far offsets handling a chain.  Mine works this way - the pbuf head is the IP and UDP header and I chain the UDP (pre-checksummed) pbuf to it.  It happened by itself because the pbuf_header failed and the current code allocates and appends when the pbuf_header fails.

Bill Auerbach <billauerbach>
Thu 25 Mar 2010 06:36:43 PM UTC, comment #28: 

No comments about my last question? If not, I'll close patch #6253 as "won't do", since I don't see how it would work (given the description in my last comment).

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Tue 16 Mar 2010 03:11:31 PM UTC, comment #27: 

I meant something different: I didn't need an u32_t but only an u16_t for the checksum (over the complete payload) and a bit in p->flags. So far it's not much different from what you did.

However, the problem arises later. Let me make an example of (pre-checksummed) udp_send:
- pbuf p passed to udp_send, p->payload == UDP-data, checksum is valid for the full payload
- pbuf_header is called, p->payload == UDP-header
-> at this point, the checksum is not valid for the full pbuf payload any more!
- inet_chksum_pseudo() is called -> it may not use p->chcksum as checksu for p->payload..(p->payload+p->len) as it is not valid any more
-> what do you suggest to do at this point?

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Tue 16 Mar 2010 12:54:36 PM UTC, comment #26: 

I used a u32_t in the pbuf so I could use a special value to say "Need checksum calcualted".  If not that value, I skipped the inet_chksum call and simply added in this field.  I made the assumption in my method that a pbuf had a checksum only for it's payload, but chained pbufs each with a checksum was allowed.

Like:

  /* iterate through all pbuf in chain */
  for(q = p; q != NULL; q = q->next) {
    LWIP_DEBUGF(INET_DEBUG, ("inet_chksum_pseudo(): checksumming pbuf %p (has next %p) \n",
      (void )q, (void )q->next));
    if(q->checksum != -1)
      acc += q->checksum;
    else
      acc += LWIP_CHKSUM(q->payload, q->len);
....

q->checksum is a u32_t which is -1 if I need the checksum calculated.

Perhaps it's that what I did is lower level than what you want to do?

Bill Auerbach <billauerbach>
Tue 16 Mar 2010 05:58:30 AM UTC, comment #25: 

The problem I saw so far (and which hasn't been solve in patch #6253) is that it's not clear to me how one 'u16_t chksum' could be enough: at the point of allocating, the checksum would be over the data (payload + len), but what part of the pbuf would be covered by the checksum after changing the payload pointer using pbuf_header()? We would then either need another u16_t 'chksum_offset' (which would have to be adapted in pbuf_header), or the inet_checksum_* functions would have to know the offset by knowing the protocol type and its length (which is hard for TCP options).

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Mon 15 Mar 2010 07:20:10 PM UTC, comment #24: 

My opinion is that if the value stored is the checksum for just the data in the pbuf then it's fine to keep it in the pbuf.  If the value stored is the checksum for the whole packet, it should be kept in whatever structure keeps the metadata for the whole packet (e.g. tcp_seg).  If it's computed for each pbuf separately that makes some things easier (e.g. resegmenting tcp_writes) with only a small cost of summing the per-buffer values together when we need to work out the whole packet.  I do want to make sure that we don't make the pbuf structure any larger though, and Simon's suggestion should do that.

Kieran Mansley <kieranm>
Project Member
Mon 15 Mar 2010 06:16:44 PM UTC, comment #23: 

>I disagree: logically a checksum should belong to the segment not the buffer (of which there may be many).


Don't these many segments have pbufs or pbuf chains?  You'll find there's less code and computation with the checksum in the pbuf.  Adding the pbuf's checksums falls out as does the even/odd aspect.  When it's in the segment, is has to be updated with every segment change.  If segment manipulation is with chaining pbufs, the checksum moves with the pbuf and there's no overhead to segment handling.  But your view and knowledge are better than mine - I only speak from what I did.

I guess it's in Simon's court and when he implements it and it will fall out.  I've done it already and with 4 or 5 lines of code for full-sized MSS packets. I think it would work for segments that are chains of pbufs too.  I didn't do what I didn't need to do (or couldn't test).  Ultimately I had to move to UDP because TCP still wasn't fast enough.  There was little change to this move with the checksum in the pbuf.

Bill Auerbach <billauerbach>
Mon 15 Mar 2010 05:48:43 PM UTC, comment #22: 

In my case, I had the ability for the hardware to send me the payload part of the packet with its checksum.  So the stack does the IP header and its checksum, and the payload was passed in with the checksum.  When the segment was created with the checksum, I filled it in (in the pbuf but just as easily could have been in the tcp_seg).  I was sending full MSS-sized data so multiple segments didn't apply (which is why it was a trivial change for me).  For UDP, I put the checksum in the pbuf and the code that checksums pbufs (and chains) just added in the checksum supplied (if supplied) or calculated it otherwise.  Note the even/odd detail fell out without any added code.  So for both UDP and TCP to work the same was the reason why I mentioned the checksum belongs in the pbuf.  There are no tcp_segs with UDP.  So maybe it doesn't belong there from a layer or design standpoint, but it looked like a better fit for TCP and UDP to work alike.  (No Kieran, I'm not trying to change your mind :-) ).

>...is it hard to change your current implementation to call udp_send_chksum / tcp_write_chksum with the checksum as an argument?


No, this is exactly what I did.  So I suppose checksum on copy is the same as what I did.  Checksum offloading is where you can throw the #define to complete turn it off.

TX and RX are separate, yes, but at the driver level there is some overlap.

I have no problem with your (good by the way) idea of adding the checksum as another type in pbuf_alloc to only the pbufs that need them.

Bill Auerbach <billauerbach>
Mon 15 Mar 2010 04:48:28 PM UTC, comment #21: 

I think we are mixing 2 things here:
a) implementation-driven checksum-in-pbuf and
b) optimizing the way the stack calculates checksums by default

Looking only at the TX side (which this task is about only), is it hard to change your current implementation to call udp_send_chksum / tcp_write_chksum with the checksum as an argument?

As to the RX side, in patch #6253, I was thinking about storing the checksum directly after the struct pbuf (which doesn't work for PBUF_REF/ROM, but we currently don't support these for RX, anyway). E.g you would have to supply a special pbuf_layer to pbuf_alloc, which reserves space for the checksum at the beginning of a packet. This way, only the first pbuf in a chain has to provide additional space for a checksum instead of all RX and TX pbufs in the complete system.

But again, I think that TX and RX are really 2 separate issues here, and that checksum-in-pbuf only applies to RX. If we agree about that, the discussion about that should be continued in patch #6253.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Mon 15 Mar 2010 04:25:11 PM UTC, comment #20: 

I hope it would only grow with this option enabled.

Bill Auerbach <billauerbach>
Mon 15 Mar 2010 04:22:55 PM UTC, comment #19: 

Oh, and in patch #6253, the main point was that struct pbuf shouldn't grow larger if we can avoid it: there are targets limited in RAM that benefit from the per-segment storage, even when mixing PBUF_RAM with PBUF_REF (tcp copy/nocopy).

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Mon 15 Mar 2010 04:18:54 PM UTC, comment #18: 

I didn't argue at any time that it wouldn't work with the checksum in the pbuf, but I also think that the checksum logically belongs to the segment.

Storing it at the buffer is only necessary when you want to pass the two around as one. This is what you want to do for rx, I think, but personally, I prefer the other way: If you already have the checksum at driver level, it's faster to quickly check whether the checksum is correct and disable checksum checking in the stack then passing the checksum around and later detecting that it is invalid.

We discussed that in patch #6253, too.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Mon 15 Mar 2010 12:47:58 PM UTC, comment #17: 

I disagree: logically a checksum should belong to the segment not the buffer (of which there may be many).

Kieran Mansley <kieranm>
Project Member
Mon 15 Mar 2010 12:45:31 PM UTC, comment #16: 

>- tcp_seg has 2 more members (u16_t chksum and u8_t chksum_swapped),


IMO this belongs in the pbuf.  A tcp_seg has a pbuf, and it must be in the pbuf to do checksum on rx.

If you ensure that one segment has one pbuf, you don't need the swapped field.  This is not a bad compromise because the gain with hw checksumming is in not touching the data, not in the combining of segments.  If the app is sending a lot of small pieces of data, I suspect transmit performance is not a big consideration.

For UDP it also works when defined in the pbuf. I've already implemented it and am using it this way for transmit.

Bill Auerbach <billauerbach>
Sun 14 Mar 2010 12:30:13 PM UTC, comment #15: 

When implementing udp_send_chksum, to prevent duplicating code, we'd have to separate calculating the checksum of (pbuf-)data from calculating the header checksum. Then, udp_send would first calculate the checksum and then go the same path as udp_send_checksum.

We might then want to create an inet_chksum_pseudo that operate on a void-pointer instead of a pbuf, since headers area always in the first pbuf (old inet_chksum_pseudo is still used for RX).

tcp_write_chksum is slightly harder to implement as it we cannot split the data into segments as we want but have to create new segments based on the given data length.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Sun 14 Mar 2010 12:08:59 PM UTC, comment #14: 

I finally found the time to work on this. For now, I've added the code as an option (LWIP_CHECKSUM_ON_COPY):
- tcp_seg has 2 more members (u16_t chksum and u8_t chksum_swapped),
- checksum is calculated when creating segments in tcp_write (checksum-on-copy when copying data, checksum only for nocopy),
- for nocopy, this still has the advantage that the data might still be in caches

This only covers the TX side and cannot take checksums from the application, yet. I plan to add 2 more funcitons:
- tcp_write_chksum that takes mss-sized data with precalculated checksum
- udp_send_chksum that takes the data checksum (over all pbufs in the chain)

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Mon 11 May 2009 07:30:15 PM UTC, comment #13: 

Yes, it can be done with lwIP not to checksum changing those #defines but it's far from ideal because the driver has to parse the outgoing packet to decide what to checksum.  IMO this is good for hardware which is parsing packets and checksumming on the fly.  Those who want to checksum with hardware but not with packet parsing need an alternative. What I want to do doesn't get the headers checksummed so my idea of wanting to do just the TCP payload isn't complete, but it checksums a majority of the data passing through which I can do in hardware.

Bill Auerbach <billauerbach>
Mon 11 May 2009 04:09:09 PM UTC, comment #12: 

1) is done by telling lwIP to not care for creating/checking a specific checksum (UDP, TCP, ICMP, IP). There are already defines for that.

2) would require a function tcp_enqueue_checksummed. I left that out in a previous post but I would implement it just like udp_enqueue_checksummed.

Talking about efficiency: we can
a) let the application pre-calculate the checksum
b) create it when copying
c) create it in the driver

c) is obviously the best solution if the hardware supports on-the-fly generation while copying the data using DMA. If not, it's not a good solution at all!
Usage of a) and b) depend on the application: If you use TCP_NO_COPY or UDP zero-copy (e.g. sockets), a) is of course better. If you copy anyway, the overhead of an improved copy-and-checksum routine to a memcpy is minimal.

I'll try to create another patch for this to discuss about when I find the time.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Mon 11 May 2009 02:52:59 PM UTC, comment #11: 

I wanted to add a point.  There are 2 checksum options to consider:

1) Offloading that the driver or stack does at some level.  I've seen an lwIP port with driver offloading.

2) Offloading where the application generates data which also can have the checksum calculated per block of data (not exceeding mss).

Although case 2 is less general it's no less important for applications which get data from some other device and the data can be streamed with a checksums included as part of that protocol.  I will be implementing case 2 and envision a tcp_enqueue where I can include the checksum.  If a segment has a 'check_sum' valid field (defaulting to off/0), then a tcp_enqueue function which receives the checksum can set this field to on/1.  Perhaps this field can be the flag too that means do not chain on this segment.  I don't know lwIP well enough to propose anything more than an idea like this.

Case 2 offloading can also be more efficient because if the checksum is done at the time the data is being queued, it's likely to be in cache and quickly accessed.  Driver offloading usually means the data has been replaced in the cache and there is less benefit offloading this late.  Of course this is system dependent.

Bill Auerbach <billauerbach>
Sun 10 May 2009 05:37:18 PM UTC, comment #10: 

Sorry, forgot something:

> My opinion on this has change a little


This and the TX-list means that checksum would not be stored per pbuf as patch #6253 requests but that it is only passed down the stack in function calls or stored in TCP segments.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Sun 10 May 2009 05:36:05 PM UTC, comment #9: 

[Referring a little bit to the discussion in patch #6253: Added csum to struct pbuf.]

My opinion on this has change a little: We have 2 main issues here, TX and RX.

My current plans for TX:
- If the application already knows the checksum, let it pass the checksum to the lwIP API (e.g. add u16_t chksum to a function udp_send_chksummed()) - this would only allow all data to be checksummed, but I think that's OK
- If the application does NOT know the checksum: create it when copying (introduce new functions checksum-memcpy, overridable for ports like current checksum creation)

For UDP/RAW the checksum would only have to be passed down, for TCP, I'd store it in the segment struct and update it when new data is appended to a segment.

The only problem is real zero-copy (UDP sockets or TCP-no-copy).

For RX, I'm still convinced that the netif driver should discard packets with invalid checksums if checksum is already created when copying the packet into RAM. We could help driver developers by providing functions that take an RX-pbuf and the checksum and do a fast check if that checksum is valid.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Thu 07 May 2009 05:47:15 AM UTC, comment #8: 

> We can have our hardware stream data and generate checksums while copying the data.


Implemented in an FPGA, I guess? A shame everyone has to create their own :-(

Anyway, the main issue is not how to generated the checksum but how to store it: For UDP it would have to be passed in with the pbuf, but it's yet unclear if only once for a pbuf chain or once in every pbuf in a chain.

For TCP, the problem lies in pbuf chains, also: you don't always create a full segment with a call to tcp_enqueue, you can also enqueue new data in an existing segment. We would then have to update the checksum in the header... And this alone is a change to the current implementation, which creates the checksum at a later point in tcp_output.

Also, I want to have standard implementations of copy_and_checksum (in inet_chksum.h) which does not require changing the TCP API. (It might be enhanced for no-copy TCP writes, though.)

To sum it up, we have to store a checksum in a pbuf for UDP/RAW and somewhere with a segment for TCP (we could also store it with a pbuf for TCP and then sum it up later), and we mainly need a decision on the first and someone who implements the latter (which moves some code from tcp_output(_segment) to tcp_enqueue).

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Wed 06 May 2009 08:58:14 PM UTC, comment #7: 

I can help out on this task.  We can have our hardware stream data and generate checksums while copying the data.  My idea was to have it drop 0 to pcb->mss bytes into memory with a u16_t word following, which is the checksum of the packet.  If a tcp_enqueue_with_checksum existed which took up to pcb->mss and the checksum for the data being sent, then a packet could be queued inserting the checksum in the header.

Posts here mention UDP - I suspect the same thing has to be done.  A call at some level passing in a precalculated checksum.

Bill

Bill Auerbach <billauerbach>
Tue 25 Mar 2008 09:11:51 PM UTC, comment #6: 

Still at it. Although testing this is a little tricky: the Win32-port didn't improve much but this might be due to the large caches of the PCs used to test... I'll test again on an Altera NIOS-II platform, when I find the time.

Anyway, what's the time schedule for 1.3.1?

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Tue 25 Mar 2008 01:13:55 PM UTC, comment #5: 

Simon - does this still interest you?

I think this would be a good thing to include.  The increase in speed would justify the extra word of memory for each segment.

Kieran Mansley <kieranm>
Project Member
Fri 04 May 2007 11:05:14 AM UTC, comment #4: 

>OOI, some network hardware, nevermind network drivers, have hardware assist for checking checksums. But in both cases that's already handled in lwIP with e.g. disabling CHECKSUM_GEN_TCP and/or CHECKSUM_CHECK_TCP. So if someone wants to write a driver that checks the checksum on receive, I think they have what they need from lwIP to do that.


Currently, I have a FIFO MAC. On receiving a frame, I use a dedicated DMA-engine to transfer it into RAM. This DMA-engine can't check/generate checksums. But since I'm copying the data using memcpy already when enqueueing TCP data, it would speed up sending significantly if I do the checksum on copying instead of reading the data twice when copying and checksummimg is separated.

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Fri 04 May 2007 11:02:23 AM UTC, comment #3: 

>Although it wouldn't be good if every pbuf had to have its size increased for UDP, and every tcp_seg had to have its size increased for the intermediate data.


Note that I would make this a (rarely used) option. So size only would have to be increased if someone wants to use this. In that case, you trade mem-usage against speed, which I think is the right thing to do!

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.
Fri 04 May 2007 10:59:20 AM UTC, comment #2: 

Actually thinking about the TCP case a bit more, I guess a seg could store some extra intermediate state to allow the checksum to be computed on an ongoing basis as things are added - 32-bit accumulator, and whether there was an odd byte yet to be added. tcp_output could then generate the checksum from that.

Although it wouldn't be good if every pbuf had to have its size increased for UDP, and every tcp_seg had to have its size increased for the intermediate data.

Jonathan Larmour <jifl>
Project Member
Fri 04 May 2007 10:23:48 AM UTC, comment #1: 

For UDP pbufs that need to be copied this is sensible, but for TCP it's more difficult - what you enqueue doesn't necessarily correspond to what gets sent - each time you enqueue something you may be adding it to an existing segment, and you don't want to have to recalculate the checksum each time something is queued.

OOI, some network hardware, nevermind network drivers, have hardware assist for checking checksums. But  in both cases that's already handled in lwIP with e.g. disabling CHECKSUM_GEN_TCP and/or CHECKSUM_CHECK_TCP. So if someone wants to write a driver that checks the checksum on receive, I think they have what they need from lwIP to do that.

Jonathan Larmour <jifl>
Project Member
Fri 04 May 2007 06:40:48 AM UTC, original submission:  

Creating the checksum for UDP/TCP pbufs could be done when copying them (e.g. in tcp_enqueue() if copy==1), which could significantly increase send performance.

This would have to be included in pbufs somehow...

And on the receive side, I'm not sure but I think theres not speed gain as long as it can't be included in the network driver (only if the network driver does copy using memcpy(), which is slow anyway...)

Simon Goldschmidt <goldsimon>
Project AdministratorIn charge of this item.

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by billauerbach (Posted a comment)
  • -email is unavailable- added by kieranm (Posted a comment)
  • -email is unavailable- added by jifl (Posted a comment)
  • -email is unavailable- added by goldsimon (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-05-16 goldsimon Open/ClosedOpen => Closed
    2010-05-16 goldsimon StatusIn Progress => Done
        Percent Complete70% => 100%
    2010-04-30 goldsimon Percent Complete40% => 70%
    2010-03-14 goldsimon StatusNone => In Progress
        Percent Complete0% => 40%
    2008-03-25 kieranm Planned ReleaseNone => 1.4.0
    2008-03-25 kieranm Priority1 - Later => 5 - Normal

    Back to the top


    Powered by Savane 3.5