tasklwIP - A Lightweight TCP/IP stack - Tasks: task #6863, Create option to use different...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #6863: Create option to use different size pools (using memp) instead of heap

Submitter:  Simon Goldschmidt <goldsimon>
Submitted:  Thu 10 May 2007 11:11:54 AM UTC
   
 
Category:  None Should Start On:  Thu 10 May 2007 12:00:00 AM UTC
Should be Finished on:  Thu 10 May 2007 12:00:00 AM UTC Priority:  3 - Low
Status:  Done Privacy:  Public
Assigned to:  goldsimon Percent Complete:  90%
Open/Closed:  Closed Planned Release:  None
Effort:  0.00

Jump to the original submission

Thu 21 Jun 2007 08:18:42 PM UTC, comment #11: 

I considere this as done, since the only remaining thing would be to get a configurable amount of pools that is compatible with our way of calculating the size of memp_memory (in memp.c).

Simon Goldschmidt <goldsimon>
Group administrator
Thu 21 Jun 2007 08:13:07 PM UTC, comment #10: 

As a start, I've checked in a version that has a fixed number of 4 different pools. If you want less, simply set element size and -count of an upper pool to 0.

The pool allocator is disabled by default!

Simon Goldschmidt <goldsimon>
Group administrator
Wed 13 Jun 2007 07:23:18 PM UTC, comment #9: 

In a first time, this is not a bad idea. I would like to have time to do a statistic analyze of number and size of mem_malloc to tune these values. Don't you have any idea of what is need (else, it could be difficult to tune these lwIP options) ?

Frédéric Bernon <fbernon>
Group Member
Wed 13 Jun 2007 07:15:29 PM UTC, comment #8: 

The only idea that I have regarding the preprocessor magic is

a) autoconf :-(
b) making the user say how much memory is needed for the pools, but that's hard to calculate given it is dependent on alignment and the overflow checks...

We could still say we always have 5 pools and their size must always be defined (much like in comment #1 but without al the '#if MEM_POOL_COUNT > X').

Any further thoughts about that?

Simon Goldschmidt <goldsimon>
Group administrator
Wed 13 Jun 2007 12:54:16 PM UTC, comment #7: 

You could remove MEM_POOL_COUNT, and check in mem_init that sizeof(memSizes)==sizeof(memCounts).

But it doesn't change your problem for memp_memory.

Frédéric Bernon <fbernon>
Group Member
Wed 13 Jun 2007 10:50:29 AM UTC, comment #6: 

I'm currently using a heap-to-pools implementation with the following defines:

#define MEM_POOL_COUNT      7
#define MEM_POOL_ELEMSIZES  {32, 64, 128, 256, 512, 1024, 1548}
#define MEM_POOL_ELEMCOUNTS {500, 100, 100, 100, 100, 100 , 100}

But in contrast to memp.c, I create the pool memory at runtime. I have 2 static arrays:

static const u32_t memSizes[MEM_POOL_COUNT] = MEM_POOL_ELEMSIZES;
static const u32_t memCounts[MEM_POOL_COUNT] = MEM_POOL_ELEMCOUNTS;

which I can loop over to create the pools using malloc(). In memp.c, however, I would have to give the pool sizes in a preprocessor-compatible way to create the static memp_memory.

And I didn't find a way to do that, yet. The only way is what Frédéric suggested: simply don't configure the pool count...
But I'd rather not have that limitation!

Simon Goldschmidt <goldsimon>
Group administrator
Tue 12 Jun 2007 03:34:56 PM UTC, comment #5: 

The problems you describe in comment #1 are indeed ugly!  I'd rather see something more elegant if possible, although don't have any preprocessor magic off the top of my head.  Would doing something like this work better:

#define MEM_POOL_COUNT 3
#define MEM_POOL_SIZES { 64,500,1500 }

struct mem_pool_size {
 int sizes[MEM_POOL_COUNT];
}

struct mem_pool_sizes pool_size = { MEM_POOL_SIZES };

for (i = 0; i < MEM_POOL_COUNT; i++) {
  printf("Pool %d is size %d\n", i, pool_size.sizes[i]);
}

Note, the above is just pseudo-code to illustrate an idea.  You might not need the struct mem_pool_size; i.e. just use an array, and statically initialise it using the #defined value.

Kieran Mansley <kieranm>
Group Member
Tue 12 Jun 2007 06:59:14 AM UTC, comment #4: 

Not yet a solution to use the preprocessor, but some remarks:

- in mem_malloc, the "if (poolnr==-1)" should be "if (poolnr>=(MEMP_MEM_POOL + MEM_POOL_COUNT))" and the initialization of the variable (-1) should be removed...

- I think your "struct memp_header" should have to be aligned, else, I'm not sure that the "void*" return per mem_malloc (sizeof("element") could be non-aligned).

- Perhaps so assert to check alignement in mem_malloc (like in mem_free)?

- I confirm that a "stats" feature could really help to determinate size and number of pool...

But it's a good idea to avoid fragmentation...

Frédéric Bernon <fbernon>
Group Member
Mon 11 Jun 2007 05:40:31 PM UTC, comment #3: 


> I will be interested to know if you have done a little evaluation about size passed to mem_malloc (with a stats feature)?


I did, but that's a long time ago and unfortunately I don't have that code any more. But it's an interesting suggestion to include in the code!

> With SNMP, isn't it possible to get something larger that 1500?


I'm using that code for nearly a year now (not exactly this code but the effect is the same) with 7 different sized pools. The biggest pool has elements of 1548 bytes and I never ran into an assert.
Unfortunately I can't tell you what is the biggest size SNMP allocates, but 1548 certainly seems to be enough...

> And in such case, how is it handle (I not yet read your code)?


mem_malloc returns NULL (and asserts) in that case.

Simon Goldschmidt <goldsimon>
Group administrator
Mon 11 Jun 2007 05:29:11 PM UTC, comment #2: 

With another stack I use, there is three pools with different size, and its pretty static (no possibility to add or remove other poll, they are named small, medium and large).

I will be interested to know if you have done a little evaluation about size passed to mem_malloc (with a stats feature)? With SNMP, isn't it possible to get something larger that 1500? And in such case, how is it handle (I not yet read your code)?

I have perhaps an idea for using the preprocessor. I will study that and tell you if I found something...


Frédéric Bernon <fbernon>
Group Member
Mon 11 Jun 2007 04:21:44 PM UTC, comment #1: 

I've finally managed to create a mem.c implementation that uses memp pools. The challenge I faced (am still facing in fact) is the following:

Initially, I wanted to have configuration options like this:

#define MEM_USE_POOLS              1
#define MEM_POOL_COUNT             3
#define MEM_POOL_SIZE              64,500,1500
#define MEM_POOL_NUM               100,20,10

But does anyone know how I then can calculate the size of each pool using the preprocessor??? To create the memp_memory, I would have to calculate 63*100, 500*20, 1500*10 using the preprocessor. Since I really don't know how to do that, I changed the defines to this:

#define MEM_USE_POOLS              1
#define MEM_POOL_COUNT             3
#define MEM_POOL_SIZE_1            64
#define MEM_POOL_NUM_1             100
#define MEM_POOL_SIZE_2            500
#define MEM_POOL_NUM_2             20
#define MEM_POOL_SIZE_3            1500
#define MEM_POOL_NUM_3             10

By doing it that way, I get many lines like '#if MEM_POOL_COUNT > X' in the memp code.

That's very bad code but it's still working and way faster than the normal heap.

I'll check it in (as default off -> using the normal heap) these days if noone comes up with a better solution!

(file #13016)

Simon Goldschmidt <goldsimon>
Group administrator
Thu 10 May 2007 11:11:54 AM UTC, original submission:  

As I currently am using it that way and I think preventing fragmentation is a good thing, I want to introduce a configurable amount of memp-pools dedicated to be used by a different mem_malloc() implementation that uses the next biggest pool to satisfy an allocation requirement.

All as a compile time option, of course (since it requires more memroy than a heap to achieve the zero-fragmentation goal).

-> Later, only as a reminder!

Simon Goldschmidt <goldsimon>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #13016:  mem_pools.patch added by goldsimon (8KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by kieranm (Posted a comment)
  • -email is unavailable- added by fbernon (Posted a comment)
  • -email is unavailable- added by goldsimon (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-06-21 goldsimon StatusIn Progress Done
        Percent Complete10% 90%
        Open/ClosedOpen Closed
    2007-06-11 goldsimon Priority1 - Later 3 - Low
        StatusNone In Progress
        Percent Complete0% 10%
        Assigned toNone goldsimon
    2007-06-11 goldsimon Attached File- Added mem_pools.patch, #13016

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code