patchmldonkey, a multi-networks file-sharing client - Patches: patch #4510, IP: Save Memory by using chars...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #4510: IP: Save Memory by using chars instead of int

Submitter:  bogeyman <bogeyman>
Submitted:  Tue 11 Oct 2005 07:49:15 PM UTC
   
 
Category:  None Priority:  3 - Low
Status:  Wont Do Assigned to:  None
Open/Closed:  Closed

Jump to the original submission

Thu 27 Oct 2005 10:40:46 AM UTC, comment #12: 

Compared vanilla 2.6.6 and 2.6.6 with 2_ints patch showed no difference in RAM usage.

spiralvoice <spiralvoice>
Group administrator
Sun 16 Oct 2005 08:13:39 PM UTC, comment #11: 

If its not broken, don“t fix it. If this patch does not save
much memory I would prefer not to use it to avoid bugs.

spiralvoice <spiralvoice>
Group administrator
Thu 13 Oct 2005 09:17:52 PM UTC, comment #10: 

Hmm, my saved memory was achieved by unseting ip_blocking_descriptions without reloading. With my patch same amount of memory is used. i'll try pangos patch tonight.

bogeyman <bogeyman>
Group Member
Thu 13 Oct 2005 03:28:33 PM UTC, comment #9: 

I wanted to compare with what ip.ml would really be using.  (The code is cut & paste to test the lib so I'm sure it was far from efficient.)

Anyway, maybe you can improve the size library if you see any obvious problems.  Since debugging ocaml apps for mem use is rather difficult, having something reliable and built in to mldonkey to reveal where all mem is going might be helpful.

Anonymous
Thu 13 Oct 2005 08:43:15 AM UTC, comment #8: 

Several things;

  • The type of t1, t2, t3 are records (of one field each, but records nonetheless). That's not exactly the same thing, and in this case it even matters:

# size_w (0, 1, 2, 3) ;;
- : int = 5
# type t = { a: int*int*int*int } ;;
# size_w { a = (0, 1, 2, 3) } ;;
- : int = 7

(I just saw that's you're testing both later on... Ok)

  • the use of @ to append elements at the end of lists silently makes your algorithm complexity raise to O(n^2), using around 500000 steps to add your 1000 elements. Try always adding elements at the head of lists, using List.rev as a last step if order matters (n*2 steps is still better than n*n/2)


  • beware of aliasing. It's non-trivial to guess when it happens.

# let lt1 = ref [] and lt2 = ref [] ;;
# for i = 1 to 1000 do
    lt1 := Int32.of_int 3 :: !lt1;
    lt2 := 3l :: !lt2;
  done ;;
# size_w !lt1 ;;
- : int = 4000
# size_w !lt2 ;;
- : int = 3001
(In that case, it's the use of the function Int32.of_int that forces each element to be rebuilt, preventing aliasing.)

  • my results:

# open Size ;;
# let nlist f n =
    let rec nlist_aux acc n =
      if n = 0 then List.rev acc
      else nlist_aux (f () :: acc) (n-1) in
    nlist_aux [] n ;;
val nlist : (unit -> 'a) -> int -> 'a list = <fun>
# ( for reference )
# size_w (nlist (fun () -> ()) 1000) ;;
- : int = 3000
# size_w (nlist (fun () -> Random.int 256, Random.int 256, Random.int 256, Random.int 256) 1000) ;;
- : int = 8000
# size_w (nlist (fun () -> Random.int 65536, Random.int 65536) 1000) ;;
- : int = 6000
# size_w (nlist (fun () -> Random.int32 Int32.max_int) 1000) ;;
- : int = 4000
# size_w (nlist (fun () -> char_of_int (Random.int 256), char_of_int (Random.int 256), char_of_int (Random.int 256), char_of_int (Random.int 256)) 1000) ;;
- : int = 8000

I'm still skeptic about the results of that library...

pango <mlpango>
Group Member
Thu 13 Oct 2005 04:04:21 AM UTC, comment #7: 

Interesting. I guess I don't understand the lt2 output in this example, which is what first led me to think the tuple list lt1 was slightly better than the char quad list lt3. 

http://pastebin.ca/25360

Anonymous
Thu 13 Oct 2005 12:42:45 AM UTC, comment #6: 

Interesting lib, I was looking for something like that...
Some cases give dubious results however, so how much can others be trusted...

# let t = 3 ;;
val t : int = 3
# size_w t ;;
- : int = 0
(ints uses no memory ? that's a good news)

I don't know what you tested, but int tuples certainly don't win:
# size_w [(0,1,2,3);(4,5,6,7)] ;;
- : int = 16
# size_w [('A','B','C','D');('E','F','G','H')] ;;
- : int = 16
# size_w [Int32.of_int 0;Int32.of_int 1] ;;
- : int = 8
Are you sure none of your values were aliased ?

pango <mlpango>
Group Member
Wed 12 Oct 2005 09:51:32 PM UTC, comment #5: 

Maybe this module can help determine the best approach: (or even added to all the mem_stats in mldonkey for some interesting memory information)

http://www.lri.fr/~filliatr/ftp/ocaml/misc/size.mli.html
Size: computes the memory size of an ocaml value.

(view the "misc" directory in the above url for all files)

(* Sizes of ocaml values (in their memory representation).
   Sizes are given in words ([size_w]), bytes ([size_b]) or kilobytes
   ([size_kb]), in a system-independent way. *)

I tried a quick test comparing size_b result of a list of char quads, int tuples, and int32s. int tuple seems to win (if this type of test is even valid?).


Anonymous
Wed 12 Oct 2005 08:04:29 PM UTC, comment #4: 

Ooops! Forgot to add Int32.t to the list of tested types.
Memory usage is the same as int * int:

# ls -lrS core.*
-rw-------  1 root root  4751360 2005-10-12 21:27 core.reference
-rw-------  1 root root 17465344 2005-10-12 21:29 core.2ints
-rw-------  1 root root 17469440 2005-10-12 22:02 core.Int32
-rw-------  1 root root 26370048 2005-10-12 21:28 core.4chars
-rw-------  1 root root 26374144 2005-10-12 21:28 core.4ints

pango <mlpango>
Group Member
Wed 12 Oct 2005 07:52:28 PM UTC, comment #3: 

Chars may be 8 bit values, it doesn't mean that they use just that amount of storage : memory is allocated in "words" (4 bytes on 32 bits architectures, 8 bytes on 64 bits architectures). The question is then, does the ocaml compiler "pack" char tuples.

Int32s are indeed boxed, which means that they also use more than 32 bits of memory. In fact, all but enumerated types are boxed, and enumerated types are stored on 31 bits, so a 32 bits value cannot use 4 bytes of memory).
A solution, as suggested on #ocaml on Freenode, may be to use int * int, storing 16 bits in each.

I wrote the program below to evaluate the different solutions. It creates 1000000 values of the tested type, using an array, then kill itselfs with SIGABRT. You can then check the size of the core dump (be sure ulimit doesn't prevent core dumps from being generated):

( compile with ocamlopt -o test unix.cmxa test.ml )
let number = 1000000
let new_ip (a:int) (b:int) (c:int) (d:int) =
( 0 ( reference ) )
( a, b, c, d ( 4 ints ) )
( char_of_int a, char_of_int b, char_of_int c, char_of_int d ( 4 chars ) )
  ((a lsl 8) lor b, (c lsl 8) lor d) ( 2 ints )
let a = Array.init number (fun _ -> new_ip 1 2 3 4)
let _ =
  Unix.kill (Unix.getpid ()) 6

Results:
-rw-------  1 root root 17465344 2005-10-12 21:29 core.2ints
-rw-------  1 root root 26370048 2005-10-12 21:28 core.4chars
-rw-------  1 root root 26374144 2005-10-12 21:28 core.4ints
-rw-------  1 root root  4751360 2005-10-12 21:27 core.reference

I wrote a patch to save IPv4 numbers as int * int, but it's still untested.

By the way, on 64 bits architectures, a simple int is enough to store an IPv4. And by the time we all switched to 64 bits boxen, we'll probably have to store IPv6 numbers ;)

pango <mlpango>
Group Member
Tue 11 Oct 2005 09:14:55 PM UTC, comment #2: 

char's are supposed to be 8 bits, while int32's are atleast 32 and boxed for GC.

http://caml.inria.fr/pub/docs/manual-ocaml/manual010.html
http://caml.inria.fr/pub/docs/manual-ocaml/libref/Int32.html

Maybe char's are the way to go even if ugly?

It would be nice to have some statistics to see if the memory savings introduced by using chars truely outweighs the ugly (read: harder to maintain and read, more likely to introduce conversion bugs, ...) code. Needs to be well tested.

Anonymous
Tue 11 Oct 2005 08:05:25 PM UTC, comment #1: 

This patch makes the code very ugly.

It has been tried as int32s, but AFAIK didn't save anything noticable.

http://download.berlios.de/pub/mldonkey/pango/store_ips_as_int32s.patch


Anonymous
Tue 11 Oct 2005 07:49:15 PM UTC, original submission:  

Please test. Maybe switch to Int32? on http://caml.inria.fr/pub/docs/manual-ocaml/libref/Pervasives.html they say Ints are 31 bits wide, so this wont work. could not find anything definit about how wide chars are. works for me, although i hoped to gain more.

bogeyman <bogeyman>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #7983:  store_ips_as_2_ints.patch added by mlpango (5KiB - text/x-patch)
file #7981:  ip_four_char.patch added by bogeyman (4KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

 

Follow 4 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2005-11-20 spiralvoice StatusNone Wont Do
    Open/ClosedOpen Closed
2005-10-12 mlpango Attached File- Added store_ips_as_2_ints.patch, #5318
2005-10-11 bogeyman Attached File- Added ip_four_char.patch, #5316

Back to the top

Powered by Savane 3.15-17aa.
Corresponding source code