taskmldonkey, a multi-networks file-sharing client - Tasks: task #5339, Reassigned to another tracker...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #5339: Reassigned to another tracker [was: ED2K: Support for file's sizes >4GB]

Submitted by:  Fabiano <fabtar>
Submitted on:  Mon 20 Mar 2006 07:50:42 PM UTC  
Votes:  200  
 
Category:  eDonkey-plugin Priority:  7 - High
Status:  In Progress Assigned to:  None
Open/Closed:  Closed Should Start On:  Mon 20 Mar 2006 12:00:00 AM UTC
Should be Finished on:  Mon 20 Mar 2006 12:00:00 AM UTC

( Jump to the original submission)

Wed 29 Nov 2006 12:08:26 AM UTC, comment #15: 

THIS ITEM WAS REASSIGNED TO PATCH #5599.

Please, do not post any new comments to this item.


spiralvoice <spiralvoice>
Project Administrator
Sat 18 Nov 2006 04:22:02 PM UTC, comment #14: 

Posted 256gb_3.patch which is an optimized version of 256gb_2.patch.

I merged the new modules QueryBloc64, Bloc64 and EmuleCompressedPart64 into the existing modules.

The border between old and new files is not
0xffffffffL (4294967295 bytes) but

eMule 0.47c, opcodes.h
#define OLD_MAX_EMULE_FILE_SIZE 4290048000ui64

Patch has been updated to reflect this using function file_is_largefile.
This patch changes much stuff, please compare it to your original
to be sure that I did not introduce any new bugs.

spiralvoice <spiralvoice>
Project Administrator
Sat 18 Nov 2006 01:27:27 PM UTC, comment #13: 

"so the tags are ok?!"

I do not know if all of them are ok, I test here step-by-step.
Working on Debian Etch I setup Wine and installed eMule.
Then I do the same steps with patched MLDonkey and eMule and
compare the packets created. Wireshark has a fine eDonkey
packet decoder which makes the comparison pretty easy.

Tested so far with positive results:
- getsources
- publish

If you like you can do the same and test all functions
necessary for large file support. If I feel the patch is
complete I will do a local transfer of a 5GB file between
eMule 0.47c and MLDonkey using a local eserver.

spiralvoice <spiralvoice>
Project Administrator
Sat 18 Nov 2006 12:33:54 PM UTC, comment #12: 

ah, now i see, why i couldnt find the problem with the servers. i assumed, mldonkey uses OP_GLOBGETSOURCES (0x9A), which doesnt requires a size, but it instead uses OP_GLOBGETSOURCES2(0x94), which indeed needs a leading 32bit zero before the size for large files. i looked at the wrong place, so the tags are ok?!

not completely, i think. a little offtopic, at the moment, the DonkeyMftp.buf_tag-code can only write 32bit-tags, even if they are 64bit, but the size could be a 64bit-tag, and in emule-source there is already a 64bit-tagtype defined:
#define TAGTYPE_UINT64 0x0B

TripleM <triplem>
Sat 18 Nov 2006 12:16:36 AM UTC, comment #11: 

New version of 256gb_2.patch to replace 256gb.patch
- publish largefiles only on servers supporting them
- print a log message if largefile could not be published on a server

spiralvoice <spiralvoice>
Project Administrator
Fri 17 Nov 2006 10:00:49 PM UTC, comment #10: 

"I removed the patch from donkeyProtoCom.ml completely"

This was wrong, for publishing files this patch is needed.
I can confirm that MLdonkey can publish > 4GB files on EDK servers
correctly which can in turn be found be eMule.
Uploaded correct version of 256gb_2.patch, but this patch is far
from complete, see comment #8

spiralvoice <spiralvoice>
Project Administrator
Fri 17 Nov 2006 01:18:12 PM UTC, comment #9: 

Computation of filesize is wrong in your patch, at least for GETSOURCES.

From eMule 0.47c

opcodes.h
=========
#define OLD_MAX_EMULE_FILE_SIZE 4290048000ui64  // (4294967295/PARTSIZE)*PARTSIZE = ~4GB^

AbstractFile.h
==============
bool IsLargeFile() const { return m_nFileSize > (uint64)OLD_MAX_EMULE_FILE_SIZE; }^

DownloadQueue.cpp, CDownloadQueue::ProcessLocalRequests
==========================================================
if (!cur_file->IsLargeFile()){^
smPacket.WriteUInt32((uint32)(uint64)cur_file->GetFileSize());^
}^
else{^
smPacket.WriteUInt32(0); // indicates that this is a large file and a uint64 follows^
smPacket.WriteUInt64(cur_file->GetFileSize());^
}^

This means for large files the int64_32 filesize is 0 and the >4GB filesize is sent afterwards. Getsources for smaller files is unchanged.
I removed the patch from donkeyProtoCom.ml completely and
added this to successfully receive sources from servers:

--- ././src/networks/donkey/donkeyProtoServer.ml 2006-11-14 19:42:59.000000000 +0100
+++ ././src/networks/donkey/donkeyProtoServer.ml 2006-11-17 14:11:05.000000000 +0100
@@ -830,7 +830,13 @@
       Printf.bprintf oc "QUERY LOCATION OF %s [%Ld]\n" (Md4.to_string t.md4) t.size

     let write buf t =
-      buf_md4 buf t.md4; buf_int64_32 buf t.size
+      buf_md4 buf t.md4;
+      if t.size > old_max_emule_file_size then
+        begin
+   buf_int64_32 buf 0L; buf_int64 buf t.size
+        end
+      else
+        buf_int64_32 buf t.size
   end

 module QueryLocationReply  = struct
--- ././src/networks/donkey/donkeyTypes.ml 2006-11-14 19:42:59.000000000 +0100
+++ ././src/networks/donkey/donkeyTypes.ml 2006-11-17 14:11:50.000000000 +0100
@@ -704,3 +708,5 @@

 let emule_proto () =
   { dummy_emule_proto with emule_version = 0 }
+
+let old_max_emule_file_size = 4290048000L

spiralvoice <spiralvoice>
Project Administrator
Wed 15 Nov 2006 09:25:56 PM UTC, comment #8: 

i used pangos 256gb-patch as a start, and added most of the missing ed2k-proto-stuff (i think, at least)

i run a test drive at the moment.

parts missing or incomplete (as known by me):
- difference in filesizeadvertisement for client<->client (64bit) and client<->server (2*32bit) (no real prob, cause client2client only occures, if share is browsable, and file is large_file)
- kad stuff
- handling of incapable clients/servers (clients should be handled right, but im not sure; large files are announced to incapable servers too, but this shouldnt be a real problem, as only the filesize known to server should be affected)

TripleM <triplem>
Wed 08 Nov 2006 07:52:15 AM UTC, comment #7: 

Copied your code fragments to the official Wiki page for EDK-4GB support:
http://mldonkey.sourceforge.net/EDK4GB#Code_fragments_found_by_TripleM

spiralvoice <spiralvoice>
Project Administrator
Wed 08 Nov 2006 07:28:06 AM UTC, comment #6: 

i browsed the sources of emule 0.47c and found some maybe known proto infos. client does not only advertise "large file capability" to server, but to clients too.

in tag "CT_EMULE_MISCOPTIONS2" (0xfe):
bit 0-3: Kad Version
bit 4: Large Files (includes support for 64bit tags)
bit 5: Ext Multipacket (Hash+Size instead of Hash)
bit 6: Reserved (ModBit)
bit 7: Supports CryptLayer
bit 8: Requests CryptLayer
bit 9: Requires CryptLayer
bit 10-31: Reserved

of course sending and requesting parts has changed.

for parts with beginoffset<0xFFFFFFFF and endoffset<=0xFFFFFFFF the old proto is used:
OP_REQUESTPARTS 0x47 // <hash>hash[3*<u32>begin][3*<u32>end]
OP_SENDINGPART 0x46 // <hash>hash<u32>begin<u32>end<data>
OP_COMPRESSEDPART 0x40 // <hash>hash<u32>begin<u32>len<len>data

for other:
OP_REQUESTPARTS_I64 0xA3 // <hash>hash[3*<u64>begin][3*<u64>end
OP_SENDINGPART_I64 0xA2 // <hash>hash<u64>begin<u64>end<data>
OP_COMPRESSEDPART_I64 0xA1 // <hash>hash<u64>begin<u32>len<len>data

some other parts of the proto have slightly changed too:

OP_OFFERFILES 0x15:
if file is "large file" and server supports large files, size is splitted into two file tags
FT_FILESIZE 0x02 // <uint32>
FT_FILESIZE_HI 0x3A // <uint32>
if server doesnt support large files, file size is advertised as 0 (i dont know, if this ever should happend)

if file is large file and other client supports large files, size is handled in one tag
FT_FILESIZE 0x02 // <uint64>
if other client doesnt support large files, file size is advertised as 0

OP_SEARCHRESULT 0x33:
simular to above (filesize is splitted for server<->client and as one tag for client<->client)

and one change in udp server requests
for "small files"
OP_GLOBGETSOURCES2 0x94 // <hash>hash<u32>size
for "large files"
OP_GLOBGETSOURCES2 0x94 // <hash>hash<u32>0<u64>size

i dont know if udp request
OP_GLOBGETSOURCES 0x9A // <hash>hash
works with large files

that's all, i found till now

TripleM <triplem>
Mon 24 Jul 2006 06:19:32 PM UTC, comment #5: 

Merged from bug #17195:
When searching for files with -minsize option, values above 4290000000 (Integer) aren't handled correctly - smaller results are returned.

Example:
s zip -minsize 4300000000
returned results with filesizes from 4292870144 to 5144805...

I'm using the precompiled mldonkey-2.7.7.static.i386-Linux_glibc-2.3.2.tar.bz2 sourceforge release, and the cmd line interface.

spiralvoice <spiralvoice>
Project Administrator
Mon 08 May 2006 10:14:11 PM UTC, comment #4: 

Mmmh forget it, required changes are much larger than that...

pango <mlpango>
Project Member
Mon 08 May 2006 09:27:19 PM UTC, comment #3: 

As far as I can see from eMule 0.46c/0.47a diff, all what's needed is to remove size limitation, and add 0x100 (SRVCAP_LARGEFILES) to "server capabilities" (bitmap of features the client reports to servers it connects to).

Totally untested patch.

pango <mlpango>
Project Member
Sun 02 Apr 2006 11:04:13 AM UTC, comment #2: 

I have looked in emule/amule forums but I have not found out something related to protocol specification.
I have also the impression that the protocol change/extension has to be applied to Kad too(from implemented feature changelog).
I think you have to look into emule's code or ask to Kry.
Perhaps Amule is  a best place to look into , I have heard Amule is well documented.

Fabiano <fabtar>
Sat 01 Apr 2006 11:56:01 PM UTC, comment #1: 

Internally mldonkey uses 64 bits offsets, so unless the protocol extension specs are complex, it should be that hard to implement...
But where are the specs ?

pango <mlpango>
Project Member
Mon 20 Mar 2006 07:50:42 PM UTC, original submission:  

THIS ITEM WAS REASSIGNED TO PATCH #5599.

Emule can manage files bigger than 4GB (... 256GB is the new limit with 0.47a version!).

Fabiano <fabtar>

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by spiralvoice (Updated the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 200 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 20 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2006-11-29 spiralvoice Reassign Itemmldonkey, a multi-networks file-sharing client, task #5339 => mldonkey, a multi-networks file-sharing client, patch #5599
    2006-11-29 spiralvoice Attached File- => Added 256gb_4.patch, #11399
    2006-11-26 spiralvoice Attached File- => Added 256gb_3.patch, #11362
    2006-11-26 spiralvoice Attached File#11361 => Removed
    2006-11-26 spiralvoice Attached File- => Added 256gb_3.patch, #11361
    2006-11-26 spiralvoice Attached File#11292 => Removed
    2006-11-18 spiralvoice Attached File- => Added 256gb_3.patch, #11292
    2006-11-18 spiralvoice Attached File- => Added 256gb_2.patch, #11291
    2006-11-18 spiralvoice Attached File#11290 => Removed
    2006-11-17 spiralvoice Attached File- => Added 256gb_2.patch, #11290
    2006-11-17 spiralvoice Priority3 - Low => 7 - High
        StatusNone => In Progress
    2006-11-17 spiralvoice Attached File#11288 => Removed
    2006-11-17 spiralvoice Attached File- => Added 256gb_2.patch, #11288
    2006-11-15 triplem Attached File- => Added 256gb.patch, #11282
    2006-10-09 mckr Carbon-CopyRemoved mckr => -
    2006-10-09 mckr Carbon-Copy- => Added -email is unavailable-
    2006-10-09 mckr Carbon-Copy- => Added mckr
    2006-05-08 mlpango Attached File- => Added 256gb_donkey_files.patch, #9924
    2006-04-13 zeleni Carbon-Copy- => Added zeleni

    Back to the top


    Powered by Savane 3.7