patchmldonkey, a multi-networks file-sharing client - Patches: patch #5599, ED2K: Support for file's sizes...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #5599: ED2K: Support for file's sizes >4GB

Submitter:  Fabiano <fabtar>
Submitted:  Wed 29 Nov 2006 12:08:26 AM UTC
   
 
Category:  Improved interfaces Priority:  5 - Normal
Status:  Done Assigned to:  None
Open/Closed:  Closed

Jump to the original submission

Sat 06 Jan 2007 06:15:48 PM UTC, comment #19: 

Committed 5599_7.patch to CVS 2007/01/06

spiralvoice <spiralvoice>
Group administrator
Thu 30 Nov 2006 08:52:48 PM UTC, comment #18: 

im so blind... now i found the error, why patch4 and following dont work correctly.
donkeyProtoClient.mldonkey_emule_proto.emule_largefiles is set to 0, and the patches dont change this. but patch6 does.

(file #11416)

TripleM <triplem>
Thu 30 Nov 2006 04:56:48 PM UTC, comment #17: 

im not right, with merging of the modules it should be able to communicate in 64bit only. i changed your patch to better reflect this ability.

but i found a logical error too.
256gb_4.patch line#207f
+  let is_largefile = file_is_largefile file || check_large ranges in
+  if is_largefile && c.client_emule_proto.emule_largefiles <> 1 then

this is not correct in this form, atm 64bit should only be used in cases, where ranges above old_emule_max_size are used, not in general, the logical or is the mistake, should be logical and. but in case that check_large ranges is true, file_is_largefile file should be true too, so this is redundant. i changed it to:
+  let is_large_request = check_large ranges in
+  if file_is_largefile file && c.client_emule_proto.emule_largefiles <> 1 then
in 256gb_5.patch

(file #11415)

TripleM <triplem>
Thu 30 Nov 2006 01:09:17 AM UTC, comment #16: 

only a small reply, it should not in any way disturb a proper working of this patch. you are partly wrong with your statment "the border is not 0xffffffffL". this is true for large file or not large file, but wrong for using 64bit-opcodes, or 32bit-opcodes.
UploadClient.cpp line#510:
if (statpos > 0xFFFFFFFF || endpos > 0xFFFFFFFF){
  packet = new Packet(OP_SENDINGPART_I64,nPacketSize+32, OP_EMULEPROT, bFromPF);
  md4cpy(&packet->pBuffer[0],GetUploadFileID());
  PokeUInt64(&packet->pBuffer[16], statpos);
  PokeUInt64(&packet->pBuffer[24], endpos);
  memfile.Read(&packet->pBuffer[32],nPacketSize);
  theStats.AddUpDataOverheadFileRequest(32);
}
else{
  packet = new Packet(OP_SENDINGPART,nPacketSize+24, OP_EDONKEYPROT, bFromPF);
  md4cpy(&packet->pBuffer[0],GetUploadFileID());
  PokeUInt32(&packet->pBuffer[16], (uint32)statpos);
  PokeUInt32(&packet->pBuffer[20], (uint32)endpos);
  memfile.Read(&packet->pBuffer[24],nPacketSize);
  theStats.AddUpDataOverheadFileRequest(24);
}

but as i stated earlier, it should not matter, large-file-capable emules accept 64bit-opcodes independent of bounderies. you could even use pure 64bit for all files with largefilecapable emules, cause, as stated in opcodes.h, the tag large_file_capable not only means this, but support for 64bit tags and opcodes too. so i dont think, this behaviour will change in the future.

what could become a problem, is, if the emule devs decide to fully exhibit this behaviour (comm with capable clients in 64bit-opcodes only), we have to rework this patch. your changes to the patch (mainly merging the modules), as appreciated they are, revoke mldonkey the ability to use the 64bit-opcodes for smaller files.

sorry, that i mentioned it that late, but i dont found the time earlier to look deeper into your changes, and to crosscheck that against emules sources.

TripleM <triplem>
Wed 29 Nov 2006 12:08:26 AM UTC, comment #15: 

This item has been reassigned from the project mldonkey, a multi-networks file-sharing client task tracker to your tracker.

The original report is still available at task #5339

Following are the information included in the original report:

[field #0]                  Item ID: 5339
[field #1]                 Group ID: 1409
[field #2]              Open/Closed: Open
[field #3]                 Severity: 3 - Normal
[field #4]                  Privacy: Public
[field #6]                    Votes: 200
[field #9]                 Category: eDonkey-plugin
[field #10]             Submitted by: fabtar
[field #11]              Assigned to: None
[field #12]             Submitted on: Montag 20.03.2006 um 20:50
[field #13]                  Summary: ED2K: Support for file s sizes >4GB
[field #14]      Original Submission: Emule can manage files bigger than 4GB (... 256GB is the new limit with 0.47a version!).

[field #16]               Item Group: None
[field #17]                   Status: In Progress
[field #18]        Component Version: None
[field #19]         Operating System: None
[field #20]          Reproducibility: None
[field #21]               Size (loc): None
[field #22]            Fixed Release: None
[field #23]          Planned Release: None
[field #24]                   Effort: 0.00
[field #28]                 Priority: 7 - High
[field #29]          Should Start On: Montag 20.03.2006 um 01:00
[field #30]    Should be Finished on: Montag 20.03.2006 um 01:00
[field #31]         Percent Complete: 0%
[field #33]                  Release: None
[field #58]     Custom Select Box #1: None
[field #59]     Custom Select Box #2: None
[field #60]     Custom Select Box #3: None
[field #61]     Custom Select Box #4: None
[field #62]     Custom Select Box #5: None
[field #63]     Custom Select Box #6: None
[field #64]     Custom Select Box #7: None
[field #65]     Custom Select Box #8: None
[field #66]     Custom Select Box #9: None
[field #67]    Custom Select Box #10: None

spiralvoice <spiralvoice>
Group administrator
Sat 18 Nov 2006 04:22:02 PM UTC, comment #14: 

Posted 256gb_3.patch which is an optimized version of 256gb_2.patch.

I merged the new modules QueryBloc64, Bloc64 and EmuleCompressedPart64 into the existing modules.

The border between old and new files is not
0xffffffffL (4294967295 bytes) but

eMule 0.47c, opcodes.h
#define OLD_MAX_EMULE_FILE_SIZE 4290048000ui64

Patch has been updated to reflect this using function file_is_largefile.
This patch changes much stuff, please compare it to your original
to be sure that I did not introduce any new bugs.

spiralvoice <spiralvoice>
Group administrator
Sat 18 Nov 2006 01:27:27 PM UTC, comment #13: 

"so the tags are ok?!"

I do not know if all of them are ok, I test here step-by-step.
Working on Debian Etch I setup Wine and installed eMule.
Then I do the same steps with patched MLDonkey and eMule and
compare the packets created. Wireshark has a fine eDonkey
packet decoder which makes the comparison pretty easy.

Tested so far with positive results:
- getsources
- publish

If you like you can do the same and test all functions
necessary for large file support. If I feel the patch is
complete I will do a local transfer of a 5GB file between
eMule 0.47c and MLDonkey using a local eserver.

spiralvoice <spiralvoice>
Group administrator
Sat 18 Nov 2006 12:33:54 PM UTC, comment #12: 

ah, now i see, why i couldnt find the problem with the servers. i assumed, mldonkey uses OP_GLOBGETSOURCES (0x9A), which doesnt requires a size, but it instead uses OP_GLOBGETSOURCES2(0x94), which indeed needs a leading 32bit zero before the size for large files. i looked at the wrong place, so the tags are ok?!

not completely, i think. a little offtopic, at the moment, the DonkeyMftp.buf_tag-code can only write 32bit-tags, even if they are 64bit, but the size could be a 64bit-tag, and in emule-source there is already a 64bit-tagtype defined:
#define TAGTYPE_UINT64 0x0B

TripleM <triplem>
Sat 18 Nov 2006 12:16:36 AM UTC, comment #11: 

New version of 256gb_2.patch to replace 256gb.patch
- publish largefiles only on servers supporting them
- print a log message if largefile could not be published on a server

spiralvoice <spiralvoice>
Group administrator
Fri 17 Nov 2006 10:00:49 PM UTC, comment #10: 

"I removed the patch from donkeyProtoCom.ml completely"

This was wrong, for publishing files this patch is needed.
I can confirm that MLdonkey can publish > 4GB files on EDK servers
correctly which can in turn be found be eMule.
Uploaded correct version of 256gb_2.patch, but this patch is far
from complete, see comment #8

spiralvoice <spiralvoice>
Group administrator
Fri 17 Nov 2006 01:18:12 PM UTC, comment #9: 

Computation of filesize is wrong in your patch, at least for GETSOURCES.

From eMule 0.47c

opcodes.h
=========
#define OLD_MAX_EMULE_FILE_SIZE 4290048000ui64  // (4294967295/PARTSIZE)*PARTSIZE = ~4GB^

AbstractFile.h
==============
bool IsLargeFile() const { return m_nFileSize > (uint64)OLD_MAX_EMULE_FILE_SIZE; }^

DownloadQueue.cpp, CDownloadQueue::ProcessLocalRequests
==========================================================
if (!cur_file->IsLargeFile()){^
smPacket.WriteUInt32((uint32)(uint64)cur_file->GetFileSize());^
}^
else{^
smPacket.WriteUInt32(0); // indicates that this is a large file and a uint64 follows^
smPacket.WriteUInt64(cur_file->GetFileSize());^
}^

This means for large files the int64_32 filesize is 0 and the >4GB filesize is sent afterwards. Getsources for smaller files is unchanged.
I removed the patch from donkeyProtoCom.ml completely and
added this to successfully receive sources from servers:

--- ././src/networks/donkey/donkeyProtoServer.ml 2006-11-14 19:42:59.000000000 +0100
+++ ././src/networks/donkey/donkeyProtoServer.ml 2006-11-17 14:11:05.000000000 +0100
@@ -830,7 +830,13 @@
       Printf.bprintf oc "QUERY LOCATION OF %s [%Ld]\n" (Md4.to_string t.md4) t.size
 
     let write buf t =
-      buf_md4 buf t.md4; buf_int64_32 buf t.size
+      buf_md4 buf t.md4;
+      if t.size > old_max_emule_file_size then
+        begin
+   buf_int64_32 buf 0L; buf_int64 buf t.size
+        end
+      else
+        buf_int64_32 buf t.size
   end
 
 module QueryLocationReply  = struct
--- ././src/networks/donkey/donkeyTypes.ml 2006-11-14 19:42:59.000000000 +0100
+++ ././src/networks/donkey/donkeyTypes.ml 2006-11-17 14:11:50.000000000 +0100
@@ -704,3 +708,5 @@
 
 let emule_proto () =
   { dummy_emule_proto with emule_version = 0 }
+
+let old_max_emule_file_size = 4290048000L

spiralvoice <spiralvoice>
Group administrator
Wed 15 Nov 2006 09:25:56 PM UTC, comment #8: 

i used pangos 256gb-patch as a start, and added most of the missing ed2k-proto-stuff (i think, at least)

i run a test drive at the moment.

parts missing or incomplete (as known by me):
- difference in filesizeadvertisement for client<->client (64bit) and client<->server (2*32bit) (no real prob, cause client2client only occures, if share is browsable, and file is large_file)
- kad stuff
- handling of incapable clients/servers (clients should be handled right, but im not sure; large files are announced to incapable servers too, but this shouldnt be a real problem, as only the filesize known to server should be affected)

TripleM <triplem>
Wed 08 Nov 2006 07:52:15 AM UTC, comment #7: 

Copied your code fragments to the official Wiki page for EDK-4GB support:
http://mldonkey.sourceforge.net/EDK4GB#Code_fragments_found_by_TripleM

spiralvoice <spiralvoice>
Group administrator
Wed 08 Nov 2006 07:28:06 AM UTC, comment #6: 

i browsed the sources of emule 0.47c and found some maybe known proto infos. client does not only advertise "large file capability" to server, but to clients too.

in tag "CT_EMULE_MISCOPTIONS2" (0xfe):
bit 0-3: Kad Version
bit 4: Large Files (includes support for 64bit tags)
bit 5: Ext Multipacket (Hash+Size instead of Hash)
bit 6: Reserved (ModBit)
bit 7: Supports CryptLayer
bit 8: Requests CryptLayer
bit 9: Requires CryptLayer
bit 10-31: Reserved

of course sending and requesting parts has changed.

for parts with beginoffset<0xFFFFFFFF and endoffset<=0xFFFFFFFF the old proto is used:
OP_REQUESTPARTS 0x47 // <hash>hash[3*<u32>begin][3*<u32>end]
OP_SENDINGPART 0x46 // <hash>hash<u32>begin<u32>end<data>
OP_COMPRESSEDPART 0x40 // <hash>hash<u32>begin<u32>len<len>data

for other:
OP_REQUESTPARTS_I64 0xA3 // <hash>hash[3*<u64>begin][3*<u64>end
OP_SENDINGPART_I64 0xA2 // <hash>hash<u64>begin<u64>end<data>
OP_COMPRESSEDPART_I64 0xA1 // <hash>hash<u64>begin<u32>len<len>data

some other parts of the proto have slightly changed too:

OP_OFFERFILES 0x15:
if file is "large file" and server supports large files, size is splitted into two file tags
FT_FILESIZE 0x02 // <uint32>
FT_FILESIZE_HI 0x3A // <uint32>
if server doesnt support large files, file size is advertised as 0 (i dont know, if this ever should happend)

if file is large file and other client supports large files, size is handled in one tag
FT_FILESIZE 0x02 // <uint64>
if other client doesnt support large files, file size is advertised as 0

OP_SEARCHRESULT 0x33:
simular to above (filesize is splitted for server<->client and as one tag for client<->client)

and one change in udp server requests
for "small files"
OP_GLOBGETSOURCES2 0x94 // <hash>hash<u32>size
for "large files"
OP_GLOBGETSOURCES2 0x94 // <hash>hash<u32>0<u64>size

i dont know if udp request
OP_GLOBGETSOURCES 0x9A // <hash>hash
works with large files

that's all, i found till now

TripleM <triplem>
Mon 24 Jul 2006 06:19:32 PM UTC, comment #5: 

Merged from bug #17195:
When searching for files with -minsize option, values above 4290000000 (Integer) aren't handled correctly - smaller results are returned.

Example:
s zip -minsize 4300000000
returned results with filesizes from 4292870144 to 5144805...

I'm using the precompiled mldonkey-2.7.7.static.i386-Linux_glibc-2.3.2.tar.bz2 sourceforge release, and the cmd line interface.

spiralvoice <spiralvoice>
Group administrator
Mon 08 May 2006 10:14:11 PM UTC, comment #4: 

Mmmh forget it, required changes are much larger than that...

pango <mlpango>
Group Member
Mon 08 May 2006 09:27:19 PM UTC, comment #3: 

As far as I can see from eMule 0.46c/0.47a diff, all what's needed is to remove size limitation, and add 0x100 (SRVCAP_LARGEFILES) to "server capabilities" (bitmap of features the client reports to servers it connects to).

Totally untested patch.

pango <mlpango>
Group Member
Sun 02 Apr 2006 11:04:13 AM UTC, comment #2: 

I have looked in emule/amule forums but I have not found out something related to protocol specification.
I have also the impression that the protocol change/extension has to be applied to Kad too(from implemented feature changelog).
I think you have to look into emule's code or ask to Kry.
Perhaps Amule is  a best place to look into , I have heard Amule is well documented.

Fabiano <fabtar>
Sat 01 Apr 2006 11:56:01 PM UTC, comment #1: 

Internally mldonkey uses 64 bits offsets, so unless the protocol extension specs are complex, it should be that hard to implement...
But where are the specs ?

pango <mlpango>
Group Member
Wed 29 Nov 2006 12:08:26 AM UTC, original submission:  

Emule can manage files bigger than 4GB (... 256GB is the new limit with 0.47a version!).

Fabiano <fabtar>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #11438:  5599_7.patch added by spiralvoice (29KiB - application/octet-stream - updated 256gb_6.patch for current CVS)
file #11416:  256gb_6.patch added by triplem (28KiB - text/x-patch)
file #11415:  256gb_5.patch added by triplem (30KiB - text/x-patch)
file #11399:  256gb_4.patch added by spiralvoice (29KiB - application/octet-stream - updated 256gb_3.patch for current post-2.8.2 CVS)
file #11362:  256gb_3.patch added by spiralvoice (30KiB - application/octet-stream - version of 256gb_2.patch, updated for current CVS)
file #11291:  256gb_2.patch added by spiralvoice (30KiB - application/octet-stream)
file #11282:  256gb.patch added by triplem (28KiB - text/x-patch)
file #9924:  256gb_donkey_files.patch added by mlpango (2KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by triplem (Posted a comment)
  • -email is unavailable- added by spiralvoice (Updated the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-01-06 spiralvoice StatusReady For Test Done
        Open/ClosedOpen Closed
    2006-12-03 spiralvoice Attached File- Added 5599_7.patch, #11438
    2006-11-30 triplem Attached File- Added 256gb_6.patch, #11416
    2006-11-30 triplem Attached File- Added 256gb_5.patch, #11415
    2006-11-29 spiralvoice CategoryNone Improved interfaces
        Priority7 - High 5 - Normal
        StatusNone Ready For Test
    2006-11-29 spiralvoice Reassign itemmldonkey, a multi-networks file-sharing client, task #5339 mldonkey, a multi-networks file-sharing client, patch #5599

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code