taskmldonkey, a multi-networks file-sharing client - Tasks: task #5339, Reassigned to another tracker...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #5339: Reassigned to another tracker [was: ED2K: Support for file's sizes >4GB]

Submitter:  Fabiano <fabtar>
Submitted:  Mon 20 Mar 2006 07:50:42 PM UTC
Votes:  200  
 
Category:  eDonkey-plugin Priority:  7 - High
Status:  In Progress Assigned to:  None
Open/Closed:  Closed Should Start On:  Mon 20 Mar 2006 12:00:00 AM UTC
Should be Finished on:  Mon 20 Mar 2006 12:00:00 AM UTC

( Jump to the original submission )

Wed 29 Nov 2006 12:08:26 AM UTC, comment #15: 

THIS ITEM WAS REASSIGNED TO PATCH #5599.

Please, do not post any new comments to this item.


spiralvoice <spiralvoice>
Project Administrator
Sat 18 Nov 2006 04:22:02 PM UTC, comment #14: 

Posted 256gb_3.patch which is an optimized version of 256gb_2.patch.

I merged the new modules QueryBloc64, Bloc64 and EmuleCompressedPart64 into the existing modules.

The border between old and new files is not
0xffffffffL (4294967295 bytes) but

eMule 0.47c, opcodes.h
#define OLD_MAX_EMULE_FILE_SIZE 4290048000ui64

Patch has been updated to reflect this using function file_is_largefile.
This patch changes much stuff, please compare it to your original
to be sure that I did not introduce any new bugs.

spiralvoice <spiralvoice>
Project Administrator
Sat 18 Nov 2006 01:27:27 PM UTC, comment #13: 

"so the tags are ok?!"

I do not know if all of them are ok, I test here step-by-step.
Working on Debian Etch I setup Wine and installed eMule.
Then I do the same steps with patched MLDonkey and eMule and
compare the packets created. Wireshark has a fine eDonkey
packet decoder which makes the comparison pretty easy.

Tested so far with positive results:
- getsources
- publish

If you like you can do the same and test all functions
necessary for large file support. If I feel the patch is
complete I will do a local transfer of a 5GB file between
eMule 0.47c and MLDonkey using a local eserver.

spiralvoice <spiralvoice>
Project Administrator
Sat 18 Nov 2006 12:33:54 PM UTC, comment #12: 

ah, now i see, why i couldnt find the problem with the servers. i assumed, mldonkey uses OP_GLOBGETSOURCES (0x9A), which doesnt requires a size, but it instead uses OP_GLOBGETSOURCES2(0x94), which indeed needs a leading 32bit zero before the size for large files. i looked at the wrong place, so the tags are ok?!

not completely, i think. a little offtopic, at the moment, the DonkeyMftp.buf_tag-code can only write 32bit-tags, even if they are 64bit, but the size could be a 64bit-tag, and in emule-source there is already a 64bit-tagtype defined:
#define TAGTYPE_UINT64 0x0B

TripleM <triplem>
Sat 18 Nov 2006 12:16:36 AM UTC, comment #11: 

New version of 256gb_2.patch to replace 256gb.patch
- publish largefiles only on servers supporting them
- print a log message if largefile could not be published on a server

spiralvoice <spiralvoice>
Project Administrator
Fri 17 Nov 2006 10:00:49 PM UTC, comment #10: 

"I removed the patch from donkeyProtoCom.ml completely"

This was wrong, for publishing files this patch is needed.
I can confirm that MLdonkey can publish > 4GB files on EDK servers
correctly which can in turn be found be eMule.
Uploaded correct version of 256gb_2.patch, but this patch is far
from complete, see comment #8

spiralvoice <spiralvoice>
Project Administrator
Fri 17 Nov 2006 01:18:12 PM UTC, comment #9: 

Computation of filesize is wrong in your patch, at least for GETSOURCES.

From eMule 0.47c

opcodes.h
=========
#define OLD_MAX_EMULE_FILE_SIZE 4290048000ui64  // (4294967295/PARTSIZE)*PARTSIZE = ~4GB^

AbstractFile.h
==============
bool IsLargeFile() const { return m_nFileSize > (uint64)OLD_MAX_EMULE_FILE_SIZE; }^

DownloadQueue.cpp, CDownloadQueue::ProcessLocalRequests
==========================================================
if (!cur_file->IsLargeFile()){^
smPacket.WriteUInt32((uint32)(uint64)cur_file->GetFileSize());^
}^
else{^
smPacket.WriteUInt32(0); // indicates that this is a large file and a uint64 follows^
smPacket.WriteUInt64(cur_file->GetFileSize());^
}^

This means for large files the int64_32 filesize is 0 and the >4GB filesize is sent afterwards. Getsources for smaller files is unchanged.
I removed the patch from donkeyProtoCom.ml completely and
added this to successfully receive sources from servers:

--- ././src/networks/donkey/donkeyProtoServer.ml 2006-11-14 19:42:59.000000000 +0100
+++ ././src/networks/donkey/donkeyProtoServer.ml 2006-11-17 14:11:05.000000000 +0100
@@ -830,7 +830,13 @@
       Printf.bprintf oc "QUERY LOCATION OF %s [%Ld]\n" (Md4.to_string t.md4) t.size

     let write buf t =
-      buf_md4 buf t.md4; buf_int64_32 buf t.size
+      buf_md4 buf t.md4;
+      if t.size > old_max_emule_file_size then
+        begin
+   buf_int64_32 buf 0L; buf_int64 buf t.size
+        end
+      else
+        buf_int64_32 buf t.size
   end

 module QueryLocationReply  = struct
--- ././src/networks/donkey/donkeyTypes.ml 2006-11-14 19:42:59.000000000 +0100
+++ ././src/networks/donkey/donkeyTypes.ml 2006-11-17 14:11:50.000000000 +0100
@@ -704,3 +708,5 @@

 let emule_proto () =
   { dummy_emule_proto with emule_version = 0 }
+
+let old_max_emule_file_size = 4290048000L

spiralvoice <spiralvoice>
Project Administrator
Wed 15 Nov 2006 09:25:56 PM UTC, comment #8: 

i used pangos 256gb-patch as a start, and added most of the missing ed2k-proto-stuff (i think, at least)

i run a test drive at the moment.

parts missing or incomplete (as known by me):
- difference in filesizeadvertisement for client<->client (64bit) and client<->server (2*32bit) (no real prob, cause client2client only occures, if share is browsable, and file is large_file)
- kad stuff
- handling of incapable clients/servers (clients should be handled right, but im not sure; large files are announced to incapable servers too, but this shouldnt be a real problem, as only the filesize known to server should be affected)

TripleM <triplem>
Wed 08 Nov 2006 07:52:15 AM UTC, comment #7: 

Copied your code fragments to the official Wiki page for EDK-4GB support:
http://mldonkey.sourceforge.net/EDK4GB#Code_fragments_found_by_TripleM

spiralvoice <spiralvoice>
Project Administrator
Wed 08 Nov 2006 07:28:06 AM UTC, comment #6: 

i browsed the sources of emule 0.47c and found some maybe known proto infos. client does not only advertise "large file capability" to server, but to clients too.

in tag "CT_EMULE_MISCOPTIONS2" (0xfe):
bit 0-3: Kad Version
bit 4: Large Files (includes support for 64bit tags)
bit 5: Ext Multipacket (Hash+Size instead of Hash)
bit 6: Reserved (ModBit)
bit 7: Supports CryptLayer
bit 8: Requests CryptLayer
bit 9: Requires CryptLayer
bit 10-31: Reserved

of course sending and requesting parts has changed.

for parts with beginoffset<0xFFFFFFFF and endoffset<=0xFFFFFFFF the old proto is used:
OP_REQUESTPARTS 0x47 // <hash>hash[3*<u32>begin][3*<u32>end]
OP_SENDINGPART 0x46 // <hash>hash<u32>begin<u32>end<data>
OP_COMPRESSEDPART 0x40 // <hash>hash<u32>begin<u32>len<len>data

for other:
OP_REQUESTPARTS_I64 0xA3 // <hash>hash[3*<u64>begin][3*<u64>end
OP_SENDINGPART_I64 0xA2 // <hash>hash<u64>begin<u64>end<data>
OP_COMPRESSEDPART_I64 0xA1 // <hash>hash<u64>begin<u32>len<len>data

some other parts of the proto have slightly changed too:

OP_OFFERFILES 0x15:
if file is "large file" and server supports large files, size is splitted into two file tags
FT_FILESIZE 0x02 // <uint32>
FT_FILESIZE_HI 0x3A // <uint32>
if server doesnt support large files, file size is advertised as 0 (i dont know, if this ever should happend)

if file is large file and other client supports large files, size is handled in one tag
FT_FILESIZE 0x02 // <uint64>
if other client doesnt support large files, file size is advertised as 0

OP_SEARCHRESULT 0x33:
simular to above (filesize is splitted for server<->client and as one tag for client<->client)

and one change in udp server requests
for "small files"
OP_GLOBGETSOURCES2 0x94 // <hash>hash<u32>size
for "large files"
OP_GLOBGETSOURCES2 0x94 // <hash>hash<u32>0<u64>size

i dont know if udp request
OP_GLOBGETSOURCES 0x9A // <hash>hash
works with large files

that's all, i found till now

TripleM <triplem>
Mon 24 Jul 2006 06:19:32 PM UTC, comment #5: 

Merged from bug #17195:
When searching for files with -minsize option, values above 4290000000 (Integer) aren't handled correctly - smaller results are returned.

Example:
s zip -minsize 4300000000
returned results with filesizes from 4292870144 to 5144805...

I'm using the precompiled mldonkey-2.7.7.static.i386-Linux_glibc-2.3.2.tar.bz2 sourceforge release, and the cmd line interface.

spiralvoice <spiralvoice>
Project Administrator
Mon 08 May 2006 10:14:11 PM UTC, comment #4: 

Mmmh forget it, required changes are much larger than that...

pango <mlpango>
Project Member
Mon 08 May 2006 09:27:19 PM UTC, comment #3: 

As far as I can see from eMule 0.46c/0.47a diff, all what's needed is to remove size limitation, and add 0x100 (SRVCAP_LARGEFILES) to "server capabilities" (bitmap of features the client reports to servers it connects to).

Totally untested patch.

pango <mlpango>
Project Member
Sun 02 Apr 2006 11:04:13 AM UTC, comment #2: 

I have looked in emule/amule forums but I have not found out something related to protocol specification.
I have also the impression that the protocol change/extension has to be applied to Kad too(from implemented feature changelog).
I think you have to look into emule's code or ask to Kry.
Perhaps Amule is  a best place to look into , I have heard Amule is well documented.

Fabiano <fabtar>
Sat 01 Apr 2006 11:56:01 PM UTC, comment #1: 

Internally mldonkey uses 64 bits offsets, so unless the protocol extension specs are complex, it should be that hard to implement...
But where are the specs ?

pango <mlpango>
Project Member
Mon 20 Mar 2006 07:50:42 PM UTC, original submission:  

THIS ITEM WAS REASSIGNED TO PATCH #5599.

Emule can manage files bigger than 4GB (... 256GB is the new limit with 0.47a version!).

Fabiano <fabtar>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by spiralvoice (Updated the item)
  •  

    There are 200 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

     

     

    Follow 20 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2006-11-29 spiralvoice Reassign Itemmldonkey, a multi-networks file-sharing client, task #5339 mldonkey, a multi-networks file-sharing client, patch #5599
    2006-11-29 spiralvoice Attached File- Added 256gb_4.patch, #11399
    2006-11-26 spiralvoice Attached File- Added 256gb_3.patch, #11362
    2006-11-26 spiralvoice Attached File#11361 Removed
    2006-11-26 spiralvoice Attached File- Added 256gb_3.patch, #11361
    2006-11-26 spiralvoice Attached File#11292 Removed
    2006-11-18 spiralvoice Attached File- Added 256gb_3.patch, #11292
    2006-11-18 spiralvoice Attached File- Added 256gb_2.patch, #11291
    2006-11-18 spiralvoice Attached File#11290 Removed
    2006-11-17 spiralvoice Attached File- Added 256gb_2.patch, #11290
    2006-11-17 spiralvoice Priority3 - Low 7 - High
        StatusNone In Progress
    2006-11-17 spiralvoice Attached File#11288 Removed
    2006-11-17 spiralvoice Attached File- Added 256gb_2.patch, #11288
    2006-11-15 triplem Attached File- Added 256gb.patch, #11282
    2006-10-09 mckr Carbon-CopyRemoved mckr -
    2006-10-09 mckr Carbon-Copy- Added -email is unavailable-
    2006-10-09 mckr Carbon-Copy- Added mckr
    2006-05-08 mlpango Attached File- Added 256gb_donkey_files.patch, #9924
    2006-04-13 zeleni Carbon-Copy- Added zeleni

    Back to the top


    Powered by Savane 3.9