patchmldonkey, a multi-networks file-sharing client - Patches: patch #4342, Large_file support for ed2k_hash...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #4342: Large_file support for ed2k_hash and MinGW MLDonkey core

Submitter:  me <skaus>
Submitted:  Wed 24 Aug 2005 07:13:10 PM UTC
   
 
Category:  None Priority:  5 - Normal
Status:  Done Assigned to:  None
Open/Closed:  Closed

Jump to the original submission

Sun 28 Aug 2005 10:48:32 AM UTC, comment #6: 

Applied 2005/08/28

spiralvoice <spiralvoice>
Group administrator
Fri 26 Aug 2005 01:56:25 PM UTC, comment #5: 

Updated patch - found more off_t's in commonHasher_c.c.

z <zet>
Group Member
Wed 24 Aug 2005 07:13:10 PM UTC, comment #4: 

This item has been reassigned from the project mldonkey, a multi-networks file-sharing client bugs tracker to your tracker.

The original report is still available at bugs #13712

Following are the information included in the original report:

[field #0]                  Item ID: 13712
[field #1]                 Group ID: 1409
[field #2]              Open/Closed: Open
[field #3]                 Severity: 3 - Normal
[field #4]                  Privacy: Public
[field #5]                 Category: None
[field #6]             Submitted by: skaus
[field #7]              Assigned to: None
[field #8]             Submitted on: Sun 07/10/05 at 13:15
[field #9]                  Summary: ed2k_hash never completes for large files
[field #10]      Original Submission: Hello,

I m running

ed2k_hash "file" >file.hash 2>&1

for all my files. For several files this command never completes, the files are, most of the time large, 1GB or more. I noticed this behaviour before with elder releases, but had time right now to investigate it further.

ed2k_hash is running and outputs:
Current locale of the target machine is ANSI_X3.4-1968
Current language of the target machine is EN
ed2k://|<>|/
sig2dat://|<>|/
    Hash: <>

then the program stalls, the urn_biprint is missing. It now runs for 2h30m real time and 79m CPU time, it allocates all remaining CPU (Idle percentage is approx 0), I have been running "strace -olog -p<>" for 20m now, the "log" file empty. The file in question is ~1.1GB.

BTW: Why is "Current ..." displayed on stdout, but the hashes themselves on stderr?

[field #12]               Item Group: Program malfunction
[field #13]                   Status: None
[field #14]        Component Version: None
[field #15]         Platform Version: Linux
[field #16]          Reproducibility: None
[field #17]               Size (loc): None
[field #18]            Fixed Release: None
[field #19]          Planned Release: None
[field #20]                   Effort: 0.00
[field #24]                 Priority: 5 - Normal
[field #27]         Percent Complete: 0%
[field #29]                  Release: 2.5.30.17
[field #54]          Binaries Origin: CVS / Self compiled
[field #55]                 CPU type: Intel x86
[field #56]                      CPU: None
[field #57]     Custom Select Box #4: None
[field #58]     Custom Select Box #5: None
[field #59]     Custom Select Box #6: None
[field #60]     Custom Select Box #7: None
[field #61]     Custom Select Box #8: None
[field #62]     Custom Select Box #9: None
[field #63]    Custom Select Box #10: None

z <zet>
Group Member
Wed 24 Aug 2005 07:13:09 PM UTC, comment #3: 

On mingw, off_t is long.  This causes files > 2 gig to fail in os_lseek. 

This updated patch uses off64_t for mingw.

Tested mld 2.6.3 on linux/cygwin/mingw, with 2 and 4.5 gig files.

z <zet>
Group Member
Wed 24 Aug 2005 05:16:49 AM UTC, comment #2: 

I receive:

Fatal error: exception Unix.Unix_error(12, "os_read", "")

when trying to ed2k_hash 2+ gig files at the bitprint part (sha1 and tigertree).  I also think that I noticed an infinite loop in tiger_block_size when the int is overflowed.

This is obviously all due to the use of ints. 

Attached is a patch I used to switch to size_t and tested on small files, 2 gig files, and 4.5 gig files. All seem to work for me now.  Tested on linux and cygwin (haven't tried mingw) with mld 2.6.3.
 
The hash results still match bitzi bitcollider and ed2k-tools.sf.net's ed2khash app, but both of those apps fail on files > 2 gigs.  I don't know of another app with which to compare.

z <zet>
Group Member
Sun 10 Jul 2005 01:19:15 PM UTC, comment #1: 

I forgot to mention that ed2k_hash has to be terminated by SIGKILL, neither 15 nor 2 nor 1 is acted on.

me <skaus>
Wed 24 Aug 2005 07:13:10 PM UTC, original submission:  

Hello,

I'm running

ed2k_hash "file" >file.hash 2>&1

for all my files. For several files this command never completes, the files are, most of the time large, 1GB or more. I noticed this behaviour before with elder releases, but had time right now to investigate it further.

ed2k_hash is running and outputs:
Current locale of the target machine is ANSI_X3.4-1968
Current language of the target machine is EN
ed2k://|<<snip>>|/
sig2dat://|<<snip>>|/
    Hash: <<snip>>

then the program stalls, the urn_biprint is missing. It now runs for 2h30m real time and 79m CPU time, it allocates all remaining CPU (Idle percentage is approx 0), I have been running "strace -olog -p<<pid>>" for 20m now, the "log" file empty. The file in question is ~1.1GB.

BTW: Why is "Current ..." displayed on stdout, but the hashes themselves on stderr?

me <skaus>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #9276:  size_t-off_t2.diff added by zet (13KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

 

Follow 8 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2005-08-28 spiralvoice StatusNone Done
    Open/ClosedOpen Closed
2005-08-26 zet Attached File- Added size_t-off_t2.diff, #5079
2005-08-26 zet Attached File#5068 Removed
2005-08-25 spiralvoice Summaryed2k_hash never completes for large files Large_file support for ed2k_hash and MinGW MLDonkey core
2005-08-25 spiralvoice SummaryReassigned item: ed2k_hash never completes for large files ed2k_hash never completes for large files
2005-08-24 zet Attached File#5067 Removed
2005-08-24 zet Reassign itemFrom group mldonkey, a multi-networks file-sharing client, bugs tracker To group mldonkey, a multi-networks file-sharing client, patch tracker

Back to the top

Powered by Savane 3.13-02a9.
Corresponding source code