patchmldonkey, a multi-networks file-sharing client - Patches: patch #4550, Create bitprint for large files

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #4550: Create bitprint for large files

Submitter:  me <skaus>
Submitted:  Tue 25 Oct 2005 09:44:08 AM UTC
   
 
Category:  None Priority:  5 - Normal
Status:  Done Assigned to:  None
Open/Closed:  Closed

Jump to the original submission

Tue 25 Oct 2005 09:47:06 AM UTC, comment #8: 

Applied 2005/10/25

spiralvoice <spiralvoice>
Group administrator
Tue 25 Oct 2005 09:44:08 AM UTC, comment #7: 

This item has been reassigned from the project mldonkey, a multi-networks file-sharing client bugs tracker to your tracker.

The original report is still available at bugs #13712

Following are the information included in the original report:

[field #0]                  Item ID: 13712
[field #1]                 Group ID: 1409
[field #2]              Open/Closed: Open
[field #3]                 Severity: 3 - Normal
[field #4]                  Privacy: Public
[field #6]                 Category: None
[field #7]             Submitted by: skaus
[field #8]              Assigned to: None
[field #9]             Submitted on: So 10.07.2005 um 15:15
[field #10]                  Summary: ed2k_hash never completes for large files
[field #11]      Original Submission: Hello,

I m running

ed2k_hash "file" >file.hash 2>&1

for all my files. For several files this command never completes, the files are, most of the time large, 1GB or more. I noticed this behaviour before with elder releases, but had time right now to investigate it further.

ed2k_hash is running and outputs:
Current locale of the target machine is ANSI_X3.4-1968
Current language of the target machine is EN
ed2k://|<>|/
sig2dat://|<>|/
    Hash: <>

then the program stalls, the urn_biprint is missing. It now runs for 2h30m real time and 79m CPU time, it allocates all remaining CPU (Idle percentage is approx 0), I have been running "strace -olog -p<>" for 20m now, the "log" file empty. The file in question is ~1.1GB.

BTW: Why is "Current ..." displayed on stdout, but the hashes themselves on stderr?

[field #12]                Closed on: Mi 24.08.2005 um 21:13
[field #13]               Item Group: Program malfunction
[field #14]                   Status: None
[field #15]        Component Version: None
[field #16]         Platform Version: Linux
[field #17]          Reproducibility: None
[field #18]               Size (loc): None
[field #19]            Fixed Release: None
[field #20]          Planned Release: None
[field #21]                   Effort: 0.00
[field #25]                 Priority: 5 - Normal
[field #28]         Percent Complete: 0%
[field #30]                  Release: 2.5.30.17
[field #55]          Binaries Origin: CVS / Self compiled
[field #56]                 CPU type: Intel x86
[field #57]                      CPU: None
[field #58]     Custom Select Box #4: None
[field #59]     Custom Select Box #5: None
[field #60]     Custom Select Box #6: None
[field #61]     Custom Select Box #7: None
[field #62]     Custom Select Box #8: None
[field #63]     Custom Select Box #9: None
[field #64]    Custom Select Box #10: None

spiralvoice <spiralvoice>
Group administrator
Mon 24 Oct 2005 09:39:45 PM UTC, comment #6: 

Quick solution would be to use unsigned long...

...
  Partial 109 : D7DEF262A127CD79096A108E7A9FC138
  Partial 110 : 8E4AF1776F86272799C3324F9AF93A6E
ed2k://|file|mlnet|1077936128|8EF4E4C0E746720ADA08A0DC0D0E4658|/
sig2dat://|File: /tmp/mlnet|Length: 1077936128 Bytes|UUHash: =kK7e2ZIs+JRup4WGNUk3JP9P+/8=|/
    Hash: 90aeded9922cf8946ea7858635493724ff4ffbff
urn:bitprint:ALKVTLHZB4RJZXTZ2SX4HBJKOCKRWMO3.HS6IGQE5E23KZQT6Q3QDX7O6L4FDMLQMJQO3PUI
  Partial   0 : MUACEID6UTVUKTRE2MTZKOPTZTMS6A2OF6B4ZNY
...
  Partial 1027 : MUACEID6UTVUKTRE2MTZKOPTZTMS6A2OF6B4ZNY
urn:bitprint:ALKVTLHZB4RJZXTZ2SX4HBJKOCKRWMO3.HS6IGQE5E23KZQT6Q3QDX7O6L4FDMLQMJQO3PUI

z <zet>
Group Member
Mon 24 Oct 2005 07:56:52 PM UTC, comment #5: 

Quoted from bug #14851:

For testing purpose, I ran these commands:

$ cd /tmp
$ dd if=/dev/zero of=mlnet bs=$(expr 1024 \* 1024) count=$(expr 1024 + 4)
$ /.../mld_hash -partial ./mlnet
[...]
Partial 109 : D7DEF262A127CD79096A108E7A9FC138
Partial 110 : 8E4AF1776F86272799C3324F9AF93A6E
ed2k://|file|mlnet|1077936128|8EF4E4C0E746720ADA08A0DC0D0E4658|/
sig2dat://|File: ./mlnet|Length: 1077936128 Bytes|UUHash: =kK7e2ZIs+JRup4WGNUk3JP9P+/8=|/
Hash: 90aeded9922cf8946ea7858635493724ff4ffbff

Then, nothing more

spiralvoice <spiralvoice>
Group administrator
Mon 24 Oct 2005 07:19:58 PM UTC, comment #4: 

I re-checked my problem again with a file of the size:

1992265570 bytes

and File: mld_hash.ml       Status: Up-to-date

   Working revision:    1.5
   Repository revision: 1.5     /cvsroot/mldonkey/mldonkey/tools/mld_hash.ml,v

After some time mld_hash stalls (I let it run for more than one hour), consumes 100% available CPU, strace does not print anything (I let it run for more than 30 minutes); and needs to be killed by SIGKILL.

The line missing is the one(s) of urn:bitprint.


me <skaus>
Wed 24 Aug 2005 07:13:09 PM UTC, comment #3: 

On mingw, off_t is long.  This causes files > 2 gig to fail in os_lseek. 

This updated patch uses off64_t for mingw.

Tested mld 2.6.3 on linux/cygwin/mingw, with 2 and 4.5 gig files.

z <zet>
Group Member
Wed 24 Aug 2005 05:16:49 AM UTC, comment #2: 

I receive:

Fatal error: exception Unix.Unix_error(12, "os_read", "")

when trying to ed2k_hash 2+ gig files at the bitprint part (sha1 and tigertree).  I also think that I noticed an infinite loop in tiger_block_size when the int is overflowed.

This is obviously all due to the use of ints. 

Attached is a patch I used to switch to size_t and tested on small files, 2 gig files, and 4.5 gig files. All seem to work for me now.  Tested on linux and cygwin (haven't tried mingw) with mld 2.6.3.
 
The hash results still match bitzi bitcollider and ed2k-tools.sf.net's ed2khash app, but both of those apps fail on files > 2 gigs.  I don't know of another app with which to compare.

z <zet>
Group Member
Sun 10 Jul 2005 01:19:15 PM UTC, comment #1: 

I forgot to mention that ed2k_hash has to be terminated by SIGKILL, neither 15 nor 2 nor 1 is acted on.

me <skaus>
Tue 25 Oct 2005 09:44:08 AM UTC, original submission:  

Hello,

I'm running

ed2k_hash "file" >file.hash 2>&1

for all my files. For several files this command never completes, the files are, most of the time large, 1GB or more. I noticed this behaviour before with elder releases, but had time right now to investigate it further.

ed2k_hash is running and outputs:
Current locale of the target machine is ANSI_X3.4-1968
Current language of the target machine is EN
ed2k://|<<snip>>|/
sig2dat://|<<snip>>|/
    Hash: <<snip>>

then the program stalls, the urn_biprint is missing. It now runs for 2h30m real time and 79m CPU time, it allocates all remaining CPU (Idle percentage is approx 0), I have been running "strace -olog -p<<pid>>" for 20m now, the "log" file empty. The file in question is ~1.1GB.

BTW: Why is "Current ..." displayed on stdout, but the hashes themselves on stderr?

me <skaus>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #7664:  ulong.diff added by zet (1KiB - application/octet-stream)
file #7666:  size_t-off_t.diff added by zet (9KiB - application/octet-stream)
file #7665:  size_t.diff added by zet (6KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

 

Follow 4 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2005-10-25 spiralvoice StatusNone Done
    Open/ClosedOpen Closed
2005-10-25 spiralvoice SummaryReassigned item: ed2k_hash never completes for large files Create bitprint for large files
2005-10-25 spiralvoice Reassign itemFrom group mldonkey, a multi-networks file-sharing client, bugs tracker To group mldonkey, a multi-networks file-sharing client, patch tracker

Back to the top

Powered by Savane 3.13-4b48.
Corresponding source code