taskmldonkey, a multi-networks file-sharing client - Tasks: task #4342, Access to disk data (free space,...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #4342: Access to disk data (free space, max file name length)

Submitted by:  None
Submitted on:  Fri 01 Jul 2005 11:31:37 PM UTC  
 
Category:  Core Priority:  8
Status:  In Progress Assigned to:  None
Open/Closed:  Closed Should Start On:  Fri 01 Jul 2005 12:00:00 AM UTC
Should be Finished on:  Fri 01 Jul 2005 12:00:00 AM UTC

( Jump to the original submission)

Wed 20 Jul 2005 04:00:07 PM UTC, comment #20: 

task moved to patch #4191

spiralvoice <spiralvoice>
Project Administrator
Sun 17 Jul 2005 05:02:23 PM UTC, comment #19: 
spiralvoice <spiralvoice>
Project Administrator
Wed 13 Jul 2005 09:47:22 PM UTC, comment #18: 

"I don`t see a sys.vfs on the OSX system, nor on FreeBSD 5.4. "

For FreeBSD it does not matter, there param.h and mount.h are used:

+#ifdef HAVE_SYS_PARAM_H
+#  include <sys/param.h>
+#  if (defined(_FreeBSD_) && _FreeBSD_version >= 503001) || defined(__OpenBSD_) || defined(_NetBSD_)
+#    include <sys/mount.h>

spiralvoice <spiralvoice>
Project Administrator
Wed 13 Jul 2005 09:39:49 PM UTC, comment #17: 

Here is my os_stubs_c.c: http://pastebin.com/312849

White_FrosT
Wed 13 Jul 2005 09:14:45 PM UTC, comment #16: 

I applied your patch, added --disable-iconv and forced HAVE_STATS to 1, but compiling of os_stubs_c.c fails:
/usr/local/bin/ocamlc.opt -verbose -ccopt "-I /byterun -o src/utils/cdk/zlibstubs.o" -ccopt "-g -O2   -D_THREAD_SAFE " -ccopt " " -cclib "" -ccopt " " -cclib "" -cclib -lz -ccopt "-D_THREAD_SAFE " -c src/utils/cdk/zlibstubs.c
+ gcc -fno-defer-pop -no-cpp-precomp -Wall -D_FILE_OFFSET_BITS=64 -D_REENTRANT  -c -I /byterun -o src/utils/cdk/zlibstubs.o -g -O2   -D_THREAD_SAFE      -D_THREAD_SAFE   -I'/usr/local/lib/ocaml' 'src/utils/cdk/zlibstubs.c'
src/utils/cdk/zlibstubs.c: In function 'camlzip_deflate':
src/utils/cdk/zlibstubs.c:87: warning: pointer targets in assignment differ in signedness
src/utils/cdk/zlibstubs.c:89: warning: pointer targets in assignment differ in signedness
src/utils/cdk/zlibstubs.c: In function 'camlzip_inflate':
src/utils/cdk/zlibstubs.c:135: warning: pointer targets in assignment differ in signedness
src/utils/cdk/zlibstubs.c:137: warning: pointer targets in assignment differ in signedness
src/utils/cdk/zlibstubs.c: In function 'camlzip_update_crc32':
src/utils/cdk/zlibstubs.c:170: warning: pointer targets in passing argument 2 of 'crc32' differ in signedness
/usr/local/bin/ocamlc.opt -ccopt "-g -O2   -D_THREAD_SAFE   -o src/utils/cdk/heap_c.o" -ccopt "" -c src/utils/cdk/heap_c.c
/usr/local/bin/ocamlc.opt -verbose -ccopt "-I /byterun -o src/config/unix/os_stubs_c.o" -ccopt "-g -O2   -D_THREAD_SAFE " -ccopt " " -cclib "" -ccopt " " -cclib "" -cclib -lz -ccopt "-D_THREAD_SAFE " -c src/config/unix/os_stubs_c.c
+ gcc -fno-defer-pop -no-cpp-precomp -Wall -D_FILE_OFFSET_BITS=64 -D_REENTRANT  -c -I /byterun -o src/config/unix/os_stubs_c.o -g -O2   -D_THREAD_SAFE      -D_THREAD_SAFE   -I'/usr/local/lib/ocaml' 'src/config/unix/os_stubs_c.c'
src/config/unix/os_stubs_c.c: In function 'os_ftruncate':
src/config/unix/os_stubs_c.c:83: warning: format '%d' expects type 'int', but argument 4 has type 'off_t'
src/config/unix/os_stubs_c.c: At top level:
src/config/unix/os_stubs_c.c:138: warning: 'struct statfs' declared inside parameter list
src/config/unix/os_stubs_c.c:138: warning: its scope is only this definition or declaration, which is probably not what you want
src/config/unix/os_stubs_c.c: In function 'copy_statfs':
src/config/unix/os_stubs_c.c:147: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c:149: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c:150: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c:151: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c:152: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c:153: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c:154: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c:175: error: dereferencing pointer to incomplete type
src/config/unix/os_stubs_c.c: In function 'statfs_statfs':
src/config/unix/os_stubs_c.c:194: error: storage size of 'buf' isn't known
src/config/unix/os_stubs_c.c:195: warning: implicit declaration of function 'statfs'
src/config/unix/os_stubs_c.c:197: warning: implicit declaration of function 'strerror'
src/config/unix/os_stubs_c.c:197: warning: passing argument 1 of 'caml_failwith' makes pointer from integer without a cast
src/config/unix/os_stubs_c.c:194: warning: unused variable 'buf'
src/config/unix/os_stubs_c.c: In function 'statfs_fstatfs':
src/config/unix/os_stubs_c.c:212: error: storage size of 'buf' isn't known
src/config/unix/os_stubs_c.c:213: warning: implicit declaration of function 'fstatfs'
src/config/unix/os_stubs_c.c:215: warning: passing argument 1 of 'caml_failwith' makes pointer from integer without a cast
src/config/unix/os_stubs_c.c:212: warning: unused variable 'buf'
make: *** [src/config/unix/os_stubs_c.o] Error 2

White_FrosT
Wed 13 Jul 2005 08:37:17 PM UTC, comment #15: 

I don`t see a sys.vfs on the OSX system, nor on FreeBSD 5.4. There is a <sys/statvfs.h>.

Relevant utput of `man statfs`:
     typedef struct { int32_t val[2]; } fsid_t;

     #define MFSNAMELEN   15 /* length of fs type name, not inc. nul */
     #define MNAMELEN     90 /* length of buffer for returned name */

     struct statfs {
         short   f_otype;    /* type of file system (reserved: zero) */
         short   f_oflags;   /* copy of mount flags (reserved: zero) */
         long    f_bsize;    /* fundamental file system block size */
         long    f_iosize;   /* optimal transfer block size */
         long    f_blocks;   /* total data blocks in file system */
         long    f_bfree;    /* free blocks in fs */
         long    f_bavail;   /* free blocks avail to non-superuser */
         long    f_files;    /* total file nodes in file system */
         long    f_ffree;    /* free file nodes in fs */
         fsid_t  f_fsid;     /* file system id (super-user only) */
         uid_t   f_owner;    /* user that mounted the file system */
         short   f_reserved1;        /* reserved for future use */
         short   f_type;     /* type of file system (reserved) */
         long    f_flags;    /* copy of mount flags (reserved) */
         long    f_reserved2[2];     /* reserved for future use */
         char    f_fstypename[MFSNAMELEN]; /* fs type name */
         char    f_mntonname[MNAMELEN];    /* directory on which mounted */
         char    f_mntfromname[MNAMELEN];  /* mounted file system */
         char    f_reserved3;        /* reserved for future use */
         long    f_reserved4[4];     /* reserved for future use */
     };

White_FrosT
Wed 13 Jul 2005 08:25:22 PM UTC, comment #14: 

The new functions are only active if <sys/param.h> or <sys/vfs.h>
are present on the system. Only then HAVE_STATS is defined to 1.
For testing you could add this line

+#  define HAVE_STATS 1

after

+#ifdef HAVE_SYS_VFS_H
+#  include <sys/vfs.h>
+#  define HAVE_STATS 1

to force the compile of the new functions.

https://savannah.nongnu.org/task/index.php#comment10
"If some C cracks are reading, the case HAVE_STATS 0 is not coded
yet because I do not know how to return "-1" values if none of
the header files and data structures are present."

spiralvoice <spiralvoice>
Project Administrator
Wed 13 Jul 2005 08:22:37 PM UTC, comment #13: 

Besides the link error for iconv, which is a different story
(please try "--disable-iconv"), could you please post the output
of the man page of "statfs" or "statvfs"?

Maybe it looks like this:
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/statfs.2.html

So you could extend the "#if defined" statements so Darwin is treated like OpenBSD. This should be added:
#if defined(_APPLE_)

Maximal file name length is a bit more complicated:
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/pathconf.2.html

This is from the patch:

+#  if defined(_OpenBSD_) || defined(_NetBSD_)
+#    include <sys/syslimits.h>
+  v = copy_int64 (NAME_MAX); caml_modify (&Field (bufv, 8), v);
+#  else
+  v = copy_int64 (buf->f_namemax); caml_modify (&Field (bufv, 8), v);
+#  endif

Please change this to:

+#  if defined(_OpenBSD_) || defined(_NetBSD_)
+#    include <sys/syslimits.h>
+  v = copy_int64 (NAME_MAX); caml_modify (&Field (bufv, 8), v);
+#  else
+#    if defined(_APPLE_)
+#      include <unistd.h>
+#      v = copy_int64 (_PC_NAME_MAX); caml_modify (&Field (bufv, 8), v);
+#    else
+       v = copy_int64 (buf->f_namemax); caml_modify (&Field (bufv, 8), v);
+#    endif
+#  endif

There is a MLDonkey command "disk <dir>" for tests and the display
of the "shares" command in HTML is extended to show some data.

spiralvoice <spiralvoice>
Project Administrator
Wed 13 Jul 2005 07:19:17 PM UTC, comment #12: 

Unfortunately, OSX exits final step with the following error:
/usr/local/bin/ocamlopt.opt -inline 10 -linkall  -o mlnet       unix.cmxa str.cmxa -ccopt " " -cclib "-lcharset  -liconv" -ccopt " " -cclib "" -cclib -lz -ccopt "-D_THREAD_SAFE "   -I build    build/cdk.cmxa    build/common.cmxa    build/client.cmxa    build/core.cmxa    build/driver.cmxa    src/daemon/common/commonMain.cmx 
/usr/bin/ld: warning multiple definitions of symbol _locale_charset
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/../../../libcharset.dylib(localcharset.o) definition of _locale_charset
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/../../../libiconv.dylib(localcharset.o) definition of _locale_charset
/usr/bin/ld: Undefined symbols:
_statfs_statfs
collect2: ld returned 1 exit status
Error during linking
make: *** [mlnet] Error 2

White_FrosT
Tue 12 Jul 2005 06:30:45 PM UTC, comment #11: 

What about Mac OSX?

spiralvoice <spiralvoice>
Project Administrator
Tue 12 Jul 2005 06:28:41 PM UTC, comment #10: 

Updated patch, should work on FreeBSD >= 5.3 and OpenBSD (tested on 5.3).
If there are older FreeBSD versions the version number in
os_stubs_c.c has to be changed, reports about older FreeBSD versions
where this patch works are welcome.

If some C cracks are reading, the case HAVE_STATS 0 is not coded
yet because I do not know how to return "-1" values if none of the
header files and data structures are present.

spiralvoice <spiralvoice>
Project Administrator
Mon 11 Jul 2005 11:32:51 PM UTC, comment #9: 

Still doesn't work on FreeBSD. Datastructure now in mldonkey:

type statfs = {
  f_type : int64;   (* type of filesystem *)
  f_bsize : int64;  (* optimal transfer block size *)
  f_blocks : int64; (* total data blocks in file system *)
  f_bfree : int64;  (* free blocks in fs *)
  f_bavail : int64; (* free blocks avail to non-superuser *)
  f_files : int64;  (* total file nodes in file system *)
  f_ffree : int64;  (* free file nodes in fs *)
  f_fsid : unit;  (* See note in statfs(2) *)
  f_fnamelen : int64; (* maximum length of filenames *)
  f_basetype : string; (* type of filesystem - Solaris, (-1) on other systems, use f_type there *)
  f_frsize : int64;  (* Fundamental file system block size, (-1) if not provided by system *)
}

Datatype decl. in FreeBSD
     struct statfs {
     uint32_t f_version;             /* structure version number */
     uint32_t f_type;                /* type of filesystem */
     uint64_t f_flags;               /* copy of mount exported flags */
     uint64_t f_bsize;               /* filesystem fragment size */
     uint64_t f_iosize;              /* optimal transfer block size */
     uint64_t f_blocks;              /* total data blocks in filesystem */
     uint64_t f_bfree;               /* free blocks in filesystem */
     int64_t  f_bavail;              /* free blocks avail to non-superuser */
     uint64_t f_files;               /* total file nodes in filesystem */
     int64_t  f_ffree;               /* free nodes avail to non-superuser */
     uint64_t f_syncwrites;          /* count of sync writes since mount */
     uint64_t f_asyncwrites;         /* count of async writes since mount */
     uint64_t f_syncreads;           /* count of sync reads since mount */
     uint64_t f_asyncreads;          /* count of async reads since mount */
     uint64_t f_spare[10];           /* unused spare */
     uint32_t f_namemax;             /* maximum filename length */
     uid_t     f_owner;              /* user that mounted the filesystem */
     fsid_t    f_fsid;               /* filesystem id */
     char      f_charspare[80];          /* spare string space */
     char      f_fstypename[MFSNAMELEN]; /* filesystem type name */
     char      f_mntfromname[MNAMELEN];  /* mounted filesystem */
     char      f_mntonname[MNAMELEN];    /* directory on which mounted */
     };

However statvfs should work on freebsd too starting from FreeBSD 5.0. I can't see light at the end of the tunnel yet.

White_FrosT
Mon 11 Jul 2005 08:16:13 PM UTC, comment #8: 

Updated for current CVS, should now work on Solaris, too.

spiralvoice <spiralvoice>
Project Administrator
Thu 07 Jul 2005 04:38:12 PM UTC, comment #7: 
spiralvoice <spiralvoice>
Project Administrator
Sun 03 Jul 2005 10:46:29 PM UTC, comment #6: 

Updated patch to compile on MinGW although no values are printed
because of the missing implementation.
Tried compiling on FreeBSD, it fails because header files are
different and the data structure is also different.

spiralvoice <spiralvoice>
Project Administrator
Sun 03 Jul 2005 02:52:11 PM UTC, comment #5: 

Summary was: commit failed for super-long name files

I renamed this task because a broader topic will be addressed here.
Attached is a patch which demonstrates the features to be applied .
For a first impression call the "shares" command in the HTML
interface, it should print some data about the shared directories,
!!temp_directory and the core directory.

It lays the foundation for accessing this data:

type statfs = {
  f_type : int64;   (* type of filesystem *)
  f_bsize : int64;  (* optimal transfer block size *)
  f_blocks : int64; (* total data blocks in file system *)
  f_bfree : int64;  (* free blocks in fs *)
  f_bavail : int64; (* free blocks avail to non-superus *)
  f_files : int64;  (* total file nodes in file system *)
  f_ffree : int64;  (* free file nodes in fs *)
  f_fsid : unit;  (* See note in statfs(2) *)
  f_fnamelen : int64; (* maximum length of filenames *)
}

For the problem with the too-long-filenames something
like this can be coded (to be done):

if Unix32.fnamelen incoming_dir < String.length filename then
  String.shorten filename (Unix32.fnamelen incoming_dir)

I am also thinking about an option hdd_space_reserve (in MB,
default 50):

if (Unix32.diskfree !!temp_directoriy) < (!!hdd_space_reserve ** Int64.megabyte) then () (* pause all downloads here,
print warnings on all interfaces, also in logfile and stop
logging *)

This should be done in the common module via add_infinite_timer.
These two things are most important and somewhat easy to code.

The rest is not so important as not all users are affected by
the problems described below.

The dllink command should also be made aware of this, if no
sparse files are used the file in !!temp_directory is created
with full length on Windows, so checking if enough space is
available and failing dllink is neccessary if not enough free
space is available. Currently only a zero-byte file and some
obscure error message is printed.

Something similar should be coded for committing if !!temp_directory
and the incoming directory are on different partitions, a check
on the incoming partition if enough space for copying the data
is available. But to reliable check if incoming and temp are
really on different physical partitions?
Unix2.rename tries renaming the file during commit, if it fails
with error code EXDEV
(see http://caml.inria.fr/pub/docs/manual-ocaml/libref/Unix.html)
it copies the file, but if this happens during multifile torrent
commit half of the files could already be copied as Unix2.rename
copies file by file and it is the only place where is notice that
different physical partitions are used.

If !!temp_directory and the incoming directory are on the same
physical partition this does not matter because the file is only
renamed so no more space is needed.

Further this patch currently only compiles on Linux,
Solaris/Sparc fails, did not test on other platforms yet.
Please report your findings, especially on Mac OSX, MinGW
(compile should fail there), Cygwin (might work) and *BSD.

spiralvoice <spiralvoice>
Project Administrator
Fri 01 Jul 2005 11:31:37 PM UTC, comment #4: 

This item has been reassigned from the project mldonkey, a multi-networks file-sharing client bugs tracker to your tracker.

The original report is still available at bugs #10758

Following are the information included in the original report:

[field #0]                  Item ID: 10758
[field #1]                 Group ID: 1409
[field #2]              Open/Closed: Open
[field #3]                 Severity: 3 - Normal
[field #4]                  Privacy: Public
[field #5]                 Category: Core
[field #6]             Submitted by: None
[field #7]              Assigned to: None
[field #8]             Submitted on: Don 21.10.2004 um 17:37
[field #9]                  Summary: commit failed for super-long name files
[field #10]      Original Submission: it s not forbidden to download a super-long-name file in mldonkey. But when it s done, it can t be commited.
I don t know the limit of the system exactly. But the only way to commit the file is to commit it as a new short name. Maybe it s not bad to check the filename length and auto-rename to a new filename with same ext name? ex: blah-blah-blah.rar -> 1.rar or blah~1.rar

ps. I got this problem because of filename with a unicode character with value 0xff,for example, is sent to mldonkey as  \255   or  %255  .
[field #12]               Item Group: Feature request
[field #13]                   Status: Postponed
[field #14]        Component Version: None
[field #15]         Platform Version: Linux
[field #16]          Reproducibility: None
[field #17]               Size (loc): None
[field #18]            Fixed Release: None
[field #19]          Planned Release: None
[field #20]                   Effort: 0.00
[field #24]                 Priority: 5 - Normal
[field #27]         Percent Complete: 0%
[field #29]                  Release: 2-5-28
[field #54]          Binaries Origin: None
[field #55]                 CPU type: Intel x86
[field #56]                      CPU: None
[field #57]     Custom Select Box #4: None
[field #58]     Custom Select Box #5: None
[field #59]     Custom Select Box #6: None
[field #60]     Custom Select Box #7: None
[field #61]     Custom Select Box #8: None
[field #62]     Custom Select Box #9: None
[field #63]    Custom Select Box #10: None

spiralvoice <spiralvoice>
Project Administrator
Fri 01 Jul 2005 11:31:35 PM UTC, comment #3: 

http://article.gmane.org/gmane.comp.lang.caml.general/29507

With this code MLDonkey is able to check fnamelen during runtime,
this bug is moved to the task tracker to remind me to implement this.

spiralvoice <spiralvoice>
Project Administrator
Fri 29 Apr 2005 09:48:46 AM UTC, comment #2: 

I would guess the file-name length limit is also different per filesystem, so i guess this would have to be an runtime check...
or an option with default 0 =unlimited .
But just printing the exception in the log would help, too :)

Amorphous <amorphous>
Tue 12 Apr 2005 10:22:47 AM UTC, comment #1: 

Unicode stuff is now in MLDonkey, try 2-5-30-4 + latest patches from here. How can we find find out the allowed length of filenames on the different systems MLDonkey can be built?

spiralvoice <spiralvoice>
Project Administrator
Fri 01 Jul 2005 11:31:37 PM UTC, original submission:  

it's not forbidden to download a super-long-name file in mldonkey. But when it's done, it can't be commited.
I don't know the limit of the system exactly. But the only way to commit the file is to commit it as a new short name. Maybe it's not bad to check the filename length and auto-rename to a new filename with same ext name? ex: blah-blah-blah.rar -> 1.rar or blah~1.rar

ps. I got this problem because of filename with a unicode character with value 0xff,for example, is sent to mldonkey as '\255'  or '%255' .

Anonymous

 

Attached Files
file #3692:  free2.patch added by spiralvoice (16KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

 

 

Follow 21 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2005-07-20 spiralvoice Reassign ItemFrom group mldonkey, a multi-networks file-sharing client, task tracker => To group mldonkey, a multi-networks file-sharing client, patch tracker
2005-07-20 spiralvoice Reassign ItemFrom group mldonkey, a multi-networks file-sharing client, task tracker => To group mldonkey, a multi-networks file-sharing client, patch tracker
2005-07-17 beedauchon StatusNone => In Progress
2005-07-12 spiralvoice Attached File#223 => Removed
2005-07-12 spiralvoice Attached File- => Added free2.patch, #227
2005-07-11 spiralvoice Attached File- => Added free.patch, #223
2005-07-11 spiralvoice Attached File#222 => Removed
2005-07-11 spiralvoice Attached File#213 => Removed
2005-07-11 spiralvoice Attached File- => Added free.patch, #222
2005-07-11 spiralvoice Priority5 - Normal => 8
2005-07-08 spiralvoice CategoryNone => Core
2005-07-07 spiralvoice Attached File#209 => Removed
2005-07-07 spiralvoice Attached File- => Added free2.patch, #213
2005-07-03 spiralvoice Attached File#208 => Removed
2005-07-03 spiralvoice Attached File- => Added free2.patch, #209
2005-07-03 spiralvoice Attached File- => Added free.patch, #208
2005-07-03 spiralvoice Attached File#207 => Removed
2005-07-03 spiralvoice Summarycommit failed for super-long name files => Access to disk data (free space, max file name length)
    Attached File- => Added free.patch, #207
2005-07-01 spiralvoice SummaryReassigned item: commit failed for super-long name files => commit failed for super-long name files
2005-07-01 spiralvoice Reassign itemFrom group mldonkey, a multi-networks file-sharing client, bugs tracker => To group mldonkey, a multi-networks file-sharing client, task tracker

Back to the top


Powered by Savane 3.7