patchmldonkey, a multi-networks file-sharing client - Patches: patch #4976, Filetype recognition with libmagic

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #4976: Filetype recognition with libmagic

Submitter:  Joakim Verona <jave>
Submitted:  Thu 16 Mar 2006 10:18:01 PM UTC
   
 
Category:  Improved interfaces Priority:  5 - Normal
Status:  Done Assigned to:  None
Open/Closed:  Closed

Jump to the original submission

Fri 12 May 2006 09:16:03 PM UTC, comment #11: 

Applied 2006/05/12

spiralvoice <spiralvoice>
Group administrator
Fri 12 May 2006 12:01:26 AM UTC, comment #10: 

Updated 4976_3.patch
- Magic values for BT sub_files (displayed in vd #num) and shared files (displayed in upstats, js popup) are now also computed
- Storing magic value strings happens now in a weak hashtable to save RAM (see mem_stats CommonFile, files_magic:)

spiralvoice <spiralvoice>
Group administrator
Mon 20 Mar 2006 01:26:23 PM UTC, comment #9: 
spiralvoice <spiralvoice>
Group administrator
Mon 20 Mar 2006 11:52:44 AM UTC, comment #8: 

added another version of this patch, using Ocaml-Magic bindings
from http://sourceforge.net/projects/ocaml-magic

  • new value impl_file_magic in CommonFile structure (Todo: GUI protocol)
  • to read CommonFile.file_magic file
  • CommonFile magic values are computed once a minute for files without it
  • new debug command: debug_dir, prints magic values for all files


spiralvoice <spiralvoice>
Group administrator
Sat 18 Mar 2006 10:06:16 PM UTC, comment #7: 

basic utilization of libmagic in magic2.patch.
show libmagic info in "vd num" and in webgui rollovers

Joakim Verona <jave>
Group Member
Thu 16 Mar 2006 10:18:01 PM UTC, comment #6: 

This item has been reassigned from the project mldonkey, a multi-networks file-sharing client task tracker to your tracker.

The original report is still available at task #5324

Following are the information included in the original report:

[field #0]                  Item ID: 5324
[field #1]                 Group ID: 1409
[field #2]              Open/Closed: Open
[field #3]                 Severity: 3 - Normal
[field #4]                  Privacy: Public
[field #6]                 Category: None
[field #7]             Submitted by: jave
[field #8]              Assigned to: None
[field #9]             Submitted on: Mi 15.03.2006 um 21:55
[field #10]                  Summary: filetype recognition with libmagic
[field #11]      Original Submission: It would be desirable to know the real filetype of a downloading file as soon as possible. The filetype given by libmagic can then be compared with the file extension. If they differ, say a "rar" is reaaly an "avi", the file is probably a fake.

Libmagic, which is used in the unix "file" command can be used for this.

There didnt seem to be any mainatined libmagic wrappers for ocaml, so I tried generating them with swig. It worked fairly well.

See attached file with the experiment.

This should of course be a configure-time option.


[field #13]               Item Group: None
[field #14]                   Status: None
[field #15]        Component Version: None
[field #16]         Platform Version: None
[field #17]          Reproducibility: None
[field #18]               Size (loc): None
[field #19]            Fixed Release: None
[field #20]          Planned Release: None
[field #21]                   Effort: 0.00
[field #25]                 Priority: 1 - Later
[field #26]          Should Start On: Mi 15.03.2006 um 01:00
[field #27]    Should be Finished on: Mi 15.03.2006 um 01:00
[field #28]         Percent Complete: 0%
[field #30]                  Release: None
[field #55]     Custom Select Box #1: None
[field #56]     Custom Select Box #2: None
[field #57]     Custom Select Box #3: None
[field #58]     Custom Select Box #4: None
[field #59]     Custom Select Box #5: None
[field #60]     Custom Select Box #6: None
[field #61]     Custom Select Box #7: None
[field #62]     Custom Select Box #8: None
[field #63]     Custom Select Box #9: None
[field #64]    Custom Select Box #10: None

spiralvoice <spiralvoice>
Group administrator
Thu 16 Mar 2006 10:17:40 PM UTC, comment #5: 

Release version of the patch, GUI stuff has to be implemented.
libmagic, part of GNU file (http://directory.fsf.org/file.html)
is now an optional dependency of MLDonkey.

new configure option: --disable-magic
new debug command: debug_dir <dir>

to get the filetype computed by libmagic use

Magic.M.magic_fileinfo file

this function returns a string, if libmagic is not used it returns "not implemented"

spiralvoice <spiralvoice>
Group administrator
Thu 16 Mar 2006 12:11:35 PM UTC, comment #4: 

For later usage: libmagic requires (among other stuff) GNU regex
for compiling on MinGW, here is how to do it:
http://www.openldap.org/faq/data/cache/302.html#2
ftp://ftp.gnu.org/gnu/regex

        $ ./configure --prefix=/mingw i686-pc-mingw32
        $ make subdirs= (ignore the error message)
        $ ar ru libregex.a regex.o
        $ cp regex.h /mingw/include
        $ cp libregex.a /mingw/lib

spiralvoice <spiralvoice>
Group administrator
Thu 16 Mar 2006 08:42:02 AM UTC, comment #3: 

The mime type given by libmagic(as seen by "file -i filename") could
be compared with the file extensions to mime-type mapping given by
mailcap, which usually resides in /etc/mime-types.

This file is unfortunately not consistent with the one libmagic
handles. rar files arent identified by mailcap, but they are by libmagic.

Joakim Verona <jave>
Group Member
Wed 15 Mar 2006 10:48:43 PM UTC, comment #2: 

Attached a dirty hack for MLDonkey, no configure checking atm.
But libmagic function should be usable from inside MLDonkey now.

spiralvoice <spiralvoice>
Group administrator
Wed 15 Mar 2006 08:58:21 PM UTC, comment #1: 

heres how to perform the test:

- make a magic.i file like this:
%module ocamlmagic
%{
 /* Includes the header in the wrapper code */
 #include "/usr/include/magic.h"
%}

/* Parse the header file to generate wrappers */
%include "/usr/include/magic.h"

- compile it like so:
swig -ocaml magic.i

we get ocaml interface files and a c wrapper.
- guess it must all be compiled and linked...

swig -ocaml -co swig.mli
swig -ocaml -co swig.ml
ocamlc -c swig.mli
ocamlc -c swig.ml
ocamlc -c -ccopt "-I/usr/include/" magic_wrap.c
ocamlc -c ocamlmagic.mli
ocamlc -c ocamlmagic.ml


- making custom toplevel
ocamlmktop -cclib -lmagic -cclib -lz   -custom magic_wrap.o swig.cmo ocamlmagic.cmo -o magic_top
ticky to get the linking right

- calling the funs
 (maybe not necessary)

./magic_top
#load "ocamlmagic.cmo";;
open Swig;;
open Ocamlmagic;;
let magiccookie = _magic_open  (C_int 0);;(just opens the lib)                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                   
_magic_load (C_list [magiccookie; (C_int 0)]);; ( returns 0 on ok)                                                                                                                                                                                                               
let fileinfo= _magic_file (C_list [magiccookie; (C_string "/tmp/142328-5.pdf")]);;                                                                                                                                                                                                 
( returns val fileinfo : Ocamlmagic.c_obj = C_string "PDF document, version 1.2" )                                                                                                                                                                                               
_magic_close magiccookie;; (close the lib)


Joakim Verona <jave>
Group Member
Thu 16 Mar 2006 10:18:01 PM UTC, original submission:  

It would be desirable to know the real filetype of a downloading file as soon as possible. The filetype given by libmagic can then be compared with the file extension. If they differ, say a "rar" is reaaly an "avi", the file is probably a fake.

Libmagic, which is used in the unix "file" command can be used for this.

There didnt seem to be any mainatined libmagic wrappers for ocaml, so I tried generating them with swig. It worked fairly well.

See attached file with the experiment.

This should of course be a configure-time option.


Joakim Verona <jave>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #9951:  4976.patch added by spiralvoice (59KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

 

Follow 25 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2006-05-12 spiralvoice StatusReady For Test Done
    Open/ClosedOpen Closed
2006-05-12 spiralvoice Attached File#9671 Removed
    Attached File#9636 Removed
2006-05-12 spiralvoice Attached File- Added 4976.patch, #9951
2006-05-12 spiralvoice Attached File#9950 Removed
2006-05-12 spiralvoice Attached File- Added 4976_2.patch, #9950
2006-05-12 spiralvoice Attached File#9692 Removed
2006-05-12 spiralvoice Attached File#9948 Removed
2006-05-12 spiralvoice Attached File- Added 4976_3.patch, #9948
2006-03-29 spiralvoice Attached File- Added 4976.patch, #6083
2006-03-29 spiralvoice Attached File#6080 Removed
2006-03-28 spiralvoice Attached File- Added 4976.patch, #6080
2006-03-28 spiralvoice Attached File#6068 Removed
2006-03-20 spiralvoice Attached File- Added 4976.patch, #6068
2006-03-20 spiralvoice Attached File#6067 Removed
2006-03-20 spiralvoice Attached File- Added 4976_2.patch, #6067
2006-03-20 spiralvoice Attached File#6064 Removed
2006-03-19 spiralvoice Attached File- Added 4976_2.patch, #6064
2006-03-18 jave Attached File- Added magic2.patch, #6060
2006-03-16 spiralvoice CategoryNone Improved interfaces
    Priority1 - Later 5 - Normal
    StatusNone Ready For Test
    SummaryReassigned item: filetype recognition with libmagic Filetype recognition with libmagic
2006-03-16 spiralvoice Reassign itemFrom group mldonkey, a multi-networks file-sharing client, task tracker To group mldonkey, a multi-networks file-sharing client, patch tracker

Back to the top

Powered by Savane 3.13-02a9.
Corresponding source code