Wed 23 Nov 2005 07:29:59 PM UTC, original submission:
I have been trying to import a subversion repository into monotone (head is 1.5g / 20k files), to compare the two systems. I created scripts that would do:
1. svn update next revision #
2. monotone ls missing, to drop removed files
3. monotone add * for each top-level folder (to add new files)
4. monotone commit
Steps 1-3 are fast. Step 4 was fast enough until the working folder size grew past the available physical memory in the machine (about 800megs). So I tried to figure out what the slowness was using strace; to import the whole repository in this way would take maybe 2 weeks of constant disk activity, and in any case saying it'll take 3+ minutes to commit a simple edit is absolutely ridiculous.
Turns out that monotone is rehashing every single file, which guarentees that once the files cannot completely fit into the disk cache that commit performance will be ridiculously slow. I tested that it was not an svn-related issue by just adding a new small file by hand and committing it, with the same results.
The obvious solution is to store the last commit/update time in the MT folder and only stat files to see if their modification time indicates they were changed after that time. Make the complete rehashing be a "--force" style option, if the user wants it. The default certainly should not be the extremely slow behavior, or at the very least there should actually be a way to do a commit quickly.
I also found a few minor perforance issues:
A) when the monotone reads from the database it does a read of 1k followed by a _llseek to the current file position, and repeats until the data is read. I couldn't follow the C++ enough to figure out why this is, but it is a significant slowdown (it takes about as long to do a syscall as to copy the 1k of data). Maybe it's needed for some sort of database consistency?
B) data reads to hash the files read in 8191 chunks. this should be 8192 to be in paged-sized chunks. it should really mmap files larger than some size though.
C) It's calling lstat on each file twice. Removing the duplicate would save maybe a tenth of a second on a large working folder.
D) seems to do an "fchmod 0100755" on each file in the working folder, which may not be the permissions the user wants (have to set umask).
|