bugmonotone - Bugs: bug #26595, put_revision and pull (and likely...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #26595: put_revision and pull (and likely commit, too) scale with tree size instead of changeset size

Submitter:  Patrick Georgi <oxygene>
Submitted:  Sat 16 May 2009 04:40:37 PM UTC
   
 
Category:  performance Severity:  3 - Normal
Item Group:  feature request Status:  Fixed
Privacy:  Public Assigned to:  None
Open/Closed:  Closed
mtn version --full: 

applies at least to 0.36 - 0.44.

Tue 09 Feb 2010 11:45:16 PM UTC, comment #2: 

This particular instance should be fixed in 85496c3f1a001e20bd3318a80acf9ab848ee0bd1 . Only deltas for non-merge revisions can be sped up this way.

Note that there are other unavoidable operations that still need to scale by tree size, like calculating/verifying the manifest_id of a revision or writing out the full roster text for the head revision to the db.

Timothy Brownawell <tbrownaw>
Group Member
Tue 26 Jan 2010 08:54:39 PM UTC, comment #1: 

Some more information:
The code in question resides in roster_delta.cc, make_roster_delta_t(..)

There, two rosters (old and new) are scanned for differences, which are then recorded in the roster_delta.
Instead, it could (optionally) take a revision as additional operand, and record the differences between both rosters for the files that appear in the revision (which already is a delta-based data structure).

The main issue for me was to make sure that the callers of make_roster_delta_t have access to the right revision.

Patrick Georgi <oxygene>
Sat 16 May 2009 04:40:37 PM UTC, original submission:  

The scalability problem comes from roster handling.

When a revision comes in, monotone creates a new roster out of the ancestor and the revision. After that, it applies a diff algorithm to both rosters to get a delta roster. Even if only 1 file changed, it has to compare all entries in the roster.

As an alternative, the roster delta could be created when the new roster is created. That's probably the least intrusive change to speed things up.

I disabled the delta roster creation for a test and reduced the runtime to convert 10 revisions of the opensolaris hg repo to mtn from 2:30 minutes to 1:20 (it used automate put_revision), and also got a significant speed up with pull, but the numbers aren't that easy to interpret there.
Of course, such a change increases the repository size a lot, so a more clever algorithm (eg. the one above) is necessary to fix this issue.

Patrick Georgi <oxygene>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tbrownaw (Posted a comment)
  • -email is unavailable- added by oxygene (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-02-09 tbrownaw StatusNone Fixed
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code