bugmonotone - Bugs: bug #26595, put_revision and pull (and likely...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #26595: put_revision and pull (and likely commit, too) scale with tree size instead of changeset size

Submitted by:  Patrick Georgi <oxygene>
Submitted on:  Sat 16 May 2009 04:40:37 PM UTC  
 
Category: performanceSeverity: 3 - Normal
Item Group: feature requestStatus: Fixed
Privacy: PublicAssigned to: None
Open/Closed: Closed
mtn version --full: 

applies at least to 0.36 - 0.44.

Tue 09 Feb 2010 11:45:16 PM UTC, comment #2:

This particular instance should be fixed in 85496c3f1a001e20bd3318a80acf9ab848ee0bd1 . Only deltas for non-merge revisions can be sped up this way.

Note that there are other unavoidable operations that still need to scale by tree size, like calculating/verifying the manifest_id of a revision or writing out the full roster text for the head revision to the db.

Timothy Brownawell <tbrownaw>
Project Member
Tue 26 Jan 2010 08:54:39 PM UTC, comment #1:

Some more information:
The code in question resides in roster_delta.cc, make_roster_delta_t(..)

There, two rosters (old and new) are scanned for differences, which are then recorded in the roster_delta.
Instead, it could (optionally) take a revision as additional operand, and record the differences between both rosters for the files that appear in the revision (which already is a delta-based data structure).

The main issue for me was to make sure that the callers of make_roster_delta_t have access to the right revision.

Patrick Georgi <oxygene>
Sat 16 May 2009 04:40:37 PM UTC, original submission:

The scalability problem comes from roster handling.

When a revision comes in, monotone creates a new roster out of the ancestor and the revision. After that, it applies a diff algorithm to both rosters to get a delta roster. Even if only 1 file changed, it has to compare all entries in the roster.

As an alternative, the roster delta could be created when the new roster is created. That's probably the least intrusive change to speed things up.

I disabled the delta roster creation for a test and reduced the runtime to convert 10 revisions of the opensolaris hg repo to mtn from 2:30 minutes to 1:20 (it used automate put_revision), and also got a significant speed up with pull, but the numbers aren't that easy to interpret there.
Of course, such a change increases the repository size a lot, so a more clever algorithm (eg. the one above) is necessary to fix this issue.

Patrick Georgi <oxygene>

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tbrownaw (Posted a comment)
  • -unavailable- added by oxygene (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 09 Feb 2010 11:45:16 PM UTCtbrownawStatusNone=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1