bugmonotone - Bugs: bug #13570, 'db load' does not work on large...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #13570: 'db load' does not work on large dump files

Submitted by:  Nathaniel Smith <njs>
Submitted on:  Wed 29 Jun 2005 06:40:05 AM UTC  
 
Category: storage systemSeverity: 1 - Wish
Item Group: crashStatus: None
Privacy: PublicAssigned to: None
Open/Closed: Closed
mtn version --full: 

Fri 25 Nov 2005 05:54:14 PM UTC, comment #2:

Fixed by :

2005-07-03 Matthew Gregan <kinetik@orcon.net.nz>

* sqlite/{sqlite3.h,tokenize.c} (sqlite3_complete_last): New
function to find the last valid SQL statement in a string; based
on sqlite3_complete. This change should be offered upstream, but
probably not before sqlite3_complete_last16 is implemented.
fixes bug 13570.

Grahame Bowland <grahame>
Project Member
Sun 03 Jul 2005 05:25:45 AM UTC, comment #1:

A fix for this bug is available in 8133bcaa6e1f80c62c04dc08d854fbb2c9f324ac.

Matthew Gregan <kinetik>
Project Member
Wed 29 Jun 2005 06:40:05 AM UTC, original submission:

'db load' reads the entire dumpfile into a string in memory. if the dumpfile is large (discoverer had a dump of a 1.2 gig database), then at some point std::string just gives up.

so we need to split the string into pieces, and execute each one separately. this is a little tricky, because we have to split it on statement boundaries, sqlite will not be happy with partial statements.

the best idea that comes to mind is to have 'db dump' put in little comments that say 'after n lines (or bytes?) there is a boundary between statements, and another comment like this one'. And then 'db load' can use these comments to split it into statement that can be fed into sqlite.

this makes the output really rigid, though, which is bad, if we want to tell people they can always dump out their file and play with it if they're unhappy. OTOH, almost anythign can occur inside a dump file, since it contains raw db data, so it's hard to scan for a magic string. I guess we could encode every piece of data somehow, like have dumps that look like 'insert into <...> values (unbase64(...), unbase64(...)', and then we have some guarantee that the string "####" won't occur anywhere except in magic comments...

it would be really nice if sqlite had a command that took a string, and executed the first complete statement occurring in it, and tells you where the statement ended; or else, if there is no complete statement, tells you that. That would be perfect.

I'd rather not write a sql parser myself, though.

Nathaniel Smith <njs>
Project Member

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can click here to add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

Please enter the title of George Orwell's famous dystopian book (it's a date):

 

 

Follows 1 latest change.

Date Changed By Updated Field Previous Value => Replaced By
Fri 25 Nov 2005 05:54:14 PM UTCgrahameOpen/ClosedOpen=>Closed

Back to the top


Powered by Savane 3.1-cleanup1