bugrdiff-backup - Bugs: bug #20570, unrecoverable failure over network

 
 

bug #20570: unrecoverable failure over network

Submitted by:  m read <mstuff>
Submitted on:  Tue 24 Jul 2007 10:43:24 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Andrew Ferguson <owsla>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 09 Aug 2007 01:39:14 PM UTC, comment #15:

Morgan,

Ok, BUY/BU1 is having network connectivity issues or a write error to disk. The trace shows that it is not possibly due to an rdiff-backup bug (see the end of rdiff-backup1_1Y.out.bz2).

Note that BUY/BU1 seems to have never finished a first backup (look at the contents of the rdiff-backup-data dir), thus you can safely delete the rdiff-backup-data dir for that mirror (like the traceback says to do) and try and pick-up from there.

BUX/BU2, on the other hand, was hitting a bug in rdiff-backup. I have fixed it in CVS. Once version 1.1.13 is released in the next week or so, your systems will be able to properly regress from a failed backup session.

I am closing this bug report since the bug has been identified and fix. Please e-mail me with any follow-up questions.

Thanks,
Andrew

Andrew Ferguson <owsla>
Project AdministratorIn charge of this item.
Thu 09 Aug 2007 08:05:50 AM UTC, comment #14:

OK, here goes. For BU2 read BUX; for BU1 read BUY (how much, no, never mind - couldn't resist:(

For /home/morgan/.rdiff-backup_include-list see attachment three to this post.

BUX
1

sudo\
rdiff-backup\
--verbosity 8\
--print-statistics\
--include-globbing-filelist /home/morgan/.rdiff-backup_include-list\
/\
root@MOZSMACHINE::/cygdrive/c/BU2\
> rdiff-backup2_1.out 2>&1

2

root@MOZSMACHINE /cygdrive/c
$ ls -l /cygdrive/c/BUX/rdiff-backup-data/
total 23559
-rw------- 1 root None 21616875 Aug 7 20:06 backup.log
-rw------- 1 root None 18 Aug 1 19:41 chars_to_quote
-rw------- 1 root None 9 Aug 3 23:45 current_mirror.2007-08-03;08423;05848;05802+12;05800.data
-rw------- 1 root None 9 Aug 4 23:05 current_mirror.2007-08-04;08423;05807;05841+12;05800.data
-rw------- 1 root None 259 Aug 1 22:46 error_log.2007-08-01;08419;05844;05826+12;05800.data.gz
-rw------- 1 root None 360 Aug 2 22:25 error_log.2007-08-02;08422;05817;05821+12;05800.data.gz
-rw------- 1 root None 334 Aug 3 23:50 error_log.2007-08-03;08423;05848;05802+12;05800.data.gz
-rw------- 1 root None 97 Aug 4 23:11 error_log.2007-08-04;08423;05807;05841+12;05800.data.gz
-rw------- 1 root None 248297 Aug 1 22:46 file_statistics.2007-08-01;08419;05844;05826+12;05800.data.gz
-rw------- 1 root None 278513 Aug 2 22:25 file_statistics.2007-08-02;08422;05817;05821+12;05800.data.gz
-rw------- 1 root None 278580 Aug 3 23:50 file_statistics.2007-08-03;08423;05848;05802+12;05800.data.gz
-rw------- 1 root None 246247 Aug 4 23:11 file_statistics.2007-08-04;08423;05807;05841+12;05800.data.gz
drwx------+ 5 root None 0 Aug 4 23:11 increments
-rwxr-xr-x 1 root None 0 Aug 1 16:35 increments.2007-08-01;08419;05844;05826+12;05800.dir
-rwxr-xr-x 1 root None 0 Aug 2 13:27 increments.2007-08-02;08422;05817;05821+12;05800.dir
-rwxr-xr-x 1 root None 0 Aug 3 16:44 increments.2007-08-03;08423;05848;05802+12;05800.dir
drwx------+ 2 root None 0 Aug 4 23:11 long_filename_data
-rw------- 1 root None 22479 Aug 2 22:25 mirror_metadata.2007-08-01;08419;05844;05826+12;05800.diff.gz
-rw------- 1 root None 21908 Aug 3 23:50 mirror_metadata.2007-08-02;08422;05817;05821+12;05800.diff.gz
-rw------- 1 root None 734692 Aug 3 23:50 mirror_metadata.2007-08-03;08423;05848;05802+12;05800.snapshot.gz
-rw------- 1 root None 638285 Aug 4 23:11 mirror_metadata.2007-08-04;08423;05807;05841+12;05800.snapshot.gz
-rw------- 1 root None 520 Aug 1 22:46 session_statistics.2007-08-01;08419;05844;05826+12;05800.data
-rw------- 1 root None 548 Aug 2 22:25 session_statistics.2007-08-02;08422;05817;05821+12;05800.data
-rw------- 1 root None 550 Aug 3 23:50 session_statistics.2007-08-03;08423;05848;05802+12;05800.data

root@MOZSMACHINE /cygdrive/c
$

3

GNU nano 1.2.2 File: ...rive/c/BUX/rdiff-backup-data/chars_to_quote

A-Z^@-^_"*/:<>?\\|^?;

See first attachment to this post

BUY
1

sudo\
rdiff-backup\
--verbosity 8\
--print-statistics\
--include-globbing-filelist /home/morgan/.rdiff-backup_include-list\
/\
root@MOZSMACHINE::/cygdrive/c/BU1\
> rdiff-backup1_1.out 2>&1

2

root@MOZSMACHINE /cygdrive/c
$ ls -l /cygdrive/c/BUY/rdiff-backup-data/
total 10430
-rw------- 1 root None 10082472 Aug 9 15:27 backup.log
-rw------- 1 root None 18 Aug 9 13:14 chars_to_quote
-rw------- 1 root None 97 Aug 9 15:27 error_log.2007-08-09;08413;05818;05809+12;05800.data.gz
-rw------- 1 root None 156767 Aug 9 15:27 file_statistics.2007-08-09;08413;05818;05809+12;05800.data.gz
drwx------+ 4 root None 0 Aug 9 13:17 increments
drwx------+ 2 root None 0 Aug 9 14:45 long_filename_data
-rw------- 1 root None 432707 Aug 9 15:27 mirror_metadata.2007-08-09;08413;05818;05809+12;05800.snapshot.gz

root@MOZSMACHINE /cygdrive/c
$

3

GNU nano 1.2.2 File: ...rive/c/BUY/rdiff-backup-data/chars_to_quote

A-Z^@-^_"*/:<>?\\|^?;

See second attachment to this post

Thanks,
M.

(file #13629, file #13630, file #13631)

m read <mstuff>
Thu 09 Aug 2007 07:40:11 AM UTC, comment #13:

Here's another couple, as per comment #11
M.

(file #13627, file #13628)

m read <mstuff>
Wed 08 Aug 2007 04:13:19 AM UTC, comment #12:

Morgan,

Great tracebacks. This is definitely something we can now sink our teeth into.

Can you either post or e-mail me three things:

1) The command line you are using to run rdiff-backup
2) The result of:
$ ls -l /cygdrive/c/BU2/rdiff-backup-data/
3) The contents of /cygdrive/c/BU2/rdiff-backup-data/chars_to_quote

Thanks! Armed with that info, I think I'll be able to unravel this.

Andrew

Andrew Ferguson <owsla>
Project AdministratorIn charge of this item.
Wed 08 Aug 2007 01:55:02 AM UTC, comment #11:

Attached two files following backup failure. rdiff-backup2_1X.out is the output and error (> rdiff-backup.out 2>&1) with -v 8 of the backup failure. rdiff-backup2_1XX.out is the subsequent rdiff-backup failure to backup to the directory which failed at rdiff-backup2_1X.out.

Now running alternative backups so when one fails, there's an alternative to fall back on, not an ideal solution but at least it's safe.

(file #13616, file #13617)

m read <mstuff>
Mon 30 Jul 2007 11:46:47 PM UTC, comment #10:

Oh, #9 is mine,
Morgan.

m read <mstuff>
Mon 30 Jul 2007 11:43:36 PM UTC, comment #9:

"devil of a problem", I'll say - I'm lost for words! Attached two rdiff-backup.out files as requested, both times rdiff-backup ran perfectly...

I'll continue running and saving the output with verbosity at 8 and see what happens - though obviously the transfers wont stress anything as they'll be quite small.

(file #13531, file #13532)

Anonymous
Sun 29 Jul 2007 10:08:08 PM UTC, comment #8:

Morgan,

Indeed, this is being a devil of a problem. Firstly, can you run with verbosity set to 8? At that level, it will not trap even "benign" tracebacks, as it is doing in this case.

Secondly, have you been redirecting the output using something like "rdiff-backup ... > rdiff-backup.out" ? If so, you should redirect STDERR as well, by doing "rdiff-backup ... > rdiff-backup.out 2>&1"

Andrew

Andrew Ferguson <owsla>
Project AdministratorIn charge of this item.
Sun 29 Jul 2007 09:32:32 PM UTC, comment #7:

Here's the out put from a successful copy using secure copy (scp), so the failure doesn't seem to be the network and doesn't seem to be ssh so, to my mind, that only leaves rdiff-backup.

Either rdiff-backup is failing, or it's not providing sufficient diagnostic information to point to where the failure is - which should be a bug in itself.

Using rdiff-backup over cifs wont work for me because I'm using cygwin to provide the network access to user accounts on the win box.

[morgan@morgansmachine ~]$ scp ~/Public/Fedora-7-i386/F-7-i386-DVD.iso root@mozsmachine:F-7-i386-DVD.iso root@mozsmachine's password:
F-7-i386-DVD.iso 100% 2766MB 317.0KB/s 2:28:56
[morgan@morgansmachine ~]$ stat ~/Public/Fedora-7-i386/F-7-i386-DVD.iso
File: `/home/morgan/Public/Fedora-7-i386/F-7-i386-DVD.iso'
Size: 2900602880 Blocks: 5670792 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 4255721 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 500/ morgan) Gid: ( 500/ morgan)
Access: 2007-07-30 02:13:03.000000000 +1200
Modify: 2007-06-03 00:30:08.000000000 +1200
Change: 2007-06-03 00:30:08.000000000 +1200
You have new mail in /var/spool/mail/morgan
[morgan@morgansmachine ~]$ ssh root@mozsmachine
root@mozsmachine's password:
Last login: Sun Jul 29 23:36:36 2007 from morgansmachine.lan

root@MOZSMACHINE ~
$ stat F-7-i386-DVD.iso
File: `F-7-i386-DVD.iso'
Size: 2900602880 Blocks: 2832620 IO Block: 1024 regular file
Device: c466326h/205939494d Inode: 1688849860303133 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 1008/ root) Gid: ( 513/ None)
Access: 2007-07-30 02:10:08.093750000 +1200
Modify: 2007-07-30 02:10:08.093750000 +1200
Change: 2007-07-30 02:10:08.093750000 +1200

root@MOZSMACHINE ~
$

m read <mstuff>
Sun 29 Jul 2007 07:01:39 PM UTC, comment #6:

Morgan,

The are two reasons why I believe that it is a network failure and not rdiff-backup's fault:

- Both "rdiff-backup_failure_output.txt" and "rdiff-backup_failure_output(2).txt" end with:

Read from remote host mozsmachine: Connection reset by peer
Fatal Error: Lost connection to the remote system

and no exception traceback from rdiff-backup. If rdiff-backup had failed, it would have printed a traceback or otherwise informative error message

- Secondly, the file error_log.2007-07-23T22:59:25+12:00.data.gz is empty and compressed. This only occurs under two conditions: the backup failed two complete AND the backup failed due to a cause outside rdiff-backup

I don't know why a copy using CIFS would work while rdiff-backup running over SSH would fail.

If you download the rdiff-backup source from CVS or wait until 1.1.13 (probably within the next week), you should be able to backup to a directory mounted over CIFS.

BTW, thanks for the earlier CIFS error report. That was most helpful.

Best,
Andrew

Andrew Ferguson <owsla>
Project AdministratorIn charge of this item.
Sun 29 Jul 2007 08:56:20 AM UTC, comment #5:

Never had problems with the network - regularly to transfers of large files - dvd's & mondoarchive - without problem. Attached is the output of a quick(ish) test; copy works fine.

Are there not any further diagnostics to run of rdiff-backup to narrow the field a little? Would gdb help or valgrind perhaps?

You say I've had a network failure that I've recovered from, how so?

Why would rdiff-backup fail where copy works?

M.

(file #13514)

m read <mstuff>
Fri 27 Jul 2007 08:45:27 PM UTC, comment #4:

Morgan,

You should be able to recover from network failures now (and it looks like you have). However, you are still having a network failure, and no message suggests that it is coming from rdiff-backup.

Is your network connection or remote machine unreliable? Can you transfer large files between them without rdiff-backup. This no longer looks like an rdiff-backup problem.

Andrew

Andrew Ferguson <owsla>
Project AdministratorIn charge of this item.
Fri 27 Jul 2007 08:34:11 PM UTC, comment #3:

root@MOZSMACHINE ~
$ rdiff-backup -V rdiff-backup 1.1.12

root@MOZSMACHINE ~
$

No change I'm afraid - still fails (no error_log produced this time). Any thing more I can do to help just ask?

Thanks,
Morgan.

(file #13500, file #13501)

m read <mstuff>
Fri 27 Jul 2007 04:33:43 AM UTC, comment #2:

root@MOZSMACHINE ~
$ rdiff-backup -V
rdiff-backup 1.1.11

root@MOZSMACHINE ~
$

Thought I had the latest, will update and check again.
Thanks,
Morgan.

m read <mstuff>
Thu 26 Jul 2007 02:25:56 PM UTC, comment #1:

Hi,

I see you are running rdiff-backup on Cygwin. What version of rdiff-backup are you running?

A number of important Cygwin fixes were implemented in 1.1.12, including one for a bug that is similar (and perhaps identical) to the one you experienced.

Thanks,
Andrew

Andrew Ferguson <owsla>
Project AdministratorIn charge of this item.
Tue 24 Jul 2007 10:43:24 AM UTC, original submission:

rdiff-backup fails, further attempts to run rdiff-backup do not cause failed directory to be undone (see --check-destination-dir).

m read <mstuff>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #13629:  chars_to_quote added by mstuff (18B - application/octet-stream)
file #13630:  chars_to_quote added by mstuff (18B - application/octet-stream)
file #13631:  .rdiff-backup_include-list added by mstuff (361B - application/octet-stream)
file #13627:  rdiff-backup1_1Y.out.bz2 added by mstuff (437KiB - application/x-bzip)
file #13628:  rdiff-backup1_1YY.out added by mstuff (5KiB - application/octet-stream)
file #13617:  rdiff-backup2_1XX.out added by mstuff (8KiB - application/octet-stream)
file #13616:  rdiff-backup2_1X.out added by mstuff (8KiB - application/octet-stream)
file #13531:  rdiff-backup1.out.tar.bz2 added by None (182KiB - application/x-bzip - bz2 archives of successful rdiff-bacup outputs (-v 8))
file #13532:  rdiff-backup2.out.tar.bz2 added by None (182KiB - application/x-bzip - bz2 archives of successful rdiff-bacup outputs (-v 8))
file #13514:  BigFileNetworkTest.txt added by mstuff (3KiB - text/plain - Jolly big file transfer test)
file #13500:  backup(2).log added by mstuff (64KiB - text/x-log - verbosity at 5)
file #13501:  rdiff-backup_failure_output(2).txt added by mstuff (66KiB - text/plain - verbosity at 5)
file #13452:  backup.log added by mstuff (456KiB - text/x-log)
file #13449:  rdiff-backup_failure_output.txt added by mstuff (458KiB - text/plain - outputs and logs, verbosity at 5)
file #13450:  2nd_failure.txt added by mstuff (3KiB - text/plain - outputs and logs, verbosity at 5)
file #13451:  error_log.2007-07-23T22:59:25+12:00.data.gz added by mstuff (87B - application/x-gzip - outputs and logs, verbosity at 5)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by owsla (Posted a comment)
  • -unavailable- added by mstuff (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 19 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 09 Aug 2007 01:39:14 PM UTCowslaStatusNone=>Fixed
      Open/ClosedOpen=>Closed
    Thu 09 Aug 2007 08:05:50 AM UTCmstuffAttached File-=>Added chars_to_quote, #13629
      Attached File-=>Added chars_to_quote, #13630
      Attached File-=>Added .rdiff-backup_include-list, #13631
    Thu 09 Aug 2007 07:40:11 AM UTCmstuffAttached File-=>Added rdiff-backup1_1Y.out.bz2, #13627
      Attached File-=>Added rdiff-backup1_1YY.out, #13628
    Wed 08 Aug 2007 04:13:19 AM UTCowslaAssigned toNone=>owsla
    Wed 08 Aug 2007 01:55:02 AM UTCmstuffAttached File-=>Added rdiff-backup2_1XX.out, #13617
    Wed 08 Aug 2007 01:55:01 AM UTCmstuffAttached File-=>Added rdiff-backup2_1X.out, #13616
    Mon 30 Jul 2007 11:43:36 PM UTCNoneAttached File-=>Added rdiff-backup1.out.tar.bz2, #13531
      Attached File-=>Added rdiff-backup2.out.tar.bz2, #13532
    Sun 29 Jul 2007 08:56:20 AM UTCmstuffAttached File-=>Added BigFileNetworkTest.txt, #13514
    Fri 27 Jul 2007 08:34:11 PM UTCmstuffAttached File-=>Added backup(2).log, #13500
      Attached File-=>Added rdiff-backup_failure_output(2).txt, #13501
    Tue 24 Jul 2007 10:58:06 AM UTCmstuffAttached File-=>Added backup.log, #13452
    Tue 24 Jul 2007 10:53:22 AM UTCmstuffAttached File-=>Added rdiff-backup_failure_output.txt, #13449
      Attached File-=>Added 2nd_failure.txt, #13450
      Attached File-=>Added error_log.2007-07-23T22:59:25+12:00.data.gz, #13451

    Back to the top


    Powered by Savane 3.1-cleanup1