bugrdiff-backup - Bugs: bug #20570, unrecoverable failure over network

 
 

bug #20570: unrecoverable failure over network

Submitter:  m read <mstuff>
Submitted:  Tue 24 Jul 2007 10:43:24 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  owsla
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 09 Aug 2007 01:39:14 PM UTC, comment #15: 

Morgan,

Ok, BUY/BU1 is having network connectivity issues or a write error to disk. The trace shows that it is not possibly due to an rdiff-backup bug (see the end of rdiff-backup1_1Y.out.bz2).

Note that BUY/BU1 seems to have never finished a first backup (look at the contents of the rdiff-backup-data dir), thus you can safely delete the rdiff-backup-data dir for that mirror (like the traceback says to do) and try and pick-up from there.

BUX/BU2, on the other hand, was hitting a bug in rdiff-backup. I have fixed it in CVS. Once version 1.1.13 is released in the next week or so, your systems will be able to properly regress from a failed backup session.

I am closing this bug report since the bug has been identified and fix. Please e-mail me with any follow-up questions.

Thanks,
Andrew

Andrew Ferguson <owsla>
Group administrator
Thu 09 Aug 2007 08:05:50 AM UTC, comment #14: 

OK, here goes.  For BU2 read BUX; for BU1 read BUY (how much, no, never mind - couldn't resist:(

For /home/morgan/.rdiff-backup_include-list see attachment three to this post.

BUX
1

sudo\
 rdiff-backup\
 --verbosity 8\
 --print-statistics\
 --include-globbing-filelist /home/morgan/.rdiff-backup_include-list\
 /\
 root@MOZSMACHINE::/cygdrive/c/BU2\
 > rdiff-backup2_1.out 2>&1

2

root@MOZSMACHINE /cygdrive/c
$ ls -l /cygdrive/c/BUX/rdiff-backup-data/
total 23559
-rw-------  1 root None 21616875 Aug  7 20:06 backup.log
-rw-------  1 root None       18 Aug  1 19:41 chars_to_quote
-rw-------  1 root None        9 Aug  3 23:45 current_mirror.2007-08-03;08423;05848;05802+12;05800.data
-rw-------  1 root None        9 Aug  4 23:05 current_mirror.2007-08-04;08423;05807;05841+12;05800.data
-rw-------  1 root None      259 Aug  1 22:46 error_log.2007-08-01;08419;05844;05826+12;05800.data.gz
-rw-------  1 root None      360 Aug  2 22:25 error_log.2007-08-02;08422;05817;05821+12;05800.data.gz
-rw-------  1 root None      334 Aug  3 23:50 error_log.2007-08-03;08423;05848;05802+12;05800.data.gz
-rw-------  1 root None       97 Aug  4 23:11 error_log.2007-08-04;08423;05807;05841+12;05800.data.gz
-rw-------  1 root None   248297 Aug  1 22:46 file_statistics.2007-08-01;08419;05844;05826+12;05800.data.gz
-rw-------  1 root None   278513 Aug  2 22:25 file_statistics.2007-08-02;08422;05817;05821+12;05800.data.gz
-rw-------  1 root None   278580 Aug  3 23:50 file_statistics.2007-08-03;08423;05848;05802+12;05800.data.gz
-rw-------  1 root None   246247 Aug  4 23:11 file_statistics.2007-08-04;08423;05807;05841+12;05800.data.gz
drwx------+ 5 root None        0 Aug  4 23:11 increments
-rwxr-xr-x  1 root None        0 Aug  1 16:35 increments.2007-08-01;08419;05844;05826+12;05800.dir
-rwxr-xr-x  1 root None        0 Aug  2 13:27 increments.2007-08-02;08422;05817;05821+12;05800.dir
-rwxr-xr-x  1 root None        0 Aug  3 16:44 increments.2007-08-03;08423;05848;05802+12;05800.dir
drwx------+ 2 root None        0 Aug  4 23:11 long_filename_data
-rw-------  1 root None    22479 Aug  2 22:25 mirror_metadata.2007-08-01;08419;05844;05826+12;05800.diff.gz
-rw-------  1 root None    21908 Aug  3 23:50 mirror_metadata.2007-08-02;08422;05817;05821+12;05800.diff.gz
-rw-------  1 root None   734692 Aug  3 23:50 mirror_metadata.2007-08-03;08423;05848;05802+12;05800.snapshot.gz
-rw-------  1 root None   638285 Aug  4 23:11 mirror_metadata.2007-08-04;08423;05807;05841+12;05800.snapshot.gz
-rw-------  1 root None      520 Aug  1 22:46 session_statistics.2007-08-01;08419;05844;05826+12;05800.data
-rw-------  1 root None      548 Aug  2 22:25 session_statistics.2007-08-02;08422;05817;05821+12;05800.data
-rw-------  1 root None      550 Aug  3 23:50 session_statistics.2007-08-03;08423;05848;05802+12;05800.data

root@MOZSMACHINE /cygdrive/c
$

3

  GNU nano 1.2.2  File: ...rive/c/BUX/rdiff-backup-data/chars_to_quote

A-Z^@-^_"*/:<>?\\|^?;

See first attachment to this post

BUY
1

sudo\
 rdiff-backup\
 --verbosity 8\
 --print-statistics\
 --include-globbing-filelist /home/morgan/.rdiff-backup_include-list\
 /\
 root@MOZSMACHINE::/cygdrive/c/BU1\
 > rdiff-backup1_1.out 2>&1

2

root@MOZSMACHINE /cygdrive/c
$ ls -l /cygdrive/c/BUY/rdiff-backup-data/
total 10430
-rw-------  1 root None 10082472 Aug  9 15:27 backup.log
-rw-------  1 root None       18 Aug  9 13:14 chars_to_quote
-rw-------  1 root None       97 Aug  9 15:27 error_log.2007-08-09;08413;05818;05809+12;05800.data.gz
-rw-------  1 root None   156767 Aug  9 15:27 file_statistics.2007-08-09;08413;05818;05809+12;05800.data.gz
drwx------+ 4 root None        0 Aug  9 13:17 increments
drwx------+ 2 root None        0 Aug  9 14:45 long_filename_data
-rw-------  1 root None   432707 Aug  9 15:27 mirror_metadata.2007-08-09;08413;05818;05809+12;05800.snapshot.gz

root@MOZSMACHINE /cygdrive/c
$

3

  GNU nano 1.2.2  File: ...rive/c/BUY/rdiff-backup-data/chars_to_quote

A-Z^@-^_"*/:<>?\\|^?;

See second attachment to this post


Thanks,
M.

(file #13629, file #13630, file #13631)

m read <mstuff>
Thu 09 Aug 2007 07:40:11 AM UTC, comment #13: 

Here's another couple, as per comment #11
M.

(file #13627, file #13628)

m read <mstuff>
Wed 08 Aug 2007 04:13:19 AM UTC, comment #12: 

Morgan,

Great tracebacks. This is definitely something we can now sink our teeth into.

Can you either post or e-mail me three things:

1) The command line you are using to run rdiff-backup
2) The result of:
$ ls -l /cygdrive/c/BU2/rdiff-backup-data/
3) The contents of /cygdrive/c/BU2/rdiff-backup-data/chars_to_quote

Thanks! Armed with that info, I think I'll be able to unravel this.

Andrew

Andrew Ferguson <owsla>
Group administrator
Wed 08 Aug 2007 01:55:02 AM UTC, comment #11: 

Attached two files following backup failure.  rdiff-backup2_1X.out is the output and error (> rdiff-backup.out 2>&1) with -v 8 of the backup failure.  rdiff-backup2_1XX.out is the subsequent rdiff-backup failure to backup to the directory which failed at rdiff-backup2_1X.out.

Now running alternative backups so when one fails, there's an alternative to fall back on, not an ideal solution but at least it's safe.

(file #13616, file #13617)

m read <mstuff>
Mon 30 Jul 2007 11:46:47 PM UTC, comment #10: 

Oh, #9 is mine,
Morgan.

m read <mstuff>
Mon 30 Jul 2007 11:43:36 PM UTC, comment #9: 

"devil of a problem", I'll say - I'm lost for words!  Attached two rdiff-backup.out files as requested, both times rdiff-backup ran perfectly...

I'll continue running and saving the output with verbosity at 8 and see what happens - though obviously the transfers wont stress anything as they'll be quite small.

(file #13531, file #13532)

Anonymous
Sun 29 Jul 2007 10:08:08 PM UTC, comment #8: 

Morgan,

Indeed, this is being a devil of a problem. Firstly, can you run with verbosity set to 8? At that level, it will not trap even "benign" tracebacks, as it is doing in this case.

Secondly, have you been redirecting the output using something like "rdiff-backup ... > rdiff-backup.out" ? If so, you should redirect STDERR as well, by doing "rdiff-backup ... > rdiff-backup.out 2>&1"

Andrew

Andrew Ferguson <owsla>
Group administrator
Sun 29 Jul 2007 09:32:32 PM UTC, comment #7: 

Here's the out put from a successful copy using secure copy (scp), so the failure doesn't seem to be the network and doesn't seem to be ssh so, to my mind, that only leaves rdiff-backup.

Either rdiff-backup is failing, or it's not providing sufficient diagnostic information to point to where the failure is - which should be a bug in itself.

Using rdiff-backup over cifs wont work for me because I'm using cygwin to provide the network access to user accounts on the win box.

[morgan@morgansmachine ~]$ scp ~/Public/Fedora-7-i386/F-7-i386-DVD.iso root@mozsmachine:F-7-i386-DVD.iso root@mozsmachine's password:
F-7-i386-DVD.iso                              100% 2766MB 317.0KB/s 2:28:56
[morgan@morgansmachine ~]$ stat ~/Public/Fedora-7-i386/F-7-i386-DVD.iso
  File: `/home/morgan/Public/Fedora-7-i386/F-7-i386-DVD.iso'
  Size: 2900602880      Blocks: 5670792    IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 4255721     Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  500/  morgan)   Gid: (  500/  morgan)
Access: 2007-07-30 02:13:03.000000000 +1200
Modify: 2007-06-03 00:30:08.000000000 +1200
Change: 2007-06-03 00:30:08.000000000 +1200
You have new mail in /var/spool/mail/morgan
[morgan@morgansmachine ~]$ ssh root@mozsmachine
root@mozsmachine's password:
Last login: Sun Jul 29 23:36:36 2007 from morgansmachine.lan

root@MOZSMACHINE ~
$ stat F-7-i386-DVD.iso
  File: `F-7-i386-DVD.iso'
  Size: 2900602880      Blocks: 2832620    IO Block: 1024   regular file
Device: c466326h/205939494d     Inode: 1688849860303133  Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1008/    root)   Gid: (  513/    None)
Access: 2007-07-30 02:10:08.093750000 +1200
Modify: 2007-07-30 02:10:08.093750000 +1200
Change: 2007-07-30 02:10:08.093750000 +1200

root@MOZSMACHINE ~
$

m read <mstuff>
Sun 29 Jul 2007 07:01:39 PM UTC, comment #6: 

Morgan,

The are two reasons why I believe that it is a network failure and not rdiff-backup's fault:

- Both "rdiff-backup_failure_output.txt" and "rdiff-backup_failure_output(2).txt" end with:

Read from remote host mozsmachine: Connection reset by peer
Fatal Error: Lost connection to the remote system

and no exception traceback from rdiff-backup. If rdiff-backup had failed, it would have printed a traceback or otherwise informative error message

- Secondly, the file error_log.2007-07-23T22:59:25+12:00.data.gz is empty and compressed. This only occurs under two conditions: the backup failed two complete AND the backup failed due to a cause outside rdiff-backup

I don't know why a copy using CIFS would work while rdiff-backup running over SSH would fail.

If you download the rdiff-backup source from CVS or wait until 1.1.13 (probably within the next week), you should be able to backup to a directory mounted over CIFS.

BTW, thanks for the earlier CIFS error report. That was most helpful.

Best,
Andrew

Andrew Ferguson <owsla>
Group administrator
Sun 29 Jul 2007 08:56:20 AM UTC, comment #5: 

Never had problems with the network - regularly to transfers of large files - dvd's & mondoarchive - without problem.  Attached is the output of a quick(ish) test; copy works fine.

Are there not any further diagnostics to run of rdiff-backup to narrow the field a little?  Would gdb help or valgrind perhaps?

You say I've had a network failure that I've recovered from, how so?

Why would rdiff-backup fail where copy works?

M.

(file #13514)

m read <mstuff>
Fri 27 Jul 2007 08:45:27 PM UTC, comment #4: 

Morgan,

You should be able to recover from network failures now (and it looks like you have). However, you are still having a network failure, and no message suggests that it is coming from rdiff-backup.

Is your network connection or remote machine unreliable? Can you transfer large files between them without rdiff-backup. This no longer looks like an rdiff-backup problem.

Andrew

Andrew Ferguson <owsla>
Group administrator
Fri 27 Jul 2007 08:34:11 PM UTC, comment #3: 

root@MOZSMACHINE ~
$ rdiff-backup -V rdiff-backup 1.1.12

root@MOZSMACHINE ~
$

No change I'm afraid - still fails (no error_log produced this time).  Any thing more I can do to help just ask?

Thanks,
Morgan.

(file #13500, file #13501)

m read <mstuff>
Fri 27 Jul 2007 04:33:43 AM UTC, comment #2: 

root@MOZSMACHINE ~
$ rdiff-backup -V
rdiff-backup 1.1.11

root@MOZSMACHINE ~
$

Thought I had the latest, will update and check again.
Thanks,
Morgan.

m read <mstuff>
Thu 26 Jul 2007 02:25:56 PM UTC, comment #1: 

Hi,

I see you are running rdiff-backup on Cygwin. What version of rdiff-backup are you running?

A number of important Cygwin fixes were implemented in 1.1.12, including one for a bug that is similar (and perhaps identical) to the one you experienced.

Thanks,
Andrew

Andrew Ferguson <owsla>
Group administrator
Tue 24 Jul 2007 10:43:24 AM UTC, original submission:  

rdiff-backup fails, further attempts to run rdiff-backup do not cause failed directory to be undone (see --check-destination-dir).

m read <mstuff>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #13629:  chars_to_quote added by mstuff (18B - application/octet-stream)
file #13630:  chars_to_quote added by mstuff (18B - application/octet-stream)
file #13631:  .rdiff-backup_include-list added by mstuff (361B - application/octet-stream)
file #13627:  rdiff-backup1_1Y.out.bz2 added by mstuff (437KiB - application/x-bzip)
file #13628:  rdiff-backup1_1YY.out added by mstuff (5KiB - application/octet-stream)
file #13617:  rdiff-backup2_1XX.out added by mstuff (8KiB - application/octet-stream)
file #13616:  rdiff-backup2_1X.out added by mstuff (8KiB - application/octet-stream)
file #13531:  rdiff-backup1.out.tar.bz2 added by None (182KiB - application/x-bzip - bz2 archives of successful rdiff-bacup outputs (-v 8))
file #13532:  rdiff-backup2.out.tar.bz2 added by None (182KiB - application/x-bzip - bz2 archives of successful rdiff-bacup outputs (-v 8))
file #13514:  BigFileNetworkTest.txt added by mstuff (3KiB - text/plain - Jolly big file transfer test)
file #13500:  backup(2).log added by mstuff (64KiB - text/x-log - verbosity at 5)
file #13501:  rdiff-backup_failure_output(2).txt added by mstuff (66KiB - text/plain - verbosity at 5)
file #13452:  backup.log added by mstuff (456KiB - text/x-log)
file #13449:  rdiff-backup_failure_output.txt added by mstuff (458KiB - text/plain - outputs and logs, verbosity at 5)
file #13450:  2nd_failure.txt added by mstuff (3KiB - text/plain - outputs and logs, verbosity at 5)
file #13451:  error_log.2007-07-23T22:59:25+12:00.data.gz added by mstuff (87B - application/x-gzip - outputs and logs, verbosity at 5)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by owsla (Posted a comment)
  • -email is unavailable- added by mstuff (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 19 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-08-09 owsla StatusNone Fixed
        Open/ClosedOpen Closed
    2007-08-09 mstuff Attached File- Added chars_to_quote, #13629
        Attached File- Added chars_to_quote, #13630
        Attached File- Added .rdiff-backup_include-list, #13631
    2007-08-09 mstuff Attached File- Added rdiff-backup1_1Y.out.bz2, #13627
        Attached File- Added rdiff-backup1_1YY.out, #13628
    2007-08-08 owsla Assigned toNone owsla
    2007-08-08 mstuff Attached File- Added rdiff-backup2_1XX.out, #13617
    2007-08-08 mstuff Attached File- Added rdiff-backup2_1X.out, #13616
    2007-07-30 None Attached File- Added rdiff-backup1.out.tar.bz2, #13531
        Attached File- Added rdiff-backup2.out.tar.bz2, #13532
    2007-07-29 mstuff Attached File- Added BigFileNetworkTest.txt, #13514
    2007-07-27 mstuff Attached File- Added backup(2).log, #13500
        Attached File- Added rdiff-backup_failure_output(2).txt, #13501
    2007-07-24 mstuff Attached File- Added backup.log, #13452
    2007-07-24 mstuff Attached File- Added rdiff-backup_failure_output.txt, #13449
        Attached File- Added 2nd_failure.txt, #13450
        Attached File- Added error_log.2007-07-23T22:59:25+12:00.data.gz, #13451

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code