bugGNU Wget - Bugs: bug #33825, 'wget -nc' race condition

 
 

bug #33825: 'wget -nc' race condition

Submitter:  Noël Köthe <nok>
Submitted:  Sat 23 Jul 2011 08:42:56 AM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Wont Fix
Privacy:  Public Assigned to:  gscrivano
Originator Name:  Open/Closed:  Open
Release:  1.12 Operating System:  GNU/Linux
Reproducibility:  None Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 03 Aug 2011 09:04:42 PM UTC, comment #1: 

wget is not designed in a way that multiple instances can write concurrently to the same set of files.  In these cases, the behaviour is undefined.

Giuseppe Scrivano <gscrivano>
Group administrator
Sat 23 Jul 2011 08:42:56 AM UTC, original submission:  

Hello,

I don't know if you aggree with the described problem but I forward it anyway to point you to this possible problem:

http://bugs.debian.org/500395

--<8--
I downloaded all the aircraft for flightgear from their ftp mirrors. To spread the
load over several mirrors I did:

wget -nc ftp://ftp.kingmont.com/flightsims/flightgear/Aircraft/*
wget -nc ftp://ftp.de.flightgear.org/pub/fgfs/Aircraft/*
wget -nc ftp://some.third.mirror/pub/fgfs/Aircraft/*

It worked basically as files started by one wget were not downloaded by other
wgets. However, after one wget decided to download a locally non existant file it
appearently created it only when the data started coming. In this interval - from
decision to file creation - other wgets sometimes would also decide to download
the same file and so two (or possibly three) wgets would download and write the
same file and waste time and bandwidth.

Maybe the destination file should be created right after wget decides to download
it. Maybe these operations can be combined atomically to avoid the race
completely.

If it later turns out that the file cannot be downloaded, it can be deleted.

That might be a new race as other wegts won't try to download a file which one
wget fails to download.

Bernhard
--8<--

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by nok (Submitted the item)
  • -email is unavailable- added by nok (Debian BTS)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-08-03 gscrivano StatusNone Wont Fix
        Assigned toNone gscrivano
    2011-07-23 nok Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code