maindavfs2 - Support: sr #108646, Low upload speed using clouds

 
 

sr #108646: Low upload speed using clouds

Submitter:  None
Submitted:  Thu 04 Sep 2014 08:57:58 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  In Progress
Privacy:  Public Assigned to:  _71007
Originator Email:  -email is unavailable- Open/Closed:  Closed
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 01 Mar 2015 09:45:23 AM UTC, comment #8: 

"I am imagine that davfs2 can have an option to configure reconnection when speed becomes too low."
I would not even know how to define "speed becomes too ow", leave alone how to measure it.

davfs2 can't do anything about slow servers.
What you could try:
Set option "read_timout" to some higher value. This might prevent davfs2 from stopping data transfer and starting it again when the server takes too much time to respond. But this is only trying, because I do not know what is wrong with the connection.

Werner

- <_71007>
Tue 24 Feb 2015 11:50:38 AM UTC, comment #7: 

I have a good vps, so I'm sure that it isn't an issue with my network. The problem is that "yandex" server is bad. Cadaver uploads data, progress bar shows 0-10%, than it stuck in 10% for about 10 seconds, than upload continues from 0%. Sometimes server closes connection, sometimes it limits the speed for about 5-10kbit/second. After upload is completed it stuck in 100% for about 1 minute.

The most dangerous behavior of its servers is random upload speed limit, but after reconnection speed becomes normal. I am imagine that davfs2 can have an option to configure reconnection when speed becomes too low.

Andrew Aladjev <puchuu>
Sat 06 Sep 2014 09:32:00 PM UTC, comment #6: 

Already tried:
$ curl -u 'user:password' -T "/home/user/Desktop/1.txt" "https://webdav.yandex.ru/1.txt" -v
Uploading was successfull but slow again with the same speed.
Although the download speed is about 10 Mbit/s.

I think the webdav traffic truncation presents at the Yandex servers, thereby my problem can be considered solved.

Anonymous
Sat 06 Sep 2014 07:31:18 PM UTC, comment #5: 

cadaver and davfs2 use the same HTTP library, Neon. davfs2 has the additional overhad of first copying into the cache, but compared to a slow internet connection it does not count.

I would be really surprised if Neon slowed down the upload. But to test this you might try curl as an alternative. It uses another library, but has a lot of parameters you might have to check.

Werner

- <_71007>
Sat 06 Sep 2014 05:31:48 PM UTC, comment #4: 

My bad. I lost sight of some details:
1) I had already set delay_upload to 0 in the config file.
2) By "upload time" I considered the execution time of the following script:
 date
 /sbin/mount.davfs https://webdav.yandex.ru /mnt/yandex/
 cp /home/user/Desktop/1.txt /mnt/yandex/
 sync
 umount /mnt/yandex/
 date
(Sorry, I didn't find out how to format it here as source code.)
Maybe it's not right way but better than first 3 "displayed" values.

By the way I forgot to describe the purpose.
I wanted to apply davfs2 to backup my personal data using gzip-compressed file streams encrypted by PGP. Your program would provide filesystem transparency with mapping a cloud resource (like NFS or SSHFS).

Thanks to you the davfs work is not transparent for me.

> If that is what you want to do, then use cadaver It is exactly designed for this.

Unfortunately cadaver shows the same speed while uploading files:
 cadaver https://webdav.yandex.ru
 dav:/> put /home/user/Desktop/1.txt /1.txt
(I used the same way (size/time) to get upload speed.)
So I dare to suppose that webdav brandwidth truncation takes place. Yandex support still have not answered about upload speed using cadaver.

Anonymous
Sat 06 Sep 2014 01:42:26 PM UTC, comment #3: 

I would have appreciated if you had answered the "how" in my question. Nothing of what you list makes clear what you did and what it is that you measured. At least example 4 gives a hint that you did not measure upload time.

So lets first clarify:

  • davfs2 is not a tool to donwload and upload files to a WebDAV server. If that is what you want to do, then use cadaver It is exactly designed for this.


  • davfs2 maps WebDAV resources into a local file system. It is designed to allow applications that are not WebDAV aware to access remote WebDAV resources through the local file system interface. It uses caching to make the local file system interface responsive. Getting files as fast as possible to the server was never a design goal.


If you copy a file to a davfs2 file system, the file is copied into the local cache and is immediately accessible by local applications. It will not be transferred to the server until the last open file descriptor for write is closed. And davfs2 will then still wait for a configurable time before uploading. This is because many applications use temporary files that would only be transferred to the server to be deletes at once.

So what you measured seems to be the time from a cp to the moment when the file appears on the server. This is obviously not the upload time. And yes, the Yandex people are right.

If for some reason you must use davfs2 for file uploads you can configure 'delay_upload 0'. But there is no way to avoid the cache file.

Werner

P.S.: Always describe exactly what you did and what happend, including complete error messages and other output from programs, when sending a support request. Don't expect people like me to trust in statements like "upload speed is 70-100 KiB" (it's KiB/s by the way) and I used iftop. If I am expected to help, I need to know what is going on, not what you think is going on.

- <_71007>
Sat 06 Sep 2014 12:20:08 PM UTC, comment #2: 


>The network and the server are the first that come to mind.

I should have noticed that with using their console program "yandex-disk" the upload/download speed is notably higher (about 5-10 MiB/s).
Provider's brandwidth is 20 Mbit/s.
The web services like speedtest.net and 2ip.ru indicates my brandwidth about 10 Mbit/s to Moscow where the main Yandex's datacenter is. But the company is surely to have local datacenter in my city (Krasnoyarsk) therefore the speed rate may be higher.

> How did you measure the upload speed?

Several ways as:
1) conky display content;
2) iftop utility;
3) cadaver status bar;
4) rough calculation: file size divided by time.
Showed the same speed apprx 70-100 KiB.

I also talked with Yandex suppport team. They explain the low speed (occured using davfs) of caching files before uploading. May they be right or the reason is contrived?

Anonymous
Sat 06 Sep 2014 08:42:36 AM UTC, comment #1: 

There are many possible reasons for slow uploads. The network and the server are the first that come to mind.

But my first question (because it may be a misunderstanding):
How did you measure the upload speed?

Werner

- <_71007>
Thu 04 Sep 2014 08:57:58 PM UTC, original submission:  

Using Debian Jessie and davfs 1.5.2.
Mounted cloud storage Yandex.Disk.
https://help.yandex.com/disk/
The download speed using /cp, /rsync etc. is acceptable but while uploading I have less than 100 KiB.
I tried to modify some parametres in /etc/davfs/davfs.conf, but it still has no effect.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by puchuu (Posted a comment)
  • -email is unavailable- added by _71007 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-15 _71007 Open/ClosedOpen Closed
    2014-09-06 _71007 StatusNone In Progress
        Assigned toNone _71007

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code