mainstoreBackup - Support: sr #108567, Support of Extended File...


sr #108567: Support of Extended File Attributes (Linux)

Submitter:  None
Submitted:  Sun 11 May 2014 11:17:47 AM UTC
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Need Info
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup

Thu 03 Nov 2016 06:49:06 PM UTC, comment #5: 

Hi, and sorry for the late response...

StoreBackup - without xAttrs - works in the following way (very short version):
It reads the sourceDir recursively and checks for new files by checking path-filename, size, ctime, mtime. It then copies, compresses or hard links the file. Additionally (!) it stores the metadata read from the original source file into .md5CheckSum.
Depending on the options, it then changes the uid, gid, mtime of the backup version of that file.

There is one problem with the hard linking. Think about Bob and Alice have the same file in the home directory. This (identical) file will be stored once in the backup, but with two hard links. But the file in the backup can only be owned by Bob or Alice, not be both. But if I restore the backup, there have to be two files - one owned by Bob in Bob's home directory and one owned by Alice in Alice's home directory. To be able to restore the data correctly, copies (or uncompresses) the file from the backup to the two different places and then restores the metadata from the information in .md5CheckSum. The result is what's wanted - two files owned one be Bob and one by Alice. (With the information in .md5CheckSum storeBackup also is able to restore hardlinks.)

So for a restore, storeBackupRecover doesn't care in any way about the file permissions or uid in the backup. (That's the reason why you can use option ignorePerms without any effect on restoring.)

I think, io handle xAttr, the way must be somehow similar.

At some point, I think, storeBackup will support xAttr. But probably this will take a lot of time for the following reasons:
- there are a lot of other features to implement with have been on the list for a long time
- my time is very limited at the moment :-(

Btw, i'm using kde and a restored my home directory several times and never had a problem with this. Do you have an idea what kde is doing with xAttr and what this means?

Best Regards,

Heinz-Josef Claes <hjclaes>
Group administrator
Wed 02 Nov 2016 08:14:25 PM UTC, comment #4: 

Sorry, i hadn't thought it through up to the end: Unfortunately it wouldn't be as simple as i thought.

Having actually already given the (bad) hint myself (explicitly testing only newly created files for reasons...), I didn't recognize the full implications: 
That changing xAttrs can't be detected at all, if file content stayed the same.
(Inode-) 'change time' would be modified though (presumably), but the md5-hashsum (that comprises only file content, but not inode metadata) wouldn't change.
Thus, modification of  extended file attributes  couldn't be 'noticed' by storeBackup, and xAttrs would only be written (quite 'by chance') or modified, if the file content had also changed.

-grmbl-. -- Modifying storeBackup in order to honor xAttrs then would mean quite more than i initially thought: Probably, to introduce one more checksum (~file-metadata-checksum~) into the concept, which would be checked if ctime changed but the md5-hash didn't.

Ok, I thought 'xAttrs' were possible by less intrusive changes to the code, all in all.  ... a pity.

Thanks anyway!
Best wishes,

Mon 31 Oct 2016 08:45:14 PM UTC, comment #3: 

Hello Heinz-Josef Claes,

I'd like to 'duplicate' on the  extended file attributes ('xAttrs')  issue.
I'm storing important tags on a lot of files (and to eventually recover -much preferedly on a per file basis- from the backups, if necessary).

This topic might perhaps be solvable easier than actually thought. Maybe I can add some more hints/infos/thoughts to this point:

- As far as I understood from the documentation, storeBackup  always?) makes use of the system's (?, see below)  cp-command    to copy the data to the backup-volume.

- That seems to be good luck here, because  GNU/Linux'  cp -a  genuinely  does support  xAttrs  where possible (e.g. on ext2/3/4 file systems and others).

- My expectation was storeBackup would therefore directly

>inherit' from  cp -a  the capability to copy all files with  (user-)xAttrs preserved.

  (Thus, user-xAttrs would just work,  trusted-xAttrs would also work (if storeBackup / cp -a  was run with root-permissions), and system/security-xAttrs probably the same(?).
   (For my personal use-case, user-xAttrs would already do the job.))

- But I found storeBackup just doesn't.  (It only copies the newly created test file(s), but without any of its attached xAttrs). While copying the test files manually via  cp -a  to the same volume worked perfectly.
  I also tried  storeBackup  with the  --cpIsGnu  option (again, newly created file, new xAttrs), but even so the xAttrs didn't get copied over.
  Also, when the volume was explicitely mounted with user-xAttr support on (sudo mount ...  -o user_xattr ... [even if cp -a already worked on gvfs-user-mounted FS's]), it didn't work, while  cp -a  worked fine.
  The test files had non-zero file length [to prevent storeBackup building up probably the same hash-sum (as for all zero-byte-files) thus not noting/assuming a file change / not updating the inode data.].
  [<- Ubuntu 12.04; storeBackup version 3.2-1; ext3 + ext4 file systems ]

The question coming up: Why doesn't  perl's / storeBackup's  'internal' call to  cp -a  (I mean from inside the program) doesn't work as expected - where an 'external' call (bash shell) just does it?
Is perl / storeBackup using a different 'entity' or 'derivative' of  the  cp-command  instead of the original  /bin/cp  binary?
Maybe  perl  merely 'simulates' some Linux binaries, somehow?  Or restricts them?
Or maybe the  -a  option finally gets lost on its way to the perl interpreter's real 'call' of the  cp binary, in the end? Or ....  or ....?

My hope is you might have some helpful idea about the  storeBackup  cp  function(-ing)  here, or some deeper insight into perl's deep-down secrets/issues  regarding the calls to a Linux binary from inside a perl program?
Maybe there is some kind of software switch / system parameter  in perl   that can be used to somehow enable full  cp  compatibility?
Or (if applies) how could one -in fact- 'pass' the whole copy thing over to the 'real'  /bin/cp  binary (together with its -a option parameter)?
Would that slow down the backup a lot (if ^ applies) ?

Thanks for your help,

P.S:  At last: Thank you for making storeBackup! This development reveals a really high sense of quality and usability/usefulness, thoughtful dedication, and professionality. Also the documentation is outstanding! From there it also becomes clear how well thought-out, tested and reliable storeBackup really must be! Just great.

Tue 03 Jun 2014 10:31:10 PM UTC, comment #2: 

Thanks a lot for your reply. I am using KDE and I am looking for a solution that preserves my file ratings and tags created by dolphin/baloo. As far as I can judge KDE uses the user namespace:

$ getfattr -d IMG_6572.JPG
# file: IMG_6572.JPG

getfattr -d text.odt
# file: text.odt

Recently, I had to restore a backup which worked flawlessly. As expected my tags and ratings weren't recovered since storeBackup doesn't support extended file attributes. I would be more than happy if storeBackup will support it in a future version. For my personal needs the user namespace would be sufficient. I don't know if this can be done in a reasonable amount of time but a solution would help me a lot.

Thank you very much.

Neo <neo111>
Sat 31 May 2014 09:01:43 AM UTC, comment #1: 

I personally didn't had to do much with extended file attributes, so I searched and found this:

Which tells me there are four namespaces for extended file attributes:
This seems to be a lot of work. (Maybe also a lot of work to understand what that means and to build the required environment (SELinux).

So what do you mean by your request? All of them or something special?

Heinz-Josef Claes <hjclaes>
Group administrator
Sun 11 May 2014 11:17:47 AM UTC, original submission:  

It would be great if storeBackup could preserve extended file attributes on linux systems (if any).



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by neo111 (Posted a comment)
  • -email is unavailable- added by hjclaes (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-05-31 hjclaes StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-12ab.
    Corresponding source code