bup

This special remote type stores file contents in a bup repository. By using git-annex in the front-end, and bup as a remote, you get an easy git-style interface to large files, and easy backups of the file contents using git.

This is particularly well suited to collaboration on projects involving large files, since both the git-annex and bup repositories can be accessed like any other git repository.

See using bup for usage examples.

Each individual key is stored in a bup remote using bup split, with a git branch named the same as the key name. Content is retrieved from bup using bup join. All other bup operations are up to you -- consider running bup fsck --generate in a cron job to generate recovery blocks, for example; or clone bup's git repository to further back it up.

configuration

These parameters can be passed to git annex initremote to configure bup:

encryption - Required. Either "none" to disable encryption of content stored in bup (ssh will still be used to transport it securely), or a value that can be looked up (using gpg -k) to find a gpg encryption key that will be given access to the remote, or "shared" which allows every clone of the repository to access the encrypted data (use with caution).

Note that additional gpg keys can be given access to a remote by running enableremote with the new key id. See encryption.
buprepo - Required. This is passed to bup as the --remote to use to store data. To create the repository,bup init will be run. Example: "buprepo=example.com:/big/mybup" or "buprepo=/big/mybup" (To use the default ~/.bup repository on the local host, specify "buprepo=")

Options to pass to bup split when sending content to bup can also be specified, by using git config annex.bup-split-options. This can be used to, for example, limit its bandwidth.

notes

git-annex-shell does not support bup, due to the wacky way that bup starts its server. So, to use bup, you need full shell access to the server.

Error with bup and gnupg

Hello,

I get this error when trying to use git-annex with bup and gnupg:

move importable_pilot_surveys.tar (gpg) (checking localaseebup...) (to localaseebup...) 
Traceback (most recent call last):
  File "/usr/lib/bup/cmd/bup-split", line 133, in 
    progress=prog)
  File "/usr/lib/bup/bup/hashsplit.py", line 167, in split_to_shalist
    for (sha,size,bits) in sl:
  File "/usr/lib/bup/bup/hashsplit.py", line 118, in _split_to_blobs
    for (blob, bits) in hashsplit_iter(files, keep_boundaries, progress):
  File "/usr/lib/bup/bup/hashsplit.py", line 86, in _hashsplit_iter
    bnew = next(fi)
  File "/usr/lib/bup/bup/helpers.py", line 86, in next
    return it.next()
  File "/usr/lib/bup/bup/hashsplit.py", line 49, in blobiter
    for filenum,f in enumerate(files):
  File "/usr/lib/bup/cmd/bup-split", line 128, in 
    files = extra and (open(fn) for fn in extra) or [sys.stdin]
IOError: [Errno 2] No such file or directory: '-'

I was able to work-around this issue by altering /usr/lib/bup/cmd/bup-split (though I don't think its a bup bug) to just pull from stdin:

files = [sys.stdin]

on ~ line 128.

Any ideas? Also, do you think that bup's data-deduplication does anything when gnupg is enabled, i.e. is it just as well to use a directory remote with gnupg?

Thanks! Git annex rules!

Albert

Comment by Albert — Mon Oct 22 16:56:56 2012

comment 2

@Albert, thanks for reporting this bug (but put them in bugs in future please).

This is specific to using the bup special remote with encryption. Without encryption it works. And no, it won't manage to deduplicate anything that's encrypted, as far as I know.

I think bup-split must have used - for stdin in the past, but now, it just reads from stdin when no file is specified, so I've updated git-annex.

Comment by joeyh.name — Tue Oct 23 16:01:43 2012

Bup remotes in git-annex assistant

Hi,

Is the bup remote available via the Assistant user interface?

Unrelated question;

If you are syncing files between two bup repos on local usb drives, does it use git to sync the changes or does it use "bup split" to re-add the file? (Basically, is the syncing as efficient as possible using git-annex or would I have to go to a lower level)

Many Thanks, Sek

Comment by sekenre — Wed Mar 13 08:54:56 2013

comment 4

I don't plan to support creating bup spefial remotes in the assistant, currently. Of course the assistant can use bup special remotes you set up.

Your two bup repos would be synced using bup-split.

Comment by joey — Wed Mar 13 12:05:50 2013

comment 5

I don't plan to support creating bup spefial remotes in the assistant, currently. Of course the assistant can use bup special remotes you set up.

Your two bup repos would be synced using bup-split.

Comment by joey — Wed Mar 13 12:05:57 2013

bup fail?

I've run into problems storing a huge number of files in the bup repo. It seems that thousands of branches are a problem. I don't know if it's a problem of git-annex, bup, or the filesystem.

How about adding an option to store tree/commit ids in git-annex instead of using branches in bup?

Comment by Tobias — Sun Mar 31 17:05:32 2013

comment 7

bup-split uses a git branch to name the objects stored in the bup repository. So it will be limited by any scalability issues affecting large numbers of git branches. I don't know what those are.

Yes, it would be possible to make git-annex store this in the git-annex branch instead.

Comment by joey — Tue Apr 2 17:24:06 2013

Comments on this page are closed.