aboutsummaryrefslogtreecommitdiffstats
path: root/README.md
blob: a75f9ac9d68b5fcbc06b34803b9447c6259da138 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
# bitpocket

Originally cloned from git://github.com/sickill/bitpocket.git    
Hosted at https://git.faster-it.com/bitpocket    
Mirrored at https://github.com/fasterit/bitpocket    


## About

This is a fork of @sickill/bitpocket we used internally since 2014.
While there are quirks (e.g. bitpocket does not backup files it deletes from the server)
we have found this tool to be very useful for controlled multi-device syncing for
tech-savy users.

We use it cross-platform for Linux, Windows/cygwin (bitpocket) and Mac OSX
with homebrew (bitpocket_mac).

If you are new to bitpocket:

**bitpocket** is a small but smart script that does 2-way directory
synchronization. It uses _rsync_ to do efficient data transfer and tracks local
file creation/removal to avoid known rsync problem when doing 2-way syncing
with deletion.

bitpocket can use any server which you have ssh access to for its central
storage.


## Installation

Clone repository and symlink `bitpocket` (or the `bitpocket_mac`) bin to somewhere within your `$PATH`:

    $ git clone git://github.com/fasterit/bitpocket.git
    $ ln -s `pwd`/bitpocket/bitpocket ~/bin/bitpocket

Or download script and place it in a directory in your `$PATH`:

    $ curl -sL https://raw.github.com/fasterit/bitpocket/master/bitpocket > ~/bin/bitpocket
    $ chmod +x ~/bin/bitpocket

For MacOSX you need _homebrew_ and have `coreutils` and `gnu-sed` installed ... eh ... "brewed".    
The BSD versions of coreutils and sed that MacOSX ships are just way too old to be useful.

### Setting up master

Create an empty directory on some host that will be the master copy of your files:

    $ ssh user@example.org
    $ mkdir ~/BitPocketMaster


### Setting up slaves

On each machine you want to synchronize initialize an empty directory as your bitpocket:

    $ mkdir ~/BitPocket
    $ cd ~/BitPocket
    $ bitpocket init user@example.org ~/BitPocketMaster


## Usage

After installation, you use the `bitpocket` command for synchronization and other tasks.
Running `bitpocket help` will display the following message.

    usage:  bitpocket [sync | push | pull | pack | log | cron | list | help]
            bitpocket init {<REMOTE_HOST> | ""} <REMOTE_PATH>

    sync    Run the sync process. If no command is specified, sync is run by default.
    push    Only push new files to the server.
    pull    Only pull new files from the server.
    init    Initialize a new bitpocket folder. Requires remote host and path params.
    pack    Pack any existing (automatic) backups into a git repository.
    cron    Run sync optimized for cron, logging output to file instead of stdout.
    log     Display the log generated by the cron command
    list    List all files in the sync set (honoring include/exclude/filter config).
    help    Show this message.

    Note: All commands (apart from help), must be run in the root of a
          new or existing bitpocket directory structure.


### Manual sync (bitpocket sync)

To synchronize your local slave with master just run _bitpocket sync_ inside your
bitpocket directory:

    $ cd ~/BitPocket
    $ bitpocket sync

Ensure that you run bitpocket at least once immediately after creating a new slave and
before adding new files to the slave directory. If there are files in the master they
will be pulled into the slave.  You may then move files into your slave directory and 
they will be detected as added.

To improve the safetely of daily operations the Faster IT version of bitpocket has
patches added which allow to only `push` or `pull`.

So

    $ cd ~/BitPocket
    $ bitpocket pull

will only pull from the server and leave files there unclobbered.
Similarly `bitpocket push` will sync new files to the server but not risk local files
when you are in a hurry.

### Maintaining backups (bitpocket pack)

bitpocket does not include a full-fledged versioning system at the moment, but
it does automatically create a local backup of any files before overwriting or 
deleting with changes from the BitPocketMaster. These backups are placed into a 
timestamped directory (one directory per sync). The path to this directory is:

    .bitpocket/backups/YYYY-MM-DD.hhmmss

As these files accumulate, you may want to remove them, but you can also run
`bitpocket pack` to combine all the backup files into a git repository contained
within the _.bitpocket_ directory:

    $ cd ~/BitPocket
    $ bitpocket pack

This requires an installation of _git_, but
allows all the space saving advantages of _git_ when making repeated changes
to the same files.

There is a discussion about potential directions for versioning direction in the
issue 15 of sickill's original bitpocket git:
[github.com/sickill/bitpocket/issues/15](https://github.com/sickill/bitpocket/issues/15)    
(tl;dr version: The discussion wasn't conclusive.)


### Redirecting output to log file (bitpocket cron)

If bitpocket is run with the cron parameter ( _bitpocket cron_ ), it will perform
a sync, but instead of showing the progress on stdout, it will redirect all 
output to a log file: 

    $ cd ~/BitPocket
    $ bitpocket cron

As the name of this parameter implies, this is mainly useful when running bitpocket
through the _cron_ command. (See "Automatic sync with cron" for more information 
about how to configure this).


### Displaying logs (bitpocket log)

When running bitpocket in cron with `bitpocket cron` it will append its output
to _.bitpocket/log_ file. You can review the tail end of an existing log file,
or watch live log as it is generated, with following command:

    $ cd ~/BitPocket
    $ bitpocket log


### Displaying list of files to be synchronized (bitpocket list)

You may want to know which files will be synchronized before actually performing
the syncronization. You can verify which files are in the synchronization set
by running `bitpocket list`:

    $ cd ~/BitPocket
    $ bitpocket list

Note that this does _not_ list changed files, it only lists all the local files 
that bitpocket will look at in determining which files to sync. Also, note that
if there are new files in the master that will be added on a sync, they will 
not be included here. This command is only intended to verify which files are
in the synchronization set. (See "Configuring file exclusion and inclusion" for
information about how to control which files are in the synchronization set).


## Configuration

### Automatic sync with cron

Add following line to your crontab to synchronize every 5 minutes:

    */5 * * * * cd ~/BitPocket && nice ~/bin/bitpocket cron

Note that cron usually has very limited environment and your ssh keys with
passphrases won't work in cron jobs as ssh-agents/keyrings don't work there.
Thus it's preferable to generate passphrase-less ssh key for bitpocket
authentication:

    $ cd ~/BitPocket
    $ ssh-keygen -t rsa -C bitpocket-`hostname` -N '' -f .bitpocket/id_rsa
    $ ssh-copy-id -i .bitpocket/id_rsa user@example.org

and uncomment line with `RSYNC_SSH` in _.bitpocket/config_ file.


### Custom rsync options

You can pass additional switches to `rsync` by setting `RSYNC_OPTS` in
_.bitpocket/config_ file. Generated config file includes (commented out)
example setting for dereferencing symlinks:

    # RSYNC_OPTS="-L"

Just uncomment it and change at will.


### Configuring file exclusion and inclusion

If you want some files to be ignored by bitpocket you can create
_.bitpocket/exclude_ file and list the paths there:

    *.avi
    jola
    /misio.txt

_*.avi_ and _jola_ will be matched anywhere in path, _misio.txt_ will be
matched at bitpocket root dir ( _~/BitPocket/misio.txt_ ).

This exclude file is passed to `rsync` as `--exclude-from` argument, check `man
rsync` for _INCLUDE/EXCLUDE PATTERN RULES_.

You can set up even more advanced exclusion/inclusion rules. In all, there
there are three files that you can create to change this configuration:

    .bitpocket/exclude
    .bitpocket/include
    .bitpocket/filter

Be aware that all the quirks from rsync exclusion/inclusion rules carry over
into bitpocket. If you decide that you need such advanced configuration, make
sure that you understand those rules very well, and consider double checking
them before syncing by running `bitpocket list`.


### Slow sync callbacks

When syncing takes more than 10 seconds (SLOW\_SYNC\_TIME setting) bitpocket
can fire off user provided command in background. This can be usefull to notify
user about long sync happening, preventing him from turning off the machine
during sync etc.

There are 3 settings that can be enabled in _.bitpocket/config_ file:

    # SLOW_SYNC_TIME=10
    # SLOW_SYNC_START_CMD="notify-send 'BitPocket sync in progress...'"
    # SLOW_SYNC_STOP_CMD="notify-send 'BitPocket sync finished'"

Just uncomment them and change at will.

You can show tray icon during long sync with
[traytor](https://github.com/sickill/traytor) and following settings:

    SLOW_SYNC_START_CMD='~/bin/traytor -t "BitPocket syncing..." -c "xdg-open ." .bitpocket/icons & echo $! >.bitpocket/traytor.pid'
    SLOW_SYNC_STOP_CMD='kill `cat .bitpocket/traytor.pid`'


## Authors

* Marcin Kulik | @sickill | https://github.com/sickill | http://ku1ik.com/
* torfason | https://github.com/torfason
* Faster IT GmbH | https://git.faster-it.com | https://www.faster-it.com

© 2014-2021 Faster IT GmbH | imprint | privacy policy