Anything-sync-daemon

anything-sync-daemon (asd) is a tiny pseudo-daemon designed to manage user specified directories referred to as sync targets from here on out, in tmpfs and to periodically sync them back to the physical disc (HDD/SSD). This is accomplished via a bind mounting step and an innovative use of rsync to maintain synchronization between a tmpfs copy and media-bound backups. Additionally, asd features several crash recovery features.

The design goals and benefits of asd are:

Transparent user experience
Reduced wear to physical drives
Speed

Since the sync target(s) is relocated into tmpfs (RAM disk), the corresponding onslaught of I/O associated with system usage of them is also redirected from the physical drive to RAM, thus reducing wear to the physical drive and also improving speed and responsiveness. The access time of RAM is on the order of nanoseconds while the access time of physical discs is on the order of milliseconds. This is a difference of six orders of magnitude or 1,000,000 times faster.

Warning: If syncing browser profiles is desired, it is recommended NOT to use asd for this purpose. Instead, use Profile-sync-daemon which has built in sanity checks for unique situations specific to running a browser profile in tmpfs. Anything-sync-daemon does not have these checks; under certain circumstances, browser profile data can be lost. You have been warned.

Installation

Install the anything-sync-daemon package.

Configuration

User managed settings are defined in /etc/asd.conf which is included in the package.

At a minimum, define the sync target(s) to be managed by asd in the WHATTOSYNC array. Syntax below.
Optionally uncomment and define the location of tmpfs in the VOLATILE variable.
Optionally enable the use of overlayfs to improve sync speed even further and use a smaller memory footprint. Note that this option requires your kernel be configured to use either the 'overlay' kernel module. See #Overlayfs mode for additional details.

Note: The default value of /tmp should work just fine for the VOLATILE setting. If using bleachbit, do NOT invoke it with the --clean system.tmp switch or you will remove a key dot file (.foo) from /tmp that asd needs to keep track of sync status. Also note that using a value of /dev/shm can cause problems with systemd's NAMESPACE spawning only when users enable the overlayfs option.

Example:

WHATTOSYNC=('/var/lib/monitorix' '/srv/http' '/foo/bar')

or

WHATTOSYNC=(
'/var/lib/monitorix'
'/srv/http'
'/foo/bar'
)

Usage

Start/enable asd.service. Additionally, a provided resync-timer will run an hourly resync from tmpfs back to the disk. The resync-timer is started automatically with asd.service so there is no need to manually start the timer.

Preview (parse) mode

Run asd parse to view what what asd will do/is doing based on the entries in /etc/asd.conf. It will also provide useful information such as dir size, paths, and if any recovery snapshots have been created.

Tips and tricks

Sync at more frequent intervals

The package provided re-sync timer triggers once per hour. Users may optionally redefine this behavior simply by extending the systemd unit. The example below changes the timer to sync once every ten minutes (note that OnUnitActiveSec needs to be cleared before being re-assigned [1]):

/etc/systemd/system/asd-resync.timer.d/frequency.conf

[Unit]
Description=Timer for Anything-sync-daemon - 10min

[Timer]
OnUnitActiveSec=
OnUnitActiveSec=10min

See systemd.timer(5) for additional options.

Overlayfs mode

Note: There are several versions of overlayfs available to the Linux kernel in production in various distributions. Versions 22 and lower have a module called 'overlayfs' while newer versions (23 and higher) have a module called 'overlay' -- note the lack of the 'fs' in the newer version. Asd will automatically detect the overlayfs available to your kernel if it is configured to use one of them.

Overlayfs is a simple union file-system mainlined in the Linux kernel version 3.18.0. Starting with asd version 5.54, overlayfs can be used to reduce the memory footprint of asd's tmpfs space and to speed up sync and unsync operations. The magic is in how the overlay mount only writes out data that has changed rather than the entire sync target. The same recovery features asd uses in its default mode are also active when running in overlayfs mode. Overlayfs mode is enabled by uncommenting the USE_OVERLAYFS="yes" line in /etc/asd.conf followed by a restart of the daemon.

Snapshots

Odds are the "last good" backup of your sync target(s) is just fine still sitting happily on your filesystem. Upon restarting asd (on a reboot for example), a check is performed to see if asd was exited in some corrupted state. If it is detected, asd will snapshot the "last good" backup before it rotates it back into place. Note that, since asd tries to decrease the disk usage, it never really "copies" the full contents of the directory and just uses the hardlinks to the previous files. And during the rsync step, it creates new files so that the previous hardlinks are untouched. So trying to modify the directory during the time asd is trying to backup can leave the directory in some corrupted state.

Note: Users can disable the snapshot/backup feature entirely by uncommenting and setting the USE_BACKUPS variable to "no" in /etc/asd.conf if desired.

You will find the snapshot in the same directory as the sync target and it will contain a date-time-stamp that corresponds to the time at which the recovery took place. For example, a /foo/bar snapshot will be /foo/.bar-backup_asd-crashrecovery-20141221_070112.tar.zstd -- of course, the date_time suffix will be different for you.

To restore your snapshots:

Stop asd.service.
Confirm that the directories created by asd is not present. If they are, asd did not stop correctly for other reasons.
Move the "bad" copy of the sync target to a backup (do not blindly delete anything).
Untar the snapshot archive to the expected sync target.

Example using /foo/bar:

$ cd /foo
$ mv bar bar-bad
$ tar -xvf .bar-backup_asd-crashrecovery-20141221_070112.tar.zstd

Clean all the snapshot with the clean mode

Running asd clean will delete ALL recovery snapshots that have accumulated. Run this only if you are sure that you want to delete them.

Note: If a sync target is owned by root or another user, and if asd is called to clean, it will throw errors based on the permissions of the sync targets. One can call this function with sudo or as root to avoid these errors.

Support

Post in the discussion thread with comments or concerns.