I am building a large software art project including a number of data files and build artifacts locally. The size of the entire generated set is around 20 gigabytes. Contained in the build results are hundreds of scripts which run unit tests, static code analysis and long running data validation. A clean build takes an hour or more, so the build is done incrementally.

To improve build iteration and testing, I exploit a continuous integration program to build this collection of assets when changes are submitted to source control. Once the build has completed, I push the data to downstream machines to run the unit test scripts, analysis and validation. In this way the primary machine builds change lists incrementally, as fast as it can, offloading long running tests to multiple downstream vms to run in parallel.

To speed up the transfer of data, I use rsync to push data to a data store, using hard links and copy-to-new-folder to create per-change list snapshots of build results. Only those files that have changed are transferred. On the testing machine side, a signal is send to restore a specific changelist snapshot via rsync, and then to run specific tests. This allows the downstream test machines to only copy over those files that have changed from one changelist to another.

This works really well. It is reliable and easy to understand.

Unfortunately as the data set has grown, the amount of time spent for rsync to scan for differences between the source and destination directory has started to grow dramatically. I only use file size and file date for compare, but still there are thousands of compares. This is slow for the primary build committing to the data store as well as the downstream test machines pulling the artefacts down. Small changes affecting only a handful of artefacts takes longer to compare differences than to copy.

Given that each snapshot of the data store is tied to a specific changelist, and each store and restore is bound to that snapshot (no mix and match from one changelist to the other is permitted), I want to exploit a true snapshot-based system where finding the required deltas when moving from one changelist to another is a constant time and fast operation and not based on scanning the different source trees.

My question is this: what snapshot file system could help me here given that I am running builds and testing them on Windows, but also restore data to Linux machines. Both local and across network. I need to go forward and backward, potentially skipping changelist to help find when bugs were introduced. Bonus points if I could also restore to the cloud.