File synchronization | Mutagen (2024)

Mutagen’s file synchronization uses a novel algorithm that combines theperformance of thersync algorithm withbidirectionality and low-latency filesystem watching. It can be used (forexample) to synchronize code between your laptop and a remote container ineffective real-time, allowing you to edit code with your editor of choice andhave it pushed to the remote environment almost instantly. Because it usesdifferential transfers, Mutagen’s synchronization also works effectively fortransferring large binary files such as images or build artifacts.

Design and architecture

Mutagen’s synchronization sessions each operate between an arbitrary pair ofendpoints, termed alpha and beta. The reason that these endpoints aren’ttermed “source” and “destination” is that Mutagen has multiple synchronizationmodes, including bidirectional modes, where these terms don’t necessarily apply.Thus, the order of endpoints provided to the mutagen sync create commandsimply establishes these identities (with the first endpoint being alpha andthe second being beta), whereas the roles of these endpoints are determined bythe synchronization mode.

Synchronization sessions are extremely flexible, allowing both endpoints to belocal, one to be local and one to be remote, or both to be remote (in which casethe local system is simply used as a proxy and controller for synchronization).This flexible topology, combined with the myriadsynchronization configuration options, allows users to createarbitrarily complex synchronization topologies.

Internally, the synchronization algorithm uses a three-way merge to reconcilechanges from both endpoints in a safe and controlled fashion. This means thatsynchronization sessions track the most-recently agreed-upon content and usethat information to detect the changes that each endpoint has made, as well asany conflicts. This algorithm operates in short cycles, with every filesystemchange triggering a cycle. Each cycle consists of a scan of both endpoints,a reconciliation of endpoint contents, staging of updated contents from oneendpoint to another, and application of changes. These cycles are designed to beextremely efficient, to the point of being imperceptibly fast for most contentand changes.

Modes

Mutagen provides four different synchronization modes:

• two-way-safe (Default): In this bidirectional synchronization mode, bothendpoints are treated with equal precedence, and conflicts are onlyautomatically resolved if they don’t result in data loss (for example,modifications on one endpoint can can overwrite deletions of the correspondingcontent on the other endpoint). If conflicts can’t be automatically resolved,they are stored in the session state (and can be enumerated with themutagen sync list command).
• two-way-resolved: This is the same as two-way-safe, except that thealpha endpoint automatically wins all conflicts, including cases where alpha’sdeletions would overwrite beta’s modifications or creations. No conflicts canoccur in this synchronization mode.
• one-way-safe: In this unidirectional synchronization mode, changes areonly allowed to propagate from alpha to beta. Deletions on beta areoverwritten by content from alpha (i.e. the content comes right back), butmodifications and creations on beta can’t be overwritten by alpha (unless bothendpoints have made the same modifications or created the same content).Conflicting contents on beta that can’t be overwritten will be recorded to thesession state (and can be enumerated with the mutagen sync list command).Extra content on beta that doesn’t conflict with contents on alpha is simplyignored.
• one-way-replica: In this unidirectional synchronization mode, betabecomes an exact replica of alpha. Any modifications or additional content onbeta are instantly overwritten or removed, respectively. No conflicts canoccur in this synchronization mode.

You can think of these modes as existing in a table:

“Safe” in this case means that conflicts are only automatically resolved if theydon’t result in the loss of unsynchronized data. “Auto-resolved” means that thealpha endpoint wins any conflict, even if it involves deleting additionalunsynchronized content from beta.

These modes can be specified on a per-session basis by passing the-m/--sync-mode=<mode> flag to the mutagen sync create command. These modescan be specified on a default per-session basis by including the followingconfiguration in ~/.mutagen.yml:

sync: defaults: mode: "<mode>"

Conflict resolution

Conflicts (which can occur in two-way-safe and one-way-safe modes) can beresolved manually by deleting the content on the endpoint which you wish to havelose the conflict. Once deleted, the conflict will no longer exist sincedeletions can be overwritten.

Endpoint URLs

Synchronization endpoint URLs are nothing more than pointers to local or remotefilesystem locations. The exact format for forwarding endpoint URLs istransport-dependent, but each contains a pathcomponent identifying the filesystem location to be synchronized. Filesystemlocations can be directory hierarchies or individual files — Mutagen cansynchronize either.

Mutagen only applies a few restrictions to synchronization endpoints:

• Symbolic links aren’t allowed as synchronization roots, though they can existas part of the parent path of a synchronization roots, and of course areallowed to exist inside of synchronization roots.
• Synchronization of directory hierarchies that span filesystem boundaries isnot allowed.

In both cases, Mutagen will detect and warn about the condition.

Configuration

Synchronization configuration is extensive, with configuration optionscontrolling:

• Synchronization modes
• Ignores
• Permissions
• Symbolic links
• Filesystem watching
• Filesystem probing and scanning
• File staging
• Size limits

Session management

Synchronization sessions are managed using the mutagen sync commands, namelycreate, list, monitor, flush, pause, resume, and terminate.Example usage for these commands can be found in theGetting started guide. Thecreate command comes with a number of flags that control the configuration ofthe sessions that it creates, and the other synchronization session managementcommands all include flags that control their behavior. For more informationabout a particular command, use:

# Show help about a particular synchronization session management command.mutagen sync <command> --help