Populate's design

Populate delivers sources to a target directory. It consists of:

• A program, which reads the sources as JSON and delivers them to the target its command line names. It knows nothing about Nix expressions: whatever produces sources as JSON can drive it.
• A Nix library, lib, with the types of sources, which describe how to write them in Nix, and evalSource, which turns sources written so into the JSON the program reads.

krops uses both: its pkgs.populate turns populate's arguments into a command line, and checks them during evaluation, so mistakes show early. So there is one program for each version of populate, instead of a script for each set of arguments, and the program can be written in any language. This directory holds the program in Rust. Another implementation, e.g. in Haskell, takes the same command line, prints the same messages, and passes the same tests.

Nix side

default.nix builds the program: nix-build in this directory, or callPackage, as krops' pkgs.populate-unwrapped. lib/default.nix is the Nix library, taking nixpkgs' lib.

krops' pkgs.populate, in its pkgs/krops/populate.nix, is populate's Nix API there. Given populate's arguments, { backup ? false, force ? false, prune ? false, source, target }, it checks them with krops' pkgs/krops/checks.nix, and returns a script running the program with the sources as JSON on file descriptor 3, so the commands populate runs keep standard input. A target string goes to the program as it is. A target attribute set, as mkTarget returns it, goes part by part, as options, with --local or --remote as krops' lib's isLocalTarget decides during evaluation, like the other functions of krops. The script puts the tools populate runs in front of PATH, and depends on the store paths in the sources, so building it builds them. krops' writeDeploy and its other functions use pkgs.populate, so their arguments stay as they are.

Command line and sources

../README.md describes populate's command line, the sources, and what populate prints, which are its interface. The program reads the sources into spec::Sources, and parses the target with spec::TargetParts, completing it with run::Origin, the user running populate and the machine's name, when it runs.

Checks

The program checks the sources and the target as krops' checks.nix does during evaluation, with the same messages:

• The target's host isn't empty, and neither it nor its user starts with -, so ssh can't take them for options. checks.nix doesn't check this, since krops' targets come from the configuration.
• Source names are portable file names: letters, digits, and ._-, not starting with -, and neither ., .., nor .populate.
• The target path is absolute, not the root directory, and contains neither .. nor control characters.
• Inputs get drawn from sources that exist, not from the source itself, and not in circles.
• The settings of a source are those lib/types/populate.nix defines for its type, of the kinds it defines, e.g. absolute paths for the paths of file, pass, and pipe sources. checks.nix doesn't repeat this, since evalSource checks the types.

Checks that depend on whether the target is local come on top, e.g. refusing to copy flakes to remote targets using sudo. krops' checks.nix does that during evaluation too, where pkgs.populate decides whether the target is local.

Safety

Populate never changes or removes anything outside the target directory, whatever the sources and whatever it finds on the target:

• The checks above keep source names and the target path inside it.
• A symlink at the target path of a source gets replaced, never written through, and populate's state directory, .populate, must not be one.
• Removing a tree makes only its directories writable, and removes what's below them, not what symlinks point to.
• Hard links to files of a source get broken before rsync writes to them.
• A Git work tree gets reused only if its .git is a directory of its own, with no other configuration than git clone and populate set, and Git runs without hooks and file system monitors.
• Temporary files get created with mktemp, and moved into place.

The tests' canaries, files around the target directories and in other places populate must not touch, check this after every phase.

Execution

Whether a target is local is what --local or --remote says, or else gets decided when the program runs, like krops' lib's isLocalTarget does: it is, if its user is LOGNAME, or null if that's empty or unset, and its host is localhost or the machine's name, from HOSTNAME, or, if that's empty or unset, the only line of /etc/hostname that is a label. For a local target, populate runs commands directly.

For a remote target, populate runs commands over ssh, sharing one master connection per target. The commands are fixed POSIX shell snippets, with values passed as positional arguments, so nothing gets quoted into them. The target needs a POSIX shell, coreutils, findutils, rsync, patch for patches, Git for git sources, and Nix; NixOS lacks Git. With sudo, the snippets run via sudo.

Populate runs rsync, Git, ssh, and Nix as programs, found in PATH.

Development

• cargo test in this directory runs the program's unit tests.
• nix-build tests -A smoke runs the smoke test, a few scenarios covering each source type and each kind of target.
• nix-build tests runs all scenarios but the exhaustive ones.
• krops' tests run populate's scenarios that expect refusals, and those of the smoke test, through krops' pkgs.populate, with tests/builder.nix.

Status

The program delivers all source types, draws inputs from sources, makes backups, and prunes. It decides what to do itself: the snippets it runs on the target send back what it needs to know, e.g. what Nix prints as JSON, where paths resolve to, or store paths. Delivering flake and flake-input sources is in src/flake_input.rs, drawing inputs and deriving copies in src/flake.rs.