aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* populate: a README of its own, krops' linking to itkrops-2tv9 hours5-535/+668
| | | | | | | | | | | | | | pkgs/populate/README.md documents populate on its own: building it, what it needs, its command line, the sources as JSON and in Nix with its Nix library, the target directory, the source types, flake inputs from sources, what it prints, and its tests. The sections about the target directory, the source types, and flake inputs from sources move there from krops' README, which links to them from sections of the same names, and from the first mention of populate. docs/design.md keeps how populate works. krops' README describes krops' own tests, and CHANGELOG what populate being a program of its own means for krops.
* tests: populate's tests of its own, krops' using themtv9 hours11-2494/+2698
| | | | | | | | | | | | | | | | | | | | | | | | populate's tests are in pkgs/populate/tests, independent of krops. They run the program, with the sources as JSON from populate's Nix library, and target strings for the remote targets, leaving the user and whether the target is local to populate, so they cover its command line too. Phases expecting populate to refuse its arguments, e.g. a source name or a target path that can't be used, expect the program's refusal, with refused, the message, since nothing checks them during evaluation there. builder.nix is what builds the scenarios, which krops' tests use as well, with a function running the phases, and whether that checks the arguments during evaluation. vm.nix takes a module for both machines, for what else the scenarios run, and fixtures.nix and smoke.nix are what both use of the fixtures and the smoke test. krops' tests are deploy, writeDeploy's arguments to nixos-rebuild, with the stand-in for nixos-rebuild in the module, and populate, populate's scenarios that expect refusals, which pkgs.populate makes evaluation fail with, and those of populate's smoke test, all through pkgs.populate with target attribute sets as mkTarget returns them. -A only selects from all of populate's scenarios and krops' own.
* populate: layer populate below kropstv9 hours12-200/+231
| | | | | | | | | | | | | | | | | | populate is a program of its own, with a Nix library, in pkgs/populate, which krops uses, and which doesn't use krops: - default.nix builds the program, with nix-build there or callPackage, as pkgs.populate-unwrapped. - lib is populate's Nix library: the types of sources and the POSIX types they use, evalSource, flakeFollows, hasGeneratedFlake, and derivesFlake, taking nixpkgs' lib. krops' lib takes them from there, so it provides them as before, and so does lib/types. pkgs.populate, the function from populate's arguments to a script running the program, is krops' own, in pkgs/krops/populate.nix, with checks.nix next to it. It takes the program as populate-unwrapped, which callPackage passes from the overlay; the flake, whose nixpkgs lacks it, passes its own.
* populate: a command line instead of a spectv9 hours7-180/+619
| | | | | | | | | | | | | | | | | | | | | The program takes populate's arguments on its command line, like other programs: populate [OPTION]... SOURCES [TARGET], with the sources as JSON in the file SOURCES, or on standard input for -. TARGET has the syntax of krops' lib's mkTarget, [USER@][HOST][:PORT][PATH], parsed the same way, and what it leaves out gets filled in when populate runs, as mkTarget would, from LOGNAME and the machine's name. Options set what the syntax can't express, or override its parts: --sudo, --ssh-arg, --user, --host, --port, --path, and --local or --remote for whether the target is the machine populate runs on, which populate otherwise decides like isLocalTarget. The sources carry no version. pkgs.populate passes a target attribute set, as mkTarget returns it, part by part, with --local or --remote as evaluation decides, so nothing changes for krops. A target string goes to the program as it is. The program also refuses target hosts and users starting with -, which ssh would take for options, now that a command line can give them.
* populate: filter the program's sources by nametv11 hours1-6/+7
| | | | | | | | The filter for the program's sources took the paths it gets relative to toString ./., which they needn't start with, e.g. if a symlink leads to the directory, so nothing matched, and the build found no Cargo.lock. It goes by the names of what it leaves out now: the Nix files, the docs, .gitignore, and cargo's target directory.
* CHANGELOG: 1.31.1, as released from krops-1tv12 hours1-0/+7
| | | | | What krops 1.31.1 has goes into a section of its own, the same as in krops-1, leaving what's new in krops 2 unreleased.
* tests: run scenarios selected by nametv13 hours2-4/+24
| | | | | | | nix-build tests -A only --argstr only REGEX runs the scenarios whose names match the regular expression REGEX in a single VM test, like the smoke test, e.g. --argstr only 'lock-hooks-.*' for lock-hooks on all three kinds of targets.
* tests: a hook in a flake populate lockstv13 hours1-0/+42
| | | | | | | | lock-hooks plants a .git in a file source that keeps it, with configuration a git source would use, but a post-index-change hook, which Nix's Git runs when it adds the lock file it writes into a git+file: flake. Drawing inputs refers to that flake as path:, so the hook doesn't run.
* populate: let git sources keep hooks, refuse them only for git+file:tv13 hours3-26/+24
| | | | | | | git sources refused work trees with hooks, but populate's Git doesn't run hooks, so git-foreign's work tree with a hook is fine to use, as it expects. Only Nix's Git runs them, in a git+file: flake, so drawing inputs refers to a work tree with hooks as path: instead.
* tests: check what populate mustn't printtv13 hours2-2/+12
| | | | | | | | A phase's unexpected lists what populate is expected not to print, whether it succeeds or not. notdir expects populate not to go on locking after it fails to find the URL of a source drawn from, which would lock the current directory, the login directory on a remote target, copying it into the store.
* populate: state what the target sends back for derived copies in codetv13 hours1-32/+140
| | | | | | | | DerivedState::parse reads what derived_state prints, and key_text builds the text whose hash is a derived copy's key, the same as before, both with unit tests, instead of decoding fields by position inside Flakes::derive. Fields from the target that don't end in a null byte fail, instead of getting dropped.
* populate: decide whether a target is local like isLocalTargettv13 hours2-13/+86
| | | | | | | | For specs whose target's local is null, the program decides whether the target is local like krops' lib's isLocalTarget: an empty LOGNAME or HOSTNAME counts as unset, as with lib.maybeEnv, and /etc/hostname's labels can be of any length, as with lib.types.label. is_local_with, pure, has unit tests.
* populate: exit with the status of the command that failedtv13 hours5-9/+42
| | | | | | | | | | | The shell implementation runs under set -e, so it exits with the status of the command that failed, e.g. 255 if ssh can't reach the target, which wrappers retrying on connection failures rely on, or 128 if git fetch fails. The program does the same, instead of exiting with 1, and with 128 plus the signal for a command a signal ended. A phase's expect.status is the exit status it expects populate to fail with; git-status checks git fetch's.
* populate: check the settings of sources in the spectv13 hours3-16/+172
| | | | | | | | | | | | The program checks that each source's settings are those lib/types/populate.nix defines for its type, with values of the kinds it defines, for specs from elsewhere than default.nix: a misspelt setting fails, as design.md promises, instead of being ignored, and so does a setting the program would treat as its default because of its kind. This subsumes the check of absolute paths. ln gets a symlink source's target after --, so it can't be taken for an option either.
* populate: fail if find fails looking for .rsync-filter filestv13 hours1-10/+11
| | | | | | | | | | Looking for .rsync-filter files that apply to a file source, a failing find counted as finding none, and only happened with rsync 3.5.0 and later. Like in the shell implementation, find runs before each transfer with filter files, and its failure stops populate with find's message, e.g. for a source that's missing, or contains a directory find can't read, which would also have switched off the check of insecureLinks. A signal during find takes effect right after it.
* populate: give sizes in rsync's log in units of 1024tv13 hours2-3/+5
| | | | | | rsync gives sizes in units of 1000 for two apostrophes, and in units of 1024 for three, which the shell implementation asks for. rsync-units checks it.
* populate: check paths in the spec, name the way into a cycletv13 hours1-4/+63
| | | | | | | | | The program checks that the paths of file, pass, and pipe sources are absolute, as lib/types/populate.nix demands, for specs from elsewhere, so none can be taken for an option, e.g. of rsync. A cycle of sources drawing inputs from each other gets named with the way into it, like checks.nix names it.
* tests: scenarios for rsync's units and pass changestv13 hours1-0/+51
| | | | | | rsync-units checks that rsync's log gives sizes in units of 1024, as --out-format asks for. pass-changes checks that a pass source gets decrypted again when its store changes, and only then.
* populate: run pipe commands through a shelltv13 hours2-2/+15
| | | | | | | | A pipe source's command is an absolute path, which a shell runs as a script if it isn't a program, e.g. lacks #!. The program ran it directly, which fails for such a command. pipe-script runs a pipe command without #!.
* populate: refer to flakes as git+file: only with a usable .gittv13 hours5-6/+63
| | | | | | | | | | | | | Writing the lock file of a git+file: flake, Nix runs Git, with the repository's configuration and hooks, which may run commands. A .git in a source's directory can come from anywhere, e.g. one planted in a materialized flake, or one a file source keeps. So drawing inputs refers to a flake as git+file: only if git_problems finds nothing wrong with its .git directory, the way git sources check their work trees, and else as path:. lock-foreign-git plants a .git in a materialized flake and in a file source that keeps it, whose configuration names an fsmonitor command.
* populate: check the hooks and submodules of Git work treestv13 hours5-6/+116
| | | | | | | | | | | | | | | git submodule update uses the .git of a submodule as it finds it in the work tree: a repository of its own, with configuration that runs commands, or a file naming a repository elsewhere. So git_problems also requires each .git below the work tree to be a file naming a repository in .git/modules. And it refuses hooks in .git/hooks, which populate's Git doesn't run, but Nix's does, writing the lock file of a flake in a git source. git-submodule plants a submodule's .git in the work tree: a repository of its own whose configuration names a filter, and a file naming a repository elsewhere. The fixture withsub has gitsrc as its submodule sub.
* populate: keep URLs, refs, and names from being taken for optionstv13 hours3-8/+34
| | | | | | | | | | A git source's url and ref are strings, and git clone, git remote, and git fetch took ones starting with - for options, e.g. --upload-pack, which runs a command where the repository is local. They come after -- now, and so does a pass source's name for git log and pass show. git-ref-option gives a ref that git fetch would take for --upload-pack.
* README: requirements, and where populate's program istv13 hours2-4/+26
| | | | | | New users learn what the deploying machine and the target need, including nixpkgs 23.11 or later for building populate, and developers where populate's program, its description, and its unit tests are.
* populate: warn about the arguments it doesn't usetv13 hours1-42/+57
| | | | | | | | dash, jq, and implementation, which overrides of the shell implementation set, come through ..., because callPackage sets named arguments, so populate can tell they're set, and warns that they're unused. Other arguments populate doesn't know fail, as they do for named arguments.
* populate: drop the shell implementationtv14 hours11-1965/+92
| | | | | | | | | | | | | | | Populate is the program in Rust only. pkgs.populate checks its arguments, and returns a script running the program with them as JSON. The arguments it took for the shell implementation, dash, jq, and implementation, stay accepted and unused, so overrides setting them keep working. Copying flakes to remote targets using sudo gets refused during evaluation, as before, by checks.nix, and by the program for specs from elsewhere. The jq test goes with the jq programs; the program's unit tests cover their equivalents.
* populate: build with Rust 1.71, as in nixpkgs 23.11tv14 hours3-3/+4
| | | | | | The program needs Rust 1.71, the oldest version its dependencies support, which Cargo.toml states, so clippy points out what newer versions add. The lock file is of version 3, which Cargo reads from version 1.53 on.
* populate: draw flake inputs natively in Rusttv15 hours4-609/+1021
| | | | | | | | | | | | | The program decides how to draw inputs and derive copies of flake and flake-input sources itself. The target runs fixed snippets, which send back what it needs to know: where paths resolve to and what's there, the lock files of flakes as nix flake metadata prints them, and what tells whether a derived copy needs to be made again. The program checks that drawn inputs exist in the lock files, computes the keys of derived copies, and writes generated flakes. Derived copies get the same keys as with the shell implementation, so switching implementations doesn't make them again.
* populate: deliver flake and flake-input sources natively in Rusttv17 hours9-623/+1032
| | | | | | | | | | | src/flake_input.rs reads what nix flake metadata and nix flake archive print as JSON itself: it selects the flake or input, decides what the target fetches or gets copied, and computes the store paths to root, which are what selected.jq, selfIsRemote.jq, selfLocked.jq, and the Nix expressions of the shell implementation do there. On the target, fixed snippets in sh::flake run Nix, getting values as positional parameters, and send JSON or store paths back. The jq test's cases are unit tests of the Rust equivalents.
* populate: deliver flakes and draw inputs with the Rust programtv17 hours6-40/+1522
| | | | | | | | | | | | | The program delivers flake and flake-input sources, draws inputs from sources, and removes state that sources don't need anymore, so it does everything the shell implementation does, and passes all scenarios. src/flake.rs builds the same shell code as shell.nix generates for these, with every value quoted, so both implementations behave alike while that code moves to Rust piece by piece. The ssh scripts in the temporary directory run the ssh that was in PATH when populate started, so one can be ssh in PATH for nix copy. The other one is for nix copy's ssh:// store, without the control socket.
* populate: deliver sources other than flakes with the Rust programtv18 hours10-17/+1435
| | | | | | | | | | | | | | | | | | | | | | The program delivers derivation, file, git, pass, passage, pipe, and symlink sources, makes backups, and prunes, as the shell implementation does, with the same messages. Commands on the target are fixed shell snippets in src/sh.rs, which get values as positional parameters, run directly for local targets and over ssh, sharing one master connection, for remote ones. rsync's transfers and its check for .rsync-filter files it can't read are in src/rsync.rs. Given flake or flake-input sources, or sources drawing inputs from others, the program fails before changing anything, saying that these aren't implemented yet. SIGHUP, SIGINT, and SIGTERM make populate stop once the command it waits for ends, stop the ssh master, remove its temporary directory, and exit with 128 plus the signal's number. The spec's target gets local, which rust.nix sets during evaluation, like the other functions of krops decide, and which the program decides when it runs if it's null. The spec comes on file descriptor 3, so the commands populate runs keep standard input, and the tools populate runs come first in PATH.
* populate docs: describe the designtv19 hours1-0/+177
| | | | | | | pkgs/populate/docs/design.md describes populate's parts, the program's invocation, the spec, the checks, what populate prints, its safety guarantees, how it runs commands, how to test changes, and what the program does so far.
* populate: add a Rust implementation, selectable for the teststv19 hours13-1616/+2247
| | | | | | | | | | | | | | | | | | | pkgs.populate takes implementation: "shell", the default, generates a script from populate's arguments, and "rust" passes them as JSON to the program in pkgs/populate, built from the crate there by program.nix. The program checks the spec as checks.nix does during evaluation, with the same messages, and reports that delivering sources isn't implemented yet. rustProgram replaces the program, e.g. with one built some other way. checks.nix holds the checks of populate's arguments during evaluation. Both implementations check the names of sources and the target path with it, and the Rust one's wrapper also checks drawing inputs, which the shell implementation checks where it draws them. nix-build tests --argstr implementation rust runs the scenarios with the Rust implementation. They expect it to refuse copying flakes to targets using sudo when it runs, not during evaluation, because only then it decides whether the target is local.
* tests: run populate with /dev/null as inputtv19 hours1-2/+4
| | | | | | | | | | The test driver runs commands in a shell whose input is the console it talks through, /dev/hvc0, which is a terminal. ssh -t, which writeDeploy's allocateTTY passes for remote targets, allocates a terminal then, and reads the console, so the deploy-*-allocateTTY scenarios of remote and sudo targets hang. Populate and writeDeploy read nothing, so their input is /dev/null, and ssh allocates no terminal, as these scenarios expect.
* tests: add a smoke testtv20 hours1-2/+45
| | | | | | | | | | nix-build tests -A smoke runs 19 scenarios in a single VM test: each source type, each kind of target, flakes drawing inputs both ways, the sentinel, prune, the checks of source names and target paths, and writeDeploy. That's quick enough to run after every change, while the complete tests stay what a change has to pass in the end. nix-build tests doesn't build it, because the complete tests run the same scenarios.
* populate, overlay: mark populate's scripts as shell, drop unused libHEADmastertv20 hours2-6/+2
| | | | | | The scripts populate writes with writeDash get /* sh */ markers like the other shell code, and the overlay doesn't import krops' lib, which it doesn't use.
* populate: omit the times of directories and symlinkstv20 hours1-0/+4
| | | | | | | rsync sets the modification time of each directory whose contents change, and logs a line for it. Nothing relies on these times, nor on the times of symlinks, so --omit-dir-times and --omit-link-times leave them alone, and the log lists only what got transferred or deleted.
* krops: add allocateTTY to writeDeploytv20 hours5-6/+41
| | | | | | | | | With allocateTTY, nixos-rebuild runs with a terminal, so it shows colours and progress: on remote targets, ssh gets -t, which allocates a terminal if its own input is one, and on the local target, expect's unbuffer provides one. The nixos-rebuild stand-in records whether its output is a terminal.
* populate: follow branches and tags of git sourcestv20 hours4-18/+69
| | | | | | | | A ref that consists of 4 to 40 hex digits is a commit, which populate fetches only if the work tree lacks it, unless fetchAlways is set. Any other ref is a branch or tag, which populate fetches on every run, resetting the work tree to the commit fetched. Resetting to the name itself resolves to a local branch, which fetching doesn't move.
* CHANGELOG: add 1.31.0v1.31.0tv32 hours1-0/+29
|
* populate: name the target alike in all messagestv21 hours2-29/+37
| | | | | | | | | | | | | What populate prints names paths on the target in different ways: populating as host:path, also for the local target, pruning and the errors about the sentinel by targetName, and replacing a symlink or a Git work tree by the path alone. All of them use targetName now: the path, prefixed by the host if the target isn't local, and by the user if that isn't root. Replacing a symlink or a Git work tree quotes the path and ends with an ellipsis, like populating and pruning do. The tests expect the names that way, and rsync-log also expects the line naming the source being populated.
* populate: reduce log spamtv22 hours1-2/+2
|
* populate: name the target's user in messages unless it's roottv29 hours2-8/+17
| | | | | | | Messages naming the target, i.e. pruning's lines and the errors about the sentinel, name the user as well, as user@host:path, unless the user is root, the usual one, so deploying as another user, e.g. with sudo, shows.
* populate prune: name the target by its host, like populate's messagestv29 hours2-26/+19
| | | | | | | | | Pruning names the target as user@host:path, like rsync's destination, but nothing else populate prints names the user: the errors about the sentinel name the host, and the commands it runs on the target name the host, with the user as ssh's option. Pruning names the target like those do, as host:path, and the errors about the sentinel use the same function for it.
* populate prune: name the target like populating doestv30 hours2-9/+17
| | | | | | | Pruning prints the path of each entry it removes, without the target's user and host, while rsync, when populating, names the target as user@host:path. Pruning names it the same way, so the lines say which machine they're about.
* tests: check the canaries' modification times and new neighbourstv30 hours1-5/+9
| | | | | | | The canary manifest records modification times, so changing a canary's time shows, as rsync does to a hard-linked file it takes for unchanged. And entries beside the target directories whose names start like the canaries' count as canaries, so a file created next to one shows, too.
* populate: fail on inputs drawn that the flake doesn't havetv30 hours4-3/+125
| | | | | | | | | | | | | | | Nix ignores an input that --override-input names but the flake doesn't have, with a warning, so a misspelt input name in inputs draws nothing, and the source keeps using its own input. Before changing anything, populate locks the flake of each source that draws inputs, with an empty flake in place of each input drawn, which may be unreachable, and fails, naming the source and the input, unless the flake has each input drawn. Below an input drawn from a source, the path continues in the flake of that source. An edge case misspells an input of the source's flake, and one below an input drawn from a source, both ways of drawing inputs.
* populate: never write through files or change hard links elsewheretv30 hours3-13/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | Populate writes some files in place, and makes files writable, which reaches beyond the target directory where such a file is a symlink or a hard link to one elsewhere: - The key of a derived copy gets written to a new file, which then replaces it, instead of written through a symlink at its place. - Lock files get copied to a new file before Nix rewrites them, instead of to flake.lock.populate, which a source may contain as a symlink. - Before rsync syncs a source, each file with other hard links gets replaced by a copy of its own. rsync fixes the mode and time of a file it takes for unchanged in place, which changes every hard link to it: one elsewhere, e.g. in /etc/nixos, or a backup's. - Removing state makes only directories writable, which is what removing what's in them needs, instead of everything, including files that are hard links to ones elsewhere. - The sentinel file only gets created if it's missing, instead of touched on every run. - A link only counts as pointing into the store if its resolved path is in the store, so a link like /nix/store/../.. gets replaced. Safety scenarios cover a symlink in place of a key, symlinks in a source named like the lock file's copies, a hard link to a read-only file outside in the target directory, and in stale state.
* populate git: refuse work trees whose .git names places elsewheretv30 hours4-3/+194
| | | | | | | | | | | | | | | | | | | | | | | | | | | | A .git directory names places elsewhere in more ways than being a symlink or a file: core.worktree, symlinks to a configuration, to objects, or to refs elsewhere, a commondir file naming another repository, and linked work trees. Git's commands follow them, so fetching, resetting, setting the remote's URL, and cleaning change what is elsewhere: with core.worktree, git clean removes everything in the work tree it names. And Git runs commands the configuration names, like core.fsmonitor, and hooks. Populate only uses a work tree whose .git directory has no symlinks, no commondir file, no linked work trees, and only configuration that git clone and populate write, where submodules' repositories may have their work tree in the work tree. Otherwise populating fails with an error telling what's wrong, unless the new option reclone replaces the work tree by a fresh clone. Git runs without hooks and without fsmonitor in any case. The work tree also gets reset to the commit if HEAD is there already: a lock file rewritten in place, or created and staged by Nix, stays otherwise once the source stops drawing inputs. A safety scenario refuses work trees with core.worktree, a symlinked configuration, a commondir file, and core.fsmonitor, replaces them with reclone, and checks that a hook doesn't run. A git scenario stops drawing inputs and checks that the work tree is clean.
* tests: move combinations no code path tells apart to exhaustive teststv31 hours3-21/+72
| | | | | | | | | | | | | | | | | | | | | The matrices cross every option of a source type with every other one, also where the code handling one never looks at the other. Scenarios for such combinations are exhaustive, and run in their own tests, which nix-build tests doesn't build, but nix-build tests -A exhaustive does: - file and git: drawing inputs combined with rsync's or Git's options, since drawing inputs happens after delivery. - flake: materialize combined with how the flake gets there, except for flakes in Git work trees and in subdirectories, since materializing copies what got delivered. - flake-input: materialize for inputs other than dep. - deploy: /etc/nixos/flake.nix combined with deploying a flake, since it only matters without one. - edge-cases: generated flakes where only delivery matters, and remote targets where drawing inputs alone matters, which happens on the target alike for every kind of target. That's 181 of 707 scenarios.
* tests: check output, test rsync versions, close gaps in what's assertedtv32 hours6-24/+207
| | | | | | | | | | | | | | | | | | | | | | Several checks would pass with the feature they cover broken, or cover it only by chance: - Phases can state what a succeeding populate prints. That covers the warnings of insecureLinks, the lines pruning prints, the one about a replaced symlink, and what rsync logs about each item. - The insecureLinks scenarios take their expectations from the version of rsync, so with an rsync older than 3.5.0 they pass without the check ever running. New tests, rsync-3-4-1 and rsync-3-5-0, run them with these versions, which tests/rsync.nix builds from their release tarballs, with ACLs and extended attributes, which backups use. - Pruning has a leftover upload to remove, and a source that stops having inputs to root leaves no empty directory of roots. - A full clone has every commit from the start, so the fetch for a commit it lacks never runs. A git scenario deploys a commit that appears on the server after cloning. - The jq test also runs the Nix function that finds a flake's subdirectory on the target, over both shapes of nix flake metadata's output.
generated by cgit v1.3.1 (git 2.54.0) at 2026-11-14 10:08:48 +0000