| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
pkgs/populate/README.md documents populate on its own: building it,
what it needs, its command line, the sources as JSON and in Nix with
its Nix library, the target directory, the source types, flake inputs
from sources, what it prints, and its tests. The sections about the
target directory, the source types, and flake inputs from sources move
there from krops' README, which links to them from sections of the same
names, and from the first mention of populate. docs/design.md keeps
how populate works.
krops' README describes krops' own tests, and CHANGELOG what populate
being a program of its own means for krops.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
populate's tests are in pkgs/populate/tests, independent of krops.
They run the program, with the sources as JSON from populate's Nix
library, and target strings for the remote targets, leaving the user
and whether the target is local to populate, so they cover its command
line too. Phases expecting populate to refuse its arguments, e.g. a
source name or a target path that can't be used, expect the program's
refusal, with refused, the message, since nothing checks them during
evaluation there.
builder.nix is what builds the scenarios, which krops' tests use as
well, with a function running the phases, and whether that checks the
arguments during evaluation. vm.nix takes a module for both machines,
for what else the scenarios run, and fixtures.nix and smoke.nix are
what both use of the fixtures and the smoke test.
krops' tests are deploy, writeDeploy's arguments to nixos-rebuild, with
the stand-in for nixos-rebuild in the module, and populate, populate's
scenarios that expect refusals, which pkgs.populate makes evaluation
fail with, and those of populate's smoke test, all through
pkgs.populate with target attribute sets as mkTarget returns them.
-A only selects from all of populate's scenarios and krops' own.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
populate is a program of its own, with a Nix library, in pkgs/populate,
which krops uses, and which doesn't use krops:
- default.nix builds the program, with nix-build there or callPackage,
as pkgs.populate-unwrapped.
- lib is populate's Nix library: the types of sources and the POSIX
types they use, evalSource, flakeFollows, hasGeneratedFlake, and
derivesFlake, taking nixpkgs' lib. krops' lib takes them from there,
so it provides them as before, and so does lib/types.
pkgs.populate, the function from populate's arguments to a script
running the program, is krops' own, in pkgs/krops/populate.nix, with
checks.nix next to it. It takes the program as populate-unwrapped,
which callPackage passes from the overlay; the flake, whose nixpkgs
lacks it, passes its own.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The program takes populate's arguments on its command line, like other
programs: populate [OPTION]... SOURCES [TARGET], with the sources as
JSON in the file SOURCES, or on standard input for -. TARGET has the
syntax of krops' lib's mkTarget, [USER@][HOST][:PORT][PATH], parsed the
same way, and what it leaves out gets filled in when populate runs, as
mkTarget would, from LOGNAME and the machine's name. Options set what
the syntax can't express, or override its parts: --sudo, --ssh-arg,
--user, --host, --port, --path, and --local or --remote for whether the
target is the machine populate runs on, which populate otherwise
decides like isLocalTarget. The sources carry no version.
pkgs.populate passes a target attribute set, as mkTarget returns it,
part by part, with --local or --remote as evaluation decides, so
nothing changes for krops. A target string goes to the program as it
is.
The program also refuses target hosts and users starting with -, which
ssh would take for options, now that a command line can give them.
|
| |
|
|
|
|
|
|
| |
The filter for the program's sources took the paths it gets relative
to toString ./., which they needn't start with, e.g. if a symlink leads
to the directory, so nothing matched, and the build found no
Cargo.lock. It goes by the names of what it leaves out now: the Nix
files, the docs, .gitignore, and cargo's target directory.
|
| |
|
|
|
|
|
| |
git sources refused work trees with hooks, but populate's Git doesn't
run hooks, so git-foreign's work tree with a hook is fine to use, as it
expects. Only Nix's Git runs them, in a git+file: flake, so drawing
inputs refers to a work tree with hooks as path: instead.
|
| |
|
|
|
|
|
|
| |
DerivedState::parse reads what derived_state prints, and key_text
builds the text whose hash is a derived copy's key, the same as before,
both with unit tests, instead of decoding fields by position inside
Flakes::derive. Fields from the target that don't end in a null byte
fail, instead of getting dropped.
|
| |
|
|
|
|
|
|
| |
For specs whose target's local is null, the program decides whether the
target is local like krops' lib's isLocalTarget: an empty LOGNAME or
HOSTNAME counts as unset, as with lib.maybeEnv, and /etc/hostname's
labels can be of any length, as with lib.types.label. is_local_with,
pure, has unit tests.
|
| |
|
|
|
|
|
|
|
|
|
| |
The shell implementation runs under set -e, so it exits with the status
of the command that failed, e.g. 255 if ssh can't reach the target,
which wrappers retrying on connection failures rely on, or 128 if git
fetch fails. The program does the same, instead of exiting with 1, and
with 128 plus the signal for a command a signal ended.
A phase's expect.status is the exit status it expects populate to fail
with; git-status checks git fetch's.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
The program checks that each source's settings are those
lib/types/populate.nix defines for its type, with values of the kinds
it defines, for specs from elsewhere than default.nix: a misspelt
setting fails, as design.md promises, instead of being ignored, and so
does a setting the program would treat as its default because of its
kind. This subsumes the check of absolute paths.
ln gets a symlink source's target after --, so it can't be taken for an
option either.
|
| |
|
|
|
|
|
|
|
|
| |
Looking for .rsync-filter files that apply to a file source, a failing
find counted as finding none, and only happened with rsync 3.5.0 and
later. Like in the shell implementation, find runs before each
transfer with filter files, and its failure stops populate with find's
message, e.g. for a source that's missing, or contains a directory find
can't read, which would also have switched off the check of
insecureLinks. A signal during find takes effect right after it.
|
| |
|
|
|
|
| |
rsync gives sizes in units of 1000 for two apostrophes, and in units of
1024 for three, which the shell implementation asks for. rsync-units
checks it.
|
| |
|
|
|
|
|
|
|
| |
The program checks that the paths of file, pass, and pipe sources are
absolute, as lib/types/populate.nix demands, for specs from elsewhere,
so none can be taken for an option, e.g. of rsync.
A cycle of sources drawing inputs from each other gets named with the
way into it, like checks.nix names it.
|
| |
|
|
|
|
|
|
| |
A pipe source's command is an absolute path, which a shell runs as a
script if it isn't a program, e.g. lacks #!. The program ran it
directly, which fails for such a command.
pipe-script runs a pipe command without #!.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Writing the lock file of a git+file: flake, Nix runs Git, with the
repository's configuration and hooks, which may run commands. A .git
in a source's directory can come from anywhere, e.g. one planted in a
materialized flake, or one a file source keeps. So drawing inputs
refers to a flake as git+file: only if git_problems finds nothing wrong
with its .git directory, the way git sources check their work trees,
and else as path:.
lock-foreign-git plants a .git in a materialized flake and in a file
source that keeps it, whose configuration names an fsmonitor command.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
git submodule update uses the .git of a submodule as it finds it in the
work tree: a repository of its own, with configuration that runs
commands, or a file naming a repository elsewhere. So git_problems
also requires each .git below the work tree to be a file naming a
repository in .git/modules. And it refuses hooks in .git/hooks, which
populate's Git doesn't run, but Nix's does, writing the lock file of a
flake in a git source.
git-submodule plants a submodule's .git in the work tree: a repository
of its own whose configuration names a filter, and a file naming a
repository elsewhere. The fixture withsub has gitsrc as its submodule
sub.
|
| |
|
|
|
|
|
|
|
|
| |
A git source's url and ref are strings, and git clone, git remote, and
git fetch took ones starting with - for options, e.g. --upload-pack,
which runs a command where the repository is local. They come after
-- now, and so does a pass source's name for git log and pass show.
git-ref-option gives a ref that git fetch would take for
--upload-pack.
|
| |
|
|
|
|
| |
New users learn what the deploying machine and the target need,
including nixpkgs 23.11 or later for building populate, and developers
where populate's program, its description, and its unit tests are.
|
| |
|
|
|
|
|
|
| |
dash, jq, and implementation, which overrides of the shell
implementation set, come through ..., because callPackage sets named
arguments, so populate can tell they're set, and warns that they're
unused. Other arguments populate doesn't know fail, as they do for
named arguments.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Populate is the program in Rust only. pkgs.populate checks its arguments,
and returns a script running the program with them as JSON. The
arguments it took for the shell implementation, dash, jq, and
implementation, stay accepted and unused, so overrides setting them keep
working.
Copying flakes to remote targets using sudo gets refused during
evaluation, as before, by checks.nix, and by the program for specs from
elsewhere.
The jq test goes with the jq programs; the program's unit tests cover
their equivalents.
|
| |
|
|
|
|
| |
The program needs Rust 1.71, the oldest version its dependencies support,
which Cargo.toml states, so clippy points out what newer versions add.
The lock file is of version 3, which Cargo reads from version 1.53 on.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The program decides how to draw inputs and derive copies of flake and
flake-input sources itself. The target runs fixed snippets, which send
back what it needs to know: where paths resolve to and what's there, the
lock files of flakes as nix flake metadata prints them, and what tells
whether a derived copy needs to be made again. The program checks that
drawn inputs exist in the lock files, computes the keys of derived
copies, and writes generated flakes.
Derived copies get the same keys as with the shell implementation, so
switching implementations doesn't make them again.
|
| |
|
|
|
|
|
|
|
|
|
| |
src/flake_input.rs reads what nix flake metadata and nix flake archive
print as JSON itself: it selects the flake or input, decides what the
target fetches or gets copied, and computes the store paths to root,
which are what selected.jq, selfIsRemote.jq, selfLocked.jq, and the Nix
expressions of the shell implementation do there. On the target, fixed
snippets in sh::flake run Nix, getting values as positional parameters,
and send JSON or store paths back. The jq test's cases are unit tests of
the Rust equivalents.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The program delivers flake and flake-input sources, draws inputs from
sources, and removes state that sources don't need anymore, so it does
everything the shell implementation does, and passes all scenarios.
src/flake.rs builds the same shell code as shell.nix generates for
these, with every value quoted, so both implementations behave alike
while that code moves to Rust piece by piece.
The ssh scripts in the temporary directory run the ssh that was in PATH
when populate started, so one can be ssh in PATH for nix copy. The
other one is for nix copy's ssh:// store, without the control socket.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The program delivers derivation, file, git, pass, passage, pipe, and
symlink sources, makes backups, and prunes, as the shell implementation
does, with the same messages. Commands on the target are fixed shell
snippets in src/sh.rs, which get values as positional parameters, run
directly for local targets and over ssh, sharing one master connection,
for remote ones. rsync's transfers and its check for .rsync-filter files
it can't read are in src/rsync.rs. Given flake or flake-input sources, or
sources drawing inputs from others, the program fails before changing
anything, saying that these aren't implemented yet.
SIGHUP, SIGINT, and SIGTERM make populate stop once the command it waits
for ends, stop the ssh master, remove its temporary directory, and exit
with 128 plus the signal's number.
The spec's target gets local, which rust.nix sets during evaluation, like
the other functions of krops decide, and which the program decides when
it runs if it's null. The spec comes on file descriptor 3, so the
commands populate runs keep standard input, and the tools populate runs
come first in PATH.
|
| |
|
|
|
|
|
| |
pkgs/populate/docs/design.md describes populate's parts, the program's
invocation, the spec, the checks, what populate prints, its safety
guarantees, how it runs commands, how to test changes, and what the
program does so far.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pkgs.populate takes implementation: "shell", the default, generates a
script from populate's arguments, and "rust" passes them as JSON to the
program in pkgs/populate, built from the crate there by program.nix.
The program checks the spec as checks.nix does during evaluation, with the
same messages, and reports that delivering sources isn't implemented yet.
rustProgram replaces the program, e.g. with one built some other way.
checks.nix holds the checks of populate's arguments during evaluation.
Both implementations check the names of sources and the target path with
it, and the Rust one's wrapper also checks drawing inputs, which the shell
implementation checks where it draws them.
nix-build tests --argstr implementation rust runs the scenarios with the
Rust implementation. They expect it to refuse copying flakes to targets
using sudo when it runs, not during evaluation, because only then it
decides whether the target is local.
|
| |
|
|
|
|
| |
The scripts populate writes with writeDash get /* sh */ markers like the
other shell code, and the overlay doesn't import krops' lib, which it
doesn't use.
|
| |
|
|
|
|
|
| |
rsync sets the modification time of each directory whose contents change,
and logs a line for it. Nothing relies on these times, nor on the times of
symlinks, so --omit-dir-times and --omit-link-times leave them alone, and
the log lists only what got transferred or deleted.
|
| |
|
|
|
|
|
|
|
| |
With allocateTTY, nixos-rebuild runs with a terminal, so it shows colours
and progress: on remote targets, ssh gets -t, which allocates a terminal
if its own input is one, and on the local target, expect's unbuffer
provides one.
The nixos-rebuild stand-in records whether its output is a terminal.
|
| |
|
|
|
|
|
|
| |
A ref that consists of 4 to 40 hex digits is a commit, which populate
fetches only if the work tree lacks it, unless fetchAlways is set. Any
other ref is a branch or tag, which populate fetches on every run,
resetting the work tree to the commit fetched. Resetting to the name
itself resolves to a local branch, which fetching doesn't move.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
What populate prints names paths on the target in different ways:
populating as host:path, also for the local target, pruning and the
errors about the sentinel by targetName, and replacing a symlink or a
Git work tree by the path alone. All of them use targetName now: the
path, prefixed by the host if the target isn't local, and by the user if
that isn't root. Replacing a symlink or a Git work tree quotes the path
and ends with an ellipsis, like populating and pruning do.
The tests expect the names that way, and rsync-log also expects the line
naming the source being populated.
|
| | |
|
| |
|
|
|
|
|
| |
Messages naming the target, i.e. pruning's lines and the errors about
the sentinel, name the user as well, as user@host:path, unless the user
is root, the usual one, so deploying as another user, e.g. with sudo,
shows.
|
| |
|
|
|
|
|
|
|
| |
Pruning names the target as user@host:path, like rsync's destination,
but nothing else populate prints names the user: the errors about the
sentinel name the host, and the commands it runs on the target name the
host, with the user as ssh's option. Pruning names the target like
those do, as host:path, and the errors about the sentinel use the same
function for it.
|
| |
|
|
|
|
|
| |
Pruning prints the path of each entry it removes, without the target's
user and host, while rsync, when populating, names the target as
user@host:path. Pruning names it the same way, so the lines say which
machine they're about.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Nix ignores an input that --override-input names but the flake doesn't
have, with a warning, so a misspelt input name in inputs draws nothing,
and the source keeps using its own input.
Before changing anything, populate locks the flake of each source that
draws inputs, with an empty flake in place of each input drawn, which
may be unreachable, and fails, naming the source and the input, unless
the flake has each input drawn. Below an input drawn from a source, the
path continues in the flake of that source.
An edge case misspells an input of the source's flake, and one below an
input drawn from a source, both ways of drawing inputs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Populate writes some files in place, and makes files writable, which
reaches beyond the target directory where such a file is a symlink or a
hard link to one elsewhere:
- The key of a derived copy gets written to a new file, which then
replaces it, instead of written through a symlink at its place.
- Lock files get copied to a new file before Nix rewrites them, instead
of to flake.lock.populate, which a source may contain as a symlink.
- Before rsync syncs a source, each file with other hard links gets
replaced by a copy of its own. rsync fixes the mode and time of a file
it takes for unchanged in place, which changes every hard link to it:
one elsewhere, e.g. in /etc/nixos, or a backup's.
- Removing state makes only directories writable, which is what
removing what's in them needs, instead of everything, including files
that are hard links to ones elsewhere.
- The sentinel file only gets created if it's missing, instead of
touched on every run.
- A link only counts as pointing into the store if its resolved path is
in the store, so a link like /nix/store/../.. gets replaced.
Safety scenarios cover a symlink in place of a key, symlinks in a source
named like the lock file's copies, a hard link to a read-only file
outside in the target directory, and in stale state.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A .git directory names places elsewhere in more ways than being a
symlink or a file: core.worktree, symlinks to a configuration, to
objects, or to refs elsewhere, a commondir file naming another
repository, and linked work trees. Git's commands follow them, so
fetching, resetting, setting the remote's URL, and cleaning change what
is elsewhere: with core.worktree, git clean removes everything in the
work tree it names. And Git runs commands the configuration names, like
core.fsmonitor, and hooks.
Populate only uses a work tree whose .git directory has no symlinks, no
commondir file, no linked work trees, and only configuration that git
clone and populate write, where submodules' repositories may have their
work tree in the work tree. Otherwise populating fails with an error
telling what's wrong, unless the new option reclone replaces the work
tree by a fresh clone. Git runs without hooks and without fsmonitor in
any case.
The work tree also gets reset to the commit if HEAD is there already: a
lock file rewritten in place, or created and staged by Nix, stays
otherwise once the source stops drawing inputs.
A safety scenario refuses work trees with core.worktree, a symlinked
configuration, a commondir file, and core.fsmonitor, replaces them with
reclone, and checks that a hook doesn't run. A git scenario stops
drawing inputs and checks that the work tree is clean.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An input below an input that gets drawn from a source is an input of the
flake in that source. If that flake draws the input on the way from
another source itself, the path goes on in the flake of that one, and
so forth. Populate follows the path one source deep only, so
x.inputs.dep = { follows = "b"; inputs.p.inputs.z.follows = "c"; } has
no effect if b draws p from a source: Nix ignores overrides below an
input that follows another one, without warning.
Populate follows such paths through the sources, for generated flakes as
well as for rewritten lock files. Where the way leads through a
generated flake, it names the generated flake's inputs, which are its
source and the sources that one draws from. So a source with a rewritten
lock file can also draw an input below one it draws from a source with a
generated flake.
Edge cases cover an input two sources deep, both ways of drawing inputs,
and a source with a rewritten lock file drawing from one with a generated
flake.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
nixos-rebuild gets generated flakes as path: URLs, but other flakes by
their bare paths, which Nix takes for the enclosing Git work tree, if the
target directory lies in one. A file source there fails to deploy,
because the work tree doesn't track it. Every flake gets deployed by a
URL: a Git work tree as git+file:, with submodules if it has any, like
populate refers to it, and anything else as path:.
The path gets resolved on the build target, because Nix refuses URLs
with symlinks on the way, which the target directory may be, and
percent-encoded.
--build-host and --target-host name no user if the target has none,
instead of failing to evaluate.
The deploy test also deploys flakes from file and git sources. Its
scenarios evaluate without LOGNAME.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Names and paths with characters that mean something to shells, to find,
or in URLs get handled as what they are, or refused:
- Pruning, and removing state that sources don't need anymore, list
entries with find instead of a loop reading names, which strips blanks
at the ends of names and splits names at newlines. So an entry named
"trail " gets pruned instead of reported as pruned and left.
- Pruning doesn't make entries writable before removing them, which
would change the mode of files elsewhere that are hard links to files
therein.
- Source names are portable file names: letters, digits, and ._-, not
starting with -. nixpkgs' writers accept nothing else in the names of
the scripts populating the sources, which contain the source names, and
fail with an assertion that doesn't mention the source.
- Paths get percent-encoded where they go into URLs for Nix, which
refuses blanks there and takes ? and # for delimiters.
- rsync gets the target path protected from the target's shell
(--protect-args), which expands wildcards in it otherwise: a file source
delivered to /var/src/a?c ends up in /var/src/abc if that exists.
A safety scenario deploys every kind of state to a target path with a
blank, wildcards, #, and %. An edge case prunes entries with blanks at
the ends of their names, with a newline, and with wildcards, and the
scenario pruning links also prunes hard links to a read-only file
outside, and lists what's left in the C locale, where ls sorts .populate
first. The scenario with unusual source names also draws inputs from a
source named self.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The shell doesn't run the EXIT trap when a signal ends it, so an
interrupted deploy leaves the ssh master, for an hour, and populate's
temporary directory. Populate traps HUP, INT, and TERM and exits, which
runs the EXIT trap.
The shell runs a trap once the command it waits for ends. A signal to
the process group, like the one of an interrupt key, of a closed terminal,
or of timeout, ends that command as well. A signal to populate alone
takes effect after the step that is running.
Phases of scenarios can end populate after some seconds. An edge case
does so while a pipe source's command sleeps, and checks that neither the
ssh master nor the temporary directory remain.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rsync reads .rsync-filter files in the directories above a source as
well, and their rules apply to the transfer. The check for files that
rsync 3.5.0 and later can't read only starts if the source itself
contains one, so a subdirectory of a repository with a filter file at
its top gets synced without it, silently. The check also starts for
filter files above the source.
The check's dry run is the transfer it stands for, as far as filtering
goes:
- It gets the source's exclude and filters, so it doesn't enter
directories the transfer skips.
- It doesn't run for transfers that apply no .rsync-filter files, i.e.
uploads of flakes and patches.
- A filter file the user may not read anyway doesn't count: rsync fails
on its own when it gets to transfer that file, and --insecure-links
changes nothing about it.
- rsync prints bytes it takes for unprintable as \#ooo, which depends on
the locale. File names get decoded before checking that they exist, so
files below, e.g., directories with umlauts in their names don't pass
for missing in the C locale.
- If rsync doesn't tell which filter files it reads at all, populate
warns that it can't tell either.
The README mentions that --insecure-links also affects the way to the
target path on local targets, and that only rsync 3.5.0 ignores filter
files below directories that may be entered but not read.
Phases of scenarios can set environment variables for populate. New
file scenarios sync a subdirectory whose filter file is above it, and a
source with a filter file below a directory with an umlaut in its name,
in the C locale.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
inputs
Delivering flakes and flake inputs gets more robust:
- With fetchOnTarget, the inputs get collected before the pipeline that
links them, so populate fails if the target can't fetch one, instead of
linking the flake without its inputs and succeeding.
- What the lock file pins gets taken from the store if it's there: the
store path is the one of a directory named source with the locked hash,
which nix-store --print-fixed-path computes. builtins.fetchTree
contacts where the input comes from even then, so redeploying unchanged
sources fails without that if a server is unreachable.
- With fetchOnTarget, a delivered input that is itself locked to a local
path the target lacks fails with populate's error naming the input, like
its inputs do.
- Inputs that get drawn from other sources are left out, with all they
pin: Nix archives an empty flake in their place. So nothing gets
fetched, copied, and rooted that no evaluation uses, and a pin that is
unreachable doesn't matter if it gets replaced anyway.
- flake-input sources get the option requireInputs. By default,
populating fails if the inputs that the input's own lock file pins
can't be delivered, e.g. because the lock file lacks an input its
flake.nix has. With false, populate warns and delivers the input with
what Nix locks now, or else without its inputs.
A remote flake that the target fetches entirely on its own collects its
inputs like everything else does. The Nix function finding its
subdirectory moves next to the jq programs.
New edge cases cover each of these.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Populate replaces what's at the target paths of sources, removes what
sources don't deliver anymore, and prunes. Nothing of that may reach
beyond the target directory, whatever names sources have and whatever the
target directory contains:
- Source names that aren't file names in the target directory don't
evaluate: the empty name, ., .., .populate, and names with slashes or
control characters. E.g. a git source named "" removes the target
directory, and a file source named .. syncs into its parent.
- Target paths that aren't absolute, that are the root directory, or that
contain .. don't evaluate.
- A symlink at the target path of a source gets replaced, not followed,
and populate says so. rsync-based and flake sources do that already.
git sources replace the link as well, and also a work tree whose .git is
a symlink or a file naming a repository elsewhere, instead of fetching,
resetting, and cleaning there. derivation sources replace a link that
doesn't point into the store, instead of failing.
- Populate refuses to use the sentinel, or a state directory therein, if
it is a symlink, instead of changing and removing what it points to.
- What populate removes recursively gets removed as a link if it is one:
chmod -R follows a link named on the command line.
- Flake and flake-input sources that get patched, get their lock file
rewritten, or are materialized replace a symlink at their target path
like every source does, and say so.
- Nix rewrites lock files in place, which changes other hard links to
them as well, e.g. in a backup or outside the target directory, and
what a lock file that is a symlink points to. Populate replaces a lock
file by a copy of its own before rewriting it.
The target directory itself may be a symlink. Nix refuses path: URLs with
symlinks on the way, so the flakes step resolves the paths it hands to
Nix.
Backups work on a sentinel that is still a file, which becomes a
directory, like for everything else that keeps state.
The tests create canaries outside the target directories: a file and a
directory beside them, and a directory elsewhere with a read-only
directory and a Git work tree with uncommitted changes. The check of
every phase of every scenario fails if a canary changed, or if a store
path that the target directory links to doesn't match its hash anymore.
A new test, safety, points links at the canaries: at the target path of
every source type, inside what sources replace, in place of populate's
state, and among what it removes and prunes. It also covers the names
and target paths that don't evaluate, a target directory that is a
symlink, and a lock file with another hard link.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sources drawing inputs from other sources get the lock file of their
flake rewritten with nix flake lock --override-input, so the flake in
the source itself uses the other sources, and needs no generated flake
to be found. file and git sources get their lock file rewritten in
place. flake and flake-input sources get delivered as a copy with the
rewritten lock file, which gets added to the store with nix-store
--add-fixed, and the target path links to that. With rewriteLock set
to false, and for symlink sources, the source stays as it is and gets
a generated flake in .populate/flakes/<name>, which writeDeploy's flake
parameter uses.
flake and flake-input sources get two more options:
- patches: patches on the deploying machine, which get uploaded to
.populate/patches/<name> and applied in order with patch -p1 on the
target, in the directory the target path shows.
- materialize: the target path becomes a writable copy instead of a
link into the store.
Sources that get patched, rewritten, or materialized get delivered to
.populate/derived/<name>/original, which also roots the delivered store
path, and the flakes step derives the target path from there. Copies
get made again only if what they derive from changes, which a key in
.populate/derived/<name> records.
The matrices of the file, flake, flake-input, and git tests cover both
ways of drawing inputs, those of flake and flake-input materialize, the
deploy test a source with a rewritten lock file, and features and
edge cases run the scenarios drawing inputs both ways. New features
cover patches, a patch that doesn't apply, patches in a subdirectory,
reusing copies, and materializing.
|
| |
|
|
|
|
|
|
|
|
|
| |
Flake and flake-input sources take the flake reference as attribute url,
like git sources take their repository, and like inputs in flake.nix:
something.flake = "github:owner/something";
otherthing.flake = {
url = "github:owner/otherthing";
inputs.nixpkgs.follows = "nixpkgs";
};
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
rsync 3.5.0 opens merge files, including per-directory .rsync-filter
files, walking the path one directory at a time, refusing symlinks that
neither root nor the user running rsync owns, and opening each directory
for reading. It treats the files it can't open that way as missing,
without saying so. That happens, e.g., below a home directory with mode
711, which the user may enter but not read, or behind another user's
symlink. Such a source then got synced without its .rsync-filter rules:
excluded files got transferred, and --delete-excluded deleted nothing.
Before syncing with rsync 3.5.0 or later a source containing .rsync-filter
files, populate asks rsync which of them it reads: a dry run with
--debug=FILTER2. If it reports an existing one as missing, the file
source type's new insecureLinks option decides: with null, the default,
populate fails with an error naming the file; with true, rsync reads such
files anyway (--insecure-links); with false, rsync syncs without them.
Both print a warning.
File scenarios sync a source through a symlink owned by nobody, with each
setting.
|
| |
|
|
|
|
| |
rsync logged only the names of the items it transferred. It logs what
changed about each item now (--itemize-changes), along with its size as
transferred and in total.
|
