* new .zig-cache subdirectory: 'v'
- stores coverage information with filename of hash of PCs that want
coverage. This hash is a hex encoding of the 64-bit coverage ID.
* build runner
* fixed bug in file system inputs when a compile step has an
overridden zig_lib_dir field set.
* set some std lib options optimized for the build runner
- no side channel mitigations
- no Transport Layer Security
- no crypto fork safety
* add a --port CLI arg for choosing the port the fuzzing web interface
listens on. it defaults to choosing a random open port.
* introduce a web server, and serve a basic single page application
- shares wasm code with autodocs
- assets are created live on request, for convenient development
experience. main.wasm is properly cached if nothing changes.
- sources.tar comes from file system inputs (introduced with the
`--watch` feature)
* receives coverage ID from test runner and sends it on a thread-safe
queue to the WebServer.
* test runner
- takes a zig cache directory argument now, for where to put coverage
information.
- sends coverage ID to parent process
* fuzzer
- puts its logs (in debug mode) in .zig-cache/tmp/libfuzzer.log
- computes coverage_id and makes it available with
`fuzzer_coverage_id` exported function.
- the memory-mapped coverage file is now namespaced by the coverage id
in hex encoding, in `.zig-cache/v`
* tokenizer
- add a fuzz test to check that several properties are upheld
Compilation errors now report a failure on rebuilds triggered by file
system watches.
Compiler crashes now report failure correctly on rebuilds triggered by
file system watches.
The compiler subprocess is restarted if a broken pipe is encountered on
a rebuild.
Changes the `make` function signature to take an options struct, which
additionally includes `watch: bool`. I intentionally am not exposing
this information to configure phase logic.
Also adds global zig cache to the compiler cache prefixes.
Closes#20600
Updates the build runner to unconditionally require a zig lib directory
parameter. This parameter is needed in order to correctly understand
file system inputs from zig compiler subprocesses, since they will refer
to "the zig lib directory", and the build runner needs to place file
system watches on directories in there.
The build runner's fanotify file watching implementation now accounts
for when two or more Cache.Path instances compare unequal but ultimately
refer to the same directory in the file system.
Breaking change: std.Build no longer has a zig_lib_dir field. Instead,
there is the Graph zig_lib_directory field, and individual Compile steps
can still have their zig lib directories overridden. I think this is
unlikely to break anyone's build in practice.
The compiler now sends a "file_system_inputs" message to the build
runner which shares the full set of files that were added to the cache
system with the build system, so that the build runner can watch
properly and redo the Compile step. This is implemented for whole cache
mode but not yet for incremental cache mode.
and add file system watching integration.
`addDirectoryWatchInput` now returns a `bool` which helps remind the
caller to
1. call addDirectoryWatchInputFromPath on any derived paths
2. but only if the dependency is not already captured by a step
dependency edge.
The make function now recursively walks all directories and adds the
found files to the cache hash rather than incorrectly only adding the
directory name to the cache hash.
closes#20571
And use it to implement InstallDir Step watch integration.
I'm not seeing any events triggered when I run `mkdir` in the watched
directory, however, and I have not yet figured out why.
This has been planned for quite some time; this commit finally does it.
Also implements file system watching integration in the make()
implementation for UpdateSourceFiles and fixes the reporting of step
caching for both.
WriteFile does not yet have file system watching integration.
I'm still learning how the fanotify API works but I think after playing
with it in this commit, I finally know how to implement it, at least on
Linux. This commit does not accomplish the goal but I want to take the
code in a different direction and still be able to reference this point
in time by viewing a source control diff.
I think the move is going to be saving the file_handle for the parent
directory, which combined with the dirent names is how we can correlate
the events back to the Step instances that have registered file system
inputs. I predict this to be similar to implementations on other
operating systems.
Now there is `captureChildProcess` which gives access to the
`std.process.Child.RunResult`, which is useful for accessing the stdout.
It also accepts and passes an optional `std.Progress.Node` to the child.
* `doc/langref` formatting
* upgrade `.{ .path = "..." }` to `b.path("...")`
* avoid using arguments named `self`
* make `Build.Step.Id` usage more consistent
* add `Build.pathResolve`
* use `pathJoin` and `pathResolve` everywhere
* make sure `Build.LazyPath.getPath2` returns an absolute path
This also makes a long-overdue change of extracting common state from
Build into a shared Graph object.
Getting the semantics right for these flags turned out to be quite
tricky. In the end it works like this:
* The override only happens when the target is fully native, with no
additional query parameters, such as versions or CPU features added.
* The override affects the resolved Target but leaves the original Query
unmodified.
* The "is native?" detection logic operates on the original, unmodified
query. This makes it possible to provide invalid host target
information, causing confusing errors to occur. Don't do that.
There are some minor breaking changes to std.Build API such as the fact
that `b.zig_exe` is now moved to `b.graph.zig_exe`, as well as a handful
of other similar flags.
* Specifically recognize stderr as a different concept than an error
message in Step results.
* Display it differently when only stderr occurs but the build proceeds
successfully.
closes#18473
Justification: exec, execv etc are unix concepts and portable version
should be called differently.
Do no touch non-Zig code. Adjust error names as well, if associated.
Closes#5853.
* introduce LazyPath.cwd_relative variant and use it for --zig-lib-dir. closes#12685
* move overrideZigLibDir and setMainPkgPath to options fields set once
and then never mutated.
* avoid introducing Build/util.zig
* use doc comments for deprecation notices so that they show up in
generated documentation.
* introduce InstallArtifact.Options, accept it as a parameter to
addInstallArtifact, and move override_dest_dir into it. Instead of
configuring the installation via Compile step, configure the
installation via the InstallArtifact step. In retrospect this is
obvious.
* remove calls to pushInstalledFile in InstallArtifact. See #14943
* rewrite InstallArtifact to not incorrectly observe whether a Compile
step has any generated outputs. InstallArtifact is meant to trigger
output generation.
* fix child process evaluation code handling of `-fno-emit-bin`.
* don't store out_h_filename, out_ll_filename, etc., pointlessly. these
are all just simple extensions appended to the root name.
* make emit_directory optional. It's possible to have nothing outputted,
for example, if you're just type-checking.
* avoid passing -femit-foo/-fno-emit-foo when it is the default
* rename ConfigHeader.getTemplate to getOutput
* deprecate addOptionArtifact
* update the random number seed of Options step caching.
* avoid using `inline for` pointlessly
* avoid using `override_Dest_dir` pointlessly
* avoid emitting an executable pointlessly in test cases
Removes forceBuild and forceEmit. Let's consider these additions separately.
Nearly all of the usage sites were suspicious.
Specifically this is to make sure llvm data layout generation doesn't
regress. The no emit bin is to allow testing targets that can't
currently be linked. The commented out targets are ones that fail in
the linker anyway when no emit bin is passed.
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
Also get rid of the TTY wrapper struct, which was exlusively used as a
namespace - this is done by the tty.zig root struct now.
detectTTYConfig has been renamed to just detectConfig, which is enough
given the new namespace. Additionally, a doc comment had been added.
The majority of these are in comments, some in doc comments which might
affect the generated documentation, and a few in parameter names -
nothing that should be breaking, however.
Previously this code asserted that a fifo's readable length was greater
than or equal to the length of its readable slice, which was an invalid
assertion.
This code avoids making that assumption.
* Remove some functions that are no longer needed since
EmulateableRunStep is gone.
* Add removeEnvironmentVariable function.
* Support printing environment variables in --verbose mode.
std.Build.addTest creates a CompileStep as before, however, this kind of
step no longer actually runs the unit tests. Instead it only compiles
it, and one must additionally create a RunStep from the CompileStep in
order to actually run the tests.
RunStep gains integration with the default test runner, which now
supports the standard --listen=- argument in order to communicate over
stdin and stdout. It also reports test statistics; how many passed,
failed, and leaked, as well as directly associating the relevant stderr
with the particular test name that failed.
This separation of CompileStep and RunStep means that
`CompileStep.Kind.test_exe` is no longer needed, and therefore has been
removed in this commit.
* build runner: show unit test statistics in build summary
* added Step.writeManifest since many steps want to treat it as a
warning and emit the same message if it fails.
* RunStep: fixed error message that prints the failed command printing
the original argv and not the adjusted argv in case an interpreter
was used.
* RunStep: fixed not passing the command line arguments to the
interpreter.
* move src/Server.zig to std.zig.Server so that the default test runner
can use it.
* the simpler test runner function which is used by work-in-progress
backends now no longer prints to stderr, which is necessary in order
for the build runner to not print the stderr as a warning message.
* std.zig.ErrorBundle: support rendering options for whether to include
the reference trace, whether to include the source line, and TTY
configuration.
* build runner: don't print progress in dumb terminals
* std.Build.CompileStep:
- add a way to expect compilation errors via the new `expect_errors`
field. This is an advanced setting that can change the intent of
the CompileStep. If this slice has nonzero length, it means that
the CompileStep exists to check for compile errors and return
*success* if they match, and failure otherwise.
- remove the object format parameter from `checkObject`. The object
format is known based on the CompileStep's target.
- Avoid passing -L and -I flags for nonexistent directories within
search_prefixes. This prevents a warning, that should probably be
upgraded to an error in Zig's CLI parsing code, when the linker
sees an -L directory that does not exist.
* std.Build.Step:
- When spawning the zig compiler process, takes advantage of the new
`std.Progress.Node.setName` API to avoid ticking up a meaningless
number at every progress update.
The problem is that one may execute too many subprocesses concurrently
that, together, exceed an RSS value that causes the OOM killer to kill
something problematic such as the window manager. Or worse, nothing, and
the system freezes.
This is a real world problem. For example when building LLVM a simple
`ninja install` will bring your system to its knees if you don't know
that you should add `-DLLVM_PARALLEL_LINK_JOBS=1`.
In particular: compiling the zig std lib tests takes about 2G each,
which at 16x at once (8 cores + hyperthreading) is using all 32GB of my
RAM, causing the OOM killer to kill my window manager
The idea here is that you can annotate steps that might use a high
amount of system resources with an upper bound. So for example I could
mark the std lib tests as having an upper bound peak RSS of 3 GiB.
Then the build system will do 2 things:
1. ulimit the child process, so that it will fail if it would exceed
that memory limit.
2. Notice how much system RAM is available and avoid running too many
concurrent jobs at once that would total more than that.
This implements (1) not with an operating system enforced limit, but by
checking the maxrss after a child process exits.
However it does implement (2) correctly.
The available memory used by the build system defaults to the total
system memory, regardless of whether it is used by other processes at
the time of spawning the build runner. This value can be overridden with
the new --maxrss flag to `zig build`. This mechanism will ensure that
the sum total of upper bound RSS memory of concurrent tasks will not
exceed this value.
This system makes it so that project maintainers can annotate
problematic subprocesses, avoiding bug reports from users, who can
blissfully execute `zig build` without worrying about the project's
internals.
Nobody's computer crashes, and the build system uses as much parallelism
as possible without risking OOM. Users do not need to unnecessarily
resort to -j1 when the build system can figure this out for them.
