This flag is used when building stage1 to omit the stage2 backends from
the compiler to save memory on the CI server. It regressed with the
merging of e8813b296bc55a13b534bd9b2a03e1f6af366915 because Value
functions started calling into Sema functions.
The end goal for this build option is to eliminate it.
With this change, we can now bake in entitlements into the binary.
Additionally, I see this as the first step towards full code signature
support which includes baking in Apple issued certificates for
redistribution, etc.
Introduce `Module.ensureFuncBodyAnalyzed` and corresponding `Sema`
function. This mirrors `ensureDeclAnalyzed` except also waits until the
function body has been semantically analyzed, meaning that inferred
error sets will have been populated.
Resolving error sets can now emit a "unable to resolve inferred error
set" error instead of producing an incorrect error set type. Resolving
error sets now calls `ensureFuncBodyAnalyzed`. Closes#11046.
`coerceInMemoryAllowedErrorSets` now does a lot more work to avoid
resolving an inferred error set if possible. Same with
`wrapErrorUnionSet`.
Inferred error set types no longer check the `func` field to determine if
they are equal. That was incorrect because an inline or comptime function
call produces a unique error set which has the same `*Module.Fn` value for
this field. Instead we use the `*Module.Fn.InferredErrorSet` pointers to
test equality of inferred error sets.
Previously, Zig ignored -lgcc_s with a warning that this dependency is
redundant because it is satisfied by compiler-rt. However, sfackler
pointed out that it also provides exception handling functions. So if
Zig sees -lgcc_s on the linker line, it needs to fulfill this dependency
with libunwind.
I also made link_libc inferred to be on if libunwind is linked since
libunwind depends on libc.
In accordance with the requesting issue (#10750):
- `zig test` skips any tests that it cannot spawn, returning success
- `zig run` and `zig build` exit with failure, reporting the command the cannot be run
- `zig clang`, `zig ar`, etc. already punt directly to the appropriate clang/lld main(), even before this change
- Native `libc` Detection is not supported
Additionally, `exec()` and related Builder functions error at run-time, reporting the command that cannot be run
First step towards #10634.
Treating stub files as C++ allows to use zig c++ as a host
compiler for nvcc.
Treating cu files as C++ allow using zig c++ as a host compiler in
CMake. CMake calls the host compiler with -E on a cu file to identify
the compiler.
Using zig c++ to directly compile CUDA code is untested.
This is only relevant for ELF files.
I also fixed a bug where passing a zig source file to `zig cc` would
incorrectly punt to clang because it thought there were no positional
arguments.
After #10656, function pointers are represented with e.g.
`*const fn()void` rather than `fn()void`.
This commit adds code to translate-c to emit different code
depending on whether the output zig source code is intended
to be compiled with stage1 or stage2.
Ideally we will have stage1 and stage2 support the exact same
Zig language, but for now they diverge because I would rather
focus on finishing and shipping stage2 than implementing the
features in stage1.
AstGen:
* rename the known_has_bits flag to known_non_opv to make it better
reflect what it actually means.
* add a known_comptime_only flag.
* make the flags take advantage of identifiers of primitives and the
fact that zig has no shadowing.
* correct the known_non_opv flag for function bodies.
Sema:
* Rename `hasCodeGenBits` to `hasRuntimeBits` to better reflect what it
does.
- This function got a bit more complicated in this commit because of
the duality of function bodies: on one hand they have runtime bits,
but on the other hand they require being comptime known.
* WipAnonDecl now takes a LazySrcDecl parameter and performs the type
resolutions that it needs during finish().
* Implement comptime `@ptrToInt`.
Codegen:
* Improved handling of lowering decl_ref; make it work for
comptime-known ptr-to-int values.
- This same change had to be made many different times; perhaps we
should look into merging the implementations of `genTypedValue`
across x86, arm, aarch64, and riscv.
Instead use the standarized option for communicating the
zig compiler backend at comptime, which is `zig_backend`. This was
introduced in commit 1c24ef0d0b09a12a1fe98056f2fc04de78a82df3.
This exposes a function from stage2 to stage1 to append symbols to automatically export them.
This happends under the following conditions:
- Target is wasm
- User has not provided --export/--rdynamic flags themselves.
This allows Zig code to perform conditional compilation based on a tag
by which a Zig compiler implementation identifies itself.
See the doc comment in this commit for more details.
Doc comments reproduced here:
This function is called by the frontend before flush(). It communicates that
`options.bin_file.emit` directory needs to be renamed from
`[zig-cache]/tmp/[random]` to `[zig-cache]/o/[digest]`.
The frontend would like to simply perform a file system rename, however,
some linker backends care about the file paths of the objects they are linking.
So this function call tells linker backends to rename the paths of object files
to observe the new directory path.
Linker backends which do not have this requirement can fall back to the simple
implementation at the bottom of this function.
This function is only called when CacheMode is `whole`.
This solves stack trace regressions on Windows and macOS because the
linker backends do not observe object file paths until flush().
Instead of juggling GPA-allocated sub_path (and ultimately dropping the
ball, in this analogy), `Compilation.create` allocates an
already-exactly-correct size `sub_path` that has the digest unpopulated.
This is then overwritten in place as necessary and used as the
`emit_bin.sub_path` value, and no allocations/frees are performed for
this file path.
Previously the code asserted source files were already loaded, but this
is not the case when cached ZIR is loaded. Now it will trigger .zig
source code to be loaded for the purposes of hashing the source for
`CacheMode.whole`.
This additionally refactors stat_size, stat_inode, and stat_mtime fields
into using the `Cache.File.Stat` struct.
This fixes a regression in this branch that can be reproduced with the
following steps:
1. `zig build-exe hello.zig`
2. delete the "hello" binary
3. `zig build-exe hello.zig`
4. observe that the "hello" binary is missing
This happened because it was a cache hit, but nothing got copied to the
output directory.
This commit sets CacheMode to incremental - even for stage1 - when the
CLI requests `disable_lld_caching` (this option should be renamed),
resulting in the main Compilation to be repeated (uncached) for stage1,
populating the binary into the cwd as expected.
For stage2 the result is even better: the incremental compilation system
will look for build artifacts to incrementally compile, and start fresh
if not found.
* Logic to check whether a bin file is not emitted is more complicated
in between `Compilation.create` and `Compilation.update`. Fixed the
logic that decides whether to build compiler-rt and other support
artifacts.
* Basically, one cannot inspect the value of `comp.bin_file.emit` until
after update() is called - fixed another instance of this happening
in the CLI.
* In the CLI, `runOrTest` is updated to properly use the result value
of `comp.bin_file.options.emit` rather than guessing whether the
output binary is.
* Don't assume that the emit output has no directory components in
sub_path. In other words, don't assume that the emit directory is the
final directory; there may be sub-directories.
The two CacheMode values are `whole` and `incremental`.
`incremental` is what we had before; `whole` is new.
Whole cache mode uses everything as inputs to the cache hash;
and when a hit occurs it skips everything including linking.
This is ideal for when source files change rarely and for backends that
do not have good incremental compilation support, for example
compiler-rt or libc compiled with LLVM with optimizations on.
This is the main motivation for the additional mode, so that we can have
LLVM-optimized compiler-rt/libc builds, without waiting for the LLVM
backend every single time Zig is invoked.
Incremental cache mode hashes only the input file path and a few target
options, intentionally relying on collisions to locate already-existing
build artifacts which can then be incrementally updated.
The bespoke logic for caching stage1 backend build artifacts
is removed since we now have a global caching mechanism for
when we want to cache the entire compilation, *including* linking.
Previously we had to get "creative" with libs.txt and a special
byte in the hash id to communicate flags, so that when the cached
artifacts were re-linked, we had this information from stage1
even though we didn't actually run it. Now that `CacheMode.whole`
includes linking, this extra information does not need to be
preserved for cache hits. So although this changeset introduces
complexity, it also removes complexity.
The main trickiness here comes from the inherent differences between the
two modes: `incremental` wants a directory immediately to operate on,
while `whole` doesn't know the output directory until the compilation is
complete. This commit deals with this problem mostly inside `update()`,
where, on a cache miss, it replaces `zig_cache_artifact_directory` with a
temporary directory, and then renames it into place once the compilation is
complete.
Items remaining before this branch can be merged:
* [ ] make sure these things make it into the cache manifest:
- @import files
- @embedFile files
- we already add dep files from c but make sure the main .c files make
it in there too, not just the included files
* [ ] double check that the emit paths of other things besides the binary
are working correctly.
* [ ] test `-fno-emit-bin` + `-fstage1`
* [ ] test `-femit-bin=foo` + `-fstage1`
* [ ] implib emit directory copies bin_file_emit directory in create() and needs
to be adjusted to be overridden as well.
* [ ] make sure emit-h is handled correctly in the cache hash
* [ ] Cache: detect duplicate files added to the manifest
Some preliminary performance measurements of wall clock time and
peak RSS used:
stage1 behavior (1077 tests), llvm backend, release build:
* cold global cache: 4.6s, 1.1 GiB
* warm global cache: 3.4s, 980 MiB
stage2 master branch behavior (575 tests), llvm backend, release build:
* cold global cache: 0.62s, 191 MiB
* warm global cache: 0.40s, 128 MiB
stage2 this branch behavior (575 tests), llvm backend, release build:
* cold global cache: 0.62s, 179 MiB
* warm global cache: 0.27s, 90 MiB
The status quo for the `build.zig` build system is preserved in
the sense that, if the user does not explicitly override
`dylib.setInstallName(...);` in their build script, the default
of `@rpath/libname.dylib` applies. However, should they want to
override the default behaviour, they can either:
1) unset it with
```dylib.setIntallName(null);```
2) set it to an explicit string with
```dylib.setInstallName("somename.dylib");```
When it comes to the command line however, the default is not to
use `@rpath` for the install name when creating a dylib. The user
will now be required to explicitly specify the `@rpath` as part
of the desired install name should they choose so like so:
1) with `build-lib`
```
zig build-lib -dynamic foo.zig -install_name @rpath/libfoo.dylib
```
2) with `cc`
```
zig cc -shared foo.c -o libfoo.dylib -Wl,"-install_name=@rpath/libfoo.dylib"
```
Notating a symbol to be exported in code will only tell the linker
where to find this symbol, so other object files can find it. However, this does not mean
said symbol will also be exported to the host environment. Currently, we 'fix' this by force
exporting every single symbol that is visible. This creates bigger binaries and means host environments
have access to symbols that they perhaps shouldn't have. Now, users can tell Zig which symbols
are to be exported, meaning all other symbols that are not specified will not be exported.
Another change is we now support `-rdynamic` in the wasm linker as well, meaning all symbols will
be put in the dynamic symbol table. This is the same behavior as with ELF. This means there's a 3rd strategy
users will have to build their wasm binary.
While investigating slow build times with [a large project](https://github.com/hexops/mach/issues/124),
I found that the compiler was reading from disk nearly every C source file in my project
when rebuilding despite no changes having been made. This accounted for several seconds of
time (approx. 20-30% of running `zig build` without any changes to the sources.)
The cause of this was that comparisons of file mtimes would _always_ fail (the mtime of the file on
disk was always newer than that stored in the cache manifest), and so the cache logic would always
fall back to byte-for-byte file content comparisons with what is on disk vs. in the cache-reading every
C source file in my project from disk during each rebuild. Because file contents were the same, a cache
hit occurred, and _despite the mtime being different the cache manifest would not be updated._
One can reproduce this by building a Zig project so the cache is populated, and then changing mtimes
of their C source files to be newer than what is in the cache (without altering file contents.)
The fix is rather simple: we should always write the updated cache manifest regardless of
whether or not a cache hit occurred (a cache hit doesn't indicate if a manifest is dirty) Luckily,
`writeManifest` already contains logic to determine if a manifest is dirty and becomes no-op if no
change to the manifest file is necessary-so we merely need to ensure it is invoked.
Signed-off-by: Stephen Gutekanst <stephen@hexops.com>
All Zig code is eligible to `@import("builtin")` which is mapped to a
generated file, build.zig, based on the target and other settings.
Zig invocations which share the same target settings will generate the
same builtin.zig file and thus the path to builtin.zig is in a shared
cache folder, and different projects can sometimes use the same file.
Before this commit, this led to race conditions where multiple
invocations of `zig` would race to write this file. If one process
wanted to *read* the file while the other process *wrote* the file, the
reading process could observe a truncated or partially written
builtin.zig file.
This commit makes the following improvements:
- limitations:
- avoid clobbering the inode, mtime in the hot path
- avoid creating a partially written file
- builtin.zig needs to be on disk for debug info / stack trace purposes
- don't mark the task as complete until the file is finished being populated
(possibly by an external process)
- strategy:
- create the `@import("builtin")` `Module.File` during the AstGen
work, based on generating the contents in memory rather than
loading from disk.
- write builtin.zig in a separate task that doesn't have
to complete until the end of the AstGen work queue so that it
can be done in parallel with everything else.
- when writing the file, first stat the file path. If it exists, we are done.
- otherwise, write the file to a temp file in the same directory and atomically
rename it into place (clobbering the inode, mtime in the cold path).
- summary:
- all limitations respected
- hot path: one stat() syscall that happens in a worker thread
This required adding a missing function to the standard library:
`std.fs.Dir.statFile`. In this commit, it does open() and then fstat()
which is two syscalls. It should be improved in a future commit to only
make one.
Fixes#9439.
