This was done by regex substitution with `sed`. I then manually went
over the entire diff and fixed any incorrect changes.
This diff also changes a lot of `callconv(.C)` to `callconv(.c)`, since
my regex happened to also trigger here. I opted to leave these changes
in, since they *are* a correct migration, even if they're not the one I
was trying to do!
The goals of this branch are to:
* compile faster when using the wasm linker and backend
* enable saving compiler state by directly copying in-memory linker
state to disk.
* more efficient compiler memory utilization
* introduce integer type safety to wasm linker code
* generate better WebAssembly code
* fully participate in incremental compilation
* do as much work as possible outside of flush(), while continuing to do
linker garbage collection.
* avoid unnecessary heap allocations
* avoid unnecessary indirect function calls
In order to accomplish this goals, this removes the ZigObject
abstraction, as well as Symbol and Atom. These abstractions resulted
in overly generic code, doing unnecessary work, and needless
complications that simply go away by creating a better in-memory data
model and emitting more things lazily.
For example, this makes wasm codegen emit MIR which is then lowered to
wasm code during linking, with optimal function indexes etc, or
relocations are emitted if outputting an object. Previously, this would
always emit relocations, which are fully unnecessary when emitting an
executable, and required all function calls to use the maximum size LEB
encoding.
This branch introduces the concept of the "prelink" phase which occurs
after all object files have been parsed, but before any Zcu updates are
sent to the linker. This allows the linker to fully parse all objects
into a compact memory model, which is guaranteed to be complete when Zcu
code is generated.
This commit is not a complete implementation of all these goals; it is
not even passing semantic analysis.
This commit amends `std.Build.ExecutableOptions` etc to have a new
field, `root_module`, which allows artifacts to be created whose root
module is an existing `*Module` rather than a freshly constructed one.
This API can be far more versatile, allowing construction of complex
module graphs before creating any compile steps, and therefore also
allowing easy reuse of modules.
The fields which correspond to module options, such as
`root_source_file`, are all considered deprecated. They may not be
populated at the same time as the `root_module` field. In the next
release cycle, these deprecated fields will be removed, and the
`root_module` field made non-optional.
At the expense of a slight special case in the build runner, we can make
the handling of dependencies between modules a little shorter and much
easier to follow.
When module and step graphs are being constructed during the "configure"
phase, we do not set up step dependencies triggered by modules. Instead,
after the configure phase, the build runner traverses the whole
step/module graph, starting from the root top-level steps, and
configures all step dependencies implied by modules. The "make" phase
then proceeds as normal. Also, the old `Module.dependencyIterator` logic
is replaced by two separate iterables. `Module.getGraph` takes the root
module of a compilation, and returns all modules in its graph; while
`Step.Compile.getCompileDependencies` takes a `*Step.Compile` and
returns all `*Step.Compile` it depends on, recursively, possibly
excluding dynamic libraries. The old `Module.dependencyIterator`
combined these two functions into one unintuitive iterator; they are now
separated, which in particular helps readability at the usage sites
which only need one or the other.
This commit changes the `root_module` field of `std.Build.Step.Compile`
to be a `*Module` rather than a `Module`. This is a breaking change, but
an incredibly minor one (the full potential extent of the breakage can
be seen in the modified standalone test).
This change will be necessary for an upcoming improvement, so it was
convenient to make it here.
The previous commit cast doubt upon the initial report about macOS
kernel behavior, identifying another reason that ENOENT could be
returned from file creation.
However, it is demonstrable that ENOENT can be returned for both cases:
1. create file race
2. handle refers to deleted directory
This commit re-introduces the workaround for the file creation race on
macOS however it does not unconditionally retry - it first tries again
with O_EXCL to disambiguate the error condition that has occurred.
Previous commits
2b0929929d67e222ca6a9523a3a594ed456c4a51
4ea2f441df36cec61e1017f4d795d4037326c98c
had this text:
> There are no dir components, so you would think that this was
> unreachable, however we have observed on macOS two processes racing to
> do openat() with O_CREAT manifest in ENOENT.
This appears to have been a misunderstanding based on the issue
report #12138 and corresponding PR #12139 in which the steps to
reproduce removed the cache directory in a loop which also executed
detached Zig compiler processes.
There is no evidence for the macOS kernel bug however the ENOENT is
easily explained by the removal of the cache directory.
This commit reverts those commits, ultimately reporting the ENOENT as an
error rather than repeating the create file operation. However this
commit also adds an explicit error set to `std.Build.Cache.hit` as well
as changing the `failed_file_index` to a proper diagnostic field that
fully communicates what failed, leading to more informative error
messages on failure to check the cache.
The equivalent failure when occuring for AstGen performs a fatal process
kill, reasoning being that the compiler has an invariant of the cache
directory not being yanked out from underneath it while executing. This
could be made a more granular error in the future but I suspect such
thing is not valuable to pursue.
Related to #18340 but does not solve it.
The goal here is to support both levels of unwind tables (sync and async) in
zig cc and zig build. Previously, the LLVM backend always used async tables
while zig cc was partially influenced by whatever was Clang's default.
We have deduced that it seems the sporadic BrokenPipe failures happening
on the CI runners (e.g.
https://github.com/ziglang/zig/actions/runs/12035916948/job/33555963190)
are likely caused by the test runner's stdin pipe abnormally closing,
likely due to the process crashing. Here, we introduce error handling
for this case, so that if these writes fail, the step is marked as
failed correctly, and we still collect the child's stderr to report.
This won't fix the CI issues, but it should promote them to proper error
messages including child stderr, which -- at least in theory -- should
allow us to ultimately track down where the errors come from.
Note that this change is desirable regardless of bugs in the test runner
or similar, since the child process could terminate abnormally for any
number of reasons (e.g. a crashing test), and such cases should be
correctly reported by the build runner.
This is necessary since isGnuLibC() is true for hurd, so we need to be able to
represent a glibc version for it.
Also add an Os.TaggedVersionRange.gnuLibCVersion() convenience function.
When using Build.Step.Run.captureStdOut with a program that prints more
than 10 megabytes of output, the build process hangs.
This is because evalGeneric returns an error without reading child's
stdin till the end, so we subsequently get stuck in `try child.wait()`.
To fix this, make sure to kill the child in case of an error!
Output before this change:
λ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
[3/8] steps
└─ run gh
^C
λ # an hour of debugging
Output after this change:
λ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
install
└─ install generated to ../tigerbeetle
└─ run build_mutliversion (tigerbeetle)
└─ run unzip
└─ run gh failure
error: unable to spawn gh: StdoutStreamTooLong
Build Summary: 3/8 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install generated to ../tigerbeetle transitive failure
└─ run build_mutliversion (tigerbeetle) transitive failure
└─ run unzip transitive failure
└─ run gh failure
error: the following build command failed with exit code 1:
/home/matklad/p/tb/work/.zig-cache/o/c0e3f5e66ff441cd16f9a1a7e1401494/build /home/matklad/p/tb/work/zig/zig /home/matklad/p/tb/work /home/matklad/p/tb/work/.zig-cache /home/matklad/.cache/zig --seed 0xc1d4efc8 -Zaecc61299ff08765 -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6
it doesn't detect and remove no longer watched things yet
it also isn't aware of any file names reported by kqueue. I'm unsure if
that functionality exists.
* Compilation.objects changes to Compilation.link_inputs which stores
objects, archives, windows resources, shared objects, and strings
intended to be put directly into the dynamic section. Order is now
preserved between all of these kinds of linker inputs. If it is
determined the order does not matter for a particular kind of linker
input, that item should be moved to a different array.
* rename system_libs to windows_libs
* untangle library lookup from CLI types
* when doing library lookup, instead of using access syscalls, go ahead
and open the files and keep the handles around for passing to the
cache system and the linker.
* during library lookup and cache file hashing, use positioned reads to
avoid affecting the file seek position.
* library directories are opened in the CLI and converted to Directory
objects, warnings emitted for those that cannot be opened.
The compiler defaults this value to off so that users whose system
shared libraries are all ELF files don't have to pay the cost of
checking every file to find out if it is a text file instead.
When a GNU ld script is encountered, the error message instructs users
about the CLI flag that will immediately solve their problem.
These are really answering questions about the Zig compiler's capacity to
provide a libc/libc++ implementation. As such, std.zig.target seems like a more
fitting place for these.
Make shared_objects a StringArrayHashMap so that deduping does not
need to happen in flush. That deduping code also was using an O(N^2)
algorithm, which is not allowed in this codebase. There is another
violation of this rule in resolveSymbols but this commit does not
address it.
This required reworking shared object parsing, breaking it into
independent components so that we could access soname earlier.
Shared object parsing had a few problems that I noticed and fixed in
this commit:
* Many instances of incorrect use of align(1).
* `shnum * @sizeOf(elf.Elf64_Shdr)` can overflow based on user data.
* `@divExact` can cause illegal behavior based on user data.
* Strange versyms logic that wasn't present in mold nor lld. The logic
was not commented and there is no git blame information in ziglang/zig
nor kubkon/zld. I changed it to match mold and lld instead.
* Use of ArrayList for slices of memory that are never resized.
* finding DT_VERDEFNUM in a different loop than finding DT_SONAME.
Ultimately I think we should follow mold's lead and ignore this
integer, relying on null termination instead.
* Doing logic based on VER_FLG_BASE rather than ignoring it like mold
and LLD do. No comment explaining why the behavior is different.
* Mutating the original ELF symbols rather than only storing the mangled
name on the new Symbol struct.
I noticed something that I didn't try to address in this commit: Symbol
stores a lot of redundant information that is already present in the ELF
symbols. I suspect that the codebase could benefit from reworking Symbol
to not store redundant information.
Additionally:
* Add some type safety to std.elf.
* Eliminate 1-3 file system reads for determining the kind of input
files, by taking advantage of file name extension and handling error
codes properly.
* Move more error handling methods to link.Diags and make them
infallible and thread-safe
* Make the data dependencies obvious in the parameters of
parseSharedObject. It's now clear that the first two steps (Header and
Parsed) can be done during the main Compilation pipeline, rather than
waiting for flush().
Embrace the Path abstraction, doing more operations based on directory
handles rather than absolute file paths. Most of the diff noise here
comes from this one.
Fix sorting of crtbegin/crtend atoms. Previously it would look at all
path components for those strings.
Make the C runtime path detection partially a pure function, and move
some logic to glibc.zig where it belongs.
The compiler actually doesn't need any functional changes for this: Sema
does reification based on the tag indices of `std.builtin.Type` already!
So, no zig1.wasm update is necessary.
This change is necessary to disallow name clashes between fields and
decls on a type, which is a prerequisite of #9938.
A compilation build step for which the binary is not required could not
be compiled previously. There were 2 issues that caused this:
- The compiler communicated only the results of the emitted binary and
did not properly communicate the result if the binary was not emitted.
This is fixed by communicating the final hash of the artifact path (the
hash of the corresponding /o/<hash> directory) and communicating this
instead of the entire path. This changes the zig build --listen protocol
to communicate hashes instead of paths, and emit_bin_path is accordingly
renamed to emit_digest.
- There was an error related to the default llvm object path when
CacheUse.Whole was selected. I'm not really sure why this didn't manifest
when the binary is also emitted.
This was fixed by improving the path handling related to flush() and
emitLlvmObject().
In general, this commit also improves some of the path handling throughout
the compiler and standard library.
