Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
* move `ptrBitWidth` from Arch to Target since it needs to know about the abi
* double isn't always 8 bits
* AVR uses 1-byte alignment for everything in GCC
This makes progress be exposed to the top-level caller of update().
I tossed in a bonus change: when the `zig build` subcommand sees exit
code 2, it omits the "following command failed" line, and the build
runner uses exit code 2 when there are compile errors. This tidies up
the output on build failure by a little bit.
Introduces std.zig.ErrorBundle which is a trivially serializeable set
of compilation errors. This is in the standard library so that both
the compiler and the build runner can use it. The idea is they will
use it to communicate compilation errors over a binary protocol.
The binary encoding of ErrorBundle is a bit problematic - I got a little
too aggressive with compaction. I need to change it in a follow-up
commit to use some indirection in the error message list, otherwise
iteration is too unergonomic. In fact it's so problematic right now that
the logic getAllErrorsAlloc() actually fails to produce a viable
ErrorBundle because it puts SourceLocation data in between the root
level ErrorMessage data.
This commit has a simplification - redundant logic for rendering AST
errors to stderr has been removed in favor of moving the logic for
lowering AST errors into AstGen. So even if we get parse errors, the
errors will get lowered into ZIR before being reported. I believe this
will be useful when working on --autofix. Either way, some redundant
brittle logic was happily deleted.
In Compilation, updateSubCompilation() is improved to properly perform
error reporting when a sub-compilation object fails. It no longer dumps
directly to stderr; instead it populates an ErrorBundle object, which
gets added to the parent one during getAllErrorsAlloc().
In package fetching code, instead of dumping directly to stderr, it now
populates an ErrorBundle object, and gets properly reported at the CLI
layer of abstraction.
In glibc.zig, there were a few instances where the zig lib dir path name
incorrectly made its way into the cache namespace for various build
artifacts, resulting in unnecessary rebuilds of glibc.
Closes#13619
Before, cache manifest files would have absolute file paths. This is
problematic for two reasons:
* Absolute file paths are not portable. Some operating systems such as
WASI have trouble with them. The files themselves are less portable;
they cannot be migrated from one user's home directory to another's.
And finally they can break due to file paths exceeding maximum path
component size.
* They would prevent some advanced use cases of Zig, where the lib dir
has a different path in a different invocation but is ultimately the
same Zig version and lib directory as before.
This commit adds a new column that specifies the prefix directory for
each file. 0 is an escape hatch and has the previous behavior. The other
two prefixes introduced are zig lib directory, and the cache directory.
This means files in zig-cache manifests can reference files local to
these directories.
In practice, this means it is possible to use a different file path for
the zig lib directory in a subsequent run of zig and have it still take
advantage of the global cache, provided that the files inside remain
unchanged.
closes#13050
Before, the code for building glibc stubs used a special case of the
Cache API that did not add any file inputs, and did not use
writeManifest(). This is not really how the Cache API is designed to
work and it shows because there was a race condition.
This commit adds as an input file the abilists file that comes with
Zig's installation, which has the added benefit of making glibc stub
caching properly detect cache invalidation when the user decides to
overwrite their abilists file. This harmonizes with the rest of how Zig
works, which intentionally allows you to hack the installation files and
have it behave properly with the cache system.
Finally, because of having any file inputs, the normal API flow of the
Cache system can be used, eliminating the one place that used the Cache
API in a special way. In other words, it uses writeManifest() now and
properly obeys the cache hit/miss semantics.
closes#13160
When linking libc and compiling natively, Zig tries to integrate with
the system C compiler. However, this caused Zig to fail when no system C
compiler is installed, despite the fact that Zig is perfectly capable of
compiling & linking libc without one.
This commit makes Zig fall back to using its own ability to provide libc
in the case that no C compiler is installed. For glibc, it means
sometimes getting the warning "zig cannot build new glibc version abc,
providing instead xyz".
Ideally, Zig would do some more validation about the system libraries
being linked against, and report an error in case it could not provide
the exact correct libc version of the system libraries (or that the
system libraries themselves conflict with each other), however, I think
it is fair to call that a separate enhancement.
Rename all references of sparcv9 to sparc64, to make Zig align more with
other projects. Also, added new function to convert glibc arch name to Zig
arch name, since it refers to the architecture as sparcv9.
This is based on the suggestion by @kubkon in PR 11847.
(https://github.com/ziglang/zig/pull/11487#pullrequestreview-963761757)
also use the common naming convention for glibc versions ("2.33" rather
than "2-33").
I also verified that these files are exactly identical to the previous
files from before zig updated to glibc 2.34.
__libc_start_main() from glibc.2.33.so or older needs to have a __libc_csu_init function callback parameter.
glibc-2.34 on the other hand has a different __libc_start_main() that does not use it,
and the start.S file from glibc-2.34 no longer construct the init function and pass null when calling __libc_start_main.
So, When targetting an older glibc, use the start.s files as they were in glibc-2.33 and construct the __libc_csu_init function.
fixes#10386#10512
This commit fixes two problems:
* `zig build-obj` regressed from the cache-mode branch. It would crash
because it assumed that dirname on the emit bin path would not be
null. This assumption was invalid when outputting to the current
working directory - a pretty common use case for `zig build-obj`.
* When using the LLVM backend, `-fno-emit-bin` combined with any other
kind of emitting, such as `-femit-asm`, emitted nothing.
Both issues are now fixed.
The two CacheMode values are `whole` and `incremental`.
`incremental` is what we had before; `whole` is new.
Whole cache mode uses everything as inputs to the cache hash;
and when a hit occurs it skips everything including linking.
This is ideal for when source files change rarely and for backends that
do not have good incremental compilation support, for example
compiler-rt or libc compiled with LLVM with optimizations on.
This is the main motivation for the additional mode, so that we can have
LLVM-optimized compiler-rt/libc builds, without waiting for the LLVM
backend every single time Zig is invoked.
Incremental cache mode hashes only the input file path and a few target
options, intentionally relying on collisions to locate already-existing
build artifacts which can then be incrementally updated.
The bespoke logic for caching stage1 backend build artifacts
is removed since we now have a global caching mechanism for
when we want to cache the entire compilation, *including* linking.
Previously we had to get "creative" with libs.txt and a special
byte in the hash id to communicate flags, so that when the cached
artifacts were re-linked, we had this information from stage1
even though we didn't actually run it. Now that `CacheMode.whole`
includes linking, this extra information does not need to be
preserved for cache hits. So although this changeset introduces
complexity, it also removes complexity.
The main trickiness here comes from the inherent differences between the
two modes: `incremental` wants a directory immediately to operate on,
while `whole` doesn't know the output directory until the compilation is
complete. This commit deals with this problem mostly inside `update()`,
where, on a cache miss, it replaces `zig_cache_artifact_directory` with a
temporary directory, and then renames it into place once the compilation is
complete.
Items remaining before this branch can be merged:
* [ ] make sure these things make it into the cache manifest:
- @import files
- @embedFile files
- we already add dep files from c but make sure the main .c files make
it in there too, not just the included files
* [ ] double check that the emit paths of other things besides the binary
are working correctly.
* [ ] test `-fno-emit-bin` + `-fstage1`
* [ ] test `-femit-bin=foo` + `-fstage1`
* [ ] implib emit directory copies bin_file_emit directory in create() and needs
to be adjusted to be overridden as well.
* [ ] make sure emit-h is handled correctly in the cache hash
* [ ] Cache: detect duplicate files added to the manifest
Some preliminary performance measurements of wall clock time and
peak RSS used:
stage1 behavior (1077 tests), llvm backend, release build:
* cold global cache: 4.6s, 1.1 GiB
* warm global cache: 3.4s, 980 MiB
stage2 master branch behavior (575 tests), llvm backend, release build:
* cold global cache: 0.62s, 191 MiB
* warm global cache: 0.40s, 128 MiB
stage2 this branch behavior (575 tests), llvm backend, release build:
* cold global cache: 0.62s, 179 MiB
* warm global cache: 0.27s, 90 MiB
Upstream, some of the nonshared functions moved to be different for hurd
and for linux. Since our glibc is linux-only we update to use the
linux-specific files.
This fixes std lib tests for x86_64 when linking glibc.
This commit introduces tools/update_glibc.zig to update the start files
for next time.
Some notable changes in recent glibc:
* abi-note.S has been changed to abi-note.c but we resist the change to
keep it easier to compile the start files.
* elf-init.c has been deleted upstream. Further testing should be done
to verify that binaries against glibc omitting elf-init.c still run
properly on oldel glibc linux systems.
Closes#4926
This commit upgrades glibc shared library stub-creating code to use the
new abilists file which is generated by the new glibc-abi-tool project:
https://github.com/ziglang/glibc-abi-tool/
The abilists file is different in these ways:
* It additionally encodes whether a symbol is a function or an object,
and if it is an object, it additionally encodes the size in bytes.
* It additionally encodes migrations of symbols from one library to
another between glibc versions.
* It is binary data instead of ascii.
* It is one file instead of three.
* It is 165 KB instead of 200 KB.
This solves three bugs:
Fixes#7667Fixes#8714Fixes#8896
* Add missing Linux headers. Closes#9837
* Update existing headers to latest Linux.
* Consolidate headers that are the same for multiple Zig target CPU
architectures. For example, Linux has only an x86 directory for both
x86_64 and x86 CPU architectures. Now Zig only ships an x86 directory
for Linux headers, and will emit the proper corresponding -isystem
flags.
* tools/update-linux-headers.zig is now available for upgrading to
newer Linux headers, and the update process is now documented on the
wiki.
* There is now a main_pkg in addition to root_pkg. They are usually the
same. When using `zig test`, main_pkg is the user's source file and
root_pkg has the test runner.
* scanDecl no longer looks for test decls outside the package being
tested. honoring `--test-filter` is still TODO.
* test runner main function has a void return value rather than
`anyerror!void`
* Sema is improved to generate better AIR for for loops on slices.
* Sema: fix incorrect capacity calculation in zirBoolBr
* Sema: add compile errors for trying to use slice fields as an lvalue.
* Sema: fix type coercion for error unions
* Sema: fix analyzeVarRef generating garbage AIR
* C codegen: fix renderValue for error unions with 0 bit payload
* C codegen: implement function pointer calls
* CLI: fix usage text
Adds 4 new AIR instructions:
* slice_len, slice_ptr: to get the ptr and len fields of a slice.
* slice_elem_val, ptr_slice_elem_val: to get the element value of
a slice, and a pointer to a slice.
AstGen gains a new functionality:
* One of the unused flags of struct decls is now used to indicate
structs that are known to have non-zero size based on the AST alone.
closes#9034
These options were listed under the
"Debug Options (Zig Compiler Development)" heading. Anything in this
section should be considered unstable and can be modified at any time
at any developer's discretion.
Our glibc stub assembly file looked something like this:
```
.globl _Exit_2_2_5
.type _Exit_2_2_5, %function;
.symver _Exit_2_2_5, _Exit@@GLIBC_2.2.5
.hidden _Exit_2_2_5
_Exit_2_2_5:
```
With clang 12, the shared objects this produced stopped having any
exported symbols. When I removed the `.hidden` directive, it resolved
the issue, however, there are now unwanted exports:
```
$ readelf -W --dyn-syms libc.so.6 | grep sys_errlist
139: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 _sys_errlist_GLIBC_2_3
147: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 _sys_errlist_GLIBC_2_4
395: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 _sys_errlist_GLIBC_2_2_5
487: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 sys_errlist_GLIBC_2_2_5
1266: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 _sys_errlist@@GLIBC_2.12
1267: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 _sys_errlist@GLIBC_2.2.5
1268: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 _sys_errlist@GLIBC_2.3
1269: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 _sys_errlist@GLIBC_2.4
2137: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 sys_errlist@@GLIBC_2.12
2138: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 sys_errlist@GLIBC_2.2.5
2139: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 sys_errlist@GLIBC_2.3
2140: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 sys_errlist@GLIBC_2.4
2156: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 sys_errlist_GLIBC_2_3
2161: 000000000001ee08 0 FUNC GLOBAL DEFAULT 7 sys_errlist_GLIBC_2_4
```
Every line here without an `@` symbol is an unwanted export. Before, the
unwanted ones had LOCAL HIDDEN linkage.
As a mitigation, I did two things:
* Added `_GLIBC_` to the unwanted exports so that they would not
conflict with anything.
* Made the default export (the `@@` one) the bare symbol name. This
appears to reduce the unwanted exports to only symbols that have more
than one symbol (which is still quite many).
This will unblock progress on this branch, however, there is now a new
issue to solve, that the provided glibc stub .so files have too many
symbols exported. We will have to find a way to avoid this.
* CLI: change to -mred-zone and -mno-red-zone to match gcc/clang.
* build.zig: remove the double negative and make it an optional bool.
This follows precedent from other flags, allowing the compiler CLI to
be the decider of what is default instead of duplicating the default
value into the build system code.
* Compilation: make it an optional `want_red_zone` instead of a
`no_red_zone` bool. The default is decided by a call to
`target_util.hasRedZone`.
* When creating a Clang command line, put -mred-zone on the command
line if we are forcing it to be enabled.
* Update update_clang_options.zig with respect to the recent {s}/{} format changes.
* `zig cc` integration with red zone preference.
