37a9a4e accidentally turned paths `b/[hash]/` into `b[hash]/` in the
global cache. This doesn't technically break anything, but it pollutes
the global cache directory. Sorry about that one!
This commit makes some big changes to how we track state for Zig source
files. In particular, it changes:
* How `File` tracks its path on-disk
* How AstGen discovers files
* How file-level errors are tracked
* How `builtin.zig` files and modules are created
The original motivation here was to address incremental compilation bugs
with the handling of files, such as #22696. To fix this, a few changes
are necessary.
Just like declarations may become unreferenced on an incremental update,
meaning we suppress analysis errors associated with them, it is also
possible for all imports of a file to be removed on an incremental
update, in which case file-level errors for that file should be
suppressed. As such, after AstGen, the compiler must traverse files
(starting from analysis roots) and discover the set of "live files" for
this update.
Additionally, the compiler's previous handling of retryable file errors
was not very good; the source location the error was reported as was
based only on the first discovered import of that file. This source
location also disappeared on future incremental updates. So, as a part
of the file traversal above, we also need to figure out the source
locations of imports which errors should be reported against.
Another observation I made is that the "file exists in multiple modules"
error was not implemented in a particularly good way (I get to say that
because I wrote it!). It was subject to races, where the order in which
different imports of a file were discovered affects both how errors are
printed, and which module the file is arbitrarily assigned, with the
latter in turn affecting which other files are considered for import.
The thing I realised here is that while the AstGen worker pool is
running, we cannot know for sure which module(s) a file is in; we could
always discover an import later which changes the answer.
So, here's how the AstGen workers have changed. We initially ensure that
`zcu.import_table` contains the root files for all modules in this Zcu,
even if we don't know any imports for them yet. Then, the AstGen
workers do not need to be aware of modules. Instead, they simply ignore
module imports, and only spin off more workers when they see a by-path
import.
During AstGen, we can't use module-root-relative paths, since we don't
know which modules files are in; but we don't want to unnecessarily use
absolute files either, because those are non-portable and can make
`error.NameTooLong` more likely. As such, I have introduced a new
abstraction, `Compilation.Path`. This type is a way of representing a
filesystem path which has a *canonical form*. The path is represented
relative to one of a few special directories: the lib directory, the
global cache directory, or the local cache directory. As a fallback, we
use absolute (or cwd-relative on WASI) paths. This is kind of similar to
`std.Build.Cache.Path` with a pre-defined list of possible
`std.Build.Cache.Directory`, but has stricter canonicalization rules
based on path resolution to make sure deduplicating files works
properly. A `Compilation.Path` can be trivially converted to a
`std.Build.Cache.Path` from a `Compilation`, but is smaller, has a
canonical form, and has a digest which will be consistent across
different compiler processes with the same lib and cache directories
(important when we serialize incremental compilation state in the
future). `Zcu.File` and `Zcu.EmbedFile` both contain a
`Compilation.Path`, which is used to access the file on-disk;
module-relative sub paths are used quite rarely (`EmbedFile` doesn't
even have one now for simplicity).
After the AstGen workers all complete, we know that any file which might
be imported is definitely in `import_table` and up-to-date. So, we
perform a single-threaded graph traversal; similar to what
`resolveReferences` plays for `AnalUnit`s, but for files instead. We
figure out which files are alive, and which module each file is in. If a
file turns out to be in multiple modules, we set a field on `Zcu` to
indicate this error. If a file is in a different module to a prior
update, we set a flag instructing `updateZirRefs` to invalidate all
dependencies on the file. This traversal also discovers "import errors";
these are errors associated with a specific `@import`. With Zig's
current design, there is only one possible error here: "import outside
of module root". This must be identified during this traversal instead
of during AstGen, because it depends on which module the file is in. I
tried also representing "module not found" errors in this same way, but
it turns out to be much more useful to report those in Sema, because of
use cases like optional dependencies where a module import is behind a
comptime-known build option.
For simplicity, `failed_files` now just maps to `?[]u8`, since the
source location is always the whole file. In fact, this allows removing
`LazySrcLoc.Offset.entire_file` completely, slightly simplifying some
error reporting logic. File-level errors are now directly built in the
`std.zig.ErrorBundle.Wip`. If the payload is not `null`, it is the
message for a retryable error (i.e. an error loading the source file),
and will be reported with a "file imported here" note pointing to the
import site discovered during the single-threaded file traversal.
The last piece of fallout here is how `Builtin` works. Rather than
constructing "builtin" modules when creating `Package.Module`s, they are
now constructed on-the-fly by `Zcu`. The map `Zcu.builtin_modules` maps
from digests to `*Package.Module`s. These digests are abstract hashes of
the `Builtin` value; i.e. all of the options which are placed into
"builtin.zig". During the file traversal, we populate `builtin_modules`
as needed, so that when we see this imports in Sema, we just grab the
relevant entry from this map. This eliminates a bunch of awkward state
tracking during construction of the module graph. It's also now clearer
exactly what options the builtin module has, since previously it
inherited some options arbitrarily from the first-created module with
that "builtin" module!
The user-visible effects of this commit are:
* retryable file errors are now consistently reported against the whole
file, with a note pointing to a live import of that file
* some theoretical bugs where imports are wrongly considered distinct
(when the import path moves out of the cwd and then back in) are fixed
* some consistency issues with how file-level errors are reported are
fixed; these errors will now always be printed in the same order
regardless of how the AstGen pass assigns file indices
* incremental updates do not print retryable file errors differently
between updates or depending on file structure/contents
* incremental updates support files changing modules
* incremental updates support files becoming unreferenced
Resolves: #22696
* Accept -fsanitize-c=trap|full in addition to the existing form.
* Accept -f(no-)sanitize-trap=undefined in zig cc.
* Change type of std.Build.Module.sanitize_c to std.zig.SanitizeC.
* Add some missing Compilation.Config fields to the cache.
Closes#23216.
`--fetch` flag now has additional optional parameter, which specifies
how lazy dependencies should be fetched:
* `needed` — lazy dependencies are fetched only if they are required
for current build configuration to work. Default and works same
as old `--fetch` flag.
* `all` — lazy dependencies are always fetched. If `--system` flag
is used after that, it's guaranteed that **any** build configuration
will not require additional download of dependencies during build.
Helpful for distro packagers and CI systems:
https://www.github.com/ziglang/zig/issues/14597#issuecomment-1426827495
If none is passed, behaviour is same as if `needed` was passed.
Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
This commits adds the following distinct integer types to std.zig.Ast:
- OptionalTokenIndex
- TokenOffset
- OptionalTokenOffset
- Node.OptionalIndex
- Node.Offset
- Node.OptionalOffset
The `Node.Index` type has also been converted to a distinct type while
`TokenIndex` remains unchanged.
`Ast.Node.Data` has also been changed to a (untagged) union to provide
safety checks.
mainly this addresses the following use case:
1. Someone creates a template with build.zig.zon, id field included
(note that zig init does not create this problem since it generates
fresh id every time it runs).
2. User A uses the template, changing package name to "example" but not
id field.
3. User B uses the same template, changing package name also to
"example", also not changing the id field.
Here, both packages have unintentional conflicting logical ids.
By making the field a combination of name checksum + random id, this
accident is avoided. "nonce" is an OK name for this.
Also relaxes errors on remote packages when using `zig fetch`.
Introduces the `id` field to `build.zig.zon`.
Together with name, this represents a globally unique package
identifier. This field should be initialized with a 16-bit random number
when the package is first created, and then *never change*. This allows
Zig to unambiguously detect when one package is an updated version of
another.
When forking a Zig project, this id should be regenerated with a new
random number if the upstream project is still maintained. Otherwise,
the fork is *hostile*, attempting to take control over the original
project's identity.
`0x0000` is invalid because it obviously means a random number wasn't
used.
`0xffff` is reserved to represent "naked" packages.
Tracking issue #14288
Additionally:
* Fix bad path in error messages regarding build.zig.zon file.
* Manifest validates that `name` and `version` field of build.zig.zon
are maximum 32 bytes.
* Introduce error for root package to not switch to enum literal for
name.
* Introduce error for root package to omit `id`.
* Update init template to generate `id`
* Update init template to populate `minimum_zig_version`.
* New package hash format changes:
- name and version limited to 32 bytes via error rather than truncation
- truncate sha256 to 192 bits rather than 40 bits
- include the package id
This means that, given only the package hashes for a complete dependency
tree, it is possible to perform version selection and know the final
size on disk, without doing any fetching whatsoever. This prevents
wasted bandwidth since package versions not selected do not need to be
fetched.
This branch regressed from master by switching to binary rather than hex
digest, allowing null bytes to end up in identifiers in the zig file.
This commit fixes it by changing the "hash" to be literally equal to the
sub_path (with a prefix '/' to indicate "global") if it can fit. If it
is too long then it is actually hashed, and that value used instead.
* fix merge conflicts
* rename the declarations
* reword documentation
* extract FixedBufferAllocator to separate file
* take advantage of locals
* remove the assertion about max alignment in Allocator API, leaving it
Allocator implementation defined
* fix non-inline function call in start logic
The GeneralPurposeAllocator implementation is totally broken because it
uses global state but I didn't address that in this commit.
heap.zig: define new default page sizes
heap.zig: add min/max_page_size and their options
lib/std/c: add miscellaneous declarations
heap.zig: add pageSize() and its options
switch to new page sizes, especially in GPA/stdlib
mem.zig: remove page_size
Instead, `source`, `tree`, and `zir` should all be optional. This is
precisely what we're actually trying to model here; and `File` isn't
optimized for memory consumption or serializability anyway, so it's fine
to use a couple of extra bytes on actual optionals here.
This moves the default value logic to Package.Module.create() instead and makes
it so that Compilation.Config.any_unwind_tables is computed similarly to
any_sanitize_thread, any_fuzz, etc. It turns out that for any_unwind_tables, we
only actually care if unwind tables are enabled at all, not at what level.
There are several test decls inside `/src` that are not currently being
tested and have bitrotted as a result. This commit revives those tests
and adds the `test-compiler-internals` set of tests which tests
everything reachable from `/src/main.zig`.
The goal here is to support both levels of unwind tables (sync and async) in
zig cc and zig build. Previously, the LLVM backend always used async tables
while zig cc was partially influenced by whatever was Clang's default.
Frame pointers make both debugging and profiling work better, and the overhead
is reportedly 1% or less for typical programs [0]. I think the pros outweigh the
cons here. People who *really* care about that 1% can simply use the
-fomit-frame-pointer option to reclaim it. For ReleaseSmall, though, it makes
sense to omit frame pointers by default for the sake of code size, as we already
strip the binary in this case anyway.
Closes#22161.
[0] https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
If the same dependency is first found as lazy and then later as eager,
the existing entry needs to be updated to eager in order for
`b.dependency()` to work.
This option, by its very nature, needs to be attached to a module. If it isn't,
the code in a module could break at random when compiled into an application
that doesn't have this option set.
After this change, skip_linker_dependencies no longer implies no_builtin in the
LLVM backend.
The re-analysis here is a little coarse; it'd be nice in the future to
have a way for an AstGen failure to preserve *all* analysis which
depends on the last success, and just hide the compile errors which
depend on it somehow. But I'm not sure how we'd achieve that, so this
works fine for now.
Resolves: #21223
The compiler actually doesn't need any functional changes for this: Sema
does reification based on the tag indices of `std.builtin.Type` already!
So, no zig1.wasm update is necessary.
This change is necessary to disallow name clashes between fields and
decls on a type, which is a prerequisite of #9938.
