Another big commit, sorry! This commit makes all fixes necessary for
incremental updates of the compiler itself (specifically, adding a
breakpoint to `zirCompileLog`) to succeed, at least on the frontend.
The biggest change here is a reform to how types are handled. It works
like this:
* When a type is first created in `zirStructDecl` etc, its namespace is
scanned. If the type requires resolution, an `interned` dependency is
declared for the containing `AnalUnit`.
* `zirThis` also declared an `interned` dependency for its `AnalUnit` on
the namespace's owner type.
* If the type's namespace changes, the surrounding source declaration
changes hash, so `zirStructDecl` etc will be hit again. We check
whether the namespace has been scanned this generation, and re-scan it
if not.
* Namespace lookups also check whether the namespace in question
requires a re-scan based on the generation. This is because there's no
guarantee that the `zirStructDecl` is re-analyzed before the namespace
lookup is re-analyzed.
* If a type's structure (essentially its fields) change, then the type's
`Cau` is considered outdated. When the type is re-analyzed due to
being outdated, or the `zirStructDecl` is re-analyzed by being
transitively outdated, or a corresponding `zirThis` is re-analyzed by
being transitively outdated, the struct type is recreated at a new
`InternPool` index. The namespace's owner is updated (but not
re-scanned, since that is handled by the mechanisms above), and the
old type, while remaining a valid `Index`, is removed from the map
metadata so it will never be found by lookups. `zirStructDecl` and
`zirThis` store an `interned` dependency on the *new* type.
Two fixes here.
* Prevent a crash when sorting the list of analysis errors when some
errors refer to lost source locations. These errors can be sorted
anywhere in the list, because they are (in theory) guaranteed to never
be emitted by the `resolveReferences` logic. This case occurs, for
instance, when a declaration has compile errors in the initial update
and is deleted in the second update.
* Prevent a crash when resolving the source location for `entire_file`
errors for a non-existent file. This is the bug underlying #20954.
Resolves: #20954.
This commit updates `Zcu.resolveReferences` to traverse the graph of
`AnalUnit` references (starting from the 1-3 roots of analysis) in order
to determine which `AnalUnit`s are referenced in an update. Errors for
unreferenced entities are omitted from the error bundle. However, note
that unreferenced `Nav`s are not removed from the binary.
This commit makes more progress towards incremental compilation, fixing
some crashes in the frontend. Notably, it fixes the regressions introduced
by #20964. It also cleans up the "outdated file root" mechanism, by
virtue of deleting it: we now detect outdated file roots just after
updating ZIR refs, and re-scan their namespaces.
Simplifies code in docs creation where we used `std.tar.output.Header`.
Writer uses that Header internally and provides higher level interface.
Updates checksum on write, handles long file names, allows setting mtime and file permission mode. Provides handy interface for passing `Dir.WalkerEntry`.
Some of this is arbitrary since spirv (as opposed to spirv32/spirv64) refers to
the version with logical memory layout, i.e. no 'real' pointers. This change at
least matches what clang does.
The type `Zcu.Decl` in the compiler is problematic: over time it has
gained many responsibilities. Every source declaration, container type,
generic instantiation, and `@extern` has a `Decl`. The functions of
these `Decl`s are in some cases entirely disjoint.
After careful analysis, I determined that the two main responsibilities
of `Decl` are as follows:
* A `Decl` acts as the "subject" of semantic analysis at comptime. A
single unit of analysis is either a runtime function body, or a
`Decl`. It registers incremental dependencies, tracks analysis errors,
etc.
* A `Decl` acts as a "global variable": a pointer to it is consistent,
and it may be lowered to a specific symbol by the codegen backend.
This commit eliminates `Decl` and introduces new types to model these
responsibilities: `Cau` (Comptime Analysis Unit) and `Nav` (Named
Addressable Value).
Every source declaration, and every container type requiring resolution
(so *not* including `opaque`), has a `Cau`. For a source declaration,
this `Cau` performs the resolution of its value. (When #131 is
implemented, it is unsolved whether type and value resolution will share
a `Cau` or have two distinct `Cau`s.) For a type, this `Cau` is the
context in which type resolution occurs.
Every non-`comptime` source declaration, every generic instantiation,
and every distinct `extern` has a `Nav`. These are sent to codegen/link:
the backends by definition do not care about `Cau`s.
This commit has some minor technically-breaking changes surrounding
`usingnamespace`. I don't think they'll impact anyone, since the changes
are fixes around semantics which were previously inconsistent (the
behavior changed depending on hashmap iteration order!).
Aside from that, this changeset has no significant user-facing changes.
Instead, it is an internal refactor which makes it easier to correctly
model the responsibilities of different objects, particularly regarding
incremental compilation. The performance impact should be negligible,
but I will take measurements before merging this work into `master`.
Co-authored-by: Jacob Young <jacobly0@users.noreply.github.com>
Co-authored-by: Jakub Konka <kubkon@jakubkonka.com>
Some projects, such as Cap'n Proto, use .c++ as their filenames. Without
this, compiling them fails because zig c++ will fall back to using the
linker.
We advertise reproducible builds for release modes, so let's help users achieve
that in C/C++ code. Users can still override this manually if they really want.
The flag makes compiler_rt and libfuzzer be in debug mode.
Also:
* fuzzer: override debug logs and disable debug logs for frequently
called functions
* std.Build.Fuzz: fix bug of rerunning the old unit test binary
* report errors from rebuilding the unit tests better
* link.Elf: additionally add tsan lib and fuzzer lib to the hash
* Add -f(no-)sanitize-coverage-trace-pc-guard CLI flag which defaults to
off. This value lowers to TracePCGuard = true (LLVM backend) and -Xclang
-fsanitize-coverage-trace-pc-guard. These settings are not
automatically included with -ffuzz.
* Add `Build.Step.Compile` flag for sanitize_coverage_trace_pc_guard
with appropriate documentation.
* Add `zig cc` integration for the respective flags.
* Avoid crashing in ELF linker code when -ffuzz -femit-llvm-ir used
together.
Exposes sanitizer coverage flags to the target machine emit function.
Makes it easier to change sancov options without rebuilding the C++
files.
This also enables PCTable = true for sancov which is needed by AFL, and
adds the corresponding Clang flag.
* Add the `-ffuzz` and `-fno-fuzz` CLI arguments.
* Detect fuzz testing flags from zig cc.
* Set the correct clang flags when fuzz testing is requested. It can be
combined with TSAN and UBSAN.
* Compilation: build fuzzer library when needed which is currently an
empty zig file.
* Add optforfuzzing to every function in the llvm backend for modules
that have requested fuzzing.
* In ZigLLVMTargetMachineEmitToFile, add the optimization passes for
sanitizer coverage.
* std.mem.eql uses a naive implementation optimized for fuzzing when
builtin.fuzz is true.
Tracked by #20702
Remove --debug-incremental
This flag is also added to the build system. Importantly, this tells
Compile step whether or not to keep the compiler running between
rebuilds. It defaults off because it is currently crashing
zirUpdateRefs.
Changes the `make` function signature to take an options struct, which
additionally includes `watch: bool`. I intentionally am not exposing
this information to configure phase logic.
Also adds global zig cache to the compiler cache prefixes.
Closes#20600
This allows the mutate mutex to only be locked during actual grows,
which are rare. For the lists that didn't previously have a mutex, this
change has little effect since grows are rare and there is zero
contention on a mutex that is only ever locked by one thread. This
change allows `extra` to be mutated without racing with a grow.
Updates the build runner to unconditionally require a zig lib directory
parameter. This parameter is needed in order to correctly understand
file system inputs from zig compiler subprocesses, since they will refer
to "the zig lib directory", and the build runner needs to place file
system watches on directories in there.
The build runner's fanotify file watching implementation now accounts
for when two or more Cache.Path instances compare unequal but ultimately
refer to the same directory in the file system.
Breaking change: std.Build no longer has a zig_lib_dir field. Instead,
there is the Graph zig_lib_directory field, and individual Compile steps
can still have their zig lib directories overridden. I think this is
unlikely to break anyone's build in practice.
The compiler now sends a "file_system_inputs" message to the build
runner which shares the full set of files that were added to the cache
system with the build system, so that the build runner can watch
properly and redo the Compile step. This is implemented for whole cache
mode but not yet for incremental cache mode.
