This logic (currently) has a non-trivial cost (particularly in terms of
peak RSS) for tracking dependencies. Until incremental compilation is in
use in the wild, it doesn't make sense for users to pay that cost.
* Functions failing codegen now set this failure on the function
analysis state. Decl analysis `codegen_failure` is reserved for
failures generating constant values.
* `liveness_failure` is consolidated into `codegen_failure`, as we do
not need to distinguish these, and Liveness.Verify is just a debugging
feature anyway.
* `sema_failure_retryable` and `codegen_failure_retryable` are removed.
Instead, retryable failures are recorded in the new
`Zcu.retryable_failures` list. On an incremental update, this list is
flushed, and all elements are marked as outdated so that we re-attempt
analysis and code generation.
Also remove the `generation` fields from `Zcu` and `Decl` as these are
not needed by our new strategy for incremental updates.
* Mark root Decls for re-analysis separately
* Check for re-analysis of root Decls
* Remove `outdated` entry when analyzing fn body
* Remove legacy `outdated` field from Decl analysis state
* Invalidate `decl_val` dependencies
* Recursively mark and un-mark all dependencies correctly
* Queue analysis of outdated dependers in `Compilation.performAllTheWork`
Introduces logic to invalidate `decl_val` dependencies after
`Zcu.semaDecl` completes. Also, recursively un-mark dependencies as PO
where needed.
With this, all dependency invalidation logic is in place. The next step
is analyzing outdated dependencies and triggering appropriate
re-analysis.
Sema now tracks dependencies appropriately. Early logic in Zcu for
resolving outdated decls/functions is in place. The setup used does not
support `usingnamespace`; compilations using this construct are not yet
supported by this incremental compilation model.
It is problematic for the cached `InternPool` state to directly
reference ZIR instruction indices, as these are not stable across
incremental updates. The existing ZIR mapping logic attempts to handle
this by iterating the existing Decl graph for a file after `AstGen` and
update ZIR indices on `Decl`s, struct types, etc. However, this is
unreliable due to generic instantiations, and relies on specialized
logic for everything which may refer to a ZIR instruction (e.g. a
struct's owner decl). I therefore determined that a prerequisite change
for incremental compilation would be to rework how we store these
indices.
This commit introduces a `TrackedInst` type which provides a stable
index (`TrackedInst.Index`) for a single ZIR instruction in the
compilation. The `InternPool` now stores these values in place of ZIR
instruction indices. This makes the ZIR mapping logic relatively
trivial: after `AstGen` completes, we simply iterate all `TrackedInst`
values and update those indices which have changed. In future, if the
corresponding ZIR instruction has been removed, we must also invalidate
any dependencies on this instruction to trigger any required
re-analysis, however the dependency system does not yet exist.
The deleted lines here are redundant because they happen first thing
inside the function call below.
Additionally, skip hashing the root source file if it is an empty
string. I explored making this field along with `root` optional but
found this to be less messy actually.
Co-authored-by: Motiejus Jakštys <motiejus@jakstys.lt>
Co-authored-by: Jakub Konka <kubkon@jakubkonka.com>
Co-authored-by: Samuel Cantero <scanterog@gmail.com>
Co-authored-by: Giorgos Georgiou <giorgos.georgiou@datadoghq.com>
Co-authored-by: Carl Åstholm <carl@astholm.se>
This branch introduced an arena allocator for temporary allocations in
Compilation.update. Almost every implementation of flush() inside the
linker code was already creating a local arena that had the lifetime of
the function call. This commit passes the update arena so that all those
local ones can be deleted, resulting in slightly more efficient memory
usage with every compilation update.
While at it, this commit also removes the Compilation parameter from the
linker flush function API since a reference to the Compilation is now
already stored in `link.File`.
This isn't technically needed since per-module -I args can suffice, but
this can produce very long CLI invocations when several --mod args are
combined with --search-prefix args since the -I args have to be repeated
for each module.
This is a partial revert of ecbe8bbf2df2ed4d473efbc32e0b6d7091fba76f.
The linker needs to know the file system path of output in the flush
function because file paths inside the build artifacts reference each
other. Fixes a regression introduced in this branch.
Without this commit, unrelated test builds using incremental cache mode
(self-hosted, no lld) would end up using the same cache namespace, which
is undesireable since concurrent builds will clobber each other's work.
This happened because of passing the root module to
addModuleToCacheHash. In the case of a test build, the root module
actually does not connect to the rest of the import table. Instead, the
main module needs to be passed, which has "root" in its import table.
The other call to addModuleTableToCacheHash which is in
addNonIncrementalStuffToCacheManifest already correctly passes the main
module.
In the future, I think this problem can be fully addressed by obtaining
an advisory lock on the output binary file. However, even in that case,
it is still valuable to make different compilations use different cache
namespaces lest unrelated compilations suffer from pointless thrashing
rather than being independently edited.
Instead of making its own inside create. 10 out of 10 calls to create()
had already an arena in scope, so this commit means that 10 instances of
Compilation now reuse an existing arena with the same lifetime rather
than creating a redundant one.
In other words, this very slightly optimizes initialization of the
frontend in terms of memory allocation.
Now, link.File will always be null when -fno-emit-bin is specified, and
in the case that LLVM artifacts are still required, the Zcu instance has
an LlvmObject.
before this commit it was trying to hash based on resolved bin_file
settings, but bin_file was always null since cache mode is always whole
when this function is called! hash based on the lf_open_opts instead.
* move wasi_emulated_libs into Compilation
- It needs to be accessed from Compilation, which needs to potentially
build those artifacts.
* Compilation: improve error reporting for two cases
- the setMiscFailure mechanism is handy - let's use it!
* fix one instance of incorrectly checking for emit_bin via
`comp.bin_file != null`. There are more instances of this that need to
be fixed in a future commit.
* fix renameTmpIntoCache not handling the case where it needs to make
the "o" directory in the zig-cache directory.
- while I'm at it, simplify the logic for handling the fact that
Windows returns error.AccessDenied rather than
error.PathAlreadyExists for failure to rename a directory over
another one.
* fix missing cache hash additions
- there are still more to add in a future commit -
addNonIncrementalStuffToCacheManifest is called when bin_file is
always null, and then it incorrectly checks if bin_file is non-null
and only then adds a bunch of stuff to the cache hash. It needs to
instead add to the cache hash based on lf_open_opts.
This is necessary because on COFF, the entry symbol name is not known
until the linker has looked at the set of global symbol names to
determine which of the four possible main entry points is present.
In Compilation.create, update the resolved config to account for the
default resolution of the root module. This makes it so that, for
example, reading comp.config.any_non_single_threaded is valid in order
to determine whether any module has single_threaded=false.
