No longer introduce build options for tests. Instead, ZIG_EXE
environment variable is added to any invocation of `zig run` or `zig
test`.
The end result of this branch is the same: there is no longer a
mandatory positional command line argument when invoking zig test
binaries directly.
Before, native glibc and dynamic linker detection attempted to use the
executable's own binary if it was dynamically linked to answer both the
C ABI question and the dynamic linker question. However, this could be
problematic on a system that uses a RUNPATH for the compiler binary,
locking it to an older glibc version, while system binaries such as
/usr/bin/env use a newer glibc version. The problem is that libc.so.6
glibc version will match that of the system while the dynamic linker
will match that of the compiler binary. Executables with these versions
mismatching will fail to run.
Therefore, this commit changes the logic to be the same regardless of
whether the compiler binary is dynamically or statically linked. It
inspects `/usr/bin/env` as an ELF file to find the answer to these
questions, or if there is a shebang line, then it chases the referenced
file recursively. If that does not provide the answer, then the function
falls back to defaults.
This commit also solves a TODO to remove an Allocator parameter to the
detect() function.
This effectively reverts 22690efcc2378222503cb8aaad26a6f4a539f5aa,
re-opening #11818.
This had the following problems:
* Buggy on some targets (macOS, Windows)
* Messy output to the terminal
These problems need to be solved before moving forward with this.
This makes it so that in a -Dsingle-threaded build of test-cases, if a
crash happens, the test case name will be printed just before the stderr
of the crash.
* proper skip_stage1 mechanism that doesn't get side-stepped with
manually added test cases.
* avoid runtime-known function pointers.
* check for type equality more simply without checking the type name.
Instead of always using std.testing.allocator, the test harness now follows
the same logic as self-hosted for choosing an allocator - that is - it
uses C allocator when linking libc, std.testing.allocator otherwise, and
respects `-Dforce-gpa` to override the decision. I did this because
I found GeneralPurposeAllocator to be prohibitively slow when doing
multi-threading, even in the context of a debug build.
There is now a second thread pool which is used to spawn each
test case. The stage2 tests are passed the first thread pool. If it were
only multi-threading the stage1 tests then we could use the same thread
pool for everything. However, the problem with this strategy with stage2
is that stage2 wants to spawn tasks and then call wait() on the main
thread. If we use the same thread pool for everything, we get a deadlock
because all the threads end up all hanging at wait() and nothing is
getting done. So we use our second thread pool to simulate a "process pool"
of sorts.
I spent most of the time working on this commit scratching my head trying
to figure out why I was getting ETXTBSY when spawning the test cases.
Turns out it's a fundamental Unix design flaw, already a known, unsolved
issue by Go and Java maintainers:
https://github.com/golang/go/issues/22315https://bugs.openjdk.org/browse/JDK-8068370
With this change, the following command, executed on my laptop, went from
6m24s to 1m44s:
```
stage1/bin/zig build test-cases -fqemu -fwasmtime -Denable-llvm
```
closes#11818
I'm not really happy with parsing compile errors; I think we should just
be checking that the expected compile error matches the actual rendered
version. I will save that change for a later date however.
Prior to this change, for an example compiler error test case with
multiple identical errors messages such as
```
:1:2: error: foo
:1:2: error: foo
```
the test harness would never increment the error index thus only
marking the very first error message as handled yielding a false
positive.
Additionally, while here, regress `dereference_anyopaque` test case
as not passing on `wasm32-wasi` target.
While calling `next` an error can occur while parsing the file.
However, we don't set the filename that is currently being processed, until `next` completed successfully.
This means that for invalid test names, the wrong filename was being displayed in the panic message.
The fix is to retrieve the correct filename when an error occurs and then setting the filename appropriately.
* migrate runtime safety tests to the new test harness
- this required adding compare output / execution support for stage1
to the test harness.
* rename `zig build test-stage2` to `zig build test-cases` since it now
does quite a bit of stage1 testing actually. I named it this way
since the main directory in the source tree associated with these
tests is "test/cases/".
* add some documentation for the test manifest format.
With this improved iterator, type of test is now inferred from
the filename, enabling us to put all cases in one common parent
directory, and iterate over that, thus automating a lot of tasks.
To correctly handle multiple backends crossed with multiple targets,
we need to push all elements in separate allocated arrays rather
than operate on raw iterators. Hence, introduce `getConfigForKeyAlloc`.
Provide default parsers for obvious config options such as
`CrossTarget` or `Backend` (or any enum for that matter).
Unroll iterator loops into multiple cases - we need to create
a Cartesian product for all possibilities specified in the
test manifest.
