109 Commits

Author SHA1 Message Date
Andrew Kelley
87c6341b61 stage2: add extern functions
and improve the C backend enough to support Hello World (almost)
2020-12-28 17:15:29 -07:00
Timon Kruiper
071417161d stage2: add initial impl of LLVM backend in self-hosted compiler 2020-12-28 21:19:40 +01:00
Andrew Kelley
f75d4cbe56 Revert "stage2: add compile log statement (#7191)"
The addition of `addDeclErr` introduced a memory leak at every call
site, and I also would like to push back on having more than 1
compilation error per `Decl`.

This reverts commit 1634d45f1d53c8d7bfefa56ab4d2fa4cc8218b6d.
2020-12-28 11:24:53 -07:00
g-w1
1634d45f1d
stage2: add compile log statement (#7191) 2020-12-26 02:40:49 +02:00
Andrew Kelley
6ab9268a90 stage2: re-use compiler runtime libs across opt modes and strip flag
Previously Zig would need to recompile runtime libs if you changed the
values of --strip or -O. Now, unless the `debug_compiler_runtime_libs`
flag is set (which is currently not exposed to the CLI), Zig will always
choose ReleaseFast or ReleaseSmall for compiler runtime libraries.

When the main application chooses ReleaseFast or ReleaseSmall, that
value is propagated to compiler runtime libraries. Otherwise a decision
is made based on the target, which is currently ReleaseSmall for
freestanding WebAssembly and ReleaseFast for everything else.

Ultimately the purpose of this commit is to have Debug and ReleaseSafe
builds of applications still get optimized builds of, e.g. libcxx and
libunwind, as well as to spend less time unnecessarily rebuilding compiler
runtime libraries.
2020-12-24 14:11:58 -07:00
Andrew Kelley
ed26b3204a stage2: tsan forces linking libc 2020-12-24 01:18:48 -07:00
Andrew Kelley
a4b134746f zig cc: support both ubsan and tsan at the same time 2020-12-24 01:18:48 -07:00
Andrew Kelley
8219d92987 stage2: fix Cache deadlock and build more of TSAN
* rename is_compiler_rt_or_libc to skip_linker_dependencies
   and set it to `true` for all sub-Compilations. I believe
   this resolves the deadlock we were experiencing on Drone
   CI and on some users' computers. I will remove the CI workaround in
   a follow-up commit.
 * enabling TSAN automatically causes the Compilation to link against
   libc++ even if not requested, because TSAN depends on libc++.
 * add -fno-rtti flags where appropriate when building TSAN objects.
   Thanks Firefox317 for pointing this out.
 * TSAN support: resolve all the undefined symbols. We are still seeing
   a dependency on __gcc_personality_v0 but will resolve this one in a
   follow-up commit.
 * static libs do not try to build libc++ or libc++abi.
2020-12-24 01:18:48 -07:00
Andrew Kelley
42b4a48bc9 WIP start adding support for TSAN 2020-12-24 01:18:47 -07:00
Andrew Kelley
177377b6e3 rework std.ResetEvent, improve std lib Darwin integration
* split std.ResetEvent into:
   - ResetEvent - requires init() at runtime and it can fail. Also
     requires deinit().
   - StaticResetEvent - can be statically initialized and requires no
     deinitialization. Initialization cannot fail.
 * the POSIX sem_t implementation can in fact fail on initialization
   because it is allowed to be implemented as a file descriptor.
 * Completely define, clarify, and explain in detail the semantics of
   these APIs. Remove the `isSet` function.
 * `ResetEvent.timedWait` returns an enum instead of a possible error.
 * `ResetEvent.init` takes a pointer to the ResetEvent instead of
   returning a copy.
 * On Darwin, `ResetEvent` is implemented using Grand Central Dispatch,
   which is exposed by libSystem.

stage2 changes:
 * ThreadPool: use a single, pre-initialized `ResetEvent` per worker.
 * WaitGroup: now requires init() and deinit() and init() can fail.
   - Add a `reset` function.
   - Compilation initializes one for the work queue in creation and
     re-uses it for every update.
   - Rename `stop` to `finish`.
   - Simplify the implementation based on the usage pattern.
2020-12-23 16:57:18 -08:00
Alex Cameron
9849e894d5 Add emit_h path to compilation hash. 2020-12-23 01:14:35 +11:00
Noam Preil
15a148db01 Fix memory leak 2020-12-23 01:14:35 +11:00
Alex Cameron
c87da2f45a Remove redundant emit_h member in Compilation struct. 2020-12-23 01:14:35 +11:00
Alex Cameron
40f0275e7c Implement emit-h 2020-12-23 01:14:35 +11:00
Andrew Kelley
1d94a68936 add an option to compile zig in single-threaded mode
And enable it for Drone CI. I hate to do this, but I need to make
progress on other fronts.
2020-12-20 15:37:58 -07:00
Andrew Kelley
aa6ef10cc6 std.Progress: make the API thread-safe
We generally get away with atomic primitives, however a lock is required
around the refresh function since it traverses the Node graph, and we
need to be sure no references to Nodes remain after end() is called.
2020-12-20 15:08:59 -07:00
Andrew Kelley
0d1cd0d482 use kprotty's ThreadPool implementation (v5) 2020-12-20 15:08:59 -07:00
Andrew Kelley
01d33855c7 stage2: protect mutable state from data races in updateCObject 2020-12-20 15:08:59 -07:00
Andrew Kelley
013efaf139 std: introduce a thread-local CSPRNG for general use
std.crypto.random

* cross platform, even freestanding
* can't fail. on initialization for some systems requires calling
  os.getrandom(), in which case there are rare but theoretically
  possible errors. The code panics in these cases, however the
  application may choose to override the default seed function and then
  handle the failure another way.
* thread-safe
* supports the full Random interface
* cryptographically secure
* no syscall required to initialize on Linux (AT_RANDOM)
* calls arc4random on systems that support it

`std.crypto.randomBytes` is removed in favor of `std.crypto.random.bytes`.

I moved some of the Random implementations into their own files in the
interest of organization.

stage2 no longer requires passing a RNG; instead it uses this API.

Closes #6704
2020-12-18 12:22:46 -07:00
Andrew Kelley
8975fa5b38 stage2: add test_evented_io to cache hash 2020-12-16 19:22:48 -07:00
Isaac Freund
1d8f33ca98
stage2: link musl dynamically by default if native
If targeting the native OS and the system libc is musl, link against it
dynamically by default.
2020-12-13 00:40:35 +01:00
Isaac Freund
307d98dc35
stage2: support dynamically linking musl libc 2020-12-13 00:40:35 +01:00
Andrew Kelley
6ab5bebed1 stage2: proper file extension stripping
Previously it used mem.split on "." and took the first iterated item.
Now it uses fs.path.extension and strips off that number of bytes.

Closes #7404
2020-12-11 17:42:13 -07:00
Andrew Kelley
c4f53d1ef6 fix deadlock with build-exe on an object for windows
The steps to repro this issue are:

zig build-obj hello.zig -target x86_64-windows-msvc
zig build-exe hello.obj -target x86_64-windows-msvc --subsystem console
-lkernel32 -lntdll

What was happening is that the main Compilation added a work item to
produce kernel32.lib. Then it added a sub-Compilation to build zig's
libc, which ended up calling a function with extern "kernel32", which
caused the sub-Compilation to also try to produce kernel32.lib. The main
Compilation and sub-Compilation do not coordinate about the set of
import libraries that they will be trying to build, so this caused a
deadlock.

This commit solves the problem by disabling the extern "foo" feature
from working when building compiler_rt or libc. Zig's linker code is now
responsible for putting the appropriate import libs on the linker line,
if any for compiler_rt and libc.

Related: #5825
2020-12-11 18:34:34 -05:00
Andrew Kelley
64a590a311 stage2: detect redundant C/C++ source files
Cache exposes BinDigest.

Compilation gains a set of a BinDigest for every C/C++ source file. We
detect when the same source/flags have already been added and emit a
compile error. This prevents a deadlock in the caching system.

Closes #7308
2020-12-11 02:33:09 -05:00
Andrew Kelley
7dd4afb224 stage2: link: properly implement passthrough mode for LLD child proc
passthrough mode does not mean always exit - it just means to pass
through stdio and exit if the child process exits, without doing any
special error reporting.
2020-12-08 22:37:01 -07:00
Andrew Kelley
24cbf1c706 always use codeview (pdb) when object_format is PE/COFF
Previously, when mixing Zig and C/C++ code for windows-gnu targets, zig
would get codeview format but the C/C++ code would not get any debug
info. Now, C/C++ code properly emits debug info in codeview format and
everything just works.
2020-12-08 13:31:39 -07:00
Andrew Kelley
2ed1ed9b32 stage2: introduce Module.failed_root_source_file
Use case:

zig build-exe non_existent_file.zig

Previous behavior:

error.FileNotFound, followed by an error return trace

Behavior after this commit:

error: unable to read non_existent_file.zig: FileNotFound
(end of stderr, exit code 1)

This turns AllErrors.Message into a tagged union which now has the
capability to represent both "plain" errors as well as source-based
errors (with file, line, column, byte offset). The "no entry point found"
error has moved to be a plain error message.
2020-12-04 17:21:55 -07:00
Jakub Konka
c4853ba7ea Version-gate appending -syslibroot flag to lld
This commit version-gates appending `-syslibroot` flag to lld. This
is predicated upon the fact that for versions of macOS lower than
11, lld would fail to find and link against frameworks with this
flag specified.

Co-authored-by: Andrew Kelley <andrew@ziglang.org>
2020-12-04 11:37:55 -08:00
Andrew Kelley
ddaf915389 add @cImport files to Compilation cache manifest
closes #7007
2020-12-03 16:07:38 -07:00
LemonBoy
c91c4dc256 stage2: Create cache manifest before calling final
If we enter the `if` because `comp.disable_c_depfile` is false the `man`
object has no manifest and calling .final on it will trip an assertion.

Closes #7096
2020-12-01 16:52:28 -07:00
Andrew Kelley
5b5097a22a stage2: add -femit-foo=bar args to the cache hash
Closes #6979
Closes #7036
2020-12-01 16:35:27 -07:00
Alexandros Naskos
8d6eff5cb9 Add package names and paths to the zig root module cache hash.
Add package names to the stage1 cache hash and package files to the
stage1 manifest file.
2020-12-01 13:56:36 -08:00
Timon Kruiper
8cbcc61c4d Make sure to include the root_name in the cache.
This fixes a bug where the caching system did not notice when the
--name flag changed.
2020-12-01 10:38:33 -08:00
Andrew Kelley
21565ca991 restore -target wasm32-freestanding-musl for C headers
See #5854

Some tiny tweaks too:
 * Use `wasm-freestanding-musl` instead of `wasm32-freestanding-musl`,
   making it pointer-size-agnostic.
 * Fix trying to build non-existent wasm musl start files.
2020-11-30 20:44:36 -07:00
Andrew Kelley
429a219f42 stage2: fix not detecting all dynamic libraries
Positional shared library arguments were not being detected as causing
dynamic linking, resulting in invalid linker lines. LLD did not have an
error message for this when targeting x86_64-linux but it did emit an
error message when targeting aarch64-linux, which is how I noticed the
problem.

This surfaced an error having to do with fifo.pipe() in the cat example
which I did not diagnose but solved the issue by doing the revamp that
was already overdue for that example.

It appears that the zig-window project was exploiting the previous
behavior for it to function properly, so this prompts the question, is
there some kind of static/dynamic executable hybrid that the compiler
should recognize? Unclear - but we can discuss that in #7240.
2020-11-30 20:25:28 -07:00
Andrew Kelley
8c9919ec7b fix regression on wasm targets
The previous commit broke wasm targets because the linking step would
look for the compiler-rt lib in the wrong place. Fixed in this commit.
2020-11-30 17:46:10 -07:00
Andrew Kelley
ff9798eb26 rework the bundle compiler-rt feature
* it is now -fcompiler-rt and -fno-compiler-rt to override the (quite
   reasonable) default of bundling compiler-rt only for executables and
   dynamic libraries.
   - the build.zig API is still called bundle_compiler_rt however it is
     now an optional bool instead of a bool. leaving it as `null` means
     to use the compiler default.
 * renamed some internal identifiers to make the source more readable
 * additionally support -fcompiler-rt when doing build-obj for ELF files
   since that target already supports linking multiple objects into one.
   - includes an error message when attempting this for non-ELF. in the
     future this could additionally be supported with a more advanced
     implementation that does not rely on the linker.
 * properly populate the linker cache hash
2020-11-30 16:35:00 -07:00
Alexandros Naskos
2fae28b6af Added bundle-compiler-rt flag 2020-11-30 14:31:41 -07:00
Jakub Konka
375bab8460 stage2 elf: refactor override_soname to soname 2020-11-27 15:42:39 -07:00
Andrew Kelley
abc717f203 modernize the PIE patch for the latest master branch
This is the part of #3960 that has to be rewritten to apply to latest
master branch code.
2020-11-22 17:28:11 -07:00
Andrew Kelley
6904cd828e add missing -m<os>-version-min CLI args to clang
This fixes some code generation issues when targeting macOS and
compiling C/C++ code.
2020-11-10 15:24:10 -07:00
Andrew Kelley
4d67c3d887 macOS: depend on bundled headers even for native
This is an alternate fix for #6773. Reverts
872bc787b56f71e53c80f4681523bc8356915b71.
2020-11-08 18:51:12 -07:00
Jakub Konka
ab69b89d52 Address review comments 2020-11-06 11:57:53 +01:00
Jakub Konka
b7c3ebcb9e Rely on ZIG_SYSTEM_LINKER_HACK instead of input flags 2020-11-06 11:34:51 +01:00
Jakub Konka
e0e3ceac19 Re-enable system linker hack
It is now possible to force linking with system linker `ld` instead
of the LLVM `lld` linker when building natively on the target. This
can be done at each stage by specifying `--system-linker-hack` flag,
and can be useful on platforms where `lld` fails to operate properly
such as macOS 11 Big Sur on ARM64 where every binary/dylib is expected
to be codesigned.

Some example invocations for each stage of compilation of Zig
toolchain:

```
cmake .. -DCMAKE_PREFIX_PATH=/path/to/llvm -DSYSTEM_LINKER_HACK=1
```

```
build/zig build test --system-linker-hack
```

```
build/zig build --prefix $(pwd)/stage2 -Denable-llvm
--system-linker-hack
```

```
build/zig build-exe hello.zig --system-linker-hack
```
2020-11-05 18:35:51 +01:00
Jakub Konka
6ecefd5903 Add std.Target.current.isDarwin() to exclude non-macs in comptime 2020-11-02 19:31:07 +01:00
Jakub Konka
f42ebd1b0a
Update src/Compilation.zig
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
2020-11-02 18:56:08 +01:00
Jakub Konka
7f5c96378d Refactor the code according to Andrew's suggestions 2020-11-02 18:38:09 +01:00
Jakub Konka
317c555a5c Fix linking issues on BigSur
This commit fixes linking issue on macOS 11 BigSur by appending
a prefix path to all lib and framework search paths known as
`-syslibroot`.

The reason this is needed is that in macOS 11, the system libraries
and frameworks are no longer readily available in the filesystem.
Instead, the new macOS ships with a built-in dynamic linker cache
of all system-provided libraries, and hence, when linking with either
`lld.ld64` or `ld64`, it is required to pass in `-syslibroot [dir]`.
The latter can usually be obtained by invoking `xcrun --show-sdk-path`.
With this commit, Zig will do this automatically when compiling natively
on macOS. However, it also provides a flag `-syslibroot` which can be
used to overwrite the automtically populated value.

To summarise, with this change, the user of Zig is not required to
generate and append their own syslibroot path. Standard invocations
such as `zig build-exe hello.zig` or `zig build` for projects will
work out of the box. The only missing bit is `zig cc` and `zig c++`
since the addition of the `-syslibroot` option would be a mismatch
between the values provided by `clang` itself and Zig's wrapper.
2020-11-02 17:06:09 +01:00