22936 Commits

Author SHA1 Message Date
Andrew Kelley
8b2d872020 fix std.Build.TranslateCStep 2023-03-15 10:48:13 -07:00
Andrew Kelley
0e078790fe multiplex compiler progress messages into the build runner 2023-03-15 10:48:13 -07:00
Andrew Kelley
81376e7205 fix UAF in build runner 2023-03-15 10:48:13 -07:00
Andrew Kelley
8acbfafefb compiler: update function accepts a std.Progress.Node
This makes progress be exposed to the top-level caller of update().

I tossed in a bonus change: when the `zig build` subcommand sees exit
code 2, it omits the "following command failed" line, and the build
runner uses exit code 2 when there are compile errors. This tidies up
the output on build failure by a little bit.
2023-03-15 10:48:13 -07:00
Andrew Kelley
85b4b6e3b3 std.Build.InstallArtifactStep: better default step name 2023-03-15 10:48:13 -07:00
Andrew Kelley
7efeedcd89 std.Build.CompileStep: better default step name 2023-03-15 10:48:13 -07:00
Andrew Kelley
7da34bd9e8 build runner: print a fancy tree with build results on failure 2023-03-15 10:48:13 -07:00
Andrew Kelley
27f136e828 build.zig: remove redundant dependency of install step on zig exe 2023-03-15 10:48:13 -07:00
Andrew Kelley
79440d2b47 std.Build.CompileStep: obtain the build output dir from protocol
Now building successfully works again.
2023-03-15 10:48:13 -07:00
Andrew Kelley
7a3dabdc47 build runner: account for debug builds in cleanExit
build runner is always compiled in debug mode, so the switch on
optimization here was silly.
2023-03-15 10:48:13 -07:00
Andrew Kelley
986a30e373 integrate the build runner and the compiler server
The compiler now provides a server protocol for an interactive session
with another process. The build runner uses this protocol to communicate
compilation errors semantically from zig compiler subprocesses to the
build runner.

The protocol is exposed via stdin/stdout, or on a network socket,
depending on whether the CLI flag `--listen=-` or e.g.
`--listen=127.0.0.1:1337` is used.

Additionally:

 * add the zig version string to the build runner cache prefix

 * remove --prominent-compile-errors CLI flag because it no longer does
   anything. Compilation errors are now unconditionally displayed at the
   bottom of the build summary output when using the terminal-based
   build runner.

 * Remove the color field from std.Build. The build steps are no longer
   supposed to interact with stderr directly. Instead they communicate
   semantically back to the build runner, which has its own logic about
   TTY configuration.

 * Use the cleanExit() pattern in the build runner.

 * Build steps can now use error.MakeFailed when they have already
   properly reported an error, or they can fail with any other error
   code in which case the build runner will create a simple message
   based on this error code.
2023-03-15 10:48:13 -07:00
Andrew Kelley
c583d14013 std.fifo: add toOwnedSlice method 2023-03-15 10:48:13 -07:00
Andrew Kelley
a42888e145 std.zig.ErrorBundle: add special representation for empty
This allows the common case of no compilation errors to be represented
without any allocations.
2023-03-15 10:48:13 -07:00
Andrew Kelley
6d88c3e935 add builtin.zig_version_string
sometimes this is more useful than SemanticVersion
2023-03-15 10:48:13 -07:00
Andrew Kelley
6f717b18f0 std.zig.ErrorBundle: rework binary encoding
* Separate into a "WIP" struct and a "finished" struct.
* Use a bit of indirection for error notes to simplify ergonomics of
  this data structure.
2023-03-15 10:48:13 -07:00
Andrew Kelley
572cb24d1a progress towards semantic error serialization
Introduces std.zig.ErrorBundle which is a trivially serializeable set
of compilation errors. This is in the standard library so that both
the compiler and the build runner can use it. The idea is they will
use it to communicate compilation errors over a binary protocol.

The binary encoding of ErrorBundle is a bit problematic - I got a little
too aggressive with compaction. I need to change it in a follow-up
commit to use some indirection in the error message list, otherwise
iteration is too unergonomic. In fact it's so problematic right now that
the logic getAllErrorsAlloc() actually fails to produce a viable
ErrorBundle because it puts SourceLocation data in between the root
level ErrorMessage data.

This commit has a simplification - redundant logic for rendering AST
errors to stderr has been removed in favor of moving the logic for
lowering AST errors into AstGen. So even if we get parse errors, the
errors will get lowered into ZIR before being reported. I believe this
will be useful when working on --autofix. Either way, some redundant
brittle logic was happily deleted.

In Compilation, updateSubCompilation() is improved to properly perform
error reporting when a sub-compilation object fails. It no longer dumps
directly to stderr; instead it populates an ErrorBundle object, which
gets added to the parent one during getAllErrorsAlloc().

In package fetching code, instead of dumping directly to stderr, it now
populates an ErrorBundle object, and gets properly reported at the CLI
layer of abstraction.
2023-03-15 10:48:12 -07:00
Andrew Kelley
4db5bc7b21 std.mem.copy: update to new for loop syntax 2023-03-15 10:48:12 -07:00
Andrew Kelley
c911de825b link.Elf: keep the logic for updates but condition on hcs 2023-03-15 10:48:12 -07:00
Andrew Kelley
4f4ddf5ef2 hot code swapping PoC working
- improve fn prototypes of process_vm_writev
 - make the memory writable in the ELF file
 - force the linker to always append the function
 - write updates with process_vm_writev
2023-03-15 10:48:12 -07:00
Andrew Kelley
50a2bb58d2 link: PTRACE_ATTACH/PTRACE_DETACH 2023-03-15 10:48:12 -07:00
Andrew Kelley
ae8e7c8f5a stage2: hot code swapping PoC
* CLI supports --listen to accept commands on a socket
 * make it able to produce an updated executable while it is running
2023-03-15 10:48:12 -07:00
Andrew Kelley
ee693bfe04 std.os.linux: add ptrace 2023-03-15 10:48:12 -07:00
Andrew Kelley
c641af3cba CI: use -j1 like a coward
Zig's build script has several race conditions preventing proper
concurrent builds from working. By using -j1 for now, finishing this
branch (concurrent zig builds) is untangled from the separate problem of
correcting concurrency issues with zig's own build script.

In other words, let's solve one problem at a time.
2023-03-15 10:48:12 -07:00
Andrew Kelley
cb09470063 zig build: add a -j<N> option for limiting concurrency 2023-03-15 10:48:12 -07:00
Andrew Kelley
96d798db8b update to new for loop syntax 2023-03-15 10:48:12 -07:00
Andrew Kelley
26486c7f23 build runner: show stderr even on successful steps run 2023-03-15 10:48:12 -07:00
Andrew Kelley
c5edd8b7f8 std.Build: better handling of stderr of child processes
With this commit, the build runner now communicates progress towards
completion of the step graph to the terminal, while also printing the
stderr of child processes as soon as possible, without clobbering each
other, and without clobbering the CLI progress output.
2023-03-15 10:48:12 -07:00
Andrew Kelley
7ebaa05bb1 std.Progress: add lock_stderr and unlock_stderr
API users can take advantage of these to freely write to the terminal
which has an ongoing progress display, similar to what Ninja does when
compiling C/C++ objects and a warning or error message is printed.
2023-03-15 10:48:12 -07:00
Andrew Kelley
8d38472293 std.Build: further enhance debug message for bad getPath()
Now it also shows the step stack trace of the step whose make function
is being run.
2023-03-15 10:48:12 -07:00
Andrew Kelley
02381c0372 std.Build: improve debugging of misconfigured steps
* Step.init() now takes an options struct
 * Step.init() now captures a small stack trace and stores it in the
   Step so that it can be accessed when printing user-friendly debugging
   information, including the lines of code that created the step in
   question.
2023-03-15 10:48:12 -07:00
Andrew Kelley
9580fbcf35 build system: capture stderr and report it later
Instead of dumping directly to stderr. This prevents processes running
simultaneously from racing their stderr against each other.

For now it only reports at the end, but an improvement would be to
report as soon as a failed step occurs.
2023-03-15 10:48:12 -07:00
Andrew Kelley
1fa1484288 build runner: proper threaded dependency management
After sorting the step stack so that dependencies can be popped before
their dependants are popped, there is still a situation left to handle
correctly:

Example:

A depends on:
  B
  C
D depends on:
  E
  F

They will be ordered like this:

A B C D E F

If there are 6+ cores, then all of them will be evaluated at once,
incorrectly evaluating A and D before their dependencies.

Starting evaluation of F and then E is correct, but waiting until they
are done is not correct because it should start working on B and C as
well.

This commit solves the problem by computing dependants in the dependency
loop checking logic, and then having workers queue up their dependants
when they finish their own work.
2023-03-15 10:48:12 -07:00
Andrew Kelley
cff86cf7a1 build_runner now executes the step graph in parallel 2023-03-15 10:48:12 -07:00
Andrew Kelley
658de75500 add std.heap.ThreadSafeAllocator
This wraps any allocator and makes it thread-safe by using a mutex.
2023-03-15 10:48:12 -07:00
Andrew Kelley
5b90fa05a4 extract ThreadPool and WaitGroup from compiler to std lib 2023-03-15 10:48:12 -07:00
Andrew Kelley
0b744d7d67 build runner: untangle dependency loop checking from making 2023-03-15 10:48:12 -07:00
Andrew Kelley
c6a895f667 extract some logic from std.Build to build_runner.zig 2023-03-15 10:48:12 -07:00
Andrew Kelley
a2c6ecd6dc
Merge pull request #14918 from jacobly0/x86_64
x86_64: fix various backend crashes
2023-03-15 13:08:26 -04:00
Jacob Young
d70955b0df x86_64: turn packed struct crashes into compile errors 2023-03-15 01:04:21 -04:00
Jacob Young
ba9d93dc9f x86_64: implement more binary immediate combinations 2023-03-15 01:04:21 -04:00
Jacob Young
05b12e6779 x86_64: handle duplicate prong deaths 2023-03-15 01:04:21 -04:00
Jacob Young
2386159840 x86_64: use short union init 2023-03-15 01:04:21 -04:00
Jacob Young
d14a9e82fe x86_64: use new for loop syntax 2023-03-15 01:04:21 -04:00
Jacob Young
bb6b9c19e0 x86_64: fix lowering of non-pointer optional is null 2023-03-15 01:04:21 -04:00
Jacob Young
c51930b060 behavior: enable passing behavior tests on stage2_x86_64 2023-03-15 01:04:21 -04:00
Jacob Young
5ab426a302 x86_64: fix store of undefined 2023-03-15 00:01:44 -04:00
Jacob Young
9a4e9215fc x86_64: fix error code paths to not have extra pops 2023-03-15 00:01:44 -04:00
Jacob Young
8be6073480 tools: fix typo in lldb command 2023-03-15 00:01:44 -04:00
Bas Westerbaan
4414f9c46e
Add Kyber post-quantum key encapsulation mechanism (#14902)
Implementation of the IND-CCA2 post-quantum secure key encapsulation
mechanism (KEM) CRYSTALS-Kyber, as submitted to the third round of the NIST
Post-Quantum Cryptography (v3.02/"draft00"), and selected for standardisation.

Co-authored-by: Frank Denis <124872+jedisct1@users.noreply.github.com>
2023-03-15 03:50:45 +00:00
Frank Denis
e17998b396
Argon2: properly handle outputs > 64 bytes in blake2Long() (#14914)
Fixes #14912
2023-03-14 21:40:02 +00:00