1171 Commits

Author SHA1 Message Date
Andrew Kelley
934573fc5d Revert "std.fmt: require specifier for unwrapping ?T and E!T."
This reverts commit 7cbd586ace46a8e8cebab660ebca3cfc049305d9.

This is causing a fail to build from source:

```
./lib/std/fmt.zig:492:17: error: cannot format optional without a specifier (i.e. {?} or {any})
                @compileError("cannot format optional without a specifier (i.e. {?} or {any})");
                ^
./src/link/MachO/Atom.zig:544:26: note: called from here
                log.debug("  RELA({s}) @ {x} => %{d} in object({d})", .{
                         ^
```

I looked at the code to fix it but none of those args are optionals.
2022-07-24 11:50:10 -07:00
InKryption
7cbd586ace
std.fmt: require specifier for unwrapping ?T and E!T.
Co-authored-by: Veikka Tuominen <git@vexu.eu>
2022-07-24 12:01:56 +03:00
Jakub Konka
cf6cfc830d macho: fix use-after-move in placeDecl
Previously, we would get a pointer to a slot in the symbol table,
apply changes to the symbol, and return the pointer. This however
didn't take into account that the symbol table may be moved in memory
in-between the modification and return from the function (`fn placeDecl`).
Prior to my rewrite, this was not possible within the body of the said
function. However, my rewrite revamped how we allocate GOT atoms and
their matching symtab indexes, which now may cause a move in memory
of the container.
2022-07-23 00:13:34 +02:00
Jakub Konka
600348283f macho: include __StaticInit input sections as GC roots 2022-07-22 21:19:33 +02:00
Jakub Konka
f8458a549b macho: dead_strip defaults: off for debug, on for release 2022-07-22 16:58:21 +02:00
Jakub Konka
0901e4805c macho: fix zig cc and c++ using stage2 llvm 2022-07-22 16:58:21 +02:00
Jakub Konka
ca74656685 macho: move GC code into dead_strip.zig module
Implement marking live atoms that reference other live atoms if
required by the compiler (via section attribute).
2022-07-22 16:58:21 +02:00
Jakub Konka
7345976261 macho: sort subsection symbols by seniority 2022-07-22 16:58:21 +02:00
Jakub Konka
39df241df4 macho: do not GC local symbols unless reference dead symbols
If a local references another local, we keep it. If it doesn't
reference anything, we keep it. Otherwise, we dead strip it.
2022-07-22 16:58:21 +02:00
Jakub Konka
a089a6dc4f macho: parse data-in-code when writing LINKEDIT segment 2022-07-22 16:58:21 +02:00
Jakub Konka
2dfc78dc03 macho: limit export info to entrypoint and mh symbol when executable 2022-07-22 16:58:21 +02:00
Jakub Konka
0f1b5d45bc macho: mark __mh_execute_header as ref'd dynamically 2022-07-22 16:58:21 +02:00
Jakub Konka
0da8ba816a macho: do not store stabs; generate on-the-fly instead 2022-07-22 16:58:21 +02:00
Jakub Konka
41b91442f4 macho: improve logs for dyld info 2022-07-22 16:58:21 +02:00
Jakub Konka
4658d857de macho: fix caching linker line in incremental setting 2022-07-22 16:58:20 +02:00
Jakub Konka
b380ed6a72 macho: deallocate globals keys 2022-07-22 16:58:20 +02:00
Jakub Konka
817939d20a macho: don't store GC roots globally
Instead, if dead-strip was requested, create a temp container and
pass it around.
2022-07-22 16:58:20 +02:00
Jakub Konka
61b4119a7d macho: link atom starting section by orig section id
In x86_64 relocs, it can so happen that the compiler
refers to the same atom by both the actual assigned symbol
and the start of the section. In this case, we need to
link the two together so add an alias.
2022-07-22 16:58:20 +02:00
Jakub Konka
35a5a4a0e4 macho: fix marking sections for pruning in GC 2022-07-22 16:58:20 +02:00
Jakub Konka
d80fcc8a0b macho: rework symbol handling for incremental stage2 builds 2022-07-22 16:58:20 +02:00
Jakub Konka
eeb6d8f045 macho: fix compilation issues on 32bit hosts 2022-07-22 16:58:20 +02:00
Jakub Konka
9eb7e5182b macho: rework symbol handling to match zld/ELF
Now, each object file will store a mutable table of symbols that it
defines. Upon symbol resolution between object files, the symbol
will be updated with a globally allocated section ordinal and address
in virtual memory. If the object defines a globally available symbol,
its location only (comprising of the symbol index and object index)
will be stored in the globals map for easy access when relocating, etc.
This approach cleans up the symbol management significantly, and matches
the status quo used in zld/ELF.

Additionally, this makes scoping symbol stabs easier too as they are
now naturally contained within each object file.
2022-07-22 16:58:20 +02:00
Jakub Konka
843701d0fe macho: remove unused fields from Atom 2022-07-22 16:58:20 +02:00
Jakub Konka
03feea0fb2 macho: split section into subsections if requested and/or possible 2022-07-22 16:58:20 +02:00
Jakub Konka
d042b88c11 macho: put parsing into an atom into a helper 2022-07-22 16:58:20 +02:00
Jakub Konka
b81c48d985 macho: read the entire file contents into memory at once 2022-07-22 16:58:20 +02:00
Jakub Konka
44ee42c6bc cli: parse -dead_strip MachO linker flag 2022-07-22 16:58:20 +02:00
Luuk de Gram
8033767082
wasm-linker: Implement linker tests (#12006)
* test/link: initial wasm support

This adds basic parsing and dumping of wasm section so they
can be tested using the new linker-test infrastructure.

* test/link: all wasm sections parsing and dumping

We now parse and dump all sections for the wasm binary format.
Currently, this only dumps the name of a custom section.
Later this should also dump symbol table, name, linking metadata and relocations.
All of those live within the custom sections.

* Add wasm linker test

This also fixes a parser mistake in reading the flags.

* test/link: implement linker tests wasm & fixes

Adds several test cases to test the wasm self-hosted linker.
This also introduces fixes that were caught during the implementation
of those tests.

* test-runner: obey omit_stage2 for standalone

When a standalone test requires stage2, but stage2 is omit
from the compiler, such test case will not be included as part
of the test suite that is being ran. This is to support CI's
where we omit stage2 to lower the memory usage.
2022-07-12 14:36:33 +02:00
Andrew Kelley
2b99182e25 stage2: cleanups to --compress-debug-sections
* make the setting in the linker backend be non-optional; by this time
   all defaults are supposed to be resolved.
 * integrate with `zig cc`
 * change the CLI parsing to match C compiler parsing, allowing
   `--compress-debug-sections` alone to choose a default encoding of
   zlib.
2022-07-11 14:18:56 -07:00
Motiejus Jakštys
1f410b500c ELF: understand -Wl,--compress-debug-sections
This argument is both a compiler and a linker flag. The linker flag was
not understood; now it is. Go likes to use it as a linker flag.

Tested with sqlite3. The size difference is significant, and I confirmed
gdb understands both binaries.

zlib: 3.66MB
------------

    CC="zig cc"
    CFLAGS="-Wl,--compress-debug-sections=zlib -O2"
    ./configure --disable-tcl
    make

        FILE SIZE        VM SIZE
     --------------  --------------
      39.1%  1.43Mi  88.4%  1.43Mi    .text
      19.6%   734Ki   0.0%       0    .debug_info
      16.4%   613Ki   0.0%       0    .debug_loc
      13.1%   492Ki   0.0%       0    .debug_line
       4.2%   157Ki   9.5%   157Ki    .rodata
       2.3%  87.6Ki   0.0%       0    .debug_ranges
       1.5%  56.2Ki   0.0%       0    .symtab
       1.1%  40.2Ki   0.0%       0    .strtab
       1.0%  38.2Ki   0.0%       0    .debug_str
       0.7%  26.2Ki   0.0%       0    .debug_frame
       0.4%  15.3Ki   0.9%  15.3Ki    .data
       0.1%  4.71Ki   0.3%  4.71Ki    .dynsym
       0.1%  3.65Ki   0.2%  3.26Ki    [16 Others]
       0.1%  2.55Ki   0.2%  2.55Ki    .rela.plt
       0.1%  2.12Ki   0.0%       0    [ELF Section Headers]
       0.0%       0   0.1%  2.02Ki    .bss
       0.0%  1.84Ki   0.1%  1.84Ki    .dynstr
       0.0%  1.72Ki   0.1%  1.72Ki    .plt
       0.0%  1.58Ki   0.1%  1.58Ki    .hash
       0.0%  1.17Ki   0.0%       0    .debug_abbrev
       0.0%  1.01Ki   0.1%  1.01Ki    .rela.dyn
     100.0%  3.66Mi 100.0%  1.62Mi    TOTAL

none: 8.56MB
------------

    CC="zig cc" CFLAGS="-O2" ./configure --disable-tcl
    make

        FILE SIZE        VM SIZE
     --------------  --------------
      41.1%  3.52Mi   0.0%       0    .debug_loc
      18.5%  1.59Mi   0.0%       0    .debug_info
      16.7%  1.43Mi  88.4%  1.43Mi    .text
      11.8%  1.01Mi   0.0%       0    .debug_line
       5.9%   515Ki   0.0%       0    .debug_ranges
       1.8%   157Ki   9.5%   157Ki    .rodata
       1.3%   118Ki   0.0%       0    .debug_frame
       1.3%   110Ki   0.0%       0    .debug_str
       0.6%  56.2Ki   0.0%       0    .symtab
       0.5%  40.2Ki   0.0%       0    .strtab
       0.2%  15.3Ki   0.9%  15.3Ki    .data
       0.1%  4.71Ki   0.3%  4.71Ki    .dynsym
       0.0%  3.64Ki   0.2%  3.26Ki    [16 Others]
       0.0%  2.98Ki   0.0%       0    .debug_abbrev
       0.0%  2.55Ki   0.2%  2.55Ki    .rela.plt
       0.0%  2.12Ki   0.0%       0    [ELF Section Headers]
       0.0%       0   0.1%  2.02Ki    .bss
       0.0%  1.84Ki   0.1%  1.84Ki    .dynstr
       0.0%  1.72Ki   0.1%  1.72Ki    .plt
       0.0%  1.58Ki   0.1%  1.58Ki    .hash
       0.0%  1.01Ki   0.1%  1.01Ki    .rela.dyn
     100.0%  8.56Mi 100.0%  1.62Mi    TOTAL
2022-07-11 13:55:29 -07:00
Andrew Kelley
6fc9f6c5f6 link.Coff: notice special windows symbols
in order to have a better default for subsystem. This brings stage2
behavior on par with stage1.
2022-07-04 17:20:01 -07:00
Andrew Kelley
d65e248ed1 stage2: ELF: improve error reporting when libc is missing
Future improvement: make plain error notes actually render as notes
rather than errors, but keep them as errors for the case of
sub-compilation errors, e.g. when compiler-rt has compilation errors.
2022-07-04 14:06:20 -07:00
Jakub Konka
b79884eaf0 macho: implement pruning of unused segments and sections
This is a prelude to a more elaborate work which will implement
`-dead_strip` flag - garbage collection of unreachable atoms. Here,
when sorting sections, we also check that the section is actually
populated with some atoms, and if not, we exclude it from the final
linked image. This can happen when we do not import any symbols
from dynamic libraries in which case we will not be populating
the stubs sections or the GOT table, implying we can skip allocating
those sections. Furthermore, we also make a check that a segment
is actually occupied too, with the exception of `__TEXT` segment
which is non-optional given that it wraps the header and load commands
and thus is required by the `dyld` to perform dynamic linking, and
`__PAGEZERO` which is generally non-optional when the linked image
is an executable. For any other segment, if its section count is
zero, we mark it as dead and skip allocating it and generating
a load command for it.

This commit also includes some minor improvements to the linker such
as refactoring of the segment allocating codepaths, skipping
`__PAGEZERO` generation for dylibs, and skipping generation of zero-sized
atoms for special symbols such as `__mh_execute_header` and `___dso_handle`.
These special symbols are only allocated local and global symbol pair
and their VM addresses is set to the start of the `__TEXT` segment,
but no `Atom` is created, as it's not necessary given that they never
carry any machine code.

Finally, we now always force-link against `libSystem` which turns out
to be required for `dyld` to properly handle `LC_MAIN` load command
on older macOS versions such as 10.15.7.
2022-07-01 22:05:22 +02:00
Jakub Konka
cd5dcfbf41 macho: annotate weak imports when linking with weak lib/framework 2022-06-28 09:19:01 +02:00
Jakub Konka
f353b59efa macho: discriminate between normal and weak dylibs
Parse `-weak-lx` and `-weak_framework x` in the CLI.
2022-06-28 09:18:54 +02:00
Jakub Konka
0dd28920da macho: implement and handle -needed-* and -needed_* family of flags
MachO linker now handles `-needed-l<name>`, `-needed_library=<name>`
and `-needed_framework=<name>`. While on macOS `-l` is equivalent
to `-needed-l`, and `-framework` to `-needed_framework`, it can be
used to the same effect as on Linux if combined with `-dead_strip_dylibs`.

This commit also adds handling for `-needed_library` which is macOS
specific flag only (in addition to `-needed-l`).

Finally, in order to leverage new linker testing harness, this commit
added ability to specify lowering to those flags via `build.zig`:
`linkSystemLibraryNeeded` (and related), and `linkFrameworkNeeded`.
2022-06-27 19:53:38 +02:00
Jakub Konka
efc5c97bff macho: implement -dead_strip_dylibs linker flag 2022-06-27 19:53:38 +02:00
Jakub Konka
08459ff1c2
Merge pull request #11933 from Luukdegram/wasm-link-bss
stage2: wasm-linker - Place decls in the correct segment and order segments
2022-06-25 22:51:06 +02:00
Luuk de Gram
140bac6395
link/wasm: Sort data segments
We now ensure the "bss" section is last, which allows us to not
emit this section and let the runtime initialize the memory with 0's instead.
This allows for smaller binaries.
The order of the other segments is arbitrary and does not matter, this may
change in the future.
2022-06-25 18:36:56 +02:00
Jakub Konka
589bf67635 macho: implement -headerpad_max_install_names 2022-06-25 18:04:40 +02:00
Jakub Konka
8c1feef4cd macho: implement -headerpad_size option
Includes both traditiona and incremental codepaths with one caveat that
in incremental case, the requested size cannot be smaller than the
default padding size due to prealloc required due to incremental nature
of linking.

Also parse `-headerpad_max_install_names`, however, not actionable just yet -
missing implementation.
2022-06-25 17:59:08 +02:00
Jakub Konka
dfdb807543 cache setting macho search strategy flags 2022-06-25 10:50:00 +02:00
Luuk de Gram
e32a5ba78b
link/wasm: Put decls into the correct segments
Decls will now be put into their respective segment.
e.g. a constant decl will be inserted into the "rodata" segment,
whereas an uninitialized decl will be put in the "bss" segment instead.
2022-06-24 22:01:41 +02:00
Jakub Konka
b60938eefd macho: verbose print all input linker args verbatim 2022-06-24 22:01:29 +02:00
Jakub Konka
0df7ed79d3 macho: implement -search_dylibs_first linker option 2022-06-24 20:25:16 +02:00
Luuk de Gram
7c87f9c828 link:clarification & enable MachO getGlobalSymbol
This adds clarification to the getGlobalSymbol doc comments,
as well as renames the `addExternFn` function for MachO to `getGlobalSymbol`.
This function will now be called from 'src/link.zig' as well.

Finally, this also enables compiling zig's libc using LLVM even though
the `fno-LLVM` flag is given.
2022-06-24 08:12:17 +02:00
Luuk de Gram
6ae898b244 wasm: more f16 support and cleanup of intrinsics
`genFunctype` now accepts calling convention, param types, and return type
as part of its function signature rather than `fnData`. This means
we no longer have to create a dummy for our intrinsic call abstraction.
This also adds support for f16 division and builtins such as `@ceil` & more.
2022-06-24 08:12:17 +02:00
Luuk de Gram
a6747d328c stage2: Enable compiler-rt when LLVM is existant
Rather than checking if the user wants to use LLVM for the current compilation,
check for the existance of LLVM as part of the compiler. This is temporarily,
until other backends gain the ability to compiler LLVM themselves.
This means that when a user passed `-fno-LLVM` we will use the native
backend for the user's code, but use LLVM for compiler-rt.

This also fixes emitting names for symbols in the Wasm linker,
by deduplicating symbol names when multiple symbols point the same object.
2022-06-24 08:12:17 +02:00
Luuk de Gram
241180216f wasm-linker: Parse object file from the archive
Rather than finding the original object file, we seekTo to the
object file's position within the archive file, and from there open
a new file handle. This file handle is passed to the `Object` parser
which will create the object.
2022-06-24 08:12:17 +02:00
Luuk de Gram
c9f929a18b fix memory leaks 2022-06-24 08:12:17 +02:00