7997 Commits

Author SHA1 Message Date
Igor Anić
257b8131ec io_uring: use kernel version test in send_zc test 2023-11-17 17:21:54 +01:00
Igor Anić
87dd8d5669 io_uring: simplify tests by checking kernel version
So far we relied on getting EINVAL in CQE for operations that kernel don't
support. The problem with that approach is that there are many other reasons
(like wrong params) to get EINVAL. The other problem is when we have an
operation that existed before and gets new behavior via different attributes,
like accept and accept_direct. Older kernels can fall back to non direct
operation although we set attributes for direct operation. Operation completes
successfully in both cases but with different results.

This commit introduces kernel version check at the start of the test. Making
body of the test free of checking for various kernel version differences.

Feature availability references:
  * https://manpages.debian.org/unstable/liburing-dev/io_uring_enter.2.en.html
  * https://kernel.dk/axboe-kr2022.pdf
  * 5acf7969bc/lib/std/os/linux.zig (L3727)
  * 5acf7969bc/lib/std/os/linux.zig (L3993)
2023-11-17 16:31:57 +01:00
Igor Anić
256384a2ec io_uring: add direct operations
Add operation on direct file descriptors. Also referred to as fixed or
registered files.

References:
https://kernel.dk/axboe-kr2022.pdf
https://lwn.net/Articles/863071/

Added functions:

IO_Uring
  accept_direct
  accept_multishot_direct
  openat_direct
  close_direct
  socket
  socket_direct
  socket_direct_alloc

Raw prepare operations:
  io_uring_prep_accept_direct
  io_uring_prep_multishot_accept_direct
  io_uring_prep_openat_direct
  io_uring_prep_close_direct
  io_uring_prep_socket
  io_uring_prep_socket_direct
  io_uring_prep_socket_direct_alloc

Tested on this kernels:

5.4.0-164-generic
2559 passed; 70 skipped; 0 failed.

5.8.0-63-generic
2573 passed; 56 skipped; 0 failed.

5.11.0-49-generic
2576 passed; 53 skipped; 0 failed.

5.13.0-52-generic
2576 passed; 53 skipped; 0 failed.

5.15.0-87-generic
2579 passed; 50 skipped; 0 failed.

5.19.0-46-geneic
2584 passed; 45 skipped; 0 failed.

6.2.0-35-generic.log
2585 passed; 44 skipped; 0 failed.

6.5.0-9-generic
2585 passed; 44 skipped; 0 failed.
2023-11-17 13:48:17 +01:00
Igor Anić
087ee497d4 io_uring: add zero-copy send operation
`send_zc` tries to avoid making intermediate copies of data. Zerocopy execution
is not guaranteed and may fall back to copying.

The flags field of the first struct io_uring_cqe may likely contain
IORING_CQE_F_MORE , which means that there will be a second completion event /
notification for the request, with the user_data field set to the same value.
The user must not modify the data buffer until the notification is posted. The
first cqe follows the usual rules and so its res field will contain the number
of bytes sent or a negative error code. The notification's res field will be set
to zero and the flags field will contain IORING_CQE_F_NOTIF. The two step model
is needed because the kernel may hold on to buffers for a long time, e.g.
waiting for a TCP ACK, and having a separate cqe for request completions allows
userspace to push more data without extra delays. Note, notifications are only
responsible for controlling the lifetime of the buffers, and as such don't mean
anything about whether the data has atually been sent out or received by the
other end. Even errored requests may generate a notification, and the user must
check for IORING_CQE_F_MORE rather than relying on the result.

Available since kernel 6.0.

References:
https://man7.org/linux/man-pages/man3/io_uring_prep_send_zc.3.html
https://man7.org/linux/man-pages/man2/io_uring_enter.2.html
2023-11-10 16:44:18 +01:00
Bogdan Romanyuk
ee8da40769 replace deprecated std.math.absCast 2023-11-09 17:39:27 +02:00
Linus Groh
e72049bc61 std.math: Add isPositiveZero() and isNegativeZero() 2023-11-09 00:55:43 -07:00
Andrew Kelley
b2ed2c4d4f
Merge pull request #17888 from AdamGoertz/zig-reduce
zig-reduce: Add reductions for `if` and `while`
2023-11-09 00:52:38 -07:00
frmdstryr
f258a391da
Speed up ast.tokenLocation 2023-11-09 01:45:25 +00:00
Veikka Tuominen
10a28bc4c4
Merge pull request #17935 from ianic/fix_io_uring_test_kernel_5_4
Fix io_uring tests on kernel 5.4
2023-11-09 00:49:22 +02:00
Igor Anić
715e5f757f fix io_uring tests on kernel 5.4
Unsupported tests are now skipped on kernel 5.4:
```
uname -a
Linux d20 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:31:37 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
zig test lib/std/std.zig --zig-lib-dir lib --main-mod-path lib/std 2>&1 | cat
709/2616 test.nop... OK
710/2616 test.readv... OK
711/2616 test.writev/fsync/readv... OK
712/2616 test.write/read... SKIP
713/2616 test.splice/read... SKIP
714/2616 test.write_fixed/read_fixed... OK
715/2616 test.openat... SKIP
716/2616 test.close... SKIP
717/2616 test.accept/connect/send/recv... SKIP
718/2616 test.sendmsg/recvmsg... OK
719/2616 test.timeout (after a relative time)... SKIP
720/2616 test.timeout (after a number of completions)... OK
721/2616 test.timeout_remove... SKIP
722/2616 test.accept/connect/recv/link_timeout... SKIP
723/2616 test.fallocate... SKIP
724/2616 test.statx... SKIP
725/2616 test.accept/connect/recv/cancel... SKIP
726/2616 test.register_files_update... SKIP
727/2616 test.shutdown... SKIP
728/2616 test.renameat... SKIP
729/2616 test.unlinkat... SKIP
730/2616 test.mkdirat... SKIP
731/2616 test.symlinkat... SKIP
732/2616 test.linkat... SKIP
733/2616 test.provide_buffers: read... SKIP
734/2616 test.remove_buffers... SKIP
735/2616 test.provide_buffers: accept/connect/send/recv... SKIP
736/2616 test.accept multishot... SKIP
```
2023-11-08 18:52:03 +01:00
Igor Anić
7a1dbbd4f9 fix io_uring timeout_remove test on kernel 5.4
There is no grantee that `copy_cqes` will return exactly wait_nr number of cqes.
If there are ready cqes it can return > 0 but < wait_nr number of cqes.
2023-11-08 18:25:29 +01:00
Frank Denis
a70d8d29d5
Curve25519.fromEdwards25519(): don't assume normalized coordinates (#17920)
The low-level `Curve25519.fromEdwards25519()` function assumed
that the X/Y coordinates were not scaled (Z=1).

But this is not guaranteed to be the case.

In most real-world applications, the coordinates are freshly decoded,
either directly or via the `X25519.fromEd25519()` function, so this
is not an issue.

However, since we offer the ability to do that conversion after
arbitrary computations, the assertion was not correct.
2023-11-08 11:56:56 +01:00
tison
ee47643b6e
std.math.big: fix sqrt with bits > limb_bits
Signed-off-by: tison <wander4096@gmail.com>
2023-11-08 08:39:08 +02:00
David Rubin
03abac3824 fix sqrt(0) 2023-11-07 14:37:43 +02:00
Jakub Konka
bf0387b6bb
Merge pull request #17873 from ziglang/elf-archive
elf: implement archiving input object files
2023-11-07 03:22:14 +01:00
Jakub Konka
261db02018 CheckObject: support parsing and dumping archive symtab for ELF 2023-11-06 21:18:26 +01:00
Adam Goertz
91570cc42d zig-reduce: Reduce if expressions
Perform these transformations in this priority order:
1. If the `else` expression is missing or an empty block, replace the condition with `if (true)` if it is not already.
2. If the `then` block is empty, replace the condition with `if (false)` if it is not already.
3. If the condition is `if (true)`, replace the `if` expression with the contents of the `then` expression.
4. If the condition is `if (false)`, replace the `if` expression with the contents of the `else` expression.
2023-11-06 04:31:55 +00:00
Andrew Kelley
702b809ea3
Merge pull request #17815 from Luukdegram/wasm-no-entry
wasm-linker: implement `-fno-entry` and correctly pass `--shared` and `--pie` when given
2023-11-05 18:44:12 -05:00
Igor Anić
bec36aa7c0 io_uring: add multishot accept operation
Server networking application typically accept multiple connections. Multishot
accept simplifies handling these situations. Applications submits once and
receives CQE whenever a new connection request comes in.

Multishot is active until it is canceled or experience error. While active, and
further notification are expected CQE completion will have IORING_CQE_F_MORE set
in the flags. If this flag isn't set, the application must re-arm this request
by submitting a new one.

Reference: [io_uring and networking in 2023](https://github.com/axboe/liburing/wiki/io_uring-and-networking-in-2023#multi-shot)
2023-11-06 00:12:39 +02:00
Andrew Kelley
c4dddcbadb std.zig.render: ability to omit variable declarations 2023-11-04 20:26:39 -07:00
Andrew Kelley
88acdb9aa6 zig reduce: delete statements from blocks 2023-11-04 18:06:11 -07:00
Andrew Kelley
31ad3af956 std.zig.render: support fixing unused parameter 2023-11-04 17:37:34 -07:00
Andrew Kelley
fc1e7a5644 zig reduce: rename identifiers when inlining an @import 2023-11-04 15:48:28 -07:00
Andrew Kelley
802c82b072 std.ArrayHashMap: add init function
for when you have keys and values array
2023-11-04 15:46:30 -07:00
Andrew Kelley
8bd01d2d9b zig reduce: add transformation for inlining file-based @import
One thing is missing for it to be useful, however, which is dealing with
ambiguous reference errors introduced by the inlining process.
2023-11-04 13:57:29 -07:00
Andrew Kelley
98dc28bbe2
Merge pull request #17852 from ziglang/zig-reduce
introduce `zig reduce` subcommand
2023-11-04 14:25:50 -04:00
Jacob Young
095c4294aa x86_64: fix miscompilations
Closes #17618
2023-11-04 09:15:05 -04:00
Andrew Kelley
31529f912b zig reduce: add transformation of replacing var init with undefined 2023-11-03 22:08:01 -07:00
Andrew Kelley
a2f4adbd19 zig reduce: add "delete unused globals" transform
While walking the AST looking for reductions, notice if any globals are
unreferenced, and if they are, add a transformation for removing the
global.
2023-11-03 21:31:32 -07:00
Jacob Young
509be7cf1f x86_64: fix std test failures 2023-11-03 23:18:21 -04:00
Andrew Kelley
9a1094c00f fix compilation regression 2023-11-03 20:05:32 -07:00
Andrew Kelley
1396479656 zig reduce: redesign
Now it works like this:
1. Walk the AST of the source file looking for independent
   reductions and collecting them all into an array list.
2. Randomize the list of transformations. A future enhancement will add
   priority weights to the sorting but for now they are completely
   shuffled.
3. Apply a subset consisting of 1/2 of the transformations and check for
   interestingness.
4. If not interesting, half the subset size again and check again.
5. Repeat until the subset size is 1, then march the transformation
   index forward by 1 with each non-interesting attempt.

At any point if a subset of transformations succeeds in producing an interesting
result, restart the whole process, reparsing the AST and re-generating the list
of all possible transformations and shuffling it again.

As for std.zig.render, the fixups operate based on AST Node Index rather
than Nth index of the function occurence. This allows precise control
over how to mutate the input.
2023-11-03 20:05:32 -07:00
Andrew Kelley
09d13104be std.BitSet: add setAll and unsetAll methods 2023-11-03 20:05:32 -07:00
Andrew Kelley
b1aaf42134 std.zig.render: fix iteration over parameters
use the handy iteration API to be correct
2023-11-03 20:05:32 -07:00
Andrew Kelley
5f1f145199 add a transformation for gutting the body of a function 2023-11-03 20:05:32 -07:00
Andrew Kelley
3263d6e6ab std.zig: move render state to struct; add fixups 2023-11-03 20:05:32 -07:00
dweiller
f6de3ec963 zstandard: fix incorrect RLE decompression into ring buffer
This reverts a change introduced in #17400 causing a bug when
decompressing an RLE block into a ring buffer.

RLE blocks contain only a single byte of data to copy into the output,
so attempting to copy a slice causes buffer overruns and incorrect
decompression.
2023-11-03 23:03:43 -04:00
XXIV
b197c4d79b
std.Atomic: remove unnecessary @ptrCast (#17825) 2023-11-03 14:55:02 -04:00
xdBronch
5bd27a2cb6 dont assume apple chips are macos exclusive 2023-11-03 14:46:53 -04:00
Ryan Liptak
ae6df9e967 start.zig: Replace kernel32 usage with ntdll
Co-authored-by: e4m2 <git@e4m2.com>
2023-11-03 19:25:46 +02:00
Luuk de Gram
c893f83715
cli: consolidate entry point flags 2023-11-03 12:48:53 +01:00
Luuk de Gram
938f9dea37
update linker tests
This updates all linker tests to include `no_entry` as well as changes
all tests to executable so they do not need to be updated later when
the in-house WebAssembly linker supports dynamic libraries.
2023-11-03 12:48:53 +01:00
Luuk de Gram
1cb7a01b25
wasm-linker: implement -fno-entry flag
This adds support for the `-fno-entry` and `-fentry` flags respectively, for
zig build-{exe/lib} and the build system. For `zig cc` we use the `--no-entry`
flag to be compatible with clang and existing tooling.

In `start.zig` we now make the main function optional when the target is
WebAssembly, as to allow for the build-exe command in combination with
`-fno-entry`.

When the execution model is set, and is set to 'reactor', we now verify
when an entry name is given it matches what is expected. When no entry
point is given, we set it to `_initialize` by default. This means the user
will also be met with an error when they use the reactor model, but did
not provide the correct function.
2023-11-03 12:48:52 +01:00
Ryan Liptak
42d4d07efa start: Make wWinMain nCmdShow setting match Windows better
It won't match perfectly; see https://github.com/ziglang/zig/issues/17808
2023-11-02 00:19:40 -04:00
XXIV
c2cf47ab28
std.os.windows: remove unnecessary @ptrCast (#17816) 2023-11-01 18:45:51 -04:00
Kai Jellinghaus
7ee7c90274 Fix regressing #17290 from #17734 2023-11-01 18:30:56 +02:00
Jordyfel
61861ef395 std.http: account for renames in docs 2023-11-01 14:46:00 +02:00
Kai Jellinghaus
084a7cf028
Use ArenaAllocator.reset in MemoryPool 2023-11-01 14:45:35 +02:00
Andrew Kelley
3fc6fc6812 std.builtin.Endian: make the tags lower case
Let's take this breaking change opportunity to fix the style of this
enum.
2023-10-31 21:37:35 -04:00
Jacob Young
149200aac5 mem: delete readIntSlice and writeIntSlice
After deleting all potentially incorrect usages, there were no more
remaining.
2023-10-31 21:37:35 -04:00