8532 Commits

Author SHA1 Message Date
Michael Dusan
70fbafacf2 std: fix macos x86_64 to link stat$INODE64
- restricted similar $INODE64 symbols to macos x86_64 only
- minor cleanup and updated comments

closes #11386
2024-02-24 13:09:03 -08:00
Andrew Kelley
6fddc9cd3d
Merge pull request #19064 from ziglang/fix-netname-deleted
std: map NETNAME_DELETED to error.ConnectionResetByPeer
2024-02-24 10:03:40 -08:00
Andrew Kelley
9812bc7b10 std: map NETNAME_DELETED to error.ConnectionResetByPeer
This was observed in writing to a network stream that was closed on the
read end.
2024-02-24 04:16:53 -07:00
Andrew Kelley
f9bf4d1d60 std.http tests: wait for server response
This avoids a race that can make the unit test fail
2024-02-24 04:16:21 -07:00
Jae B
88b3c14426 fix compilation issues
ie.
C:\zig\current\lib\std\debug.zig:726:23: error: no field or member function named 'getDwarfInfoForAddress' in 'dwarf.DwarfInfo'
        if (try module.getDwarfInfoForAddress(unwind_state.debug_info.allocator, unwind_state.dwarf_context.pc)) |di| {
                ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
C:\zig\current\lib\std\dwarf.zig:663:23: note: struct declared here
pub const DwarfInfo = struct {
                      ^~~~~~
referenced by:
    next_internal: C:\zig\current\lib\std\debug.zig:737:29
    next: C:\zig\current\lib\std\debug.zig:654:31
    remaining reference traces hidden; use '-freference-trace' to see all reference traces
C:\zig\current\lib\std\debug.zig:970:31: error: no field or member function named 'getSymbolAtAddress' in 'dwarf.DwarfInfo'
    const symbol_info = module.getSymbolAtAddress(debug_info.allocator, address) catch |err| switch (err) {
                        ~~~~~~^~~~~~~~~~~~~~~~~~~
C:\zig\current\lib\std\dwarf.zig:663:23: note: struct declared here
pub const DwarfInfo = struct {
2024-02-23 23:33:28 -08:00
Jae B
7f11c19450 use @trap instead of system.abort for emscripten 2024-02-23 22:23:51 -08:00
Brandon Botsch
5df6c70a32
std.hash_map: Fix integer overflow bugs (#19048) 2024-02-24 00:56:00 -05:00
Jae B
774487038f setup PATH_MAX for emscripten 2024-02-23 21:18:27 -08:00
Andrew Kelley
229b87cab6
Merge pull request #19053 from ianic/tar_fuzzing2
std:tar fix two crashes found by fuzzing
2024-02-23 21:11:27 -08:00
Igor Anić
0a86b117bf std.tar fix integer overflow in header size parse
Found by fuzzing. Fixing code and adding test.
2024-02-23 21:57:40 +01:00
Igor Anić
f67aa8b9b3 std.tar fix parsing mode field in tar header
Found by fuzzing. Previous numeric function assumed that is is getting
buffer of size 12, mode is size 8. Fuzzing found overflow.
Fixing and adding test cases.
2024-02-23 21:57:15 +01:00
Andrew Kelley
653d4158cd std.http.Server: expose arbitrary HTTP headers
Ultimate flexibility, just be sure to destroy the correct amount of
information when looking at them.
2024-02-23 02:58:02 -07:00
Andrew Kelley
5b34a1b718 std.http: disable the test that was never passing on windows
We didn't know it wasn't passing before because it wasn't actually being
run.
2024-02-23 02:37:11 -07:00
Andrew Kelley
d7ac8c8e65 wasi: don't try to test http
wasi does not support networking
2024-02-23 02:37:11 -07:00
Andrew Kelley
483b63d301 std.http: migrate remaining test/standalone/http.zig to std lib
These tests were not being actually run. Now they are run along with
testing the standard library.
2024-02-23 02:37:11 -07:00
Andrew Kelley
10beb19ce7 std.http: assert against \r\n in headers
The HTTP specification does not provide a way to escape \r\n in headers,
so it's the API user's responsibility to ensure the header names and
values do not contain \r\n. Also header names must not contain ':'.

It's an assertion, not an error, because the calling code very likely is
using hard-coded values or server-provided values that do not need to be
checked, and the error would be unreachable anyway.

Untrusted user input must not be put directly into into HTTP headers.
2024-02-23 02:37:11 -07:00
Andrew Kelley
d051b13963 std.http.Server: implement respondStreaming with unknown len
no content-length header
no transfer-encoding header
2024-02-23 02:37:11 -07:00
Andrew Kelley
737e7be46c std.http: refactor unit tests
avoid a little bit of boilerplate
2024-02-23 02:37:11 -07:00
Andrew Kelley
abde76a808 std.http.Server: handle expect: 100-continue requests
The API automatically handles these requests as expected. After
receiveHead(), the server has a chance to notice the expectation and do
something about it. If it does not, then the Server implementation will
handle it by sending the continuation header when the read stream is
created.

Both respond() and respondStreaming() send the continuation header as
part of discarding the request body, only if the read stream has not
already been created.
2024-02-23 02:37:11 -07:00
Andrew Kelley
380916c0f8 std.http.Server.Request.Respond: support all transfer encodings
Before I mistakenly thought that missing content-length meant zero when
it actually means to stream until the connection is closed.

Now the respond() function accepts transfer_encoding which can be left
as default (use content.len for content-length), set to none which makes
it omit the content-length, or chunked, which makes it format the
response as a chunked transfer even though the server has the entire
contents already buffered.

The echo-content tests are moved from test/standalone/http.zig to the
standard library where they are actually run.
2024-02-23 02:37:11 -07:00
Andrew Kelley
40ed3c4d24 std.http.Client: add keep_alive option to fetch 2024-02-23 02:37:11 -07:00
Andrew Kelley
c0d8ac83eb std.http.Server: fix handling of HEAD + chunked 2024-02-23 02:37:11 -07:00
Andrew Kelley
17291e072b std.ArrayList: fixedWriter
A writer that appends to the list, returning error.OutOfMemory rather
than attempting to increase capacity.
2024-02-23 02:37:11 -07:00
Andrew Kelley
2e7d8062ca std.http.Server: fix seeing phantom request 2024-02-23 02:37:11 -07:00
Andrew Kelley
b4b9f6aa4a std.http.Server: reimplement chunked uploading
* Uncouple std.http.ChunkParser from protocol.zig
* Fix receiveHead not passing leftover buffer through the header parser.
* Fix content-length read streaming

This implementation handles the final chunk length correctly rather than
"hoping" that the buffer already contains \r\n.
2024-02-23 02:37:11 -07:00
Andrew Kelley
a8958c99a9 std.net: fix std lib test regression. fixup 2024-02-23 02:37:11 -07:00
Andrew Kelley
d943ce5dc7 std.io.Reader: add discard function
Reads the stream until the end, ignoring all the data.
Returns the number of bytes discarded.
2024-02-23 02:37:11 -07:00
Andrew Kelley
12a9e0f415 std.net.listen: fix Windows API use
In a previous commit I removed a load-bearing use of `@hasDecl` to
detect whether the SO.REUSEPORT option should be set. `@hasDecl` should
not be used for OS feature detection because it can hide bugs.

The new logic checks for the operating system specifically and then does
the thing that is supposed to be done on that operating system directly.
2024-02-23 02:37:11 -07:00
Andrew Kelley
6395ba852a std.http.Server: rework the API entirely
Mainly, this removes the poorly named `wait`, `send`, `finish`
functions, which all operated on the same "Response" object, which was
actually being used as the request.

Now, it looks like this:
1. std.net.Server.accept() gives you a std.net.Server.Connection
2. std.http.Server.init() with the connection
3. Server.receiveHead() gives you a Request
4. Request.reader() gives you a body reader
5. Request.respond() is a one-shot, or Request.respondStreaming() creates
   a Response
6. Response.writer() gives you a body writer
7. Response.end() finishes the response; Response.endChunked() allows
   passing response trailers.

In other words, the type system now guides the API user down the correct
path.

receiveHead allows extra bytes to be read into the read buffer, and then
will reuse those bytes for the body or the next request upon connection
reuse.

respond(), the one-shot function, will send the entire response in one
syscall.

Streaming response bodies no longer wastefully wraps every call to write
with a chunk header and trailer; instead it only sends the HTTP chunk
wrapper when flushing. This means the user can still control when it
happens but it also does not add unnecessary chunks.

Empirically, in my example project that uses this API, the usage code is
significantly less noisy, it has less error handling while handling
errors more correctly, it's more obvious what is happening, and it is
syscall-optimal.

Additionally:
* Uncouple std.http.HeadParser from protocol.zig
* Delete std.Server.Connection; use std.net.Server.Connection instead.
  - The API user supplies the read buffer when initializing the
    http.Server, and it is used for the HTTP head as well as a buffer
    for reading the body into.
* Replace and document the State enum. No longer is there both "start"
  and "first".
2024-02-23 02:37:11 -07:00
Andrew Kelley
9129fb28dc std.ArrayList: add writerAssumeCapacity
Useful when you want to use an ArrayList to operate on a static buffer.
2024-02-23 02:37:11 -07:00
Andrew Kelley
2df3de1e20 std.http.Server: no more dynamic allocation
In particular remove OutOfMemory from the error set for parsing client
http headers.
2024-02-23 02:37:11 -07:00
Andrew Kelley
68d3e103b7 full send 2024-02-23 02:37:11 -07:00
Andrew Kelley
968d08af6d std.http.Server.Connection: remove dead code 2024-02-23 02:37:11 -07:00
Andrew Kelley
c7fc2d76ce std.http.Server: move closing bool
It does not belong in the Connection struct
2024-02-23 02:37:11 -07:00
Andrew Kelley
6129ecd4fe std.net, std.http: simplify 2024-02-23 02:37:11 -07:00
Andrew Kelley
f1565e3d09 std.http.Server.accept can no longer fail from OOM 2024-02-23 02:37:11 -07:00
Andrew Kelley
c44a902836 fix zstd compilation errors from previous commit 2024-02-23 02:37:11 -07:00
dweiller
5c12783094 std.compress.zstd: make DecompressStream options runtime 2024-02-23 02:37:11 -07:00
Andrew Kelley
63acc856c1 std.http.Client: remove invalid use of refAllDecls 2024-02-23 02:37:11 -07:00
dweiller
accbba3cd8 std.compress.zstd: disable failing wasm32 tests
This commit can be reverted after
https://github.com/ziglang/zig/pull/18971 is merged.
2024-02-23 02:37:11 -07:00
dweiller
ac1b957e79 std.compress.zstd: remove allocation from DecompressStream 2024-02-23 02:37:11 -07:00
dweiller
73f6d3afb5 std.compress.zstd: fix decompressStreamOptions 2024-02-23 02:37:11 -07:00
dweiller
63fa151f1c std.compress.zstandard: fix buffer sizes
This change corrects the size of various internal buffers used. The
previous behavior did not cause validity problems but wasted space.
2024-02-23 02:37:11 -07:00
Andrew Kelley
e204b2ca92 std.http.Client.connectUnix: handle unsupported OS at compile time 2024-02-23 02:37:11 -07:00
Andrew Kelley
6de8748b05 std.http: skip tests on wasi and single-threaded
WASI does not support networking, and these tests require threads.
2024-02-23 02:37:11 -07:00
Andrew Kelley
651aa5e8e4 std.http.Client: eliminate arena allocator usage
Before, this code constructed an arena allocator and then used it when
handling redirects.

You know what's better than having threads fight over an allocator?
Avoiding dynamic memory allocation in the first place.

This commit reuses the http headers static buffer for handling
redirects. The new location is copied to the beginning of the static
header buffer and then the subsequent request uses a subslice of that
buffer.
2024-02-23 02:37:11 -07:00
Andrew Kelley
107992d50e std.Uri: refactor std.mem.Allocator -> Allocator 2024-02-23 02:37:11 -07:00
Andrew Kelley
743a0c966d std.http.Client: remove bad decisions from fetch()
* "storage" is a better name than "strategy".
* The most flexible memory-based storage API is appending to an
  ArrayList.
* HTTP method should default to POST if there is a payload.
* Avoid storing unnecessary data in the FetchResult
* Avoid the need for a deinit() method in the FetchResult

The decisions that this logic made about how to handle files is beyond
repair:
- fail to use sendfile() on a plain connection
- redundant stat
- does not handle arbitrary streams
So, file-based response storage is no longer supported. Users should use
the lower-level open() API which allows avoiding these pitfalls.
2024-02-23 02:37:11 -07:00
Andrew Kelley
0ddcb83418 std.http.Client.fetch: remove inappropriate seek
no
2024-02-23 02:37:11 -07:00
Andrew Kelley
7036644ed2 std.http.Client: remove advisory file lock on fetch
This is not an appropriate place to put this code. It belongs in the
caller's code, if at all.
2024-02-23 02:37:11 -07:00