This is theoretically a bugfix as well, since it enforces the correct
limit on the first write after writing the header. This theoretical bug
hasn't been hit in practice though as far as I know.
* std.Io.Reader: fix confused semantics of rebase. Before it was
ambiguous whether it was supposed to be based on end or seek. Now it
is clearly based on seek, with an added assertion for clarity.
* std.crypto.tls.Client: fix panic due to not enough buffer size
available. Also, avoid unnecessary rebasing.
* std.http.Reader: introduce max_head_len to limit HTTP header length.
This prevents crash in underlying reader which may require a minimum
buffer length.
* std.http.Client: choose better buffer sizes for streams and TLS
client. Crucially, the buffer shared by HTTP reader and TLS client
needs to be big enough for all http headers *and* the max TLS record
size. Bump HTTP header size default from 4K to 8K.
fixes#24872
I have noticed however that there are still fetch problems
`limit` in chunkedSendFile applies only to the file, not the entire
chunk. `limit` in sendFileHeader does not include the header.
Additionally adds a comment to clarify what `limit` applies to in
sendFileHeader and fixed a small bug in it (`drain` is able to return
less then `header.len`).
It's a bit counter-intuitive, but there are two streams here: the
implementation here, and the connected output stream.
When we say "unflushed" we mean don't flush the connected output stream
because that's managed externally. But an "end" operation should always
flush the implementation stream.
added adapter to AnyWriter and GenericWriter to help bridge the gap
between old and new API
make std.testing.expectFmt work at compile-time
std.fmt no longer has a dependency on std.unicode. Formatted printing
was never properly unicode-aware. Now it no longer pretends to be.
Breakage/deprecations:
* std.fs.File.reader -> std.fs.File.deprecatedReader
* std.fs.File.writer -> std.fs.File.deprecatedWriter
* std.io.GenericReader -> std.io.Reader
* std.io.GenericWriter -> std.io.Writer
* std.io.AnyReader -> std.io.Reader
* std.io.AnyWriter -> std.io.Writer
* std.fmt.format -> std.fmt.deprecatedFormat
* std.fmt.fmtSliceEscapeLower -> std.ascii.hexEscape
* std.fmt.fmtSliceEscapeUpper -> std.ascii.hexEscape
* std.fmt.fmtSliceHexLower -> {x}
* std.fmt.fmtSliceHexUpper -> {X}
* std.fmt.fmtIntSizeDec -> {B}
* std.fmt.fmtIntSizeBin -> {Bi}
* std.fmt.fmtDuration -> {D}
* std.fmt.fmtDurationSigned -> {D}
* {} -> {f} when there is a format method
* format method signature
- anytype -> *std.io.Writer
- inferred error set -> error{WriteFailed}
- options -> (deleted)
* std.fmt.Formatted
- now takes context type explicitly
- no fmt string
Note that the `_ = Address` statements in tests previously were a nop,
and now actually check that the type is valid. However, on WASI, the
type is *not* valid.
* Uncouple std.http.ChunkParser from protocol.zig
* Fix receiveHead not passing leftover buffer through the header parser.
* Fix content-length read streaming
This implementation handles the final chunk length correctly rather than
"hoping" that the buffer already contains \r\n.
Mainly, this removes the poorly named `wait`, `send`, `finish`
functions, which all operated on the same "Response" object, which was
actually being used as the request.
Now, it looks like this:
1. std.net.Server.accept() gives you a std.net.Server.Connection
2. std.http.Server.init() with the connection
3. Server.receiveHead() gives you a Request
4. Request.reader() gives you a body reader
5. Request.respond() is a one-shot, or Request.respondStreaming() creates
a Response
6. Response.writer() gives you a body writer
7. Response.end() finishes the response; Response.endChunked() allows
passing response trailers.
In other words, the type system now guides the API user down the correct
path.
receiveHead allows extra bytes to be read into the read buffer, and then
will reuse those bytes for the body or the next request upon connection
reuse.
respond(), the one-shot function, will send the entire response in one
syscall.
Streaming response bodies no longer wastefully wraps every call to write
with a chunk header and trailer; instead it only sends the HTTP chunk
wrapper when flushing. This means the user can still control when it
happens but it also does not add unnecessary chunks.
Empirically, in my example project that uses this API, the usage code is
significantly less noisy, it has less error handling while handling
errors more correctly, it's more obvious what is happening, and it is
syscall-optimal.
Additionally:
* Uncouple std.http.HeadParser from protocol.zig
* Delete std.Server.Connection; use std.net.Server.Connection instead.
- The API user supplies the read buffer when initializing the
http.Server, and it is used for the HTTP head as well as a buffer
for reading the body into.
* Replace and document the State enum. No longer is there both "start"
and "first".
I originally removed these in 402f967ed5339fa3d828b7fe1d57cdb5bf38dbf2.
I allowed them to be added back in #15299 because they were smuggled in
alongside a bug fix, however, I wasn't kidding when I said that I wanted
to take the design of std.http in a different direction than using this
data structure.
Instead, some headers are provided via explicit field names populated
while parsing the HTTP request/response, and some are provided via
new fields that support passing extra, arbitrary headers.
This resulted in simplification of logic in many places, as well as
elimination of the possibility of failure in many places. There is
less deinitialization code happening now.
Furthermore, it made it no longer necessary to clone the headers data
structure in order to handle redirects.
http_proxy and https_proxy fields are now pointers since it is common
for them to be unpopulated.
loadDefaultProxies is changed into initDefaultProxies to communicate
that it does not actually load anything from disk or from the network.
The function now is leaky; the API user must pass an already
instantiated arena allocator. Removes the need to deinitialize proxies.
Before, proxies stored arbitrary sets of headers. Now they only store
the authorization value.
Removed the duplicated code between https_proxy and http_proxy. Finally,
parsing failures of the environment variables result in errors being
emitted rather than silently ignoring the proxy.
error.CompressionNotSupported is renamed to
error.CompressionUnsupported, matching the naming convention from all
the other errors in the same set.
Removed documentation comments that were redundant with field and type
names.
Disabling zstd decompression in the server for now; see #18937.
I found some apparently dead code in src/Package/Fetch/git.zig. I want
to check with Ian about this.
I discovered that test/standalone/http.zig is dead code, it is only
being compiled but not being run. Furthermore it hangs at the end if you
run it manually. The previous commits in this branch were written under
the assumption that this test was being run with
`zig build test-standalone`.
* Add missing period in Stack's description
This looks fine in the source, but looks bad when seen on the documentation website.
* Correct documentation for attachSegfaultHandler()
The description for attachSegfaultHandler() looks pretty bad without indicating that the stuff at the end is code
* Added missing 'the's in Queue.put's documentation
* Fixed several errors in Stack's documentation
`push()` and `pop()` were not styled as code
There was no period after `pop()`, which looks bad on the documentation.
* Fix multiple problems in base64.zig
Both "invalid"s in Base64.decoder were not capitalized.
Missing period in documentation of Base64DecoderWithIgnore.calcSizeUpperBound.
* Fix capitalization typos in bit_set.zig
In DynamicBitSetUnmanaged.deinit's and DynamicBitSet.deinit's documentation, "deinitializes" was uncapitalized.
* Fix typos in fifo.zig's documentation
Added a previously missing period to the end of the first line of LinearFifo.writableSlice's documentation.
Added missing periods to both lines of LinearFifo.pump's documentation.
* Fix typos in fmt.bufPrint's documentation
The starts of both lines were not capitalized.
* Fix minor documentation problems in fs/file.zig
Missing periods in documentation for Permissions.setReadOnly, PermissionsWindows.setReadOnly, MetadataUnix.created, MetadataLinux.created, and MetadataWindows.created.
* Fix a glaring typo in enums.zig
* Correct errors in fs.zig
* Fixed documentation problems in hash_map.zig
The added empty line in verify_context's documentation is needed, otherwise autodoc for some reason assumes that the list hasn't been terminated and continues reading off the rest of the documentation as if it were part of the second list item.
* Added lines between consecutive URLs in http.zig
Makes the documentation conform closer to what was intended.
* Fix wrongfully ended sentence in Uri.zig
* Handle wrongly entered comma in valgrind.zig.
* Add missing periods in wasm.zig's documentation
* Fix odd spacing in event/loop.zig
* Add missing period in http/Headers.zig
* Added missing period in io/limited_reader.zig
This isn't in the documentation due to what I guess is a limitation of autodoc, but it's clearly supposed to be. If it was, it would look pretty bad.
* Correct documentation in math/big/int.zig
* Correct formatting in math/big/rational.zig
* Create an actual link to ZIGNOR's paper.
* Fixed grammatical issues in sort/block.zig
This will not show up in the documentation currently.
* Fix typo in hash_map.zig
Some servers will respond with the identity encoding, meaning no
encoding, especially when responding to range-get requests. Adding the
identity encoding stops the header parser from failing when it
encounters this.
The majority of these are in comments, some in doc comments which might
affect the generated documentation, and a few in parameter names -
nothing that should be breaking, however.
* extract http protocol into protocol.zig, as it is shared between client and server
* coalesce Request and Response back into Client.zig, they don't contain
any large chunks of code anymore
* http.Server is implemented as basic as possible, a simple example below:
```zig
fn handler(res: *Server.Response) !void {
while (true) {
defer res.reset();
try res.waitForCompleteHead();
res.headers.transfer_encoding = .{ .content_length = 14 };
res.headers.connection = res.request.headers.connection;
try res.sendResponseHead();
_ = try res.write("Hello, World!\n");
if (res.connection.closing) break;
}
}
pub fn main() !void {
var server = Server.init(std.heap.page_allocator, .{ .reuse_address = true });
defer server.deinit();
try server.listen(try net.Address.parseIp("127.0.0.1", 8080));
while (true) {
const res = try server.accept(.{ .dynamic = 8192 });
const thread = try std.Thread.spawn(.{}, handler, .{res});
thread.detach();
}
}
```
closes#14204
In order to add tests for this I need to implement an HTTP server in the
standard library (#910) so that's probably the next thing I'll do.
RFC 9110 section 15:
Values outside the range 100..599 are invalid. Implementations often use
three-digit integer values outside of that range (i.e., 600..999) for
internal communication of non-HTTP status (e.g., library errors). A
client that receives a response with an invalid status code SHOULD
process the response as if it had a 5xx (Server Error) status code.
* std.http.Status.Class: add a "nonstandard" enum tag. Instead of
having `class` return an optional value, it can potentially return
nonstandard.
* extract out std.http.Client.Connection from std.http.Client.Request
- this code abstracts over plain/TLS only
- this is the type that will potentially be stored in a client's LRU
connection map
* introduce two-staged HTTP header parsing
- API users can rely on a heap-allocated buffer with a maximum limit,
which defaults to 16 KB, or they can provide a static buffer that
is borrowed by the Request instance.
- The entire HTTP header is buffered because there are strings in
there and they must be accessed later, such as with the case of
HTTP redirects.
- When buffering the HTTP header, the parser only looks for the
\r\n\r\n pattern. Further validation is done later.
- After the full HTTP header is buffered, it is parsed into
components such as Content-Length and Location.
* HTTP redirects are handled, with a maximum redirect count option that
defaults to 3.
- Connection: close is always used for now; implementing keep-alive
connections and an LRU connection pool in std.http.Client is a task
for another day.
see #2007
This is a streaming HTTP header parser. All it currently does is detect
the end of headers. This will be a non-allocating parser where one can
bring supply their own buffer if they want to handle custom headers.
This commit also improves std.http.Client to not return the HTTP headers
with the read functions.
