Now it's based on calling fillMore rather than an illegal aliased stream
into the Reader buffer.
This commit also includes a disambiguation block inspired by #25162. If
`StreamTooLong` was added to `RebaseError` then this logic could be
replaced by removing the exit condition from the while loop. That error
code would represent when `buffer` capacity is too small for an
operation, replacing the current use of asserts.
Fix `takeDelimiter` and `takeDelimiterExclusive` tossing too many bytes
(#25132)
Also add/improve test coverage for all delimiter and sentinel methods,
update usages of `takeDelimiterExclusive` to not rely on the fixed bug,
tweak a handful of doc comments, and slightly simplify some logic.
I have not fixed#24950 in this commit because I am a little less
certain about the appropriate solution there.
Resolves: #25132
Co-authored-by: Andrew Kelley <andrew@ziglang.org>
Previously, the logic in peekDelimiterInclusive (when the delimiter was not found in the existing buffer) used the `n` returned from `r.vtable.stream` as the length of the slice to check, but it's valid for `vtable.stream` implementations to return 0 if they wrote to the buffer instead of `w`. In that scenario, the `indexOfScalarPos` would be given a 0-length slice so it would never be able to find the delimiter.
This commit changes the logic to assume that `r.vtable.stream` can both:
- return 0, and
- modify seek/end (i.e. it's also valid for a `vtable.stream` implementation to rebase)
Also introduces `std.testing.ReaderIndirect` which helps in being able to test against Reader implementations that return 0 from `stream`/`readVec`
Fixes#25428
This reverts commit 27aba2d776caf59bb6569934626af587fdba9c75.
I'd like to review this contribution more carefully, particularly with
the alternate implementation that is also open as a pull request
(#25109).
Reopens#25093
If `r.end` is updated in the `stream` implementation, then it's possible that `r.end += ...` will behave unexpectedly. What seems to happen is that it reverts back to its value before the function call and then the increment happens. Here's a reproduction:
```zig
test "fill when stream modifies `end` and returns 0" {
var buf: [3]u8 = undefined;
var zero_reader = infiniteZeroes(&buf);
_ = try zero_reader.fill(1);
try std.testing.expectEqual(buf.len, zero_reader.end);
}
pub fn infiniteZeroes(buf: []u8) std.Io.Reader {
return .{
.vtable = &.{
.stream = stream,
},
.buffer = buf,
.end = 0,
.seek = 0,
};
}
fn stream(r: *std.Io.Reader, _: *std.Io.Writer, _: std.Io.Limit) std.Io.Reader.StreamError!usize {
@memset(r.buffer[r.seek..], 0);
r.end = r.buffer.len;
return 0;
}
```
When `fill` is called, it will call into `vtable.readVec` which in this case is `defaultReadVec`. In `defaultReadVec`:
- Before the `r.end += r.vtable.stream` line, `r.end` will be 0
- In `r.vtable.stream`, `r.end` is modified to 3 and it returns 0
- After the `r.end += r.vtable.stream` line, `r.end` will be 0 instead of the expected 3
Separating the `r.end += stream();` into two lines fixes the problem (and this separation is done elsewhere in `Reader` so it seems possible that this class of bug has been encountered before).
Potentially related issues:
- https://github.com/ziglang/zig/issues/4021
- https://github.com/ziglang/zig/issues/12064
I measured this against master branch and found no statistical
difference. Since this code is simpler and logically superior due to
always leaving sufficient unused capacity when growing, it is preferred
over status quo.
This reverts commit ac42eaaadd0650ffc281f9a1ed1a642fde8984b7, reversing
changes made to 9fa2394f8c00d060931d69fb6f342f7f2e3d826e.
I would like a chance to review this, please. I already spotted some
issues.
Before this commit, calling appendRemaining with an ArrayList where list.items.len != list.capacity could result in illegal behavior if the Writer.Allocating resized the list during the appendRemaining call.
Fixes#25057
* extend std.Io.Reader.peekDelimiterExclusive test to repeat successful end-of-stream path (fails)
* fix std.Io.Reader.peekDelimiterExclusive to not advance seek position in successful end-of-stream path
Writer.sendFileAll() asserts non-zero buffer capacity in the case that
the fallback is hit. It also requires the caller to flush. The buffer
may be bypassed as an optimization but this is not a guarantee.
Also improve the Writer documentation and add an earlier assert on
buffer capacity in sendFileAll().
* std.Io.Reader: fix confused semantics of rebase. Before it was
ambiguous whether it was supposed to be based on end or seek. Now it
is clearly based on seek, with an added assertion for clarity.
* std.crypto.tls.Client: fix panic due to not enough buffer size
available. Also, avoid unnecessary rebasing.
* std.http.Reader: introduce max_head_len to limit HTTP header length.
This prevents crash in underlying reader which may require a minimum
buffer length.
* std.http.Client: choose better buffer sizes for streams and TLS
client. Crucially, the buffer shared by HTTP reader and TLS client
needs to be big enough for all http headers *and* the max TLS record
size. Bump HTTP header size default from 4K to 8K.
fixes#24872
I have noticed however that there are still fetch problems
* std.Io.Reader: appendRemaining no longer supports alignment and has
different rules about how exceeding limit. Fixed bug where it would
return success instead of error.StreamTooLong like it was supposed to.
* std.Io.Reader: simplify appendRemaining and appendRemainingUnlimited
to be implemented based on std.Io.Writer.Allocating
* std.Io.Writer: introduce unreachableRebase
* std.Io.Writer: remove minimum_unused_capacity from Allocating. maybe
that flexibility could have been handy, but let's see if anyone
actually needs it. The field is redundant with the superlinear growth
of ArrayList capacity.
* std.Io.Writer: growingRebase also ensures total capacity on the
preserve parameter, making it no longer necessary to do
ensureTotalCapacity at the usage site of decompression streams.
* std.compress.flate.Decompress: fix rebase not taking into account seek
* std.compress.zstd.Decompress: split into "direct" and "indirect" usage
patterns depending on whether a buffer is provided to init, matching
how flate works. Remove some overzealous asserts that prevented buffer
expansion from within rebase implementation.
* std.zig: fix readSourceFileToAlloc returning an overaligned slice
which was difficult to free correctly.
fixes#24608
`limit` in chunkedSendFile applies only to the file, not the entire
chunk. `limit` in sendFileHeader does not include the header.
Additionally adds a comment to clarify what `limit` applies to in
sendFileHeader and fixed a small bug in it (`drain` is able to return
less then `header.len`).
This "get" is useless noise and was copied from FixedBufferWriter.
Since this API has not yet landed in a release, now is a good time
to make the breaking change to fix this.
This is one way of partially addressing https://github.com/ziglang/zig/issues/24767
- These functions are unused
- These functions are untested
- These functions are broken
+ The same dangling pointer bug from 6219c015d8e8c958d96e5caa5ef0dbab9c414996 exists in `writePreserve`
+ The order of the bytes preserved in relation to the `bytes` being written can differ depending on unused buffer capacity at the time of the call and the drain implementation.
If there ends up being a need for these functions, they can be fixed and added back.
