117 Commits

Author SHA1 Message Date
Andrew Kelley
a72292513e add std.Thread.Pool.spawnWg
This function accepts a WaitGroup parameter and manages the reference
counting therein. It also is infallible.

The existing `spawn` function is still handy when the job wants to
further schedule more tasks.
2024-05-03 20:58:02 -07:00
Jonathan Marler
a96b78c170 add std.zip and support zip files in build.zig.zon
fixes #17408

Helpful reviewers/testers include Joshe Wolfe, Auguste Rame, Andrew
Kelley and Jacob Young.

Co-authored-by: Joel Gustafson <joelg@mit.edu>
2024-05-03 16:58:53 -04:00
Igor Anić
9cfac4718d fetch: combine unpack error validation in a single function
Follow up of: #19500
[discussion](https://github.com/ziglang/zig/pull/19500#discussion_r1558238138)
2024-04-11 15:44:40 -07:00
Jacob Young
c4587dc9f4 Uri: propagate per-component encoding
This allows `std.Uri.resolve_inplace` to properly preserve the fact
that `new` is already escaped but `base` may not be.  I originally tried
just moving `raw_uri` around, but it made uri resolution unmanagably
complicated, so I instead added per-component information to `Uri` which
allows extra allocations to be avoided when constructing uris with
components from different sources, and in some cases, deferring the work
all the way to when the uri is printed, where an allocator may not even
be needed.

Closes #19587
2024-04-10 02:11:54 -07:00
Igor Anić
4151e6c31b fetch: use arena allocator for diagnostic/UnpackResult
Reference:
https://github.com/ziglang/zig/pull/19500#discussion_r1556476973

Arena is now used for Diagnostic (tar and git). `deinit` is not called on Diagnostic
allowing us to reference strings from Diagnostic in UnpackResult without
dupe.

That seamed reasonable to me. Instead of using gpa for Diagnostic, and
then dupe to arena. Or using arena for both and making dupe so we can deinit
Diagnostic.
2024-04-09 15:00:22 +02:00
Igor Anić
f8dd2a1064 fetch: add executable bit test 2024-04-09 15:00:22 +02:00
Igor Anić
8c58b8fe01 fetch: refactor package root in errors
Use stripRoot in less places. Strip it while copying error from
diagnostic to unpack result so other palaces can be free of this logic.
2024-04-09 15:00:22 +02:00
Igor Anić
b422e4a202 fetch: save test cases
Prepare test cases, store them in Fetch/testdata.
They cover changes in this PR as well from previous one #19111.
2024-04-09 15:00:22 +02:00
Igor Anić
3d5a9237f7 fetch: use empty string instead of null for root_dir
Make it consistent with Cache.Path sub_path.
Remove null check in many locations.
2024-04-09 15:00:22 +02:00
Igor Anić
5f0b434f90 fetch: remove root_dir from error messages
To be consistent with paths in manifest.
2024-04-09 15:00:22 +02:00
Igor Anić
22e9c50376 fetch: fix test tarball
Should include folder structure, at least root folder so it can be found
in pipeToFileSystem.
2024-04-09 15:00:21 +02:00
Igor Anić
fc745fb05c fetch: remove test with mount and net dependencies 2024-04-09 15:00:21 +02:00
Igor Anić
84fac2242c fix zig fmt 2024-04-09 15:00:21 +02:00
Igor Anić
dc61c2e904 add comments 2024-04-09 15:00:21 +02:00
Igor Anić
373d48212f fetch: add pathological packages test
Using test cases from:
https://github.com/ianprime0509/pathological-packages repository.
Depends on existence of the FAT32 file system. Folder is in FAT32 file
system because it is case insensitive and and does not support symlinks.

It is complicated test case requires internet connection, depends on
existence of FAT32 in the specific location. But it is so valuable for
development. Running `zig test Package.zig` is so much faster than
building zig binary and running `zig fetch URL`. Committing it here
although it should probably be removed.
2024-04-09 15:00:21 +02:00
Igor Anić
5a38924a7d fetch.git: collect file create diagnostic errors
On case insensitive file systems, don't overwrite files with same name
in different casing. Add diagnostic error so caller could decide what to do.
2024-04-09 15:00:21 +02:00
Igor Anić
ad60b6c1ed fetch: update comments 2024-04-09 15:00:21 +02:00
Igor Anić
dfec4918a3 fetch: remove absolute path from tests 2024-04-09 15:00:21 +02:00
Igor Anić
4d6a7e074b fetch: filter unpack errors
Report only errors which are not filtered by paths in build.zig.zon.
2024-04-09 15:00:21 +02:00
Igor Anić
a0790914b4 fetch: return UnpackResult from unpackResource
Test that we are still outputing same errors.
2024-04-09 15:00:21 +02:00
Igor Anić
b3339f3cd5 tar: store diagnostic errors into UnpackResult
Test that UnpackResult prints same error output as diagnostic.
2024-04-09 15:00:21 +02:00
Igor Anić
9e47857ae9 fetch: make failing test 2024-04-09 15:00:21 +02:00
Ian Johnson
129de47a71 Fix first-time package fetches
Closes #19557
Closes #19561

Previously, `build.zig` was not being detected correctly by
`computeHash` for packages where there is a containing root directory.
2024-04-07 09:07:37 +02:00
Igor Anić
34bb670bb6 package manager: set executable bit
Based on file content. Detects elf magic header or shebang line.

Executable bit is ignored in hash calculation, as it was before this. So
packages hashes are not changed.

Reference:
https://github.com/ziglang/zig/issues/17463#issuecomment-1984798880

Fixes: 17463

Test is here:
7c4600d7bb/src/main.zig (L307)
(if #19500 got accepted I'll move this test to the Fetch.zig)
2024-04-06 13:00:57 -07:00
Igor Anić
a60b7af2c1 fetch: fix manifest included paths filtering
Filter should be applied on path where package root folder (if
there is any) is stripped. Manifest is inside package root and has paths
relative to package root not temporary directory root.
2024-04-04 01:59:15 +02:00
Igor Anić
8545cb0147 fetch: use package root detection pipeToFileSystem
Based on:
https://github.com/ziglang/zig/pull/19111#discussion_r1548620985

In pipeToFileSystem we are already iterating over all files in tarball.
Inspecting them there for existence of single root folder saves two
syscalls later.
2024-04-03 21:20:39 +02:00
Igor Anić
363acf4991 fetch: update comments 2024-04-03 21:17:57 +02:00
Igor Anić
1431e34cb9 fetch: remove one openDir in runResource
Based on comment:
https://github.com/ziglang/zig/pull/19111#discussion_r1548640939

computeHash finds all files in temporary directory. There is no
difference on what path are they. When calculating hash normalized_path
must be set relative to package root. That's the place where we strip
root if needed.
2024-04-03 21:01:29 +02:00
Igor Anić
24304a4385 fetch: save syscall, and add comment
From review comments: https://github.com/ziglang/zig/pull/19111
2024-04-03 17:08:41 +02:00
Igor Anić
c4261bf562 fetch: find package root only for archives 2024-04-03 17:06:20 +02:00
Igor Anić
15ca0c9471 fix typo 2024-04-03 17:06:20 +02:00
Igor Anić
a5a928b966 package manager: don't strip components in tar
Unpack tar without removing leading root folder. Then find package root
in unpacked tmp folder.
2024-04-03 17:06:20 +02:00
Igor Anić
bc5076715b fetch: fix failing test
Prior to
[this](ef9966c985 (diff-08c935ef8c633bb630641d44230597f1cff5afb0e736d451e2ba5569fa53d915R805))
commit tar was not a valid extension. After that this one is valid case.
2024-04-03 17:06:20 +02:00
Andrew Kelley
7bc0b74b6d move Package.Path to std.Build.Cache.Path 2024-03-21 16:16:47 -07:00
Andrew Kelley
cd62005f19 extract std.posix from std.os
closes #5019
2024-03-19 11:45:09 -07:00
Andrew Kelley
f60c24c73c
Merge pull request #19155 from ianic/tar_max_file_size
std.tar: error on insufficient buffers provided to iterator
2024-03-11 17:03:44 -07:00
Igor Anić
f8f43ca3e1 package: remove git fetch zlib lookahead fix
My first zlib implementation broke git fetch because it introduce
[lookahead](https://github.com/ziglang/zig/issues/18967). That resulted
in workarounds [1](80f3ef6e14)
[2](d00faa2407)

After [fixing](https://github.com/ziglang/zig/pull/19163) lookahead in
zlib decompressor this fixes are no longer necessary.
2024-03-11 16:49:21 -07:00
Tristan Ross
9d70d614ae
std.builtin: make link mode fields lowercase 2024-03-11 07:09:10 -07:00
Igor Anić
0cca7e732e std.tar: fix broken public interface 2024-03-11 12:25:51 +01:00
mlugg
38331b1cab
Package.Module: actually set File.path_digest for builtin modules 2024-03-07 06:26:25 +00:00
mlugg
b41a0b4768
Package.Module: deduplicate identical builtin modules
Previously, when multiple modules had builtin modules with identical
sources, two distinct `Module`s and `File`s were created pointing at the
same file path. This led to a bug later in the frontend. These modules
are now deduplicated with a simple hashmap on the builtin source.
2024-03-06 21:26:39 +00:00
Andrew Kelley
d661f0f35b compiler: JIT zig fmt
See #19063
2024-02-26 22:26:19 -07:00
Andrew Kelley
08e886b8fe package management: fix regression of printing expected hash
Regressed in ed4ccea7bae99c1baa634716941f308d9f922985. The early exit
path was only supposed to happen in case of --system mode.
2024-02-25 09:27:39 -08:00
Andrew Kelley
c44a902836 fix zstd compilation errors from previous commit 2024-02-23 02:37:11 -07:00
dweiller
ac1b957e79 std.compress.zstd: remove allocation from DecompressStream 2024-02-23 02:37:11 -07:00
Andrew Kelley
99a5de9dbb git fetching: fix redirect handling
I mistakenly thought this was dead code in an earlier commit in this
branch. This commit restores the proper behavior.
2024-02-23 02:37:11 -07:00
Andrew Kelley
4d401e6159 std.http: remove Headers API
I originally removed these in 402f967ed5339fa3d828b7fe1d57cdb5bf38dbf2.
I allowed them to be added back in #15299 because they were smuggled in
alongside a bug fix, however, I wasn't kidding when I said that I wanted
to take the design of std.http in a different direction than using this
data structure.

Instead, some headers are provided via explicit field names populated
while parsing the HTTP request/response, and some are provided via
new fields that support passing extra, arbitrary headers.

This resulted in simplification of logic in many places, as well as
elimination of the possibility of failure in many places. There is
less deinitialization code happening now.

Furthermore, it made it no longer necessary to clone the headers data
structure in order to handle redirects.

http_proxy and https_proxy fields are now pointers since it is common
for them to be unpopulated.

loadDefaultProxies is changed into initDefaultProxies to communicate
that it does not actually load anything from disk or from the network.
The function now is leaky; the API user must pass an already
instantiated arena allocator. Removes the need to deinitialize proxies.

Before, proxies stored arbitrary sets of headers. Now they only store
the authorization value.

Removed the duplicated code between https_proxy and http_proxy. Finally,
parsing failures of the environment variables result in errors being
emitted rather than silently ignoring the proxy.

error.CompressionNotSupported is renamed to
error.CompressionUnsupported, matching the naming convention from all
the other errors in the same set.

Removed documentation comments that were redundant with field and type
names.

Disabling zstd decompression in the server for now; see #18937.

I found some apparently dead code in src/Package/Fetch/git.zig. I want
to check with Ian about this.

I discovered that test/standalone/http.zig is dead code, it is only
being compiled but not being run. Furthermore it hangs at the end if you
run it manually. The previous commits in this branch were written under
the assumption that this test was being run with
`zig build test-standalone`.
2024-02-23 02:37:11 -07:00
Andrew Kelley
90bd4f226e std.http: remove the ability to heap-allocate headers
The buffer for HTTP headers is now always provided via a static buffer.
As a consequence, OutOfMemory is no longer a member of the read() error
set, and the API and implementation of Client and Server are simplified.

error.HttpHeadersExceededSizeLimit is renamed to
error.HttpHeadersOversize.
2024-02-23 02:37:10 -07:00
Igor Anić
a5326c5ef8 return few previous fixes
Review: https://github.com/ziglang/zig/pull/19032#pullrequestreview-1896251841
2024-02-22 17:20:41 +01:00
Igor Anić
d00faa2407 use BufferedTee in Fetch/git.zig 2024-02-22 15:18:03 +01:00