121 Commits

Author SHA1 Message Date
Gungun974
21a0885ae7 Make zig fetch handle jar like zip 2025-06-06 16:59:40 -04:00
Andrew Kelley
f32a5d349d std: eradicate u29 and embrace std.mem.Alignment 2025-04-13 02:20:32 -04:00
Eric Joldasov
27c1f2b3a0 zig build: allow to choose "lazy mode" for fetching process
`--fetch` flag now has additional optional parameter, which specifies
how lazy dependencies should be fetched:
 * `needed` — lazy dependencies are fetched only if they are required
   for current build configuration to work. Default and works same
   as old `--fetch` flag.
 * `all` — lazy dependencies are always fetched. If `--system` flag
   is used after that, it's guaranteed that **any** build configuration
   will not require additional download of dependencies during build.
   Helpful for distro packagers and CI systems:
   https://www.github.com/ziglang/zig/issues/14597#issuecomment-1426827495

If none is passed, behaviour is same as if `needed` was passed.

Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
2025-03-26 19:00:33 +01:00
Matthew Lugg
d0911786c9
Merge pull request #22397 from Techatrix/type-safe-ast
improve type safety of std.zig.Ast
2025-03-12 02:22:41 +00:00
Mathias Lafeldt
7e751e8040
Merge pull request #23193 from mlafeldt/fix-macho-detection
Fetch: enhance Mach-O executable detection for modern Macs

closes #21044
2025-03-11 06:12:13 +00:00
Techatrix
ca6fb30e99
std.zig.Ast: improve type safety
This commits adds the following distinct integer types to std.zig.Ast:
- OptionalTokenIndex
- TokenOffset
- OptionalTokenOffset
- Node.OptionalIndex
- Node.Offset
- Node.OptionalOffset

The `Node.Index` type has also been converted to a distinct type while
`TokenIndex` remains unchanged.

`Ast.Node.Data` has also been changed to a (untagged) union to provide
safety checks.
2025-03-07 22:22:01 +01:00
Carl Åstholm
8363b95178 Fix "dependency path outside project" error for nested local path dependencies
Closes #23076
2025-03-04 17:27:21 -05:00
Andrew Kelley
de43f5eb6a rename "nonce" to "fingerprint" 2025-02-26 11:42:04 -08:00
Andrew Kelley
0fc7c9f57c switch from "id" to "nonce"
mainly this addresses the following use case:

1. Someone creates a template with build.zig.zon, id field included
   (note that zig init does not create this problem since it generates
   fresh id every time it runs).
2. User A uses the template, changing package name to "example" but not
   id field.
3. User B uses the same template, changing package name also to
   "example", also not changing the id field.

Here, both packages have unintentional conflicting logical ids.

By making the field a combination of name checksum + random id, this
accident is avoided. "nonce" is an OK name for this.

Also relaxes errors on remote packages when using `zig fetch`.
2025-02-26 11:42:03 -08:00
Andrew Kelley
d6a88ed74d introduce package id and redo hash format again
Introduces the `id` field to `build.zig.zon`.

Together with name, this represents a globally unique package
identifier. This field should be initialized with a 16-bit random number
when the package is first created, and then *never change*. This allows
Zig to unambiguously detect when one package is an updated version of
another.

When forking a Zig project, this id should be regenerated with a new
random number if the upstream project is still maintained. Otherwise,
the fork is *hostile*, attempting to take control over the original
project's identity.

`0x0000` is invalid because it obviously means a random number wasn't
used.

`0xffff` is reserved to represent "naked" packages.

Tracking issue #14288

Additionally:

* Fix bad path in error messages regarding build.zig.zon file.
* Manifest validates that `name` and `version` field of build.zig.zon
  are maximum 32 bytes.
* Introduce error for root package to not switch to enum literal for
  name.
* Introduce error for root package to omit `id`.
* Update init template to generate `id`
* Update init template to populate `minimum_zig_version`.
* New package hash format changes:
  - name and version limited to 32 bytes via error rather than truncation
  - truncate sha256 to 192 bits rather than 40 bits
  - include the package id

This means that, given only the package hashes for a complete dependency
tree, it is possible to perform version selection and know the final
size on disk, without doing any fetching whatsoever. This prevents
wasted bandwidth since package versions not selected do not need to be
fetched.
2025-02-26 11:42:03 -08:00
Andrew Kelley
a57b0a0f2f fix generated hash of by-path dependencies
This branch regressed from master by switching to binary rather than hex
digest, allowing null bytes to end up in identifiers in the zig file.

This commit fixes it by changing the "hash" to be literally equal to the
sub_path (with a prefix '/' to indicate "global") if it can fit. If it
is too long then it is actually hashed, and that value used instead.
2025-02-26 11:42:03 -08:00
Andrew Kelley
12355cfb4c Package: new hash format
legacy format is also supported.

closes #20178
2025-02-26 11:42:03 -08:00
Meghan Denny
9142482372
std.ArrayList: popOrNull() -> pop() [v2] (#22720) 2025-02-10 04:21:31 +00:00
Andrew Kelley
284de7d957 adjust runtime page size APIs
* fix merge conflicts
* rename the declarations
* reword documentation
* extract FixedBufferAllocator to separate file
* take advantage of locals
* remove the assertion about max alignment in Allocator API, leaving it
  Allocator implementation defined
* fix non-inline function call in start logic

The GeneralPurposeAllocator implementation is totally broken because it
uses global state but I didn't address that in this commit.
2025-02-06 14:23:23 -08:00
Archbirdplus
439667be04 runtime page size detection
heap.zig: define new default page sizes
heap.zig: add min/max_page_size and their options
lib/std/c: add miscellaneous declarations
heap.zig: add pageSize() and its options
switch to new page sizes, especially in GPA/stdlib
mem.zig: remove page_size
2025-02-06 14:23:23 -08:00
Fabio Arnold
97b97ae620
Package fetch: add executable detection for Mach-O file headers (#21555) 2025-01-21 11:48:19 -08:00
Ian Johnson
5217da57fc zig fetch: support SHA-256 Git repositories
Closes #21888
2024-12-13 08:49:45 -05:00
Carl Åstholm
de3db8a1a6 Fix undefined behavior in package extraction code 2024-12-13 08:49:45 -05:00
Carl Åstholm
b352595aa2 Add compiler internals tests
There are several test decls inside `/src` that are not currently being
tested and have bitrotted as a result. This commit revives those tests
and adds the `test-compiler-internals` set of tests which tests
everything reachable from `/src/main.zig`.
2024-12-13 08:49:02 -05:00
Tw
aa7d138462 zig fetch: add missing path separator in error message
Signed-off-by: Tw <tw19881113@gmail.com>
2024-11-29 18:50:21 -05:00
Ian Johnson
dceab4502a zig fetch: handle redirects for Git packages
Closes #21976
2024-11-19 00:35:00 -08:00
Carl Åstholm
4fc295dc02 Take eagerness into account when deduplicating dependencies
If the same dependency is first found as lazy and then later as eager,
the existing entry needs to be updated to eager in order for
`b.dependency()` to work.
2024-11-13 14:59:46 -08:00
mlugg
54611e32d7 Package.Fetch: add another non-standard Content-Type
For instance, the official download site for libvterm uses this MIME
type for tar.gz tarballs.
2024-09-12 19:53:02 -07:00
Linus Groh
8588964972 Replace deprecated default initializations with decl literals 2024-09-12 16:01:23 +01:00
Travis Staloch
204107c499 package-manager: add application/x-tar-gz mime type
Closes #21314

This allows the package manger to download tar.gz bitbucket urls.
2024-09-06 10:37:29 -07:00
mlugg
0fe3fd01dd
std: update std.builtin.Type fields to follow naming conventions
The compiler actually doesn't need any functional changes for this: Sema
does reification based on the tag indices of `std.builtin.Type` already!
So, no zig1.wasm update is necessary.

This change is necessary to disallow name clashes between fields and
decls on a type, which is a prerequisite of #9938.
2024-08-28 08:39:59 +01:00
Andrew Kelley
eb4028bf30 add std.fmt.hex
converts an unsigned integer into an array
2024-07-20 01:06:29 -07:00
Igor Anić
e0350859bb use unreachable keyword for unreachable code path 2024-07-03 12:12:53 -07:00
Igor Anić
035c1b6522 std.tar: add strip components error to diagnostics
This was the only kind of error which was raised in pipeToFileSystem and
not added to Diagnostics.
Shell tar silently ignores paths which are stripped out when used with
`--strip-components` switch. This enables that same behavior, errors
will be collected in diagnostics but caller is free to ignore that type
of diagnostics errors.
Enables use case where caller knows structure of the tar file and want
to extract only some deeply nested folders ignoring upper files/folders.

Fixes: #17620 by giving caller options:
- not provide diagnostic and get errors
- provide diagnostics and analyze errors
- provide diagnostics and ignore errors
2024-07-03 12:12:50 -07:00
Ryan Liptak
76fb2b685b std: Convert deprecated aliases to compile errors and fix usages
Deprecated aliases that are now compile errors:

- `std.fs.MAX_PATH_BYTES` (renamed to `std.fs.max_path_bytes`)
- `std.mem.tokenize` (split into `tokenizeAny`, `tokenizeSequence`, `tokenizeScalar`)
- `std.mem.split` (split into `splitSequence`, `splitAny`, `splitScalar`)
- `std.mem.splitBackwards` (split into `splitBackwardsSequence`, `splitBackwardsAny`, `splitBackwardsScalar`)
- `std.unicode`
  + `utf16leToUtf8Alloc`, `utf16leToUtf8AllocZ`, `utf16leToUtf8`, `fmtUtf16le` (all renamed to have capitalized `Le`)
  + `utf8ToUtf16LeWithNull` (renamed to `utf8ToUtf16LeAllocZ`)
- `std.zig.CrossTarget` (moved to `std.Target.Query`)

Deprecated `lib/std/std.zig` decls were deleted instead of made a `@compileError` because the `refAllDecls` in the test block would trigger the `@compileError`. The deleted top-level `std` namespaces are:

- `std.rand` (renamed to `std.Random`)
- `std.TailQueue` (renamed to `std.DoublyLinkedList`)
- `std.ChildProcess` (renamed/moved to `std.process.Child`)

This is not exhaustive. Deprecated aliases that I didn't touch:
  + `std.io.*`
  + `std.Build.*`
  + `std.builtin.Mode`
  + `std.zig.c_translation.CIntLiteralRadix`
  + anything in `src/`
2024-06-13 10:18:59 -04:00
Andrew Kelley
3b77f1ed7e rename zig-cache to .zig-cache
closes #20077
2024-05-29 10:20:15 -07:00
Andrew Kelley
f97c2f28fd update the codebase for the new std.Progress API 2024-05-27 20:56:48 -07:00
Christofer Nolander
8f6b1f2c38
zig fetch: resolve branch/tag names to commit SHA (#19941)
* Revert "Revert "Merge pull request #19349 from nolanderc/save-commit""

This reverts commit 6ca4ed5948d8eaab28fc5e3706aeb1b113a210af.

* update to new URI changes, rework `--save` type

* initialize `latest_commit` to null everywhere
2024-05-11 14:19:35 -07:00
Andrew Kelley
6ca4ed5948 Revert "Merge pull request #19349 from nolanderc/save-commit"
This reverts commit 7fa2357d0586cef742bf691d69a6cffdd353b496, reversing
changes made to cb77bd672c3b398e3c5f6be80af03243bf8638e3.
2024-05-10 16:41:15 -07:00
Andrew Kelley
7fa2357d05
Merge pull request #19349 from nolanderc/save-commit
`zig fetch`: resolve branch/tag names to commit SHA
2024-05-10 16:27:30 -07:00
Andrew Kelley
a72292513e add std.Thread.Pool.spawnWg
This function accepts a WaitGroup parameter and manages the reference
counting therein. It also is infallible.

The existing `spawn` function is still handy when the job wants to
further schedule more tasks.
2024-05-03 20:58:02 -07:00
Jonathan Marler
a96b78c170 add std.zip and support zip files in build.zig.zon
fixes #17408

Helpful reviewers/testers include Joshe Wolfe, Auguste Rame, Andrew
Kelley and Jacob Young.

Co-authored-by: Joel Gustafson <joelg@mit.edu>
2024-05-03 16:58:53 -04:00
Igor Anić
9cfac4718d fetch: combine unpack error validation in a single function
Follow up of: #19500
[discussion](https://github.com/ziglang/zig/pull/19500#discussion_r1558238138)
2024-04-11 15:44:40 -07:00
Jacob Young
c4587dc9f4 Uri: propagate per-component encoding
This allows `std.Uri.resolve_inplace` to properly preserve the fact
that `new` is already escaped but `base` may not be.  I originally tried
just moving `raw_uri` around, but it made uri resolution unmanagably
complicated, so I instead added per-component information to `Uri` which
allows extra allocations to be avoided when constructing uris with
components from different sources, and in some cases, deferring the work
all the way to when the uri is printed, where an allocator may not even
be needed.

Closes #19587
2024-04-10 02:11:54 -07:00
Igor Anić
4151e6c31b fetch: use arena allocator for diagnostic/UnpackResult
Reference:
https://github.com/ziglang/zig/pull/19500#discussion_r1556476973

Arena is now used for Diagnostic (tar and git). `deinit` is not called on Diagnostic
allowing us to reference strings from Diagnostic in UnpackResult without
dupe.

That seamed reasonable to me. Instead of using gpa for Diagnostic, and
then dupe to arena. Or using arena for both and making dupe so we can deinit
Diagnostic.
2024-04-09 15:00:22 +02:00
Igor Anić
f8dd2a1064 fetch: add executable bit test 2024-04-09 15:00:22 +02:00
Igor Anić
8c58b8fe01 fetch: refactor package root in errors
Use stripRoot in less places. Strip it while copying error from
diagnostic to unpack result so other palaces can be free of this logic.
2024-04-09 15:00:22 +02:00
Igor Anić
b422e4a202 fetch: save test cases
Prepare test cases, store them in Fetch/testdata.
They cover changes in this PR as well from previous one #19111.
2024-04-09 15:00:22 +02:00
Igor Anić
3d5a9237f7 fetch: use empty string instead of null for root_dir
Make it consistent with Cache.Path sub_path.
Remove null check in many locations.
2024-04-09 15:00:22 +02:00
Igor Anić
5f0b434f90 fetch: remove root_dir from error messages
To be consistent with paths in manifest.
2024-04-09 15:00:22 +02:00
Igor Anić
22e9c50376 fetch: fix test tarball
Should include folder structure, at least root folder so it can be found
in pipeToFileSystem.
2024-04-09 15:00:21 +02:00
Igor Anić
fc745fb05c fetch: remove test with mount and net dependencies 2024-04-09 15:00:21 +02:00
Igor Anić
84fac2242c fix zig fmt 2024-04-09 15:00:21 +02:00
Igor Anić
dc61c2e904 add comments 2024-04-09 15:00:21 +02:00
Igor Anić
373d48212f fetch: add pathological packages test
Using test cases from:
https://github.com/ianprime0509/pathological-packages repository.
Depends on existence of the FAT32 file system. Folder is in FAT32 file
system because it is case insensitive and and does not support symlinks.

It is complicated test case requires internet connection, depends on
existence of FAT32 in the specific location. But it is so valuable for
development. Running `zig test Package.zig` is so much faster than
building zig binary and running `zig fetch URL`. Committing it here
although it should probably be removed.
2024-04-09 15:00:21 +02:00