* std.c: consolidate some definitions, making them share code. For
example, freebsd, dragonfly, and openbsd can all share the same
`pthread_mutex_t` definition.
* add type safety to std.c.O
- this caught a bug where mode flags were incorrectly passed as the
open flags.
* 3 fewer uses of usingnamespace keyword
* as per convention, remove purposeless field prefixes from struct field
names even if they have those prefixes in the corresponding C code.
* fix incorrect wasi libc Stat definition
* remove C definitions from incorrectly being in std.os.wasi
* make std.os.wasi definitions type safe
* go through wasi native APIs even when linking libc because the libc
APIs are problematic and wasteful
* don't expose WASI definitions in std.posix
* remove std.os.wasi.rights_t.ALL: this is a footgun. should it be all
future rights too? or only all current rights known? both are
the wrong answer.
As suggested by @matu3ba, it can be better to use Security Attributes
directly while creating the handle instead of creating the handle then
setting the handle to inherit. Doing so can prevent potentially leaking
to other parallel spawned processes which would inherit the opened `\Device\Null`
handle.
This change also allows windows.OpenFile to handle when bInheritHandle
is set.
Note that we are using the same `saAttr`, but since it's taken as a
pointer to a const in all calls, it's never mutated, and OpenFile never alters it.
This also saves 1 kernel call for setting the handle to inherit.
This commit allows write access to the `\\Device\\Null` Handle.
Without a write access, it's not possible for the child process to write
SdOut to Null. As a requirement `SetHandleInformation` was also changed
to mark the handle as iheritable (by adding it to Flags) by the spawned process.
This allows the child to access the NUL device that was opened.
This also makes the Windows part to behave similarly to `spawnPosix`.
The old implementation had a bug in it in that it didn't quote empty strings, but it also didn't properly follow the special quoting rules required for the first argument (the executable name). This new implementation serializes the argv correctly such that it can be parsed by the `CommandLineToArgvW` algorithm.
* move std.atomic.Atomic to std.atomic.Value
* fix incorrect argument order passed to testing.expectEqual
* make the functions be a thin wrapper over the atomic builtins and
stick to the naming conventions.
* remove pointless functions loadUnchecked and storeUnchecked. Instead,
name the field `raw` instead of `value` (which is redundant with the
type name).
* simplify the tests by not passing every possible combination. Many
cases were iterating over every possible combinations but then not
even using the for loop element value!
* remove the redundant compile errors which are already implemented by
the language itself.
* remove dead x86 inline assembly. this should be implemented in the
language if at all.
Use inline to vastly simplify the exposed API. This allows a
comptime-known endian parameter to be propogated, making extra functions
for a specific endianness completely unnecessary.
Justification: exec, execv etc are unix concepts and portable version
should be called differently.
Do no touch non-Zig code. Adjust error names as well, if associated.
Closes#5853.
Previously, a relative path like `..` would:
- Attempt to be normalized (i.e. remove . and .. without any path resolution), but would error with TooManyParentDirs
- This would make wToPrefixedFileW run it through `RtlGetFullPathName_U` to do the necessary path resolution, but `RtlGetFullPathName_U` always resolves relative paths relative to the CWD
Instead, when TooManyParentDirs occurs, we now look up the path of the passed in `dir` (if it's non-null) and append the relative path to it before giving it to `RtlGetFullPathName_U`. If `dir` is null, then we just give it RtlGetFullPathName_U directly and let it resolve it relative to the CWD.
Closes#16779
When calling NtCreateFile with a UNC path, if either `\\server` or `\\server\share` are not found, then the statuses `BAD_NETWORK_PATH` or `BAD_NETWORK_NAME` are returned (respectively).
These statuses are not translated into `error.FileNotFound` because they convey more information than the typical FileNotFound error. For example, if you were trying to call `Dir.makePath` with an absolute UNC path like `\\MyServer\MyShare\a\b\c\d`, then knowing that `\\MyServer\MyShare` was not found allows for returning after trying to create the first directory instead of then trying to create `a\b\c`, `a\b`, etc. when it's already known that they will all fail in the same way.
This fixes a regression caused by https://github.com/ziglang/zig/pull/13993
As an optimization, the first call to `NtQueryDirectoryFile` would only ask for a single result and assume that if the result returned did not match the app_name exactly, then the unappended app_name did not exist. However, this relied on the assumption that the unappended app_name would always be returned first, but that only seems to be the case on NTFS. On FAT filesystems, the order of returned files can be different, which meant that it could assume the unappended file doesn't exist when it actually does.
This commit fixes that by fully iterating the wildcard matches via `NtQueryDirectoryFile` and taking note of any unappended/PATHEXT-appended filenames it finds. In practice, this strategy does not introduce a speed regression compared to the previous (buggy) implementation.
Benchmark 1 (10 runs): winpathbench-master.exe
measurement mean ± σ min … max outliers delta
wall_time 508ms ± 4.08ms 502ms … 517ms 1 (10%) 0%
peak_rss 3.62MB ± 2.76KB 3.62MB … 3.63MB 0 ( 0%) 0%
Benchmark 2 (10 runs): winpathbench-fat32-fix.exe
measurement mean ± σ min … max outliers delta
wall_time 500ms ± 21.4ms 480ms … 535ms 0 ( 0%) - 1.5% ± 2.8%
peak_rss 3.62MB ± 2.76KB 3.62MB … 3.63MB 0 ( 0%) - 0.0% ± 0.1%
---
Partially addresses #16374 (it fixes `zig build` on FAT32 when no `zig-cache` is present)
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
There are many different types of Windows paths, and there are a few different possible namespaces on top of that. Before this commit, NT namespaced paths were somewhat supported, and for Win32 paths (those without a namespace prefix), only relative and drive absolute paths were supported. After this commit, all of the following are supported:
- Device namespaced paths (`\\.\`)
- Verbatim paths (`\\?\`)
- NT-namespaced paths (`\??\`)
- Relative paths (`foo`)
- Drive-absolute paths (`C:\foo`)
- Drive-relative paths (`C:foo`)
- Rooted paths (`\foo`)
- UNC absolute paths (`\\server\share\foo`)
- Root local device paths (`\\.` or `\\?` exactly)
Plus:
- Any of the path types and namespace types can be mixed and matched together as appropriate.
- All of the `std.os.windows.*ToPrefixedFileW` functions will accept any path type, prefixed or not, and do the appropriate thing to convert them to an NT-prefixed path if necessary.
This is achieved by making the `std.os.windows.*ToPrefixedFileW` functions behave like `ntdll.RtlDosPathNameToNtPathName_U`, but with a few differences:
- Does not allocate on the heap (this is why we can't use `ntdll.RtlDosPathNameToNtPathName_U` directly, it does internal heap allocation).
- Relative paths are kept as relative unless they contain too many .. components, in which case they are treated as 'drive relative' and resolved against the CWD (this is how it behaved before this commit as well).
- Special case device names like COM1, NUL, etc are not handled specially (TODO)
- `.` and space are not stripped from the end of relative paths (potential TODO)
Most of the non-trivial conversion of non-relative paths is done via `ntdll.RtlGetFullPathName_U`, which AFAIK is used internally by `ntdll.RtlDosPathNameToNtPathName_U`.
Some relevant reading on Windows paths:
- https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html
- https://chrisdenton.github.io/omnipath/Overview.htmlCloses#8205
Might close (untested) #12729
Note:
- This removes checking for illegal characters in `std.os.windows.sliceToPrefixedFileW`, since the previous solution (iterate the whole string and error if any illegal characters were found) was naive and won't work for all path types. This is further complicated by things like file streams (where `:` is used as a delimiter, e.g. `file.ext:stream_name:$DATA`) and things in the device namespace (where a path like `\\.\GLOBALROOT\??\UNC\localhost\C$\foo` is valid despite the `?`s in the path and is effectively equivalent to `C:\foo`). Truly validating paths is complicated and would need to be tailored to each path type. The illegal character checking being removed may open up users to more instances of hitting `OBJECT_NAME_INVALID => unreachable` when using `fs` APIs.
+ This is related to https://github.com/ziglang/zig/issues/15607
The majority of these are in comments, some in doc comments which might
affect the generated documentation, and a few in parameter names -
nothing that should be breaking, however.
Notably the Darwin (XNU) kernel the maxrss field is number of bytes
and not kilobytes (kibibytes) like other platforms (e.g. Linux, BSD).
watchOS and tvOS are not supported because they do not have the ability
to spawn a child process. iOS is enabled but due to OS sandboxing it
should fail with a permission error.
`GetProcessMemoryInfo` is implemented using `NtQueryInformationProcess`
with `ProcessVmCounters` to obtain `VM_COUNTERS`. The structs, enum
definitions are found in `winternl.h` or `ntddk.h` in the latest WDK.
This should give the same results as using `K32GetProcessMemoryInfo`
In Windows, the equivalent to maxrss is PeakWorkingSetSize which is
found in PROCESS_MEMORY_COUNTERS in bytes.
Currently, this is done by calling `GetProcessMemoryInfo` in kernel32.
* Eliminate all uses of `std.debug.print` in make() functions, instead
properly using the step failure reporting mechanism.
* Introduce the concept of skipped build steps. These do not cause the
build to fail, and they do allow their dependants to run.
* RunStep gains a new flag, `skip_foreign_checks` which causes the
RunStep to be skipped if stdio mode is `check` and the binary cannot
be executed due to it being a foreign executable.
- RunStep is improved to automatically use known interpreters to
execute binaries if possible (integrating with flags such as
-fqemu and -fwasmtime). It only does this after attempting a native
execution and receiving a "exec file format" error.
- Update RunStep to use an ArrayList for the checks rather than this
ad-hoc reallocation/copying mechanism.
- `expectStdOutEqual` now also implicitly adds an exit_code==0 check
if there is not already an expected termination. This matches
previously expected behavior from older API and can be overridden by
directly setting the checks array.
* Add `dest_sub_path` to `InstallArtifactStep` which allows choosing an
arbitrary subdirectory relative to the prefix, as well as overriding
the basename.
- Delete the custom InstallWithRename step that I found deep in the
test/ directory.
* WriteFileStep will now update its step display name after the first
file is added.
* Add missing stdout checks to various standalone test case build
scripts.
Today I found out that posix_spawn is trash. It's actually implemented
on top of fork/exec inside of libc (or libSystem in the case of macOS).
So, anything posix_spawn can do, we can do better. In particular, what
we can do better is handle spawning of child processes that are
potentially foreign binaries. If you try to spawn a wasm binary, for
example, posix spawn does the following:
* Goes ahead and creates a child process.
* The child process writes "foo.wasm: foo.wasm: cannot execute binary file"
to stderr (yes, it prints the filename twice).
* The child process then exits with code 126.
This behavior is indistinguishable from the binary being successfully
spawned, and then printing to stderr, and exiting with a failure -
something that is an extremely common occurrence.
Meanwhile, using the lower level fork/exec will simply return ENOEXEC
code from the execve syscall (which is mapped to zig error.InvalidExe).
The posix_spawn behavior means the zig build runner can't tell the
difference between a failure to run a foreign binary, and a binary that
did run, but failed in some other fashion. This is unacceptable, because
attempting to excecve is the proper way to support things like Rosetta.
Previously, this API had pid, to be used on POSIX systems, and handle,
to be used on Windows.
This commit unifies the API, defining an Id type that is either the pid
or the HANDLE depending on the target OS.
This commit also prepares for the future by allowing one to import via
`std.process.Child` which is the fully qualified namespace that I intend
to migrate to in the future.
