This reverts commit 0c99ba1eab63865592bb084feb271cd4e4b0357e, reversing
changes made to 5f92b070bf284f1493b1b5d433dd3adde2f46727.
This caused a CI failure when it landed in master branch due to a
128-bit `@byteSwap` in std.mem.
Most of this migration was performed automatically with `zig fmt`. There
were a few exceptions which I had to manually fix:
* `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten
* `@truncate`'s fixup is incorrect for vectors
* Test cases are not formatted, and their error locations change
* Scan from line start when finding tag in tokenizer
This resolves a crash that can occur for invalid bytes like carriage
returns that are valid characters when not parsed from within literals.
There are potentially other edge cases this could resolve as well, as
the calling code for this function didn't account for any potential
'pending_invalid_tokens' that could be queued up by the tokenizer from
within another state.
* Fix carriage return crash in multiline string
Follow the guidance of #38:
> However CR directly before NL is interpreted as only a newline and not part of the multiline string. zig fmt will delete the CR.
Zig fmt already had code for deleting carriage returns, but would still
crash - now it no longer does so. Carriage returns encountered before
line-feeds are now appropriately removed on program compilation as well.
* Only accept carriage returns before line feeds
Previous commit was much less strict about this, this more closely
matches the desired spec of only allow CR characters in a CRLF pair, but
not otherwise.
* Fix CR being rejected when used as whitespace
Missed this comment from ziglang/zig-spec#83:
> CR used as whitespace, whether directly preceding NL or stray, is still unambiguously whitespace. It is accepted by the grammar and replaced by the canonical whitespace by zig fmt.
* Add tests for carriage return handling
For some errors if the found token is not on the same line as
the previous token, point to the end of the previous token.
This usually results in more helpful errors.
- adds initial support for the operators +|, -|, *|, <<|, +|=, -|=, *|=, <<|=
- uses operators in addition to builtins in behavior test
- adds binOpExt() and assignBinOpExt() to AstGen.zig. these need to be audited
We already have a LICENSE file that covers the Zig Standard Library. We
no longer need to remind everyone that the license is MIT in every single
file.
Previously this was introduced to clarify the situation for a fork of
Zig that made Zig's LICENSE file harder to find, and replaced it with
their own license that required annual payments to their company.
However that fork now appears to be dead. So there is no need to
reinforce the copyright notice in every single file.
It now displays the byte with proper printability handling. This makes
the relevant compile error test case no longer a regression in quality
from stage1 to stage2.
In order to not regress the quality of compile errors, some improvements
had to be made.
* std.zig.parseCharLiteral is improved to return more detailed parse
failure information.
* tokenizer is improved to handle null bytes in the middle of strings,
character literals, and line comments.
* validating how many unicode escape digits in string literals is moved
to std.zig.parseStringLiteral rather than handled in the tokenizer.
* when a tokenizer error occurs, if the reported token is the 'invalid'
tag, an error note is added to point to the invalid byte location.
Further improvements would be:
- Mention the expected set of allowed bytes at this location.
- Display the invalid byte (if printable, print it, otherwise
escape-print it).
By requiring the source file to be null-terminated, we avoid extra
branching while simplifying the logic at the same time.
Running ast-check on a large zig source file (udivmodti4_test.zig),
master branch compared to this commit:
* 4% faster wall clock
* 7% fewer cache misses
* 1% fewer branches
* Remove parser error on double ampersand
* Add failing test for double ampersand case
* Add error when encountering double ampersand in AstGen
"Bit and" operator should not make sense when one of its operands
is an address.
* Check that 2 ampersands are adjacent to each other in source string
* Remove cases of unused variables in tests
Instead require `1e9` and `0x1p9`, disallowing the trailing dot.
This change to the grammar is consistent with forbidding `1.` and `0x1.`
as float literals and ensures there is only one way to do things here.
Achieve this by reducing the amount of special casing to handle EOF so
that the already correct logic for normal comments does not need to be
duplicated.
