300 Commits

Author SHA1 Message Date
Andrew Kelley
7e58c56ca7 self-hosted: implement Decl lookup
* Take advantage of coercing anonymous struct literals to struct types.
 * Reworks Module to favor Zig source as the primary use case.
   Breaks ZIR compilation, which will have to be restored in a future commit.
 * Decl uses src_index rather then src, pointing to an AST Decl node
   index, or ZIR Module Decl index, rather than a byte offset.
 * ZIR instructions have an `analyzed_inst` field instead of Module
   having a hash table.
 * Module.Fn loses the `fn_type` field since it is redundant with
   its `owner_decl` `TypedValue` type.
 * Implement Type and Value copying. A ZIR Const instruction's TypedValue
   is copied to the Decl arena during analysis, which allows freeing the
   ZIR text instructions post-analysis.
 * Don't flush the ELF file if there are compilation errors.
 * Function return types allow arbitrarily complex expressions.
 * AST->ZIR for function calls and return statements.
2020-06-18 17:12:56 -04:00
Vexu
e7207bc267
add workaround for #5599 2020-06-14 20:13:02 +03:00
Jonathan Marler
7481582774 support Writer instead of OutStream
Start implementing https://github.com/ziglang/zig/issues/4917 which is to rename instream/outstream to reader/writer.  This first change allows code to use Writer/writer instead of OutStream/outStream, but still maintains the old outstream names with "Deprecated" comments.
2020-06-08 22:59:28 -04:00
Alexandros Naskos
aa1a727284 Allow carriare return in comments 2020-06-02 00:56:05 -04:00
Vexu
a47257d9b0
fix std.zig rejecting literal tabs in comments 2020-06-01 14:37:36 +03:00
Vexu
010c58e303
fix zig fmt out of bounds on empty file 2020-05-30 23:07:51 +03:00
Andrew Kelley
1e0de896b8
Merge pull request #5452 from squeek502/comptime-string-map
Add std.ComptimeStringMap based on the tokenizer optimization in #5442
2020-05-28 23:06:40 -04:00
Ryan Liptak
a9dd79d293 Use std.ComptimeStringMap in zig's tokenizer
Should have no performance change, the implementation was just made generic/re-usable
2020-05-26 21:36:58 -07:00
Jakub Konka
e61e8c94be Reenable zig parser tests disabled targeting Wasm
I'm not sure why I disabled them when landing extended Wasm/WASI
support, but they pass the parser tests just fine now, so I'm gonna
go ahead and re-enable them.
2020-05-26 21:01:54 -04:00
Andrew Kelley
ba41a9d5d7 different strategy for tokenizing keywords
throughput: 279 MiB/s => 347 MiB/s
2020-05-26 18:59:49 -04:00
Vexu
e07b467c7c
fix missing compile error on while/for missing block 2020-05-25 23:25:06 +03:00
Andrew Kelley
69ff89fd12 stage2 parser: heuristics to pre-allocate token arrays
throughput: 72.2 MiB/s => 75.3 MiB/s

I also tried the idea from the deleted comment in this commit and it
made the throughput worse.
2020-05-25 15:12:23 -04:00
Jethro Nederhof
4b1a846471
target: Implement running OS version detection for FreeBSD (#5416)
* target: Implement OS version detection for FreeBSD

Fixes #4582

* freebsd version: handle errors explicitly
2020-05-24 20:52:13 -04:00
Andrew Kelley
dd05f2be80 run zig fmt on std lib 2020-05-24 10:04:09 -04:00
Andrew Kelley
1c0b7ddda8 fix regression in AST node iteration
the new iteration code caused an integer underflow for function
prototypes with no parameters. now fixed.
2020-05-24 10:03:26 -04:00
Andrew Kelley
f771545a7e revert std.zig.ast iterator changes back to master branch API 2020-05-23 20:14:38 -04:00
Andrew Kelley
8df0841d6a stage2 parser: token ids in their own array
To prevent cache misses, token ids go in their own array, and the
start/end offsets go in a different one.

perf measurement before:
         2,667,914      cache-misses:u
     2,139,139,935      instructions:u
       894,167,331      cycles:u

perf measurement after:
         1,757,723      cache-misses:u
     2,069,932,298      instructions:u
       858,105,570      cycles:u
2020-05-22 12:34:12 -04:00
Andrew Kelley
295bca9b5f stage2 parser: don't append doc comments to the list
The DocComment AST node now only points to the first doc comment token.
API users are expected to iterate over the following tokens directly.

After this commit there are no more linked lists in use in the
self-hosted AST API.

Performance impact is negligible. Memory usage slightly reduced.
2020-05-22 00:28:59 -04:00
Andrew Kelley
8252c8b9d6 stage2 parser: different multiline string literal parsing strategy
and using flat memory rather than singly linked list

roughly equivalent performance, slightly reduced memory usage, better
API.
2020-05-21 23:25:15 -04:00
Andrew Kelley
19de259936 stage2 parser: arrays and structs directly in memory after the node
Slightly reduced memory usage. Roughly the same API and perf.
2020-05-21 22:52:45 -04:00
Andrew Kelley
9377af934f stage2 parser: SwitchCase uses intrusive array instead of linkedlist
no perf impact, but the API is better
2020-05-21 22:28:30 -04:00
Andrew Kelley
d37b81d43b stage2 parser performance/API improvements
* Extract Call ast node tag out of SuffixOp; parameters go in memory
   after Call.
 * Demote AsmInput and AsmOutput from AST nodes to structs inside the
   Asm node.
 * The following ast nodes get their sub-node lists directly following
   them in memory:
   - ErrorSetDecl
   - Switch
   - BuiltinCall
 * ast.Node.Asm gets slices for inputs, outputs, clobbers instead of
   singly linked lists

Performance changes:

throughput: 72.7 MiB/s => 74.0 MiB/s
maxrss: 72 KB => 69 KB (nice)
2020-05-21 22:01:16 -04:00
Andrew Kelley
32ecb416f3 fix compile errors when setting NodeIndex/TokenIndex to u32 2020-05-21 00:30:08 -04:00
Andrew Kelley
d57d9448aa stage2 parsing: rework block statements AST memory layout
block statements are now directly following the Block AST node rather
than a singly linked list. This had negligible impact on performance:

throughput: 72.3 MiB/s => 72.7 MiB/s

however it greatly improves the API since the statements are laid out in
a flat array in memory.
2020-05-20 23:47:04 -04:00
Andrew Kelley
688aa114e4 Revert "stage2 parser: elide memcpy of large initialization lists"
This reverts commit 84df1d4f3d0312553f5a3857ed67042319c20846.

Not worth the complexity! Always memcpy initialization lists into the
arena.
2020-05-20 22:42:43 -04:00
Andrew Kelley
84df1d4f3d stage2 parser: elide memcpy of large initialization lists
throughput: 71.4 MiB/s => 72.9 MiB/s
2020-05-20 22:41:18 -04:00
Andrew Kelley
897f23f20f stage2 parser: split off some SuffixOp AST nodes into separate tags
These SuffixOp nodes have their own ast.Node tags now:
 * ArrayInitializer
 * ArrayInitializerDot
 * StructInitializer
 * StructInitializerDot

Their sub-expression lists are general-purpose-allocator allocated
and then copied into the arena after completion of parsing.

throughput: 72.9 MiB/s => 74.4 MiB/s
maxrss: 68 KB => 72 KB

The API is also nicer since the sub expression lists are now flat arrays
instead of singly linked lists.
2020-05-20 19:18:14 -04:00
Andrew Kelley
7c2c0e36f8 stage2 parser: different memory layout of ParamDecl
Instead of being its own node, it's a struct inside FnProto.
Instead of FnProto having a SinglyLinkedList of ParamDecl nodes,
ParamDecls are appended directly in memory after the FnProto.

throughput: 72.2 MiB/s => 72.9 MiB/s
maxrss: 70 KB => 68 KB

Importantly, the API is improved as well since the data is arranged
linearly in memory.
2020-05-20 17:39:54 -04:00
Andrew Kelley
64c149ca19 fields and decls: ArrayList appending, memcpy to ast arena
This makes fields and decl ast nodes part of the Root and ContainerDecl
AST nodes.

Surprisingly, it's a performance regression from using a singly-linked
list for these nodes:

throughput: 76.5 MiB/s => 69.4 MiB/s

However it has much better memory usage:

maxrss: 392 KB => 77 KB

It's also better API for consumers of the parser, since it is a flat
list in memory.
2020-05-20 16:13:02 -04:00
Andrew Kelley
8c10178a1e golly jeepers it's taking a long time to update translate-c 2020-05-20 13:53:53 -04:00
Andrew Kelley
4617c5907a fix zig fmt test regressions 2020-05-20 00:19:00 -04:00
Andrew Kelley
82225f6568 zig fmt parser tests are compiling again 2020-05-19 23:08:48 -04:00
Andrew Kelley
93384f7428 use singly linked lists for std.zig.parse
std.ast uses a singly linked list for lists of things. This is a
breaking change to the self-hosted parser API.

std.ast.Tree has been separated into a private "Parser" type which
represents in-progress parsing, and std.ast.Tree which has only
"output" data. This means cleaner, but breaking, API for parse results.
Specifically, `tokens` and `errors` are no longer SegmentedList but a
slice.

The way to iterate over AST nodes has necessarily changed since lists of
nodes are now singly linked lists rather than SegmentedList.

From these changes, I observe the following on the
self-hosted-parser benchmark from ziglang/gotta-go-fast:

throughput: 45.6 MiB/s => 55.6 MiB/s
maxrss: 359 KB => 342 KB

This commit breaks the build; more updates are necessary to fix API
usage of the self-hosted parser.
2020-05-19 21:22:52 -04:00
Jakub Konka
3d267bab71 Re-enable refAllDecls gen and check in std.zig 2020-05-18 21:05:29 +02:00
Vexu
28d449b38d
fix zig fmt regression 2020-05-17 15:13:19 +03:00
Vexu
eda03354dc
Merge pull request #5358 from Vexu/parser
Fix infinite loop with invalid comptime
2020-05-16 20:49:37 +03:00
Vexu
081ffe24cf
fix infinite loop with invalid comptime 2020-05-16 19:23:59 +03:00
Vexu
74f7d710bb
Merge pull request #5032 from LakeByTheWoods/redo_translate_c
Translate C: Redo Add comment containing c source location for failed decls.
2020-05-16 17:37:19 +03:00
Vexu
a05011d403
move ParamType to Node.ParamDecl 2020-05-16 12:54:51 +03:00
Vexu
ed62081d38
recover from missing semicolon after if stmt 2020-05-16 12:29:01 +03:00
Vexu
6ca0def499
recover from invalid global error set access 2020-05-16 12:09:34 +03:00
Vexu
b2f16d4484
fix infinite loop with mismatced bracket 2020-05-15 23:50:47 +03:00
Vexu
a00fd6e254
properly handle extra closing braces at top level 2020-05-15 14:30:49 +03:00
Vexu
440189a04a
cleanup* remove unecessary error* properly handle ReturnTypeInvalid in ast.zig functions* assert that the tree is clean in render.zig* simplify parser recovery with top level decls 2020-05-15 11:35:37 +03:00
Vexu
c77fee0344
fix infinite loop
findToken wasn't as generic as I thought it was
2020-05-14 19:56:55 +03:00
Vexu
c4552ee8ed
store rparen in ReturnType.Invalid
This is useful for getting a partial function signature
2020-05-14 17:18:14 +03:00
Vexu
a32e240540
improve recovery from invalid container members
Instead of trying to find the end of the block or the next comma/semicolon
we no try to find the next token that can start a container member.
2020-05-14 12:09:40 +03:00
Vexu
89f2923a8a
recover from missing semicolon 2020-05-14 11:19:50 +03:00
Vexu
ac319b2734
remove useless nosuspend parsing
nosuspend cannot be used in a type expression and
all other use casesare covered by PrimaryExpr
2020-05-14 11:19:14 +03:00
Vexu
c3b76d0913
recover from invalid builtin/async call 2020-05-14 00:16:56 +03:00