Andrew Kelley 3f680abbe2 stage2: tokenizer: require null terminated source
By requiring the source file to be null-terminated, we avoid extra
branching while simplifying the logic at the same time.

Running ast-check on a large zig source file (udivmodti4_test.zig),
master branch compared to this commit:
 * 4% faster wall clock
 * 7% fewer cache misses
 * 1% fewer branches
2021-07-02 13:27:35 -07:00
..
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-05-08 15:15:30 +03:00
2021-06-17 16:37:38 -04:00
2020-12-31 15:45:24 -08:00
2020-12-31 15:45:24 -08:00
2021-06-21 17:03:03 -07:00
2021-06-16 14:46:25 -04:00
2021-05-08 15:15:30 +03:00
2020-12-31 15:45:24 -08:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2020-12-31 15:45:24 -08:00
2021-05-08 15:15:30 +03:00
2021-06-27 12:54:31 +03:00
2020-12-31 15:45:24 -08:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-10 20:13:43 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-21 17:03:03 -07:00
2021-06-30 18:39:04 -07:00
2021-05-08 15:15:30 +03:00
2021-05-08 15:15:30 +03:00