adrien/ZipponDB
1
0
Fork 0
Code Actions Releases Activity

Moved stuffs to new dir

Browse Source
This commit is contained in:
adrien
2024-11-09 16:41:28 +01:00
parent 9fa6f25459
commit 18c55194d5
8 changed files with 252 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Benchmark.md
+67
View File
@@ -0,0 +1,67 @@
***Benchmark are set to quicly evolve. I have currently multiple ideas to improve perf***
# Intro
In this example I create a random dataset of 50 000 000 Users using this shema:
```lua
User (
name: str,
age: int,
email: str,
bday: date,
last_order: datetime,
a_time: time,
scores: []int,
friends: []str,
)
```
Here a user example:
```
run "ADD User (name = 'Diana Lopez',age = 2,email = 'allisonwilliams@example.org',scores=[37 85 90 71 88 85 68],friends = [],bday=1973/11/13,last_order=1979/07/18-15:05:26.590261,a_time=03:04:06.862213)
```
First let's do a query that parse all file but dont return anything, so we have the time to read and evaluate file but not writting and sending output.
```
run "GRAB User {name = 'asdfqwer'}"
```
## 50 000 000 Users
This take 6.24GB space on disk, seperated into xx files of xx MB
| Thread | Time (s) | Usage (%) |
| --- | --- | --- |
| 1 | 40 | 10 |
| 2 | 21 | 18 |
| 3 | 15 | 25 |
| 4 | 12 | 30 |
| 6 | 8.3 | 45 |
| 8 | 6.6 | 55 |
| 12 | 5.1 | 85 |
| 16 | 4.3 | 100 |
![alt text](https://github.com/MrBounty/ZipponDB/blob/v0.1.4/charts/time_usage_per_thread_50_000_000.png)
## 1 000 000
This take 127MB space on disk, sperated into 24 files of 5.2MB
| Thread | Time (ms) |
| --- | --- |
| 1 | 790 |
| 2 | 446 |
| 3 | 326 |
| 4 | 255 |
| 6 | 195 |
| 8 | 155 |
| 12 | 136 |
| 16 | 116 |
![alt text](https://github.com/MrBounty/ZipponDB/blob/v0.1.4/charts/time_usage_per_thread_1_000_000.png)
## TODO
- [ ] Benchmark per files size, to find the optimal one. For 10kB, 5MB, 100MB, 1GB
- [ ] Create a build command to benchmark. For 1_000, 1_000_000, 50_000_000 users
- [ ] Create a random dataset
- [ ] Do simple query, get average and +- time by set of 25 query
- [ ] Return the data to do a chart
docs/TODO v0.2.md
+37
View File
@@ -0,0 +1,37 @@
- [ ] Delete the .new file if an error happend
- [ ] Create a struct that manage the schema
Relationships
- [X] Update the schema Parser and Tokenizer
- [X] Include the name of the link struct with the schema_struct
- [ ] New ConditionValue that is an array of UUID
- [ ] When relationship found in filter, check if the type is right and exist
- [ ] When parseFilter, get list of UUID as value for relationship
- [ ] Add new operation in Filter evalue: IN and !IN
- [ ] parseNewData can use filter like in "Add User (friends = [10] {age > 20})" to return UUID
- [ ] parseFilter can use sub filter. "GRAB User {friends IN {age > 20}}" At least one friend in a list of UUID
- [ ] When send, send the entities in link specify between []
Optimizations
- [X] Parse file one time for all conditions, not once per condition
- [X] parse file in parallel, multi threading
- [X] GRAB
- [X] DELETE
- [X] UPDATE
- [ ] Radix Tries ofr UUID list
ADD User (name='Bob', age = 44, best_friend = {id=0000-0000}) => new_user => UPDATE User {id = 0000-0000} TO (best_friend = new_user)
Happy to annonce the v0.2 of my database. New feature include:
- Relationship
- Huge performance increase with multi threading
- Date, time and datetime type
- Linked query
- Compressed binary files
- Logs
All core features of the query language is working, v0.3 will focus on adding things around ot, including:
- Schema migration
- Dump/Bump data
- Recovery
- Better CLI
docs/Technical docs.md
+121
View File
@@ -0,0 +1,121 @@
# Intro
TODO
***Note: Code snipped do not necessary represent the actual codebase but are use to explain principle.***
# Tokenizers
All `Tokenizer` work similary and are based on the [zig tokenizer.](https://github.com/ziglang/zig/blob/master/lib/std/zig/tokenizer.zig)
The `Tokenizer` role is to take a buffer string and convert it into a list of `Token`. A token have an enum `Tag` that represent what the token is, for example `=` is the tag `equal`, and a `Loc` with a `start` and `end` usize that represent the emplacement in the buffer.
The `Tokenizer` itself have 2 methods: `next` that return the next `Token`. And `TODO` that return the slice of the buffer that represent the `Token`, using it's `Loc`.
This is how to use it:
```zig
const toker = Tokenizer.init(buff);
const token = toker.next();
std.debug.print("{s}", .{toker.xxxx(token)});
```
I usually use a `Tokenizer` in a loop until the `Tag` is `end`. And in each loop I take the next token and will use a switch on the `Tag` to do stuffs.
Here a simple example:
```zig
const toker = Tokenizer.init(buff);
var token = toker.next();
while (token.tag != .end) : (token = toker.next()) switch (token.tag) {
.equal => std.debug.print("{s}", .{toker.xxxx(token)}),
else => {},
}
```
### All tokenizers
There is 4 differents tokenizer in ZipponDB, I know, that a lot. Here the list:
- **ZiQL:** Tokenizer for the query language.
- **cli:** Tokenizer the commands.
- **schema:** Tokenizer for the schema file.
- **data:** Tokenizer for csv file.
They all have different `Tag` and way to parse the array of bytes but overall are very similar. The only noticable difference is that some use a null terminated string (based on the zig tokenizer) and other not.
Mostly because I need to use dupeZ to get a new null terminated array, not necessary.
# Parser
`Parser` are the next step after the tokenizer. Its role is to take `Token` and do stuff or raise error. There is 3 `Parser`, the main one is for ZiQL, one for the schema and one for the cli.
Note that the cli one is just the `main` function in `main.zig` and not it's own struct but overall do the same thing.
A `Parser` have a `State` and a `Tokenizer` as member and have a `parse` method. Similary to `Tokenizer`, it will enter a while loop. This loop will continue until the `State` is `end`.
Let's take as example the schema parser that need to parse this file:
```
User (name: str)
```
When I run the `parse` method, it will init the `State` as `start`. When in `start`, I check if the `Token` is a identifier (a variable name), if it is one I add it to the list of struct in the current schema, if not I raise an error pointing to this token.
Here the idea for a `parse` method:
```zig
var state = .start;
var token = self.toker.next();
while (state != .end) : (token = self.toker.next()) switch (state) {
.start => switch (token.tag) {
.identifier => self.addStruct(token),
else => printError("Error: Expected a struct name.", token),
},
else => {},
}
```
The issue here is obviously that we are in an infinite loop that just going to add struct or print error. I need to change the `state` based on the combinaison of the current `state` and `token.tag`. For that I usually use very implicite name for `State`.
For example in this situation, after a struct name, I expect `(` so I will call it something like `expect_l_paren`. Here the idea:
```zig
var state = .start;
var token = self.toker.next();
while (state != .end) : (token = self.toker.next()) switch (state) {
.start => switch (token.tag) {
.identifier => {
self.addStruct(token);
state = .expect_l_parent;
},
else => printError("Error: Expected a struct name.", token),
},
.expect_l_parent => switch (token.tag) {
.l_paren => {},
else => printError("Error: Expected (.", token),
},
else => {},
}
```
And that's basicly it, the entire `Parser` work like that. It is fairly easy to debug as I can print the `state` and `token.tag` at each iteration and follow the path of the `Parser`.
Note that the `ZiQLParser` use different methods for parsing:
- **parse:** The main one that will then use the other.
- **parseFilter:** This will create a `Filter`, this is a tree that contain all condition in the query, what is between `{}`.
- **parseCondition:** Create a `Condition` based on a part of what is between `{}`. E.g. `name = 'Bob'`.
- **parseAdditionalData:** Populate the `AdditionalData` that represent what is between `[]`.
- **parseNewData:** Return a string map with key as member name and value as value of what is between `()`. E.g. `(name = 'Bob')` will return a map with one key `name` with the value `Bob`.
- **parseOption:** Not done yet. Parse what is between `||`
# FileEngine
The `FileEngine` is that is managing files, everything that need to read or write into files is here.
I am not goind into too much detail here as I think this will change in the future.
# Multi-threading
How do I do multi-threading ? Basically all struct are saved in multiples `.zid` files. Each files have
a size limit defined in the config and a new one is created when no previous one is found with space left.
When I run a GRAB query and parse all files and evaluate each struct, I use a thread pool and give a file
to each thread. Each thread have it's own buffered writer and once all finished, I concatenate all writer
and send it.
The only atomic value share by all threads are the number of founded struct (to stop thread if enough are found when
[10] is use). And the number of finished thread, so I know when I can concatenate and send stuffs.
Like that this keep things simple and easy to implement. I dont have parallel thread that run different
that need to access the same file.
docs/ZiQL.md
+298
View File
@@ -0,0 +1,298 @@
# ZipponQL
ZipponDB uses its own query language, ZipponQL or ZiQL for short. Here are the key points to remember:
- 4 actions available: `GRAB`, `ADD`, `UPDATE`, `DELETE`
- All queries start with an action followed by a struct name
- `{}` are filters
- `[]` specify how much and what data
- `()` contain new or updated data (not already in the file)
- `||` are additional options
- By default, all members that are not links are returned
- To return links or only some members, specify them between []
***Disclaimer: A lot of features are still missing, and the language may change over time.***
# Making errors
When you make an error writing ZiQL, you should see something like this to help you understand where you made a mistake:
```
Error: Expected string
GRAB User {name = Bob}
^^^
```
```
Error: Expected ( or member name.
GRAB User {name = 'Bob' AND {age > 10}}
^
```
# Data types
There is 5 data types for the moment:
- `int`: 64 bit integer
- `float`: 64 bit float. Need to have a dot, `1.` is a float `1` is an integer.
- `bool`: Boolean, can be `true` or `false`
- `string`: Character array between `''`
- `UUID`: Id in the UUID format, used for relationship, ect. All struct have an id member.
- `date`: A date in yyyy/mm/dd
- `time`: A time in hh:mm:ss.mmmm
- `datetime`: A date time in yyyy/mm/dd-hh:mm:ss:mmmm
All data types can be an array of those types using `[]` in front of it. So `[]int` is an array of integer.
All data types can also be `null`. Except arrays that can only be empty.
# Examples
## GRAB
The main action is `GRAB`, this will parse files and return data.
Here's how to return all `User` entities without any filtering:
```js
GRAB User
```
To get all `User` entities above 18 years old:
```js
GRAB User {age > 18}
```
To return only the `name` member of `User` entities:
```js
GRAB User [name] {age > 18}
```
To return the 10 first `User` entities:
```js
GRAB User [10] {age > 18}
```
You can combine these options:
```js
GRAB User [10; name] {age > 18}
```
Use multiple conditions:
```js
GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}
```
GRAB queries return a list of JSON objects with the data inside, e.g:
```
[{id:"1e170a80-84c9-429a-be25-ab4657894653", name: "Gwendolyn Ray", age: 70, email: "austin92@example.org", scores: [ 77 ], friends: [], }, ]
```
#### Not yet implemented
To order the results by `name`:
```js
GRAB User [10; name] {age > 10} |ASC name|
```
You can specify how much data to return and which members to include, even for links inside structs. In this example, I get 1 friend's name for 10 `User` entities:
```js
GRAB User [10; friends [1; name]]
```
##### Using IN
You can use the `IN` operator to check if something is in an array:
```js
GRAB User { age > 10 AND name IN ['Adrien' 'Bob']}
```
This also works by using other filters. Here I get `User` entities that have a best friend named Adrien:
```js
GRAB User { bestfriend IN { name = 'Adrien' } }
```
When using an array with `IN`, it will return all User entities that have at least one friend named Adrien:
```js
GRAB User { friends IN { name = 'Adrien' } }
```
To get `User` entities with all friends named Adrien:
```js
GRAB User { friends ALLIN { name = 'Adrien' } }
```
You can use `IN` on itself. Here I get all `User` entities that liked a `Comment` from 2024. Both queries return the same result:
```js
GRAB User { IN Comment {at > '2024/01/01'}.like_by}
GRAB Comment.like_by { at > '2024/01/01'}
```
You can obtain a similar result with this query, but it will return a list of `Comment` entities with a `liked_by` member that is similar to the `User` entities above. If you take all `liked_by` members inside all `Comment` entities, it will be the same list, but you can end up with duplicates since one `User` can like multiple `Comment` entities.
```js
GRAB Comment [liked_by] {at > '2024/01/01'}
```
##### Return relationship
You can also return a relationship only. The filter will be applied to `User` entities, but will return `Comment` entities:
```js
GRAB User.comments {name = 'Bob'}
```
You can do it as much as you like. This will return all `User` that liked comments from Bob:
```js
GRAB User.comments.like_by {name = 'Bob'}
```
This can also be used inside filters. Note that we need to specify `User` because it is a different struct than `Post`. Here, I get all `Post` entities that have a comment from Bob:
```js
GRAB Post {comments IN User{name = 'Bob'}.comments}
```
You can also do the same but only for the first Bob found:
```js
GRAB Post {comments IN User [1] {name = 'Bob'}.comments}
```
Be careful; this will return all `User` entities that liked a comment from 10 `User` entities named Bob:
```js
GRAB User.comments.like_by [10] {name = 'Bob'}
```
To get 10 `User` entities that liked a comment from any `User` entity named Bob, you need to use:
```js
GRAB User.comments.like_by [comments [like_by [10]]] {name = 'Bob'}
```
##### Using !
You can use `!` to return the opposite. When used with `IN`, it checks if something is NOT in the list. When used with filters, it returns entities that do not match the filter.
This will return all `User` entities that didn't like a `Comment` in 2024:
```js
GRAB User { !IN Comment {at > '2024/01/01'}.like_by}
```
Be careful because this does not return the same thing as above; it returns all `User` entities that liked a `Comment` not in 2024:
```js
GRAB Comment.like_by !{ at > '2024/01/01'}
```
Which is the same as:
```js
GRAB Comment.like_by { at < '2024/01/01'}
```
## ADD
The `ADD` action adds one entity to the database. The syntax is similar to `GRAB`, but uses `()`. This signifies that the data is not yet in the database.
Here's an example:
```js
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82])
```
You need to specify all members when adding an entity (default values are coming soon).
#### Not yet implemented
The `ADD` query will return a list of added IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
And you can also add them in batch
```js
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82]) (name = 'Bob2', age = 33, email = 'bob2@email.com', scores = [])
```
You don't need to specify the members in the second entity as long as the order is respected:
```js
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82]) ('Bob2', 33, 'bob2@email.com', [])
```
## DELETE
Similar to `GRAB` but deletes all entities found using the filter and returns a list of deleted UUIDs.
```js
DELETE User {name = 'Bob'}
```
#### Not yet implemented
The `DELETE` query will return a list of deleted IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
## UPDATE
A mix of `GRAB` and `ADD`. It takes a filter first, then the new data.
Here, we update the first 5 `User` entities named 'adrien' to capitalize the name and become 'Adrien':
```js
UPDATE User [5] {name='adrien'} TO (name = 'Adrien')
```
Note that, compared to `ADD`, you don't need to specify all members between `()`. Only the ones specified will be updated.
#### Not yet implemented
The `UPDATE` query will return a list of updated IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
You can use operations on values themselves when updating:
```js
UPDATE User {name = 'Bob'} TO (age += 1)
```
You can also manipulate arrays, like adding or removing values:
```js
UPDATE User {name='Bob'} TO (scores APPEND 45)
UPDATE User {name='Bob'} TO (scores APPEND [45 99])
UPDATE User {name='Bob'} TO (scores REMOVEAT [0 1 2])
```
Currently, there will be four keywords for manipulating lists:
- `APPEND`: Adds a value to the end of the list.
- `REMOVE`: Checks the list, and if the same value is found, deletes it.
- `REMOVEAT`: Deletes the value at a specific index.
- `CLEAR`: Removes all values from the array.
Except for `CLEAR`, which takes no value, each keyword can use one value or an array of values. If you choose an array, it will perform the operation on all values in the array.
For relationships, you can use filters:
```js
UPDATE User {name='Bob'} TO (comments APPEND {id = '000'})
UPDATE User {name='Bob'} TO (comments REMOVE { at < '2023/12/31'})
```
I may include more options later.
# Date
***WIP***
## Date
To write a date, you use this format: `yyyy/mm/dd`.
Like that: `2024/10/19`.
***Note: You cant use negative years***
## Time
To write a time, you use this format: `hh:mm:ss.mmmmm`.
Like that: `12:45:00.0000`.
Millisecond and second are optional so this work too: `12:45:00` and `12:45`
## Datetime
Mix of both, to write a datetime, you use this format: `yyyy/mm/dd-hh:mm:ss.mmmmm`.
Like that: `2024/10/19-12:45:00.0000`.
Millisecond and second are optional so this work too: `2024/10/19-12:45:00` and `2024/10/19-12:45`
GRAB User {birthday > 2020/10/19 AND birthday < 2024/01/01 }
GRAB IOT {id = '0000-0000' AND .TemperatureSensor.Temperature.timestamp > 22-45-50.0000}
GRAB IOT {.TemperatureSensor.TemperatureRecord IN TemperatureRecord{.timestamp > 22-45-50.0000 AND version = 3}}
GRAB IOT {.TemperatureSensor.TemperatureRecord IN .{timestamp > 22-45-50.0000} AND .TemperatureSensor IN .{version = 3}}