adrien/ZipponDB
1
0
Fork 0
Code Actions Releases Activity

Good docs update, stillo need some work. Need to take a step back and think

Browse Source
This commit is contained in:
Adrien Bouvais 2025-01-14 22:48:50 +01:00
parent 9ae9b65f1e
commit e848be94c1
23 changed files with 722 additions and 522 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
README.md
View File

@ -86,51 +86,6 @@ Here, we update the first 5 `User` entities named 'bob' to capitalize the name a
UPDATE User [5] {name='bob'} TO (name = 'Bob')
```
## Vs SQL
```sql
SELECT * FROM User
```
```
GRAB User
```
```sql
SELECT *
FROM your_table_name
WHERE name = 'Bob'
AND (age > 30 OR age < 10);
```
```
GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}
```
```sql
SELECT u1.name AS user_name, GROUP_CONCAT(u2.name || ' (' || u2.age || ')') AS friends_list
FROM User u1
LEFT JOIN User u2 ON ',' || u1.friends || ',' LIKE '%,' || u2.id || ',%'
WHERE u1.age > 30
GROUP BY u1.name;
```
```
GRAB User [name, friends [name, age]] {age > 30}
```
```sql
SELECT u1.name AS user_name, GROUP_CONCAT(u2.name || ' (' || u2.age || ')') AS friends_list
FROM User u1
LEFT JOIN User u2 ON ',' || u1.friends || ',' LIKE '%,' || u2.id || ',%'
WHERE u2.age > 30
GROUP BY u1.name;
```
```
GRAB User [name, friends [name, age] {age > 30}]
```
## Link query - Not yet implemented
You can also link query. Each query returns a list of UUID of a specific struct. You can use it in the next query.

4
docs/Benchmark.md
View File

@ -1,10 +1,10 @@
# Benchmark
***Benchmark are set to quicly evolve. I have currently multiple ideas to improve perf***
***Benchmark are set to quicly evolve. I have currently multiple ideas to improve performance.***
## Command
You can run `zig build benchmark` if you clone the repo to benchmark your machine.
You can run `zig build benchmark` if you clone the repo to benchmark your machine. [More info.](/ZipponDB/build)
Here an example on my machine:

35
docs/Quickstart.md
View File

@ -6,34 +6,23 @@ This guide will help you set up and start using ZipponDB quickly.
Obtain a binary for your architecture by:
- Building from source code (tutorial coming soon)
- Downloading a pre-built binary from the releases page (coming soon)
- Downloading a pre-built binary from the [releases page](https://github.com/MrBounty/ZipponDB/releases)
- Building from [source code](https://mrbounty.github.io/ZipponDB/build)
## Step 2: Create a Database
Once with the binary, run it to get access to the CLI.
Run the binary to start the Command Line Interface. Create a new database by running:
## Step 2: Select a Database
``` bash
db new path/to/directory
```
This will create a new ZipponDB directory. Verify the creation by running:
``` bash
db metrics
```
## Step 3: Select a Database
Select a database by running:
Once in the CLI, create a database by running:
```bash
db use path/to/ZipponDB
db use path/to/dir
```
Alternatively, set the `ZIPPONDB_PATH` environment variable to the path of a valid ZipponDB directory (containing DATA, BACKUP, and LOG directories).
Alternatively, set the `ZIPPONDB_PATH` environment variable.
## Step 4: Attach a Schema
## Step 3: Attach a Schema
Define a schema (see the next section for details) and attach it to the database by running:
Define a [schema](/ZipponDB/Schema) and attach it to the database by running:
```bash
schema init path/to/schema.txt
@ -45,7 +34,9 @@ This will create the necessary directories and empty files for data storage. Tes
schema describe
```
## Step 5: Use the Database
Alternatively, set the `ZIPPONDB_SCHEMA` environment variable.
## Step 4: Use the Database
Start using the database by sending queries, such as:
@ -53,4 +44,4 @@ Start using the database by sending queries, such as:
run "ADD User (name = 'Bob')"
```
You're now ready to explore the features of ZipponDB!
[Learn more about ZiQL.](/ZipponDB/ziql/intro)

18
docs/Schema.md
View File

@ -42,10 +42,26 @@ Comment (
*Note: The [] symbol preceding a type indicates an array of that type. For example, []User represents an array of User structs.*
## Schema Migration (Coming Soon)
## Schema Migration - Not yet implemented
In future releases, ZipponDB will support schema updates, allowing you to modify existing structs or add new ones, and then apply these changes to your database. Currently, schema modifications are not possible once the database has been initialized.
I will add commands or query to add, delete and update struct in the schema. Maybe the schema will be a Struct itself with a schema like that:
```
Struct (
name: str,
members: []Member,
)
Member (
name: str,
type: int,
)
```
Like that can just do ZiQL qquery directly on it.
### Planned Migration Features
- Add new members to existing structs

16
docs/Single_file.md
View File

@ -1,15 +1,15 @@
# Single file
TODO: In the future I will migrate into a single file database like SQLite.
In the future I will migrate into a single file database like SQLite.
## FileBlok
A blok is like memory but for file. It has a starting point and a size and can hold entities or FileVar.
Everything I will save will be linked into a Blok and I can create and delete them at will. The size is in the config and can't change.
In SQLite, it is what they call it a page I think. Similare thing.
## FileVar
A FileVar is a single value but saved in a file. This have a Blok, a position, a size and a type.
For string, I will create something different I think, with a size max and a len, and can request more if needed.
## FileBlok
A blok is like memory but for file. It has a starting point and a size and can hold entities or FileVar.
Everything I will save will be linked into a Blok and I can create and delete it at will. The size it given in the config and can't change.
In SQLite, it is what they call a page I think. Similare thing.

298
docs/ZiQL.md
View File

@ -1,298 +0,0 @@
# ZipponQL
TODO: Cut this into multiple .md
ZipponDB uses its own query language, ZipponQL or ZiQL for short. Here are the key points to remember:
- 4 actions available: `GRAB`, `ADD`, `UPDATE`, `DELETE`
- All queries start with an action followed by a struct name
- `{}` are filters
- `[]` specify how much and what data
- `()` contain new or updated data (not already in files)
- `||` are additional options
- By default, all members that are not links are returned
***Disclaimer: A lot of features are still missing, and the language may change over time.***
## Making errors
When you make an error writing ZiQL, you should see something like this to help you understand where you made a mistake:
```lua
Error: Expected string
GRAB User {name = Bob}
^^^
```
```
Error: Expected ( or member name.
GRAB User {name = 'Bob' AND {age > 10}}
^
```
## Filters
What is between `{}` are filters. You can see it as a list of condition. This filter is use when parsing files and evaluate every struct one by one and return `true`
or `false`.
For example `{ name = 'Bob' }` will return `true` if the member `name` of the evaluated struct is equal to `Bob`. This is the most important thing in ZipponDB.
You can see it as `WHERE` in SQL.
## Link query - Not yet implemented
You can also link query. Each query returns a list of UUID of a specific struct. You can use it in the next query.
Here an example where I create a new `Comment` that I then append to the list of comment of one specific `User`.
```python
ADD Comment (content='Hello world', at=NOW, like_by=[]) => added_comment => UPDATE User {id = '000'} TO (comments APPEND added_comment)
```
The name between `=>` is the variable name of the list of UUID used for the next queries, you can have multiple one if the link has more than 2 queries.
You can also just use one `=>` but the list of UUID is discarded in that case.
This can be use with GRAB too. So you can create variable before making the query. Here an example:
```python
GRAB User {name = 'Bob'} => bobs =>
GRAB User {age > 18} => adults =>
GRAB User {IN adults AND !IN bobs}
```
Which is the same as:
```js
GRAB User {name != 'Bob' AND age > 18}
```
## Examples
### GRAB
The main action is `GRAB`, this will parse files and return data.
#### Basic
Here's how to return all `User` entities without any filtering:
```python
GRAB User
```
To get all `User` entities above 30 years old:
```python
GRAB User {age > 30}
```
To return only the `name` member of `User` entities:
```python
GRAB User [name] {age > 30}
```
To return the 10 first `User` entities:
```lua
GRAB User [10] {age > 30}
```
You can combine these options:
```lua
GRAB User [10; name] {age > 30}
```
Use multiple conditions:
```lua
GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}
```
GRAB queries return a list of JSON objects with the data inside, e.g:
```
[{id:"1e170a80-84c9-429a-be25-ab4657894653", name: "Gwendolyn Ray", age: 70, email: "austin92@example.org", scores: [ 77 ], friends: [], }, ]
```
#### Ordering
**Not yet implemented**
To order the results by `name`:
```js
GRAB User [10; name] {age > 10} |ASC name|
```
You can specify how much data to return and which members to include, even for links inside structs. In this example, I get 1 friend's name for 10 `User` entities:
```js
GRAB User [10; friends [1; name]]
```
#### Array
You can use the `IN` operator to check if something is in an array:
```js
GRAB User { age > 10 AND name IN ['Adrien' 'Bob']}
```
TODO: More
#### Relationship
2 main things to remember with relationship:
- You can use the dot `.` to refer to a relationship.
- You can use filter inside filter.
Get `User` that have a best friend named Adrien:
```js
GRAB User { bestfriend IN { name = 'Adrien' } }
```
---
When using `IN`, it return all `User` that have AT LEAST one friend named Adrien:
```js
GRAB User { friends IN { name = 'Adrien' } }
```
---
To get `User` entities with all friends named Adrien:
```js
GRAB User { friends ALLIN { name = 'Adrien' } }
```
---
You can use `IN` on itself. Here I get all `User` entities that liked a `Comment` from 2024. Both queries return the same result:
```js
GRAB User { IN Comment {at > '2024/01/01'}.like_by}
GRAB Comment.like_by { at > '2024/01/01'}
```
---
You can also return a relationship only. The filter will be applied to `User` entities, but will return `Comment` entities:
```js
GRAB User.comments {name = 'Bob'}
```
---
You can do it as much as you like. This will return all `User` that liked comments from Bob:
```js
GRAB User.comments.like_by {name = 'Bob'}
```
---
This can also be used inside filters. Note that we need to specify `User` because it is a different struct than `Post`. Here, I get all `Post` entities that have a comment from Bob:
```js
GRAB Post {comments IN User{name = 'Bob'}.comments}
```
---
You can also do the same but only for the first Bob found:
```js
GRAB Post {comments IN User [1] {name = 'Bob'}.comments}
```
---
Be careful; this will return all `User` that liked a comment from 10 `User` named Bob:
```js
GRAB User.comments.like_by [10] {name = 'Bob'}
```
---
To get 10 `User` that liked a comment from any `User` named Bob, you need to use:
```js
GRAB User.comments.like_by [comments [like_by [10]]] {name = 'Bob'}
```
#### !
You can use `!` to return the opposite. When used with `IN`, it checks if something is NOT in the list. When used with filters, it returns entities that do not match the filter.
This will return all `User` entities that didn't like a `Comment` in 2024:
```js
GRAB User { !IN Comment {at > '2024/01/01'}.like_by}
```
Be careful because this does not return the same thing as above; it returns all `User` entities that liked a `Comment` not in 2024:
```js
GRAB Comment.like_by !{ at > '2024/01/01'}
```
Which is the same as:
```js
GRAB Comment.like_by { at < '2024/01/01'}
```
### ADD
The `ADD` action adds one entity to the database. The syntax is similar to `GRAB`, but uses `()`. This signifies that the data is not yet in the database.
Here's an example:
```lua
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82])
```
You need to specify all members when adding an entity (default values in roadmap).
The `ADD` query will return a list of added IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
**Not yet implemented**
And you can also add them in batch
```js
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82]) (name = 'Bob2', age = 33, email = 'bob2@email.com', scores = [])
```
You don't need to specify the members in the second entity as long as the order is respected:
```js
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82]) ('Bob2', 33, 'bob2@email.com', [])
```
### DELETE
Similar to `GRAB` but deletes all entities found using the filter and returns a list of deleted UUIDs.
```js
DELETE User {name = 'Bob'}
```
The `DELETE` query will return a list of deleted IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
### UPDATE
A mix of `GRAB` and `ADD`. It takes a filter first, then the new data.
Here, we update the first 5 `User` entities named 'adrien' to capitalize the name and become 'Adrien':
```js
UPDATE User [5] {name='adrien'} TO (name = 'Adrien')
```
Note that, compared to `ADD`, you don't need to specify all members between `()`. Only the ones specified will be updated.
The `UPDATE` query will return a list of updated IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
**Not yet implemented**
You can use operations on values themselves when updating:
```js
UPDATE User {name = 'Bob'} TO (age += 1)
```
You can also manipulate arrays, like adding or removing values:
```js
UPDATE User {name='Bob'} TO (scores APPEND 45)
UPDATE User {name='Bob'} TO (scores APPEND [45 99])
UPDATE User {name='Bob'} TO (scores REMOVEAT [0 1 2])
```
Currently, there will be four keywords for manipulating lists:
- `APPEND`: Adds a value to the end of the list.
- `REMOVE`: Checks the list, and if the same value is found, deletes it.
- `REMOVEAT`: Deletes the value at a specific index.
- `CLEAR`: Removes all values from the array.
Except for `CLEAR`, which takes no value, each keyword can use one value or an array of values. If you choose an array, it will perform the operation on all values in the array.
For relationships, you can use filters:
```js
UPDATE User {name='Bob'} TO (comments APPEND {id = '000'})
UPDATE User {name='Bob'} TO (comments REMOVE { at < '2023/12/31'})
```
I may include more options later.

109
docs/ZipponData.md
View File

@ -95,7 +95,7 @@ pub fn main() !void {
***Note: The dir can be null and it will use cwd.***
# Array
## Array
All data type have an array equivalent. To write an array, you need to first encode it using `allocEncodArray` before writing it.
This use an allocator so you need to free what it return.
@ -155,22 +155,10 @@ pub fn main() !void {
}
```
# Benchmark
## Benchmark
Done on a AMD Ryzen 7 7800X3D with a Samsung SSD 980 PRO 2TB (up to 7,000/5,100MB/s for read/write speed) on one thread.
| Rows | Write Time (ms) | Average Write Time (μs) | Read Time (ms) | Average Read Time (μs) | File Size (kB) |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.01 | 13.63 | 0.025 | 25.0 | 0.04 |
| 10 | 0.01 | 1.69 | 0.03 | 3.28 | 0.4 |
| 100 | 0.04 | 0.49 | 0.07 | 0.67 | 4.0 |
| 1_000 | 0.36 | 0.36 | 0.48 | 0.48 | 40 |
| 10_000 | 3.42 | 0.34 | 4.67 | 0.47 | 400 |
| 100_000 | 36.39 | 0.36 | 48.00 | 0.49 | 4_000 |
| 1_000_000 | 361.41 | 0.36 | 481.00 | 0.48 | 40_000 |
TODO: Update number to use Unix one. Benchmark on my laptop and maybe on some cloud VM.
Data use:
```zig
const schema = &[_]DType{
@ -192,87 +180,16 @@ const data = &[_]Data{
};
```
***Note: You can check Benchmark.md in ZipponDB to see performance using multi-threading. Was able to parse 1_000_000 users in less than 100ms***
Result:
# Importing the package
Create a `build.zig.zon` next to `build.zig` if not already done.
Add this dependencies in `build.zig.zon`:
```zig
.ZipponData = .{
.url = "git+https://github.com/MrBounty/ZipponData",
//the correct hash will be suggested by zig},
```
Here what my complete `build.zig.zon` is for my project ZipponDB:
```zig
.{
.name = "ZipponDB",
.version = "0.1.4",
.dependencies = .{
.ZipponData = .{
.url = "git+https://github.com/MrBounty/ZipponData",
//the correct hash will be suggested by zig},
},
.paths = .{
"",
},
}
```
And in `build.zig` you can import the module like this:
```zig
const zid = b.dependency("ZipponData", .{});
exe.root_module.addImport("ZipponData", zid.module("ZipponData"));
```
And you can now import it like std in your project:
```zig
const zid = @import("ZipponData");
zid.createFile("Hello.zid", null);
```
# What you can't do
You can't update files. You gonna need to implement that yourself. The easier way (and only I know), is to parse the entire file and write it into another.
Here an example that evaluate all struct using a `Filter` and write only struct that are false. (A filter can be like `age > 20`, if the member `age` of the struct is `> 20`, it is true):
```zig
pub fn delete(file_name: []const u8, dir: std.fs.Dir, filter: Filter) !void {
// 1. Create the iterator of the current file
var iter = try zid.DataIterator.init(self.allocator, file_name, dir, sstruct.zid_schema);
defer iter.deinit();
// 2. Create a new file
const new_path_buff = try std.fmt.allocPrint(self.allocator, "{s}.new", .{file_name});
defer self.allocator.free(new_path_buff);
try zid.createFile(new_path_buff, dir);
// 3. Create a writer of the new data
var new_writer = try zid.DataWriter.init(new_path_buff, dir);
defer new_writer.deinit();
// 4. For all struct, evaluate and write to new file if false
while (try iter.next()) |row| {
if (!filter.evaluate(row)) {
try new_writer.write(row);
}
}
// 5. Flush, delete old file and rename new file to previous file
try new_writer.flush();
try dir.deleteFile(path_buff);
try dir.rename(new_path_buff, path_buff);
}
```
# Potential update
I don't plan do update this but it will depend if my other project need it.
- Functions to update files
- Add a header with the data type at the beginning of the file so no need to make a schema and I can check everytime I write if it's in the good format
- More type
- Multi threading
| Number of Entity | Total Write Time (ms) | Average Write Time / entity (μs) | Total Read Time (ms) | Average Read Time / entity (μs) | File Size (kB) |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.01 | 13.63 | 0.025 | 25.0 | 0.04 |
| 10 | 0.01 | 1.69 | 0.03 | 3.28 | 0.4 |
| 100 | 0.04 | 0.49 | 0.07 | 0.67 | 4.0 |
| 1_000 | 0.36 | 0.36 | 0.48 | 0.48 | 40 |
| 10_000 | 3.42 | 0.34 | 4.67 | 0.47 | 400 |
| 100_000 | 36.39 | 0.36 | 48.00 | 0.49 | 4_000 |
| 1_000_000 | 361.41 | 0.36 | 481.00 | 0.48 | 40_000 |
***Note: You can check [benchmark](/ZipponDB/Benchmark) to see performance of the real database using multi-threading. Was able to parse 1_000_000 users in less than 100ms***

33
docs/about.md
View File

@ -0,0 +1,33 @@
# About
## Me
ZipponDB is developped by a single person, [Adrien Bouvais](https://github.com/MrBounty).
I started this project because I wanted to learn a lot of things:
* Database
* Zig
* CI/CD
* Data structures
* Algorithms
* Documentation
* Memory management
* High performance
* Low level programming
* ...
### Why
I wanted to create a "real" project, not just some TODO app. But something original with actually everything.
That's why I added automatic documentation update when push in main. Or automatic build and release when I push a new tag. And why I make it look like a proper open-source project.
### How
I came into this project with 0 real knowledge of how a database work and a small week-end of coding zig. I didn't read that much about how other people make databases.
My goal was more to learn by mistake, doing it myself. I spended a lot of time at the beguining planing how the database will work inside, before coding.
For the how itself, I solved one small problem at the time. 'How to open a file ?', 'How to read and write bytes ?', 'How to do a tokenizer ?', etc... And litle by litle,
you go closer and closer to your objectif.
I learned a lot of thing with this project. And even if I would do some part differently if I do it again, I think it is a fairly decent piece of software.

60
docs/build.md Normal file
View File

@ -0,0 +1,60 @@
# Build
On this page, I will show you how to build ZipponDB from source.
## 1. Get Zig
First thing first, <a href="https://ziglang.org/" target="_blank">go get zig</a>.
## 2. Clone repo
Simple enough, clone ZipponDB repository and cd into it.
```bash
git clone https://github.com/MrBounty/ZipponDB
cd ZipponDB
```
## 3. Config
In `lib/config.zig` you will find a config file. There is few parameters and they are comptime for now (can't change from cli). But more will be added.
Parameter | Default | Description
----- | ----- | ---------------
MAX_FILE_SIZE | 1Mb | Max size of each individual file where data is store.
CPU_CORE | 16 | Number of thread the pool will use. (At least 4 recommended for db > 100Mb)
## 4. build
```
zig build
```
Create 2 binaries in `zig-out/bin`:
* **zippondb:** The database CLI.
* **benchmark:** Run and print a benchmark.
### build run
```
zig build run
```
Build and run the CLI.
### build benchmark
```
zig build benchmark
```
Build and run the benchmark.
### build test
```
zig build test
```
Build and run tests.

56
docs/cli.md
View File

@ -8,9 +8,11 @@ Run a ZiQL query on the selected database.
**Usage:**
```go
run "QUERY" // (1)!
```
run QUERY
```
1. Note that query need to be between ""
## db
@ -21,15 +23,9 @@ Print some metrics from the db, including: Size on disk and number of entities s
**Usage:**
```
db metrics [OPTIONS]
db metrics
```
**Options:**
Name | Type | Description | Default
---- | ---- | ------------------- | ----
TODO | TODO | TODO | TODO
### db new
Create a new empty directory that can be then initialize with a schema.
@ -37,15 +33,9 @@ Create a new empty directory that can be then initialize with a schema.
**Usage:**
```
db new path/to/dir [OPTIONS]
db new path/to/dir
```
**Options:**
Name | Type | Description | Default
---- | ---- | ------------------- | ----
TODO | TODO | TODO | TODO
### db use
Select an already created database with `db new`.
@ -53,15 +43,9 @@ Select an already created database with `db new`.
**Usage:**
```
db use path/to/dir [OPTIONS]
db use path/to/dir
```
**Options:**
Name | Type | Description | Default
---- | ---- | ------------------- | ----
TODO | TODO | TODO | TODO
### db state
Return the state of the database, either `Ok` or `MissingDatabase` if no database selected or `MissingSchema` if no schema was initialize.
@ -74,22 +58,16 @@ db state
## schema
### schema init
### schema use
Initialize the database using a schema file.
Attach a schema to the database using a schema file.
**Usage:**
```
schema use path/to/schema.file [OPTIONS]
schema use path/to/schema.file
```
**Options:**
Name | Type | Description | Default
---- | ---- | ------------------- | ----
TODO | TODO | TODO | TODO
### schema describe
Print the schema use by the selected database.
@ -97,9 +75,21 @@ Print the schema use by the selected database.
**Usage:**
```
schema use path/to/schema.file [OPTIONS]
schema describe
```
## dump
Export the entier database in a specific format.
**Usage:**
```
dump [FORMAT] [PATH]
```
FORMAT options: `csv`, `json`, `zid`
## quit
Quit the CLI.

3
docs/features.md
View File

@ -1,3 +0,0 @@
# Features
TODO

56
docs/index.md
View File

@ -1,22 +1,54 @@
# ZipponDB: A Lightweight Relational Database in Zig
ZipponDB is a relational database built from the ground up in Zig, with zero external dependencies. Designed for simplicity, performance, and portability, it's ideal for small to medium applications that prioritize reliability and ease of use over complex features.
<style>
.md-content .md-typeset h1 { display: none; }
</style>
<p align="center">
<a href="/ZipponDB"><img src="images/banner.png" alt="ZipponDB"></a>
</p>
<p align="center">
<em>Relational Database written in Zig</em>
</p>
---
**Documentation**: <a href="/ZipponDB" target="_blank">https://mrbounty.github.io/ZipponDB</a>
**Source Code**: <a href="https://github.com/MrBounty/ZipponDB" target="_blank">https://github.com/MrBounty/ZipponDB</a>
---
ZipponDB is a relational database built from the ground up in Zig, with zero external dependencies. Designed for simplicity,
performance, and portability, it's almost usable for small to
medium applications that want a quick and simple database.
## Key Features
- **Relational Model:** ZipponDB is being developed to support easy relationship.
- **Simple Query Language:** A straightforward and minimalist query language.
- **Lightweight and Fast:** ZipponDB's small footprint and efficient design ensure quick performance.
- **Portable:** Built with Zig, ZipponDB can be easily compiled and deployed across various platforms.
* **Small:** ZipponDB binary is small, around 2-3Mb.
* **Fast:** ZipponDB can parse millions of entities in milliseconds.
* **Relationship:** ZipponDB is build with focus on easy relationship.
* **Query Language:** ZipponDB use it's own simple query language named ZipponQL.
* **No dependencies:** ZipponDB depend on nothing, every line of code is in the codebase.
* **Open-source:** ZipponDB is open-source under MIT licence.
* **Portable:** ZipponDB can be easily compiled and deployed across various platforms.*
## Why Choose ZipponDB?
<small>* Plan for more platforms like arm, 32 bit system.</small>
If you need a database that is:
### To come
- **Easy to integrate:** ZipponDB's minimal design and simple query language make integration into your projects straightforward.
- **Resource-efficient:** Its lightweight nature minimizes resource consumption, making it suitable for resource-constrained environments.
- **Reliable:** Built with a focus on correctness and stability.
- **Cross-platform:** Deployable wherever Zig is supported.
* **Auth:** Be able to auth users, maybe third-party OAuth.
* **HTTP server:** Be able to start a simple HTTP server and send json.
* **Interface:** Small package to interact with ZipponDB from different programming language.
* **Single file:** Like SQLite, ZipponDB's database aim to be a single file.
* **Web interface:** Similare to EdgeDB, getting a webapp to query and config the DB.
Then ZipponDB might be the right choice for you.
<small>More info in the Roadmap.</small>
### Maybe
Those are idea for very long term with 0 promesse.
* **Per user database:** Like Turso.
* **Performace:** I'm sure I can do better.
* **Client side database:** Run it on the client side with WASM.

35
docs/interface.md Normal file
View File

@ -0,0 +1,35 @@
# Interface
Interfaces are way to use ZipponDB from other programming language.
## Python
**Not yet implemented, to give a general idea. Exact code may change.**
```python
from zippondb import Client
db = Client("data")
db.run("ADD User (name='Bob')")
```
### Pydantic
I will most likely implement with Pydantic. Something like that:
```python
from zippondb import Client, Model
class User(Model):
name: str
age: int
db = Client("data")
users = db.run("GRAB User {}", model=User)
```
`Model` is like `BaseModel` from pydantic but all member are optional. If no model provided, return a list of dict.
## Golang
TODO

3
docs/logs.md Normal file
View File

@ -0,0 +1,3 @@
# Log
I have logging implemented but badly and I don't log enough. I need to check this part.

2
docs/release.md
View File

@ -7,3 +7,5 @@ First release.
Mostly to test the release pipeline with Github actions.
Only include CLI, most ZiQL features here but still missing some part.
Mostly missing thing around the database to make it usable.
[Link](https://github.com/MrBounty/ZipponDB/releases/tag/0.1.8)

29
docs/ziql/add.md Normal file
View File

@ -0,0 +1,29 @@
# ADD
The `ADD` action adds entities to the database. The syntax is similar to `GRAB`, but uses `()`. This signifies that the data is not yet in the database.
Here's an example:
```lua
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82])
```
You need to specify all members when adding an entity (default values comming).
The `ADD` query will return a list of added IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
And you can also add them in batch
```
ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82]) (name = 'Bob2', age = 33, email = 'bob2@email.com', scores = [])
```
You don't need to specify member's name for the second entity as long as the order is respected:
```
ADD User
(name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82])
('Bob2', 33, 'bob2@email.com', [])
```

11
docs/ziql/delete.md Normal file
View File

@ -0,0 +1,11 @@
# DELETE
Similar to `GRAB` but deletes all entities found using the filter and returns a list of deleted UUIDs.
```js
DELETE User {name = 'Bob'}
```
The `DELETE` query will return a list of deleted IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```

120
docs/ziql/grab.md Normal file
View File

@ -0,0 +1,120 @@
# GRAB
The main action is `GRAB`, this will parse files and return data.
#### Basic
Here's how to return all `User` entities without any filtering:
```
GRAB User
```
---
To get all `User` entities above 30 years old:
```
GRAB User {age > 30}
```
---
To return only the `name` member of `User` entities:
```
GRAB User [name] {age > 30}
```
---
To return the 10 first `User` entities:
```
GRAB User [10] {age > 30}
```
---
You can combine these options:
```
GRAB User [10; name] {age > 30}
```
---
Use multiple conditions:
```
GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}
```
---
GRAB queries return a list of JSON objects with the data inside, e.g:
```
[{id:"1e170a80-84c9-429a-be25-ab4657894653", name: "Gwendolyn Ray", age: 70, email: "austin92@example.org", scores: [ 77 ], friends: [], }, ]
```
---
#### Ordering - Not yet implemented
To order the results by `name`:
```js
GRAB User [10; name] {age > 10} |ASC name|
```
#### Array
You can use the `IN` operator to check if something is in an array:
```js
GRAB User { age > 10 AND name IN ['Adrien' 'Bob']}
```
---
#### Relationship
2 main things to remember with relationship:
* You can use filter inside filter.
* You can use the dot `.` to refer to a relationship. (Not yet implemented)
Get `User` that have a best friend named Adrien:
```js
GRAB User { bestfriend IN { name = 'Adrien' } }
```
---
You can specify how much data to return and which members to include, even for links inside entity. In this example, I get 1 friend's name for 10 `User`:
```js
GRAB User [10; friends [1; name]]
```
---
When using `IN`, it return all `User` that have AT LEAST one friend named Adrien:
```js
GRAB User { friends IN { name = 'Adrien' } }
```
---
To get `User` entities with all friends named Adrien:
```js
GRAB User { friends ALLIN { name = 'Adrien' } }
```
---
You can use `!` to say not in:
```js
GRAB User { friends !IN { name = 'Adrien' } }
```
---
#### Dot - Not yet implemented
You can use `.` if you just want to do one comparison. Here I get all `User` that ordered at least one book:
```js
GRAB User { orders.products.category.name = 'Book' }
```
Same as:
```
GRAB User {orders IN { products IN { category IN { name = 'Book'} } } }
```

144
docs/ziql/intro.md Normal file
View File

@ -0,0 +1,144 @@
# ZipponQL
ZipponDB uses its own query language, ZipponQL or ZiQL for short.
The language goal is to be minimal, small and simple.
Yet allowing powerful relationship.
Here are the key points to remember:
* 4 actions available: `GRAB`, `ADD`, `UPDATE`, `DELETE`
* All queries start with an action followed by a struct name
* `{}` are filters
* `[]` specify how much and what data
* `()` contain new or updated data (not already in files)
* `||` for ordering
* By default, all members that are not links are returned
***Disclaimer: The language may change a bit over time.***
## Making errors
When you make an error writing ZiQL, you should see something like this to help you understand where you made a mistake:
```lua
Error: Expected string
GRAB User {name = Bob}
^^^
```
```
Error: Expected ( or member name.
GRAB User {name = 'Bob' AND {age > 10}}
^
```
## To return
What is between `[]` are what data to return. You can see it as the column name after `SELECT` in SQL.
Here I return just the name of all users:
```
GRAB User [name] {}
```
Here the 100 first users:
```
GRAB User [100] {}
```
Here the name of the 100 first users:
```
GRAB User [100; name]
```
### For relationship
You can also specify what data to return for each relationship returned. By default, query do not return any relationship.
This will return the name and best friend of all users:
```
GRAB User [name, best_friend] {}
```
You can also specify what the best friend return:
```
GRAB User [name, best_friend [name, age]] {}
```
## Filters
What is between `{}` are filters, basically as a list of condition. This filter is use when parsing files and evaluate entities. You can see it as `WHERE` in SQL.
For example `{ name = 'Bob' }` will return `true` if the member `name` of the evaluated entity is equal to `Bob`. This is the most important thing in ZipponDB.
Here an example in a query:
```
GRAB User {name = 'Bob' AND age > 44}
```
### For relationship
Filter can be use inside filter. This allow simple yet powerfull relationship.
This query will return all users that have a best friend named 'Bob'.
```
GRAB User {best_friend IN {name = 'Bob'}}
```
You are obviously not limited to one depth. This will return all users that ordered at least one book in 2024:
```go
GRAB User {
orders IN {
products IN {
category IN {
name = 'Book'
}
},
date > 2024/01/01
}
}
```
Same as:
```go
GRAB User {orders IN { products.category.name = 'Book' AND date > 2024/01/01} } // (1)!
```
1. Dot not yet implemented
## Link query - Not yet implemented
You can also link query. Each query returns a list of UUID of a specific struct. You can use it in the next query.
Here an example where I create a new `Comment` that I then append to the list of comment of one specific `User`.
```
ADD Comment (content='Hello world', at=NOW, like_by=[], by={id='000'})
=> added_comment =>
UPDATE User {id = '000'} TO (comments APPEND added_comment)
```
The name between `=>` is the variable name of the list of UUID used for the next queries, you can have multiple one if the link has more than 2 queries.
You can also just use one `=>` but the list of UUID is discarded in that case.
This can be use with GRAB too. So you can create variable before making the query. Here an example:
```
GRAB User {name = 'Bob'} => bobs =>
GRAB User {age > 18} => adults =>
GRAB User {IN adults AND !IN bobs}
```
Which is the same as:
```
GRAB User {name != 'Bob' AND age > 18}
```
Another example:
```
GRAB Product [1] {category.name = 'Book'} => book =>
GRAB Order {date > 2024/01/01 AND products IN book} => book_orders =>
GRAB User [100] {orders IN book_orders}
```

45
docs/ziql/update.md Normal file
View File

@ -0,0 +1,45 @@
# UPDATE
A mix of `GRAB` and `ADD`. It takes a filter first, then the new data.
Here, we update the first 5 `User` entities named 'adrien' to capitalize the name and become 'Adrien':
```js
UPDATE User [5] {name='adrien'} TO (name = 'Adrien')
```
Note that, compared to `ADD`, you don't need to specify all members between `()`. Only the ones specified will be updated.
The `UPDATE` query will return a list of updated IDs, e.g.:
```
["1e170a80-84c9-429a-be25-ab4657894653", "1e170a80-84c9-429a-be25-ab4657894654", ]
```
## Not yet implemented
You can use operations on values themselves when updating:
```js
UPDATE User {name = 'Bob'} TO (age += 1)
```
You can also manipulate arrays, like adding or removing values:
```js
UPDATE User {name='Bob'} TO (scores APPEND 45)
UPDATE User {name='Bob'} TO (scores APPEND [45 99])
UPDATE User {name='Bob'} TO (scores REMOVEAT [0 1 2])
```
Currently, there will be four keywords for manipulating lists:
- `APPEND`: Adds a value to the end of the list.
- `REMOVE`: Checks the list, and if the same value is found, deletes it.
- `REMOVEAT`: Deletes the value at a specific index.
- `CLEAR`: Removes all values from the array.
Except for `CLEAR`, which takes no value, each keyword can use one value or an array of values. If you choose an array, it will perform the operation on all values in the array.
For relationships, you can use filters:
```js
UPDATE User {name='Bob'} TO (comments APPEND {id = '000'})
UPDATE User {name='Bob'} TO (comments REMOVE [1] { at < '2023/12/31'})
```
I may include more options later.

106
docs/ziql/vssql.md Normal file
View File

@ -0,0 +1,106 @@
# Vs SQL
A good way to see how ZipponQl work is to compare it to SQL.
## Select everything
```
SELECT * FROM User
```
```
GRAB User
or
GRAB User {}
```
## Selection on condition
```
SELECT *
FROM Users
WHERE name = 'Bob'
AND (age > 30 OR age < 10);
```
```go
GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}
```
## Select something
```
SELECT name, age
FROM Users
LIMIT 100
```
```go
GRAB User [100; name, age] {}
```
## Relationship
### List of other entity
```
SELECT u1.name AS user_name, GROUP_CONCAT(u2.name || ' (' || u2.age || ')') AS friends_list
FROM Users u1
LEFT JOIN User u2 ON ',' || u1.friends || ',' LIKE '%,' || u2.id || ',%'
WHERE u1.age > 30
GROUP BY u1.name;
```
```go
GRAB User [name, friends [name, age]] {age > 30}
```
### Join
#### Simple one
SQL:
```
SELECT Users.name, Orders.orderID, Orders.orderDate
FROM Users
INNER JOIN Orders ON Users.UsersID = Orders.CustomerID;
```
ZiQL:
```go
GRAB User [name, order [id, date]] {}
```
#### More complexe one
SQL:
```
SELECT
U.name AS UserName,
O.orderID,
O.orderDate,
P.productName,
C.categoryName,
OD.quantity,
FROM
Users U
INNER JOIN Orders O ON U.UserID = O.UserID
INNER JOIN OrderDetails OD ON O.OrderID = OD.OrderID
INNER JOIN Products P ON OD.ProductID = P.ProductID
INNER JOIN Categories C ON P.CategoryID = C.CategoryID
WHERE
O.orderDate >= '2023-01-01'
AND C.categoryName != 'Accessories'
ORDER BY
O.orderDate DESC;
```
ZiQL
```go
GRAB User
[ name, orders [id, date, details [quantity]], product [name], category [name] ]
{ orders IN {date >= 2023/01/01} AND category IN {name != 'Accessories'} }
| orders.date DESC | // (1)!
```
1. Ordering not yet implemented

13
mkdocs.yml
View File

@ -49,17 +49,26 @@ extra_css:
nav:
- Home: index.md
- Features: features.md
- Learn:
- Quickstart: Quickstart.md
- Schema: Schema.md
- ZipponQL: ZiQL.md
- ZipponQL:
- Intro: ziql/intro.md
- Vs SQL: ziql/vssql.md
- GRAB: ziql/grab.md
- ADD: ziql/add.md
- UPDATE: ziql/update.md
- DELETE: ziql/delete.md
- Data types: Data type.md
- Command Line Interface: cli.md
- Logs: logs.md
- Interface: interface.md
- Technical:
- Benchmark: Benchmark.md
- Technical: Technical docs.md
- Build: build.md
- ZipponData: ZipponData.md
- Single file: Single_file.md
- About: about.md
- Roadmap: Roadmap.md
- Release: release.md

3
src/file/utils.zig
View File

@ -28,6 +28,9 @@ pub fn writeDbMetrics(self: *Self, buffer: *std.ArrayList(u8)) ZipponError!void
const main_size = getDirTotalSize(main_dir) catch 0;
writer.print("Total size: {d:.2}Mb\n", .{@as(f64, @floatFromInt(main_size)) / 1024.0 / 1024.0}) catch return ZipponError.WriteError;
writer.print("CPU core: {d}\n", .{config.CPU_CORE}) catch return ZipponError.WriteError;
writer.print("Max file size: {d:.2}Mb\n", .{@as(f64, @floatFromInt(config.MAX_FILE_SIZE)) / 1024.0 / 1024.0}) catch return ZipponError.WriteError;
const log_dir = main_dir.openDir("LOG", .{ .iterate = true }) catch return ZipponError.CantOpenDir;
const log_size = getDirTotalSize(log_dir) catch 0;
writer.print("LOG: {d:.2}Mb\n", .{@as(f64, @floatFromInt(log_size)) / 1024.0 / 1024.0}) catch return ZipponError.WriteError;