1 line
58 KiB
JSON
1 line
58 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"ZipponDB: A Minimalist Lightweight General Purpose Database in Zig","text":"<p> Minimalist Lightweight General Purpose Database written in Zig </p> <p>Documentation: https://mrbounty.github.io/ZipponDB</p> <p>Source Code: https://github.com/MrBounty/ZipponDB</p> <p>ZipponDB is a database built from the ground up in Zig, with zero external dependencies. Designed for simplicity, performance, and portability, it's almost usable for small to medium applications that want a quick and simple database.</p>"},{"location":"#key-features","title":"Key Features","text":"<ul> <li>Small Binary: ~300kb.</li> <li>Fast: Parse millions of entities in milliseconds.*</li> <li>Relationship: Build with focus on easy relationship.</li> <li>Query Language: Use it's own stupid query language.</li> <li>No dependencies: Depend on nothing, every line of code running is in the codebase and written for ZipponDB.</li> <li>Open-source: Open-source under MIT licence.</li> </ul> <p>* Check benchmark.</p>"},{"location":"#planned","title":"Planned","text":"<ul> <li>Interface: Small package to interact with ZipponDB from different programming language.</li> <li>Single file: Like SQLite, ZipponDB's database aim to be a single file.</li> <li>Schema migration: Update dynamically database schema.</li> <li>Custom index: Speed up query with custom indexing.</li> <li>Better QL: Improving ZiQL and adding features.</li> <li>Safety and Performance: Improve general safty and performance.</li> </ul> <p>More info in the Roadmap.</p>"},{"location":"#maybe","title":"Maybe","text":"<p>Those are idea for very long term with 0 promesse, maybe as extension.</p> <ul> <li>Web interface: Similare to EdgeDB, getting a webapp to query and config the DB.</li> <li>HTTP server: Be able to start a simple HTTP server and send json.</li> <li>Auth: Be able to auth users, maybe third-party OAuth.</li> <li>Per user database: Like Turso.</li> <li>Performace: I'm sure I can do better.</li> <li>Client side database: Run it on the client side with WASM.</li> </ul>"},{"location":"Benchmark/","title":"Benchmark","text":"<p>Benchmark are set to evolve. I have currently multiple ideas to improve performance.</p> <p>ZipponDB is fairly fast and can easely query millions of entities. Current limitation is around 5GB I would say, depending of CPU cores, usage and kind of data saved. After that query can start to become slow as optimizations are still missing.</p> <p>Most of query's time is writing entities into a JSON format. Parsing the file itself take little time. For example in the benchmark report bellow, I parse 100 000 users in around 40ms if there is no entities to send and 130ms if all 100 000 entities are to send.</p> <p>I choosed to release ZipponDB binary with the small release. Zig has a fast and safe release, but the fast release isn't that much faster in my case, if not at all. If you want you can build it with it.</p>"},{"location":"Benchmark/#command","title":"Command","text":"<p>You can run <code>zig build benchmark</code>, if you clone the repo to benchmark your machine. More info on how to build from source.</p> <p>Here an example on my machine with 16 core:</p> <pre><code>=====================================\n\nPopulating with 5000 users.\nPopulate duration: 0.035698 seconds\n\nDatabase path: benchmarkDB\nTotal size: 0.36Mb\nCPU core: 16\nMax file size: 5.00Mb\nLOG: 0.02Mb\nBACKUP: 0.00Mb\nDATA: 0.33Mb\n Item: 0.00Mb | 19 entities | 1 files\n User: 0.33Mb | 5000 entities | 1 files\n Order: 0.00Mb | 0 entities | 1 files\n Category: 0.00Mb | 4 entities | 1 files\n\n--------------------------------------\n\nQuery: GRAB User {}\nTime: 16.90 \u00b1 25.22 ms | Min 8.25ms | Max 92.55ms\n\nQuery: GRAB User {name='asd'}\nTime: 2.62 \u00b1 0.10 ms | Min 2.52ms | Max 2.85ms\n\nQuery: GRAB User [1] {}\nTime: 0.16 \u00b1 0.01 ms | Min 0.15ms | Max 0.18ms\n\nQuery: GRAB User [name] {}\nTime: 7.88 \u00b1 11.69 ms | Min 3.91ms | Max 42.94ms\n\nQuery: GRAB User {name = 'Charlie'}\nTime: 3.87 \u00b1 0.16 ms | Min 3.70ms | Max 4.17ms\n\nQuery: GRAB Category {}\nTime: 0.20 \u00b1 0.07 ms | Min 0.17ms | Max 0.41ms\n\nQuery: GRAB Item {}\nTime: 0.21 \u00b1 0.02 ms | Min 0.19ms | Max 0.25ms\n\nQuery: GRAB Order {}\nTime: 0.14 \u00b1 0.01 ms | Min 0.13ms | Max 0.18ms\n\nQuery: GRAB Order [from, items, quantity, at] {}\nTime: 4.18 \u00b1 12.01 ms | Min 0.15ms | Max 40.21ms\n\nQuery: DELETE User {}\nTime: 0.64 \u00b1 1.13 ms | Min 0.23ms | Max 4.04ms\n\nRead: 1907698 Entity/second *Include small condition\nWrite: 350200 Entity/second\n\n=====================================\n\nPopulating with 100000 users.\nPopulate duration: 0.707605 seconds\n\nDatabase path: benchmarkDB\nTotal size: 6.62Mb\nCPU core: 16\nMax file size: 5.00Mb\nLOG: 0.02Mb\nBACKUP: 0.00Mb\nDATA: 6.59Mb\n Item: 0.00Mb | 19 entities | 1 files\n User: 6.59Mb | 100000 entities | 2 files\n Order: 0.00Mb | 0 entities | 1 files\n Category: 0.00Mb | 4 entities | 1 files\n\n--------------------------------------\n\nQuery: GRAB User {}\nTime: 126.99 \u00b1 3.05 ms | Min 123.37ms | Max 133.56ms\n\nQuery: GRAB User {name='asd'}\nTime: 38.12 \u00b1 1.60 ms | Min 36.48ms | Max 41.88ms\n\nQuery: GRAB User [1] {}\nTime: 0.19 \u00b1 0.02 ms | Min 0.16ms | Max 0.22ms\n\nQuery: GRAB User [name] {}\nTime: 59.33 \u00b1 1.29 ms | Min 58.02ms | Max 61.47ms\n\nQuery: GRAB User {name = 'Charlie'}\nTime: 53.29 \u00b1 1.00 ms | Min 51.50ms | Max 54.78ms\n\nQuery: GRAB Category {}\nTime: 0.19 \u00b1 0.01 ms | Min 0.18ms | Max 0.22ms\n\nQuery: GRAB Item {}\nTime: 5.51 \u00b1 13.43 ms | Min 0.22ms | Max 45.22ms\n\nQuery: GRAB Order {}\nTime: 0.16 \u00b1 0.01 ms | Min 0.15ms | Max 0.18ms\n\nQuery: GRAB Order [from, items, quantity, at] {}\nTime: 0.17 \u00b1 0.02 ms | Min 0.15ms | Max 0.21ms\n\nQuery: DELETE User {}\nTime: 5.96 \u00b1 17.04 ms | Min 0.26ms | Max 57.07ms\n\nRead: 2623338 Entity/second *Include small condition\nWrite: 1125278 Entity/second\n\n=====================================\n\nPopulating with 1000000 users.\nPopulate duration: 7.029142 seconds\n\nDatabase path: benchmarkDB\nTotal size: 65.96Mb\nCPU core: 16\nMax file size: 5.00Mb\nLOG: 0.02Mb\nBACKUP: 0.00Mb\nDATA: 65.93Mb\n Item: 0.00Mb | 19 entities | 1 files\n User: 65.93Mb | 1000000 entities | 14 files\n Order: 0.00Mb | 0 entities | 1 files\n Category: 0.00Mb | 4 entities | 1 files\n\n--------------------------------------\n\nQuery: GRAB User {}\nTime: 250.77 \u00b1 6.74 ms | Min 247.08ms | Max 270.61ms\n\nQuery: GRAB User {name='asd'}\nTime: 67.90 \u00b1 0.42 ms | Min 67.31ms | Max 68.78ms\n\nQuery: GRAB User [1] {}\nTime: 8.92 \u00b1 24.86 ms | Min 0.55ms | Max 83.51ms\n\nQuery: GRAB User [name] {}\nTime: 110.08 \u00b1 5.27 ms | Min 106.86ms | Max 125.21ms\n\nQuery: GRAB User {name = 'Charlie'}\nTime: 73.65 \u00b1 2.79 ms | Min 69.24ms | Max 79.22ms\n\nQuery: GRAB Category {}\nTime: 0.19 \u00b1 0.04 ms | Min 0.16ms | Max 0.33ms\n\nQuery: GRAB Item {}\nTime: 0.21 \u00b1 0.02 ms | Min 0.19ms | Max 0.26ms\n\nQuery: GRAB Order {}\nTime: 0.15 \u00b1 0.01 ms | Min 0.14ms | Max 0.17ms\n\nQuery: GRAB Order [from, items, quantity, at] {}\nTime: 0.17 \u00b1 0.01 ms | Min 0.16ms | Max 0.18ms\n\nQuery: DELETE User {}\nTime: 11.74 \u00b1 34.19 ms | Min 0.29ms | Max 114.30ms\n\nRead: 14727354 Entity/second *Include small condition\nWrite: 5468517 Entity/second\n\n=====================================\n\nPopulating with 10000000 users.\nPopulate duration: 72.675680 seconds\n\nDatabase path: benchmarkDB\nTotal size: 659.33Mb\nCPU core: 16\nMax file size: 5.00Mb\nLOG: 0.02Mb\nBACKUP: 0.00Mb\nDATA: 659.30Mb\n Item: 0.00Mb | 19 entities | 1 files\n User: 659.30Mb | 10000000 entities | 132 files\n Order: 0.00Mb | 0 entities | 1 files\n Category: 0.00Mb | 4 entities | 1 files\n\n--------------------------------------\n\nQuery: GRAB User {}\nTime: 2535.29 \u00b1 86.92 ms | Min 2448.39ms | Max 2712.78ms\n\nQuery: GRAB User {name='asd'}\nTime: 684.75 \u00b1 39.96 ms | Min 649.09ms | Max 797.13ms\n\nQuery: GRAB User [1] {}\nTime: 6.65 \u00b1 1.00 ms | Min 5.36ms | Max 8.75ms\n\nQuery: GRAB User [name] {}\nTime: 1106.21 \u00b1 33.57 ms | Min 1056.57ms | Max 1172.61ms\n\nQuery: GRAB User {name = 'Charlie'}\nTime: 690.56 \u00b1 20.41 ms | Min 661.51ms | Max 718.07ms\n\nQuery: GRAB Category {}\nTime: 0.21 \u00b1 0.03 ms | Min 0.18ms | Max 0.31ms\n\nQuery: GRAB Item {}\nTime: 0.23 \u00b1 0.04 ms | Min 0.19ms | Max 0.32ms\n\nQuery: GRAB Order {}\nTime: 0.15 \u00b1 0.01 ms | Min 0.13ms | Max 0.17ms\n\nQuery: GRAB Order [from, items, quantity, at] {}\nTime: 0.17 \u00b1 0.02 ms | Min 0.15ms | Max 0.21ms\n\nQuery: DELETE User {}\nTime: 109.55 \u00b1 326.64ms | Min 0.47ms | Max 1089.46ms\n\nRead: 14603810 Entity/second *Include small condition\nWrite: 5403847 Entity/second\n=====================================\n</code></pre>"},{"location":"Data%20type/","title":"Data types","text":"<p>ZipponDB have a little set of types. This is on purpose, to keep the database simple and fast. But more type may be added in the future.</p>"},{"location":"Data%20type/#primary-data-types","title":"Primary Data Types","text":"<p>ZipponDB supports 8 primary data types:</p> Type Description Example int 32-bit integer 42 float 64-bit float (must include a decimal point) 3.14 bool Boolean value true or false string Character array enclosed in single quotes 'Hello, World!' UUID Universally Unique Identifier 123e4567-e89b-12d3-a456-426614174000 date Date in yyyy/mm/dd format 2024/10/19 time Time in hh:mm:ss.mmmm format 12:45:00.0000 datetime Combined date and time 2024/10/19-12:45:00.0000"},{"location":"Data%20type/#array-types","title":"Array Types","text":"<p>Any of these data types can be used as an array by prefixing it with <code>[]</code>. For example:</p> <ul> <li><code>[]int</code>: An array of integers</li> <li><code>[]string</code>: An array of strings</li> </ul>"},{"location":"Quickstart/","title":"Quickstart","text":"<p>This guide will help you set up and start using ZipponDB quickly.</p>"},{"location":"Quickstart/#step-1-get-a-binary","title":"Step 1: Get a Binary","text":"<p>Obtain a binary for your architecture by:</p> <ul> <li>Downloading a pre-built binary from the releases page</li> <li>Building from source code</li> </ul> <p>Once with the binary, run it to get access to the CLI.</p>"},{"location":"Quickstart/#step-2-select-a-database","title":"Step 2: Select a Database","text":"<p>Once in the CLI, create a database by running: <pre><code>db use path/to/dir\n</code></pre></p> <p>Alternatively, set the <code>ZIPPONDB_PATH</code> environment variable.</p>"},{"location":"Quickstart/#step-3-attach-a-schema","title":"Step 3: Attach a Schema","text":"<p>Define a schema and attach it to the database by running:</p> <pre><code>schema use path/to/schema.txt\n</code></pre> <p>This will create the necessary directories and empty files for data storage. Test the current database schema by running:</p> <pre><code>schema describe\n</code></pre> <p>Alternatively, set the <code>ZIPPONDB_SCHEMA</code> environment variable.</p>"},{"location":"Quickstart/#step-4-use-the-database","title":"Step 4: Use the Database","text":"<p>Start using the database by sending queries, such as:</p> <pre><code>run \"ADD User (name = 'Bob')\"\n</code></pre> <p>Learn more about ZiQL.</p>"},{"location":"Roadmap/","title":"Roadmap","text":"<p>Note: This will probably evolve over time.</p>"},{"location":"Roadmap/#alpha","title":"Alpha","text":""},{"location":"Roadmap/#v01-base","title":"v0.1 - Base","text":"<ul> <li>[X] UUID </li> <li>[X] CLI </li> <li>[X] Tokenizers </li> <li>[X] ZiQL parser</li> <li>[X] Schema engine </li> <li>[X] File engine </li> </ul>"},{"location":"Roadmap/#v02-usable","title":"v0.2 - Usable","text":"<ul> <li>[X] Relationships </li> <li>[X] Custom data file</li> <li>[X] Date</li> <li>[X] Logs</li> <li>[X] Query multi threading</li> <li>[X] Arrays manipulation</li> </ul>"},{"location":"Roadmap/#v03-qol","title":"v0.3 - QoL","text":"<ul> <li>[X] Docs website</li> <li>[ ] Ordering</li> <li>[ ] Dot operator</li> <li>[ ] Linked query </li> <li>[ ] Schema migration </li> <li>[ ] Dump/Bump data </li> <li>[ ] Recovery</li> <li>[ ] Better CLI</li> </ul>"},{"location":"Roadmap/#v04-usability","title":"v0.4 - Usability","text":"<ul> <li>[ ] Single file</li> <li>[ ] Docker </li> <li>[ ] YAML config file</li> <li>[ ] Python interface </li> <li>[ ] Go interface </li> </ul>"},{"location":"Roadmap/#v05-in-memory","title":"v0.5 - In memory","text":"<ul> <li>[ ] In memory option </li> <li>[ ] Cache</li> </ul>"},{"location":"Roadmap/#v06-performance","title":"v0.6 - Performance","text":"<ul> <li>[ ] Transaction </li> <li>[ ] Other multi threading</li> <li>[ ] Query optimization </li> <li>[ ] Index</li> </ul>"},{"location":"Roadmap/#v07-safety","title":"v0.7 - Safety","text":"<ul> <li>[ ] Metrics</li> <li>[ ] Durability</li> </ul>"},{"location":"Roadmap/#v08-docs","title":"v0.8 - Docs","text":"<ul> <li>[ ] ZiQL tuto </li> <li>[ ] Deployment tuto </li> <li>[ ] Code docs </li> <li>[ ] CLI help</li> </ul>"},{"location":"Schema/","title":"Schema","text":"<p>In ZipponDB, data is organized and manipulated using structures, or structs, rather than traditional tables. A struct is defined by a name, such as <code>User</code>, and members, such as <code>name</code> and <code>age</code>.</p>"},{"location":"Schema/#defining-a-schema","title":"Defining a Schema","text":"<p>To declare structs for use in your database, create a separate file containing the schema definitions. Below is an example of a simple schema file: <pre><code>User (\n name: str,\n email: str,\n best_friend: User,\n)\n</code></pre></p> <p>In this example, the <code>best_friend</code> member is a reference to another <code>User</code>.</p> <p>Here's a more complex example featuring multiple structs: <pre><code>User (\n name: str,\n email: str,\n friends: []User,\n posts: []Post,\n comments: []Comment,\n)\n\nPost (\n title: str,\n image: str,\n at: date,\n like_by: []User,\n comments: []Comment,\n)\n\nComment (\n content: str,\n at: date,\n like_by: []User,\n)\n</code></pre></p> <p>Note: The [] symbol preceding a type indicates an array of that type. For example, []User represents an array of User structs.</p>"},{"location":"Schema/#schema-migration-not-yet-implemented","title":"Schema Migration - Not yet implemented","text":"<p>In future releases, ZipponDB will support schema updates, allowing you to modify existing structs or add new ones, and then apply these changes to your database. Currently, schema modifications are not possible once the database has been initialized.</p> <p>I will add commands or query to add, delete and update struct in the schema. Maybe the schema will be a Struct itself with a schema like that:</p> <pre><code>Struct (\n name: str,\n members: []Member,\n)\n\nMember (\n name: str,\n type: int,\n)\n</code></pre> <p>Like that can just do ZiQL query directly on it.</p>"},{"location":"Schema/#planned-migration-features","title":"Planned Migration Features","text":"<ul> <li>Add new members to existing structs</li> <li>Modify or remove existing members</li> <li>Rename structs or members</li> <li>Update relationships between structs</li> <li>More...</li> </ul>"},{"location":"Single_file/","title":"Single file","text":"<p>In the future I will migrate into a single file database like SQLite.</p>"},{"location":"Single_file/#fileblok","title":"FileBlok","text":"<p>A blok is like memory but for file. It has a starting point and a size and can hold entities or FileVar. Everything I will save will be linked into a Blok and I can create and delete them at will. The size is in the config and can't change.</p> <p>In SQLite, it is what they call it a page I think. Similare thing.</p>"},{"location":"Single_file/#filevar","title":"FileVar","text":"<p>A FileVar is a single value but saved in a file. This have a Blok, a position, a size and a type. For string, I will create something different I think, with a size max and a len, and can request more if needed.</p>"},{"location":"TODO%20v0.2/","title":"TODO v0.2","text":"<ul> <li>[ ] Array manipulation</li> <li>[ ] Some time keyword like NOW</li> <li>[X] Benchmark command</li> </ul>"},{"location":"TODO%20v0.2/#run-in-wasm-for-a-demo","title":"Run in WASM for a demo","text":"<p>This could be fun, make a small demo where you get a wasm that run the database locally in the browser.</p>"},{"location":"Technical%20docs/","title":"Intro","text":"<p>TODO</p> <p>Note: Code snippets in this documentation are simplified examples and may not represent the actual codebase.</p>"},{"location":"Technical%20docs/#tokenizers","title":"Tokenizers","text":"<p>Tokenizers are responsible for converting a buffer string into a list of tokens. Each token has a <code>Tag</code> enum that represents its type, such as <code>equal</code> for the <code>=</code> symbol, and a <code>Loc</code> struct with start and end indices that represent its position in the buffer.</p> <p>All tokenizers work similarly and are based on the zig tokenizer. They have two main methods: next, which returns the next token, and getTokenSlice, which returns the slice of the buffer that represents the token.</p> <p>Here's an example of how to use a tokenizer: <pre><code>const toker = Tokenizer.init(buff);\nconst token = toker.next();\nstd.debug.print(\"{s}\", .{toker.getTokenSlice(token)});\n</code></pre></p> <p>Tokenizers are often used in a loop until the <code>end</code> tag is reached. In each iteration, the next token is retrieved and processed based on its tag. Here's a simple example: <pre><code>const toker = Tokenizer.init(buff);\nvar token = toker.next();\nwhile (token.tag != .end) : (token = toker.next()) switch (token.tag) {\n .equal => std.debug.print(\"{s}\", .{toker.getTokenSlice(token)}),\n else => {},\n}\n</code></pre></p>"},{"location":"Technical%20docs/#available-tokenizers","title":"Available Tokenizers","text":"<p>There are four different tokenizers in ZipponDB:</p> <ul> <li>ZiQL: Tokenizer for the query language.</li> <li>cli: Tokenizer the commands.</li> <li>schema: Tokenizer for the schema file.</li> </ul> <p>Each tokenizer has its own set of tags and parsing rules, but they all work similarly.</p>"},{"location":"Technical%20docs/#parser","title":"Parser","text":"<p>Parsers are the next step after tokenization. They take tokens and perform actions or raise errors. There are three parsers in ZipponDB: one for ZiQL, one for schema files, and one for CLI commands.</p> <p>A parser has a <code>State</code> enum and a <code>Tokenizer</code> instance as members, and a parse method that processes tokens until the <code>end</code> state is reached.</p> <p>Here's an example of how a parser works: <pre><code>var state = .start;\nvar token = self.toker.next();\nwhile (state != .end) : (token = self.toker.next()) switch (state) {\n .start => switch (token.tag) {\n .identifier => self.addStruct(token),\n else => printError(\"Error: Expected a struct name.\", token),\n },\n else => {},\n}\n</code></pre></p> <p>The parser's state is updated based on the combination of the current state and token tag. This process continues until the <code>end</code> state is reached.</p> <p>The ZiQL parser uses different methods for parsing:</p> <ul> <li><code>parse</code>: The main parsing method that calls other methods.</li> <li><code>parseFilter</code>: Creates a filter tree from the query.</li> <li><code>parseCondition</code>: Creates a condition from a part of the query.</li> <li><code>parseAdditionalData</code>: Populates additional data from the query.</li> <li><code>parseNewData</code>: Returns a string map with key-value pairs from the query.</li> <li><code>parseOption</code>: Not implemented yet.</li> </ul>"},{"location":"Technical%20docs/#file-parsing","title":"File parsing","text":"<p>File parsing is done through a small library that I did named ZipponData. </p> <p>It is minimal and fast, it can parse 1_000_000 entity in 0.3s on one thread on a 7 7800X3D at around 4.5GHz with a Samsung SSD 980 PRO 2TB (up to 7,000/5,100MB/s for read/write speed).</p> <p>To read a file, you create an iterator for a single file and then you can iterate with <code>.next()</code>. It will return an array of <code>Data</code>. This make everything very easy to use.</p> <pre><code>const std = @import(\"std\");\n\npub fn main() !void {\n const allocator = std.testing.allocator;\n\n // 0. Make a temporary directory\n try std.fs.cwd().makeDir(\"tmp\");\n const dir = try std.fs.cwd().openDir(\"tmp\", .{});\n\n // 1. Create a file\n try createFile(\"test\", dir);\n\n // 2. Create some Data\n const data = [_]Data{\n Data.initInt(1),\n Data.initFloat(3.14159),\n Data.initInt(-5),\n Data.initStr(\"Hello world\"),\n Data.initBool(true),\n Data.initUnix(2021),\n };\n\n // 3. Create a DataWriter\n var dwriter = try DataWriter.init(\"test\", dir);\n defer dwriter.deinit(); // This just close the file\n\n // 4. Write some data\n try dwriter.write(&data);\n try dwriter.write(&data);\n try dwriter.flush(); // Dont forget to flush !\n\n // 5. Create a schema\n // A schema is how the iterator will parse the file. \n // If you are wrong here, it will return wrong/random data\n // And most likely an error when iterating in the while loop\n const schema = &[_]DType{\n .Int,\n .Float,\n .Int,\n .Str,\n .Bool,\n .Unix,\n };\n\n // 6. Create a DataIterator\n var iter = try DataIterator.init(allocator, \"test\", dir, schema);\n defer iter.deinit();\n\n // 7. Iterate over data\n while (try iter.next()) |row| {\n std.debug.print(\"Row: {any}\\n\", .{ row });\n }\n\n // 8. Delete the file (Optional ofc)\n try deleteFile(\"test\", dir);\n try std.fs.cwd().deleteDir(\"tmp\");\n}\n</code></pre>"},{"location":"Technical%20docs/#engines","title":"Engines","text":"<p>ZipponDB segregate responsability with Engines.</p> <p>For example the <code>FileEngine</code> is the only place where files used, for both writting and reading. This simplify refactoring, testing, etc.</p>"},{"location":"Technical%20docs/#dbengine","title":"DBEngine","text":"<p>This is just a wrapper around all other Engines to keep them at the same place. This doesnt do anything except storing other Engines.</p> <p>This can be find in <code>main.zig</code>, in the <code>main</code> function.</p>"},{"location":"Technical%20docs/#fileengine","title":"FileEngine","text":"<p>The <code>FileEngine</code> is responsible for managing files, including reading and writing.</p> <p>Most methods will parse all files of a struct and evaluate them with a filter and do stuff if <code>true</code>. For example <code>parseEntities</code> will parse all entities and if the filter return <code>true</code>, will write using the writer argument a JSON object with the entity's data.</p> <p>Those methods are usually sperated into 2 methods. The main one and a <code>OneFile</code> version, e.g. <code>parseEntitiesOneFile</code>. The main one will call a thread for each file using multiple <code>OneFile</code> version. This is how multi-threading is done.</p>"},{"location":"Technical%20docs/#schemaengine","title":"SchemaEngine","text":"<p>The <code>SchemaEngine</code> manage everything related to schemas. </p> <p>This is mostly used to store a list of <code>SchemaStruct</code>, with is just one struct as defined in the schema. With all member names, data types, links, etc.</p> <p>This is also here that I store the <code>UUIDFileIndex</code>, that is a map of UUID to file index. So I can quickly check if a UUID exist and in which file it is store. This work well but use a bit too much memory for me, around 220MB for 1_000_000 entities. I tried doing a Radix Trie but it doesn't use that much less memory, maybe I did a mistake somewhere.</p>"},{"location":"Technical%20docs/#threadengine","title":"ThreadEngine","text":"<p>The <code>ThreadEngine</code> manage the thread pool of the database.</p> <p>This is also where is stored the <code>ThreadSyncContext</code> that is use for each <code>OneFile</code> version of parsing methods in the <code>FileEngine</code>. This is the only atomix value currently used in the database.</p>"},{"location":"Technical%20docs/#multi-threading","title":"Multi-threading","text":"<p>ZipponDB uses multi-threading to improve performance. Each struct is saved in multiple <code>.zid</code> files, and a thread pool is used to process files concurrently. Each thread has its own buffered writer, and the results are concatenated and sent once all threads finish.</p> <p>The only shared atomic values between threads are the number of found structs and the number of finished threads. This approach keeps things simple and easy to implement, avoiding parallel threads accessing the same file.</p>"},{"location":"Technical%20docs/#data-structures","title":"Data Structures","text":""},{"location":"Technical%20docs/#additionaldata","title":"AdditionalData","text":"<p>AdditionalData keep what is between <code>[]</code>. It is composed of 2 struct <code>AdditionalData</code> and <code>AdditionalDataMember</code>.</p> <p><code>AdditionalDataMember</code> have the name of the member, it's position in the schema file and an <code>AdditionalData</code>.</p> <p><code>AdditionalData</code> have a limit (the first part <code>[100]</code>), and a list of <code>AdditionalDataMember</code>.</p>"},{"location":"Technical%20docs/#filters","title":"Filters","text":"<p>A filter is a series of condition. It use a tree approach, by that I mean the filter have a root node that is either a condition or have 2 others nodes (left and right).</p> <p>For example the filter <code>{name = 'Bob'}</code> have one root node that is the condition. So when I evaluate the struct, I just check this condition.</p> <p>Now for like <code>{name = 'Bob' AND age > 0}</code>, the root node have as left node the condition <code>name = 'Bob'</code> and right node the condition <code>age > 0</code>.</p> <p>To look like that: <pre><code> AND\n / \\\n OR OR\n / \\ / \\\n name name age age\n ='A' ='B' >80 <20\n</code></pre></p>"},{"location":"Technical%20docs/#condition","title":"Condition","text":"<p>A condition is part of filters. It is one 'unit of condition'. For example <code>name = 'Bob'</code> is one condition. <code>name = 'Bob' and age > 0</code> are 2 conditions and one filter. It is created inside <code>parseCondition</code> in the <code>ziqlParser</code>.</p> <p>A condition have those infos:</p> <ul> <li>value: ConditionValue. E.g. <code>32</code></li> <li>operation: ComparisonOperator. E.g. <code>equal</code> or <code>in</code></li> <li>data_type: DataType. E.g. <code>int</code> or <code>str</code></li> <li>data_index: usize. This is the index in when parsing returned by zid <code>DataIterator</code></li> </ul>"},{"location":"Technical%20docs/#newdata","title":"NewData","text":"<p>NewData is a map with member name as key and ConditionValue as value, it is created when parsing and is use to add data into a file. I transform ConditionValue into Zid Data. Maybe I can directly do a map member name -> zid Data ?</p>"},{"location":"Technical%20docs/#relationmap","title":"RelationMap","text":"<p>A <code>RelationMap</code> is use when I need to return relationship. Let's say we have this query <code>GRAB User [orders [date]]</code>.</p> <p>The <code>RelationMap</code> have a struct_name, here Order. A member name, here orders. A map with UUID as key and a string as value.</p> <p>When I first init the map, I am parsing the first struct (here User). So it populate the map with empty string for entities that I want to return. Here it will be UUID of Order. </p> <p>Then I parse Order file and add the string to the right UUID, skipping UUID that are not in the map. Once that done, I parse the JSON response that I generated when parsing User. Where a relationship should be, there is <code>{<|[16]u8|>}</code>, where <code>[16]u8</code> is the UUID of the entity that shoud be here. So now I can just replace it by the right key in the map.</p>"},{"location":"Technical%20docs/#entitywriter","title":"EntityWriter","text":"<p>This is responsable to transform the raw Data into a JSON, Table or other output format to send send to end user. Basically the last step before sending.</p>"},{"location":"ZipponData/","title":"ZipponData","text":"<p>ZipponData is a library developped in the context of ZipponDB.</p> <p>The library intent to create a simple way to store and parse data from a file in the most efficient and fast way possible. </p> <p>There is 6 data type available in ZipponData:</p> Type Zig type Bytes in file int i32 4 float f64 8 bool bool 1 str []u8 4 + len uuid [16]u8 16 unix u64 8 <p>Each type have its array equivalent.</p>"},{"location":"ZipponData/#quickstart","title":"Quickstart","text":"<ol> <li>Create a file with <code>createFile</code></li> <li>Create some <code>Data</code></li> <li>Create a <code>DataWriter</code></li> <li>Write the data</li> <li>Create a schema</li> <li>Create an iterator with <code>DataIterator</code></li> <li>Iterate over all value</li> <li>Delete the file with <code>deleteFile</code></li> </ol> <p>Here an example of how to use it: <pre><code>const std = @import(\"std\");\n\npub fn main() !void {\n const allocator = std.testing.allocator;\n\n // 0. Make a temporary directory\n try std.fs.cwd().makeDir(\"tmp\");\n const dir = try std.fs.cwd().openDir(\"tmp\", .{});\n\n // 1. Create a file\n try createFile(\"test\", dir);\n\n // 2. Create some Data\n const data = [_]Data{\n Data.initInt(1),\n Data.initFloat(3.14159),\n Data.initInt(-5),\n Data.initStr(\"Hello world\"),\n Data.initBool(true),\n Data.initUnix(2021),\n };\n\n // 3. Create a DataWriter\n var dwriter = try DataWriter.init(\"test\", dir);\n defer dwriter.deinit(); // This just close the file\n\n // 4. Write some data\n try dwriter.write(&data);\n try dwriter.write(&data);\n try dwriter.write(&data);\n try dwriter.write(&data);\n try dwriter.write(&data);\n try dwriter.write(&data);\n try dwriter.flush(); // Dont forget to flush !\n\n // 5. Create a schema\n // A schema is how the iterator will parse the file. \n // If you are wrong here, it will return wrong/random data\n // And most likely an error when iterating in the while loop\n const schema = &[_]DType{\n .Int,\n .Float,\n .Int,\n .Str,\n .Bool,\n .Unix,\n };\n\n // 6. Create a DataIterator\n var iter = try DataIterator.init(allocator, \"test\", dir, schema);\n defer iter.deinit();\n\n // 7. Iterate over data\n while (try iter.next()) |row| {\n std.debug.print(\"Row: {any}\\n\", .{ row });\n }\n\n // 8. Delete the file (Optional ofc)\n try deleteFile(\"test\", dir);\n try std.fs.cwd().deleteDir(\"tmp\");\n}\n</code></pre></p> <p>Note: The dir can be null and it will use cwd.</p>"},{"location":"ZipponData/#array","title":"Array","text":"<p>All data type have an array equivalent. To write an array, you need to first encode it using <code>allocEncodArray</code> before writing it. This use an allocator so you need to free what it return.</p> <p>When read, an array is just the raw bytes. To get the data itself, you need to create an <code>ArrayIterator</code>. Here an example:</p> <pre><code>pub fn main() !void {\n const allocator = std.testing.allocator;\n\n // 0. Make a temporary directory\n try std.fs.cwd().makeDir(\"array_tmp\");\n const dir = try std.fs.cwd().openDir(\"array_tmp\", .{});\n\n // 1. Create a file\n try createFile(\"test\", dir);\n\n // 2. Create and encode some Data\n const int_array = [4]i32{ 32, 11, 15, 99 };\n const data = [_]Data{\n Data.initIntArray(try allocEncodArray.Int(allocator, &int_array)), // Encode\n };\n defer allocator.free(data[0].IntArray); // DOnt forget to free it\n\n // 3. Create a DataWriter\n var dwriter = try DataWriter.init(\"test\", dir);\n defer dwriter.deinit();\n\n // 4. Write some data\n try dwriter.write(&data);\n try dwriter.flush();\n\n // 5. Create a schema\n const schema = &[_]DType{\n .IntArray,\n };\n\n // 6. Create a DataIterator\n var iter = try DataIterator.init(allocator, \"test\", dir, schema);\n defer iter.deinit();\n\n // 7. Iterate over data\n var i: usize = 0;\n if (try iter.next()) |row| {\n\n // 8. Iterate over array\n var array_iter = ArrayIterator.init(&row[0]); // Sub array iterator\n while (array_iter.next()) |d| {\n try std.testing.expectEqual(int_array[i], d.Int);\n i += 1;\n }\n\n }\n\n try deleteFile(\"test\", dir);\n try std.fs.cwd().deleteDir(\"array_tmp\");\n} \n</code></pre>"},{"location":"ZipponData/#benchmark","title":"Benchmark","text":"<p>Done on a AMD Ryzen 7 7800X3D with a Samsung SSD 980 PRO 2TB (up to 7,000/5,100MB/s for read/write speed) on one thread.</p> <p>Data use: <pre><code>const schema = &[_]DType{\n .Int,\n .Float,\n .Int,\n .Str,\n .Bool,\n .Unix,\n};\n\nconst data = &[_]Data{\n Data.initInt(1),\n Data.initFloat(3.14159),\n Data.initInt(-5),\n Data.initStr(\"Hello world\"),\n Data.initBool(true),\n Data.initUnix(2021),\n};\n</code></pre></p> <p>Result:</p> Number of Entity Total Write Time (ms) Average Write Time / entity (\u03bcs) Total Read Time (ms) Average Read Time / entity (\u03bcs) File Size (kB) 1 0.01 13.63 0.025 25.0 0.04 10 0.01 1.69 0.03 3.28 0.4 100 0.04 0.49 0.07 0.67 4.0 1_000 0.36 0.36 0.48 0.48 40 10_000 3.42 0.34 4.67 0.47 400 100_000 36.39 0.36 48.00 0.49 4_000 1_000_000 361.41 0.36 481.00 0.48 40_000 <p>Note: You can check benchmark to see performance of the real database using multi-threading. Was able to parse 1_000_000 users in less than 100ms</p>"},{"location":"about/","title":"About","text":""},{"location":"about/#me","title":"Me","text":"<p>ZipponDB is developped by a single person, Adrien Bouvais.</p> <p>I started this project because I wanted to learn a lot of things:</p> <ul> <li>Database</li> <li>Zig</li> <li>CI/CD</li> <li>Data structures</li> <li>Algorithms</li> <li>Documentation</li> <li>Memory management</li> <li>High performance</li> <li>Low level programming</li> <li>...</li> </ul>"},{"location":"about/#why","title":"Why","text":"<p>I wanted to create a \"real\" project, not just some TODO app. But something original with actually everything. That's why I added automatic documentation update when push in main. Or automatic build and release when I push a new tag. And why I make it look like a proper open-source project.</p> <p>I didn't research that much how other db do stuff. I wanted to just do it, figure it out and make silly mistake. That how you learn. I may take this knowledge and try to do more niche database, something maybe a bit more through upfront. Because here I just wrote the simpliest query I could think of that can do easy relationship. But let's be honest, that's kind of a mess. I more want to do like a vector db or tigerbeetle and there transaction database.</p> <p>Or just just simply a SQL db. I mean I don't think it would be that difficult based on what I already did.</p>"},{"location":"about/#how","title":"How","text":"<p>I came into this project with 0 real knowledge of how a database work and a small week-end of coding zig. I didn't read that much about how other people make databases. My goal was more to learn by mistake, doing it myself. I spended a lot of time at the beguining planing how the database will work inside, before coding.</p> <p>For the how itself, I solved one small problem at the time. 'How to open a file ?', 'How to read and write bytes ?', 'How to do a tokenizer ?', etc... And litle by litle, you go closer and closer to your objectif.</p> <p>I learned a lot of thing with this project. And even if I would do some part differently if I do it again, I think it is a fairly decent piece of software.</p>"},{"location":"build/","title":"Build","text":"<p>On this page, I will show you how to build ZipponDB from source.</p>"},{"location":"build/#1-get-zig","title":"1. Get Zig","text":"<p>First thing first, go get zig.</p>"},{"location":"build/#2-clone-repo","title":"2. Clone repo","text":"<p>Simple enough, clone ZipponDB repository and cd into it.</p> <pre><code>git clone https://github.com/MrBounty/ZipponDB\ncd ZipponDB\n</code></pre>"},{"location":"build/#3-config","title":"3. Config","text":"<p>In <code>lib/config.zig</code> you will find a config file. There is few parameters and they are comptime for now (can't change from cli). But more will be added.</p> Parameter Default Description MAX_FILE_SIZE 1Mb Max size of each individual file where data is store. CPU_CORE 16 Number of thread the pool will use. (At least 4 recommended for db > 100Mb)"},{"location":"build/#4-build","title":"4. build","text":"<pre><code>zig build\n</code></pre> <p>Create 2 binaries in <code>zig-out/bin</code>:</p> <ul> <li>zippondb: The database CLI.</li> <li>benchmark: Run and print a benchmark.</li> </ul>"},{"location":"build/#build-run","title":"build run","text":"<pre><code>zig build run\n</code></pre> <p>Build and run the CLI.</p>"},{"location":"build/#build-benchmark","title":"build benchmark","text":"<pre><code>zig build benchmark\n</code></pre> <p>Build and run the benchmark.</p>"},{"location":"build/#build-test","title":"build test","text":"<pre><code>zig build test\n</code></pre> <p>Build and run tests.</p>"},{"location":"cli/","title":"Command Line Interface","text":"<p>ZipponDB use a CLI to interact, there is few commands available for now as focus was given to ZiQL. But more commands will be added in the future.</p>"},{"location":"cli/#run","title":"run","text":"<p>Run a ZiQL query on the selected database.</p> <p>Usage:</p> <pre><code>run \"QUERY\" // (1)!\n</code></pre> <ol> <li>Note that query need to be between \"\"</li> </ol>"},{"location":"cli/#db","title":"db","text":""},{"location":"cli/#db-metrics","title":"db metrics","text":"<p>Print some metrics from the db, including: Size on disk and number of entities stored.</p> <p>Usage:</p> <pre><code>db metrics\n</code></pre>"},{"location":"cli/#db-new","title":"db new","text":"<p>Create a new empty directory that can be then initialize with a schema.</p> <p>Usage:</p> <pre><code>db new path/to/dir\n</code></pre>"},{"location":"cli/#db-use","title":"db use","text":"<p>Select an already created database with <code>db new</code>.</p> <p>Usage:</p> <pre><code>db use path/to/dir\n</code></pre>"},{"location":"cli/#db-state","title":"db state","text":"<p>Return the state of the database, either <code>Ok</code> or <code>MissingDatabase</code> if no database selected or <code>MissingSchema</code> if no schema was initialize.</p> <p>Usage:</p> <pre><code>db state\n</code></pre>"},{"location":"cli/#schema","title":"schema","text":""},{"location":"cli/#schema-use","title":"schema use","text":"<p>Attach a schema to the database using a schema file.</p> <p>Usage:</p> <pre><code>schema use path/to/schema.file \n</code></pre>"},{"location":"cli/#schema-describe","title":"schema describe","text":"<p>Print the schema use by the selected database.</p> <p>Usage:</p> <pre><code>schema describe\n</code></pre>"},{"location":"cli/#dump","title":"dump","text":"<p>Export the entier database in a specific format.</p> <p>Usage:</p> <pre><code>dump [FORMAT] [PATH]\n</code></pre> <p>FORMAT options: <code>csv</code>, <code>json</code>, <code>zid</code></p>"},{"location":"cli/#quit","title":"quit","text":"<p>Quit the CLI.</p> <p>Usage:</p> <pre><code>quit\n</code></pre>"},{"location":"cli/#help","title":"help","text":"<p>Write an help message.</p> <p>Usage:</p> <pre><code>help\n</code></pre>"},{"location":"interface/","title":"Interface","text":"<p>Interfaces are way to use ZipponDB from other programming language.</p>"},{"location":"interface/#python","title":"Python","text":"<p>Not yet implemented, to give a general idea. Exact code may change.</p> <pre><code>from zippondb import Client\n\ndb = Client(\"data\")\ndb.run(\"ADD User (name='Bob')\")\n</code></pre>"},{"location":"interface/#pydantic","title":"Pydantic","text":"<p>I will most likely implement with Pydantic. Something like that:</p> <pre><code>from zippondb import Client, Model\n\nclass User(Model):\n name: str\n age: int\n\ndb = Client(\"data\")\nusers = db.run(\"GRAB User {}\", model=User)\n</code></pre> <p><code>Model</code> is like <code>BaseModel</code> from pydantic but all member are optional. If no model provided, return a list of dict.</p>"},{"location":"interface/#golang","title":"Golang","text":"<p>TODO</p>"},{"location":"logs/","title":"Log","text":"<p>I have logging implemented but badly and I don't log enough. I need to check this part.</p>"},{"location":"release/","title":"Release","text":""},{"location":"release/#018","title":"0.1.8","text":"<p>First release.</p> <p>Mostly to test the release pipeline with Github actions. Only include CLI, most ZiQL features here but still missing some part. Mostly missing thing around the database to make it usable.</p> <p>Link</p>"},{"location":"ziql/add/","title":"ADD","text":"<p>The <code>ADD</code> action adds entities to the database. The syntax is similar to <code>GRAB</code>, but uses <code>()</code>. This signifies that the data is not yet in the database.</p> <p>Here's an example: <pre><code>ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82])\n</code></pre></p> <p>You need to specify all members when adding an entity (default values comming).</p> <p>The <code>ADD</code> query will return a list of added IDs, e.g.: <pre><code>[\"1e170a80-84c9-429a-be25-ab4657894653\", \"1e170a80-84c9-429a-be25-ab4657894654\", ]\n</code></pre></p> <p>And you can also add them in batch <pre><code>ADD User (name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82]) (name = 'Bob2', age = 33, email = 'bob2@email.com', scores = [])\n</code></pre></p> <p>You don't need to specify member's name for the second entity as long as the order is respected: <pre><code>ADD User \n(name = 'Bob', age = 30, email = 'bob@email.com', scores = [1 100 44 82])\n('Bob2', 33, 'bob2@email.com', [])\n</code></pre></p> <ul> <li>Default value</li> <li>Array default is empty</li> <li>Link default is none</li> </ul>"},{"location":"ziql/delete/","title":"DELETE","text":"<p>Similar to <code>GRAB</code> but deletes all entities found using the filter and returns a list of deleted UUIDs. <pre><code>DELETE User {name = 'Bob'}\n</code></pre></p> <p>The <code>DELETE</code> query will return a list of deleted IDs, e.g.: <pre><code>[\"1e170a80-84c9-429a-be25-ab4657894653\", \"1e170a80-84c9-429a-be25-ab4657894654\", ]\n</code></pre></p>"},{"location":"ziql/grab/","title":"GRAB","text":"<p>The main action is <code>GRAB</code>, this will parse files and return data. </p>"},{"location":"ziql/grab/#basic","title":"Basic","text":"<p>Here's how to return all <code>User</code> without any filtering: <pre><code>GRAB User\n</code></pre></p> <p>To get all <code>User</code> above 30 years old: <pre><code>GRAB User {age > 30}\n</code></pre></p> <p>To return only the member <code>name</code> of <code>User</code>: <pre><code>GRAB User [name] {age > 30}\n</code></pre></p> <p>To return the 10 first <code>User</code>: <pre><code>GRAB User [10] {age > 30}\n</code></pre></p> <p>You can combine these options: <pre><code>GRAB User [10; name] {age > 30}\n</code></pre></p> <p>Use multiple conditions: <pre><code>GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}\n</code></pre></p> <p>GRAB queries return a list of JSON objects with the data inside, e.g: <pre><code>[{id:\"1e170a80-84c9-429a-be25-ab4657894653\", name: \"Gwendolyn Ray\", age: 70, email: \"austin92@example.org\", scores: [ 77 ], friends: [], }, ]\n</code></pre></p>"},{"location":"ziql/grab/#ordering-not-yet-implemented","title":"Ordering - Not yet implemented","text":"<p>To order the results by <code>name</code>: <pre><code>GRAB User [10; name] {age > 10} |ASC name|\n</code></pre></p>"},{"location":"ziql/grab/#array","title":"Array","text":"<p>You can use the <code>IN</code> operator to check if something is in an array: <pre><code>GRAB User { age > 10 AND name IN ['Adrien' 'Bob']}\n</code></pre></p>"},{"location":"ziql/grab/#relationship","title":"Relationship","text":"<p>2 main things to remember with relationship:</p> <ul> <li>You can use filter inside filter.</li> <li>You can use the dot <code>.</code> to refer to a relationship. (Not yet implemented)</li> </ul> <p>Get <code>User</code> that have a best friend named Adrien: <pre><code>GRAB User { bestfriend IN { name = 'Adrien' } }\n</code></pre></p> <p>You can specify how much data to return and which members to include, even for links. In this example, I get 1 friend's name for 10 <code>User</code>: <pre><code>GRAB User [10; friends [1; name]]\n</code></pre></p> <p>When using <code>IN</code>, it return all <code>User</code> that have AT LEAST one friend named Adrien: <pre><code>GRAB User { friends IN { name = 'Adrien' } }\n</code></pre></p> <p>To get <code>User</code> with all friends named Adrien: <pre><code>GRAB User { friends ALLIN { name = 'Adrien' } }\n</code></pre></p> <p>You can use <code>!</code> to say not in: <pre><code>GRAB User { friends !IN { name = 'Adrien' } }\n</code></pre></p>"},{"location":"ziql/grab/#dot-not-yet-implemented","title":"Dot - Not yet implemented","text":"<p>You can use <code>.</code> if you just want to do one comparison. Here I get all <code>User</code> that ordered at least one book: <pre><code>GRAB User { orders.products.category.name = 'Book' }\n</code></pre></p> <p>Same as: <pre><code>GRAB User {orders IN { products IN { category IN { name = 'Book'} } } }\n</code></pre></p> <p>You can also use the dot like that:</p> <pre><code>GRAB User.orders {name = 'Bob'}\n</code></pre> <p>The filter is done on User but it return Order. It return all Order from User named 'Bob'.</p>"},{"location":"ziql/intro/","title":"ZipponQL","text":"<p>ZipponDB uses its own query language, ZipponQL or ZiQL for short. The language goal is to be minimal, small and simple. Yet allowing powerful relationship.</p> <p>Here are the key points to remember:</p> <ul> <li>4 actions available: <code>GRAB</code>, <code>ADD</code>, <code>UPDATE</code>, <code>DELETE</code></li> <li>All queries start with an action followed by a struct name</li> <li><code>{}</code> are filters</li> <li><code>[]</code> specify how much and what data</li> <li><code>()</code> contain new or updated data (not already in files)</li> <li><code>||</code> for ordering</li> <li>By default, all members that are not links are returned</li> </ul> <p>Disclaimer: The language may change a bit over time.</p>"},{"location":"ziql/intro/#making-errors","title":"Making errors","text":"<p>When you make an error writing ZiQL, you should see something like this to help you understand where you made a mistake: <pre><code>Error: Expected string\nGRAB User {name = Bob}\n ^^^ \n</code></pre></p> <pre><code>Error: Expected ( or member name.\nGRAB User {name = 'Bob' AND {age > 10}}\n ^ \n</code></pre>"},{"location":"ziql/intro/#to-return","title":"To return","text":"<p>What is between <code>[]</code> are what data to return. You can see it as the column name after <code>SELECT</code> in SQL and number after <code>LIMIT</code>.</p> <p>Here I return just the name of all users: <pre><code>GRAB User [name] {}\n</code></pre></p> <p>Here the 100 first users: <pre><code>GRAB User [100] {}\n</code></pre></p> <p>Here the name of the 100 first users: <pre><code>GRAB User [100; name]\n</code></pre></p> <p>You can also specify what data to return for each relationship returned. By default, query do not return any relationship.</p> <p>This will return the name and best friend of all users: <pre><code>GRAB User [name, best_friend] {}\n</code></pre></p> <p>You can also specify what the best friend return:</p> <pre><code>GRAB User [name, best_friend [name, age]] {}\n</code></pre>"},{"location":"ziql/intro/#filters","title":"Filters","text":"<p>What is between <code>{}</code> are filters, basically as a list of condition. This filter is use when parsing files and evaluate entities. You can see it as <code>WHERE</code> in SQL.</p> <p>For example <code>{ name = 'Bob' }</code> will return <code>true</code> if the member <code>name</code> of the evaluated entity is equal to <code>Bob</code>. This is the most important thing in ZipponDB.</p> <p>Here an example in a query:</p> <pre><code>GRAB User {name = 'Bob' AND age > 44}\n</code></pre>"},{"location":"ziql/intro/#relationship","title":"Relationship","text":"<p>Filter can be use inside filter. This allow simple yet powerfull relationship.</p> <p>This query will return all users that have a best friend named 'Bob'.</p> <pre><code>GRAB User {best_friend IN {name = 'Bob'}}\n</code></pre> <p>You are obviously not limited to one depth. This will return all users that ordered at least one book in 2024:</p> <pre><code>GRAB User {\n orders IN {\n products IN {\n category IN {\n name = 'Book'\n }\n },\n date > 2024/01/01\n }\n}\n</code></pre> <p>Same as: <pre><code>GRAB User {orders IN { products.category.name = 'Book' AND date > 2024/01/01} } // (1)!\n</code></pre></p> <ol> <li>Dot not yet implemented</li> </ol>"},{"location":"ziql/intro/#link-query-not-yet-implemented","title":"Link query - Not yet implemented","text":"<p>You can also link query. Each query returns a list of UUID of a specific struct. You can use it in the next query. Here an example where I create a new <code>Comment</code> that I then append to the list of comment of one specific <code>User</code>. <pre><code>ADD Comment (content='Hello world', at=NOW, like_by=[], by={id='000'}) \n=> added_comment =>\nUPDATE User {id = '000'} TO (comments APPEND added_comment)\n</code></pre></p> <p>The name between <code>=></code> is the variable name of the list of UUID used for the next queries, you can have multiple one if the link has more than 2 queries. You can also just use one <code>=></code> but the list of UUID is discarded in that case.</p> <p>This can be use with GRAB too. So you can create variable before making the query. Here an example: <pre><code>GRAB User {name = 'Bob'} => bobs =>\nGRAB User {age > 18} => adults =>\nGRAB User {IN adults AND !IN bobs}\n</code></pre></p> <p>Which is the same as: <pre><code>GRAB User {name != 'Bob' AND age > 18}\n</code></pre></p> <p>Note that the query are completly isolated, so files will be parse 3 times.</p> <p>Another example: <pre><code>GRAB Product [1] {category.name = 'Book'} => book =>\nGRAB Order {date > 2024/01/01 AND products IN book} => book_orders =>\nGRAB User [100] {orders IN book_orders}\n</code></pre></p>"},{"location":"ziql/intro/#actions","title":"Actions","text":""},{"location":"ziql/intro/#grab","title":"GRAB","text":"<p>The main action is <code>GRAB</code>, it parse files and return data. <pre><code>GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}\n</code></pre></p> <p>Using <code>[]</code> before the filter tell what to return. <pre><code>GRAB User [id, email] {name = 'Bob'}\n</code></pre></p> <p>Relationship use filter within filter. <pre><code>GRAB User {best_friend IN {name = 'Bob'}}\n</code></pre></p> <p>GRAB queries return a list of JSON objects, e.g: <pre><code>[{id:\"1e170a80-84c9-429a-be25-ab4657894653\", name: \"Gwendolyn Ray\", age: 70, email: \"austin92@example.org\", scores: [ 77 ], friends: [], }, ]\n</code></pre></p> <p>More info.</p>"},{"location":"ziql/intro/#add","title":"ADD","text":"<p>The <code>ADD</code> action adds one entity to the database. The syntax is similar to <code>GRAB</code>, but uses <code>()</code>. This signifies that the data is not yet in the database. <pre><code>ADD User (name = 'Bob', age = 30)\n</code></pre></p> <p>More info.</p>"},{"location":"ziql/intro/#delete","title":"DELETE","text":"<p>Similar to <code>GRAB</code> but deletes all entities found using the filter and returns a list of deleted UUIDs. <pre><code>DELETE User {name = 'Bob'}\n</code></pre></p> <p>More info.</p>"},{"location":"ziql/intro/#update","title":"UPDATE","text":"<p>A mix of <code>GRAB</code> and <code>ADD</code>. It takes a filter first, then the new data. Here, we update the first 5 <code>User</code> entities named 'bob' to capitalize the name and become 'Bob': <pre><code>UPDATE User [5] {name='bob'} TO (name = 'Bob')\n</code></pre></p> <p>More info.</p>"},{"location":"ziql/update/","title":"UPDATE","text":"<p>A mix of <code>GRAB</code> and <code>ADD</code>. It takes a filter first, then the new data. Here, we update the first 5 <code>User</code> entities named 'adrien' to capitalize the name and become 'Adrien': <pre><code>UPDATE User [5] {name='adrien'} TO (name = 'Adrien')\n</code></pre></p> <p>Note that, compared to <code>ADD</code>, you don't need to specify all members between <code>()</code>. Only the ones specified will be updated.</p> <p>The <code>UPDATE</code> query will return a list of updated IDs, e.g.: <pre><code>[\"1e170a80-84c9-429a-be25-ab4657894653\", \"1e170a80-84c9-429a-be25-ab4657894654\", ]\n</code></pre></p>"},{"location":"ziql/update/#not-yet-implemented","title":"Not yet implemented","text":"<p>You can use operations on values themselves when updating: <pre><code>UPDATE User {name = 'Bob'} TO (age += 1)\n</code></pre></p>"},{"location":"ziql/update/#array-not-yet-implemented","title":"Array - Not yet implemented","text":"<p>You can also manipulate arrays, like adding or removing values: <pre><code>UPDATE User {name='Bob'} TO (scores APPEND 45)\nUPDATE User {name='Bob'} TO (scores APPEND [45 99])\nUPDATE User {name='Bob'} TO (scores REMOVEAT [0 1 2])\n</code></pre></p> <p>Currently, there will be four keywords for manipulating lists:</p> <ul> <li><code>APPEND</code>: Adds a value to the end of the list.</li> <li><code>REMOVE</code>: Checks the list, and if the value is found, deletes it.</li> <li><code>REMOVEAT</code>: Deletes the value at a specific index.</li> <li><code>CLEAR</code>: Removes all values from the array.</li> <li><code>POP</code>: Remove the last value.</li> </ul> <p>Except for <code>CLEAR</code> and <code>POP</code>, which takes no value, each keyword can use one value or an array of values. If you choose an array, it will perform the operation on all values in the array.</p> <p>For relationships, you can use filters: <pre><code>UPDATE User {name='Bob'} TO (comments APPEND {id = '000'})\nUPDATE User {name='Bob'} TO (comments REMOVE [1] { at < '2023/12/31'})\n</code></pre></p> <p>I may include more options later.</p>"},{"location":"ziql/vssql/","title":"Vs SQL","text":"<p>A good way to see how ZipponQl work is to compare it to SQL.</p>"},{"location":"ziql/vssql/#select-everything","title":"Select everything","text":"<pre><code>SELECT * FROM User\n</code></pre> <pre><code>GRAB User {}\n</code></pre>"},{"location":"ziql/vssql/#selection-on-condition","title":"Selection on condition","text":"<pre><code>SELECT *\nFROM Users\nWHERE name = 'Bob'\nAND (age > 30 OR age < 10);\n</code></pre> <pre><code>GRAB User {name = 'Bob' AND (age > 30 OR age < 10)}\n</code></pre>"},{"location":"ziql/vssql/#select-something","title":"Select something","text":"<pre><code>SELECT name, age\nFROM Users\nLIMIT 100\n</code></pre> <pre><code>GRAB User [100; name, age] {}\n</code></pre>"},{"location":"ziql/vssql/#relationship","title":"Relationship","text":"<pre><code>SELECT u1.name AS user_name, GROUP_CONCAT(u2.name || ' (' || u2.age || ')') AS friends_list\nFROM Users u1\nLEFT JOIN User u2 ON ',' || u1.friends || ',' LIKE '%,' || u2.id || ',%'\nWHERE u1.age > 30\nGROUP BY u1.name;\n</code></pre> <pre><code>GRAB User [name, friends [name, age]] {age > 30}\n</code></pre> <p>SQL: <pre><code>SELECT Users.name, Orders.orderID, Orders.orderDate\nFROM Users\nINNER JOIN Orders ON Users.UserID = Orders.UserID;\n</code></pre></p> <p>ZiQL: <pre><code>GRAB User [name, order [id, date]] {}\n</code></pre></p> <p>SQL: <pre><code>SELECT \n U.name AS UserName,\n O.orderID,\n O.orderDate,\n P.productName,\n C.categoryName,\n OD.quantity,\nFROM \n Users U\nINNER JOIN Orders O ON U.UserID = O.UserID\nINNER JOIN OrderDetails OD ON O.OrderID = OD.OrderID\nINNER JOIN Products P ON OD.ProductID = P.ProductID\nINNER JOIN Categories C ON P.CategoryID = C.CategoryID\nWHERE \n O.orderDate >= '2023-01-01'\n AND C.categoryName != 'Accessories'\nORDER BY \n O.orderDate DESC;\n</code></pre></p> <p>ZiQL: <pre><code>GRAB User\n[ name, orders [id, date, details [quantity, products [name, category [name]]]]]\n{ orders IN {date >= 2023/01/01 AND details.products.category.name != 'Accessories' } } // (1)!\n| orders.date DESC | // (2)!\n</code></pre></p> <ol> <li>Dot not yet implemented. But you can do it with: <pre><code>details IN { products IN {category IN {name != 'Accessories'}}}\n</code></pre></li> <li>Ordering not yet implemented</li> </ol> <p>SQL: <pre><code>UPDATE orders o\nJOIN customers c ON o.customer_id = c.customer_id\nSET o.status = 'Priority'\nWHERE c.membership_level = 'Premium' AND o.order_date > '2023-01-01';\n</code></pre></p> <p>ZiQL: <pre><code>GRAB User.orders { membership_level = 'Premium' } // (1)!\n=> premium_order => // (2)!\nUPDATE Order {id IN premium_order AND date > 2023/01/01}\nTO (status = 'Priority')\n</code></pre></p> <ol> <li> <p>Not yet implemented. Can't do it now. Here that mean filter are done on User but it return Order. It return all order of User with a premium membership.</p> </li> <li> <p>Linked query not implemented</p> </li> </ol>"}]} |