121 Commits

Author SHA1 Message Date
1495e779c9 Speed up date
Date was taking a long time for parsing when using ADD in batch, speeded
up by like x50
2025-01-11 15:27:17 +01:00
7d012b527f Write every 1000 map
Now if I run a big ADD query, write every 1000 entities to prevent a
maps array too big and go OOM
2025-01-11 13:50:32 +01:00
d533eaff98 Working multi struct schema with relationship!
Noce
2025-01-08 23:53:41 +01:00
71e5f6eb1e Started to debug schema with multiple struct and some time keyword
Added NOW already and now debuging some stuff regarding filter and
parsing file of one struct when it should be another

Also moved query test into a seperated test file.
And some fix and changed in docs
2025-01-08 10:09:15 +01:00
a712fe3e83 Save schema file when using new 2025-01-07 17:15:09 +01:00
65fa7ccfc4 Now delete empty file when deleting entities exept 0.zid 2025-01-07 14:09:22 +01:00
2a4842432d Speed up batch ADD and better bechmark
Now I flush only when the file is full and I check the the currently
used file if it is big enough.

So I dont get stat of all files and flush everytime like before
2025-01-07 13:55:02 +01:00
e3264d8553 Random ADD for benchmark 2025-01-06 20:45:16 +00:00
900b9e5305 Moved benchmark outside of src and making it better 2025-01-02 13:28:23 +00:00
b075f8b89a Moved config to libs 2025-01-02 12:19:05 +00:00
4f421f7977 Added ADD in batch 2024-12-30 15:31:16 +01:00
a535ce5283 Basic benchmark command 2024-12-30 14:09:59 +01:00
a7d076ec80 Added !IN 2024-12-30 12:08:25 +01:00
2e2dae3df1 Some typo 2024-12-29 10:19:04 +01:00
b2dd6fe627 Basic working relationship
It is working !

The basic GRAB [name, best_friend] work and do return the best friend
infos.

I added more test and some do not work yet, working on it now
2024-12-28 20:12:20 +01:00
3b47007ca4 Base reparsing
This dosnt work yet but I implemented the reparsing of the files to
return relationship.

So GRAB User [name, friends] should return all infos of all friends and
not just the UUID of those friends like before
2024-12-27 12:42:51 +01:00
0127daa330 Push before leaving for christmas 2024-12-21 15:13:50 +01:00
e7056efec9 Now query with relationship will write the UUID bytes betwen {|<>|}
So then I can parse the file again, create a map of UUID sub json and
just iterate over the first json looking for UUID and update with new
data from map
2024-12-20 22:29:02 +01:00
7fb0b5f63e Changed memory in AdditionalData
Now it need to use an arena because there is no deinit function.

And additionalData of member are null by default instead of an empty
list.
2024-11-30 10:33:28 +01:00
57a9eecdb3 Renamed members of AdditionalData 2024-11-30 09:49:54 +01:00
c6fe428270 Dont send link if not in AdditionalData 2024-11-29 21:56:52 +01:00
4ee79c9629 Created an EntityWriter responsable for writing into JSON and table into a writer 2024-11-29 21:38:05 +01:00
7f27557ca2 Fix parsing array
Before if the array is [ 1 ] I get [ 0, 1, 0] because I get 2 empty
string that when I try to make into int, it become 0
2024-11-29 21:20:12 +01:00
3f53da5220 Making link_array working, now can do query with filter for array of ofther struct 2024-11-28 19:20:44 +01:00
12ae7f9a62 Added db state command and fixed printError
I was using the same buffer at 2 place :/
2024-11-26 22:59:36 +01:00
08cae48cbc Added a writeEntityTable and fixed date and time 2024-11-24 22:26:54 +01:00
1333b75981 Dont trow and error on basic relationship query 2024-11-24 20:10:21 +01:00
bf1f5916d7 Can now ADD using a filter 2024-11-24 16:55:54 +01:00
ceab51da27 NewData map now use ConditionValue 2024-11-24 16:26:26 +01:00
23e8ed8709 Changed a bit how the memory work
Now SchemaEngine have an arena at the root of the file. So UUIDFileIndex
can be as big as we want. The idea is nice, maybe I will use it for the
FileEngine too.
2024-11-22 11:03:39 +01:00
3f5c929a11 Big memory improvement
Started to use fixed buffer at the root of each file. I like that
because that mean I can estimate the amount of memory that the app use.

The only thing that still use an non fixed allocator is the
UUIDFileIndex, but I dont like it. I think it should also use a fixed
lenght buffer. But then how can I be sure it is enough ? :/

I mean I get it, I need an allocator here. But if this is the only place
I use one, this a shame
2024-11-20 23:52:36 +01:00
979690fec4 IN filter working
Now if I do like GRAB User { best_friends IN {name = 'Bob'}} it should
work

At least now the condition value is a hashmap with key as UUID.
2024-11-17 21:52:11 +01:00
7b2a754d81 Check if additional data count is reached at each iteration
Before I was checking after the filter.evaluate is true. So I can end up
with the limit reach, but the thread will continue parsing until it find
one that is true, add it and then check if the limit is reach.

Which mean I endup with more than what I want and the thread could stop
before, making things faster.
2024-11-17 20:45:58 +01:00
65ba0263cc Condition value now can have a HashMap of UUID/void 2024-11-17 18:11:54 +01:00
ca5de646d6 Populate the UUIDFileIndex
Now each SchemaStruct have a UUIDFileIndex that allow to find the file
index of where an entity is based on it's UUID
2024-11-17 16:40:12 +01:00
5c8d2b6dc7 Create ThreadEngine
Now the DBEngine have a ThreadEngine that have a pool. Like that it is
always the same pool and I dont create one everytime I run a parseing
methods of the FileEngine
2024-11-17 15:25:54 +01:00
303dc81e2a Created an unused UUIDFileIndex
Created a RadixTree but it is too slow and doest provide that much
reduced memory, mostly because I did it badly. So I will juste use a
small wrapper around an AutoHashMap UUID usize
2024-11-17 00:11:34 +01:00
ef84200b34 Parsing filter can now parse sub filter, waiting to be use with the parsing to find relationship 2024-11-14 20:39:04 +01:00
6e7d1d150c Pass test with added stuffs for relationship 2024-11-13 22:44:27 +01:00
13cba58dbc Fix 2024-11-12 21:57:13 +01:00
5e9c64caf6 Removed useless code 2024-11-12 21:18:19 +01:00
3a61e81024 Added not in keyword 2024-11-12 21:17:49 +01:00
f5d93c94f6 Created a ThreadSyncContext
This keep the atomic value that are shared by thread
2024-11-12 21:17:06 +01:00
91b1f61e66 Working new SchemaEngine 2024-11-10 17:10:42 +01:00
e1c957b3b6 Started cleaning by creating a SchemaEngine 2024-11-10 11:38:37 +01:00
9fa6f25459 Removed CSV_DELIMITER and some old comment 2024-11-09 16:39:40 +01:00
e5d0a122e1 Added multi threading to delete 2024-11-06 16:04:07 +01:00
65be757440 Added multithreading to update 2024-11-06 16:01:55 +01:00
900b6c5f54 Replace some path_buff to utils.preintOpenDir/File 2024-11-04 23:18:49 +01:00
96e77a5ad4 Removed last trace of null 2024-11-04 22:57:19 +01:00