Now I flush only when the file is full and I check the the currently
used file if it is big enough.
So I dont get stat of all files and flush everytime like before
It is working !
The basic GRAB [name, best_friend] work and do return the best friend
infos.
I added more test and some do not work yet, working on it now
This dosnt work yet but I implemented the reparsing of the files to
return relationship.
So GRAB User [name, friends] should return all infos of all friends and
not just the UUID of those friends like before
So then I can parse the file again, create a map of UUID sub json and
just iterate over the first json looking for UUID and update with new
data from map
Now SchemaEngine have an arena at the root of the file. So UUIDFileIndex
can be as big as we want. The idea is nice, maybe I will use it for the
FileEngine too.
Started to use fixed buffer at the root of each file. I like that
because that mean I can estimate the amount of memory that the app use.
The only thing that still use an non fixed allocator is the
UUIDFileIndex, but I dont like it. I think it should also use a fixed
lenght buffer. But then how can I be sure it is enough ? :/
I mean I get it, I need an allocator here. But if this is the only place
I use one, this a shame
Before I was checking after the filter.evaluate is true. So I can end up
with the limit reach, but the thread will continue parsing until it find
one that is true, add it and then check if the limit is reach.
Which mean I endup with more than what I want and the thread could stop
before, making things faster.
Now the DBEngine have a ThreadEngine that have a pool. Like that it is
always the same pool and I dont create one everytime I run a parseing
methods of the FileEngine
Created a RadixTree but it is too slow and doest provide that much
reduced memory, mostly because I did it badly. So I will juste use a
small wrapper around an AutoHashMap UUID usize
- Added a new data type self, that represent the id of the intity itself
- Fixed multi threading for parsing, now each thread use it's own writer
and I concat them at the end
- Added a schemaStruct id to the list
- Other fixe and stuff to go with the rest
New step, multi threading for all function then finally relationship
For very importatnt stuff like the writter that write data when parsing,
started to use fixed length because it take the majority of the time to
write, not to parse =/
Gonna need to improve that
When I use s2t, or string to type. Like to transform a string array into
an array. "[1 4 21]" to []i32{ 1, 4, 21}.
Before I return an ArrayList, now it use toOwnedSlice
Created a new function to replace parseAndFindUUIDWithFilter.
Now it parse the file, evaluate with the filter and write directly to a
buffer. There is no UUID that is return, then file are parse again.
Should be a huge perf improve.
Some bugs with additionData tho, got a name: "<UUID>" =(
SchemaStruct member now are [] and not ArrayList. Started to use and
understand toOwnedSlice.
Implemented the hashmap that keep all UUID -> File index. Inside each
SchemaStruct
Condition now store the index of the value in the file instead of the
member_name. So when I parse, I can just use the list array []Data and
the index to compare both value
Now the parseFilter function return a Filter struct.
It is a tree of Condition and Logic operator that will then be use when
parsing row. Need to do the evaluate function with the []Data and test
some error
adrien