Think about how to handle large files about iota HOT 6 OPEN

Yeah, I was thinking about this the other day.
If we are to only load chunks of files at a time, can we still do things like search, line count, char count etc? We will need to decide what is considered a large file, so we can impose these kind of restrictions on anything above that size.

from iota.

We will also have to change how files are saved to accommodate large files too. Right now files are saved to a temporary directory then moved to replace the old version. This won't work well for large files.

from iota.

For a data structure, vis uses a piece table, to store the file in memory. This is very efficient for large files, since you can just mmap the file into memory.

Also, you might want to read this paper.

from iota.

A piece table might be a nice half-way step towards a rope, which I think would, in principle, also support mmap-ing files into a giant leaf node.

Thanks for that paper @ellisadigvom, this stuff is really helpful for someone as inexperienced as myself!

from iota.

@ellisadigvom that's very interesting, thanks for sharing! I have been leaning quite heavily towards a Rope recently, but have not had the time to sit down and try it out. This looks like a good alternative, though, definitely worth looking into. I had actually never heard of a piece table before..

Gonna give vis a shot and see how it handles some big files :)

from iota.

So, anyone know of a piece table implementation in rust? :)

from iota.