Compact storage with shorter tag and field key prefixes about tinyflux HOT 4 CLOSED

I will add a feature that allows you to change the prefix of field and tag key/values from the defaults of _field_ and _tag_ here shortly. Might take a couple of days.

from tinyflux.

no rush we have actually modified directly on the point module, it's just a suggestion to make the storage object more portable

from tinyflux.

do you have a fork on github? id be interested to see how you implemented it as i work on this feature

from tinyflux.

Okay I've added a feature to 0.3 where you can specify to use a compact key prefix for tags and fields upon insertion. I didn't implement custom prefixes because the prefixes are necessary for deserialization, and I don't expect users to understand the reasoning behind this- specifically the requirement that a prefix for a tag key and field key must be different. I also wanted to keep the prefix somewhat human-readable in the CSV so users aren't terribly confused so I kept the trailing underscore in the prefix. So now when you specify the compact_key_prefixes=True option, you will get t_myTagKey for tag keys, and f_myFieldKey for field keys, instead of _tags_myTagKey and _fields_myFieldKey respectively. You can also insert points with compact key prefixes into a database that already has the longer, default prefixes.

Make sure to update tinyflux:

$ pip install --upgrade tinyflux

To use in your python interpreter:

>>> from tinyflux import Point, TinyFlux
>>> my_db = TinyFlux("test.tinyflux")
>>> p = Point(tags={"myTagKey": "foo"}, fields={"myFieldKey": 0.0})
>>> my_db.insert(p, compact_key_prefixes=True)
1
>>> # Inspect the CSV data for the compact key prefixes.
>>> import csv
>>> with open("test.tinyflux") as f:
...     data = list(csv.reader(f))
>>> data # t_ and f_ prefixes
[['2023-03-21T22:50:02.313069', '_default', 't_myTagKey', 'foo', 'f_myFieldKey', '0.0']]

Again, this will also work with a mix of compact and non-compact (default) key prefixes, which may be the case if you now use tinyflux v0.3 on a database that was built with <0.3:

>>> from tinyflux import Point, TinyFlux
>>> my_db = TinyFlux("test.tinyflux")
>>> p1 = Point(tags={"myTagKey": "foo"})
>>> my_db.insert(p1) # compact_key_prefixes is False by default
1
>>> p2 = Point(tags={"myTagKey": "bar"})
>>> my_db.insert(p2, compact_key_prefixes=True)
1
>>> # Inspect the CSV data for compact and default key prefixes.
>>> import csv
>>> with open("test.tinyflux") as f:
...     data = list(csv.reader(f))
>>> data[0] # _tag_ prefix
['2023-03-21T22:54:35.202775', '_default', '_tag_myTagKey', 'foo']
>>> data[1] # t_ prefix
['2023-03-21T22:55:03.869054', '_default', 't_myTagKey', 'bar']
>>> my_db.all()[0].tags # default tag key prefix is parsed normally
{'myTagKey': 'foo'}
>>> my_db.all()[1].tags # compact tag key prefix is parsed normally
{'myTagKey': 'bar'}

This option also works with the .insert_multiple() method. Read the documentation on inserting points for more.

Finally, to update all key prefixes to the compact version, you can cntl-f and replace, but it might hang your text editor if its a regular GUI application. I would use sed in bash. Make a backup of the file, and hopefully your database has no instances of _tag_ or _field_ besides in the tag and field keys:

$ # Replaces all instances of _tag_ with t_.
$ sed -i '' 's/_tag_/t_/g' /path/to/your/file.csv
$ # Replaces all instances of _field_ with f_.
$ sed -i '' 's/_field_/f_/g' /path/to/your/file.csv

This could take several seconds but your database will be good to go upon update.

There are some other considerations you can make for saving space in your files in the documentation here, including keeping tag and field keys short and concise.

from tinyflux.