citrusvanilla / tinyflux Goto Github PK

Describe the bug
Tag and Field keys can be removed during an update if an in-place removal is contained in the updater callable.

To Reproduce
Steps to reproduce the behavior:

>>> from tinyflux import TinyFlux, Point
>>> p = Point(tags={"a": "1"})
>>> db = TinyFlux("test.tinyflux")
>>> db.insert(p)
>>> def my_bad_updater(x):
>>>     x.pop("a")
>>>     return x
>>> db.update_all(tags=my_bad_updater)
>>> db.all()[0]
Point(time=2023-03-27T18:24:24.549741+00:00, measurement=_default)

Expected behavior
Tag and Field keys should not be able to be deleted using the update function.

Functionality to improve and goal

Though tinyflux codebase is widely type annotated, the type annotations supplied with the tinyflux package are not used during static analysis (i.e. mypy) when imported as a dependency in other projects.

Specifically, this causes mypy to output the following error and render any class imported from tinyflux package as imprecise.

error: Skipping analyzing "tinyflux": module is installed, but missing library stubs or py.typed marker  [import]

Proposal

Add py.typed file at the root of the package directory.

Links and references

PEP 561

Describe the bug
Field values of 0.0 are serialized to the DB as the string "_none", resulting in a deserialized value of None in Python. They should be serialized as the string "0.0".

To Reproduce
Steps to reproduce the behavior:

>>> import tinyflux
>>> p = tinyflux.Point(fields={"a": 0.0})
>>> p._serialize_to_list()
('2022-11-22T18:36:47.633509', '_default', '_field_a', '_none')

Expected behavior
Points with values of 0.0 for any field should serialize to "0.0", not "_none".

Is your feature request related to a problem? Please describe.
Will TinyFlux work out of the box on Python 3.12?

Describe the solution you'd like
TinyFlux should work with Python 3.12 interpreters.

Describe alternatives you've considered
There is no alternative.

Description
I just tried to use tinyflux as an "in-memory time-series database" using the storage=storages.MemoryStorage option but found that my query returns different results then when using a "real" csv file as store.

To reproduce, simple comment/uncomment near method1/method2 to use the different constructors

#!/usr/bin/env python3

from tinyflux import TinyFlux, Point, TagQuery, FieldQuery, MeasurementQuery, TimeQuery, storages
import datetime

# method1
tiny_flux = TinyFlux(storage=storages.MemoryStorage)
# method2
#tiny_flux = TinyFlux("test.csv")

data = {
    "measurement": "sma",
    "tags": {
        "id": "should",
    },
    "time": "0",
    "fields": {
        "value" : 1.0,
    }
}

p1 = Point(time = datetime.datetime.now(), measurement = data["measurement"], fields = data["fields"], tags = data["tags"])
tiny_flux.insert(p1)
data["tags"]["id"] = "not"
p2 = Point(time = datetime.datetime.now(), measurement = data["measurement"], fields = data["fields"], tags = data["tags"])
tiny_flux.insert(p2)

tags = TagQuery()
result = tiny_flux.search(tags.id == "should") # different behaviour with MemoryStorage and CSV?
print(result)

Is this the intended behaviour?

Is your feature request related to a problem? Please describe.
The need would be to keep the data locally on an IoT device but also sync it with a cloud database for backup. Could be Influx or something else.

Describe the solution you'd like
It would be amazing to have a feature allowing syncing local TinyFlux with external Database

Describe alternatives you've considered
I guess MQTT could be used to set up such workflow, but I'm trying to reduce as much as possible the number of tools needed.

Additional context
The device would run on the field could be damaged or lost but has a 4G connectivity, and we should be able to back up the data.

Describe the bug
When a Point containing a tag value of "" is inserted into the database and read back out, it returns as the string "_none".

To Reproduce

>>> from tinyflux import Point
>>> p = Point(tags={"a": ""})
>>> # Simulate and write/read op.
>>> new_p = tf.Point()._deserialize_from_list(p._serialize_to_list())
>>> new_p.tags["a"]
'_none'

Expected behavior
The tag value should remain an empty string after write/read ops.

Additional context
tinyflux version 0.2.6

Hi,
I am using this pretty library in my small application.
After porting my app to Windows 11, tinyflux shows an error when creating and reopening the file because it adds empty lines to the file when writing:

env\lib\site-packages\tinyflux\point.py", line 251, in _deserialize_from_list

    p_time = datetime.fromisoformat(row[0]).replace(tzinfo=timezone.utc)

IndexError: list index out of range

I did some research and found out that adding newline='' when opening the file solves the problem:

# Open the file for reading/writing
self._handle = open(path, mode=self._mode, encoding=encoding, newline=newline)

About newline param

Thanks!

Describe the bug
Updating a Point after insertion using CSVStorage throws a TypeError.

To Reproduce

from tinyflux import Point, TinyFlux
db = TinyFlux("my_db.tinyflux")
p = Point(fields={"a": 1})
db.insert(p)
db.update_all(fields=lambda x: {"b": 2})

Expected behavior
A TypeError should not be thrown.

Is your feature request related to a problem? Please describe.
I am trying to get the very last entry inserted in the database. From the doc I can't see any way to do this efficiently.
I was hoping to be able to query by index but this is not an option.

Describe the solution you'd like
Ideally a db.last() function to return the last inserted point would work well.
If not I would be happy to know any efficient way to get the last entry of the db.

Describe alternatives you've considered
Well I could maybe query by time with a start time early enough to grab at least one point.
Then collect the last entry in the list. Not great.

Additional context
nothing to add

My use case is with a few GB of data and trying to create a compact version of csvstorge to trim down data size. what I wanted to do is replacing _field_name with fname on the csv row, however, I notice I actually need to modify the serialize and deserialize from list of the point function.

do you have any solution for a more portable storage implementation?

and thanks for the awesome tinyflux

hi, when i run the following codes, got errors

from tinyflux import TinyFlux,TagQuery,FieldQuery
db = TinyFlux('tm_db/features_store.csv')

Error Traceback (most recent call last)
File :2, in

File ~/miniconda3/lib/python3.8/site-packages/tinyflux/database.py:181, in TinyFlux.init(self, auto_index, *args, **kwargs)
179 # Reindex if auto_index is True.
180 if self._auto_index and not self._storage._initially_empty:
--> 181 self.reindex()

File ~/miniconda3/lib/python3.8/site-packages/tinyflux/database.py:754, in TinyFlux.reindex(self)
751 return
753 # Build the index.
--> 754 self._index.build(
755 self._storage._deserialize_storage_item(i) for i in self._storage
756 )
758 return

File ~/miniconda3/lib/python3.8/site-packages/tinyflux/index.py:185, in Index.build(self, points)
182 # A buffer for the new timestamps and their storage positions.
183 timestamp_buffer: List[Tuple[float, int]] = []
--> 185 for idx, point in enumerate(points):
186 self._num_items += 1
187 self._insert_measurements(idx, point.measurement)

File ~/miniconda3/lib/python3.8/site-packages/tinyflux/database.py:754, in (.0)
751 return
753 # Build the index.
--> 754 self._index.build(
755 self._storage._deserialize_storage_item(i) for i in self._storage
756 )
758 return

Error: line contains NUL

Since update is used to modify an point's fileds dict, new fileds will be added and existing fields will not be deleted.

Let me explain it by the code:

>>> tags = TagQuery()
>>> q = tags.id == "some-unique-value"
>>> db.get(q)
Point(time=2023-02-01T19:26:28+00:00, measurement=_default, tags=id:some-unique-value, fields=sensor1:58)
... def generalize_sensors(fields):
...     if "sensor1" in fields:
...         return {"sensor": fields["sensor1"]}
...     else:
...         return fields
>>> db.update(q, fields=generalize_sensors)
>>> db.get(q)
Point(time=2023-02-01T19:26:28+00:00, measurement=_default, tags=id:some-unique-value, fields=sensor1:58; sensor:58)

I would expect that generalize_sensors will remove sensor1 from the fields since it does not use **fields in the returning dict.

Typing should be improved by using mypy in strict mode.