citrusvanilla / tinyflux Goto Github PK
View Code? Open in Web Editor NEWThe tiny time series database optimized for your happiness.
License: MIT License
The tiny time series database optimized for your happiness.
License: MIT License
Describe the bug
Updating a Point after insertion using CSVStorage throws a TypeError.
To Reproduce
from tinyflux import Point, TinyFlux
db = TinyFlux("my_db.tinyflux")
p = Point(fields={"a": 1})
db.insert(p)
db.update_all(fields=lambda x: {"b": 2})
Expected behavior
A TypeError should not be thrown.
Describe the bug
Tag and Field keys can be removed during an update if an in-place removal is contained in the updater callable.
To Reproduce
Steps to reproduce the behavior:
>>> from tinyflux import TinyFlux, Point
>>> p = Point(tags={"a": "1"})
>>> db = TinyFlux("test.tinyflux")
>>> db.insert(p)
>>> def my_bad_updater(x):
>>> x.pop("a")
>>> return x
>>> db.update_all(tags=my_bad_updater)
>>> db.all()[0]
Point(time=2023-03-27T18:24:24.549741+00:00, measurement=_default)
Expected behavior
Tag and Field keys should not be able to be deleted using the update function.
Is your feature request related to a problem? Please describe.
The need would be to keep the data locally on an IoT device but also sync it with a cloud database for backup. Could be Influx or something else.
Describe the solution you'd like
It would be amazing to have a feature allowing syncing local TinyFlux with external Database
Describe alternatives you've considered
I guess MQTT could be used to set up such workflow, but I'm trying to reduce as much as possible the number of tools needed.
Additional context
The device would run on the field could be damaged or lost but has a 4G connectivity, and we should be able to back up the data.
Is your feature request related to a problem? Please describe.
I am trying to get the very last entry inserted in the database. From the doc I can't see any way to do this efficiently.
I was hoping to be able to query by index but this is not an option.
Describe the solution you'd like
Ideally a db.last() function to return the last inserted point would work well.
If not I would be happy to know any efficient way to get the last entry of the db.
Describe alternatives you've considered
Well I could maybe query by time with a start time early enough to grab at least one point.
Then collect the last entry in the list. Not great.
Additional context
nothing to add
Is your feature request related to a problem? Please describe.
Will TinyFlux work out of the box on Python 3.12?
Describe the solution you'd like
TinyFlux should work with Python 3.12 interpreters.
Describe alternatives you've considered
There is no alternative.
Since update
is used to modify an point's fileds dict, new fileds will be added and existing fields will not be deleted.
Line 1223 in 24ab88d
Let me explain it by the code:
>>> tags = TagQuery()
>>> q = tags.id == "some-unique-value"
>>> db.get(q)
Point(time=2023-02-01T19:26:28+00:00, measurement=_default, tags=id:some-unique-value, fields=sensor1:58)
... def generalize_sensors(fields):
... if "sensor1" in fields:
... return {"sensor": fields["sensor1"]}
... else:
... return fields
>>> db.update(q, fields=generalize_sensors)
>>> db.get(q)
Point(time=2023-02-01T19:26:28+00:00, measurement=_default, tags=id:some-unique-value, fields=sensor1:58; sensor:58)
I would expect that generalize_sensors
will remove sensor1
from the fields since it does not use **fields
in the returning dict.
My use case is with a few GB of data and trying to create a compact version of csvstorge to trim down data size. what I wanted to do is replacing _field_name with fname on the csv row, however, I notice I actually need to modify the serialize and deserialize from list of the point function.
do you have any solution for a more portable storage implementation?
and thanks for the awesome tinyflux
hi, when i run the following codes, got errors
from tinyflux import TinyFlux,TagQuery,FieldQuery
db = TinyFlux('tm_db/features_store.csv')
Error Traceback (most recent call last)
File :2, in
File ~/miniconda3/lib/python3.8/site-packages/tinyflux/database.py:181, in TinyFlux.init(self, auto_index, *args, **kwargs)
179 # Reindex if auto_index is True.
180 if self._auto_index and not self._storage._initially_empty:
--> 181 self.reindex()
File ~/miniconda3/lib/python3.8/site-packages/tinyflux/database.py:754, in TinyFlux.reindex(self)
751 return
753 # Build the index.
--> 754 self._index.build(
755 self._storage._deserialize_storage_item(i) for i in self._storage
756 )
758 return
File ~/miniconda3/lib/python3.8/site-packages/tinyflux/index.py:185, in Index.build(self, points)
182 # A buffer for the new timestamps and their storage positions.
183 timestamp_buffer: List[Tuple[float, int]] = []
--> 185 for idx, point in enumerate(points):
186 self._num_items += 1
187 self._insert_measurements(idx, point.measurement)
File ~/miniconda3/lib/python3.8/site-packages/tinyflux/database.py:754, in (.0)
751 return
753 # Build the index.
--> 754 self._index.build(
755 self._storage._deserialize_storage_item(i) for i in self._storage
756 )
758 return
Error: line contains NUL
Description
I just tried to use tinyflux as an "in-memory time-series database" using the storage=storages.MemoryStorage option but found that my query returns different results then when using a "real" csv file as store.
To reproduce, simple comment/uncomment near method1/method2 to use the different constructors
#!/usr/bin/env python3
from tinyflux import TinyFlux, Point, TagQuery, FieldQuery, MeasurementQuery, TimeQuery, storages
import datetime
# method1
tiny_flux = TinyFlux(storage=storages.MemoryStorage)
# method2
#tiny_flux = TinyFlux("test.csv")
data = {
"measurement": "sma",
"tags": {
"id": "should",
},
"time": "0",
"fields": {
"value" : 1.0,
}
}
p1 = Point(time = datetime.datetime.now(), measurement = data["measurement"], fields = data["fields"], tags = data["tags"])
tiny_flux.insert(p1)
data["tags"]["id"] = "not"
p2 = Point(time = datetime.datetime.now(), measurement = data["measurement"], fields = data["fields"], tags = data["tags"])
tiny_flux.insert(p2)
tags = TagQuery()
result = tiny_flux.search(tags.id == "should") # different behaviour with MemoryStorage and CSV?
print(result)
Is this the intended behaviour?
Typing should be improved by using mypy in strict mode.
Describe the bug
Some documentation pages of this project have errors in example code snippets, and more accurately tinydb
is used instead of tinyflux
. You can search for all occurances here.
To Reproduce
Steps to reproduce the behavior:
from tinydb import
Expected behavior
tinyflux
shall be used instead.
Additional context
Someone has made a pull request for this issue.
Describe the bug
When a Point containing a tag value of ""
is inserted into the database and read back out, it returns as the string "_none"
.
To Reproduce
>>> from tinyflux import Point
>>> p = Point(tags={"a": ""})
>>> # Simulate and write/read op.
>>> new_p = tf.Point()._deserialize_from_list(p._serialize_to_list())
>>> new_p.tags["a"]
'_none'
Expected behavior
The tag value should remain an empty string after write/read ops.
Additional context
tinyflux version 0.2.6
Though tinyflux
codebase is widely type annotated, the type annotations supplied with the tinyflux
package are not used during static analysis (i.e. mypy
) when imported as a dependency in other projects.
Specifically, this causes mypy
to output the following error and render any class imported from tinyflux
package as imprecise.
error: Skipping analyzing "tinyflux": module is installed, but missing library stubs or py.typed marker [import]
Add py.typed
file at the root of the package directory.
Hi,
I am using this pretty library in my small application.
After porting my app to Windows 11, tinyflux shows an error when creating and reopening the file because it adds empty lines to the file when writing:
env\lib\site-packages\tinyflux\point.py", line 251, in _deserialize_from_list
p_time = datetime.fromisoformat(row[0]).replace(tzinfo=timezone.utc)
IndexError: list index out of range
I did some research and found out that adding newline='' when opening the file solves the problem:
# Open the file for reading/writing
self._handle = open(path, mode=self._mode, encoding=encoding, newline=newline)
Thanks!
Describe the bug
Field values of 0.0 are serialized to the DB as the string "_none", resulting in a deserialized value of None
in Python. They should be serialized as the string "0.0".
To Reproduce
Steps to reproduce the behavior:
>>> import tinyflux
>>> p = tinyflux.Point(fields={"a": 0.0})
>>> p._serialize_to_list()
('2022-11-22T18:36:47.633509', '_default', '_field_a', '_none')
Expected behavior
Points with values of 0.0 for any field should serialize to "0.0", not "_none".
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.