tobgu / qcache Goto Github PK

Seems like numpy decides that float is the proper type. If passing a numerical stand-in the following types should be applied:

0.0 => type float
0 => type int

It's not possible to run any queries using standin-columns in the query headers. QCache crashes.

This is a regression in Pandas 0.21 it seems, see:
pandas-dev/pandas#18186

It's currently possible to define stand-in-columns for data that may not be available in the dataset that is being pushed but for which you would later like to make queries against.

This should also be possible when querying data for added flexibility.

Support gzip (for compatibility) and lz4 (for greatness).

In Pandas 0.19 it is possible to define categorials directly when reading CSVs. Investigate this to see if it is feasible for categorials support.

https://github.com/tobgu/qcache/blob/master/qcache/qframe/pandas_filter.py#L95
should ignore NaNs as described here: https://stackoverflow.com/questions/28311655/ignoring-nans-with-str-contains.

In this case the dataset size when inserting the dataset into the cache may differ from the size when evicting it. This causes the cache size to slip and the memory usage to grow with no bounds.

This would allow you to write queries that currently require two queries against the same dataset.

Pandas interprets a number of strings as NaN when reading CSVs. It should not. Only empty string should be considered NaN.

I wonder what is the advantage to use this instead of having an in memory sqlite or another in memory database engine (a huge advantage is that database engine work for multiple tables)

Currently posting new data for a key that already exists overwrites the current entry. This is probably a good default behavior. There may be times however where the key-value relationship is immutable and a certain key always represents a certain value. In these cases an update would just be a wasteful no-op.

It should be possible to decide the update strategy upon startup of the server (probably not needed per update request). Strategy "replace" would act as today. Strategy "keep" would simply do nothing if the key already exists and just return a 201 to the client.