aimhubio / aimrecords Goto Github PK

• B+ tree data structure to hold indexes and pointers
• Provide an appropriate API for indexing

Where it will be used
In main panel:

• when selecting runs which have accuracy greater than 0.9
• when selecting specific epochs and iterations of runs
• selecting by any other field

Notes
aimrecords is not aware of the data model scheme and actual data that it receives and stores(it gets encoded binary records), so for secondary indexing the field key: value should be passed. An appropriate API should be provided e.g

writer.append_record(data, secondary_index={
  value: v, # it will hold secondary index by value
  epoch: e, # it will hold secondary index by epoch
  …
})

Problem
When using AimRecords in a distributed environments, non-ACID transactions of saving records would cause inconsistencies in data. When several nodes are trying to save into the same records log.

Solution
Implement ACID transactions.