dbiir / realtimeanalysis Goto Github PK

Make plan based on given metadata

Read source code of Hive connector. Up to down. Focused on how to provide record set suitable for Presto.

Add support for PARTITION, LOAD and etc. Details

Convert received SQL queries into Presto-style SQL queries, and get entityId, timestamp start and timestamp end in user query, then check meta store for related HDFS files.

Read source code of example http connector. Grasp the main view of connector implementation.

Using Hive API, create Hive external table and query through Presto engine.

Read source code of base jdbc connector. Just for reference. Focused on how to get metadata.

A simple interactive shell. Based on the glued code, receiving queries and returning results (along with running time statistics.)

A simple web ui where user can submit queries, and return results. More importantly, this ui can demonstrate how queries are running like the web ui of Presto.

Basic implementation. Support reading parquet file. Basic skip reading support without optimisation.

Using symlink instead of copy to group files into the same folder

Read source code of Hive connector. Up to down. Focused on how to provide record set suitable for Presto.

Add support for PARTITION, LOAD and etc. Details

Design connector modules and interfaces. Ready for implementation

Glue everything up. Make a whole module, ready to accept queries and return results based on our own meta server.