support bulk load by creating and ingesting SST files about incubator-pegasus HOT 3 CLOSED

可以参考TiDB的做法： https://zhuanlan.zhihu.com/p/55397024

from incubator-pegasus.

@hycdong 把设计文档初稿记录在这里

from incubator-pegasus.

The basic idea is just as zuoyan's comment. We firstly create rocksdb SST files on remote storage such as HDFS, and ingest those files into pegasus. Rocksdb support Creating-and-Ingesting-SST-files.

Basic design

1. Creating SST files on HDFS
   • Using RocksDBJava and Spark or MapReduce, write key-value pairs into SST files on HDFS

Tips:
We plan to support converting files in specific format or HBase HFile into pegasus key structure, more details about this are already in discussion.

2. Replica servers download SST files from HDFS
   • Client send start bulk load request(including app and remote file storage informations) to meta server
   • Meta server send bulk load request to primary replica of each partition
   • Primary replica receive request from meta server and download SST files and notify secondaries download files and report its download progress
   • Secondary replica download SST files and report download progress through group_check
   • Primary replica report download progress to meta servers
   • Meta server will wait all partitions including primary and secondaries finish downloading

Tips:

1. Primary and secondaries all download files from HDFS, so that we can avoid learn and data transfer through replicas which may affect normal read, write operations.
2. Server will restrict how many replica can download files at the same time
3. Download files task code has lower priority than normal write operations

In one word, we should guarantee downloading staging has minimal affect to normal user operations.

3. Replica servers ingestion SST files
   • Meta server send ingestion request to primary replica of each partition.
   • Primary replica will consider ingestion request as a special write request, execute 2pc staging
   • Write an empty request after primary commit to ensure secondaries commit ingestion request

Tips:
Rocksdb will prevent write operation during ingestion files, as a result, replica during ingestion staging will also reject all user operation. We should guarantee such time as short as possible.

Relation will other functions

• Write operations: ingestion staging will prevent user write operations
• Partition split: can not execute at the same time, because split will change partition count which affect SST files generation
• Access control: user should have write permission of target cluster
• Duplication, cold backup and cu calculation: WIP

from incubator-pegasus.