Code Monkey home page Code Monkey logo

Comments (1)

krasserm avatar krasserm commented on August 16, 2024

From a conversation:

  • Let all journals use the same snapshot storage API ... This is currently not the case: for example, LevelDB uses java.io.File whereas HBase uses org.apache.hadoop.fs.FileSystem.
  • The Hadoop FileSystem abstraction has implementations for local filesystem, HDFS, S3, FTP etc.
  • In a first step we let all journals use the Hadoop FileSystem abstraction and choose a reasonable default implementation for each journal. For example, RawLocalFileSystem for LevelDB, NativeS3FileSystem for DynamoDB etc. Applications can override these defaults, if needed. This way we could implement snapshotting for all journals very quickly. I just need to move the Hadoop FileSystem snapshot support from hbase to a common place and other journals can re-use it by providing different root URIs.
  • In a second steps we could introduce an extension to local snapshot storage. ... the local storage could serve as a cache and a background process/thread could do the the synchronization with a backup location (HDFS, S3, FTP etc).
  • The above strategy would make local caching an option and is an implementation detail of local snapshot storage (except that applications need to configure which backup storage they want to have).

from eventsourced.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.