Code Monkey home page Code Monkey logo

pg_cryogen's Introduction

Build Status experimental

pg_cryogen

pg_cryogen is an append-only compressed pluggable storage.

Dependencies

pg_cryogen requires liblz4 and libzstd development packages installed.

Build

After clone just run in command line:

make install

Or if postgres binaries are not in the PATH:

make install PG_CONFIG=/path/to/pg_config

Using

pg_cryogen implements pluggable storage API and hence requires PostgreSQL 12 or later. To use it as a storage it needs to be specified as an access method while creating a table. E.g.:

create extension pg_cryogen;
create table my_table (...) using pg_cryogen;

pg_cryogen is an append only storage meaning that only INSERT and COPY commands are supported for data modification. Due to specifics of current implementation only inserts to a single table per transaction are possible. Also storage is optimized for bulk inserts and will create at least one separate block for each INSERT/COPY command.

Both index scan and bitmap scans are implemented.

Compression parameters

pg_cryogen offers two compression methods: lz4, which provides on average better compression/decompression speed, and zstd, which provides better compression ratio. Since pluggable storage API currently does not provide ways to set custom parameters for storages, the only way to set them is by using GUCs:

  • pg_cryogen.compression_method: possible options are lz4 and zstd (default).
  • pg_cryogen.lz4_acceleration: lz4 specific parameter which specifies how fast compression would be; valid values are [1..50], where 1 is better compression ratio and higher value is better speed (default is 1);
  • pg_cryogen.zstd_compression_level: zstd specific parameter for compression level; valid values are [-5..22] (default is 1).

Those GUCs can be set in either in postgresql.conf or for any particular session.

Internals

Please refer to a separate document for details.

pg_cryogen's People

Contributors

0xflotus avatar za-arthur avatar zilder avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pg_cryogen's Issues

ERROR: only heap AM is supported

I got an error when using it with timescaledb.

PostgreSQL Version: 12.3
TimescaleDB Version: 1.7.0
OS Version: CentOS 7.6
Kernel Version: 5.7.3-1.el7.elrepo.x86_64

CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
CREATE EXTENSION IF NOT EXISTS pg_cryogen;

CREATE TABLE "public"."log" (
"time" timestamptz(6) NOT NULL,
"device_id" TEXT NOT NULL,
"message_id" integer NOT NULL,
"project_id" TEXT NOT NULL,
"createtime" timestamp(6) NOT NULL,
"service_type" integer NOT NULL,
"message_type" integer NOT NULL,
"data_type" TEXT NOT NULL,
"metrics_name" TEXT NOT NULL,
"metrics_value" TEXT NOT NULL
) using pg_cryogen;

SELECT create_hypertable('log', 'time', chunk_time_interval => interval '1 hours', create_default_indexes => false);
ERROR: only heap AM is supported
SQL State: 0A000

Please let me know if there is a problem with my usage. Thanks~

cannot to compile

fix

diff --git a/compression.h b/compression.h
index 7d875c5..865b7ce 100644
--- a/compression.h
+++ b/compression.h
@@ -10,17 +10,17 @@ typedef enum
     COMP_ZSTD
 } CompressionMethod;
 
-int compression_method_guc;
-int lz4_acceleration_guc;
-int zstd_compression_level_guc;
+extern int compression_method_guc;
+extern int lz4_acceleration_guc;
+extern int zstd_compression_level_guc;
 
-char *cryo_compress(CompressionMethod method,
+extern char *cryo_compress(CompressionMethod method,
                     const char *data,
                     Size *compressed_size);
-bool cryo_decompress(CompressionMethod method,
+extern bool cryo_decompress(CompressionMethod method,
                      const char *compressed,
                      Size compressed_size,
                      char *out);
-void cryo_define_compression_gucs(void);
+extern void cryo_define_compression_gucs(void);
 
 #endif /* __COMPRESSION_H__ */

too much memory is required for hign levels of compressions

I tested insert 1M rows

create table test(a int);
create table test2 (like test) using pg_cryogen;
insert into test select random()*100000 from generate_series(1,1000000);

with

set pg_cryogen.zstd_compression_level=22;

the statement

insert into test2 select * from test;

requires about 4GB RAM (original table has 35MB, compressed table has 27MB).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.