zifter / clickhouse-migrations Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 5.0 104 KB

Simple migration tool for clickhouse database

License: MIT License

Makefile 5.25% Python 94.75%

clickhouse database migration python

clickhouse-migrations's Introduction

Hi there 👋

I'm Oleg! I'm a Software Developer based in Vilnius, Lithuania 🇱🇹, originally from Minsk, Belarus.

👷 Check out what I'm currently working on

zifter/u-trust-bot - Telegram Personal Assistent (1 month ago)
zifter/clickhouse-migrations - Simple migration tool for clickhouse database (1 month ago)
zifter/unireport - (1 month ago)

🔨 My recent Pull Requests

prepare for c4 on Cyclenerd/google-cloud-pricing-cost-calculator (2 weeks ago)
add c4 instance series on Cyclenerd/google-cloud-compute-machine-types (2 weeks ago)
Allow default db name #24 on zifter/clickhouse-migrations (1 month ago)

🚀 Latest releases I've contributed to

zifter/u-trust-bot v0.6.1 (3 days ago)
zifter/clickhouse-migrations v0.7.1 (1 month ago)
zifter/unireport v0.0.9 (1 month ago)

📄 Latest posts (personal blog in Russian)

Latest posts (medium in English)

Movie blooper in “American Gods” (2 years ago)
Automation of domain expiration checking with prometheus-stack in Kubernetes (2 years ago)

💬 Feedback

Say Hello, I don't bite!

📫 How to reach me

Blog: https://zifter.github.io/
Email: [email protected]

clickhouse-migrations's People

Contributors

Stargazers

Watchers

Forkers

shawnd solitaryurt infinipoint chartmogul dimoschi

clickhouse-migrations's Issues

Is there a hope to get rid of heavyweight pandas ?

Pandas is heavy to pull it to a tiny microservice container every time we just need to run migrations before the start of this microservice.
I see you use it in migrator.py to merge lists of migrations by certain criterion. May it be rewritten in pure python or by the means of lighter libs, please? ;)

apperantly sql semicolon delimeter is kind of broken atm

good time of the day,

so after 0.3.2 I had issues with same sql request files which worked fine with 0.3.0
for example this query

create table testing.migration 
(
    `app_id`     Int16,
    `created_at` DateTime64(3, 'Europe/Moscow') CODEC (Delta(2), ZSTD(22)),
    `uid`        String CODEC (ZSTD(22)),
    `user_level` Nullable(Int64) CODEC (Delta(2), ZSTD(22)),
    `session_id` Nullable(String) CODEC (ZSTD(22)),
    uid_old      Nullable(String) CODEC (ZSTD(22)),
    id           Int64,
    level_old    Nullable(Int64),
    social       Nullable(String) CODEC (ZSTD(22))
)
    ENGINE = MergeTree()
        PARTITION BY toYYYYMM(created_at)
        PRIMARY KEY app_id
        ORDER BY (app_id, created_at)
        SETTINGS index_granularity = 8192;

create table testing.refund 
(
    `app_id`       Int16,
    `created_at`   DateTime64(3, 'Europe/Moscow') CODEC (Delta(2), ZSTD(22)),
    `uid`          String CODEC (ZSTD(22)),
    transaction_id Nullable(String) CODEC (ZSTD(22)),
    id             Int64
)
    ENGINE = MergeTree()
        PARTITION BY toYYYYMM(created_at)
        PRIMARY KEY app_id
        ORDER BY (app_id, created_at)
        SETTINGS index_granularity = 8192;

separates like this for some reason

create table testing.migration         --that on is fine
(
    `app_id`     Int16,
    `created_at` DateTime64(3, 'Europe/Moscow') CODEC (Delta(2), ZSTD(22)),
    `uid`        String CODEC (ZSTD(22)),
    `user_level` Nullable(Int64) CODEC (Delta(2), ZSTD(22)),
    `session_id` Nullable(String) CODEC (ZSTD(22)),
    uid_old      Nullable(String) CODEC (ZSTD(22)),
    id           Int64,
    level_old    Nullable(Int64),
    social       Nullable(String) CODEC (ZSTD(22))
)
    ENGINE = MergeTree()
        PARTITION BY toYYYYMM(created_at)
        PRIMARY KEY app_id
        ORDER BY (app_id, created_at)
        SETTINGS index_granularity = 8192;

create table testing.refund 
(
    `app_id`       Int16,
    `created_at`   DateTime64(3, 'Europe/Moscow') CODEC (Delta(2), ZSTD(22)),
    `uid`          String CODEC (ZSTD(22)), 
    transaction_id Nullable(Strin   --this one is missing some if the lines|

anyways, please ignore this if you know its on my end
ty for reading