Integrate with Pipelinedb? about timescaledb HOT 21 CLOSED

We haven't tested integration with PipelineDB yet but once they release their extension, it shouldn't be that difficult.

We're working hard on the extension refactor, and we look forward to working with you guys to figure out an easy, legitimate way for users to synergize our extensions.

from timescaledb.

https://www.pipelinedb.com/blog/pipelinedb-1-0-0-high-performance-time-series-aggregation-for-postgresql pipelinedb is now an extension \o/

from timescaledb.

I too would LOVE LOVE LOVE this combination.

Honestly, it might be worthwhile to consider merging these two projects. Keep them functional on their own via configuration, but maintain and release them together.

from timescaledb.

Since we've released continuous aggregates and PipelineDB isn't going to be updating with new versions, I'm going to go ahead and close out!

from timescaledb.

Just heard from Jeff Ferguson | PipelineDB via email – they're actively working on the PostgreSQL extension refactor now and it should be completed this quarter.

from timescaledb.

Any news on this topic? Thanks!

from timescaledb.

Given that pipelinedb is now dead in the water(?), (the team joined confluent, and pipelinedb will be stuck at 1.0). I wonder how this will proceed. Will timescaledb pick up the features at pipelinedb?

from timescaledb.

We haven't tested integration with PipelineDB yet but once they release their extension, it shouldn't be that difficult.

from timescaledb.

Hate to post on an old issue, but perhaps this issue has been lost in the sands of time. Has there been any status updates since April 2017?

from timescaledb.

Hi @balupton we're still waiting on PipelineDB to refactor as a PostgreSQL extension. Uncertain as to their progress / status.

from timescaledb.

The other side of the fence.
pipelinedb/pipelinedb#1876

from timescaledb.

@derekjn @stalltron is this being looked at?

from timescaledb.

@pratikpparikh @windbender @eugene-bright @allan-simon @joshhopkins @balupton It's been a while coming, but looking into whether or not it makes sense for us to integrate better here. Anyone willing to share use cases on when they need PipelineDB + TimescaleDB? Also, if you'd prefer to do this over a call, I'm available at diana at timescale.com

from timescaledb.

@dianasaur323 I have time on monday to connect via call or email.

from timescaledb.

@pratikpparikh that would be great! Mind emailing me at diana at timescale.com so that I can schedule a time with you?

from timescaledb.

My use-cases:
Tracking data (e.g. from sensors) in TimescaleDB (and used for ad-hoc queries - detailed insights) but PipelineDB used for configured (= known queries) Dashboards and reporting. As you'd need the actual raw data to create useful views for a dashboard in an exploratory way, TimescaleDB is required.

Also you'd sometimes want to prove that the data in the generated reports are actually correct, which is why you'd need the raw data (or to fix some historical errors and rebuild the PipelineDB aggregates).

I am currently living with the triggers option but that feels inefficient, especially as will don't have support for transition tables, yet: #1084

Just FYI: having TimescaleDB as a backup store / history for the actual raw data (to this date: TimescaleDB v1.2.1 + PipelineDB v1.0.0), it seems to be best (performance-wise) to use a Continuous Transform on an input data stream which writes to the desired hypertable, e.g.:

-- create trigger function as usual for AFTER INSERT triggers:
CREATE FUNCTION fnt_forward_raw_page() RETURNS trigger
    LANGUAGE plpgsql AS
$$BEGIN
    INSERT INTO
        t_raw_data_page (c_timestamp, c_session, c_request, c_client, c_path)
    VALUES
        (NEW.c_timestamp, NEW.c_session, NEW.c_request, NEW.c_client, NEW.c_path)
    ;
    RETURN NEW;
END;$$
;

-- create continuous transform at the page stream:
CREATE VIEW v_trans_raw_data_page
    WITH (action=transform, outputfunc=fnt_forward_raw_page)
    AS
        SELECT c_timestamp, c_session, c_request, c_client, c_path FROM stream_tracking_page
;

...which is performing magnitudes better than the other way around (trigger at the hypertable to forward data to the stream):

INSERT 0 1838298
Time: 73442.984 ms (01:13.443)

(9 continuous views also and a TimescaleDB chunk creation included)

This also enables one to just turn off raw data writing but keep the aggregates.

from timescaledb.

my usecase is very similar to @ancoron more of a data lineage and master data management.

from timescaledb.

@jonathan-s we haven't decided to take up pipelinedb, but would love to see where the community takes this. That being said, definitely try out our new continuous aggregates feature to see if it meets part of your needs.

from timescaledb.

There really is work going on to eliminate the need for PipelineDB and have continuous aggregates a first-class citizen as of 1.3.0: #1179 (comment)

from timescaledb.

PinelineDB also recommends Timescale's continuous aggregates as an alternative.

from timescaledb.

It'd be great if timescaledb will also include pipelinedb aggregates:
http://docs.pipelinedb.com/aggregates.html

from timescaledb.