Which problem is this feature request solving? The postgresql dest

Just waiting for permission to contribute per <div class="Box Box--condensed my-2"

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

feat: PostgreSQL destination delete stale requires sequential scan about cloudquery HOT 3 OPEN

kd7lxl commented on June 1, 2024 1

feat: PostgreSQL destination delete stale requires sequential scan

from cloudquery.

Comments (3)

kd7lxl commented on June 1, 2024 1

The scope of this request is to create an index to directly support the query used in

cloudquery/plugins/destination/postgresql/client/delete.go

Lines 18 to 25 in f649bed

    
           sb.WriteString("delete from ") 
        
           sb.WriteString(pgx.Identifier{msg.TableName}.Sanitize()) 
        
           sb.WriteString(" where ") 
        
           sb.WriteString(schema.CqSourceNameColumn.Name) 
        
           sb.WriteString(" = $1 and ") 
        
           sb.WriteString(schema.CqSyncTimeColumn.Name) 
        
           sb.WriteString(" < $2") 
        
           batch.Queue(sb.String(), msg.SourceName, msg.SyncTime)

This is not a generic request from the SDK to create indexes, since the SDK or source plugin is not responsible for the design of this delete query. The query is defined in plugins/destination/postgresql/client/delete.go, so the matching index will be created in plugins/destination/postgresql/client/migrate.go.

Supporting indexes for other columns via the SDK would be an interesting discussion, but beyond the scope here.

from cloudquery.

kd7lxl commented on June 1, 2024

Just waiting for permission to contribute per

cloudquery/CONTRIBUTING.md

Line 20 in 85f0d9b

    
           The CloudQuery framework, SDK and CLI are open source while plugins available under `plugins` are **open core**, hence not all contributions to plugins directory will be accepted if they are part of the commercial plugin offering - please [file an issue](https://github.com/cloudquery/cloudquery/issues/new/choose) before opening a PR.

from cloudquery.

erezrokah commented on June 1, 2024

Hi @kd7lxl, thanks for suggesting a contribution. Maybe you can start with a draft PR to show how the implementation would look like? We'll need to also discuss this internally if we want the SDK schema to tell destinations if they should create indexes for columns, or it should be coded in the destination per column

from cloudquery.

Recommend Projects

feat: PostgreSQL destination delete stale requires sequential scan about cloudquery HOT 3 OPEN

Comments (3)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	sb.WriteString("delete from ")
	sb.WriteString(pgx.Identifier{msg.TableName}.Sanitize())
	sb.WriteString(" where ")
	sb.WriteString(schema.CqSourceNameColumn.Name)
	sb.WriteString(" = $1 and ")
	sb.WriteString(schema.CqSyncTimeColumn.Name)
	sb.WriteString(" < $2")
	batch.Queue(sb.String(), msg.SourceName, msg.SyncTime)