mongey / ksql Goto Github PK
View Code? Open in Web Editor NEWA barebones go client for interacting with Confluent's KSQL Server
A barebones go client for interacting with Confluent's KSQL Server
DROP
caused by the persistent queries dependencies needing prior termination for the STREAMS
& TABLES
instantiated with CREATE AS SELECT
statement.CREATE AS SELECT
fails on DROP
because their persistent queries needs to be terminated before the drop.TERMINATE
the QUERIES
.Running a DROP
on a STREAMS
or a TABLES
SHOULD terminate the underlying persistent query before the drop if present BUT SHOULD NOT manage cascading drops as this is a resource dependency issue related to Confulent KSQL.
DESCRIBE
query for STREAMS
& TABLES
information listing.TERMINATE
query for QUERIES
termination.Run DESCRIBE
on the resource
In order to check if there is persistent queries related to a STREAM
or a TABLE
, the KSQL client
should check the resource status with the DESCRIBE
query before running a DROP
operation.
As stated in the doc
Query:
{
"ksql": "DESCRIBE [RESOURCE_NAME];",
"streamsProperties": {}
}
Result:
[
{
"@type": "sourceDescription",
"statementText": "DESCRIBE [RESOURCE_NAME];",
"sourceDescription": {
"name": "[RESOURCE_NAME]",
"readQueries": [
/* => resource dependencies */
/* Other queries listening for data */
],
"writeQueries": [
{
"sinks": [
"[RESOURCE_NAME]"
],
"id": "[QUERY_ID]",
"queryString": "CREATE ..."
}
],
"fields": [
{
"name": "ROWTIME",
"schema": {
"type": "BIGINT",
"fields": null,
"memberSchema": null
}
},
{
"name": "ROWKEY",
"schema": {
"type": "STRING",
"fields": null,
"memberSchema": null
}
},
{
"name": "[CUSTOM_NAME]",
"schema": {
"type": "[CUSTOM_TYPES]",
"fields": "[CUSTOM_FIELDS]",
"memberSchema": "[CUSTOM_MEMBER_SCHEMA]"
}
},
...
],
"type": "[STREAM|TABLE]",
"key": "[value from the `WITH KEY` option]",
"timestamp": "",
"statistics": "",
"errorStats": "",
"extended": false,
"format": "[AVRO|JSON|DELIMITED]",
"topic": "[THE_TOPIC_RELATED_TO_THE_OUTPUT_DATA]",
"partitions": 0,
"replication": 0
}
}
]
From a code perspective, this should look like this:
sourceDescription, err := c.Describe(RESOURCE_NAME)
if err != nil {
return err
}
Check if there's some read queries
Read queries should be perceived as downstream dependencies and should therefore imply a CASCADE DROP
. As this behavior is not intended and is an issue related to the KSQL development, the DROP
should fail under this condition, specifying that a subset of resource depends on the current resource.
Concretely, from the received payload from the operation above, the code should look like this:
if len(sourceDescription.ReadQueries) > 0 {
dependency := sourceDescription.readQueries[0].sinks[0]
return fmt.Errorf("could not drop '%s', '%s' needs to be dropped before.", RESOURCE_NAME, dependency)
}
Try to terminate the write queries related to the current resource.
To determine if a persistent query is running for a certain resource, we would expect the query sinks to be an array containing exclusively the resource name. If this is not the case, the function should return an error notifying that some underlying queries needs to be looked at before the resource can be dropped.
On the other hand, if the query is only related to the resource to be dropped, then the query should be terminated.
Because the writeQueries
object is an array, the routine must be applied to all of its entries.
for _, q := range sourceDescription.WriteQueries {
expectedSinks := []string{RESOURCE_NAME}
if q.Sinks != expectedSinks {
return fmt.Errorf("could not drop '%s', the query '%s' should sinks '%v' but '%v' was found instead.", RESOURCE_NAME, q.ID, expectedSinks, q.Sinks)
}
err := c.Terminate(q.ID)
if err != nil {
return err
}
}
Drop the resource
After the precedent steps, it should now be safe to drop the resource. Error occuring at this point should only refer to standard errors caused by network connection or invalid operations flow.
Looking into the cm-update-confluent
the final function return in the code should remain:
return c.qTOerr(req)
Currently there is no obvious way to add Authorization headers to requests. From what I can see KSQL supports Basic Auth.
Use case: Mongey/terraform-provider-ksql#9
Use case: Mongey/terraform-provider-ksql#9
Currently there are two functions to instantiate a client, both of which hide access to the HTTP client configuration.
It would be nice to be able to add a TL configuration (for example: path to CA certs, or to ignore TLS verification).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.