TL;DR;
Remove defining webhooks by custom ConfigMap and switch to CRs - Cluster(WebHook) and Cluster(WebHooksScenario).
Reasons
At the moment webhooks are applied in Rafter by custom ConfigMap or inline in (Cluster)Assets (this is not a problem). Problems with ConfigMap (and not only):
- webhooks (for
(Cluster)AssetGroups) are applied in global scope (in meaning: applied for all cluster and namespaces resources), and user cannot disable webhook for appropriate ClusterAssetGroup or AssetGroup.
- poor configuration for scenario (in meaning: order in which Rafter processes them) - at the moment we have hardcoded this part: first is mutation, then validation, and at the end metadata (extractor).
- we haven't Procedural webhook for custom action (not only for mutation, extractor or validation) - for examples: request to another service if asset will be
md file. So from my perspective, user should have possibility to write custom hook.
- we have hardcoded external ports through which Rafter communicate with services (and user cannot change it) - I would like to ask for correction because I did not find anything about it in the code, but I have heard it several times and I am not sure now whether it is true
New types
Rafter would have such webhooks:
- Mutation webhook - modifies fetched assets before the Asset Controller uploads them into the bucket.
- Extraction webhook - extract metadata from assets and inserts it under the
status.assetRef.files.metadata field in the Asset/ClusterAsset CR.
- Procedural webhook - perform custom actions like checking if content of file meets some criteria (like an old validation hook).
and CRs pointed to character of webhooks:
- ClusterWebHook/WebHook CR - specific characteristic of webhook with fields:
- spec.type - type of webhook (types are describe above):
mutation | extraction | procedural.
- spec.sourceTypes - array of types admitted to the webhook (based on type field of source in
(Cluster)AssetGroup CR and in (Cluster)Asset - we must also fix it, because we have only type field on sources of AssetGroup CR). OPTIONAL field, because user can allows all types.
- endpoint - endpoint to perform action. Example
https://example.com/v1/convert.
- spec.serviceRef - reference to service which exposes endpoint for mutation/extraction/other actions.
- spec.serviceRef.name - name of service.
- spec.serviceRef.namespace - namespace of service.
- spec.serviceRef.endpoint - endpoint of service to perform action.
- spec.parameters - additional parameters (in object) which are passed to endpoint. OPTIONAL field.
- spec.filter - determines what files are to be sent from the asset to the webhook by regex. OPTIONAL field.
NOTE: One of spec.endpoint or spec.serviceRef must be specified. Otherwise an error will occur.
Examples:
apiVersion: rafter.kyma-project.io/v1beta1
kind: ClusterWebHook/WebHook
metadata:
name: asyncapi-converter
namespace: kyma-system # if kind is WebHook
spec:
type: "mutation"
sourceTypes:
- asyncapi
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: barapiVersion: rafter.kyma-project.io/v1beta1
kind: ClusterWebHook/WebHook
metadata:
name: asyncapi-converter
namespace: kyma-system # if kind is WebHook
spec:
type: "mutation"
sourceTypes:
- asyncapi
serviceRef:
name: rafter-asyncapi-service
namespace: kyma-system
endpoint: "/v1/convert"
filter: "\\.yaml$"
parameters:
foo: bar
- ClusterWebHooksScenario/WebHooksScenario CR - scenario of webhooks (in meaning: order in which Rafter processes them):
- spec.scenario - list of webhooks pointed by name. Element of list must contains:
- webhookRef - reference to
ClusterWebHook/WebHook CR.
- webhookRef.kind -
ClusterWebHook or WebHook type.
- webhookRef.name - name of webhook.
- webhook - implementation of spec field of
ClusterWebHook/WebHook.
Example:
apiVersion: rafter.kyma-project.io/v1beta1
kind: ClusterWebHooksScenario/WebHooksScenario
metadata:
name: sample-webhook-scenario
namespace: kyma-system # if kind is WebHooksScenario
spec:
scenario:
- webhookRef:
kind: ClusterWebHook
name: markdown-front-matter-extractor
- webhookRef:
kind: WebHook
name: asyncapi-converter # WebHook in kyma-system namespace
# we can also create anonymous WebHooks in scenario with this same fields like in `ClusterWebHook/WebHook`
- webhook:
type: "mutation"
sourceTypes:
- asyncapi
serviceRef:
name: rafter-asyncapi-service
namespace: kyma-system
endpoint: "/v1/convert"
filter: "\\.yaml$"
parameters:
foo: bar
- webhook:
type: "mutation"
sourceTypes:
- asyncapi
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: barAs we can see, we have a limitation for scenarios:
- for
WebHooksScenarios we can bind ClusterWebHooks and WebHooks (of course with this same namespace as WebHooksScenario).
- for
ClusterWebHooksScenarios we can bind only ClusterWebHooks.
Failure of one hook in the scenario interrupts further operations and sets resource status to Failed.
Procedural webhook
I introduce Procedural webhooks, because they allow custom actions independent of the webhook type and I think that separating the validation process into a separate one (webhook type) is not necessary, so validation hook will be implemented in a procedural hook from now.
Binding WebHooks in (Cluster)Assets and (Cluster)AssetGroups
As I mentioned in Reason section, webhooks defined in ConfigMap are applied in global scope (in meaning: applied for all cluster and namespaces Assets CR) and user cannot disable webhook for appropriate ClusterAssetGroup or AssetGroup.
So, my proposition is inject webhooks by ClusterWebHooksScenario/WebHooksScenario and/or single ClusterWebHook/WebHook:
For ClusterAsset/Asset
apiVersion: rafter.kyma-project.io/v1beta1
kind: Asset
metadata:
name: sample-asset
namespace: kyma-system
spec:
...
webhooks:
- scenarioRef:
kind: WebHooksScenario
name: sample-webhook-scenario
- webhookRef:
kind: ClusterWebHook
name: markdown-front-matter-extractor
- webhook:
type: "mutation"
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: barFor ClusterAssetGroup/AssetGroup
apiVersion: rafter.kyma-project.io/v1beta1
kind: AssetGroup
metadata:
name: sample-assetgroup
namespace: kyma-system
spec:
...
webhooks:
- scenarioRef:
kind: WebHooksScenario
name: sample-webhook-scenario
sources:
- ...
webhooks:
- webhookRef:
kind: ClusterWebHook
name: markdown-front-matter-extractor
- webhook:
type: "mutation"
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: barIn case, when user defines webhooks for single assets, then webhooks injected in ClusterAssetGroup/AssetGroup will be completely overwritten for them (this same behavior like now).
Service specification requirements
-
service for Mutation webhook must expose endpoints that:
- accept parameters and content properties, where parameters is additional parameters defined in
ClusterWebHook/WebHook CR or inline in ClusterWebHooksScenario/WebHooksScenario CR, and content is a content of file.
- return the
200 response with new file content.
- return the
304 response informing that the file content was not modified.
-
service for Extraction webhook must expose endpoints that:
- accept parameters and content properties, where parameters is additional parameters defined in
ClusterWebHook/WebHook CR or inline in ClusterWebHooksScenario/WebHooksScenario CR, and content is a content of file.
- return the
200 response confirming that validation succeeded.
- return the
422 response informing why validation failed.
-
service for Procedural webhook must expose endpoints that:
- accept parameters and content properties, where parameters is additional parameters defined in
ClusterWebHook/WebHook CR or inline in ClusterWebHooksScenario/WebHooksScenario CR, and content is a content of file.
- return the
2** response with custom message about successful operation.
- return the
4** response with custom message about failed operation.
Comments and suggestions are welcome!
I am still thinking about the 4th type of webhook: StatusWebhook, which would perform custom action like Proceural, but based of status of (Cluster)Asset or (Cluster)Asset, but I know it can be done in a different way, unrelated to the Rafter himself, so for now I abandoned investment in this case.
In addition, we must define the option of defining default hooks per namespace or cluster (at the moment we have injecting them in all Cluster(AssetGroup) by ConfigMap). A labels probably will be the best for that case, but we can think about that together :)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent