kyma-project / rafter Goto Github PK
View Code? Open in Web Editor NEWKubernetes-native S3-like files/assets store based on CRDs and powered by MinIO
License: Apache License 2.0
Kubernetes-native S3-like files/assets store based on CRDs and powered by MinIO
License: Apache License 2.0
Description
Improve the AssetGroup custom resource lifecycle
document in the Rafter documentation - its structure should actually more resemble the structure of two other similar docs Asset custom resource lifecycle
& Bucket custom resource lifecycle
which show what happens when you create, modify, and remove the given custom resource.
So far, the document shows AssetGroup CR dependencies and details so perhaps it would be better to extract its current content to a separate document and provide the create, modify, and remove subsections with proper diagrams instead.
Description
Create a Helm Chart that will work without Kyma, but can be reused in Kyma.
AC:
All exposed values are described in chart Readme
Leader election is enabled when controller deployment has more than one instance
Globals from Kyma are not used
Chart should also install CRD by default, but there should be a way to disable it with a value like install_crd true or something like that
Once above points are done and accepted - Have it in Helm Repo (create PR). We know we depend on helm people to approve it, but still, we need to do it.
Reasons
Improve Rafter adoption in Kubernetes users.
Related issue(s)
kyma-project/kyma#5585
Description
Write integration tests for Rafter:
Reasons
This part of functionality hasn't been covered with integration tests. We have to make sure that it works properly.
This is a follow-up of kyma-project/kyma#2656
Description
After consolidation and migration to the new repository, makefiles probably not work and we don't have pipelines that build whole Rafter applications.
AC:
pr
folder in docker registryReasons
Having everything as automated and simplified as possible.
Related issue(s)
kyma-project/kyma#5585
Description
private
Reasons
There are use cases when you want to use the Asset Store to store files privately. You should be able to store files in a bucket and allow to read it only if the reader is authorized. Some kind of proxy is needed
Description
It's become very hard to update MinIO images as the dependencies are very old (minio/mc image is almost a year old), and the minio subchart is very old too (we have v2.5, newest is v5.0.15!!!). Minio-go is out of date also, but in this case I've tried updating it and no breaking changes were encountered.
The charts here and in Kyma repo are also old.
Reasons
If we want to develop this project in the future we need to work with most patched version.
Also the reason I've tried to update Minio was security -> minio images use vulnerable images, old alpine
s etc
Attachments
Description
Currently, the Asset Upload Service gives nice functionality for uploading content to Minio.
We need to have also an option to remove that uploaded content. For example, it could be done manually by making DELETE
call on resource endpoint or automatically garbage collected if none of ClusterDocsTopics/DocTopics are using that.
AC
Description
This task is a follow up to this PR which introduced the possibility of adding the source files of (Cluster)Assets CRs as ConfigMaps.
Find a good place in Rafter docs (here as it is not used in Kyma, or just a general remark in Kyma docs that such an option is possible) to document:
configmap
mode in Asset CRs and ClusterAsset CRs.Reasons
Document the new functionality that allows you to add sources of (Cluster)Asset CRs in a ConfigMap.
Description
After consolidation, we have to simplify the codebase:
Reasons
Improve the maintainability and extensibility of Rafter
Related issue(s)
kyma-project/kyma#5585
Description
Create a new main README.md
that explains what Rafter is. Follow the official template.
Related issue(s)
kyma-project/kyma#5585
Description
It happened really often that tests for Gateway mode are failing on CI. We should make them more resilient.
Things to consider:
Expected result
Tests are stable
Actual result
Tests are failing on communication with GCP/Azure
Steps to reproduce
Execute tests on PR
Troubleshooting
Description
Reasons
Sometimes you just need to store 1 or 2 assets using an application and creating a bucket separately might be overhead.
Description
Currently, the Asset has no readable and unique name. We need it to distinguish different assets in ServiceCatalog UI (tab titles, to be precise).
Please read kyma-project/console#1617 to deeper understand the problem.
Reasons
Related issue(s)
Rafter PR: #72
Documentation + Bumpss in Kyma: kyma-project/kyma#7864
Console issue: kyma-project/console#1617
@derberg commented on Tue Jun 18 2019
Description
Reasons
At the moment status of resources is not properly updated. If you for example remove one source, or do some other changes and at the end you know that Asset resource is processed, the status should switch to Pending and then again to Ready or Failed.
Description
Add tracing to Asset Store.
Reasons
It will help us to monitor, debug and optimize Asset Store code.
Attachments
Recently, I noticed that all issues were closed with the message:
We dont want to invest in rafter within Kyma organisation.
The rafter use case is no longer valid for kyma runtime
For example #50 (comment)
What that it means for the Rafter? Is there a plan to move it to a different organization? Or maybe it will be archived?
Since the beginning, Rafter was designed to work as a standalone project that doesn't require Kyma, so maybe it is no longer valid for Kyma, but it still can be used by someone in the community.
Description
AssetGroup is not only a new name for DocsTopic, but it also provided a new functionality - the possibility to provide Bucket reference for created Assets. Labels should be also improved.
More details: kyma-project/kyma#5585 (comment)
Reasons
Give a more control on buckets when working with AssetGroup
Related issue(s)
kyma-project/kyma#5585
Description
v0.1.0
- importantReasons
it is a mature project that is already used in production for over 6months so it is time to release it officially
Description
Reasons
Description
After migration from Asset Store and hCMS we left some mentions of asset-store in source code, due to the fact that we had to hurry and it just worked
. This has to be updated.
Reasons
Improvements in source code and quality of Rafter as a project
Description
Rafter manages buckets in MinIO and in remote storage by Bucket CR. Unfortunately, it is not possible to communicate with MinIO by Bucket CR name. The problem is that changes in buckets may generate new names in storage and client needs to monitor CR to be up-to-date.
We need to create a simple proxy service that will be able to translate Bucket CR to remote names and will be still compatible with S3 API. Thanks to that you can use any S3 client and communicate by Bucket CR names.
Whats need to be done:
Reasons
It will definitely simplify the usage of Rafter and links generated by it. You will no longer see ugly hashes in links and also the link will be static, not regenerated with changes on remote bucket names.
Another benefit is that it will be possible to use Rafter as storage for in-cluster Docker Registry.
All services have a problem with counting returned status codes other than 200
on the Grafana
dashboard.
I observed this issue when I was trying to cause errors on the Upload Service then the Grafana
didn't count their
Description
Reasons
We expect users to use Rafter with Minio Gateway. Minio supports many providers but we should always make sure they are supported within Asset Store and then we need to officially mention it in documentation
Description
This task is followup to kyma-project/kyma#5085 - what needs to be done:
Copy pasted stretch, which is now in scope too:
Reasons
Have metrics for controllers and services to be able to see how solution behaves and if the load handling matches the defined SLO
Description
Implement integration tests and pipelines for whole Rafter. Execute tests on Kind.
Be careful with credentials in Gateway modes.
AC:
Reasons
Automated validation if Rafter works.
Related issue(s)
kyma-project/kyma#5585
Description
Reasons
Make sure customers do not misunderstand the scope of the component and do not store private data with Rafter
Description
We will be implementing all of the logic for pipelines in bash, as it is faster to do so right now, and we really need to get it asap, but it is not a good solution overall. This POC shows that it is possible, and not that hard to achieve.
Reasons
Bash is sick magic, Go is more testable/easier to read/less error prone etc, etc, so we should migrate to that eventually in every aspect of development.
Description
This issue is a follow-up of issue 9 for adding README.md
files to Rafter components under the cmd
folder.
The following need to be verified and added in these README.md
files:
Issue(s)
Follow-up of #9
Description
After moving Rafter to a separate repo, it should have README.md
docs explaining Rafter services and the controller. Reuse the existing README.md
docs from the kyma
repository (+remember about renaming).
Related issue(s)
kyma-project/kyma#5585
Remove defining webhooks by custom ConfigMap and switch to CRs - Cluster(WebHook)
and Cluster(WebHooksScenario)
.
At the moment webhooks are applied in Rafter by custom ConfigMap or inline in (Cluster)Assets
(this is not a problem). Problems with ConfigMap (and not only):
(Cluster)AssetGroups
) are applied in global scope (in meaning: applied for all cluster and namespaces resources), and user cannot disable webhook for appropriate ClusterAssetGroup or AssetGroup.md
file. So from my perspective, user should have possibility to write custom hook.Rafter would have such webhooks:
status.assetRef.files.metadata
field in the Asset/ClusterAsset CR.and CRs pointed to character of webhooks:
mutation
| extraction
| procedural
.(Cluster)AssetGroup
CR and in (Cluster)Asset
- we must also fix it, because we have only type field on sources of AssetGroup
CR). OPTIONAL field, because user can allows all types.https://example.com/v1/convert
.NOTE: One of spec.endpoint or spec.serviceRef must be specified. Otherwise an error will occur.
Examples:
apiVersion: rafter.kyma-project.io/v1beta1
kind: ClusterWebHook/WebHook
metadata:
name: asyncapi-converter
namespace: kyma-system # if kind is WebHook
spec:
type: "mutation"
sourceTypes:
- asyncapi
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: bar
apiVersion: rafter.kyma-project.io/v1beta1
kind: ClusterWebHook/WebHook
metadata:
name: asyncapi-converter
namespace: kyma-system # if kind is WebHook
spec:
type: "mutation"
sourceTypes:
- asyncapi
serviceRef:
name: rafter-asyncapi-service
namespace: kyma-system
endpoint: "/v1/convert"
filter: "\\.yaml$"
parameters:
foo: bar
ClusterWebHook/WebHook
CR.ClusterWebHook
or WebHook
type.ClusterWebHook/WebHook
.Example:
apiVersion: rafter.kyma-project.io/v1beta1
kind: ClusterWebHooksScenario/WebHooksScenario
metadata:
name: sample-webhook-scenario
namespace: kyma-system # if kind is WebHooksScenario
spec:
scenario:
- webhookRef:
kind: ClusterWebHook
name: markdown-front-matter-extractor
- webhookRef:
kind: WebHook
name: asyncapi-converter # WebHook in kyma-system namespace
# we can also create anonymous WebHooks in scenario with this same fields like in `ClusterWebHook/WebHook`
- webhook:
type: "mutation"
sourceTypes:
- asyncapi
serviceRef:
name: rafter-asyncapi-service
namespace: kyma-system
endpoint: "/v1/convert"
filter: "\\.yaml$"
parameters:
foo: bar
- webhook:
type: "mutation"
sourceTypes:
- asyncapi
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: bar
As we can see, we have a limitation for scenarios:
WebHooksScenarios
we can bind ClusterWebHooks
and WebHooks
(of course with this same namespace as WebHooksScenario).ClusterWebHooksScenarios
we can bind only ClusterWebHooks
.Failure of one hook in the scenario interrupts further operations and sets resource status to Failed
.
I introduce Procedural
webhooks, because they allow custom actions independent of the webhook type and I think that separating the validation process into a separate one (webhook type) is not necessary, so validation hook will be implemented in a procedural hook from now.
As I mentioned in Reason
section, webhooks defined in ConfigMap are applied in global scope (in meaning: applied for all cluster and namespaces Assets
CR) and user cannot disable webhook for appropriate ClusterAssetGroup or AssetGroup.
So, my proposition is inject webhooks by ClusterWebHooksScenario/WebHooksScenario
and/or single ClusterWebHook/WebHook
:
For ClusterAsset/Asset
apiVersion: rafter.kyma-project.io/v1beta1
kind: Asset
metadata:
name: sample-asset
namespace: kyma-system
spec:
...
webhooks:
- scenarioRef:
kind: WebHooksScenario
name: sample-webhook-scenario
- webhookRef:
kind: ClusterWebHook
name: markdown-front-matter-extractor
- webhook:
type: "mutation"
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: bar
For ClusterAssetGroup/AssetGroup
apiVersion: rafter.kyma-project.io/v1beta1
kind: AssetGroup
metadata:
name: sample-assetgroup
namespace: kyma-system
spec:
...
webhooks:
- scenarioRef:
kind: WebHooksScenario
name: sample-webhook-scenario
sources:
- ...
webhooks:
- webhookRef:
kind: ClusterWebHook
name: markdown-front-matter-extractor
- webhook:
type: "mutation"
endpoint: https://example.com/v1/convert
filter: "\\.yaml$"
parameters:
foo: bar
In case, when user defines webhooks for single assets, then webhooks injected in ClusterAssetGroup/AssetGroup
will be completely overwritten for them (this same behavior like now).
service for Mutation webhook must expose endpoints that:
ClusterWebHook/WebHook
CR or inline in ClusterWebHooksScenario/WebHooksScenario
CR, and content is a content of file.200
response with new file content.304
response informing that the file content was not modified.service for Extraction webhook must expose endpoints that:
ClusterWebHook/WebHook
CR or inline in ClusterWebHooksScenario/WebHooksScenario
CR, and content is a content of file.200
response confirming that validation succeeded.422
response informing why validation failed.service for Procedural webhook must expose endpoints that:
ClusterWebHook/WebHook
CR or inline in ClusterWebHooksScenario/WebHooksScenario
CR, and content is a content of file.2**
response with custom message about successful operation.4**
response with custom message about failed operation.I am still thinking about the 4th type of webhook: StatusWebhook
, which would perform custom action like Proceural
, but based of status of (Cluster)Asset
or (Cluster)Asset
, but I know it can be done in a different way, unrelated to the Rafter himself, so for now I abandoned investment in this case.
In addition, we must define the option of defining default hooks per namespace or cluster (at the moment we have injecting them in all Cluster(AssetGroup)
by ConfigMap). A labels probably will be the best for that case, but we can think about that together :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.