Comments (9)
Hey everyone!
I took a shot at creating a new catalog module that breaks the AWS Org processor into its own module and Entity Provider here: #24994. I wanted to comment here for context as I believe it is related.
from backstage.
I noticed that the GitHub and GitLab catalog modules have broken out their "organization" providers in to a separate modules/packages that basically just declares the backend module for those providers but the implementation remain in the main GitHub/GitLab packages.
What I'm unclear about here is whether we should:
- Break these processors out to separate packages similar to above (seems like a lot of work?)
- Rebrand the existing
catalogModuleAwsS3EntityProvider
as something likecatalogModuleAws
and register all of the S3 entity provider plus the processors in the same module. - Something else?
from backstage.
yes, the way to go is to create a new module for each of the processors.
It seems more work from a plugin's author perspective but it will much easier for adopters to install the modules using the new backend system.
from backstage.
To play devils advocate a bit: I'm not sure thats better for anyone? For the maintainers its more overhead and for the adopters its more steps to import multiple packages and likely more documentation to look through. And I'm not 100% clear on the benefits of splitting them. I understand a boundary has to be drawn somewhere and if this is the general guidance then I get where its coming from but looking specifically at a case like this I'm not sure I understand what anyone gains from the separate packages.
But if thats the recommended direction I can start to figure out what this will look like.
from backstage.
The way the backend system is made, it wants to consume the default export of packages. And there can only be one default export of course.
I wonder if this is the point where we could start talking about subpath exports. Ping @Rugvip
from backstage.
Perhaps related to this I was browsing #10183 and wondered whether these processors would also benefit from being converted to providers.
For example with many AWS customers I expect that their accounts change infrequently enough that being able to control the interval for the AWS Organizations processor would reduce the amount of API calls.
I also see that ingesting GKE clusters as Resource
kinds is implemented as a provider instead of a processor in GkeEntityProvider. The EKS version is a processor.
It could be that the path here is to port these to providers and export those rather than exporting the existing processors. These processors are largely undocumented anyway. However I'm not sure what the downsides of switching to providers would be.
from backstage.
Yep, entity providers is the preferred pattern for ingesting additional entities. Definitely makes sense to migrate any such processors that don't currently have an equivalent provider.
In many cases we also keep supporting providers for backwards compatibility, and it may be that we want to start deprecation some of them.
Regarding the export structure in the new backend system, I think we can keep any processor modules on separate named exports, and stick to exporting the provider modules on the default exports. This effectively means that you need a bit of extra code to wire up a processor discovery module in the new system, but that's fine because it isn't the recommended approach.
from backstage.
For my own education: are there negative consequences to just adding these processors in the existing module alongside the S3 provider? Not introducing more modules.
from backstage.
@niallthomson Yep, right now you can register any type of location through a Location
entity. That means that anyone with the ability to add a Location
entity to the catalog can also set up any form of ingestion with the available processors in the catalog. Worst case that can lead to information leakage as the catalog might be authorized to read content that the user is not.
The typical setup of custom processors is through catalog.locations
config though, and I think it would be very reasonable to lock down what location types you're able to define in a Location
entity. If we did that I don't see any reason we couldn't register all processors by default. Then again, discovery processors aren't really a thing we want to be encouraging in the first place, entity providers do the same thing but better.
from backstage.
Related Issues (20)
- 🐛 Bug Report: Not able to add `common-library` plugin using `yarn add` HOT 4
- catalog error Error: secretOrPrivateKey must be an asymmetric key when using RS256 HOT 1
- 🐛 Bug Report: GCP IAP Provider has wrong provider name in documentation HOT 1
- 🐛 Bug Report: pymdownx.snippets `base_path` setting does not work for base directory of repository HOT 1
- 🐛 Bug Report: Error in <Select> component that has empty string as placeholder HOT 8
- 🐛 Bug Report: GithubIntegration Doc links to itself HOT 3
- 🐛 Bug Report: relations consumesApi and providesApi not showing at same time HOT 1
- 🐛 Bug Report: Scaffolder Actions failing with JWTClaimValidationFailed HOT 1
- 🐛 Bug Report: Adding `pagination` property on `<CatalogIndexPage />` removes "Actions" column HOT 3
- 🚀 Feature: @backstage/plugin-auth-backend-module-oidc-proxy-provider
- 🐛 Bug Report: Unable to unregister a component HOT 1
- 🚀 Feature: introduce lifecycle in catalog-model for Resource entity kind
- 🐛 Bug Report: frequency format ISO 8601 not working for schedule
- Software catalogs integration with gitlab - facing error
- The `T` property on the `ExtensionPoint` type should always be set to null in runtime HOT 2
- [Core] Evaluate removing the callback form on the `TestBackendOptions.features` property
- [Helper] Prefix `MockDirectoryOptions` with the `createMockDirectory` method name
- [Lifecycle] Inline types on the `LifecycleService` or ensure that the separate types are nicely prefixed (e.g. `LifecycleServiceShutdownOptions`) HOT 5
- 🐛 Bug Report: Kubernetes plugin - cluster selection annotation doesn't work anymore HOT 2
- [Permissions] Replace `token` with `credentials` on the `PermissionsServiceRequestOptions` type HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from backstage.