Comments (19)
I'd love to see this potentially be extended into either loading common libs from ConfigMaps as is kinda-sorta the case for vanilla OPA, or else see a CRD created for libs. If I define a lib in one ConstraintTemplate, can it be referenced by name from another ConstraintTemplate?
Not currently. Gatekeeper currently requires each template be self-contained. The benefit of that is that you avoid dependency conflicts.
Imagine one template relied on a function:
less_than(a, b)
And another template relied on a newer version of that function that allowed fuzzy matching:
less_than(a, b, delta)
The example is a bit contrived, but there are plenty of examples of dependency conflicts in the wild
libs
is a way to make it easier to embed hard-copies of a dependency into a template so that template consumers don't need to worry about conflicts happening live on their cluster.
There is also a potential security issue where one template would be able to maliciously extend another template's library if they had access to a shared namespace.
from gatekeeper-library.
+1 to supporting better code reuse.
We have the beginnings of that in the libs
field of the constraint template. That is intended so that code may be safely shared without worrying about naming collisions.
There is definitely a lot of tooling work to be done around automatically bundling common libraries into constraint templates as part of a build process.
from gatekeeper-library.
Only usage of libs
I could find is here is https://github.com/open-policy-agent/gatekeeper/blob/1312aee9bc3e772003f5113ce4e454be7367e8d7/test/bats/tests/templates/k8scontainterlimits_template.yaml
- Docs here on how libs work would be nice
- I could not find anything in the rego docs on
import
, runningopa test
ignoresimport
(does not fail / does not do anything). - Since I'm not keeping my rego code in yaml files, I'd need to make it also work when using
opa test
cli too.
... so ideally import
would work locally by loading in a relative file and when I generate my constraints I either generate them into libs
or dump helpers and code into the rego field.
from gatekeeper-library.
Found some example setup here https://github.com/plexsystems/konstraint/tree/main/examples
... so I think the idea is that:
- during local operation all files get loaded -> all packages are defined
- when generating the constraints the libs get passed in via an extra field so they are available too
from gatekeeper-library.
@grosser, yep that's the idea currently. Anything slated for potential re-use would go into libs
and the rego would then import it.
But as @maxsmythe points of, still going back and forth on the best way to more systematically import what you need, rather than import the world every time.
from gatekeeper-library.
FYI I ended up making small libs and then generating the policy by scanning for which imports are needed
libs = rego.scan(/^import data\.lib\.([^.\s]+)/).flatten(1).map do |lib|
File.read("policies/lib/#{lib}.rego")
end
from gatekeeper-library.
@grosser Konstraint does that as well. For every rego, it'll look at the import statements and add the imports to the lib section.
from gatekeeper-library.
I'd love to see this potentially be extended into either loading common libs from ConfigMaps as is kinda-sorta the case for vanilla OPA, or else see a CRD created for libs. If I define a lib in one ConstraintTemplate, can it be referenced by name from another ConstraintTemplate?
from gatekeeper-library.
This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
from gatekeeper-library.
this is similar to #206
from gatekeeper-library.
It seems like this issue should be on the main Gatekeeper project, no?
Regardless, adding libs to existing templates using kustomize or scripting is one thing, but doesn't seem like close to an ideal solution as it involves grossly expanding the size of the objects with lots of duplicate code. It seems like some kind of new CRD Library
is needed to house common code that can be imported into arbitrary templates. For example, a template could declare its Library
dependencies using label selectors or by name, and the controller makes those files available.
As for conflicts, it seems like each template should have its own scope (otherwise shouldn't those conflicts already exist? Say two templates declaring a rule with the same name). So if a template tries to extend the library, it doesn't influence any other template
The main issue I see (knowing nothing about the internals, really) is integration with gator
. Locally, if I declare a dependency, gator
needs to know how to load that dependency without talking to a cluster. Solvable, I think, by having a way to configure it to "gather" the relevant libraries locally from the repo so you can simulate the import clientside
from gatekeeper-library.
It seems like this issue should be on the main Gatekeeper project, no?
I think this is about the ability to re-use Rego in the writing process, not necessarily avoiding duplicating compiled code.
doesn't seem like close to an ideal solution as it involves grossly expanding the size of the objects with lots of duplicate code.
There is definitely a tradeoff where we are gaining a lack of need to worry about dependency conflicts (and security/speed as I'll get into in a second) for space. So far I haven't heard any complaints about memory footprint due to copied Rego, so definitely curious if that's a problem. By far the largest consumer of RAM is usually the informer cache for synced resources (at least on large clusters).
WRT the speed gain: each constraint template is compiled into a separate Rego environment, so they are unable to share code. The security/speed benefits of this relative to compiling everything to the same Rego environment (which we used to do) are:
-
When adding a new template, the entire corpus of Rego no longer needs to be recompiled, which was expensive
-
There is no need to sandbox templates from each other by statically analyzing the Rego to prevent over-broad access, which makes it easier to ensure templates cannot affect each other in multi-tenant environments.
-
We are able to offload common operations, like evaluating match criteria, to pure golang, which tends to be more performant (there are maybe ways to do this while still keeping everything in the same Rego environment, but coupled with the benefits to compilation time, this was a win)
See this doc for more info about the performance improvements.
It seems like some kind of new CRD Library is needed to house common code that can be imported into arbitrary templates. For example, a template could declare its Library dependencies using label selectors or by name, and the controller makes those files available.
Aside from raising the need to resolve dependency conflicts (maybe solvable if you, say, require the users to reference their dependency via semver), there would also be a usability concern here. How would these common libraries be packaged and redistributed?
As for conflicts, it seems like each template should have its own scope (otherwise shouldn't those conflicts already exist? Say two templates declaring a rule with the same name). So if a template tries to extend the library, it doesn't influence any other template
Currently each template has its own scope for its local code, but that doesn't change the problem of dependency conflicts, since we are talking about shared libraries, which implies each template would be referencing the same rego code. This is the same sort of problem as static vs. dynamic linking of libraries.
The "maliciously extend" mentioned in an above comment was less about whether a template has its own scope and more about if they are also importing a common scope (e.g. data.libs.common
). In the case where they are and data.libs.common
is in any way writeable (say by adding two libraries that have the same package path), then there is opportunity for exploitation. Preventing exploitation may be possible, but avoiding shared code altogether seems a stronger guarantee.
The main issue I see (knowing nothing about the internals, really) is integration with gator. Locally, if I declare a dependency, gator needs to know how to load that dependency without talking to a cluster. Solvable, I think, by having a way to configure it to "gather" the relevant libraries locally from the repo so you can simulate the import clientside
I think, at that point, the libraries could live next to, and be loaded with, the constraint templates (using the same methods a user is leveraging to load the templates into gator).
Finally, there is the elephant-in-the-room that is the CEL KEP. When dealing with language-specific features, we should be careful to remain generic so that we don't accidentally bake something into the system that makes it hard to adapt to the changing landscape. The use of common libraries, if adopted, would be one of those things that should be generic.
There are more pedestrian concerns such as code complexity (e.g. dealing with the edge cases of a multi-resource controller), that are surmountable, but definitely not trivial.
In summary, I don't think it's impossible to have shared libraries, but I'd definitely want to see that there is a real need, that the benefits outweigh the costs and get a sense of how the CEL KEP reshapes the environment before treading too deep into a design.
from gatekeeper-library.
Just for more background, we use a COTS product called Styra as a replacement for Gatekeeper. It's the main feature missing in Gatekeeper that we use heavily in Styra (in our case, we have ~1000 lines of Rego code which includes all the library code plus the business logic, so you can see the benefit of abstracting rules into a library). Now granted, Styra doesn't store policy as CRDs in the cluster. So there's a slightly different architecture and different concerns. But that's where I'm coming from
from gatekeeper-library.
if duplicating them in CRDs is not getting too large you should be able to do that
from gatekeeper-library.
It's the main feature missing in Gatekeeper that we use heavily in Styra (in our case, we have ~1000 lines of Rego code which includes all the library code plus the business logic, so you can see the benefit of abstracting rules into a library).
I'm curious if all policies use all library code, or if selective importing would limit the duplication? How many of those 1000 lines are library code?
Also, in terms of RAM usage, I'm curious how many constraint templates you'd be looking to use?
Assuming an average of 100 characters/line, 1000 lines is around 98 kB (though unsure what the compiled size would be, worth looking into), which would be require ~100 templates for 10 MB of RAM usage for the string representation. Let's assume 50 MB of RAM to account for duplicate caching and size of compiled code (it could be much better or worse than that, I haven't benchmarked shared Rego as TBH the template library has only a few examples that are much smaller than this).
These are all obviously very rough rules-of-thumb, but I'm interested in figuring out when the management complexity vs. infrastructure spend cost/benefit analysis starts to tip for most people.
At a macro level, some amount of code duplication is implicit in most containerized/sandboxed applications. Docker containers share a kernel, but have differing binaries/libraries/runtimes, for example. IMO it's the same issue here: what are the tradeoffs between operational complexity, operational fragility, and infrastructure cost?
I'm also curious... it seems like you're looking to use multiple templates, which implies you see value in sharding your existing code across templates vs. creating one mega-template. I can make guesses, but don't what to assume: what benefits are you looking for in switching to constraints/templates?
from gatekeeper-library.
if duplicating them in CRDs is not getting too large you should be able to do that
That's what we'll do for now, perhaps using something like Helm templates to generate everything. We have a little bit of work to do to migrate everything though so I haven't been able to test it yet to see if we hit any limitations
Also, in terms of RAM usage, I'm curious how many constraint templates you'd be looking to use?
That's a question I had as well. We have lots of use cases, not only for native Kubernetes objects, but lots of CRDs for different commercial and other products we use. Essentially, we use OPA to implement more secure multitenancy in our clusters where the controllers are lacking (typical examples are preventing cross-namespace references, validating schema where the CRDs are lacking, enforcing certain settings within object specs, deconflicting global settings like ingress hostnames, etc.)
I'm curious if all policies use all library code, or if selective importing would limit the duplication? How many of those 1000 lines are library code?
They almost all use some kind of library code. But I don't know enough yet to say how much duplication we'll need to introduce. For example, a common complexity is objects that have similar shapes. Think all objects that include Pod somewhere in their data -- Job, Deployment, StatefulSet, CronJob. We use some products that have similar characteristics where we need multiple rules to select the different varieties -- object A has the data at spec.x.y.z, object B has it at spec.a[].b[].c, etc. -- in some cases the nesting goes very deep and there are many different varieties. Some of these products also nest things like Pod or other native objects we want to govern.
what benefits are you looking for in switching to constraints/templates?
To be honest, mainly we would like to stop paying for Styra. We really only use it for Kubernetes policy and not any of its other features
from gatekeeper-library.
Think all objects that include Pod somewhere in their data -- Job, Deployment, StatefulSet, CronJob. We use some products that have similar characteristics where we need multiple rules to select the different varieties -- object A has the data at spec.x.y.z, object B has it at spec.a[].b[].c, etc
For the general model of "resources that wrap other resources", especially if those resources are created in the cluster, expansion templates may be helpful, letting you only worry about writing policies against pods. They also play a bit nicer with match criteria, which may improve the enforcement accuracy.
Definitely curious how things go. Keep us updated!
from gatekeeper-library.
@maxsmythe Thanks, I'll look into that. Not sure we're at a Gatekeeper version that has that alpha feature, but I suppose we can upgrade to try it out
from gatekeeper-library.
This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
from gatekeeper-library.
Related Issues (20)
- Consider validating pod generic ephemerals in K8sStorageClass HOT 2
- Consolidating Kubernetes PSP-related ConstraintTemplates into a Single Template for Streamlined Migration HOT 1
- bump mutate assign api version from alpha to v1
- Website generator appears to only retain the final mutation sample per directory HOT 2
- Any interest in policies/constraints that apply to custom resources? HOT 7
- Workflow Upload artifacts: overwrites the matrixed job logs HOT 1
- k8spsphostnetworkingports exemptImages does not allow hostNetwork HOT 4
- automount-serviceaccount-token ConstraintTemplate does not reflect ServiceAccount settings HOT 1
- Not able to create statefulset without storageclass with policy k8sallowedstorageclas is used HOT 1
- Should apparmor always view unconfined as complaint? HOT 4
- The example of disallowed/allowed ingress resources in the unique ingress host example has incorrect hostnames HOT 10
- Add colon in message for consistency
- Example of pod mutation adding init-container HOT 4
- Example k8scontainerlimits does not throw error for a deployment but does on a plain Pod creation HOT 2
- poddisruptionbudget policy query HOT 3
- K8sRequiredResources ConstraintTemplate doesn't work properly HOT 1
- `variables.anyObject` should be used for required labels CEL
- Add CEL code for PSP Policies in library
- Improve Rego testing for library HOT 1
- K8sPSPHostNetworkingPorts constraint template not handling exemptImages parameter properly HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gatekeeper-library.