Comments (17)
cccs-rs
commented on August 12, 2024
1
We could probably make it a global configuration option via services.registries.
https://cybercentrecanada.github.io/assemblyline4_docs/odm/models/config/#serviceregistry
This will be more a global configuration than at a per DockerConfig change. So as a long as the configuration is present for the deployment, it should use the appropriate settings per container registry endpoint.
@cccs-douglass thoughts on whether or not we want to go this route?
from assemblyline.
cccs-rs
commented on August 12, 2024
1
Since we're basing this authentication flow off a pre-existing standard it should be fine to go with this route. 😁
from assemblyline.
ed4wg
commented on August 12, 2024
1
Hi @cccs-rs - any update on this one? The reason I ask is I was able to publish our hatching-triage AL service and I'd like to make it available in the community services list. However, it's hosted on ghcr (as that's what is available to us) and as mentioned above, the only way i've found to make it work for a standard install of AL is to hardcode the version in the manifest when installing or updating.
from assemblyline.
cccs-rs
commented on August 12, 2024
1
Sorry, have been pulled away from this on more pressing tasks. Should be able to revisit it and complete the integration soon!
from assemblyline.
cccs-rs
commented on August 12, 2024
1
Good catch! I didn't account for the difference in schema versions for the Docker Registry API:
https://docker-docs.uclv.cu/registry/spec/manifest-v2-2/
But since it got far enough to retrieve the image tags (hopefully with no warning log saying otherwise), I'm convinced that the feature is working as it should. We can include this as part of a new stable release on Monday.
The error pertaining to getting the OS is irrelevant once merged into the stable branch (was related to an old feature that's yet to be complete but was in testing: #38).
from assemblyline.
ed4wg
commented on August 12, 2024
1
Looks like it's working. Thanks for getting this feature added! 🥳
from assemblyline.
cccs-rs
commented on August 12, 2024
Sounds similar to this issue: #100
from assemblyline.
ed4wg
commented on August 12, 2024
Yeah i agree they are similar. One additional thing this issue requests is to allow the public (no-creds) option to work for the ghcr.io registry.
from assemblyline.
cccs-rs
commented on August 12, 2024
This release is available for testing: https://github.com/CybercentreCanada/assemblyline/releases/tag/v4.5.1.dev19
from assemblyline.
ed4wg
commented on August 12, 2024
I tried updating the updater deployment to 4.5.1.dev20 (dev19 was gone) but it's not working.
Looking at the diff i noticed a few things: CybercentreCanada/assemblyline-core@master...feature/registry_token_auth
This approach will require the user to derive the bearer token and put it in the password field. I don't know if it can be assumed that the token will remain static. In the ghcr example, the github PAT token would of course remain static until the user changes it, but i'm not sure the bearer token will.
e.g.
bearer_token=$(curl -s "https://ghcr.io/token?service=ghcr.io" -u "${username}:${ghcr_token}")
I did derive the bearer token and then placed it in the password field. The updater is giving me this error:
Crash in dispatcher: container_versionsTraceback (most recent call last):
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/server_base.py\", line 275, in with_logs
fn(*args, **kwargs)
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/updater/run_updater.py\", line 678, in container_versions
image_name, tag_name, auth, _ = get_latest_tag_for_service(service, self.config, self.log, prefix=\"[CV] \")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/updater/helper.py\", line 225, in get_latest_tag_for_service
os = registry.get_image_os(image_name, tag_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/updater/helper.py\", line 83, in get_image_os
return json.loads(resp['history'][0]['v1Compatibility'])['os']
~~~~^^^^^^^^^^^
KeyError: 'history'
Also, any chance you're going to consider adding the bearer token flow for anonymous access as described in the original request? This would allow public access to registries like ghcr.io without the user having to create any token at all.
from assemblyline.
Hanochar
commented on August 12, 2024
Based on: https://distribution.github.io/distribution/spec/auth/token/
I think to set up anonymous access, you have to specify the authorization server to get the token from because there're no guarantees the same host that's hosting the registry is the same host to get the token from. In GHCR, it seems to use the same host, but in the example mentioned in the docs they were different.
from assemblyline.
ed4wg
commented on August 12, 2024
Any chance that can be a config? perhaps it defaults to the same, but allows you to specify an alternate if needed.
from assemblyline.
cccs-rs
commented on August 12, 2024
@ed4wg v4.5.1.dev67 should be a good candidate to test with 😁
from assemblyline.
ed4wg
commented on August 12, 2024
When i add the service manifest i get the below error in the ui logs.
Looks like image_name isn't being passed in to the DockerRegistry.__init__(). Seems HarborRegistry has the same issue.
I'm testing with this service which is hosted on ghcr.
Exception - /api/v4/service/
Traceback (most recent call last):
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 2190, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 1484, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_ui/api/base.py\", line 181, in base
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_ui/api/v4/service.py\", line 260, in add_service
_, tag_name, _, os = get_latest_tag_for_service(tmp_service, config, LOGGER)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/updater/helper.py\", line 227, in get_latest_tag_for_service
registry: ContainerRegistry = REGISTRY_TYPE_MAPPING[registry_type](**registry_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/updater/helper.py\", line 81, in __init__
token_url = f\"https://{token_server}/token?scope=repository:{image_name}:pull\"
^^^^^^^^^^
NameError: name 'image_name' is not defined
from assemblyline.
cccs-rs
commented on August 12, 2024
Should be patched in dev68.
from assemblyline.
ed4wg
commented on August 12, 2024
I'm getting the following error now.
Traceback (most recent call last):
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 2190, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 1484, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/flask/app.py\", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_ui/api/base.py\", line 181, in base
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_ui/api/v4/service.py\", line 260, in add_service
_, tag_name, _, os = get_latest_tag_for_service(tmp_service, config, LOGGER)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/updater/helper.py\", line 260, in get_latest_tag_for_service
os = registry.get_image_os(image_name, tag_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_core/updater/helper.py\", line 100, in get_image_os
return json.loads(resp['history'][0]['v1Compatibility'])['os']
~~~~^^^^^^^^^^^
KeyError: 'history'
I manually made the same http request as get_image_os() and this is what the output looks like.
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 18207,
"digest": "sha256:dd208bb83d364c99fb1612522e2ed349590c45ccb4728486d5dcb01a60ba7229"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 29124181,
"digest": "sha256: 8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 3507699,
"digest": "sha256:1103112ebfc46e01c0f35f3586e5a39c6a9ffa32c1a362d4d5f20e3783c6fdd7"
},
...
from assemblyline.
cccs-rs
commented on August 12, 2024
Feature should be included in 4.5.0.11 release
from assemblyline.
Related Issues (20)
- Feature Request: (Badlist) Show which badlist source was matched against in the heuristics. HOT 3
- Feature Request: (Badlist) Allow setting expiration time on Badlist IOCs HOT 7
- Feature Request: (Badlist) Allow setting a heuristic score per update source HOT 3
- bug in assemblyline charts with release 4.5.stable HOT 2
- Service Manifest Typo - DeobfuScripter & Swiffer HOT 1
- Deobfuscripter missed obvious IoC HOT 3
- Malware Archive: supplementary files are always included
- Finding related submissions doesn't work for supplementary files HOT 1
- Feature Request: Enhanced API Key Management, Auditing, and Expiry System
- EPUB identified as java/jar HOT 4
- EBook document preview HOT 1
- Configuration for Privileged vs Non-Privileged services HOT 3
- Feature Request: Regional Storage of Malware Samples Due to Legislative Changes HOT 3
- Sorting extracted files HOT 2
- Full-text search in submission files HOT 2
- S3 IAM role authentication
- Support for ASAR Archives HOT 2
- Allow setting some metadata when manually submitting a file HOT 2
- Cannot submit archived expired file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from assemblyline.