Comments (32)
My AWS case is already escalated to the internal ec2 team. Will keep you posted!
from test-infra.
@renilthomas AWS has updated the AMIs and the updated AMI uses kernel version 5.4.238-148.347.amzn2.x86_64
. We did an initial testing and falco probe is getting downloaded properly and is working properly now.
cc: @FedeDP
from test-infra.
I still don't get where does this kernel version come from; doing a
wget amazonlinux.us-west-2.amazonaws.com/2/extras/kernel-5.4/latest/x86_64/mirror.list
yelds a mirror.list file with
http://amazonlinux.us-east-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61
as a content. That is what kernel-crawler is discovering too.
There should not be any other mirror, as far as i could find, for kernel-5.4.
from test-infra.
We are also facing similar issue with latest AWS EKS AMI for 1.23. We have tried to downgrade the ami and it seems to be working.
@renilthomas Thanks for working with the ec2 team. Please share if you hear any updates.
from test-infra.
So, this was an issue on the AWS side right?
Great to hear it's getting fixed!:)
from test-infra.
Hi All, Just wanted to update that we have tested falco versions 2.0.18,2.5.5,3.0.0 and 3.1.3 with k8s version 1.22 to 1.26. And the falco daemonset is working as expected with ebpf probe getting loaded and we are getting threats data as well
from test-infra.
Hi!
We have 5.4.238-148.347
: https://download.falco.org/driver/site/index.html?lib=4.0.0%2Bdriver&target=amazonlinux2&arch=x86_64&kind=kmod&search=falco_amazonlinux2_5.4.238-148.347.amzn2.x86_64_1.ko
I think you need to upgrade your kernel to latest version! We did never build 5.4.238-148.346
unfortunately; most probably it was immediately patched to .347
and since our pipeline runs weekly, we only discovered and built the latter.
from test-infra.
I got this URL from the instance with the specific kernel version
from test-infra.
In the meantime, i am trying to figure out what are we missing :)
Also, @EXONER4TED (gentle tag ahah) is the 👑 when finding weird kernel-crawler issues :D
from test-infra.
oh right, checking
from test-infra.
You are right, there seems to be none for 346 😞
from test-infra.
That is correct, they responded to my issue awslabs/amazon-eks-ami#1266 (comment)
And I had also escalated it within AWS and last I heard earlier was they were working on the fix.
So, all good now @FedeDP
from test-infra.
uhm thank you for reporting this, probably the kernel crawler still searching for this kernel @FedeDP
from test-infra.
The EKS latest AMI release has pinned 5.4.238-148.346.amzn2
for k8s v1.23 or below. Can the kernel crawler build based on a specific version?
At this stage, there are some critical vulnerabilities that is addressed in this new AMI but we have only two options:
- Disable falco and upgrade AMI
- Upgrade k8s - which obviously requires scheduling a maintenance and planning towards it.
from test-infra.
Hi! It seems like we fail to crawl the kernel; i will look into this asap :)
from test-infra.
Thank you @FedeDP 🙇
from test-infra.
Mmh my guess is that the 5.4.238-148.346.amzn2
kernel is not present in any mirror anymore; as i said previously, we can only scrape 5.4.238-148.347.amzn2
.
Are you able to share the url for the 5.4.238-148.346.amzn2
package?
from test-infra.
For reference, this is what the crawler finds instead:
http://amazonlinux.us-west-2.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61/../../../../../../blobstore/fa5caa71652bca6f4a8aeff9356d18bd1785d8620080398768acbb53a7fb5756/kernel-devel-5.4.238-148.347.amzn2.x86_64.rpm
from test-infra.
Yes i have seen that running kernel-crawler myself. Is it possible to build with that URL I shared? Thank you in advance!
from test-infra.
The quickest fix, since you have the full URL, is to add a new config on test-infra for it by eg: copy/pasting the 347
one: https://github.com/falcosecurity/test-infra/blob/master/driverkit/config/4.0.0%2Bdriver/x86_64/amazonlinux2_5.4.238-148.347.amzn2.x86_64_1.yaml
If you open a PR (please add it for all 3 supported driver versions!) i will be happy to accept it :)
from test-infra.
Brilliant, on it.. Thanks @FedeDP
from test-infra.
Wait! i just notice: you are sharing the kernel URL, not the kernel-devel
one.
Perhaps there is no kernel-devel package for it?
from test-infra.
So, did they push an updated kernel without pushing updated headers? That would be really weird...
from test-infra.
I am checking with AWS directly, yes if that is the case it is truly absurd.
from test-infra.
We also suffer from this problem. It's happened before, about two years ago I think.
I've already asked AWS Support to fix this. Indeed, there are no entries for the newest AMI in their package list. We build the falco module via the driverkit, but it fails as there are no package entries for this new AMI kernel version:
Step by step; when you retrieve this:
http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/latest/x86_64/mirror.list
..it returns this URL: http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61
When retrieving this as an sqlite database:
http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61/repodata/primary.sqlite.gz
...we get a database that holds no info for the latest AMI version, but several entries for the ones before:
% gunzip primary.sqlite.gz ; sqlite3 primary.sqlite
SQLite version 3.39.5 2022-10-14 20:58:05
Enter ".help" for usage hints.
sqlite> select rpm_sourcerpm from packages where rpm_sourcerpm like "%5.4.235-144.344%";
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
sqlite> select rpm_sourcerpm from packages where rpm_sourcerpm like "%5.4.238-148.346.amzn2.x86_64%";
sqlite>
My AWS support person has not yet understood this is the problem, or cannot find the people involved for this sqlite database. My guess is not the latter, unfortunately.
Cheers,
Eelko
from test-infra.
Yep, our kernel-crawler does exactly the same thing; and indeed it cannot find this specific kernel.
To me, this seems like an aws internal issue; let's see.
from test-infra.
from test-infra.
Release details are here: https://github.com/awslabs/amazon-eks-ami/releases
from test-infra.
I am going to pin this issue since it caused multiple headaches :D
from test-infra.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
from test-infra.
from test-infra.
@tigardis: Closing this issue.
In response to this:
/close
On Tue, Jul 25, 2023 at 6:32 AM poiana @.***> wrote:
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually
close.If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
—
Reply to this email directly, view it on GitHub
#1093 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQ3DKIUV6MJNDUEE7QBVTHLXR7DG7ANCNFSM6AAAAAAW2YSN7Y
.
You are receiving this because you authored the thread.Message ID:
@.***>
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from test-infra.
Related Issues (20)
- Terraform version is not downloaded as excpected
- GH runner misses toleration for ARM architecture nodes
- Increase max capacity of jobs on x86 node group
- Validation jobs should run on the jobs node group HOT 5
- Integrate Kubernetes objects with external secrets manager HOT 7
- Introduce GitOps delivery of Prow HOT 4
- Cleanup scripts and unused assets HOT 4
- Drivers build not triggered for x86_64 HOT 19
- Validate driver configs for AL2023 miss toleration for single AZ
- Node group ARM cannot scale-up HOT 1
- Move Terraform CircleCI jobs to GitHub Actions HOT 4
- [UMBRELLA] Move CI/CD out of CircleCI HOT 4
- Bootstrap the GitOps operator HOT 4
- Ensure Prow Kubernetes configurations are fully declarative HOT 4
- Failed to download prebuilt module falco_amazonlinux2_5.10.186-179.751.amzn2.x86_64_1.ko on EKS AMI AMI Release v20230816 HOT 4
- Port driverkit build grid away from makefile and bash scripts HOT 5
- Build drivers more frequently (ideally every 2 hours) HOT 7
- Allow repository archival in Peribolos job
- Run peribolos-syncer jobs on all repositories
- Create AWS IAM role for infrastructure read
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from test-infra.