[Missing Driver]: falco_amazonlinux2_5.4.238-148.346.amzn2.x86_64_1.ko about test-infra HOT 32 CLOSED

My AWS case is already escalated to the internal ec2 team. Will keep you posted!

from test-infra.

@renilthomas AWS has updated the AMIs and the updated AMI uses kernel version 5.4.238-148.347.amzn2.x86_64. We did an initial testing and falco probe is getting downloaded properly and is working properly now.
cc: @FedeDP

from test-infra.

I still don't get where does this kernel version come from; doing a

wget amazonlinux.us-west-2.amazonaws.com/2/extras/kernel-5.4/latest/x86_64/mirror.list

yelds a mirror.list file with

http://amazonlinux.us-east-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61

as a content. That is what kernel-crawler is discovering too.
There should not be any other mirror, as far as i could find, for kernel-5.4.

from test-infra.

We are also facing similar issue with latest AWS EKS AMI for 1.23. We have tried to downgrade the ami and it seems to be working.

@renilthomas Thanks for working with the ec2 team. Please share if you hear any updates.

from test-infra.

So, this was an issue on the AWS side right?
Great to hear it's getting fixed!:)

from test-infra.

Hi All, Just wanted to update that we have tested falco versions 2.0.18,2.5.5,3.0.0 and 3.1.3 with k8s version 1.22 to 1.26. And the falco daemonset is working as expected with ebpf probe getting loaded and we are getting threats data as well

from test-infra.

Hi!
We have 5.4.238-148.347: https://download.falco.org/driver/site/index.html?lib=4.0.0%2Bdriver&target=amazonlinux2&arch=x86_64&kind=kmod&search=falco_amazonlinux2_5.4.238-148.347.amzn2.x86_64_1.ko
I think you need to upgrade your kernel to latest version! We did never build 5.4.238-148.346 unfortunately; most probably it was immediately patched to .347 and since our pipeline runs weekly, we only discovered and built the latter.

from test-infra.

I got this URL from the instance with the specific kernel version

https://amazonlinux-2-repos-us-west-2.s3.dualstack.us-west-2.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/2d43c5da4c1aee506efc1db66eca9864d37d1f92076bd0eb8b357c32189079bf/../../../../../../blobstore/6e296e6f7ea2f7f03d201ac21967f5c61bdb7904485492cc822b34d6892af4c8/kernel-5.4.238-148.346.amzn2.x86_64.rpm

from test-infra.

In the meantime, i am trying to figure out what are we missing :)
Also, @EXONER4TED (gentle tag ahah) is the 👑 when finding weird kernel-crawler issues :D

from test-infra.

oh right, checking

from test-infra.

You are right, there seems to be none for 346 😞

from test-infra.

That is correct, they responded to my issue awslabs/amazon-eks-ami#1266 (comment)
And I had also escalated it within AWS and last I heard earlier was they were working on the fix.
So, all good now @FedeDP

from test-infra.

uhm thank you for reporting this, probably the kernel crawler still searching for this kernel @FedeDP

from test-infra.

The EKS latest AMI release has pinned 5.4.238-148.346.amzn2 for k8s v1.23 or below. Can the kernel crawler build based on a specific version?
At this stage, there are some critical vulnerabilities that is addressed in this new AMI but we have only two options:

1. Disable falco and upgrade AMI
2. Upgrade k8s - which obviously requires scheduling a maintenance and planning towards it.

from test-infra.

Hi! It seems like we fail to crawl the kernel; i will look into this asap :)

from test-infra.

Thank you @FedeDP 🙇

from test-infra.

Mmh my guess is that the 5.4.238-148.346.amzn2 kernel is not present in any mirror anymore; as i said previously, we can only scrape 5.4.238-148.347.amzn2.
Are you able to share the url for the 5.4.238-148.346.amzn2 package?

from test-infra.

For reference, this is what the crawler finds instead:
http://amazonlinux.us-west-2.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61/../../../../../../blobstore/fa5caa71652bca6f4a8aeff9356d18bd1785d8620080398768acbb53a7fb5756/kernel-devel-5.4.238-148.347.amzn2.x86_64.rpm

from test-infra.

Yes i have seen that running kernel-crawler myself. Is it possible to build with that URL I shared? Thank you in advance!

from test-infra.

The quickest fix, since you have the full URL, is to add a new config on test-infra for it by eg: copy/pasting the 347 one: https://github.com/falcosecurity/test-infra/blob/master/driverkit/config/4.0.0%2Bdriver/x86_64/amazonlinux2_5.4.238-148.347.amzn2.x86_64_1.yaml

If you open a PR (please add it for all 3 supported driver versions!) i will be happy to accept it :)

from test-infra.

Brilliant, on it.. Thanks @FedeDP

from test-infra.

Wait! i just notice: you are sharing the kernel URL, not the kernel-devel one.
Perhaps there is no kernel-devel package for it?

from test-infra.

So, did they push an updated kernel without pushing updated headers? That would be really weird...

from test-infra.

I am checking with AWS directly, yes if that is the case it is truly absurd.

from test-infra.

We also suffer from this problem. It's happened before, about two years ago I think.

I've already asked AWS Support to fix this. Indeed, there are no entries for the newest AMI in their package list. We build the falco module via the driverkit, but it fails as there are no package entries for this new AMI kernel version:

Step by step; when you retrieve this:

http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/latest/x86_64/mirror.list

..it returns this URL: http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61

When retrieving this as an sqlite database:
http://amazonlinux.eu-central-1.amazonaws.com/2/extras/kernel-5.4/stable/x86_64/437c15a145ce03d3a48b7fd7559df3851f3b58fd9d7a78960f44d029f145bd61/repodata/primary.sqlite.gz

...we get a database that holds no info for the latest AMI version, but several entries for the ones before:

% gunzip primary.sqlite.gz ; sqlite3 primary.sqlite
SQLite version 3.39.5 2022-10-14 20:58:05
Enter ".help" for usage hints.
sqlite> select rpm_sourcerpm from packages where rpm_sourcerpm like "%5.4.235-144.344%";
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
kernel-5.4.235-144.344.amzn2.src.rpm
sqlite> select rpm_sourcerpm from packages where rpm_sourcerpm like "%5.4.238-148.346.amzn2.x86_64%";
sqlite>

My AWS support person has not yet understood this is the problem, or cannot find the people involved for this sqlite database. My guess is not the latter, unfortunately.

Cheers,
Eelko

from test-infra.

Yep, our kernel-crawler does exactly the same thing; and indeed it cannot find this specific kernel.
To me, this seems like an aws internal issue; let's see.

from test-infra.

from test-infra.

Release details are here: https://github.com/awslabs/amazon-eks-ami/releases

from test-infra.

I am going to pin this issue since it caused multiple headaches :D

from test-infra.

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

from test-infra.

from test-infra.

@tigardis: Closing this issue.

from test-infra.