Comments (37)
Are the self-hosted runners only x86_64?
There is one aarch64 one. (It is required because GitHub doesn't offer aarch64 Linux boxes, and Google Cloud doesn't offer an aarch64 CPU that can run armhf 32-bit binaries)
from secp256k1.
It would be more productive, if we somehow prioritise them to spend our time until Sept. 1st more effectively.
I'd say the Windows/macOS ones are probably easier, since they don't require write permission and don't have to deal with docker image caching.
from secp256k1.
qemu-arm is a bit slower than native aarch64. You can use the already existing persistent worker, if you want:
(Currently not set up for this repo, but should be some time this week)
from secp256k1.
Sure, that's an easy option. I just think we're currently playing around with the idea to move everything to GHA, if it's feasible for this repo.
from secp256k1.
Have you tried the supplied suppression file (LouisBrunner/valgrind-macos@
main
/darwin19.supp)?
Yes, I have. It does not change the outcome.
UPD. I used https://github.com/LouisBrunner/valgrind-macos/blob/main/darwin22.supp as we run Ventura.
from secp256k1.
FWIW, it works with no additional suppressions on macos-12.
Oh ok, should we then just use this for now?
I can take care if you don't have the bandwidth.
It would be nice because I have no x86_64 macOS Ventura available.
I don't have any macOS available. ;)
from secp256k1.
Do you think maintaining the suppressions is a problem? I don't think it's a big deal.
You mean, in this repository?
Yes... I don't think it will be a lot of work, but I guess we should still submit it upstream first.
See LouisBrunner/valgrind-macos#96 as a first step.
from secp256k1.
Ok, then it probably makes sense to do what I suggested in #1153, namely move ARM tasks to Linux, and reduce the number of our macOS tasks.
from secp256k1.
moving everything to GitHub Actions is a bit cleaner for us
Sounds interesting. I wonder how (and if) docker images can be cached, along with ccache, etc...
from secp256k1.
moving everything to GitHub Actions is a bit cleaner for us
Sounds interesting. I wonder how (and if) docker images can be cached, along with ccache, etc...
Yeah, we'll need to see.
And I agree that "in the short run it seems easier to stick to Cirrus for now, because the diff is a lot smaller (just replace container: in the yml with persistent_worker:, etc)" (bitcoin/bitcoin#28098 (comment)). We should probably do this first, and then see if we're interested in moving to GitHub Actions fully.
edit: I updated the roadmap above.
from secp256k1.
For macOS, we need to take also #1153 into account. It seems that GitHub-hosted macOS runners are on x86_64. The good news is that Valgrind should work again then...
For such a case, it is good to see some progress in #1274 :)
from secp256k1.
moving everything to GitHub Actions is a bit cleaner for us
Sounds interesting. I wonder how (and if) docker images can be cached, along with ccache, etc...
See #1396.
from secp256k1.
- Move native Windows tasks to GitHub Actions (solved by ci: Run "Windows (VS 2022)" job on GitHub Actions #1389)
- Move native macOS tasks to GitHub Actions
- (Optional, in the long run:) Move Linux tasks to GitHub actions
There are open PRs for all of the mentioned items. It would be more productive, if we somehow prioritise them to spend our time until Sept. 1st more effectively.
from secp256k1.
Yes, we should in principle proceed in the order of the list above. But it doesn't need to be very strict. For example, if it turns out that #1396 is ready by Sep 1st, we can skip "Move Linux tasks to the Bitcoin Core persistent workers".
from secp256k1.
- Move Linux tasks to the Bitcoin Core persistent workers
It seems reasonable to split this task in two ones, depending on the underlying architecture: x86_64
and arm64
, because the GitHub hosted runners lack support for arm64
.
from secp256k1.
@hebasto Hm, we currently don't have native Linux arm64 jobs, so we can't "move" them over. We could add some (see #1163 and #1394 (comment)).
I tend to think that is also acceptable to wait for github/roadmap#528, it's currently planned for the end of the year. Then we could move macOS back to ARM. Until that happens, perhaps we can add a QEMU jobs that run the ctimetests on MSan (clang-only) at least. Note to self: We need apt-get install libclang-rt-dev:arm64
and this works with
HOST="aarch64-linux-gnu" CC="clang --target=aarch64-linux-gnu" WRAPPER_CMD="qemu-aarch64"
(The real tests fail with msan enabled on qemu. I think this is because the stack will explode.)
I updated the list above with optional items.
from secp256k1.
- Enable Valgrind on macOS again now that the macOS tasks run on x86_64 (Constant-time tests on macOS #1151)
While it worked on macOS Catalina back in time, it seems a couple of suppression for /usr/lib/libSystem.B.dylib
and /usr/lib/dyld
are needed.
Branch (POC) -- https://github.com/hebasto/secp256k1/tree/230824-valgrind
CI -- https://github.com/hebasto/secp256k1/actions/runs/5967987235
from secp256k1.
Oh thanks for checking. Have you tried the supplied suppression file (https://github.com/LouisBrunner/valgrind-macos/blob/main/darwin19.supp)? If it doesn't solve the problem, we could try to upstream the additional suppressions, see also LouisBrunner/valgrind-macos#15.
from secp256k1.
Do you think maintaining the suppressions is a problem? I don't think it's a big deal.
UPD. I used LouisBrunner/valgrind-macos@main/darwin22.supp as we run Ventura.
Okay, sure, I got confused and looked at the wrong file.
from secp256k1.
Do you think maintaining the suppressions is a problem? I don't think it's a big deal.
You mean, in this repository?
from secp256k1.
Do you think maintaining the suppressions is a problem? I don't think it's a big deal.
You mean, in this repository?
Yes... I don't think it will be a lot of work, but I guess we should still submit it upstream first. If they merge it quickly, then it's easiest for us. I can take care if you don't have the bandwidth.
from secp256k1.
- Enable Valgrind on macOS again now that the macOS tasks run on x86_64 (Constant-time tests on macOS #1151)
While it worked on macOS Catalina back in time, it seems a couple of suppression for
/usr/lib/libSystem.B.dylib
and/usr/lib/dyld
are needed.
FWIW, it works with no additional suppressions on macos-12
.
from secp256k1.
I can take care if you don't have the bandwidth.
It would be nice because I have no x86_64 macOS Ventura available.
from secp256k1.
FWIW, it works with no additional suppressions on macos-12.
Oh ok, should we then just use this for now?
Done in #1412.
from secp256k1.
- Add a task for ctimetest on ARM64/Linux/Valgrind on Cirrus CI using free minutes or the self-hosted runner
Hm, it appears that Cirrus' "Dockerfile as a CI environment" feature won't work with persistent workers (see #1418). Now that I think about it, that's somewhat expected (e.g., where should the built images be pushed?).
Alternatives:
- Set up our own Docker pipeline on Cirrus (but that seems overkill)
- Use the free minutes (but they're shared among all repos of bitcoin-core)
- Go back to the QEMU approach and try to compile compiler-rt on our own (but that's a rabbit hole and doesn't cover gcc)
- Don't use a Dockerfile (but that means we won't get gcc-snapshot, unless we build it every time or cache it somehow)
- Do nothing (and wait for native ARM64 on GHA)
I think we should do one of the last two?
from secp256k1.
A persistent worker will persist the docker image itself, after the first run on the hardware. I think all you need to do is call
podman image --file $docker_file --name --env $bla --name $bla_image_name && podman container kill $ci_bla_name && podman run -it --rm --name $ci_bla_name $bla_image_name ./ci.sh
Alternatively it may be possible to find a sponsor to cover the cost (if it is not too high) on cirrus directly, while native arm64 isn't on GHA.
I can look at the llvm issue next week, if time permits.
from secp256k1.
A persistent worker will persist the docker image itself, after the first run on the hardware.
Thanks for chiming in. Wouldn't we also need to make sure that images get pruned from time to time? Or does podman handle this automatically?
podman image --file $docker_file --name --env $bla --name $bla_image_name && podman container kill $ci_bla_name && podman run -it --rm --name $ci_bla_name $bla_image_name ./ci.sh
I assume the first step performs the caching automatically, rebuildung layers only as necessary? Sorry, I'm not familiar with podman, I have only used Docker so far.
Alternatively it may be possible to find a sponsor to cover the cost (if it is not too high) on cirrus directly, while native arm64 isn't on GHA.
Right, yeah, I'm just not sure if I want to spend time on this.
I can look at the llvm issue next week, if time permits.
Ok sure, but I recommend not spending too much time on it. It also won't help with GCC (I added a note above).
from secp256k1.
Thanks for chiming in. Wouldn't we also need to make sure that images get pruned from time to time? Or does podman handle this automatically?
Yeah, you can also run podman image prune
, if you want. Pull requests to bitcoin-core/gui should already run it on the same machines, but that seems fragile to rely on.
See:
I assume the first step performs the caching automatically, rebuildung layers only as necessary? Sorry, I'm not familiar with podman, I have only used Docker so far.
Yes, it is the same. You should be able to use docker
as well, if you want, which is podman-docker
.
Right, yeah, I'm just not sure if I want to spend time on this.
If you mean reaching out to a sponsor, I am happy to reach out, if there is a cost estimate.
from secp256k1.
Okay, then I think this approach is probably simpler than I expected. I'm not sure if I have the time this week, but I'll look into that soon. (Or @hebasto, if you want to give it a try, feel free to go ahead, of course. My plan was to simply "abuse" the existing Dockerfile to avoid maintaining a second one, at the cost of a somewhat larger image. The existing file should build fine except that debian won't let you install an arm64 cross-compiler on arm64. So we'd need to add some check to skip these packages when we're on arm64, see https://github.com/bitcoin-core/secp256k1/pull/1163/files#diff-751ef1d9fd31c5787e12221f590262dcf7d96cfb166d456e06bd0ccab115b60d .)
If you mean reaching out to a sponsor, I am happy to reach out, if there is a cost estimate.
Okay, thanks, but let's first try docker/podman then.
from secp256k1.
Anything left to be done here?
from secp256k1.
The migration is done, but there are still a few unticked checkboxes. (And I've just added two.) None of them are crucial, but I plan to work on them soon, so I'd like to keep this open for now. We could also close this issue here and add a new tracking issue, or open separate issues for the remaining items, if people think that makes tracking easier.
from secp256k1.
from secp256k1.
"With today’s launch, our macOS larger runners will be priced at $0.16/minute for XL and $0.12/minute for large."
from secp256k1.
"With today’s launch, our macOS larger runners will be priced at $0.16/minute for XL and $0.12/minute for large."
This is a price decrease for private repos, and GHA remains free for public repos.
from secp256k1.
Are large runners available for public repos?
from secp256k1.
Are large runners available for public repos?
Ha, okay, you're right. No, "larger runners" are always billed per minute, i.e., they're not free for public repos. And it seems that they're not planning to provide M1 "standard runners". At least github/roadmap#528 (comment) has been closed now. That means we should stick to the Cirrus runners for ARM.
from secp256k1.
Related Issues (20)
- build: benchmarks and examples should be disabled when external callbacks are enabled HOT 4
- build error: implicit declaration of function 'time' is invalid in C99 HOT 2
- Wrong documentation for secp256k1_ec_seckey_tweak_add
- secp256k1_u128_rshift should use __shiftright128 intrinsic on MSVC
- ci: MSVC fails after rebuild of Docker image
- "sizeof(secp256k1_ge_storage) != 64" path not tested HOT 1
- scalar: Verify invariants on every entry HOT 2
- `-Wmaybe-uninitialized` when compiling with `-O1` HOT 3
- Schnorr is not compatible with BIP340 HOT 5
- group: ge(j) should have as invariant that the curve equation holds HOT 3
- Remove `SECP256K1_WARN_UNUSED_RESULT` from `secp256k1_keypair_*` functions HOT 3
- Policy for VERIFY_CHECK and #ifdef VERIFY HOT 5
- ci: Add a "compile the headers" job
- ci: Add a constant-time test job that uses MSan HOT 2
- mingw-w64: "visibility attribute not supported in this configuration; ignored" under `-flto` HOT 5
- Silent Payments (BIP352) module -- discussion about scope and interface HOT 2
- Compile with: --with-ecmult-window=24 --with-ecmult-gen-precision=8
- ARM assembly: "missing .note.GNU-stack section implies executable stack" with recent ld
- ci: Random Valgrind MacOS build failures HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from secp256k1.