shapehq / tartelet Goto Github PK

⚙️💻 A macOS app that makes it a breeze to manage multiple GitHub Actions runners in ephemeral virtual machines on a single host machine. The benefits are that runners can run in parallel, and each job runs in an isolated environment.

License: MIT License

Swift 99.87% Shell 0.13%

github-actions macos runner self-hosted-runner swift

tartelet's Introduction

👋 Welcome to Tartelet - a macOS app that launches self-hosted GitHub Actions runners in virtual machines using Tart.

Tartelet makes it a breeze to manage up to two GitHub Actions runners in ephemeral virtual machines on a single host machine. The benefits are that runners can run in parallel and each job runs in an isolated environment that is recreated after each GitHub Actions job has finished.

🚀 Getting Started

Please refer to the following articles in the wiki to get started with Tartelet.

👨‍🔧 How does it work?

Tartelet uses Tart for managing the virtual machines and Tart which in turn uses Apple's Apple's Virtualization framework. The lifecycle of a GitHub Actions runner managed by Tartelet is as follows:

Tartelet uses Tart to clone a virtual machine.
The virtual machine is booted.
After the machine is booted, a setup script is being run. The script downloads the newest version of GitHub's runner application and registers the runner on the GitHub organization.
The runner listens for a job and executes it.
After executing the job, the runner automatically removes itself from the GitHub organization.
The virtual machine is shutdown.
Tartelet uses Tart to delete the virtual machine.

After the last step the process starts over.

🏎 How is the performance?

The performance depends on the hardware that the app is running on. When testing on a Mac mini M1 from 2020 with 16 GB memory, we found that our jobs run 3 - 4 times faster than on GitHub's runners.

We found that our jobs run about 12% slower when running two virtual machines in parallel compared to running a single virtual machine. We find this performance cost negligible as running two virtual machines significantly increases our throughput at a low monetary cost.

This means that Tartelet can run two virtual machines at once. This the maximum number of virtual machines that Apple’s Virtualization framework allows us to run at once.

After a job has finished, the virtual machine that ran the job is shut down and destroyed, and a new virtual machine is created and booted. This process takes about 25 - 30 seconds. However, that overhead is insignificant in most cases as Tartelet creates a new virtual machine after each job has finished. This means that a new virtual machine and its GitHub Actions runner are ready to process the next job. Unless there are more jobs queued on GitHub than the number of available virtual machines, the overhead of creating and booting a virtual machine becomes negligible.

These numbers were last updated in January/February 2023.

👩‍💻 How can I contribute?

Pull requests with bugfixes and new features are much appreciated. We are happy to review PRs and merge them once they are ready, as long as they contain changes that fit within the vision of Tartelet.

Clone the repository to get started working on the project.

git clone [email protected]:shapehq/tartelet.git

Generating a Project File with XcodeGen

After cloning the repository you will notice that the project does not contain a .xcodeproj file. This should be generated using XcodeGen. Install XcodeGen using Homebrew by running the following command in your terminal.

brew install xcodegen

After installing XcodeGen the project file can be generated by running the following command.

xcodegen generate

Generating Resource Constants with SwiftGen

We use SwiftGen to generate constants for images, colors, and localizations. Install SwiftGen using Homebrew by running the following command in your terminal.

brew install swiftgen

Constants for images, colors, and localizations are then generated by running the following command in your terminal.

swiftgen

The swiftgen.yml file at the root of the repository describes how constants are generated.

Configuring the project to run on your machine

To run the project locally, it is necessary to edit the Tartelet.entitlements file to specify a keychain access group that you control. Then you will need to edit the Composers.swift file to ensure the keychain is initialized with the keychain access group specified in the entitlements file. If you do not do this, the app will not be able to persist settings to the keychain.

In other words, you will need to search for $(AppIdentifierPrefix)dk.shape.Tartelet and 566MC7D8D4.dk.shape.Tartelet in the project and replace the occurrences with references to your keychain access group.

Linting the Codebase with SwiftLint

We use SwiftLint to ensure uniformity in the code. Install SwiftLint using Homebrew by running the following command in your terminal.

brew install swiftlint

🤨 Why is it named Tartelet?

The app is named Tartelet because it builds upon Tart, a source-available CLI for managing macOS virtual machines. Tartelet makes it easy to run multiple virtual machines using Tart. The Danish word for "easy" is "let". "Tart" + "e" + "let" = "Tartelet" and a "tartelet" is a traditional Danish food.

🙏 Acknowledgements

Tart does all the heavy-lifting of creating, cloning, and running virtual machines.
Tartelet is heavily inspired by Cilicon.

Tartelet is built with ❤️ by Shape in Denmark. Oh, and we are hiring 🤗

tartelet's People

Contributors

Stargazers

Watchers

Forkers

williamtheaker woshizilong rahul0x24 maxgoedjen mattia stmitt xavierlowmiller jon889 jamesmunro futuredapp haikusw hisaac youcruit giovanninibarbosa nikoroberts greg-cook

tartelet's Issues

Runners not starting

I'm experiencing the same issue(s) as #63. I'm creating a new issue since that other issue is closed.

Problem:
Starting a runner (Virtual Machine --> Start) doesn't start a runner.

Debug logs:
The debug log from Tartelet mentions this error:

2024-02-29T12:44:16Z [VirtualMachineResourcesCopier] [Info] Failed removing resources at file:///Users/anne/Library/Application%20Support/dk.shape.Tartelet/runner-1/.Trashes: “.Trashes” couldn’t be removed because you don’t have permission to access it.
2024-02-29T12:44:16Z [VirtualMachineResourcesCopier] [Info] Failed copying resources from file:///Users/anne/Library/Application%20Support/dk.shape.Tartelet/Virtual%20Machine%20Resources/.Trashes/ to file:///Users/anne/Library/Application%20Support/dk.shape.Tartelet/runner-1/.Trashes: “.Trashes” couldn’t be copied because you don’t have permission to access “runner-1”.
2024-02-29T12:44:16Z [EphemeralVirtualMachineFactory] [Error] Failed making ephemeral virtual machine as resources could not be created: “.Trashes” couldn’t be copied because you don’t have permission to access “runner-1”.
2024-02-29T12:44:16Z [VirtualMachineFleetLive] [Error] Could not create virtual machine named runner-1: “.Trashes” couldn’t be copied because you don’t have permission to access “runner-1”.

When I delete this Trashes directory manually via sudo rm -rf ~/Library/Application\ Support/dk.shape.Tartelet/Virtual\ Machine\ Resources/.Trashes a runner is started, but exits immediately with the following error:

Steps to reproduce:

I was able to reproduce it with the normal installation steps as described in the wiki: https://github.com/shapehq/tartelet/wiki/Configuring-Tartelet.
Using a newer IPSW file did not make a difference.

# old
tart create runner --disk-size=120 --from-ipsw=https://updates.cdn-apple.com/2023WinterFCS/fullrestores/032-48346/EFF99C1E-C408-4E7A-A448-12E1468AF06C/UniversalMac_13.2.1_22D68_Restore.ipsw

# new
tart create runner --disk-size=120 --from-ipsw=https://updates.cdn-apple.com/2024WinterFCS/fullrestores/052-40770/72916BCC-D357-422D-A4A2-EF1DEDF6968C/UniversalMac_14.3.1_23D60_Restore.ipsw

Other details:

Tartelet shows an 'Unknown' VM

These files exist in my Virtual Machine Resources:

Used versions

Tartelet: Version 0.7.1 (1)
Tart: 2.6.1

Tartelet did work in the past without any issues! I'm not sure when it exactly broke, but I did upgrade MacOS (to 14.3.1 (23D60)) and maybe updated Tart via a brew upgrade.

Does somebody have an idea what is wrong?

Reboot doesn't always start VMs

I had the host mini reboot from a power failure (ours isn't on UPS yet) and despite Tartlet set to launch on login & start VMs on startup it didn't start either VM. I hit stop/start and it started 1, had to repeat once more to get both VMs going.

Are there logs somewhere to share? Haven't found them if so.

Debug mode or logging output

I am trying out this project for a client and so far it seems very promising, thanks for sharing it with the community.

However I was stuck during the setup for a very long time because I forgot the "install" the GitHub App I created inside our GitHub organisation. It was a very small and stupid error but I could not find any kind of debug mode or logging which could have easily told me the authorization with GitHub was failing.

Is there a way that could have helped me troubleshoot Tartelet? I'm not too familiar with macOS so I could have missed something. Or is this something that is still on the roadmap for the app?

Thanks!

Tartelet product vision check

Hi 👋

I would love to use Tartelet with GitLab, on the first glance the current architecture should be flexible enough to add new "provider" along with GitHub.

Will you consider this feature inline with overall product vision or would you prefer to stay GitHub exclusive?

Option to start VM fleet automatically on launch

Currently, you can add Tartelet as a login item, but that just starts the app without starting the fleet. An option to start automatically would very useful, in case the host machine restarts or updates automatically.

[Question] Access Runner Name from within VM

I'm trying to set up some monitoring for our build infrastructure (using Datadog), and I have a need to be able to determine the name of the tart runner from within the runner.

A little more context:

When Datadog reports vitals on a computer, it uses the computer's hostname to report information back to Datadog. Because the same virtual machine image is being used for all our Tart runners, the hostname is identical (runners-Virtual-Machine.local). Datadog handles this by only reporting on one virtual machine and ignoring the others.

So I'm trying to find a way of accessing the VM's name from within the VM, so that I can set the host name within a script.

Do you have any ideas on how I could accomplish this in Tartelet's current state? Or is there a change to Tartelet that could be made that might help here? I'd be happy to put in the work to make the change if so.

Here are a couple possible ideas I've thought of:

Give Tartelet the ability to set the Tart VM's hostname after it has been started (maybe in the boot.sh script, or via command over SSH).
Give Tartelet the ability to set or modify global environment variables on the VM after it's been booted (maybe by creating/modifying a file on the VM that can then be sourced or read from within .zshrc or .bashrc files, etc.)
Give Tartelet the ability to run a post-boot script that runs before (or after) the boot.sh script, with the option of passing some variables into the script that could be useful.

Random Runner Crashes After Upgrading to 0.8

We recently upgraded to Tartelet 0.8, and since then, we've been experiencing random crashes of our runners. This issue is impacting our CI/CD workflow, and we're trying to identify the cause.

We are now using Tartelet version 0.8.3, tart version 2.9.0, and are using the xcode 15.3 image from cirruslabs and the host is running sonoma 14.4.1.

This is how the VM's look when connection to the host:

We are unsure whether this problem is related to Tartelet itself, the underlying Tart tool, the VM image, or our GitHub Actions job. Has anyone else experienced similar issues after upgrading? Any insights or suggestions would be greatly appreciated.

Thank you for your assistance!

My Shared Files not showing

A few months ago, I set up a Mac mini M2 with Tartelet. For this, I followed the entire wiki (setting up the host, installing Tartelet and configuring Tartelet). Everything worked perfect.

Due to circumstances, I had to install the machine again. However, when I now follow the same tutorial, I don't see the "My Shared Files" in finder when editing the VM. I even downgraded the version of Tartelet to 0.6.0.

I also tried to export the logs, but hitting that button doesn't seem to do anything. Does anyone have any idea what might be the issue?

Enable Tartelet to update the virtual machines from a registry.

We've just set up Tartelet and learned a lot about tart. This whole project is fantastic!

We're trying to figure out how to manage several physical machines running VMs with Tartelet and keep everything in sync.

We're using a registry to store our VM image which tart can access via pull / clone. But I'm not sure of the best way to propagate changes through the system automatically.

Would it be feasible to have Tartelet run a shell script before launching a runner when it's restarted? That would provide a hook to run some tart commands to pull the latest image from the registry.

Of course it would be nice to have a UI for this, but I'm afraid it's going to be specific to the registry. For example, we're using Amazon's ECR registry which requires authenticating via the aws cli tool before accessing the registry. It's simple enough to script, but impossible to build a UI around.

I think this is a feature request to be able to run a script between stopping a runner and restarting it.

Register runners under repositories to support individual accounts

Tartelet assumes that the GitHub Actions runners should be registered for an organization and as such, it isn't possible to use Tartelet as an individual.

There is not such thing as registering runners on individual accounts, however, they can be registered under a repository. It would be great to support this in Tartelet to make the app useful for individuals.

Right now we have no use for this at @shapehq, and as such, it is not a priority of ours to implement this but we are happy to merge PRs that address it. I'm not ruling out that we'll get around to adding it eventually either.

Memory leak

Hi, I'm experiencing extensive memory usage of the Tartelet process. Tartelet runs for less than one day and then does not process jobs anymore, due to the system being out of memory.

I've configured Tartelet to run two VMs.

System Info

Hardware Overview:
Model Name: Mac mini
Model Identifier: Mac14,3
Model Number: Z16K000EYD/A
Chip: Apple M2
Total Number of Cores: 8 (4 performance and 4 efficiency)
Memory: 16 GB
System Firmware Version: 8422.121.1
OS Loader Version: 8422.121.1
Activation Lock Status: Disabled

System Software Overview:
System Version: macOS 13.4 (22F66)
Kernel Version: Darwin 22.5.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled

Allow GitHub URLs to be set in settings to support GitHub Enterprise and Enterprise Cloud

Tartelet hardcodes the Server and API URLs to GitHub.com. It would be nice if these could be overridden in settings to support GitHub Enterprise and Enterprise cloud.

Does start the virtual machine

I'm using the latest version (0.6.0) and when I start it, it does not start any virtual machine and it does not show any error message.

Unable to start VMs

I'm seeing the following in the console:

Could not create virtual machine named runner-2: The GitHub app has not been installed. Please install it from the developer settings.

I have installed the app in the GitHub org settings, though, so not sure what else to try?

It appears that the private key isn't saved to the keychain

Overview

macOS: 13.2.1
tartelet: 0.2.0

The app appears to not be saving the private key as indicated in point 8 and the screenshot:

CleanShot000034.mp4

Unexpected termination status: 64

Hi,
I'm experiencing issues when using the latest versions of Tartlet (0.4) and Tart (1.7.0) together. Specifically, when I click "Start," the virtual machines don't start. In the logs, I see the following information:

2023-06-29T13:07:58Z [VirtualMachineFleetLive] [Info] Start virtual machine named macos-ventura-xcode:14.3.1-1
2023-06-29T13:07:58Z [EphemeralTartVirtualMachine] [Info] Clone Tart image named macos-ventura-xcode:14.3.1 to macos-ventura-xcode:14.3.1-1...
2023-06-29T13:07:58Z [EphemeralTartVirtualMachine] [Info] Run Tart image named macos-ventura-xcode:14.3.1-1...
2023-06-29T13:07:58Z [VirtualMachineFleetLive] [Info] Virtual machine named macos-ventura-xcode:14.3.1-1 stopped with message: Unexpected termination status: 64

However, if I manually start the virtual machine through the "Edit Virtual Machine" option, it starts as expected.

GitLab Support

Hey, is there anyway this can be modified to support GitLab?

Automatically check for updates

At the moment Tartelet does not have a builtin mechanism to update itself; a user has to be aware of a new release an manually download and install and update.
This capability could be implemented by leveraging a tool like Sparkle; it would still require the user to manually install the update but it would be nevertheless an improvement for the user experience.

Is this a feature that the project would like to integrate? I would be interested in taking a stab at it, even if I haven't done it before.
If it is ok, I may try to implement it in the upcoming weeks

Best practice for setting environment variables?

I think I have a plan, but want to make sure:

Are there any best practices for setting environment variables on runner VM instances?

Background: I'm trying to set up our runners for Android builds, and need to set ANDROID_HOME, which ideally should always be available.

I'm looking into using launchctl setenv (as described here), but I'm wondering if there is an "official" tartelet way of doing this?

Question: Should runner account be standard or admin?

I had a question about a recommendation made in Tartelet's Wiki.

In the Setting Up a Host Machine page, it advises:

Create an account named "runner" through the Users & Groups pane in System Settings. It is important that this is a standard account, that is, it should not be an administrator. We will use this account to run the Tartelet application.

It later goes on to outline how to grant the account sudo privileges, which sort of makes it not exactly a standard account anyway.

Could you clarify why it's important that this is a standard account and not an admin? I'm sure there's a good reason, so I'd just like to understand it better.

Upgrading from 0.3 got me in a reboot loop

Replaced Tartlet 0.3 with 0.8.x (previously configured to boot VMs on start)
Launched it
It proceeded to try to boot VMs but failed to ssh login & kept restarting
Stop/start was missing from menu item

Entering settings & turning off start-on-boot & restarting Tartlet broke the loop & got me start/stop item so was able to configure things after that & now it's running great.

No ephemeral VMs created

I've set up tart & Tartelet on my 2021 M1 Pro MBP running Ventura 13.0 and a 2020 M1 Mac Mini at Macstadium. I was able to setup and configure a VM without issues. Followed the guides, which are great btw., step by step. Everything seems to be set up correctly, but when I hit Start, no runners how up on Github. tart CLI also reports no additional machines being setup.

My log file looks like this for the MBP and I see the same message on the Macstadium instance:

2023-05-12T12:08:33Z [VirtualMachine] [Error] Failed making ephemeral virtual machine as resources could not be created: <private>
2023-05-12T12:08:33Z [VirtualMachine] [Error] Could not create virtual machine named "<private>": <private>
2023-05-12T12:08:33Z [VirtualMachine] [Error] Failed making ephemeral virtual machine as resources could not be created: <private>
2023-05-12T12:08:33Z [VirtualMachine] [Error] Could not create virtual machine named "<private>": <private>

Any hint what I'm doing wrong? I glanced at the code and could happily clone machines via the tart CLI. Is there something else I can inspect?

Sometimes the second virtual machine is not started

There's an issue where sometimes Tartelet will go from maintaining two virtual machines to only one, even when configured to maintain two.

Workaround:

Stop Tartelet.
Start Tartelet.

This will address the issue for a while but it may eventually happen again.

I'll find some time to look into this issue. I think it can be resolved by vastly simplifying the way virtual machines are maintained.

Failed making ephemeral virtual machine, missing permissions

Hi, I'm getting this error when starting the VM's.

Failed making ephemeral virtual machine as resources could not be created: “.Trashes” couldn’t be copied because you don’t have permission to access “runner-1”.

Unsure what needs permission to perform the copy action?

System

Mac mini, M1 2020
macOS Sonoma 14.3.1

Using fine-grained tokens instead of an app for github?

Aloha!

Firstly, thanks for developing this amazing project 💌 !

I was wondering why an app is required for registering the github actions runners in the repositories and if supporting fine-grained tokens for it is in the roadmap. I'm running the kubernetes github-actions-runners-controller which makes use of just a token to do the exact same job, so in principle, it should be possible for tartalet as well.

Thanks in advance!