GitHub PR Comments

Placeholder

Misc

• Create Buildkite service account for API tokens
  • Replace token for Agent Manager
  • Replace token for PR bot

Monitoring and Alerting

Implement monitoring and alerting for Buildkite and Buildkite-related GCP resources.

• GCP
  • Instances
  • Quotas
• Buildkite
  • Agent Queues (jobs waiting for agents, current agents per queue, etc)
  • Jobs (status, duration)

Baseline Pipeline

Implement a pipeline to replace the "Baseline" pipeline currently running in Jenkins.

Agent Manager

• Change secs configs to mins
• Documentation
• Test coverage
• Balance agents cross numerous GCP zones
• Hard and soft time limits for GCP instances
• APM instrumentation
• Delete orphaned GCP instances (isntance online, not connected to an agent)
• Scaling idle timeouts
  • e.g. the first agent for a configuration might have an idle timeout of 1 hour, but the 200th might be 5 minutes
• Automatically replace agents if disk usage is too high?

Test Issue

• Test task
• Test task

Intrastructure

• New GCP project for Kibana CI
• IAM + Roles
• Migrate current Buildkite infra to new GCP project

Notes:

• Separate GCP project for CI
• Things that will live here:
  • GCS Buckets
  • Artifact Registry
  • Cloud Build
  • Cloud Run
    • PR bot, Slack bot?
  • GKE Auto-Pilot
    • Agent Manager
  • Agent Instances / Images
  • Cloud NAT + Cloud Router (for agents)
• Cloud NAT
  • To scale beyond about 1,800 agents, we will likely need to implement Cloud NAT and remove external IP addresses from our agent instances
  • Needs Cloud Router set up as well
  • Automatic IP allocation works like this:
    • Each VM is reserved a specified number of ports per IP address in Cloud NAT
    • When the ports are maxed, a new IP address is added to the pool, which counts against static IP quota (confirm static IP quota?)
    • Each IP can have about 64K ports
    • Each VM can reserve up to 1024 ports
    • So, with auto scaling of IPs, at least 64 agents will share an IP
• Allow SSH via IAP
  • IAP allows authorized users (auth via GCP/Google IAM) to SSH into agent instances that don't have public IPs
  • This is already done, except for assigning user/group permissions

GitHub PR Bot

Placeholder

Slack Notifications

Placeholder

Agent Images

Work related to what's contained in the agent images, rather than the management of the images.

• Beats

Agent Image Management

• Periodically create any necessary images
• Test images after creation
• Promote images after test
• Rollback image after promotion (e.g. promote old image or pin an image)
• Delete old images

What's the best way to "promote" and "pin" images? Do we need separate families that we copy images between?