meanpug / folding-together Goto Github PK

Function to start the CloudFormation stack with any required input parameters

This function will be responsible for tracking the available balance for running compute over time. As time passes this value should go down and help represent a near-real-time balance of the funds available for a particular donor.

• run at regular intervals
• decrement available funds based on instance running costs and scheduling frequency

For example, if we run the function every minute and the target instance currently costs $2.40/hr, we should decrement by $0.04

We can report on team stats like:

• wus completed
• score
• number of team members
• global rank

Relevant endpoints:

• https://api.foldingathome.org/team/255839

Sample Response:

{
    "founder": "CygnusXI",
    "id": 224497,
    "logo": "http://i.imgur.com/PIdLHV1.jpg",
    "name": "Curecoin",
    "rank": 1,
    "score": 1114003226872,
    "url": "http://curecoin.net",
    "wus": 25490973
}

If this gets really popular, we will have to worry about service quotas (soft limits) such as:

• Running EC2 instances
• Active CloudFormation stacks
• Queue size?

Database to track balance on a per-donor basis

Thinking DynamoDB with strong consistency

https://covid-global-hackathon.devpost.com/updates/12472-day-3-wrap-up

Currently we default to no team. A donor should be able to specify a team.

The donor-team combo can be a unique value in the balance tracker.

We want a "failsafe" mechanism to prevent o proactively alert when costs are accruing without the donations to back them. For example, if an instance fails to terminate when scheduled, we want to find out before it runs for several hours and generates costs that the project maintainers will have to bear.

As a way to improve #4, we should publish to the SQS queue async (through either celery worker or migration to asgi+asyncio)

Map CloudFormation stack ids to donors in the ddb table

Function to terminate the stack

On the detail page, we can include information like:

• The donor name
• The donation amount
• The status of the the donation (unprocessed vs. processing vs. processed)
• Stats around the donation as provided by F@H API. These stats can include things like: WU run, score, and amount of work run relative to other donations on platform (e.g. percentile)

Trigger: probably cron #26. Frequency as often as one minute but less frequent will also work as long as the drawdown is proportional to the execution frequency.

Steps:

1. Query balances < $0 (or some other stop threshold)
2. For each balance above $0, run a function (#12) to decrement the remaining amount depending on instance cost (usually in $/hr but accrued per-second with Linux) and time since last adjustment.
3. Check to see if an instance already exists. Make a decision on what to do.
   • If one does not exist and funds are available, create one. #1, #24
   • If one does not exist and there are insufficient funds, do nothing.
   • If one exists and funds are available, do nothing.
   • If one exits and there are insufficient funds, terminate (#13)
4. Update the DDB table with the stack id (#25)

This will be a requirement for #2 .

We have two options here, and I'm not yet sure which is the right one:

Option 1

For this, we back compute with EC2 resources and an ASG, so we'll need:

• a launch template or configuration that includes the software to run folding@home software on start
• an Autoscaling Group

Option 2

We back compute with Fargate resources. A pre-requisite here is that we can get folding@home working properly in a container and the container can be juiced with enough access to the host compute resources:

• Fargate cluster
• Docker image pre-loaded with software to run folding@home on launch

The API should accept a donor ID and retrieve the status (queued vs. processing vs. finished) for the donor

Pick a frequency and write a cron expression for it in a CloudWatch event that can be used for the state machine trigger

The gateway simply needs to allow for basic user registration and payments. Upon successful payment, it can queue a task to schedule compute for the available credits.

cc @atkinsonm

Helpful Links:

• https://foldingathome.org/support/faq/installation-guides/linux/command-line-options/
• https://foldingathome.org/support/faq/installation-guides/linux/manual-installation-advanced/

Goal
To get the folding@home software set up and working on a single EC2 tiny instance. Documentation of any issues / steps can be attached as comments

I would like to have a way to measure whether an instance is actually running folding@home jobs. If an instance (or container) is running but sitting idle (because it hasn't received assignment, potentially do to upstream saturation), it's not useful to anyone and should be hibernated or terminated to save resources.

This is the function which runs when a donation is received. It is responsible for identifying the donor username and amount and updating the available balance for that donor.

• trigger when messages added to queue
• read from the queue
• add funds to the metric under donor name
• remove processed message

There is a hard limit of 200 CloudFormation stacks per region. If this solution gets popular, the 1 stack per donor will no longer be a viable model.

We can avoid this with custom calls against the EC2 API. We might be able to reuse bits like security groups, instance profiles, AMIs, etc.

Relates to #17

currently, when someone makes an anonymous donation through the Anonymous user, the balance of the associated anonymous record is incremented appropriately, however because we don't update the stack ID to empty upon completion and subsequent termination, a new stack is never launched.

The proposed fix is to update stack ID to empty upon termination of FAH instances, which we should probably be doing anyway

Essentially, this service should understand how many blocks of time have been purchased and - based on this - adjust the ASG parameters. For instance, assume donor A purchased 20m of time and donor B 40m of time. The scheduler would then understand that an instance may be added to the ASG for 60m before a new scaling event. Assume then that a new donor purchases, 60m of time. Rather than line up blocks in serial, a new instance should be scheduled immediately and the ASG scaled up to 2 for 1 hour (rather than 1 for 2 hours).

The TlDr here is, schedule in 60m increments

I think it would be cool to have a FoldingTogether team to serve as a way to showcase the contributions from this team and its donors and other contributors.