- Java 7 or higher.
- Maven 3.3 or higher (not a strict prerequisite)
Suppose you want to offer a service that sets up cloud-based Spark clusters for users. One thing you will need is a resource manager to assign Spark workers to the cloud instances you acquire. Based on the size of the requested cluster, the resource manager should request a set of instances from a cloud provider and place containers (lightweight virtual machines) on these instances (where each container runs a Spark worker). When there is a request to shut down a cluster, the resource manager should unplace containers from instances.
The goal of this assignment is to implement a resource manager. The resource manager should take in three parameters: the cloud provider, the memory size in GB of the instances that the cloud provider provides, and the memory size in GB of the containers you want to place on the instances. It should then respond to requests for new containers by managing a pool of available instances, tracking the containers that have already been placed on them, and requesting new instances as necessary. Your teammate has already implemented an interface for placing and unplacing containers on a single instance.
- The resource manager should create the minimal number of instances when placing multiple
containers. For example, if the instance size is 60 GB and the container size is 6 GB, placing up
to 10 containers should only result in a single new instance.
- The resource manager should NOT make any instance requests when there is enough space on an *existing instance or a pending instance (i.e., an instance that has been requested but has not yet been returned by the cloud provider)*. For example, if all instances are full and a user requests another container, the resource manager should request a new instance on which to place that container. If, while waiting for that new instance to become available, the resource manager receives another request to place a container, it should not request another instance, since it will eventually be able to place the container on the pending instance.
placeContainers
should return within constant time with respect to the number of instances requested by the call, the number of containers requested by the call, and the number of concurrentplaceContainers
calls. This constant time should be dominated by the time for the Cloud Provider to create one instance. You may assume that the Cloud Provider is scalable, i.e. the time for the Cloud Provider to create one instance is the same as the time to create multiple instances.- The resource manager must load the current state of the world from the cloud provider on startup.
- Your code should have good readability and modularity. It should be easy to understand what your code is doing and why.
- Your submission should include unit tests that test all functionality of your code (i.e., in a suite called ResourceManagerImplTest.java). We recommend using JUnit and a mocking library (Mockito is a good one) for these tests.
- Your unit tests should run very quickly (on the order of seconds).
- Your implementation must be in Java.
- There will be no failures when making requests to the cloud provider for instances.
- The cloud provider creates instances in parallel, i.e., the time for the cloud provider to create one instance is the same as the time to create 10 instances.
- The resource manager's state (i.e., the instances and the containers objects) can fit in memory.
- You will never have to handle more than 100 simultaneous requests across all of the resource manager's methods.
- You may assume that there will only be a single instance of the resource manager running at any given time.
- The resource manager can hold onto instances from the cloud provider even after it has unplaced all containers from those instances.