requests.exceptions.HTTPError: 400 Client Error: Bad Request for url about e2e-mlops HOT 7 CLOSED

The time to execute the actual job should not take more than 2-5 minutes, however it can potentially take an equivalent time to acquire the VMs for the cluster. Note the comment just under where you specify the node and worker types.

If you have the permissions to do so I would set up an instance pool and supply the pool ID to the commented out driver_instance_pool_id and instance_pool_id attributes. This means that for each job you can acquire resources from that warm pool of instances. Note that you will then have to comment out node_type_id and driver_node_type_id.

Your deployment.yml would look something then like the following:

from e2e-mlops.

Creating a starter job is not required. The first job you submit against the pool may take a few mins to acquire those VMs for the pool. But then that pool will remain warm to the time period you have set. Subsequent jobs which use this pool will then take a shorter time to acquire resources as they will acquire VMs from the now warm pool.

from e2e-mlops.

It looks like you're running on an Azure workspace, but supplying an AWS instance type.

In deployment.yml we define node_type_id and driver_node_type_id , using i3.xlarge instance types. These are AWS-specific. Ultimately, the choice of instance type is up to the user, but the nearest equivalent on Azure would be Standard_DS3_v2. I would suggest replacing i3.xlarge with Standard_DS3_v2 and give it a go.

from e2e-mlops.

Thanks @niall-turbitt . Setting the node_type_id and driver_node_type_id to Standard_DS3_v2 solves it. Now the launch command can be executed. The speed for running this job seems quite slow though. Would using a larger instance likely to help? Like Standard_DS4_v2 or Standard_DS5_v2?

from e2e-mlops.

Thanks. Just to double check, what should be the compatible preloaded Databricks Runtime version? Should it be 11.0 ML (Scala 2.12, Spark 3.3.0) or 11.0 (Scala 2.12, Spark 3.3.0) ?

from e2e-mlops.

You should select DBR 11.0 ML (Scala 2.12, Spark 3.3.0)

from e2e-mlops.

Thanks! I have created the pool now. Do I also need to add a starter job as the video explained? However, the interface is quite different. I couldn't find a place to specify the use of pool here.

Is there any sample code I could copy over for the starter job?

Thank you.

from e2e-mlops.