This is very promising. I was wondering if there is a way to store the results (let sa

Store results in blob about azureml-and-dask HOT 1 CLOSED

AliCharkhi commented on July 19, 2024

Store results in blob

from azureml-and-dask.

Comments (1)

danielsc commented on July 19, 2024

As you will have found out by now, the datastore that is being passed in to the RunConfig can be written to, so you just need to write to that folder. In startDask.py, I am saving the datastore path to the runhistory from which it can be retrieved easily:

output_path = run.get_metrics()['datastore'] + '/output/' + output_uuid + '.parquet'

print('save parquet to ', output_path)

taxi_df.to_parquet(output_path)

see https://github.com/danielsc/azureml-and-dask/blob/master/batch/dask/startDask.py
and https://github.com/danielsc/azureml-and-dask/blob/master/batch/dask/batch.py

In case of a batch run that terminates properly, you can just write to ./outputs -- that folder will be uploaded to blob storage after the run completes.

Thirdly, you can use abfs to write directly to blob storage

run = Run.get_context()
ws = run.experiment.workspace
ds = ws.get_default_datastore()
ACCOUNT_NAME = ds.account_name
ACCOUNT_KEY = ds.account_key
CONTAINER = ds.container_name

output_uuid = uuid.uuid1().hex
run.log('output_uuid', output_uuid)
print('save parquet to ', f"abfs://{CONTAINER}/output/{output_uuid}.parquet")
taxi_df.to_parquet(f"abfs://{CONTAINER}/output/{output_uuid}.parquet", 
                   storage_options=STORAGE_OPTIONS, engine='pyarrow')