I'm currently using the Kedro-SageMaker plugin to run a Kedro pipeline on SageMaker. My Kedro project has multiple parameters (using several parameters YAML files).

While executing kedro sagemaker run -i <image name>, I encountered this error:

ClientError: ValidationError for UpdatePipeline operation: Unable to parse pipeline definition. Model validation failed - container ProcessingEnvironmentMap length (132) exceeds maximum limit (100).

Here are the steps to reproduce the issue.

• Add a dummy_parameter.yml file under conf/base/parameters/ in the kedro space flight example code. (check dummy_pamaramter.yml file content below)
• Simply run kedro sagemaker run --auto-build

dummy_parameter.yml file content:

dummy_params:
  parameter1: 1
  parameter2: 2
  parameter3: 3
  parameter4: 4
  parameter5: 5
  .
  .
  .
  .
  .
 parameter100: 100

While trying to implement spaceflight tutorial but i am getting error when running "kedro sagemaker run --auto-build" command.

Using following package versions:
kedro == 0.18.13
kedro-datasets ==1.7.0
kedro-sagemaker == 0.3.0
kedro-viz == 6.5.0

Error:
ValidationError: 1 validation error for KedroSageMakerPluginConfig
root
KedroSageMakerPluginConfig expected dict not NoneType (type=type_error)

Hello and thanks for this plugin. I have three questions:

1. I want to assign a tag (in addition to those assigned automatically like 'sagemaker:pipeline-execution-arn' and 'sagemaker:pipeline-step-name') to the Processing Jobs launched through this plugin. Is there a way to do this automatically without interfacing through the AWS management console?
2. I want that all the nodes runs on the same instance. The default behaviour is that when a step of the pipeline is executed a new instance is start-up and when it finishes the instance is shut-down increasing the total execution time. Is there a way to make it executing on the same instance to optimize execution time?
3. Moreover I have some nodes that can be executed in parallel, can I do this configuring the plugin in a certain way?

Hi,

Thanks for this cool project! I would like to integrate Kedro with Sagemaker. If I understand things correctly, using kedro-sagemaker I can run a Kedro Pipeline in Sagemaker Pipelines. This would result in a trained model that I can deploy as a Sagemaker Endpoint for inference.

When the deployed model receives a request, the data still needs to be transformed (e.g. scaling, one-hot encoding, ...). Is there a way to run the Kedro Pipeline as part of the Sagemaker Endpoint using transformations that are fitted during training?

I want to give a flask app.py as the main file while building docker. The endpoint in this flask will call the kedro cli, build the pipeline and deploy to sagemaker. Does it seem feasible?

Adding as a reminder of things to do/consider.