Comments (3)
jarokaz
commented on August 25, 2024
Increasing the capacity of the cluster hosting KFP will not help. Most of the time is spent in starting and executing Dataflow and AI Platform Pipelines jobs. The long execution time for AI Platform training does not look right.
from mlops-on-gcp.
jarokaz
commented on August 25, 2024
I have fixed the slow training in AI Platform Training. The issue was a Tensorboard callback that was writing to GCS. The next step is to look at optimizing Dataflow, which most likely will be just an incremental change
from mlops-on-gcp.
dougkelly
commented on August 25, 2024
Thanks, excellent detective work and fix! I just ran through the lab again and saw a significant improvement in runtimes of 0:45:28 and 0:46:12 (last one even added InfraValidator component to DAG as well). This is great and gives us a lot more flexibility to re-use this lab scoped to take about an hour end-2-end (reading + explanation + pipeline runtime) in a number of different training formats going forward e.g. workshops, ASL, future Qwiklab conversion, etc. Marking this as successfully fixed.
from mlops-on-gcp.
Related Issues (20)
- Bad argument name in "chicago_taxi_dag.py"
- TFX on Cloud AI Platform Pipelines: "Activate Cloud Shell" unnecessary HOT 1
- TFX on Cloud AI Platform Pipelines: GKE cluster not provisioned on lab startup HOT 1
- TFX on Cloud AI Platform Pipelines: kfp not installed by default
- TFX on Cloud AI Platform Pipelines: Bucket not created by default HOT 2
- TFX on Cloud AI Platform Pipelines: Skaffold not deployed on Notebooks instance
- TFX on Cloud AI Platform Pipelines: TFX pipeline run cannot write into defined bucket (403 Insufficient Permission)
- Missing `covertype_training_pipeline.py`
- MLFLOW_TRACKING_EXTERNAL_URI is empty. It seems MLFlow Tracking Server is not properly deployed HOT 8
- ml-ops examples do not work
- mlops-composer-mlflow/install.sh old composer version issue
- Missing CS dataset
- build HOT 1
- [TFX on Cloud AI Platform Pipelines] Cannot pass Deploy the pipeline package to AI Platform Pipelines HOT 2
- Something
- `dag` parameter not required because the operator is in the DAG context already
- [TFX Standard Components Walkthrough] Outdated library dependencies HOT 4
- Typos in workshops/tfx-caip-tf23/lab-01-tfx-walkthrough
- workshops/tfx-caip-tf23/lab-02-tfx-pipeline lab's last two checkpoints cannot be satisfied HOT 1
- `Pusher.png` mistakenly labeled as "InfraValidator"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mlops-on-gcp.