Comments (6)
cherusk
commented on July 28, 2024
1
@nzw0301 yes, you can see the import also in the code snippet I provided.
But I ran it manually now also in the airflow container:
airflow@56d394fe9f60:/opt/airflow$ python3
Python 3.9.15 (main, Oct 25 2022, 05:49:37)
[GCC 10.2.1 20210110] on linux
>>> import optuna
>>>
It succeeded as you can see. That the error stems from the dask workers I doubt, because it fails already at the optuna.integration.DaskStorage step.
from optuna-examples.
cherusk
commented on July 28, 2024
1
Thanks, can you import |DaskStorage| like
***@***.***:/opt/airflow$ python3
Python 3.9.15 (main, Oct 25 2022, 05:49:37)
[GCC 10.2.1 20210110] on linux
>>> from optuna.integration import DaskStorage
Yes, also that works when done manually.
I've done that now manually for all components to be imported, with success:
```
***@***.***:/opt/airflow$ python3
Python 3.9.15 (main, Oct 25 2022, 05:49:37)
[GCC 10.2.1 20210110] on linux
>> from optuna.integration import DaskStorage
>> from dask.distributed import Client
>> from dask.distributed import wait
>>
```
from optuna-examples.
cherusk
commented on July 28, 2024
Maybe @jrbourbeau, because he authored the example code!?
from optuna-examples.
nzw0301
commented on July 28, 2024
Could you try import optuna on your environment to make sure your env is correctly setup?
from optuna-examples.
cherusk
commented on July 28, 2024
The example does also does not state the version requirements at the point when it worked. 😢
from optuna-examples.
cherusk
commented on July 28, 2024
I've found the issue.
The installs on the scheduler and the workers was done with conda, that's where this was coming from.
File "/opt/conda/lib/python3.9/site-packages/distributed/core.py", line 820, in _handle_comm
File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 3217, in run
File "/opt/conda/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 73, in loads
Installing the pre-release version of optuna on the target dask cluster solved the issue. 😳
from optuna-examples.
Related Issues (20)
- AttributeError: 'Trainer' object has no attribute 'training_type_plugin' HOT 5
- Implement cross-validation examples for Optuna Terminator 🤖 HOT 3
- Add `plot_rank` and `plot_timeline` to visualisation examples HOT 2
- Trial Fail HOT 1
- Optuna DDP with Slurm Cluster HOT 1
- License of the optuna-examples repo? HOT 2
- optuna.integration.XGBoostPruningCallback is not xgboost TrainingCallback HOT 2
- Updating the PyTorch Lightning example to >= 2.0 HOT 5
- Pruning Not Working in Pytorch HOT 1
- Support Lighting instead/besides Pytorch Lighting HOT 1
- Flax example HOT 2
- optuna-examples/xgboost /xgboost_integration.py error HOT 1
- Add `README.md` to `./dashboard` HOT 3
- Pruning only on some models? HOT 2
- Add Python 3.12 to the CI HOT 8
- Add README to each directory if it contains multiple examples HOT 2
- XGBoost callback problem HOT 2
- The problem of DDP training with pytorch lightning HOT 1
- NSGAIISampler number trials per generation HOT 1
- intermediate values and objective value use different metrics in `LightGBMPruningCallback` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from optuna-examples.