ytopt-team / ytopt Goto Github PK
View Code? Open in Web Editor NEWytopt: machine-learning-based search methods for autotuning
License: BSD 2-Clause "Simplified" License
ytopt's People
Contributors
Stargazers
Watchers
Forkers
vinusreenivasan looptilingbenchmark tlranda xiehuohuo andreabrantes jaehoonkoo kerilk nchristensen jmlarson1 jlnav hpcrl sameeranjoshi tupendra malithj prav2508 wuxf99ytopt's Issues
I'm trying to replicate the experiments on xsbench-omp but I keep running into the following error
I'm using the following command to run the experiment from the problem.py directory
python3 -m ytopt.search.ambs --evaluator ray --problem problem.Problem --max-evals=10 --learner RF
I get the following error when trying to use this:
Uncaught exception <class 'IndexError'>: list index out of rangeTraceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/akashdutta/Documents/git_repos/ytopt/ytopt/search/ambs.py", line 109, in <module>
search.main()
File "/home/akashdutta/Documents/git_repos/ytopt/ytopt/search/ambs.py", line 80, in main
XX = self.optimizer.ask_initial(n_points=self.num_workers)
File "/home/akashdutta/Documents/git_repos/ytopt/ytopt/search/optimizer/optimizer.py", line 103, in ask_initial
XX = self._optimizer.ask(n_points=n_points)
File "/home/akashdutta/Documents/git_repos/scikit-optimize/skopt/optimizer/optimizer.py", line 426, in ask
x = opt.ask()
File "/home/akashdutta/Documents/git_repos/scikit-optimize/skopt/optimizer/optimizer.py", line 399, in ask
return self._ask()
File "/home/akashdutta/Documents/git_repos/scikit-optimize/skopt/optimizer/optimizer.py", line 465, in _ask
return self.space.rvs(Xi=self.Xi, random_state=self.rng)[0]
IndexError: list index out of range
Lies count against Skopt.Optimizer's _n_initial_points
In reference to ytopt.search.optimizer.Optimizer._ask():103 and ytopt.search.optimizer.Optimizer.ask_initial():130.
In the lines referenced above, ytopt leverages lies to the underlying Scikit-optimize Optimizer object as part of its sampling strategy. However, this means that ytopt reduces that object's internal counter for initial points on BOTH during the lie and during the real .tell() when actual evaluation results are available, leading to unexpected behavior if Scikit-optimize is supposed to behave differently after its initial warmup phase.
Since the ytopt wrapping optimizer is using the lie as part of its point-collecting strategy, it should increment the underlying optimizer's _n_initial_points variable by one with each lie so that Scikit-opt continues to generate configurations as expected between its initial warmup phase and post-warmup execution. This can be done by simply adding the instruction: self._optimizier._n_initial_points += 1 after each of the lines referenced in this issue.
Is this runtime error message harmful?
I tried following the instructions in the matrix multiplication tutorial. Is the numpy warning that I am getting an issue/ can I ignore it? The final point that I am getting is not the best runtime. How can I determine the best runtime? Should I be looking at the csv/json file?
python -m ytopt.search.ambs --evaluator ray --problem problem.Problem --max-evals=5 --learner RF
2024-03-14 14:52:57,321 INFO worker.py:1724 -- Started a local Ray instance.
(compute_objective pid=119908) CONFIG: {'BLOCK_SIZE': 5}
(compute_objective pid=119908) OUTPUT: 0.134
(compute_objective pid=119908) CONFIG: {'BLOCK_SIZE': 2}
(compute_objective pid=119908) OUTPUT: 0.464
(compute_objective pid=119908) CONFIG: {'BLOCK_SIZE': 8}
(compute_objective pid=119908) OUTPUT: 0.055
/home/reetog/miniconda3/envs/ytune/lib/python3.10/site-packages/numpy/core/numeric.py:330: RuntimeWarning: invalid value encountered in cast
multiarray.copyto(a, fill_value, casting='unsafe')
(compute_objective pid=119908) CONFIG: {'BLOCK_SIZE': 6}
(compute_objective pid=119908) OUTPUT: 0.093
/home/reetog/miniconda3/envs/ytune/lib/python3.10/site-packages/numpy/core/numeric.py:330: RuntimeWarning: invalid value encountered in cast
multiarray.copyto(a, fill_value, casting='unsafe')
/home/reetog/miniconda3/envs/ytune/lib/python3.10/site-packages/numpy/core/numeric.py:330: RuntimeWarning: invalid value encountered in cast
multiarray.copyto(a, fill_value, casting='unsafe')
(compute_objective pid=119908) CONFIG: {'BLOCK_SIZE': 7}
(compute_objective pid=119908) OUTPUT: 0.087
Running in parallel
I am using ytopt for running a few thousand sampling tests. I want to know if there is any way to run them in parallel or speed up the searching somehow.
I have inputs such that almost 60% of sampling inputs are false positives, so even reducing them from the total sampling space would also be another pointer I wish to know if any.
I tried to use the conditionals and forbidden but they don't seem to be applicable for the use case which I am trying to get working, my usecase is that I want to keep the multiplication of tx * ty which are 2 integer hyperparameters below 1024 which refers to the threadIdx.x andthreadIdx.ywhich should be less than 1024 the total thread limit on a GPU, any suggestions would be helpful.
Thanks
Requirement 'tensorflow==1.14.0' from setup.py is not found in PyPi.
I assume its because my Ubuntu installation uses python38, and old TensorFlow version are not provided for newer versions of python. However, it seems to work find with the newest tensorflow.
ConfigSpace objects break SkOptimizer instantiation
The Optimizer class appears to have handling for ConfigurationSpace objects (
ytopt/ytopt/search/optimizer/optimizer.py
Line 43 in 8c8f1f3
Reproducer:
from autotune import TuningProblem
from autotune.space import *
from skopt.space import Real
from ytopt.search.ambs import AMBS
import ConfigSpace as cs
input_space = cs.ConfigurationSpace("autotuning_space")
input_space.add_hyperparameter(cs.Categorical("a", [0]))
output_space = Space([Real(0.0, inf, name="avg_time")])
def obj_func(p):
return 1
problem = TuningProblem(
task_space=None,
input_space=input_space,
output_space=output_space,
objective=obj_func,
constraints=None,
model=None)
searcher = AMBS(problem=problem, evaluator="ray")
searcher.main()Stack trace:
Uncaught exception <class 'ValueError'>: Dimension has to be a list or tuple.Traceback (most recent call last):
File "/home/njchris2/Workspace/tagtune/ytopt_reproducer.py", line 22, in <module>
searcher = AMBS(problem=problem, evaluator="subprocess")
File "/home/njchris2/Workspace/ytopt/ytopt/search/ambs.py", line 48, in __init__
self.optimizer = Optimizer(
File "/home/njchris2/Workspace/ytopt/ytopt/search/optimizer/optimizer.py", line 44, in __init__
self._optimizer = SkOptimizer(
File "/home/njchris2/miniconda3/envs/dev/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", line 271, in __init__
self.space = Space(dimensions)
File "/home/njchris2/miniconda3/envs/dev/lib/python3.9/site-packages/skopt/space/space.py", line 771, in __init__
self.dimensions = [check_dimension(dim) for dim in dimensions]
File "/home/njchris2/miniconda3/envs/dev/lib/python3.9/site-packages/skopt/space/space.py", line 771, in <listcomp>
self.dimensions = [check_dimension(dim) for dim in dimensions]
File "/home/njchris2/miniconda3/envs/dev/lib/python3.9/site-packages/skopt/space/space.py", line 91, in check_dimension
raise ValueError("Dimension has to be a list or tuple.")
ValueError: Dimension has to be a list or tuple.
Tutorial improvement
mmm-block tutorial
- I suggest to explain what BLOCK_SIZE is and why we would want to optimize it.
- Going through the details of Plopper seems a bit much for a tutorial
- Creating the TuningProblem object seems central to the tutorial and should be explained more.
- Explaining the command line paramters wouls be nice
- "Once autotuning kick off, ytopt.log, results.csv, and results.json will be rendered." Usually, "render" does not refer to writing to a file. The output I see:
/home/meinersbur/build/ytopt/tutorial/lib/python3.8/site-packages/ray/_private/services.py:238: UserWarning: Not all Ray Dashboard dependencies were found. To use the dashboard please install Ray using pip install ray[default]. To disable this message, set RAY_DISABLE_IMPORT_WARNING env var to '1'.
warnings.warn(warning_message)
(pid=4874) CONFIG: {'BLOCK_SIZE': 9}
(pid=4874) OUTPUT:%f 0.024
(pid=4874) CONFIG: {'BLOCK_SIZE': 10}
(pid=4874) OUTPUT:%f 0.023
(pid=4874) CONFIG: {'BLOCK_SIZE': 6}
(pid=4874) OUTPUT:%f 0.034
(pid=4874) CONFIG: {'BLOCK_SIZE': 2}
(pid=4874) OUTPUT:%f 0.14
Somehow it also launches a redis server and wants access through my firewall.
- Dumping the entire ytopt.log seem unnecessary in a tutorial.
Breaking issue with YTOPT.
We have been running the tool for some internal work since last 5 months.
I have tried to recently rerun and some strange errors are seen recently.
~/workspace/SANDBOX/ytopt/ytopt/ytopt/ytopt/benchmark/mmm-block/mmm_problem$ python3 -m ytopt.search.ambs --evaluator ray --problem problem.Problem --max-evals=10 --learner RF
2023-07-26 15:21:33,509 INFO worker.py:1621 -- Started a local Ray instance.
Uncaught exception <class 'TypeError'>: Expected list, got setTraceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/sameeran/workspace/SANDBOX/ytopt/ytopt/ytopt/ytopt/search/ambs.py", line 125, in <module>
search = AMBS(**vars(args))
File "/home/sameeran/workspace/SANDBOX/ytopt/ytopt/ytopt/ytopt/search/ambs.py", line 48, in __init__
self.optimizer = Optimizer(
File "/home/sameeran/workspace/SANDBOX/ytopt/ytopt/ytopt/ytopt/search/optimizer/optimizer.py", line 44, in __init__
self._optimizer = SkOptimizer(
File "/home/sameeran/workspace/SANDBOX/ytopt/ytopt/scikit-optimize/skopt/optimizer/optimizer.py", line 315, in __init__
self.space = Space(dimensions)
File "/home/sameeran/workspace/SANDBOX/ytopt/ytopt/scikit-optimize/skopt/space/space.py", line 920, in __init__
cond_hps = self.config_space.get_all_conditional_hyperparameters()
File "ConfigSpace/configuration_space.pyx", line 1024, in **ConfigSpace.configuration_space.ConfigurationSpace.get_all_conditional_hyperparameters
TypeError: Expected list, got set**
Here's my git diff.
sameeran@echo11:~/workspace/SANDBOX/ytopt/ytopt/ytopt/ytopt/benchmark/mmm-block/mmm_problem$ git diff
diff --git a/ytopt/benchmark/mmm-block/mmm_problem/problem.py b/ytopt/benchmark/mmm-block/mmm_problem/problem.py
index aa7253a..ab9877f 100644
--- a/ytopt/benchmark/mmm-block/mmm_problem/problem.py
+++ b/ytopt/benchmark/mmm-block/mmm_problem/problem.py
@@ -13,8 +13,8 @@ from plopper import Plopper
# create an object of ConfigSpace
cs = CS.ConfigurationSpace(seed=1234)
#block size for openmp dynamic schedule
-# p0= CSH.OrdinalHyperparameter(name='BLOCK_SIZE', sequence=['1','2','3','4','5','6','7','8','9','10'], default_value='5')
-p0= CSH.UniformIntegerHyperparameter(name='BLOCK_SIZE', lower=1, upper=10, default_value=5)
+p0= CSH.OrdinalHyperparameter(name='BLOCK_SIZE', sequence=['1','2','3','4','5','6','7','8','9','10'], default_value='5')
+#p0= CSH.UniformIntegerHyperparameter(name='BLOCK_SIZE', lower=1, upper=10, default_value=5)
cs.add_hyperparameters([p0])
# problem space
Related publications
It would be great if publications associated with ytopt could be linked in the Readme.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.