Code Monkey home page Code Monkey logo

lakas's Introduction

Lakas

Game parameter optimizer using nevergrad framework.

Explore on sample interactive plot once an optimization session is done.

Sample plot

Optimization overview

We run an engine vs engine match of 1000 games or so, one engine is called the test_engine and the other is called base_engine. The base_engine will use the default parameter values while the test_engine will use the parameter values that are suggested by the optimizer. Match result is (wins + draws/2) / games. If the result is 0.55 in favor of the test_engine, that means that the parameters suggested by the optimizer is better than the default. We then report back this result to the optimizer as 1 - 0.55 or 0.45 since we are minimizing. All match result should be reported to the optimizer. In the next match we will ask another parameter values to try for the test_engine then play it against the base_engine. This process continues until we reach the maximum budgets. The parameter values that will be recommended by the optimizer at the end of the optimization will be our best parameter values.

A. Setup

Specific installation example

A guide to setup Lakas on virtual environment for windows 10.

General installation guide

  • Install python 3.8 or later.
  • Install nevergrad.
    • pip install nevergrad
  • Install hiplot for plotting.
    • pip install hiplot
  • Install psutil to log cpu and memory usage of python.
    • pip install psutil
  • Download Lakas

B. Command line

See also the examples folder.

python lakas.py --match-manager-path c:/chess/cutechess/cutechess-cli.exe --concurrency 1 --optimizer bayesopt --bo-utility-kind ucb --output-data-file bayesopt_ucb.dat --input-data-file bayesopt_ucb.dat --optimizer-log-file opt_log_plot_ucb.txt --base-time-sec 5 --inc-time-sec 0.05 --budget 100 --games-per-budget 200 --engine ./engines/stockfish-modern/stockfish.exe --input-param "{'RazorMargin': {'init':527, 'lower':100, 'upper':700}, 'KingAttackWeights[2]': {'init':81, 'lower':0, 'upper':150}, 'eMobilityBonus[0][7]': {'init':20, 'lower':-20, 'upper':50}, 'mMobilityBonus[1][7]': {'init':63, 'lower':-50, 'upper':150}, 'eMobilityBonus[3][21]': {'init':168, 'lower':50, 'upper':250}, 'eThreatByRook[1]': {'init':46, 'lower':0, 'upper':150}, 'mThreatByMinor[3]': {'init':77, 'lower':0, 'upper':150}, 'eThreatByKing': {'init':89, 'lower':10, 'upper':150}, 'eThreatByPawnPush': {'init':39, 'lower':0, 'upper':100}}" --opening-file ./start_opening/ogpt_chess_startpos.epd

Help

python lakas.py -h
See help page in the wiki.

Saving hiplot plot

python lakas.py --optimizer-log-file opt_oneplusone.txt ...
After the optimization, plots will be saved in opt_oneplusone.txt.html file.

Parameter Input

Given a uci engine with uci option:

  • option name mThreatByKing type spin default 24 min 0 max 400
  • option name mThreatByPawnPush type spin default 48 min 0 max 400
python lakas.py --input-param "{'mThreatByKing': {'init': 24, 'lower': 0, 'upper': 400}, 'mThreatByPawnPush': {'init': 48, 'lower': 0, 'upper': 400}}" ...

If there is a space in the parameter name like option name Futility Margin type spin default 100 min 0 max 200
--input-param "{'\"Futility Margin\"': {'init': 24, 'lower': 0, 'upper': 400}}"

Parameter Categorical and float values

Given option name mThreatByKing type spin default 24 min 0 max 400
--input-param "{'mThreatByKing': [24, 16, 20, 28, 32]}"
Note the first value should be the default, in this case its 24.

Or with true or false.
--input-param "{'MultiGather': ['false', 'true']}"

or with float value
--input-param "{'CPuct': [1.745, 1.675, 1.875]}"

can also be
--input-param "{'CPuct': {'init': 1.745, 'lower': 1.0, 'upper': 3.0}}"

or a combination
--input-param "{'MultiGather': ['false', 'true'], 'CPuct': {'init': 1.745, 'lower': 1.0, 'upper': 3.0}}"

C. Supported NeverGrad Optimizers

--optimizer oneplusone ...
--optimizer tbpsa ...
--optimizer bayesopt ...
--optimizer spsa ...
--optimizer cmaes ...
--optimizer ngopt ...

D. Sample optimization sessions

E. Resuming a cancelled optimization

Use the option
--output-data-file myopt.dat ...
to save the optimization data into the file myopt.dat. You may resume the optimization by using the option
--input-data-file myopt.dat ...

Note

If you use oneplusone optimizer, you should use it to resume the oneplusone optimization.
Example:
python lakas.py --output-data-file oneplusone.dat --optimizer oneplusone ...

After 2 budgets or so, optimization is cancelled. To resume:
python lakas.py --input-data-file oneplusone.dat --output-data-file oneplusone.dat --optimizer oneplusone ...

The 2 budgets stored in oneplusone.dat are still there and new budgets will be added on that same data file.

If your optimizer is tbpsa save it to a different file.
python lakas.py --output-data-file tbpsa.dat --optimizer tbpsa ...

F. Credits

lakas's People

Contributors

fsmosca avatar

Stargazers

avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

joergoster claes1981 sarona-rd sarona-cccf cristivlas

lakas's Issues

UCI parameters with blank in the name

It seems impossible to pass common parameters with a blank in the name to cutechess.
E.g. I tried to set "Use NNUE" to "false" to tune some classic eval parameters. Cutechess needs to get a parameter "option.Use NNUE=false" with(!) quotation marks but Lakas seems not able to pass them.

Or I am just missing some trick...

EPD test suite

are the EPDs taken randomly or serially from ogpt_chess_startpos.epd? (ie same positions for each budget or random?)

are the games back to back (eg one side plays black, then reverse colours same epd?)

Error using spsa related to float/integer conversion

With latest changes I'm getting the following error when using spsa optimizer:

python -u lakas.py --optimizer spsa --input-data-file bay_nodes100000.dat --output-data-file bay_nodes100000.dat --optimizer-log-file log_bay-nodes100000.txt --nodes 100000 --base-time-sec 10 --inc-time-sec 1 --budget 5 --games-per-budget 500 --concurrency 14 --engine ./engines/RubiChess-AVX2.exe --input-param "{'NnueOutBias': {'init':-1613, 'lower':-1800, 'upper':-1400}, 'NnueOutWeight_00': {'init':-17, 'lower':-97, 'upper':63}, 'NnueOutWeight_01': {'init':-14, 'lower':-94, 'upper':66}, 'NnueOutWeight_02': {'init':23, 'lower':-57, 'upper':103}, 'NnueOutWeight_03': {'init':-87, 'lower':-128, 'upper':-7}, 'NnueOutWeight_04': {'init':4, 'lower':-76, 'upper':84}, 'NnueOutWeight_05': {'init':-44, 'lower':-124, 'upper':36}, 'NnueOutWeight_06': {'init':40, 'lower':-40, 'upper':120}, 'NnueOutWeight_07': {'init':-77, 'lower':-128, 'upper':3}, 'NnueOutWeight_08': {'init':89, 'lower':9, 'upper':127}, 'NnueOutWeight_09': {'init':10, 'lower':-70, 'upper':90}, 'NnueOutWeight_10': {'init':-32, 'lower':-112, 'upper':48}, 'NnueOutWeight_11': {'init':66, 'lower':-14, 'upper':127}, 'NnueOutWeight_12': {'init':-22, 'lower':-102, 'upper':58}, 'NnueOutWeight_13': {'init':-8, 'lower':-88, 'upper':72}, 'NnueOutWeight_14': {'init':-115, 'lower':-128, 'upper':-35}, 'NnueOutWeight_15': {'init':50, 'lower':-30, 'upper':127}, 'NnueOutWeight_16': {'init':-32, 'lower':-112, 'upper':48}, 'NnueOutWeight_17': {'init':-19, 'lower':-99, 'upper':61}, 'NnueOutWeight_18': {'init':-51, 'lower':-128, 'upper':29}, 'NnueOutWeight_19': {'init':-16, 'lower':-96, 'upper':64}, 'NnueOutWeight_20': {'init':127, 'lower':47, 'upper':127}, 'NnueOutWeight_21': {'init':51, 'lower':-29, 'upper':127}, 'NnueOutWeight_22': {'init':-19, 'lower':-99, 'upper':61}, 'NnueOutWeight_23': {'init':19, 'lower':-61, 'upper':99}, 'NnueOutWeight_24': {'init':-23, 'lower':-103, 'upper':57}, 'NnueOutWeight_25': {'init':-16, 'lower':-96, 'upper':64}, 'NnueOutWeight_26': {'init':92, 'lower':12, 'upper':127}, 'NnueOutWeight_27': {'init':-26, 'lower':-106, 'upper':54}, 'NnueOutWeight_28': {'init':20, 'lower':-60, 'upper':100}, 'NnueOutWeight_29': {'init':-9, 'lower':-89, 'upper':71}, 'NnueOutWeight_30': {'init':-11, 'lower':-91, 'upper':69}, 'NnueOutWeight_31': {'init':-10, 'lower':-90, 'upper':70}}" --opening-file ./start_opening/noob_3moves.epd --opening-file-format epd

python : Traceback (most recent call last):
In Zeile:1 Zeichen:1
+ python -u lakas.py --optimizer spsa --input-data-file spsa80_nodes100 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
  File "lakas.py", line 905, in <module>
    main()
  File "lakas.py", line 867, in main
    loss = objective.run(**x.kwargs)
  File "C:\Schach\game_param_tuner\Lakas-main\myvenv\lib\site-packages\nevergrad\parametrization\container.py", line 89, in kwargs
    return self[1].value  # type: ignore
  File "C:\Schach\game_param_tuner\Lakas-main\myvenv\lib\site-packages\nevergrad\parametrization\core.py", line 555, in value
    return {k: as_parameter(p).value for k, p in self._content.items()}
  File "C:\Schach\game_param_tuner\Lakas-main\myvenv\lib\site-packages\nevergrad\parametrization\core.py", line 555, in <dictcomp>
    return {k: as_parameter(p).value for k, p in self._content.items()}
  File "C:\Schach\game_param_tuner\Lakas-main\myvenv\lib\site-packages\nevergrad\parametrization\data.py", line 503, in value
    return float(self._value[0]) if not self.integer else int(np.round(self._value[0]))
ValueError: cannot convert float NaN to integer

Maybe related to a844798

Feature proposal: Linux support

I noticed it was not as simple as changing line 320 from
tour_manager = Path(Path.cwd(), './tourney_manager/cutechess/cutechess-cli.exe')
to
tour_manager = '/usr/bin/cutechess-cli'

result:

python lakas.py --engine /partitions/Sandisk/xfs/media/data/chess/engines/Cfish/Cfish-20100303_pgo_extra --base-time-sec 1 --inc-time-sec 0.5 --optimizer bayesopt --output-data-file test1.dat --optimizer-log-file test1.txt --input-param "{'Threads': {'init': 8, 'lower': 1, 'upper': 8}}" --common-param "{'EvalFile': '/partitions/Sandisk/xfs/media/data/chess/engines/Stockfish_development_versions/nets/nn-baeb9ef2d183.nnue', 'Contempt': 11}" --opening-file /home/claes/Lakas/start_pos.pgn --opening-file-format pgn
2021-03-10 11:56:41,687 | INFO  | Lakas v0.33.0
2021-03-10 11:56:41,687 | INFO  | input param: OrderedDict([('Threads', {'init': 8, 'lower': 1, 'upper': 8})])

2021-03-10 11:56:41,687 | INFO  | total budget: 1000
2021-03-10 11:56:41,687 | INFO  | games/budget: 100
2021-03-10 11:56:41,687 | INFO  | move control: base_time_sec: 1, inc_time_sec: 0.5, depth=None, nodes=None
2021-03-10 11:56:41,688 | INFO  | parameter dimension: 1
2021-03-10 11:56:41,688 | INFO  | deterministic function: False
2021-03-10 11:56:41,688 | INFO  | use best param: False
2021-03-10 11:56:41,688 | INFO  | optimizer: bayesopt, initialization: Hammersley, init_budget: None, middle_point: False, utility_kind: ucb, utility_kappa: 2.576, utility_xi: 0.0, gp_parameters: {'alpha': 0.001, 'normalize_y': True, 'n_restarts_optimizer': 5, 'random_state': None}

2021-03-10 11:56:41,696 | INFO  | budget: 1
2021-03-10 11:56:41,696 | INFO  | recommended param: {'Threads': 8}
2021-03-10 11:56:41,696 | INFO  | best param: {'Threads': 8}
2021-03-10 11:56:41,697 | INFO  | best loss: 0.5
2021-03-10 11:56:41,697 | INFO  | init param: {'Threads': 8}
2021-03-10 11:56:41,697 | INFO  | common param: {'EvalFile': '/partitions/Sandisk/xfs/media/data/chess/engines/Stockfish_development_versions/nets/nn-baeb9ef2d183.nnue', 'Contempt': 11}
2021-03-10 11:56:41,697 | INFO  | recommended vs init
Traceback (most recent call last):
  File "lakas.py", line 931, in <module>
    main()
  File "lakas.py", line 893, in main
    loss = objective.run(**x.kwargs)
  File "lakas.py", line 244, in run
    result = engine_match(self.engine_file, test_options, base_options,
  File "lakas.py", line 398, in engine_match
    process = Popen(str(tour_manager) + command, stdout=PIPE, text=True)
  File "/usr/lib64/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.8/subprocess.py", line 1706, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/cutechess-cli -concurrency 1 -tournament round-robin -pgnout nevergrad_games.pgn fi -each tc=0/0:1+0.5 -engine cmd=/partitions/Sandisk/xfs/media/data/chess/engines/Cfish/Cfish-20100303_pgo_extra name=test option.Threads=8 option.EvalFile=/partitions/Sandisk/xfs/media/data/chess/engines/Stockfish_development_versions/nets/nn-baeb9ef2d183.nnue option.Contempt=11 proto=uci -engine cmd=/partitions/Sandisk/xfs/media/data/chess/engines/Cfish/Cfish-20100303_pgo_extra name=base option.Threads=8 option.EvalFile=/partitions/Sandisk/xfs/media/data/chess/engines/Stockfish_development_versions/nets/nn-baeb9ef2d183.nnue option.Contempt=11 proto=uci -rounds 50 -games 2 -repeat 2 -recover -wait 5000 -openings file=/home/claes/Lakas/start_pos.pgn order=random format=pgn -resign movecount=6 score=700 twosided=true -draw movenumber=30 movecount=6 score=1'

Entering /usr/bin/cutechess-cli -concurrency 1 -tournament round-robin -pgnout nevergrad_games.pgn fi -each tc=0/0:1+0.5 -engine cmd=/partitions/Sandisk/xfs/media/data/chess/engines/Cfish/Cfish-20100303_pgo_extra name=test option.Threads=8 option.EvalFile=/partitions/Sandisk/xfs/media/data/chess/engines/Stockfish_development_versions/nets/nn-baeb9ef2d183.nnue option.Contempt=11 proto=uci -engine cmd=/partitions/Sandisk/xfs/media/data/chess/engines/Cfish/Cfish-20100303_pgo_extra name=base option.Threads=8 option.EvalFile=/partitions/Sandisk/xfs/media/data/chess/engines/Stockfish_development_versions/nets/nn-baeb9ef2d183.nnue option.Contempt=11 proto=uci -rounds 50 -games 2 -repeat 2 -recover -wait 5000 -openings file=/home/claes/Lakas/start_pos.pgn order=random format=pgn -resign movecount=6 score=700 twosided=true -draw movenumber=30 movecount=6 score=1 on the command line starts a cutechess-cli match as expected, so the command seems to work there.

Concurrency and CPU usage

Testing using AMD with 64 cores.

If run at concurrency 50, works for a budget or two then gives 'game incomplete' fail.
Same situation concurrency at 40, 30.
Concurrency 20 works (so far).

Observing the CPU usage, there's a demand spike at the end of each game run (one budget), which for concurrency 30, 40, 50 takes CPU usage to 100% and the system gets its knickers in a twist and starts reporting fails.

At concurrency 20. CPU usage is 44%, 255 processes, 3372 threads, and 79Gb RAM in use.
At the end of each game run (one budget), CPU usage doubles to 80%, processes to 333, threads to 3590 and RAM to 131 GB. I'm getting away with concurrency 20, so far.

Guessing here, but it looks like, at the end of each budget for some reason, it fires up both master and test engines (2x20) and does some processing, using 40(?) cores for a few seconds?
It fails for high concurrency, because 60, 80, 100 core fire ups are too much for it? Well, too much when using my engine.
Is there anyway to lower the increased core demand at the end of each budget?

fixed time per move

Hi Ferdy,

Is it possible to ask for a fixed time to move option? Probably I am being dumb, but I can't manage to set anything other then a depth limit, when I'ld prefer to set something like movetime=0.01

Tx if possible.

Feature request: Fixed nodes per move

I would really like to see support for cutechess "nodes=N" mode in Lakas.

Reason:
I found out that training with a fixed depth doesn't work. With fixed depth training I always get parameter sets where the engines runs in a lot of fail-high/fail-lows (which gives many nodes while staying inside the max depth) while playing real games with real time control with this parameter set you just don't get the needed depth and fail badly.

I could believe that training with fixed nodes works better while still avoiding real time controls and the related inaccuracy by CPU load etc.

Some questions about how the optimizer works and best practice in usage

No issue but some questions came to my mind while I'm using your very useful and promising tool (thanks for developing it)...

  1. Lets say ran the optimizer for 100 budgets, each budget playing 500 games and I notice that I get high values for the loss function, most of them > 0.9 and all far from beating the 0.5. Is it okay to continue the optimizer on this data file for another 100 or even more budgets but now with reduced games per budget, e.g. 200 so save some time assuming that the optimizer needs a lot more runs to get even close to the 0.5 area? Or is it important to use a constant games_per_budget for the complete process?
  2. Lets assume the optimizer found a very good set of values with a loss value of 0.45. And lets assume this good score was a fluke, maybe caused by too small games_per_budget. Will these "fluked" parameter set ever be retested? Will other value sets with a "real" loss value of 0.47 ever get a change to replace the fluked one?
  3. Maybe related to the last question: When continuing an optimization, should I replace the original init values (I usually use the defaults of the engine when starting the optimization for the first time) with the best values found so far (e.g. the values from the fluked set in last question). Can I still use the same --input-data-file (= --output-data-file of last run) even if I change the inits?
  4. Replacing the base engine... should I ever do that? Probably only when I found something that passes the verification test.

I could probably write some more questions but lets go step by step.
Thanks in advance.

Feature/Help request

It would be awesome to have a version which simply calls the engine with a configurable command, like 'stockfish bench 128 1 4 1000.epd depth mixed', instead of running a tournament manager, e.g. cutechess-cli. I think many other engines provide a similar functionality, too.

I've written a very basic Alpha-Beta search for Stockfish. Now I want to work on move-ordering but not in the usual way. I don't want to call a MovePicker object, but want to score all moves and sort them before searching. (I know this will be slow!)
I want to tune this scoring algorithm.
The nice thing with a plain AB search is that move-ordering alone decides about how many nodes it takes to finish a given depth. The best move and eval don't change.

Maybe you already have something like this, or you can at least give some advice what I need to change?

Test Lakas with new versions of dependent modules

Upcoming changes on the requirements.txt

Major changes

nevergrad==0.4.3.post8, was nevergrad==0.4.3
scikit-learn==1.0, was scikit-learn==0.23.2
scipy==1.7.1, was scipy==1.5.2
cma==3.1.0, was cma==3.0.3
hiplot==0.1.29, was hiplot==0.1.19
numpy==1.21.2, was numpy==1.19.2

Optimizer tests

  • --optimizer oneplusone
  • --optimizer tbpsa
  • --optimizer bayesopt
  • --optimizer spsa
  • --optimizer cmaes
  • --optimizer ngopt

Suggestion : export html at each step 

diff --git a/lakas.py b/lakas.py
index 3196a93..5faa693 100644
--- a/lakas.py
+++ b/lakas.py
@@ -1032,6 +1037,9 @@ def main():
         if output_data_file is not None:
             optimizer.dump(output_data_file)
 
+        exp = nevergrad_logger.to_hiplot_experiment()
+        exp.to_html(f'{optimizer_log_file}.html')
+
     # Optimization done, get the best param.
     recommendation = optimizer.provide_recommendation()
     best_param = recommendation.value

Suggesting more optimization methods from Nevergrad

Hello! just suggesting adding the new optimization methods from Nevergrad, that might be more relevant in your context:

  • you can now specify "deterministic_function = False" in Nevergrad, when your objective function is stochastic; not sure this is the case for you.
  • we have a NGOpt method, which automatically selects one of the optimization methods depending on the characteristics of your problem. In particular in the noisy context this might switch to SQP or DiscreteNoisyOnePlusOne or other noisy optimization methods which are not bad on our own game benchmarks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.