Question about .fit() in 2.3.0 about d3rlpy HOT 5 CLOSED

Installing d3rlpy==1.1.1 and after downgrading numpy already to 1.24.3 and cython to 0.29.35 instead their latest versions (which resolved one issue) i get the following error in my python terminal of visual studio code:
Building wheels for collected packages: d3rlpy
Building wheel for d3rlpy (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [131 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\argument_utility.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\base.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\cli.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\constants.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\containers.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\context.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\datasets.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\decorators.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\gpu.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\itertools.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\logger.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\torch_utility.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy_version.py -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy
creating build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\awac.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\base.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\bc.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\bcq.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\bear.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\combo.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\cql.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\crr.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\ddpg.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\dqn.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\iql.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\mopo.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\nfq.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\plas.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\random_policy.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\sac.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\td3.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\td3_plus_bc.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos\utility.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
copying d3rlpy\algos_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos
creating build\lib.win-amd64-cpython-311\d3rlpy\dynamics
copying d3rlpy\dynamics\base.py -> build\lib.win-amd64-cpython-311\d3rlpy\dynamics
copying d3rlpy\dynamics\probabilistic_ensemble_dynamics.py -> build\lib.win-amd64-cpython-311\d3rlpy\dynamics
copying d3rlpy\dynamics_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\dynamics
creating build\lib.win-amd64-cpython-311\d3rlpy\envs
copying d3rlpy\envs\wrappers.py -> build\lib.win-amd64-cpython-311\d3rlpy\envs
copying d3rlpy\envs_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\envs
creating build\lib.win-amd64-cpython-311\d3rlpy\iterators
copying d3rlpy\iterators\base.py -> build\lib.win-amd64-cpython-311\d3rlpy\iterators
copying d3rlpy\iterators\random_iterator.py -> build\lib.win-amd64-cpython-311\d3rlpy\iterators
copying d3rlpy\iterators\round_iterator.py -> build\lib.win-amd64-cpython-311\d3rlpy\iterators
copying d3rlpy\iterators_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\iterators
creating build\lib.win-amd64-cpython-311\d3rlpy\metrics
copying d3rlpy\metrics\comparer.py -> build\lib.win-amd64-cpython-311\d3rlpy\metrics
copying d3rlpy\metrics\scorer.py -> build\lib.win-amd64-cpython-311\d3rlpy\metrics
copying d3rlpy\metrics_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\metrics
creating build\lib.win-amd64-cpython-311\d3rlpy\models
copying d3rlpy\models\builders.py -> build\lib.win-amd64-cpython-311\d3rlpy\models
copying d3rlpy\models\encoders.py -> build\lib.win-amd64-cpython-311\d3rlpy\models
copying d3rlpy\models\optimizers.py -> build\lib.win-amd64-cpython-311\d3rlpy\models
copying d3rlpy\models\q_functions.py -> build\lib.win-amd64-cpython-311\d3rlpy\models
copying d3rlpy\models_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\models
creating build\lib.win-amd64-cpython-311\d3rlpy\online
copying d3rlpy\online\buffers.py -> build\lib.win-amd64-cpython-311\d3rlpy\online
copying d3rlpy\online\explorers.py -> build\lib.win-amd64-cpython-311\d3rlpy\online
copying d3rlpy\online\iterators.py -> build\lib.win-amd64-cpython-311\d3rlpy\online
copying d3rlpy\online\utility.py -> build\lib.win-amd64-cpython-311\d3rlpy\online
copying d3rlpy\online_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\online
creating build\lib.win-amd64-cpython-311\d3rlpy\ope
copying d3rlpy\ope\fqe.py -> build\lib.win-amd64-cpython-311\d3rlpy\ope
copying d3rlpy\ope_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\ope
creating build\lib.win-amd64-cpython-311\d3rlpy\preprocessing
copying d3rlpy\preprocessing\action_scalers.py -> build\lib.win-amd64-cpython-311\d3rlpy\preprocessing
copying d3rlpy\preprocessing\reward_scalers.py -> build\lib.win-amd64-cpython-311\d3rlpy\preprocessing
copying d3rlpy\preprocessing\scalers.py -> build\lib.win-amd64-cpython-311\d3rlpy\preprocessing
copying d3rlpy\preprocessing\stack.py -> build\lib.win-amd64-cpython-311\d3rlpy\preprocessing
copying d3rlpy\preprocessing_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\preprocessing
creating build\lib.win-amd64-cpython-311\d3rlpy\wrappers
copying d3rlpy\wrappers\sb3.py -> build\lib.win-amd64-cpython-311\d3rlpy\wrappers
copying d3rlpy\wrappers_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\wrappers
creating build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\awac_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\base.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\bcq_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\bc_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\bear_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\combo_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\cql_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\crr_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\ddpg_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\dqn_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\iql_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\plas_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\sac_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\td3_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\td3_plus_bc_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch\utility.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
copying d3rlpy\algos\torch_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\algos\torch
creating build\lib.win-amd64-cpython-311\d3rlpy\dynamics\torch
copying d3rlpy\dynamics\torch\base.py -> build\lib.win-amd64-cpython-311\d3rlpy\dynamics\torch
copying d3rlpy\dynamics\torch\probabilistic_ensemble_dynamics_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\dynamics\torch
copying d3rlpy\dynamics\torch_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\dynamics\torch
creating build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch\distributions.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch\dynamics.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch\encoders.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch\imitators.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch\parameters.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch\policies.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch\v_functions.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
copying d3rlpy\models\torch_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch
creating build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions\base.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions\ensemble_q_function.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions\fqf_q_function.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions\iqn_q_function.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions\mean_q_function.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions\qr_q_function.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions\utility.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
copying d3rlpy\models\torch\q_functions_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\models\torch\q_functions
creating build\lib.win-amd64-cpython-311\d3rlpy\ope\torch
copying d3rlpy\ope\torch\fqe_impl.py -> build\lib.win-amd64-cpython-311\d3rlpy\ope\torch
copying d3rlpy\ope\torch_init_.py -> build\lib.win-amd64-cpython-311\d3rlpy\ope\torch
copying d3rlpy\dataset.pyi -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\py.typed -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\dataset.pyx -> build\lib.win-amd64-cpython-311\d3rlpy
copying d3rlpy\dataset.pxd -> build\lib.win-amd64-cpython-311\d3rlpy
running build_ext
building 'd3rlpy.dataset' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for d3rlpy
Running setup.py clean for d3rlpy
Failed to build d3rlpy
ERROR: Could not build wheels for d3rlpy, which is required to install pyproject.toml-based projects

from d3rlpy.

I think I found something in the 2.3.0 documentation what resembles to eval_episodes in the 1.1.1 version but I am not sure if it has the same function; it is possible in 2.3.0 to incorporate the validation/test episodes into for instance the TDErrorEvaluator metric. has this the same purpose as these lines from 1.1.1 documentation?
dqn.fit(train_episodes,
eval_episodes=test_episodes,
scorers={
'td_error': td_error_scorer,
'value_scale': average_value_estimation_scorer,
'environment': evaluate_on_environment(env)
}

from d3rlpy.

@stijnmeels Thanks for the issue.

Regarding the first question about number of epochs setting, in v2, there is no longer n_epochs. Instead, please use n_steps and n_steps_per_epoch to specify how many steps you train the agent.

For the second question about test episode, it's been changed since v1. Please check documentation.
https://d3rlpy.readthedocs.io/en/v2.3.0/references/metrics.html

from d3rlpy.

@takuseno thanks a lot, a last follow up question on this topic before closing the topic: How was the number of steps per epoch defined when in version 1 only n_eopchs was entered?

For my feeling it should be something in the trend of:

batch_size = 100
num_epochs = 100
total_samples = len(train_episodes) * len(train_episodes[0])
steps_per_epoch = total_samples // batch_size
total_steps = steps_per_epoch * num_epochs

from d3rlpy.

Yes, it was exactly something like that.

from d3rlpy.