Оптимизация долгосрочного портфеля акций
License: The Unlicense
Master branch содержит находящуюся в разработке новую реализацию (version 3) с web интерфейсом. Предыдущая версия содержится в ветке version_2 и будет поддерживаться и обновляться до момента завершения работы над новой версией.
Для предварительного ознакомления на MacOS необходимо:
.env.template и переименовать его в .env% task install% task run
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Загрузка данных по торговым дням
Сохранение ID(package='data', group='trading_dates', name='trading_dates')
TradingDayEnded(date=datetime.date(2022, 2, 4))
Загрузка инфляции
Загрузка данных
Загрузка данных
(2022-02-04, 2022-02-04)
Загрузка данных
Сохранение ID(package='data', group='usd', name='usd')
Сохранение ID(package='data', group='risk_free', name='risk_free')
Сохранение ID(package='data', group='CPI', name='CPI')
Traceback (most recent call last):
File "%PATH_TO%\poptimizer\poptimizer\data\adapters\html\parser.py", line 33, in _get_table_from_html
table = soup.find_all("table")[table_num]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\PythonProjects\anaconda3\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\PythonProjects\anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "%PATH_TO%\poptimizer\poptimizer_main.py", line 7, in
from poptimizer.data.views import div_status
File "%PATH_TO%\poptimizer\poptimizer\data\views\div_status.py", line 10, in
from poptimizer.data.app import bootstrap, viewers
File "%PATH_TO%\poptimizer\poptimizer\data\app\bootstrap.py", line 47, in
BUS, VIEWER = start_app()
File "%PATH_TO%\poptimizer\poptimizer\data\app\bootstrap.py", line 42, in start_app
bus.handle_event(event)
File "%PATH_TO%\poptimizer\poptimizer\shared\app.py", line 74, in handle_event
loop.run_until_complete(self._handle_event(event))
File "D:\PythonProjects\anaconda3\lib\asyncio\base_events.py", line 642, in run_until_complete
return future.result()
File "%PATH_TO%\poptimizer\poptimizer\shared\app.py", line 89, in _handle_event
pending |= self._create_tasks(task.result())
File "%PATH_TO%\poptimizer\poptimizer\shared\app.py", line 100, in _handle_one_command
return await self._event_handler.handle_event(event, repo)
File "%PATH_TO%\poptimizer\poptimizer\data\domain\handlers.py", line 69, in trading_day_ended
*itertools.chain.from_iterable(await asyncio.gather(*aws)),
File "%PATH_TO%\poptimizer\poptimizer\data\domain\handlers.py", line 29, in _load_by_id_and_handle_event
return await table.handle_event(event)
File "%PATH_TO%\poptimizer\poptimizer\data\domain\tables\base.py", line 71, in handle_event
df_new = await self._prepare_df(event)
File "%PATH_TO%\poptimizer\poptimizer\data\domain\tables\dividends.py", line 90, in _prepare_df
dfs = [await gw() for gw in self._gateways]
File "%PATH_TO%\poptimizer\poptimizer\data\domain\tables\dividends.py", line 90, in
dfs = [await gw() for gw in self._gateways]
File "%PATH_TO%\poptimizer\poptimizer\data\adapters\gateways\moex_status.py", line 57, in call
df = await parser.get_df_from_url(URL, TABLE_INDEX, cols_desc)
File "%PATH_TO%\poptimizer\poptimizer\data\adapters\html\parser.py", line 87, in get_df_from_url
return get_df_from_html(html, table_num, cols_desc)
File "%PATH_TO%\poptimizer\poptimizer\data\adapters\html\parser.py", line 78, in get_df_from_html
table = _get_table_from_html(html, table_num)
File "%PATH_TO%\poptimizer\poptimizer\data\adapters\html\parser.py", line 35, in _get_table_from_html
raise description.ParserError(f"На странице нет таблицы {table_num}")
poptimizer.data.adapters.html.description.ParserError: На странице нет таблицы 2
Видимо что то поменяли...
Добрый день!
Подскажите пожалуйста, каким образом запустить Телеграмм бота? Зашел в каталог data прошелся по командам makefile. Обновил Go до 1.18, заполнил .env TOKEN=<токен от бота>, CHAT_ID=<ID моего аккаунта>. Собрал бинарник, лежит в ~/poptimizer/data/.bin/podata
Бинарник запускать отдельно или он сам запустится вместе с evolve?
При пустой базе падает на этом месте, возможно, её тоже нужно поместить в try.
лог:
https://gist.github.com/RomaKoks/d7cf5a03bf58a1315af1f3fed4662c4b#file-max_history-error-log
Добрый вечер
Подскажите пожалуйста с чем может быть связана ошибка?
Появилась пару дней назад, с тех пор не может выкачать котировки за предыдущий день (падает при выгрузке каждый день)
Traceback (most recent call last):
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/connector.py", line 985, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
transport, protocol = await self._create_connection_transport(
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/asyncio/base_events.py", line 1103, in _create_connection_transport
transport = self._make_ssl_transport(
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/asyncio/selector_events.py", line 78, in _make_ssl_transport
_SelectorSocketTransport(self, rawsock, ssl_protocol,
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/asyncio/selector_events.py", line 774, in init
base_events._set_nodelay(self._sock)
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/asyncio/base_events.py", line 199, in _set_nodelay
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
OSError: [Errno 22] Invalid argument
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/startEvolution.py", line 9, in
ev.evolve()
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/evolve/evolve.py", line 56, in evolve
step, current = self._step_setup(step, current)
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/evolve/evolve.py", line 90, in _step_setup
dates = listing.all_history_date(self._tickers, start=self._end)
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/data/views/listing.py", line 26, in all_history_date
bus.handle_event(event)
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/shared/app.py", line 74, in handle_event
loop.run_until_complete(self._handle_event(event))
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/shared/app.py", line 89, in _handle_event
pending |= self._create_tasks(task.result())
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/shared/app.py", line 100, in _handle_one_command
return await self._event_handler.handle_event(event, repo)
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/data/domain/handlers.py", line 92, in ticker_traded
return list(itertools.chain.from_iterable(await asyncio.gather(*aws)))
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/data/domain/handlers.py", line 29, in _load_by_id_and_handle_event
return await table.handle_event(event)
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/data/domain/tables/base.py", line 71, in handle_event
df_new = await self._prepare_df(event)
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/data/domain/tables/quotes.py", line 39, in _prepare_df
df_new = await self._load_df(event)
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/data/domain/tables/quotes.py", line 70, in _load_df
await asyncio.gather(*aws),
File "/Users/Documents/PythonProjects/poptimizer/poptimizer/poptimizer/data/adapters/gateways/moex.py", line 141, in call
json = await aiomoex.get_market_candles(
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiomoex/candles.py", line 116, in get_market_candles
return await request_helpers.get_long_data(session, url, table, query)
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiomoex/request_helpers.py", line 137, in get_long_data
table_dict = await iss.get_all()
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiomoex/client.py", line 112, in get_all
async for block in self:
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiomoex/client.py", line 131, in _iterator_maker
respond = await self.get(start)
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiomoex/client.py", line 94, in get
async with self._session.get(url, params=query) as respond:
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in aenter
self._resp = await self._coro
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/connector.py", line 906, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/connector.py", line 1205, in _create_direct_connection
raise last_exc
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/connector.py", line 1174, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/Users/opt/miniconda3/envs/torch39/lib/python3.9/site-packages/aiohttp/connector.py", line 991, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host iss.moex.com:443 ssl:default [Invalid argument]
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 63, in evolve
next_, new = self._step(current)
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 127, in _step
if self._eval_organism(hunter) is None:
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 185, in _eval_organism
if self._is_dead(organism):
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 198, in _is_dead
maximum, median, upper = _select_worst_bound(
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 246, in _select_worst_bound
median, upper, maximum = min(
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 267, in _aligned_tests
candidate = candidate["llh"][candidate_start:]
TypeError: list indices must be integers or slices, not str
DateCheckRequired(timestamp=datetime.datetime(2021, 12, 7, 20, 31, 47, 843809))
2021-12-06: Шаг эволюции — 214
LLH - (-2.2139, 2.2601, 2.4676)
RET - (-0.7104, 0.4548, 4.0908)
Организмов - 211 / Максимум оценок - 21
Доля принятых - 36.69%
Родитель:
LLH — 1.6468: 1.7021, 1.6963, 1.6906, 1.6870, 1.6750, 1.6597, 1.6637, 1.6427, 1.6464, 1.6546, 1.6557, 1.6473, 1.6450, 1.6418, 1.6266, 1.6362, 1.6305, 1.6292, 1.6254, 1.6308
RET — -0.1679: -0.5417, -0.5741, -0.5027, -0.4789, -0.4114, -0.2440, -0.2873, -0.1028, -0.1737, -0.2066, -0.3161, -0.1621, -0.0832, 0.0163, 0.1652, 0.0561, 0.0891, 0.1086, 0.1810, 0.0807
Timer — 0:30:35 / Scores - 20
Data: {'batch_size': 7095.271077633346, 'history_days': 104.2664946854577, 'ticker_on': 4976.2716520817985, 'day_of_year_on': -3.709624878883848, 'day_of_period_on': 137.4568151927667, 'prices_on': -16.121645323896313, 'dividends_on': -4605.032017093387, 'turnover_on': 78727.19365161215, 'average_turnover_on': 2126.666392327704, 'rvi_on': -57826.07390087617, 'mcftrr_on': -1739.7378353702825, 'imoex_on': 460.7037304850061, 'ticker_type_on': 383706.00156781333, 'usd_on': 202413.2533475839, 'open_on': 47.910467251496684, 'high_on': -569.0460289604246, 'low_on': 4.311145782034553, 'meogtrr_on': -67.47850617523062}
Model: {'start_bn': 8.526469978101927, 'kernels': 27.67607274460994, 'sub_blocks': 11.077653647532772, 'gate_channels': 7.36760712976323, 'residual_channels': 5.830611696867351, 'skip_channels': 175.2040654474389, 'end_channels': 129.06367751044317, 'mixture_size': 406.9241719003154}
Optimizer: {'betas': 0.9512142169103824, 'eps': 0.000124510967630482, 'weight_decay': 13.612907456891998}
Scheduler: {'max_lr': 0.0499352822631719, 'epochs': 3.6580907291933498, 'pct_start': 0.820676400666021, 'anneal_strategy': -3730.263470910985, 'base_momentum': 0.9972310235283417, 'max_momentum': 0.9201040129780192, 'div_factor': 1430.0763899622098, 'final_div_factor': 1573377.7010648176}
Претендент - новый организм:
LLH — -
RET — -
Timer — 0:00:00 / Scores - 0
Data: {'batch_size': 6895.328881408579, 'history_days': 112.0798316286424, 'ticker_on': 4478.858597534277, 'day_of_year_on': -3.8270234650609622, 'day_of_period_on': 159.05112347948247, 'prices_on': -217.4771456933069, 'dividends_on': 7007.702613656215, 'turnover_on': 71872.51718327151, 'average_turnover_on': 1227.43596951955, 'rvi_on': 380738.18073469825, 'mcftrr_on': -1722.3528881635725, 'imoex_on': 392.29323879244976, 'ticker_type_on': 194817.4720718744, 'usd_on': 146121.75177522193, 'open_on': 43.19711628339914, 'high_on': -550.3772067225993, 'low_on': 5.6694529090453925, 'meogtrr_on': -115.39740648167755}
Model: {'start_bn': 2.1897676303046563, 'kernels': 26.807849195937457, 'sub_blocks': 11.080826090600356, 'gate_channels': 9.482139014483748, 'residual_channels': 5.022165621453517, 'skip_channels': 200.0585995947543, 'end_channels': 381.17119402983553, 'mixture_size': 376.6544359345419}
Optimizer: {'betas': 0.9373288828589553, 'eps': 0.00014087801919633505, 'weight_decay': 129.54538277095145}
Scheduler: {'max_lr': 0.049719869950901545, 'epochs': 9650.153309416133, 'pct_start': 0.832731169529873, 'anneal_strategy': -2578.2779830306904, 'base_momentum': 0.9677651595266519, 'max_momentum': 0.9208398561827363, 'div_factor': 39353.97274675947, 'final_div_factor': 1499376.0759575109}
DateCheckRequired(timestamp=datetime.datetime(2021, 12, 7, 20, 31, 47, 910822))
Epochs - 9650.15 / Train size - 215015
Количество слоев / параметров - 435 / 704524
--
~~> Train: 0%| | 1/308805 [00:18<1587:17:08, 18.50s/it, 0.86492]
т.е. ориентировочное время проверки данного организма 1587/24 = 66 дней, за 6.5 часов даже 1 % тренировки не прошло.
Можно ли как-то ограничить максимальное кол-во эпох ? Наверняка такое кол-во эпох слишком избыточно
Запуск эволюции не совсем выход, так как основная цель - переобучить имеющиеся модели (а не искать параметры для новых) и обновить их показатели "качества" на обновленном наборе. Грубо говоря, как сделать переинициализацию, но не случайным образом, а с текущими параметрами моделей.
Вымерла популяция моделей до 10 шт. Поднял из резервной копии базу с обученными моделями за 18 ноября, модели 256 обучились на данных с 19.11 до 08.12. Сегодня ночью на тестах данных от 09.12 опять популяция сократилась до 13. Запустил сегодня с нуля эволюцию на 28м шаге из 100 начальных осталось 83.
Куда копать?
`INFO:Mapper:Сохранение ID(package='data', group='quotes', name='KOGK')
INFO:Mapper:Сохранение ID(package='data', group='quotes', name='WMT-RM')
INFO:Mapper:Сохранение ID(package='data', group='quotes', name='VTRS-RM')
INFO:Mapper:Сохранение ID(package='data', group='quotes', name='VZRZ')
[E:pyppeteer.connection] connection unexpectedly closed
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-4231' coro=<Connection._async_send() done, defined at /home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/pyppeteer/connection.py:69> exception=InvalidStateError('invalid state')>
Traceback (most recent call last):
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
message = await self.read_message()
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
frame = await self.read_data_frame(max_size=self.max_size)
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
frame = await self.read_frame(max_size)
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
frame = await Frame.read(
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
data = await reader(2)
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/asyncio/streams.py", line 721, in readexactly
raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/pyppeteer/connection.py", line 73, in _async_send
await self.connection.send(msg)
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/websockets/protocol.py", line 555, in send
await self.ensure_open()
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/pyppeteer/connection.py", line 79, in _async_send
await self.dispose()
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/pyppeteer/connection.py", line 170, in dispose
await self._on_close()
File "/home/UNAME/anaconda3/envs/finan/lib/python3.8/site-packages/pyppeteer/connection.py", line 151, in _on_close
cb.set_exception(_rewriteError(
asyncio.exceptions.InvalidStateError: invalid state
`
Сломается если в портфеле нет открытых позиций.
poptimizer/poptimizer/data/domain/tables/securities.py
Lines 25 to 26 in defb9db
Видимо, с появлением VEON-RX тут стоит модифицировать условие на проверку типа акции, иначе загрузка данных упадет сегодня ночью.
Доброго дня!
Система OSX, логин пользователя в системе с точкой <имя>.<фамилия>
Приложение запущено из домашнего каталога /Users/<имя>.<фамилия>/poptimazer/
При ротации логов, в функции "namer" /poptimizer/poptimizer/shared/log.py из-за "split(".")" получаю следующее расположение для файла с бэкапом '/Users/<имя>_2021-12-07_18-00-34_610884.log' и соответственно ошибку "Permission denied"
Замена
name = default_name.split(".")[0]
на
name = default_name.split(".log")[0]
исправило ситуацию
Изучил репозиторий, но не смог найти тестовый (не реальный, а просто заглушку) yaml конфиг для портфеля.
DateCheckRequired(timestamp=datetime.datetime(2022, 4, 12, 14, 50, 3, 162156))
Загрузка данных по торговым дням
Сохранение ID(package='data', group='trading_dates', name='trading_dates')
TradingDayEnded(date=datetime.date(2022, 4, 11))
Загрузка инфляции
Загрузка данных
(2022-04-11, 2022-04-11)
Загрузка данных
Сохранение ID(package='data', group='div_new', name='div_new')
Сохранение ID(package='data', group='usd', name='usd')Сохранение ID(package='data', group='usd', name='usd')
Traceback (most recent call last):
File "/PATH_TO/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/PATH_TO/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/PATH_TO/poptimizer/poptimizer/main.py", line 7, in
from poptimizer.data.views import div_status
File "/PATH_TO/poptimizer/poptimizer/data/views/div_status.py", line 10, in
from poptimizer.data.app import bootstrap, viewers
File "/PATH_TO/poptimizer/poptimizer/data/app/bootstrap.py", line 47, in
BUS, VIEWER = start_app()
File "/PATH_TO/poptimizer/poptimizer/data/app/bootstrap.py", line 42, in start_app
bus.handle_event(event)
File "/PATH_TO/poptimizer/poptimizer/shared/app.py", line 74, in handle_event
loop.run_until_complete(self._handle_event(event))
File "/PATH_TO/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/PATH_TO/poptimizer/poptimizer/shared/app.py", line 89, in _handle_event
pending |= self._create_tasks(task.result())
File "/PATH_TO/poptimizer/poptimizer/shared/app.py", line 100, in _handle_one_command
return await self._event_handler.handle_event(event, repo)
File "/PATH_TO/poptimizer/poptimizer/data/domain/handlers.py", line 73, in trading_day_ended
*itertools.chain.from_iterable(await asyncio.gather(*aws)),
File "/PATH_TO/poptimizer/poptimizer/data/domain/handlers.py", line 29, in _load_by_id_and_handle_event
return await table.handle_event(event)
File "/PATH_TO/poptimizer/poptimizer/data/domain/tables/base.py", line 71, in handle_event
df_new = await self._prepare_df(event)
File "/PATH_TO/poptimizer/poptimizer/data/domain/tables/cpi.py", line 32, in _prepare_df
return await self._gateway()
File "/PATH_TO/poptimizer/poptimizer/data/adapters/gateways/cpi.py", line 99, in call
df = await _load_and_parse_xlsx(self._session)
File "/PATH_TO/poptimizer/poptimizer/data/adapters/gateways/cpi.py", line 38, in _load_and_parse_xlsx
page = await _loap_cpi_page(session)
File "/PATH_TO/poptimizer/poptimizer/data/adapters/gateways/cpi.py", line 55, in _loap_cpi_page
return await resp.text("cp1251")
File "/PATH_TO/python3.9/site-packages/aiohttp/client_reqrep.py", line 1085, in text
return self._body.decode( # type: ignore[no-any-return,union-attr]
File "/PATH_TO/python3.9/encodings/cp1251.py", line 15, in decode
return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in position 4050: character maps to
Например, 3-4 GB.
***2021-11-24: Шаг эволюции — 243***
LLH - (0.7407, 2.5590, 2.7341)
RET - (0.2098, 0.2959, 1.6163)
Организмов - 213 / Максимум оценок - 17
Доля принятых - 5.23%
Родитель:
LLH — 1.2893: 1.2804, 1.2822, 1.2810, 1.2915, 1.2956, 1.2911, 1.2909, 1.2924, 1.2983
RET — 0.2794: 0.1963, 0.1026, 0.2753, 0.1421, 0.3180, 0.2902, 0.5003, 0.3608, 0.3287
Timer — 0:03:34
Data: {'batch_size': 139811.11778702133, 'history_days': 112.57072564138315, 'ticker_on': -5.161145416073824, 'day_of_year_on': -5.766080083064525, 'day_of_period_on': 46.563581618325806, 'prices_on': 1.8714902291930082, 'dividends_on': 900.9217629177548, 'turnover_on': 8736.373690688984, 'average_turnover_on': 439.3670786982759, 'rvi_on': 178.46045539547123, 'mcftrr_on': 968.230619669111, 'imoex_on': -46.300169774610076, 'ticker_type_on': -67.33822514824053, 'usd_on': 8.864144689780463, 'open_on': -26.257885080709183, 'high_on': -3.3051814304821723, 'low_on': 6.348455320734517, 'meogtrr_on': -0.7318042946129198}
Model: {'start_bn': -26.462877096492438, 'kernels': 11.916432720113374, 'sub_blocks': 4.716133114531003, 'gate_channels': 4.21401462977202, 'residual_channels': 7.489649605598338, 'skip_channels': 7.173944290463009, 'end_channels': 20.032993444408106, 'mixture_size': 15.623952530217979}
Optimizer: {'betas': 0.9957827712084104, 'eps': 3.6858605799557665e-06, 'weight_decay': 0.1425642364851929}
Scheduler: {'max_lr': 0.0160832031038165, 'epochs': 1.7182526110863983, 'pct_start': 0.3032371187129486, 'anneal_strategy': 39.65421914074793, 'base_momentum': 0.3685056171200781, 'max_momentum': 0.9472647638288794, 'div_factor': 127.86830964018056, 'final_div_factor': 4734830.833439709}
Претендент - новый организм:
LLH — -
RET — -
Timer — 0:00:00
Data: {'batch_size': 140064.03402124936, 'history_days': 167.33439844456754, 'ticker_on': -6.144128940470949, 'day_of_year_on': -5.611961091686764, 'day_of_period_on': 44.249412809116926, 'prices_on': 1.4586902179030707, 'dividends_on': 900.8428701059249, 'turnover_on': 9036.3562121416, 'average_turnover_on': 64696.6463580999, 'rvi_on': 178.32575135899867, 'mcftrr_on': 1123.6452450364195, 'imoex_on': -46.36415827468592, 'ticker_type_on': -67.48653659762039, 'usd_on': 9.634472546572647, 'open_on': -24.91638751552406, 'high_on': -4.429428934478038, 'low_on': 7.154702049966035, 'meogtrr_on': -0.8945665994377038}
Model: {'start_bn': -26.35370627032382, 'kernels': 12.079054746747497, 'sub_blocks': 91.42960596734237, 'gate_channels': 4.38845540845128, 'residual_channels': 7.4173728933107865, 'skip_channels': 7.028698071864793, 'end_channels': 19.747718108684705, 'mixture_size': 15.034516535964439}
Optimizer: {'betas': 0.9957821493164769, 'eps': 3.6870724712961566e-06, 'weight_decay': 0.12075289055584597}
Scheduler: {'max_lr': 0.017162015274579553, 'epochs': 1.724127112019961, 'pct_start': 0.303242939386313, 'anneal_strategy': -140.7051815141791, 'base_momentum': 0.3778324858225567, 'max_momentum': 0.9472441358367255, 'div_factor': 124.43565265605905, 'final_div_factor': 2771693.987494981}
INFO:EventBus:DateCheckRequired(timestamp=datetime.datetime(2021, 11, 25, 11, 10, 23, 449185))
Epochs - 1.72 / Train size - 202030
Количество слоев / параметров - 3693/524148
~~> Train: 0%| | 0/4 [00:00<?, ?it/s]
Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
Модели обучались достаточно долго, несколько дней и несколько перезапусков обучения. Но градиенты (в таблице с оптимизацией) выходят очень странные, не подскажете в чем может быть дело? Я даже пробовал удалять вручную модели с отрицательным LLH...
КЛЮЧЕВЫЕ МЕТРИКИ ПОРТФЕЛЯ
Дней в истории - (154 <-> 154 <-> 154)
Корреляция - (11.46% <-> 11.46% <-> 11.46%)
Сжатие - (48.72% <-> 48.72% <-> 48.72%)
MEAN STD BETA SHARPE GRAD
AFKS 3.204968 4.629894 1.231849 1.467736 0.118649
AFLT 3.166615 4.560440 1.102449 1.703298 0.269892
AGRO 3.127170 4.489847 0.957616 1.996904 0.149288
AKRN 3.100019 4.441739 0.888641 2.154438 0.839037
ALRS 3.113174 4.464998 1.021626 1.904257 -0.003673
ASSB 3.251409 4.715091 0.691915 2.168823 0.855619
BANEP 5.312154 8.600042 2.153156 0.225667 0.073186
BISVP 5.307807 8.524845 2.323929 0.000012 -0.006607
CHMF 3.148694 4.528265 1.066614 1.818065 -0.018042
DIOD 3.107570 4.455079 0.924197 2.114640 0.169674
DSKY 3.171264 4.568817 1.121548 1.658117 -0.155296
ENRU 3.154212 4.538153 1.148158 1.594587 -0.204739
ETLN 3.183147 4.590282 1.057278 1.794551 0.408340
FEES 3.187027 4.597302 1.190608 1.520222 0.079570
FIVE 3.171316 4.568912 1.010849 1.890005 0.478861
FLOT 3.174314 4.574317 0.997278 1.922789 0.496678
GAZP 3.070079 4.389137 1.149134 1.668626 -0.187119
GCHE 3.103978 4.448727 0.879583 2.190835 0.808174
GEMA 3.091245 4.426274 0.844295 2.229632 0.374504
GLTR 3.136846 4.507086 0.905029 2.128026 0.770320
GMKN 3.168407 4.563666 1.190821 1.557652 -0.242545
HIMCP 5.264921 9.233613 1.117715 0.700741 0.300420
HYDR 3.165939 4.559224 1.120969 1.663747 -0.154222
INGR 3.194879 4.611546 1.148208 1.700457 0.384517
IRKT 3.189549 4.601875 0.869642 2.188435 1.197704
KRKNP 5.292915 8.771701 1.400830 0.959465 0.076452
KRSBP 5.298620 8.450047 1.545946 1.807477 1.275892
LKOH 3.132780 4.499835 1.058643 1.880179 -0.026335
LSNGP 5.283682 8.924782 1.376169 0.157453 -0.036225
LSRG 3.194960 4.611691 1.079832 1.794044 0.492629
MAGN 3.150917 4.532246 0.972474 1.974543 0.121681
MGNT 3.127490 4.490416 0.978385 1.975141 0.460485
MGTSP 5.331383 8.699933 1.778324 0.962198 0.507715
MOEX 3.166586 4.560389 0.917351 2.100634 0.757402
MRKC 3.165662 4.558728 1.221350 1.416269 -0.340729
MRKP 3.160485 4.549416 1.259407 1.367453 -0.411439
MRKU 3.146641 4.524588 1.005334 1.947126 0.648389
MRKV 3.202542 4.625479 1.173594 1.543648 0.179176
MRKY 3.167125 4.561359 0.929031 2.077309 0.837539
MRKZ 3.162848 4.553664 1.189472 1.566602 0.195656
MSNG 3.167546 4.562116 1.173398 1.508187 -0.251635
MSRS 3.158387 4.545645 1.121375 1.656769 -0.154976
MTSS 3.168592 4.564001 1.137999 1.617904 -0.185864
MVID 3.204474 4.628995 0.928676 2.101563 0.891404
NLMK 3.173746 4.573294 1.196550 1.500221 -0.294650
NMTP 3.173816 4.573421 1.106982 1.735987 0.370997
OGKB 3.186273 4.595938 1.274338 1.394256 -0.021429
PHOR 3.101170 4.443768 0.903964 2.159595 0.274886
PIKK 3.112196 4.463267 0.829574 2.241210 0.331837
PLZL 3.171311 4.568903 1.123241 1.703844 0.357671
PMSBP 5.283428 9.047125 1.154882 1.034200 0.392335
POLY 3.183546 4.590999 1.071657 1.724546 -0.062601
ROSN 3.153130 4.536211 1.087752 1.839024 0.290681
RSTIP 5.296736 8.467317 2.531629 0.018040 -0.334410
RTKMP 5.299729 8.455023 2.239283 0.594266 -0.104502
RZSB 3.124781 4.485599 0.549115 2.629627 1.187588
SBERP 5.269168 9.297883 1.397491 0.959770 0.030040
SELG 3.121978 4.480617 0.629526 2.525985 1.607021
SIBN 3.110951 4.461061 1.008135 1.930227 0.329039
SMLT 2.813536 3.956683 0.617218 2.450539 0.780686
SNGSP 5.301840 8.577883 2.042314 0.189751 0.157277
STSBP 5.271825 9.825322 1.098090 1.047107 0.490332
TATN 3.192469 4.607172 1.189091 1.574371 0.240026
TATNP 5.308748 8.574111 2.384996 0.414851 -0.248883
TGKA 3.176812 4.578826 1.194220 1.500736 -0.290323
TORSP 5.287541 8.743272 1.943046 0.261794 0.145359
TRMK 3.178876 4.582558 0.854261 2.268269 1.177782
TTLK 3.163277 4.554433 0.992519 1.951164 0.568677
UPRO 3.177164 4.579463 1.027868 1.854759 0.444589
VSMO 3.107503 4.454958 0.775840 2.345838 0.806486
VTBR 3.125041 4.486059 0.951888 2.058716 0.184680
CASH 0.000000 0.000000 0.000000 1.593405 -0.067010
PORTFOLIO 2.994268 1.795054 1.000000 1.630736 0.000000
Безрисковая ставка: 0.0670
Консервативная доходность: 1.4785
Консервативный риск: 61886987453406.8281
Консервативный Шарп: 0.0000
ОПТИМИЗАЦИЯ ПОРТФЕЛЯ
forecasts = 17
p-value = 5.00%
LOWER UPPER COSTS PRIORITY
SELG 495.834097 1.190011e+06 1.547099 494.286998
GCHE 484.605576 1.729713e+05 0.870767 483.734809
VSMO 480.559256 6.335420e+04 1.037603 479.521652
GLTR 439.192997 3.585873e+05 0.397014 438.795983
SIBN 437.125744 5.444899e+04 0.183037 436.942708
MOEX 408.018295 3.990379e+05 0.117106 407.901189
AKRN 388.964979 8.883661e+04 0.981696 387.983283
ROSN 357.948987 2.323769e+05 0.069020 357.879967
TTLK 356.694888 1.664968e+05 2.176313 354.518575
FLOT 307.756917 1.176961e+05 0.445324 307.311593
MRKU 307.815032 1.897442e+05 4.677403 303.137629
FIVE 301.634823 1.722830e+05 0.134978 301.499845
TRMK 297.189818 7.825663e+05 0.617536 296.572282
MRKY 289.363975 4.699842e+05 4.980147 284.383828
UPRO 276.082351 1.186205e+05 0.355809 275.726542
NMTP 270.786931 1.042383e+05 0.694294 270.092637
INGR 236.940652 3.518466e+05 1.160851 235.779801
IRKT 213.355757 7.707623e+05 3.011886 210.343871
MVID 207.513256 4.597404e+05 0.332590 207.180666
LSRG 204.820328 2.287192e+05 0.450817 204.369511
MRKZ 200.024523 2.937171e+04 3.683783 196.340740
TATN 191.415092 4.137502e+05 0.084563 191.330530
AFKS 128.909840 1.883633e+04 0.125729 128.784112
PLZL 17.443099 3.028069e+04 0.096593 17.346506
ETLN 8.827274 2.400846e+04 0.759360 8.067914
SBERP 1.918771 6.353486e+07 0.228545 1.690225
SNGSP 0.927868 6.760736e+07 0.228444 0.699424
TATNP 0.718077 5.339666e+07 0.334104 0.383973
ДАННЫЕ ПО ДИВИДЕНДАМ ТРЕБУЮТ ОБНОВЛЕНИЯ
MAGN, MGNT, MVID, CHMF, PHOR, DSKY, NLMK, ETLN, LKOH
INFO:EventBus:AppStarted(timestamp=datetime.datetime(2021, 5, 12, 5, 35, 25, 243556))
Forecasts: 90it [03:55, 2.62s/it]
Traceback (most recent call last):
File "path_to\poptimizer\poptimizer\store\database.py", line 48, in __setitem__
self._collection.replace_one({"_id": key}, value, upsert=True)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\collection.py", line 939, in replace_one
common.validate_ok_for_replace(replacement)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\common.py", line 539, in validate_ok_for_replace
validate_is_mapping("replacement", replacement)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\common.py", line 494, in validate_is_mapping
raise TypeError("%s must be an instance of dict, bson.son.SON, or "
TypeError: replacement must be an instance of dict, bson.son.SON, or any other type that inherits from collections.Mapping
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "path_to\poptimizer\loop.py", line 13, in opt
optimize(date)
File "path_to\poptimizer\poptimizer\__main__.py", line 30, in optimize
opt = Optimizer(port)
File "path_to\poptimizer\poptimizer\portfolio\optimizer.py", line 53, in __init__
self._metrics = metrics.MetricsResample(portfolio)
File "path_to\poptimizer\poptimizer\portfolio\metrics.py", line 138, in __init__
for forecast in evolve.get_forecasts(tickers, date):
File "path_to\poptimizer\poptimizer\evolve\forecaster.py", line 70, in get_forecasts
mongodb[FORECAST] = forecasts
File "path_to\poptimizer\poptimizer\store\database.py", line 51, in __setitem__
self._collection.replace_one({"_id": key}, value, upsert=True)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\collection.py", line 943, in replace_one
self._update_retryable(
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\collection.py", line 868, in _update_retryable
return self.__database.client._retryable_write(
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\mongo_client.py", line 1498, in _retryable_write
return self._retry_with_session(retryable, func, s, None)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\mongo_client.py", line 1384, in _retry_with_session
return self._retry_internal(retryable, func, session, bulk)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\mongo_client.py", line 1416, in _retry_internal
return func(session, sock_info, retryable)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\collection.py", line 860, in _update
return self._update(
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\collection.py", line 829, in _update
result = sock_info.command(
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\pool.py", line 699, in command
self._raise_connection_failure(error)
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\pool.py", line 683, in command
return command(self, dbname, spec, slave_ok,
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\network.py", line 135, in command
message._raise_document_too_large(
File "path_to\anaconda3\envs\finan\lib\site-packages\pymongo\message.py", line 1092, in _raise_document_too_large
raise DocumentTooLarge("%r command document too large" % (operation,))
pymongo.errors.DocumentTooLarge: 'update' command document too large
вот что он
poptimizer/poptimizer/store/database.py
Line 50 in 09150f0
ДАННЫЕ ПО ДИВИДЕНДАМ ТРЕБУЮТ ОБНОВЛЕНИЯ
JNJ-RM, VTRS-RM
Обучение шло по тикерам из base.yaml
Сутки крутилось на видеокарточке... После попытался с оптимизировать портфель на вчерашнюю даты выдал что нет дивидентов по этим двум тикерам.
Создал отдельный портфель в котором исключил тикеры с -RM на конце. и попробовал оптимизировать только этот портфель в итоге он выдал
poptimizer.evolve.population.ForecastError: Отсутствуют прогнозы - необходимо обучить модели. SymDiff: {'ABBV-RM', 'CMCSA-RM', 'KHC-RM', 'MRN
A-RM', 'IBM-RM', 'AMZN-RM', 'NEE-RM', 'BIIB-RM', 'MSFT-RM', 'AAPL-RM', 'NFLX-RM', 'BIDU-RM', 'PYPL-RM', 'WMT-RM', 'CRM-RM', 'HPQ-RM', 'PFE-RM
', 'NEM-RM', 'TGT-RM', 'JNJ-RM', 'AVGO-RM', 'DIS-RM', 'GM-RM', 'BABA-RM', 'SBUX-RM', 'CVX-RM', 'EA-RM', 'AMD-RM', 'PG-RM', 'ATVI-RM', 'MCD-RM
', 'TWTR-RM', 'BMY-RM', 'F-RM', 'TSLA-RM', 'ADBE-RM', 'MA-RM', 'VTRS-RM', 'FDX-RM', 'ABT-RM', 'NKE-RM', 'QCOM-RM', 'GPS-RM', 'INTC-RM', 'FB-R
M', 'NVDA-RM', 'UBER-RM', 'GOOG-RM', 'EBAY-RM', 'T-RM', 'V-RM', 'CAT-RM', 'KO-RM', 'AAL-RM', 'GE-RM', 'CSCO-RM', 'XOM-RM'}
как раз удалённых тикеров по которым как раз модели и обучались...
Здравствуйте!
Тоже решил познакомится с вашим решением.
Собрал ваше приложение в контейнере и запустил evolve
Загрузку дивидентов прошла успешно. А дальше вылезло вот это....
Корректность работы pyppeteer проверил.
python3.9 -m poptimizer evolve
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/app/poptimizer/__main__.py", line 46, in <module>
app(prog_name="poptimizer")
File "/usr/local/lib/python3.9/dist-packages/typer/main.py", line 214, in __call__
return get_command(self)(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/typer/main.py", line 500, in wrapper
return callback(**use_params) # type: ignore
File "/opt/app/poptimizer/__main__.py", line 17, in evolve
ev.evolve()
File "/opt/app/poptimizer/evolve/evolve.py", line 54, in evolve
step, current = self._step_setup(step, current)
File "/opt/app/poptimizer/evolve/evolve.py", line 85, in _step_setup
self._end = dates[1]
File "/usr/local/lib/python3.9/dist-packages/pandas/core/indexes/extension.py", line 279, in __getitem__
result = self._data[key]
File "/usr/local/lib/python3.9/dist-packages/pandas/core/arrays/datetimelike.py", line 322, in __getitem__
result = super().__getitem__(key)
File "/usr/local/lib/python3.9/dist-packages/pandas/core/arrays/_mixins.py", line 194, in __getitem__
result = self._ndarray[key]
IndexError: index 1 is out of bounds for axis 0 with size 0
Потомок:
Побед — 0
Data: {'batch_size': 177.7385721058952, 'history_days': 55.70134040397355, 'ticker_on': -90.52231354907431, 'day_of_year_on': 10.266435523458313, 'day_of_period_on': -59.75439292031155, 'prices_on': 464.5733644522056, 'dividends_on': -6.221069051894889, 'turnover_on': 0.6707214118303866, 'average_turnover_on': -64.84367367009655, 'rvi_on': 451.557234503783, 'mcftrr_on': 5.060442543491146, 'imoex_on': -2605.016051899486, 'ticker_type_on': 426.15916544317963, 'usd_on': -17.02749236435758, 'open_on': 36.30055136508749, 'high_on': 41.832617063123394, 'low_on': -42.290686709031405}
Model: {'start_bn': 3.68131633535158, 'kernels': 1.8169208373228016, 'sub_blocks': 1.172485135834774, 'gate_channels': 9.603954794985695, 'residual_channels': 14.822437221051137, 'skip_channels': 21.963910979579275, 'end_channels': 23.69156068767206, 'mixture_size': 12.36557598230185}
Optimizer: {'betas': 0.998975098900571, 'eps': 6.043738817960838e-09, 'weight_decay': 0.017549188685557042}
Scheduler: {'max_lr': 0.026301493988270034, 'epochs': 0.975965210378401, 'pct_start': 0.2750600642864314, 'anneal_strategy': -58.68592289332554, 'base_momentum': 0.6260647115426332, 'max_momentum': 0.7563211017511952, 'div_factor': 23.49088257256306, 'final_div_factor': 885334.0228427305}
Количество слоев - 74
Количество параметров - 13761
Epochs - 0.98
Train size - 197585
~~> Train: 27%|████████████████████████ | 295/1091 [00:13<00:35, 22.13it/s, -0.22357]
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/main.py", line 158, in
app(prog_name="poptimizer")
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/typer/main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/main.py", line 17, in evolve
ev.evolve()
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/evolve/evolve.py", line 76, in evolve
if _eval_and_print(child, tickers, end) is None:
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/evolve/evolve.py", line 112, in _eval_and_print
fitness = organism.evaluate_fitness(tickers, end)
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/evolve/population.py", line 83, in evaluate_fitness
llh, ir = model.quality_metrics
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 114, in quality_metrics
self._llh = self._eval_llh()
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 148, in _eval_llh
model = self.prepare_model(loader)
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 126, in prepare_model
self._model = self._train_model()
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 258, in _train_model
loss, means, _ = loss_fn(model, batch)
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 62, in log_normal_llh_mix
dist = model.dist(batch)
File "/home/torchok/PycharmProjects/poptimizer/poptimizer/dl/models/wave_net.py", line 318, in dist
weights_dist = distributions.Categorical(logits=logits)
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/torch/distributions/categorical.py", line 64, in init
super(Categorical, self).init(batch_shape, validate_args=validate_args)
File "/home/torchok/PycharmProjects/poptimizer/venv/lib/python3.9/site-packages/torch/distributions/distribution.py", line 53, in init
raise ValueError("The parameter {} has invalid values".format(param))
ValueError: The parameter logits has invalid values
Потомок:
Побед - 0
Data: {'batch_size': 204.9562127867058, 'history_days': 3.3102549152105727, 'ticker_on': -5776.724982106083, 'day_of_year_on': -1780.9273236372533, 'day_of_period_on': -13489.714181493244, 'prices_on': -30336.533005284495, 'dividends_on': -548625.4968190438, 'turnover_on': -397529.18654028786, 'average_turnover_on': -1324596.296612373, 'rvi_on': -954.3014412478325, 'mcftrr_on': -0.5122750395318638}
Model: {'start_bn': -580395.1587646941, 'kernels': 3.538092510358175, 'sub_blocks': 1.191626480806004, 'gate_channels': 294.91508063470513, 'residual_channels': 145.71541125887177, 'skip_channels': 135.19470189097453, 'end_channels': 201.1516020489509}
Optimizer: {'betas': 0.9893778147420297, 'eps': 2.735814118142896e-07, 'weight_decay': 0.49070839808123856, 'amsgrad': -258547.40207594028}
Scheduler: {'max_lr': 0.007796431125684897, 'epochs': 8.460116105978639, 'pct_start': 0.35364452023660325, 'anneal_strategy': -29558.046276696186, 'base_momentum': 0.9862028678552708, 'max_momentum': 0.5580588206266746, 'div_factor': 365.9928552054215, 'final_div_factor': 25411.010159529764}
Количество слоев - 30
Количество параметров - 769546
Epochs - 8.46
Train size - 189826
~~> Train: 0%| | 0/7877 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/main.py", line 19, in evolve
ev.evolve(port)
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/evolve/evolve.py", line 56, in evolve
child_fitness = self._eval_and_print(child, tickers, end)
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/evolve/evolve.py", line 84, in _eval_and_print
fitness = organism.evaluate_fitness(tickers, end)
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/evolve/population.py", line 76, in evaluate_fitness
llh = model.llh
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 104, in llh
self._llh = self._eval_llh()
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 123, in _eval_llh
model = self.get_model(loader)
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 159, in get_model
self._model = self._train_model()
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/dl/model.py", line 229, in _train_model
output = model(batch)
File "/home/user_name/anaconda3/envs/finan/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/dl/models/wave_net.py", line 273, in forward
y, skip = block(y)
File "/home/user_name/anaconda3/envs/finan/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/dl/models/wave_net.py", line 121, in forward
y = sub_block(y)
File "/home/user_name/anaconda3/envs/finan/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/user_name/PycharmProjects/poptimizer/poptimizer/dl/models/wave_net.py", line 51, in forward
y_signal = self.signal_conv(y)
File "/home/user_name/anaconda3/envs/finan/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/user_name/anaconda3/envs/finan/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 256, in forward
return F.conv1d(input, self.weight, self.bias, self.stride,
RuntimeError: Given groups=1, weight of size [294, 145, 3], expected input[1, 1, 3] to have 145 channels, but got 1 channels instead
Подскажите пожалуйста с чем эта может быть связана, и как её лечить/избегать:
Forecasts: 101it [23:21, 13.88s/it]
Traceback (most recent call last):
File "PATH_TO\poptimizer\optimize.py", line 12, in opt
optimize(date, ports=ports)
File "PATH_TO\poptimizer\poptimizer\__main__.py", line 23, in optimize
opt = Optimizer(port)
File "PATH_TO\poptimizer\poptimizer\portfolio\optimizer.py", line 38, in __init__
self._metrics = metrics.MetricsResample(portfolio)
File "PATH_TO\poptimizer\poptimizer\portfolio\metrics.py", line 158, in __init__
for forecast in evolve.get_forecasts(tickers, date):
File "PATH_TO\poptimizer\poptimizer\evolve\forecaster.py", line 142, in get_forecasts
return cache()
File "PATH_TO\poptimizer\poptimizer\evolve\forecaster.py", line 98, in __call__
return self._create_cache()
File "PATH_TO\poptimizer\poptimizer\evolve\forecaster.py", line 116, in _create_cache
forecasts = Forecasts(self._tickers, self._date)
File "PATH_TO\poptimizer\poptimizer\evolve\forecaster.py", line 31, in __init__
self._forecasts = forecasts or _prepare_forecasts(tickers, date)
File "PATH_TO\poptimizer\poptimizer\evolve\forecaster.py", line 61, in _prepare_forecasts
for organism in tqdm.tqdm(population.get_oldest(), desc="Forecasts"):
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\tqdm\std.py", line 1185, in __iter__
for obj in iterable:
File "PATH_TO\poptimizer\poptimizer\evolve\population.py", line 249, in get_oldest
for id_dict in collection.aggregate(pipeline):
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\command_cursor.py", line 259, in next
doc = self._try_next(True)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\command_cursor.py", line 270, in _try_next
self._refresh()
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\command_cursor.py", line 196, in _refresh
self.__send_message(
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\command_cursor.py", line 139, in __send_message
response = client._run_operation_with_response(
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\mongo_client.py", line 1368, in _run_operation_with_response
return self._retryable_read(
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\mongo_client.py", line 1471, in _retryable_read
return func(session, server, sock_info, slave_ok)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\mongo_client.py", line 1360, in _cmd
return server.run_operation_with_response(
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\server.py", line 135, in run_operation_with_response
_check_command_response(first, sock_info.max_wire_version)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pymongo\helpers.py", line 162, in _check_command_response
raise CursorNotFound(errmsg, code, response, max_wire_version)
pymongo.errors.CursorNotFound: cursor id 9024041513465701594 not found, full error: {'ok': 0.0, 'errmsg': 'cursor id 9024041513465701594 not found', 'code': 43, 'codeName': 'CursorNotFound'}
Если дать больше вычислительных ресурсов, то отрабатывает нормально:
Forecasts: 116it [10:30, 5.43s/it]
ПОРТФЕЛЬ - 2021-10-29
Первая ссылка по поводу говорит, что сюда
poptimizer/poptimizer/evolve/population.py
Line 239 in 084ac14
no_cursor_timeout=True
в ту сторону смотрю? там правда есть уточнение not valid at server, а здесь такого нет...
Traceback (most recent call last):
File "pandas\_libs\index.pyx", line 460, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas\_libs\hashtable_class_helper.pxi", line 2131, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas\_libs\hashtable_class_helper.pxi", line 2140, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 1636416000000000000
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\base.py", line 3361, in get_loc
return self._engine.get_loc(casted_key)
File "pandas\_libs\index.pyx", line 429, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas\_libs\index.pyx", line 462, in pandas._libs.index.DatetimeEngine.get_loc
KeyError: Timestamp('2021-11-09 00:00:00')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\datetimes.py", line 702, in get_loc
return Index.get_loc(self, key, method, tolerance)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\base.py", line 3363, in get_loc
raise KeyError(key) from err
KeyError: Timestamp('2021-11-09 00:00:00')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 151, in price
price = price.loc[self.date]
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexing.py", line 931, in __getitem__
return self._getitem_axis(maybe_callable, axis=axis)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexing.py", line 1164, in _getitem_axis
return self._get_label(key, axis=axis)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexing.py", line 1113, in _get_label
return self.obj.xs(label, axis=axis)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\generic.py", line 3773, in xs
loc = index.get_loc(key)
File "PATH_TO\anaconda3\envs\finan\lib\site-packages\pandas\core\indexes\datetimes.py", line 704, in get_loc
raise KeyError(orig_key) from err
KeyError: Timestamp('2021-11-09 00:00:00')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "PATH_TO\PycharmProjects\poptimizer\optimize.py", line 12, in opt
optimize(date, ports=ports)
File "PATH_TO\PycharmProjects\poptimizer\poptimizer\__main__.py", line 24, in optimize
print(opt.portfolio)
File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 66, in __str__
self._positions_stats(),
File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 91, in _positions_stats
weights = self.weight.iloc[:-2]
File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 176, in weight
value = self.value
File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 165, in value
value = self.price * self.shares
File "PATH_TO\PycharmProjects\poptimizer\poptimizer\portfolio\portfolio.py", line 153, in price
raise config.POptimizerError(
poptimizer.config.POptimizerError: Для даты 2021-11-09 отсутствуют исторические котировки
Кроме как удалением всех котировок, можно как-то лечить? Запуск переобучения моделей почему-то не подтягивает свежие котировки. Ошибка возникает при оптимизации. Видел её раньше, приходилось удалять все котировки (включая индексы и misc) и качать заново.
Приветствую!
В первую очередь спасибо за разработку.
Мой сетап:
OS: CentOS 7 64bit
Python 3.6.8
Пытаюсь запустить на последнем коммите из ветки мастер:
$ python3 -m poptimizer
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/doka/proj/2022/poptimizer/poptimizer/__main__.py", line 6, in <module>
from poptimizer import config
File "/home/doka/proj/2022/poptimizer/poptimizer/config.py", line 9, in <module>
from poptimizer.shared.log import get_handlers
File "/home/doka/proj/2022/poptimizer/poptimizer/shared/log.py", line 48, in <module>
) -> list[Handler]:
TypeError: 'type' object is not subscriptable
pytest жалуется на аналогичную проблему:
$ pytest poptimizer -v --cov=poptimizer --cov-report=term-missing --cov-report=xml --setup-show
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.6.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /home/doka/proj/2022/poptimizer/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/doka/proj/2022/poptimizer
plugins: asyncio-0.16.0, cov-3.0.0, mock-3.6.1
collected 0 items / 1 error
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/adapters/gateways/cpi.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/app/viewers.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/factory.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/tables/cbr.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/tables/cpi.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/tables/dividends.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/tables/indexes.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/tables/quotes.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/tables/trading_dates.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/data/domain/tables/usd.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/dl/features/turnover.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/dl/model.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/evolve/evolve.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/evolve/population.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/portfolio/portfolio.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/shared/adapters.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/shared/connections.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
/home/doka/proj/2022/poptimizer/.venv/lib64/python3.6/site-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/home/doka/proj/2022/poptimizer/poptimizer/store/database.py' (couldnt-parse)
coverage._warn(msg, slug="couldnt-parse")
========================================================================================================================================= ERRORS =========================================================================================================================================
_____________________________________________________________________________________________________________________________ ERROR collecting test session ______________________________________________________________________________________________________________________________
/usr/lib64/python3.6/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:994: in _gcd_import
???
<frozen importlib._bootstrap>:971: in _find_and_load
???
<frozen importlib._bootstrap>:941: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:994: in _gcd_import
???
<frozen importlib._bootstrap>:971: in _find_and_load
???
<frozen importlib._bootstrap>:941: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:994: in _gcd_import
???
<frozen importlib._bootstrap>:971: in _find_and_load
???
<frozen importlib._bootstrap>:955: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:665: in _load_unlocked
???
<frozen importlib._bootstrap_external>:678: in exec_module
???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
???
poptimizer/reports/__init__.py:2: in <module>
from poptimizer.reports.pdf import report
poptimizer/reports/pdf.py:5: in <module>
from poptimizer.config import REPORTS_PATH, POptimizerError
poptimizer/config.py:9: in <module>
from poptimizer.shared.log import get_handlers
poptimizer/shared/log.py:48: in <module>
) -> list[Handler]:
E TypeError: 'type' object is not subscriptable
----------- coverage: platform linux, python 3.6.8-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
-------------------------------------------------------------------------------------------------
poptimizer/__init__.py 0 0 0 0 100%
poptimizer/__main__.py 20 20 0 0 0% 2-36
poptimizer/config.py 36 29 4 0 18% 12-65
poptimizer/conftest.py 8 3 0 0 62% 11-13
poptimizer/data/__init__.py 0 0 0 0 100%
poptimizer/data/adapters/__init__.py 0 0 0 0 100%
poptimizer/data/adapters/gateways/__init__.py 0 0 0 0 100%
poptimizer/data/adapters/gateways/bcs.py 53 53 14 0 0% 2-83
poptimizer/data/adapters/gateways/cbr.py 28 28 4 0 0% 2-57
poptimizer/data/adapters/gateways/close_reestry.py 52 52 14 0 0% 2-91
poptimizer/data/adapters/gateways/conomy.py 54 54 4 0 0% 2-105
poptimizer/data/adapters/gateways/dividends.py 24 24 8 0 0% 2-50
poptimizer/data/adapters/gateways/dohod.py 24 24 2 0 0% 2-50
poptimizer/data/adapters/gateways/finrange.py 38 38 4 0 0% 2-71
poptimizer/data/adapters/gateways/gateways.py 17 17 8 0 0% 2-39
poptimizer/data/adapters/gateways/invest_mint.py 54 54 12 0 0% 2-111
poptimizer/data/adapters/gateways/moex.py 56 56 14 0 0% 2-173
poptimizer/data/adapters/gateways/moex_status.py 25 25 4 0 0% 2-58
poptimizer/data/adapters/gateways/nasdaq.py 35 35 4 0 0% 2-72
poptimizer/data/adapters/gateways/smart_lab.py 19 19 2 0 0% 2-48
poptimizer/data/adapters/gateways/street_insider.py 25 25 2 0 0% 2-52
poptimizer/data/adapters/html/__init__.py 0 0 0 0 100%
poptimizer/data/adapters/html/cell_parser.py 46 46 10 0 0% 2-64
poptimizer/data/adapters/html/chromium.py 28 28 6 0 0% 2-56
poptimizer/data/adapters/html/description.py 27 27 10 0 0% 2-56
poptimizer/data/adapters/html/parser.py 52 52 13 0 0% 2-87
poptimizer/data/adapters/odm.py 49 49 10 0 0% 2-107
poptimizer/data/app/__init__.py 0 0 0 0 100%
poptimizer/data/app/bootstrap.py 19 19 2 0 0% 2-47
poptimizer/data/domain/__init__.py 0 0 0 0 100%
poptimizer/data/domain/events.py 31 31 12 0 0% 2-64
poptimizer/data/domain/handlers.py 51 51 12 0 0% 2-123
poptimizer/data/domain/tables/__init__.py 0 0 0 0 100%
poptimizer/data/domain/tables/base.py 64 64 24 0 0% 2-123
poptimizer/data/domain/tables/securities.py 39 39 10 0 0% 2-86
poptimizer/data/ports.py 12 12 0 0 0% 2-15
poptimizer/data/views/__init__.py 0 0 0 0 100%
poptimizer/data/views/crop/__init__.py 0 0 0 0 100%
poptimizer/data/views/crop/div.py 17 17 2 0 0% 2-43
poptimizer/data/views/crop/not_div.py 17 17 2 0 0% 2-37
poptimizer/data/views/div_status.py 58 58 14 0 0% 2-124
poptimizer/data/views/go.py 8 8 0 0 0% 2-13
poptimizer/data/views/indexes.py 28 28 0 0 0% 2-65
poptimizer/data/views/listing.py 24 24 0 0 0% 2-52
poptimizer/data/views/quotes.py 34 34 8 0 0% 2-107
poptimizer/dl/__init__.py 4 4 0 0 0% 2-5
poptimizer/dl/data_loader.py 38 38 12 0 0% 2-86
poptimizer/dl/features/__init__.py 18 18 0 0 0% 2-19
poptimizer/dl/features/data_params.py 79 79 12 0 0% 2-170
poptimizer/dl/features/day_of_period.py 14 14 2 0 0% 2-29
poptimizer/dl/features/day_of_year.py 16 16 2 0 0% 2-32
poptimizer/dl/features/dividends.py 16 16 2 0 0% 2-28
poptimizer/dl/features/feature.py 18 18 4 0 0% 2-40
poptimizer/dl/features/high.py 20 20 2 0 0% 2-37
poptimizer/dl/features/imoex.py 19 19 2 0 0% 2-38
poptimizer/dl/features/label.py 28 28 2 0 0% 2-41
poptimizer/dl/features/low.py 20 20 2 0 0% 2-37
poptimizer/dl/features/mcftrr.py 19 19 2 0 0% 2-37
poptimizer/dl/features/meogtrr.py 19 19 2 0 0% 2-33
poptimizer/dl/features/open.py 21 21 2 0 0% 2-39
poptimizer/dl/features/prices.py 17 17 2 0 0% 2-27
poptimizer/dl/features/rvi.py 19 19 2 0 0% 2-37
poptimizer/dl/features/ticker.py 16 16 2 0 0% 2-26
poptimizer/dl/features/ticker_type.py 17 17 2 0 0% 2-32
poptimizer/dl/features/usd.py 19 19 2 0 0% 2-38
poptimizer/dl/forecast.py 19 19 2 0 0% 2-28
poptimizer/dl/ledoit_wolf.py 33 33 0 0 0% 2-82
poptimizer/dl/ledoit_wolf_nonlinear.py 52 52 4 0 0% 1-141
poptimizer/dl/models/__init__.py 1 1 0 0 0% 2
poptimizer/dl/models/wave_net.py 112 112 42 0 0% 2-334
poptimizer/evolve/__init__.py 2 2 0 0 0% 2-3
poptimizer/evolve/chromosomes/__init__.py 6 6 0 0 0% 2-7
poptimizer/evolve/chromosomes/chromosome.py 46 46 16 0 0% 2-134
poptimizer/evolve/chromosomes/data.py 22 22 34 0 0% 2-154
poptimizer/evolve/chromosomes/model.py 12 12 4 0 0% 2-75
poptimizer/evolve/chromosomes/optimizer.py 6 6 4 0 0% 2-33
poptimizer/evolve/chromosomes/scheduler.py 11 11 4 0 0% 2-73
poptimizer/evolve/chromosomes/utility.py 5 5 2 0 0% 2-37
poptimizer/evolve/forecaster.py 66 66 18 0 0% 2-137
poptimizer/evolve/genotype.py 33 33 10 0 0% 2-80
poptimizer/evolve/seq.py 22 22 6 0 0% 10-109
poptimizer/evolve/store.py 79 79 20 0 0% 2-151
poptimizer/portfolio/__init__.py 1 1 0 0 0% 2
poptimizer/portfolio/metrics.py 176 176 42 0 0% 2-346
poptimizer/portfolio/optimizer_hmean.py 99 99 14 0 0% 2-199
poptimizer/portfolio/optimizer_resample.py 81 81 10 0 0% 2-191
poptimizer/reports/__init__.py 2 1 0 0 50% 3
poptimizer/reports/excel.py 14 14 4 0 0% 1-25
poptimizer/reports/pdf.py 60 57 10 0 4% 6-153
poptimizer/reports/pdf_lower.py 62 62 4 0 0% 2-144
poptimizer/reports/pdf_middle.py 75 75 6 0 0% 3-130
poptimizer/reports/pdf_style.py 36 36 0 0 0% 2-76
poptimizer/reports/pdf_upper.py 85 85 8 0 0% 3-172
poptimizer/reports/simple.py 49 49 2 0 0% 2-107
poptimizer/shared/__init__.py 0 0 0 0 100%
poptimizer/shared/app.py 60 60 22 0 0% 2-133
poptimizer/shared/col.py 21 21 0 0 0% 2-30
poptimizer/shared/domain.py 35 35 14 0 0% 2-78
poptimizer/shared/log.py 33 20 4 0 41% 16, 23-29, 33-37, 41, 50-65
poptimizer/store/__init__.py 0 0 0 0 100%
-------------------------------------------------------------------------------------------------
TOTAL 3025 2996 599 0 1%
Coverage XML written to file coverage.xml
================================================================================================================================ short test summary info =================================================================================================================================
ERROR - TypeError: 'type' object is not subscriptable
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================================================================== 1 error in 2.17s ====================================================================================================================================
Я что-то делаю не так или есть какой-то более стабильно работающий коммит?
После обновления evolve аварийно завершается с трейсом:
Traceback (most recent call last):
File "/home/parsee/poptimizer/poptimizer/evolve/store.py", line 48, in __get__
return data_dict[self._name]
KeyError: 'llh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/parsee/poptimizer/./poptimizer/__main__.py", line 44, in <module>
app(prog_name="poptimizer")
File "/home/parsee/miniconda3/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
return get_command(self)(*args, **kwargs)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/home/parsee/poptimizer/./poptimizer/__main__.py", line 19, in evolve
ev.evolve(port)
File "/home/parsee/poptimizer/poptimizer/evolve/evolve.py", line 67, in evolve
_kill_weakest(child)
File "/home/parsee/poptimizer/poptimizer/evolve/evolve.py", line 114, in _kill_weakest
print(f"LLH: {weakest.llh:.4f}")
File "/home/parsee/poptimizer/poptimizer/evolve/population.py", line 63, in llh
return self._doc.llh
File "/home/parsee/poptimizer/poptimizer/evolve/store.py", line 50, in __get__
raise AttributeError(f"'{owner.__name__}' object has no attribute {error}")
AttributeError: 'Doc' object has no attribute 'llh'
Полный лог: https://gist.github.com/YoukaiCat/b3ef8830ad0241c7d1bbfabea4367047
Воспроизводится на master ветке без каких-либо моих изменений.
File "/home/PycharmProjects/poptimizer/poptimizer/main.py", line 19, in evolve
ev.evolve(port)
File "/home/PycharmProjects/poptimizer/poptimizer/evolve/evolve.py", line 56, in evolve
child_fitness = self._eval_and_print(child, tickers, end)
File "/home/PycharmProjects/poptimizer/poptimizer/evolve/evolve.py", line 84, in _eval_and_print
fitness = organism.evaluate_fitness(tickers, end)
File "/home/PycharmProjects/poptimizer/poptimizer/evolve/population.py", line 87, in evaluate_fitness
data.save()
File "/home/PycharmProjects/poptimizer/poptimizer/evolve/store.py", line 113, in save
collection.update_one(filter={ID: self.id}, update={"$set": self._update}, upsert=True)
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/collection.py", line 1019, in update_one
self._update_retryable(
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/collection.py", line 868, in _update_retryable
return self.__database.client._retryable_write(
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1498, in _retryable_write
return self._retry_with_session(retryable, func, s, None)
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
return self._retry_internal(retryable, func, session, bulk)
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1416, in _retry_internal
return func(session, sock_info, retryable)
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/collection.py", line 860, in _update
return self._update(
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/collection.py", line 829, in _update
result = sock_info.command(
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/pool.py", line 699, in command
self._raise_connection_failure(error)
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/pool.py", line 683, in command
return command(self, dbname, spec, slave_ok,
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/network.py", line 135, in command
message._raise_document_too_large(
File "/home/anaconda3/envs/finan/lib/python3.8/site-packages/pymongo/message.py", line 1092, in _raise_document_too_large
raise DocumentTooLarge("%r command document too large" % (operation,))
pymongo.errors.DocumentTooLarge: 'update' command document too large
Кажется, что из-за него возможны появления невозможных вариантов сделок:
cash: 1737
SELL Q_SELL BUY Q_BUY GRAD_DIFF TURNOVER P_VALUE
1 SBERP 1 KLSB 1 0.059933 0.812051 0.000027
2 MVID 1 KRSBP 1 0.046613 0.847026 0.000007
3 AKRN 1 URKZ 1 0.046562 0.913750 0.034296 <- вот эта невозможна,
так как 1737 + 6100 < 15600
ORUP переведен в категорию бумаг повышенного риска c 11.02.2022 (https://www.moex.com/n40783?utm_source=www.moex.com&utm_term=orup), выдаёт KeyError "['ORUP']" not in index
Добрый день, Михаил.
На тестовом портфеле стал возникать дамп. Подскажите пож, как с ним бороться.
RUR: 100000
USD: 0
positions:
QIWI: 5
FXMM: 10
FXRU: 35
GAZP: 104
SBER: 145
-- lgr.lgr:
2022-01-08 08:58:22.698136 | INFO | Config | config.py | line#46 | pid#2548 | {}
2022-01-08 08:58:23.708614 | INFO | EventBus | thread.py | line#52 | pid#2548 | DateCheckRequired(timestamp=datetime.datetime(2022, 1, 8, 8, 58, 23, 706201))
2022-01-08 08:58:27.317486 | INFO | root | main.py | line#33 | pid#2548 | ПОРТФЕЛЬ [base] - 2022-01-04
Количество бумаг - 5
Открытых позиций - 5
Эффективных позиций - 3
LOT_SIZE SHARES PRICE VALUE WEIGHT TURNOVER
SBER 10 145.0 304.24 44114.8 0.188128 60465.750212
GAZP 10 104.0 352.55 36665.2 0.156359 83952.855062
FXRU 1 35.0 953.00 33355.0 0.142242 69.284629
FXMM 1 10.0 1728.40 17284.0 0.073708 124.557150
QIWI 1 5.0 615.00 3075.0 0.013113 232.440619
CASH 1 100000.0 1.00 100000.0 0.426450 144844.887673
PORTFOLIO 1 1.0 234494.00 234494.0 1.000000 144844.887673
2022-01-08 08:58:27.492025 | INFO | root | main.py | line#34 | pid#2548 |
КЛЮЧЕВЫЕ МЕТРИКИ ПОРТФЕЛЯ
History days - (63 <-> 104 <-> 191)
Correlation - (0.47% <-> 1.49% <-> 2.49%)
Shrinkage - (8.67% <-> 10.61% <-> 14.96%)
Risk aversion - (1.00 <-> 1.00 <-> 1.00)
Error tolerance - (0.00 <-> 0.00 <-> 0.01)
MEAN STD BETA R_ADJ GRAD
FXMM 0.922696 2.211875 0.297528 1.074571 0.689361
FXRU 0.934053 2.207661 0.341905 1.057338 0.718068
GAZP 0.965154 2.218428 2.478487 0.204596 0.016911
QIWI 0.955321 2.226396 1.248873 0.453727 0.209521
SBER 0.938340 2.218428 2.758477 0.172311 -0.027261
CASH 0.000000 0.000000 0.000000 0.217704 -0.129250
PORTFOLIO 0.543051 0.659848 1.000000 0.235044 0.000000
Консервативная доходность: -43.72%
Консервативный риск: 299.60%
Консервативная метрика: -127.74%
Оценка просадки: 80.18%
--
C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\metrics.py:88: RuntimeWarning: invalid value encountered in matmul
portfolio_var = weight.reshape(1, -1) @ cov @ weight.reshape(-1, 1) # noqa: WPS221
C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\metrics.py:100: RuntimeWarning: invalid value encountered in matmul
beta = cov @ weight.reshape(-1, 1)
--- Logging error ---
Traceback (most recent call last):
File "C:\Python\Anaconda3\lib\logging\handlers.py", line 73, in emit
if self.shouldRollover(record):
File "C:\Python\Anaconda3\lib\logging\handlers.py", line 191, in shouldRollover
msg = "%s\n" % self.format(record)
File "C:\Python\Anaconda3\lib\logging_init_.py", line 927, in format
return fmt.format(record)
File "C:\Python\Anaconda3\lib\logging_init_.py", line 663, in format
record.message = record.getMessage()
File "C:\Python\Anaconda3\lib\logging_init_.py", line 365, in getMessage
msg = str(self.msg)
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 45, in str
df = self._for_trade()
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 68, in _for_trade
conf_int = self._prepare_bounds()
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 114, in _prepare_bounds
conf_int = conf_int.apply(
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\frame.py", line 8740, in apply
return op.apply()
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\apply.py", line 688, in apply
return self.apply_standard()
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\apply.py", line 812, in apply_standard
results, res_index = self.apply_series_generator()
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\apply.py", line 828, in apply_series_generator
results[i] = self.f(v)
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 115, in
lambda grad: _grad_conf_int(grad, p_value),
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 184, in _grad_conf_int
interval = stats.bootstrap(
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\scipy\stats_bootstrap.py", line 454, in bootstrap
ci_l = percentile_fun(theta_hat_b, interval[0]*100)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\scipy\stats_bootstrap.py", line 78, in _percentile_along_axis
percentiles[indices] = np.percentile(theta_hat_b_i, alpha_i)
File "<array_function internals>", line 180, in percentile
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\numpy\lib\function_base.py", line 4133, in percentile
raise ValueError("Percentiles must be in the range [0, 100]")
ValueError: Percentiles must be in the range [0, 100]
Call stack:
File "C:\Python\Anaconda3\lib\runpy.py", line 197, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Python\Anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Python\poptimizer-master_2022_01\poptimizer_main.py", line 46, in
app(prog_name="poptimizer")
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\typer\main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1128, in call
return self.main(*args, **kwargs)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1053, in main
rv = self.invoke(ctx)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1659, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 754, in invoke
return callback(*args, **kwargs)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\typer\main.py", line 500, in wrapper
return callback(**use_params) # type: ignore
File "C:\Python\poptimizer-master_2022_01\poptimizer_main.py", line 35, in optimize
LOGGER.info(opt)
Message: <poptimizer.portfolio.optimizer_resample.Optimizer object at 0x000000062D81AA60>
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
File "C:\Python\Anaconda3\lib\logging_init.py", line 1083, in emit
msg = self.format(record)
File "C:\Python\Anaconda3\lib\logging_init.py", line 927, in format
return fmt.format(record)
File "C:\Python\Anaconda3\lib\logging_init.py", line 663, in format
record.message = record.getMessage()
File "C:\Python\Anaconda3\lib\logging_init.py", line 365, in getMessage
msg = str(self.msg)
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 45, in str
df = self._for_trade()
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 68, in _for_trade
conf_int = self._prepare_bounds()
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 114, in _prepare_bounds
conf_int = conf_int.apply(
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\frame.py", line 8740, in apply
return op.apply()
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\apply.py", line 688, in apply
return self.apply_standard()
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\apply.py", line 812, in apply_standard
results, res_index = self.apply_series_generator()
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\pandas\core\apply.py", line 828, in apply_series_generator
results[i] = self.f(v)
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 115, in
lambda grad: _grad_conf_int(grad, p_value),
File "C:\Python\poptimizer-master_2022_01\poptimizer\portfolio\optimizer_resample.py", line 184, in _grad_conf_int
interval = stats.bootstrap(
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\scipy\stats_bootstrap.py", line 454, in bootstrap
ci_l = percentile_fun(theta_hat_b, interval[0]*100)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\scipy\stats_bootstrap.py", line 78, in _percentile_along_axis
percentiles[indices] = np.percentile(theta_hat_b_i, alpha_i)
File "<array_function internals>", line 180, in percentile
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\numpy\lib\function_base.py", line 4133, in percentile
raise ValueError("Percentiles must be in the range [0, 100]")
ValueError: Percentiles must be in the range [0, 100]
Call stack:
File "C:\Python\Anaconda3\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python\Anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Python\poptimizer-master_2022_01\poptimizer_main.py", line 46, in
app(prog_name="poptimizer")
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\typer\main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1128, in call
return self.main(*args, **kwargs)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1053, in main
rv = self.invoke(ctx)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 754, in invoke
return _callback(*args, **kwargs)
File "C:\Users\sn\AppData\Roaming\Python\Python39\site-packages\typer\main.py", line 500, in wrapper
return callback(**use_params) # type: ignore
File "C:\Python\poptimizer-master_2022_01\poptimizer_main.py", line 35, in optimize
LOGGER.info(opt)
Message: <poptimizer.portfolio.optimizer_resample.Optimizer object at 0x000000062D81AA60>
Arguments: ()
Добрый день. У вас ошибка в теге репозитория (это ерунда конечно но все же).
deap-learning -> deep-learning
Довольно регулярно вылетает такая ошибка (вероятно, в результате некорректного завершения работы) и приходится искать модель без генотипа. Предлагаю внести маленькое изменение:
поместить этот print
poptimizer/poptimizer/evolve/evolve.py
Line 102 in 197f39f
(ModelError, AttributeError)В разделе FAQ указано
У меня появилось сообщение ДАННЫЕ ПО ДИВИДЕНДАМ ТРЕБУЮТ ОБНОВЛЕНИЯ - что делать?
Вся необходимая для работы программы информация обновляется автоматически, кроме данных по дивидендам, которые необходимо обновлять в ручную в базе данныхsource. После ввода информации необходимо запустить команду dividends для дополнительной сверки с разными внешними источниками данных и перемещения информации в основную рабочую базу data.
Вот тут не понятно как именно обновлять вручную. Базу source из дампа я поднял. откуда для нее брать данные и как именно обновлять в ручную?
Внезапно появилась такая ошибка и "накрутила" всем побед. Какие меры мне лучше предпринять?
{"t":{"$date":"2021-04-06T13:24:33.059+03:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2021-04-06T13:24:33.061+03:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2021-04-06T13:24:33.062+03:00"},"s":"I", "c":"NETWORK", "id":4648602, "ctx":"main","msg":"Implicit TCP FastOpen in use."}
{"t":{"$date":"2021-04-06T13:24:33.062+03:00"},"s":"I", "c":"STORAGE", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":748,"port":27017,"dbPath":"PATH_TO/poptimizer/db","architecture":"64-bit","host":"iRU"}}
{"t":{"$date":"2021-04-06T13:24:33.062+03:00"},"s":"I", "c":"CONTROL", "id":23398, "ctx":"initandlisten","msg":"Target operating system minimum version","attr":{"targetMinOS":"Windows 7/Windows Server 2008 R2"}}
{"t":{"$date":"2021-04-06T13:24:33.062+03:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.0","gitVersion":"563487e100c4215e2dce98d0af2a6a5a2d67c5cf","modules":[],"allocator":"tcmalloc","environment":{"distmod":"windows","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2021-04-06T13:24:33.062+03:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Microsoft Windows 10","version":"10.0 (build 19042)"}}}
{"t":{"$date":"2021-04-06T13:24:33.062+03:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"localhost"},"storage":{"dbPath":"PATH_TO\poptimizer\db","directoryPerDB":true}}}}
{"t":{"$date":"2021-04-06T13:24:33.069+03:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"PATH_TO/poptimizer/db","storageEngine":"wiredTiger"}}
{"t":{"$date":"2021-04-06T13:24:33.069+03:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=15820M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2021-04-06T13:24:33.088+03:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1617704673:88162][748:140703820895568], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 457 through 458"}}
{"t":{"$date":"2021-04-06T13:24:33.136+03:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1617704673:136034][748:140703820895568], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 458 through 458"}}
{"t":{"$date":"2021-04-06T13:24:33.181+03:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1617704673:180914][748:140703820895568], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 457/222080 to 458/256"}}
{"t":{"$date":"2021-04-06T13:24:33.263+03:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1617704673:263693][748:140703820895568], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 457 through 458"}}
{"t":{"$date":"2021-04-06T13:24:33.320+03:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1617704673:320541][748:140703820895568], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 458 through 458"}}
{"t":{"$date":"2021-04-06T13:24:33.363+03:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1617704673:363426][748:140703820895568], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
{"t":{"$date":"2021-04-06T13:24:33.375+03:00"},"s":"I", "c":"STORAGE", "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":306}}
{"t":{"$date":"2021-04-06T13:24:33.375+03:00"},"s":"I", "c":"RECOVERY", "id":23987, "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2021-04-06T13:24:33.377+03:00"},"s":"I", "c":"STORAGE", "id":22262, "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2021-04-06T13:24:33.378+03:00"},"s":"W", "c":"CONTROL", "id":22120, "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","tags":["startupWarnings"]}
{"t":{"$date":"2021-04-06T13:24:33.381+03:00"},"s":"I", "c":"STORAGE", "id":20536, "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2021-04-06T13:24:33.546+03:00"},"s":"W", "c":"FTDC", "id":23718, "ctx":"initandlisten","msg":"Failed to initialize Performance Counters for FTDC","attr":{"error":{"code":179,"codeName":"WindowsPdhError","errmsg":"PdhExpandCounterPathW failed with 'Указанные объекты не найдены на этом компьютере.' for counter '\Processor(_Total)\% Idle Time'"}}}
{"t":{"$date":"2021-04-06T13:24:33.546+03:00"},"s":"I", "c":"FTDC", "id":20625, "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"PATH_TO/poptimizer/db/diagnostic.data"}}
{"t":{"$date":"2021-04-06T13:24:33.548+03:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"127.0.0.1"}}
{"t":{"$date":"2021-04-06T13:24:33.548+03:00"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
INFO:EventBus:AppStarted(timestamp=datetime.datetime(2021, 4, 6, 10, 24, 34, 635047))
Forecasts: 17it [00:23, 1.39s/it]
Traceback (most recent call last):
File "PATH_TO\poptimizer\poptimizer\evolve\store.py", line 48, in get
return data_dict[self._name]
KeyError: 'genotype'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "PATH_TO\poptimizer\loop.py", line 13, in opt
optimize(date)
File "PATH_TO\poptimizer\poptimizer_main_.py", line 30, in optimize
opt = Optimizer(port)
File "PATH_TO\poptimizer\poptimizer\portfolio\optimizer.py", line 36, in init
self._metrics = metrics.MetricsResample(portfolio)
File "PATH_TO\poptimizer\poptimizer\portfolio\metrics.py", line 138, in init
for forecast in evolve.get_forecasts(tickers, date):
File "PATH_TO\poptimizer\poptimizer\evolve\forecaster.py", line 69, in get_forecasts
forecasts = Forecasts(tickers, date)
File "PATH_TO\poptimizer\poptimizer\evolve\forecaster.py", line 29, in init
forecast = organism.forecast(tickers, date)
File "PATH_TO\poptimizer\poptimizer\evolve\population.py", line 139, in forecast
model = Model(tickers, end, self.genotype.get_phenotype(), pickled_model)
File "PATH_TO\poptimizer\poptimizer\evolve\population.py", line 47, in genotype
return self._data.genotype
File "PATH_TO\poptimizer\poptimizer\evolve\store.py", line 50, in get
raise AttributeError(f"'{owner.name}' object has no attribute {error}")
AttributeError: 'Doc' object has no attribute 'genotype'
NOW is 2021-04-06 13:24:59.426034
Имеется 100 организмов из 100
2021-04-05: Шаг переоценки существующих организмов - 1/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 282895
Data: {'batch_size': 129.11398996479213, 'history_days': 37.015207014300145, 'ticker_on': 1.4961829685606562, 'day_of_year_on': -1.3184921148222324, 'day_of_period_on': -0.3581207837799719, 'prices_on': 2.9821826510974634, 'dividends_on': -1.218817060144394, 'turnover_on': 0.914842380640876, 'average_turnover_on': -0.699126762530033, 'rvi_on': -2.792763447791985, 'mcftrr_on': -1.93265326300987, 'imoex_on': -5.306833839096052, 'ticker_type_on': -3.339556376347452, 'usd_on': -2.0302456959424413, 'open_on': 2.377407126067244}
Model: {'start_bn': -2.170820364329111, 'kernels': 1.2304203372717688, 'sub_blocks': 1.0165487399288449, 'gate_channels': 3.655193907692617, 'residual_channels': 3.110450587130084, 'skip_channels': 3.6009756675600033, 'end_channels': 12.543127189713942}
Optimizer: {'betas': 0.9989469716669452, 'eps': 9.990772823439389e-09, 'weight_decay': 0.009851862286420794}
Scheduler: {'max_lr': 0.008438415665195705, 'epochs': 0.9942368141030508, 'pct_start': 0.30569184948124123, 'anneal_strategy': 1.1834934807741755, 'base_momentum': 0.8543202286943965, 'max_momentum': 0.9501062028416284, 'div_factor': 24.872879360917633, 'final_div_factor': 10594.662867350402}
LLH: 1.0127
Timer: 34
2021-04-05: Шаг переоценки существующих организмов - 2/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283266
Data: {'batch_size': 130.71316719246317, 'history_days': 45.254623552239096, 'ticker_on': 4.414210986145641, 'day_of_year_on': -3.162958601682031, 'day_of_period_on': 1.408086465864079, 'prices_on': 3.43601163648374, 'dividends_on': 0.8990273569914335, 'turnover_on': 1.2812019467955387, 'average_turnover_on': -0.5149759332821369, 'rvi_on': -6.835235543125595, 'mcftrr_on': -1.2141754347540386, 'imoex_on': 0.8270908692909673, 'ticker_type_on': -0.45949887305703196, 'usd_on': 1.749619114397614, 'open_on': 0.8941885000762019}
Model: {'start_bn': -0.14486289430410093, 'kernels': 2.3093182631129996, 'sub_blocks': 1.3869500871513922, 'gate_channels': 3.8775953733659234, 'residual_channels': 3.8575443778678387, 'skip_channels': 4.313933338601139, 'end_channels': 8.859915011287493}
Optimizer: {'betas': 0.9989802718243128, 'eps': 9.97206879489969e-09, 'weight_decay': 0.009973884393479118}
Scheduler: {'max_lr': 0.01175190008029012, 'epochs': 0.9917025866227448, 'pct_start': 0.29579159180383385, 'anneal_strategy': 1.138798664510326, 'base_momentum': 0.8549236452613173, 'max_momentum': 0.9524465372866897, 'div_factor': 25.212607987459382, 'final_div_factor': 9872.104572793327}
LLH: 1.0109
Timer: 51
2021-04-05: Шаг переоценки существующих организмов - 3/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283232
Data: {'batch_size': 129.14643040318686, 'history_days': 54.50594029150181, 'ticker_on': 2.115290052257728, 'day_of_year_on': -0.605611354736193, 'day_of_period_on': -0.8520918255997714, 'prices_on': 1.6323379506740474, 'dividends_on': -0.3279877259664346, 'turnover_on': 1.2664755935505292, 'average_turnover_on': -0.7406823893061762, 'rvi_on': -1.1590445449003424, 'mcftrr_on': -0.3492122657367346, 'imoex_on': -2.89406703917413, 'ticker_type_on': -1.3013872878613275, 'usd_on': -0.7449143409561085, 'open_on': 0.6925362788889224}
Model: {'start_bn': -0.4717546217081192, 'kernels': 2.183638757726063, 'sub_blocks': 1.1323090645053433, 'gate_channels': 5.280161228661479, 'residual_channels': 4.217797691082319, 'skip_channels': 4.239420257019827, 'end_channels': 8.506476109331658}
Optimizer: {'betas': 0.9989834591232805, 'eps': 9.994310399452697e-09, 'weight_decay': 0.01007220385406504}
Scheduler: {'max_lr': 0.007764736034557539, 'epochs': 1.0000526529820437, 'pct_start': 0.30321315779622576, 'anneal_strategy': -0.8663086938527101, 'base_momentum': 0.8515211468352687, 'max_momentum': 0.9461697684068386, 'div_factor': 24.562424675016622, 'final_div_factor': 10067.283703884208}
LLH: 1.0079
Timer: 30
2021-04-05: Шаг переоценки существующих организмов - 4/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283316
Data: {'batch_size': 127.3890773382612, 'history_days': 35.297505822749606, 'ticker_on': 2.804127796026597, 'day_of_year_on': -0.9359065818905388, 'day_of_period_on': -1.0395959466803584, 'prices_on': 1.6323379506740474, 'dividends_on': -0.3279877259664346, 'turnover_on': 1.1471947724675142, 'average_turnover_on': -0.16509257766528385, 'rvi_on': -4.568702974814391, 'mcftrr_on': -0.4989881291618443, 'imoex_on': -2.89406703917413, 'ticker_type_on': -1.0103945596892876, 'usd_on': -0.9229464056875802, 'open_on': 0.6925362788889224}
Model: {'start_bn': 0.48476424269648216, 'kernels': 3.5617878436362433, 'sub_blocks': 1.2593684197709267, 'gate_channels': 3.0315556948935356, 'residual_channels': 5.82514672195885, 'skip_channels': 5.096100627824891, 'end_channels': 8.506476109331658}
Optimizer: {'betas': 0.9990019691432226, 'eps': 9.977581253256297e-09, 'weight_decay': 0.009981105321278523}
Scheduler: {'max_lr': 0.0036661261806283756, 'epochs': 0.9994436809991594, 'pct_start': 0.30291731203982464, 'anneal_strategy': -0.43525336483691457, 'base_momentum': 0.8487618611039887, 'max_momentum': 0.9552621810907912, 'div_factor': 24.92997399906494, 'final_div_factor': 9935.61692246235}
LLH: 1.0077
Timer: 33
2021-04-05: Шаг переоценки существующих организмов - 5/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283221
Data: {'batch_size': 129.5487944564173, 'history_days': 32.36127680856102, 'ticker_on': 0.7011247282330246, 'day_of_year_on': -1.857850553183083, 'day_of_period_on': 1.4195213570117273, 'prices_on': 0.6964216386220465, 'dividends_on': -1.153634625953878, 'turnover_on': 2.1102497474778406, 'average_turnover_on': -3.1913740055639495, 'rvi_on': -2.5393398074285485, 'mcftrr_on': -4.69851087964688, 'imoex_on': -3.104498844510889, 'ticker_type_on': -0.05772789379846566, 'usd_on': -0.8680799383593495, 'open_on': -0.6190260639010782}
Model: {'start_bn': 0.7557948842868958, 'kernels': 2.584219888776984, 'sub_blocks': 1.3561183949595448, 'gate_channels': 3.18883626383162, 'residual_channels': 5.300472732305459, 'skip_channels': 4.75163379239307, 'end_channels': 10.611426433296922}
Optimizer: {'betas': 0.9989841633358855, 'eps': 9.977691237490357e-09, 'weight_decay': 0.010072505024148722}
Scheduler: {'max_lr': 0.02854526387104548, 'epochs': 1.0018595658098304, 'pct_start': 0.2999523000194047, 'anneal_strategy': 0.8365494562790383, 'base_momentum': 0.8489453360449295, 'max_momentum': 0.9539226049209667, 'div_factor': 24.887657226441803, 'final_div_factor': 10481.752750294121}
LLH: 1.0045
Timer: 29
2021-04-05: Шаг переоценки существующих организмов - 6/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283264
Data: {'batch_size': 124.21062605176519, 'history_days': 32.67785817149473, 'ticker_on': 1.3060997077503325, 'day_of_year_on': -7.936433292459158, 'day_of_period_on': 2.060009238765374, 'prices_on': 1.9148294054983712, 'dividends_on': 2.6038430925449703, 'turnover_on': 1.4819486657028693, 'average_turnover_on': -1.8464205620187308, 'rvi_on': -6.602621776577463, 'mcftrr_on': -1.7546778704878725, 'imoex_on': -0.4755212965667067, 'ticker_type_on': -1.0996231793309064, 'usd_on': 2.1194544077971615, 'open_on': 1.9804186477453523}
Model: {'start_bn': -1.5651408192057843, 'kernels': 3.6605500750289717, 'sub_blocks': 1.4072307553497114, 'gate_channels': 4.855084404400753, 'residual_channels': 4.326888841913921, 'skip_channels': 5.788781656780704, 'end_channels': 7.469472088956564}
Optimizer: {'betas': 0.9990311787852406, 'eps': 9.988894574332353e-09, 'weight_decay': 0.010091130670425842}
Scheduler: {'max_lr': 0.027795188069731806, 'epochs': 0.9967358598507899, 'pct_start': 0.29886806636072116, 'anneal_strategy': -0.9397714362228164, 'base_momentum': 0.84913177441488, 'max_momentum': 0.9572537007436318, 'div_factor': 25.07582657094894, 'final_div_factor': 10552.913826574493}
LLH: 1.0027
Timer: 45
2021-04-05: Шаг переоценки существующих организмов - 7/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283337
Data: {'batch_size': 129.11398996479213, 'history_days': 43.29744444905284, 'ticker_on': 0.735412176157566, 'day_of_year_on': -1.557348897095807, 'day_of_period_on': -0.3581207837799719, 'prices_on': 0.6002072410011461, 'dividends_on': 1.532254271286333, 'turnover_on': 2.334680820809848, 'average_turnover_on': -0.699126762530033, 'rvi_on': -2.792763447791985, 'mcftrr_on': -1.93265326300987, 'imoex_on': -1.60552005974125, 'ticker_type_on': -0.4156808853894721, 'usd_on': -0.344491486165805, 'open_on': 1.6349581093280006}
Model: {'start_bn': -1.7040348988448693, 'kernels': 2.783507604278526, 'sub_blocks': 1.1950114355829806, 'gate_channels': 4.745658473792522, 'residual_channels': 4.146579751612516, 'skip_channels': 4.436116925717199, 'end_channels': 12.543127189713942}
Optimizer: {'betas': 0.9989637727531835, 'eps': 9.990772823439389e-09, 'weight_decay': 0.010169437865328644}
Scheduler: {'max_lr': 0.008438415665195705, 'epochs': 0.9951716403269572, 'pct_start': 0.30236494392487, 'anneal_strategy': 1.8304205574661059, 'base_momentum': 0.8527166703274321, 'max_momentum': 0.9466437535961111, 'div_factor': 24.724676788284086, 'final_div_factor': 10080.827151780597}
LLH: 0.9998
Timer: 37
2021-04-05: Шаг переоценки существующих организмов - 8/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283574
Data: {'batch_size': 127.49862615207198, 'history_days': 51.73668384768355, 'ticker_on': 1.6668398265693205, 'day_of_year_on': -2.104375905901345, 'day_of_period_on': 0.04797090715941338, 'prices_on': 2.8019867245624948, 'dividends_on': 1.2010200611503368, 'turnover_on': 0.8563418456881949, 'average_turnover_on': 0.3390305390974662, 'rvi_on': -2.1128372471109405, 'mcftrr_on': -1.230338548465791, 'imoex_on': -1.578237345982047, 'ticker_type_on': -1.7271048289836193, 'usd_on': -0.10131940210631403, 'open_on': 2.2762163613346527}
Model: {'start_bn': -0.12199213957494909, 'kernels': 2.285662196403919, 'sub_blocks': 1.2230820979811132, 'gate_channels': 3.2806968717556853, 'residual_channels': 3.616627605518013, 'skip_channels': 4.415009367881208, 'end_channels': 6.553571494928423}
Optimizer: {'betas': 0.998998952856435, 'eps': 9.97161216448502e-09, 'weight_decay': 0.01004943288181694}
Scheduler: {'max_lr': 0.02681743387930703, 'epochs': 0.9989498885224218, 'pct_start': 0.300482004619608, 'anneal_strategy': -0.034917066508703076, 'base_momentum': 0.8542522503715722, 'max_momentum': 0.9516106331592764, 'div_factor': 24.436444158148685, 'final_div_factor': 10237.05214805306}
LLH: 0.9969
Timer: 39
2021-04-05: Шаг переоценки существующих организмов - 9/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 283854
Data: {'batch_size': 131.7136368739434, 'history_days': 32.20042114567875, 'ticker_on': 1.2026228332380475, 'day_of_year_on': -1.7204774542706809, 'day_of_period_on': -1.262185962559439, 'prices_on': 1.2131344308186114, 'dividends_on': 1.6341928782411455, 'turnover_on': 0.6764887695329795, 'average_turnover_on': -0.23209959524970836, 'rvi_on': -2.6445266787475883, 'mcftrr_on': -4.104668521341506, 'imoex_on': -1.0946394188163844, 'ticker_type_on': 2.0057922750462263, 'usd_on': -1.919167917482044, 'open_on': -2.4146226127606893}
Model: {'start_bn': -1.7040348988448693, 'kernels': 2.5750853981289055, 'sub_blocks': 1.2278142469968878, 'gate_channels': 3.693980601755917, 'residual_channels': 3.7063630006197155, 'skip_channels': 4.063800033496327, 'end_channels': 12.543127189713942}
Optimizer: {'betas': 0.9990227337594352, 'eps': 1.0060624238616954e-08, 'weight_decay': 0.010208150886727793}
Scheduler: {'max_lr': 0.019806999084712333, 'epochs': 0.9876311874452565, 'pct_start': 0.29928579068225475, 'anneal_strategy': -0.7489924140354225, 'base_momentum': 0.8485622740061275, 'max_momentum': 0.9526890035830102, 'div_factor': 24.3989712333599, 'final_div_factor': 9888.659764290083}
LLH: 0.9958
Timer: 30
2021-04-05: Шаг переоценки существующих организмов - 10/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 284086
Data: {'batch_size': 127.17356135002618, 'history_days': 38.125731599994666, 'ticker_on': 1.0709456448978822, 'day_of_year_on': -2.3298473772626394, 'day_of_period_on': 0.4527593332187303, 'prices_on': 1.6290063743667915, 'dividends_on': 1.1396869976768111, 'turnover_on': 0.8126634506060731, 'average_turnover_on': -1.0820504558139223, 'rvi_on': -0.40665114711689354, 'mcftrr_on': 1.0722365273559096, 'imoex_on': -2.6127075396767814, 'ticker_type_on': -1.7466876629796155, 'usd_on': -1.244601353580486, 'open_on': 0.7032420333172618}
Model: {'start_bn': -1.1953885845219354, 'kernels': 1.9628285614808827, 'sub_blocks': 1.2096970640967675, 'gate_channels': 4.149128095154571, 'residual_channels': 4.40043979610523, 'skip_channels': 4.072633926779601, 'end_channels': 6.911928494344707}
Optimizer: {'betas': 0.9989747330989738, 'eps': 9.965976651810387e-09, 'weight_decay': 0.010136475921371771}
Scheduler: {'max_lr': 0.02089162150903083, 'epochs': 0.9924477811966503, 'pct_start': 0.3030005207746779, 'anneal_strategy': 0.493977283368806, 'base_momentum': 0.8519067460302696, 'max_momentum': 0.9513717801768419, 'div_factor': 25.285776742919897, 'final_div_factor': 10281.722280226806}
LLH: 0.9954
Timer: 46
2021-04-05: Шаг переоценки существующих организмов - 11/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 284636
Data: {'batch_size': 132.09895965625606, 'history_days': 37.94209432123361, 'ticker_on': 3.9198905889472524, 'day_of_year_on': -2.3615728185797966, 'day_of_period_on': 2.1500001370097377, 'prices_on': 0.7851002444870305, 'dividends_on': 0.34894018552039696, 'turnover_on': 0.05049546602830868, 'average_turnover_on': -0.41393985043823145, 'rvi_on': -1.2308050372675108, 'mcftrr_on': -6.75195231977609, 'imoex_on': -0.18592898779016886, 'ticker_type_on': 0.9742815695233601, 'usd_on': 2.13706085368269, 'open_on': -3.5382062808082666}
Model: {'start_bn': 3.0008226183831015, 'kernels': 3.426055906780645, 'sub_blocks': 1.3282073148928848, 'gate_channels': 5.861725774349978, 'residual_channels': 5.156465439482679, 'skip_channels': 5.546817024471426, 'end_channels': 10.161418182618414}
Optimizer: {'betas': 0.9990218556415129, 'eps': 1.0038378058617539e-08, 'weight_decay': 0.010111874343043106}
Scheduler: {'max_lr': 0.02657908619908253, 'epochs': 0.9987220155975899, 'pct_start': 0.29485364473079284, 'anneal_strategy': 2.4786462105162306, 'base_momentum': 0.8547921377195049, 'max_momentum': 0.9505139957854383, 'div_factor': 24.70007035729723, 'final_div_factor': 10232.242537969347}
LLH: 0.9942
Timer: 43
2021-04-05: Шаг переоценки существующих организмов - 12/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 284937
Data: {'batch_size': 129.06413826957555, 'history_days': 48.411970887829845, 'ticker_on': 1.185883239521181, 'day_of_year_on': -3.1161282031870736, 'day_of_period_on': -0.12978014889844874, 'prices_on': 2.8844185841578764, 'dividends_on': 0.13215395456730664, 'turnover_on': 1.4572986265779178, 'average_turnover_on': -1.6838949023006808, 'rvi_on': -3.2377957588186073, 'mcftrr_on': -1.7065778676110945, 'imoex_on': -5.538660309379201, 'ticker_type_on': 0.9423668463756475, 'usd_on': -0.5677760330728062, 'open_on': 2.7595291638017825}
Model: {'start_bn': -0.009889836379681965, 'kernels': 2.9382256501335324, 'sub_blocks': 1.113612418952892, 'gate_channels': 4.444970526990106, 'residual_channels': 3.7496116552604772, 'skip_channels': 4.399243510403746, 'end_channels': 8.093747138719504}
Optimizer: {'betas': 0.9989834414597125, 'eps': 9.958269634578969e-09, 'weight_decay': 0.010034420549575863}
Scheduler: {'max_lr': 0.014894741340022515, 'epochs': 0.9945015447909407, 'pct_start': 0.3004539977445932, 'anneal_strategy': 0.4021360233671351, 'base_momentum': 0.8514024131701574, 'max_momentum': 0.9581642259761585, 'div_factor': 25.138310729911666, 'final_div_factor': 10247.648300695622}
LLH: 0.9940
Timer: 36
2021-04-05: Шаг переоценки существующих организмов - 13/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 285189
Data: {'batch_size': 129.61500214665156, 'history_days': 55.31868831412401, 'ticker_on': 0.931404936842664, 'day_of_year_on': -3.638659870481857, 'day_of_period_on': 1.4078050556191406, 'prices_on': 2.1910137188916474, 'dividends_on': 0.11498553075282236, 'turnover_on': 1.9454675354678514, 'average_turnover_on': -0.6676346129338598, 'rvi_on': -4.772375470636296, 'mcftrr_on': -1.4855613793562803, 'imoex_on': -1.6268119573030444, 'ticker_type_on': -0.2716349789855247, 'usd_on': 1.901268382316126, 'open_on': 0.8214387648272354}
Model: {'start_bn': -0.8727644546676028, 'kernels': 2.7083244786339216, 'sub_blocks': 1.3815790453466792, 'gate_channels': 3.921986310255915, 'residual_channels': 3.548003770777458, 'skip_channels': 5.489351825152683, 'end_channels': 8.762312333954057}
Optimizer: {'betas': 0.9989822687380243, 'eps': 9.978709315399259e-09, 'weight_decay': 0.010122757605364196}
Scheduler: {'max_lr': 0.013021483486942823, 'epochs': 0.9955112438293735, 'pct_start': 0.29526694953823235, 'anneal_strategy': 1.9797802825050212, 'base_momentum': 0.8535224664921386, 'max_momentum': 0.957107346764789, 'div_factor': 25.567671388733405, 'final_div_factor': 9924.13915344965}
LLH: 0.9937
Timer: 44
2021-04-05: Шаг переоценки существующих организмов - 14/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 285188
Data: {'batch_size': 125.56142942984138, 'history_days': 39.03428177378684, 'ticker_on': 1.0241321934227394, 'day_of_year_on': -8.773181388318761, 'day_of_period_on': 1.8233026915462622, 'prices_on': 2.3739640054458757, 'dividends_on': 1.7700724375538193, 'turnover_on': 0.4744135848107208, 'average_turnover_on': -1.3934578224900216, 'rvi_on': -6.717151175202348, 'mcftrr_on': -2.1573266534963085, 'imoex_on': -0.9478032880823322, 'ticker_type_on': -0.057849855565923036, 'usd_on': 2.2371891465696234, 'open_on': 0.6906769315788238}
Model: {'start_bn': -1.1502211731323875, 'kernels': 3.171661850173492, 'sub_blocks': 1.380012963264939, 'gate_channels': 4.409707262282689, 'residual_channels': 4.717634654952425, 'skip_channels': 5.0383045057230955, 'end_channels': 8.327922381265513}
Optimizer: {'betas': 0.9990035902743758, 'eps': 9.980403015094026e-09, 'weight_decay': 0.010063787039953098}
Scheduler: {'max_lr': 0.02670206855767571, 'epochs': 0.9955391912587991, 'pct_start': 0.2988314343863867, 'anneal_strategy': -0.292902107633636, 'base_momentum': 0.8480438759048313, 'max_momentum': 0.9590899362627602, 'div_factor': 25.556593046976428, 'final_div_factor': 10034.32162597855}
LLH: 0.9936
Timer: 47
2021-04-05: Шаг переоценки существующих организмов - 15/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 285356
Data: {'batch_size': 124.14288731919436, 'history_days': 35.2783702088608, 'ticker_on': 0.4760106640175248, 'day_of_year_on': -9.327063242868723, 'day_of_period_on': 1.2689905103448793, 'prices_on': 3.0554795554437213, 'dividends_on': 2.7695848176814932, 'turnover_on': 0.029861897297741913, 'average_turnover_on': -0.6851659544431169, 'rvi_on': -2.6445266787475883, 'mcftrr_on': -4.104668521341506, 'imoex_on': -1.5976359820568033, 'ticker_type_on': 3.546322596962837, 'usd_on': -0.7369311155721017, 'open_on': 3.6692192094297926}
Model: {'start_bn': 1.5700899378755133, 'kernels': 3.816654544582109, 'sub_blocks': 1.2438965334796046, 'gate_channels': 6.745644303700853, 'residual_channels': 3.580647864175917, 'skip_channels': 4.063800033496327, 'end_channels': 11.1768826215664}
Optimizer: {'betas': 0.9990324187105801, 'eps': 9.998723180249824e-09, 'weight_decay': 0.009993970944875015}
Scheduler: {'max_lr': 0.03565489868375743, 'epochs': 0.9832781980468731, 'pct_start': 0.29859567262437514, 'anneal_strategy': 0.15671900518369952, 'base_momentum': 0.8570873641030302, 'max_momentum': 0.9562006268497444, 'div_factor': 23.914192282824015, 'final_div_factor': 9716.059387251045}
LLH: 0.9935
Timer: 40
2021-04-05: Шаг переоценки существующих организмов - 16/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 285540
Data: {'batch_size': 128.16606046388947, 'history_days': 30.928971409826925, 'ticker_on': 1.9782511811771557, 'day_of_year_on': -3.266398071242293, 'day_of_period_on': -0.38931533660365814, 'prices_on': 2.2065510451931667, 'dividends_on': 0.06841875958225627, 'turnover_on': 1.403049420665596, 'average_turnover_on': -0.7076719892046817, 'rvi_on': -4.568702974814391, 'mcftrr_on': -2.0428598261369526, 'imoex_on': -5.240053886752641, 'ticker_type_on': -1.0103945596892876, 'usd_on': -0.5051735246868357, 'open_on': 1.2252659762885214}
Model: {'start_bn': -1.0518595772471402, 'kernels': 1.8777499668839939, 'sub_blocks': 1.1752107252908797, 'gate_channels': 3.8001687776109114, 'residual_channels': 3.668155571431715, 'skip_channels': 5.403237349032759, 'end_channels': 8.209692915692491}
Optimizer: {'betas': 0.9989695464307408, 'eps': 9.995859268415831e-09, 'weight_decay': 0.009981105321278523}
Scheduler: {'max_lr': 0.018645541159247676, 'epochs': 0.9994436809991594, 'pct_start': 0.3020645250449366, 'anneal_strategy': 0.4240996795127956, 'base_momentum': 0.8487618611039887, 'max_momentum': 0.9542086806491754, 'div_factor': 25.040189519820228, 'final_div_factor': 10205.296970918454}
LLH: 0.9928
Timer: 34
2021-04-05: Шаг переоценки существующих организмов - 17/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285541
Родитель:
Побед - 285541
Data: {'batch_size': 128.02451740493672, 'history_days': 74.90270576456544, 'ticker_on': 1.2423475420476409, 'day_of_year_on': -0.5404438350947776, 'day_of_period_on': -0.026814566664855405, 'prices_on': 1.9766172547706324, 'dividends_on': 0.36968902430741085, 'turnover_on': 0.4398467489715099, 'average_turnover_on': 0.8447546511404818, 'rvi_on': -1.0906776772540918, 'mcftrr_on': -0.2469610066229145, 'imoex_on': -2.2318207233287284, 'ticker_type_on': -0.7982001712992459, 'usd_on': -0.8769582965798767, 'open_on': 0.7374743491927982}
Model: {'start_bn': -0.26689366888959487, 'kernels': 2.17314783525574, 'sub_blocks': 1.2702420526026488, 'gate_channels': 4.691816784873492, 'residual_channels': 4.548800728206341, 'skip_channels': 5.66065097256378, 'end_channels': 5.118295201407214}
Optimizer: {'betas': 0.9990029166402363, 'eps': 9.99986981424181e-09, 'weight_decay': 0.010071213431929597}
Scheduler: {'max_lr': 0.006487685225365178, 'epochs': 1.0012833521263194, 'pct_start': 0.3007243366993013, 'anneal_strategy': 0.5026070320339392, 'base_momentum': 0.850864336790906, 'max_momentum': 0.9520295726343669, 'div_factor': 25.027491866892902, 'final_div_factor': 10148.252966746757}
LLH: 0.9928
Timer: 39
2021-04-05: Шаг переоценки существующих организмов - 18/100
LLH - (0.9165, 0.9678, 1.0127)
Максимум побед - 285542
Родитель:
Побед - 16
'Doc' object has no attribute 'genotype'
Traceback (most recent call last):
File "PATH_TO\poptimizer\poptimizer\evolve\store.py", line 48, in get
return data_dict[self._name]
KeyError: 'genotype'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "PATH_TO\poptimizer\loop.py", line 29, in
evolve()
File "PATH_TO\poptimizer\poptimizer_main_.py", line 19, in evolve
ev.evolve(port)
File "PATH_TO\poptimizer\poptimizer\evolve\evolve.py", line 42, in evolve
self._eval_exiting(portfolio)
File "PATH_TO\poptimizer\poptimizer\evolve\evolve.py", line 98, in _eval_exiting
parent_fitness = self._eval_and_print(parent, tickers, end)
File "PATH_TO\poptimizer\poptimizer\evolve\evolve.py", line 112, in _eval_and_print
print(organism)
File "PATH_TO\poptimizer\poptimizer\evolve\population.py", line 37, in str
return str(self._data.genotype)
File "PATH_TO\poptimizer\poptimizer\evolve\store.py", line 50, in get
raise AttributeError(f"'{owner.name}' object has no attribute {error}")
AttributeError: 'Doc' object has no attribute 'genotype'
Process finished with exit code 0
В начале очередного обучения
DateCheckRequired(timestamp=datetime.datetime(2022, 1, 14, 10, 14, 52, 239306))
Epochs - 0.49 / Train size - 229297
Количество слоев / параметров - 104 / 468639
~~> Train: 0%| | 0/13 [00:02<?, ?it/s$
Traceback (most recent call last):
Вылетело с ошибкой
RuntimeError: CUDA out of memory. Tried to allocate 72.00 MiB (GPU 0; 1.95 GiB total capacity; 1.03 GiB already allocated; 46.62 MiB free; 1.17 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Что происходит с обучением если оно прерывается и можно ли в таком случае запускать на CPU?
За 25.11 в программе:
YNDX 635.6000
Факт за 25.11:
YNDX 2629
После обновления перестала работать эволюция.
Выводит трейс:
Родитель нападает на добычу:
Traceback (most recent call last):
File "/home/parsee/poptimizer/./poptimizer/__main__.py", line 37, in <module>
app(prog_name="poptimizer")
File "/home/parsee/miniconda3/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
return get_command(self)(*args, **kwargs)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/parsee/miniconda3/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/home/parsee/poptimizer/./poptimizer/__main__.py", line 12, in evolve
ev.evolve()
File "/home/parsee/poptimizer/poptimizer/evolve/evolve.py", line 70, in evolve
if self._prey_killed(parent, prey):
File "/home/parsee/poptimizer/poptimizer/evolve/evolve.py", line 148, in _prey_killed
_, p_value = stats.ttest_ind(
TypeError: ttest_ind() got an unexpected keyword argument 'permutations'
Полный лог: https://gist.github.com/YoukaiCat/a31a1acabb877f66c6880cc452aab9c0
$ python -m poptimizer optimize 2021.12.14
{'P_VALUE': 0.05, 'FORECAST_DAYS': 21, 'HISTORY_DAYS_MIN': 63, 'TARGET_POPULATION': 100, 'START_EVOLVE_HOUR': 1, 'STOP_EVOLVE_HOUR': 1, 'COSTS': 0.025, 'MARKET_IMPACT_FACTOR': 1, 'OPTIMIZER': 'resample'}
DateCheckRequired(timestamp=datetime.datetime(2021, 12, 15, 7, 40, 13, 992642))
Forecasts: 1it [00:20, 20.82s/it]
Traceback (most recent call last):
File "/PATH_TO/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3361, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'MAIL'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/PATH_TO/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/PATH_TO/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/PATH_TO/poptimizer/poptimizer/main.py", line 46, in
app(prog_name="poptimizer")
File "/PATH_TO/lib/python3.9/site-packages/typer/main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "/PATH_TO/lib/python3.9/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/PATH_TO/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/PATH_TO/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/PATH_TO/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/PATH_TO/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/PATH_TO/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
return callback(**use_params) # type: ignore
File "/PATH_TO/poptimizer/poptimizer/main.py", line 32, in optimize
opt = opt_type(port)
File "/PATH_TO/poptimizer/poptimizer/portfolio/optimizer_resample.py", line 41, in init
self._metrics = metrics.MetricsResample(portfolio)
File "/PATH_TO/poptimizer/poptimizer/portfolio/metrics.py", line 154, in init
for forecast in evolve.get_forecasts(tickers, date):
File "/PATH_TO/poptimizer/poptimizer/evolve/forecaster.py", line 140, in get_forecasts
return cache()
File "/PATH_TO/poptimizer/poptimizer/evolve/forecaster.py", line 96, in call
return self._create_cache()
File "/PATH_TO/poptimizer/poptimizer/evolve/forecaster.py", line 114, in _create_cache
forecasts = Forecasts(self._tickers, self._date)
File "/PATH_TO/poptimizer/poptimizer/evolve/forecaster.py", line 31, in init
self._forecasts = forecasts or _prepare_forecasts(tickers, date)
File "/PATH_TO/poptimizer/poptimizer/evolve/forecaster.py", line 64, in _prepare_forecasts
forecast = organism.forecast(tickers, date)
File "/PATH_TO/poptimizer/poptimizer/evolve/population.py", line 156, in forecast
forecast = model.forecast()
File "/PATH_TO/poptimizer/poptimizer/dl/model.py", line 288, in forecast
loader = data_loader.DescribedDataLoader(
File "/PATH_TO/poptimizer/poptimizer/dl/data_loader.py", line 67, in init
data_sets = [OneTickerDataset(ticker, params) for ticker in tickers]
File "/PATH_TO/poptimizer/poptimizer/dl/data_loader.py", line 67, in
data_sets = [OneTickerDataset(ticker, params) for ticker in tickers]
File "/PATH_TO/poptimizer/poptimizer/dl/data_loader.py", line 19, in init
self.features = [
File "/PATH_TO/poptimizer/poptimizer/dl/data_loader.py", line 20, in
getattr(features, feat_name)(ticker, params) for feat_name in params.get_all_feat()
File "/PATH_TO/poptimizer/poptimizer/dl/features/ticker_type.py", line 23, in init
ticker_type = listing.ticker_types()[ticker]
File "/PATH_TO/lib/python3.9/site-packages/pandas/core/series.py", line 942, in getitem
return self._get_value(key)
File "/PATH_TO/lib/python3.9/site-packages/pandas/core/series.py", line 1051, in _get_value
loc = self.index.get_loc(label)
File "/PATH_TO/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3363, in get_loc
raise KeyError(key) from err
KeyError: 'MAIL'
Добрый день, Михаил.
Первый раз прервал эволюцию на 4ххх шаге, подумал что баг. После повторного ситуация повторилась.
Посмотрите пожалуйста логи https://drive.google.com/drive/folders/1aonlwT0NKgGX2yuU4uYijlHzqFtCx1YB?usp=sharing , может быть я зря панику поднял. Можно поискать в файлах по слову "Delta" - там всегда "28".
Может быть можно с Вами онлайн проконсультироваться по конкретной инсталляции poptimizer, например за оплату :) ?
добрый день! Скажите, а почему вы публикуете свою разработку? Что вас толкает на этот шаг?
INFO:Config:{'P_VALUE': 0.05, 'FORECAST_DAYS': 21, 'HISTORY_DAYS_MIN': 63, 'TARGET_POPULATION': 100, 'START_EVOLVE_HOUR': 1, 'STOP_EVOLVE_HOUR': 1, 'COSTS': 0.025, 'MARKET_IMPACT_FACTOR': 1, 'OPTIMIZER': 'resample'}
INFO:EventBus:DateCheckRequired(timestamp=datetime.datetime(2021, 12, 2, 14, 18, 51, 211020))
Forecasts: 99it [19:11, 11.64s/it]
ПОРТФЕЛЬ [base, test] - 2021-12-01
Количество бумаг - 199
Открытых позиций - 8
Эффективных позиций - 4
LOT_SIZE SHARES PRICE VALUE WEIGHT TURNOVER
.
.
.
PORTFOLIO ...........
PATH_TO/poptimizer/portfolio/metrics.py:78: RuntimeWarning: invalid value encountered in matmul
portfolio_var = weight.reshape(1, -1) @ cov @ weight.reshape(-1, 1) # noqa: WPS221
PATH_TO/poptimizer/portfolio/metrics.py:90: RuntimeWarning: invalid value encountered in matmul
beta = cov @ weight.reshape(-1, 1)
КЛЮЧЕВЫЕ МЕТРИКИ ПОРТФЕЛЯ
Дней в истории - (89 <-> 146 <-> 196)
Корреляция - (7.16% <-> 8.09% <-> 8.82%)
Сжатие - (40.07% <-> 46.09% <-> 52.70%)
MEAN STD BETA R_GEOM GRAD
AAL-RM ...
.
.
.
CASH ...
PORTFOLIO ...
Консервативная доходность: 25.69%
Консервативный риск: 51.69%
Консервативная метрика: 16.23%
Оценка просадки: 36.07%
Traceback (most recent call last):
File "PATH_TO/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "PATH_TO/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "PATH_TO/poptimizer/main.py", line 42, in
app(prog_name="poptimizer")
File "PATH_TO/lib/python3.9/site-packages/typer/main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "PATH_TO/lib/python3.9/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "PATH_TO/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "PATH_TO/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "PATH_TO/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "PATH_TO/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "PATH_TO/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
return callback(**use_params) # type: ignore
File "PATH_TO/poptimizer/main.py", line 31, in optimize
print(opt)
File "PATH_TO/poptimizer/portfolio/optimizer_resample.py", line 45, in str
df = self._for_trade()
File "PATH_TO/poptimizer/portfolio/optimizer_resample.py", line 68, in _for_trade
conf_int = self._prepare_bounds()
File "PATH_TO/poptimizer/portfolio/optimizer_resample.py", line 111, in _prepare_bounds
conf_int = conf_int.apply(
File "PATH_TO/lib/python3.9/site-packages/pandas/core/frame.py", line 8740, in apply
return op.apply()
File "PATH_TO/lib/python3.9/site-packages/pandas/core/apply.py", line 688, in apply
return self.apply_standard()
File "PATH_TO/lib/python3.9/site-packages/pandas/core/apply.py", line 812, in apply_standard
results, res_index = self.apply_series_generator()
File "PATH_TO/lib/python3.9/site-packages/pandas/core/apply.py", line 828, in apply_series_generator
results[i] = self.f(v)
File "PATH_TO/poptimizer/portfolio/optimizer_resample.py", line 112, in
lambda grad: _grad_conf_int(grad, p_value),
File "PATH_TO/poptimizer/portfolio/optimizer_resample.py", line 179, in _grad_conf_int
interval = stats.bootstrap(
File "PATH_TO/lib/python3.9/site-packages/scipy/stats/_bootstrap.py", line 454, in bootstrap
ci_l = percentile_fun(theta_hat_b, interval[0]*100)
File "PATH_TO/lib/python3.9/site-packages/scipy/stats/_bootstrap.py", line 78, in _percentile_along_axis
percentiles[indices] = np.percentile(theta_hat_b_i, alpha_i)
File "<array_function internals>", line 5, in percentile
File "PATH_TO/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3866, in percentile
raise ValueError("Percentiles must be in the range [0, 100]")
ValueError: Percentiles must be in the range [0, 100]
2020-10-01: Шаг эпохи - 100/100
LLH - (2.0130, 2.1284, 26548.4739)
Максимум побед - 30
Родитель:
Побед - 30
Data: {'batch_size': 127.45135180958202, 'history_days': 253.82088632293883, 'ticker_on': -2.127426227385521, 'day_of_year_on': -5.0167343543925575, 'day_of_period_on': 0.1399657547039146, 'prices_on': -1.7791840595416208, 'dividends_on': -2.5585755251108013, 'turnover_on': -3.2084742407103635, 'average_turnover_on': -0.22674459319728557}
Model: {'start_bn': -2.1176982922820216, 'kernels': 2.300783417110007, 'sub_blocks': 1.360626893510361, 'gate_channels': 2.460099421116139, 'residual_channels': 5.8362842700907, 'skip_channels': 5.782226700386957, 'end_channels': 2.127746428553457}
Optimizer: {'betas': 0.9991061935123207, 'eps': 1.009124393366369e-08, 'weight_decay': 0.010240856523939598, 'amsgrad': 3.3038695826083053}
Scheduler: {'max_lr': 0.010750924886635432, 'epochs': 1.004782005843315, 'pct_start': 0.3038539102518628, 'anneal_strategy': 0.47784917341259225, 'base_momentum': 0.8558587213774674, 'max_momentum': 0.9469542283955424, 'div_factor': 24.67513505453672, 'final_div_factor': 10421.271524928849}
LLH: 26548.4739
Timer: 45
Потомок:
Побед - 0
Data: {'batch_size': 127.45135180958202, 'history_days': 280.89739122307157, 'ticker_on': -22.0823140380744, 'day_of_year_on': -0.4995328699316497, 'day_of_period_on': 6.173325860767405, 'prices_on': 2.73561726442144, 'dividends_on': -67.65783995739321, 'turnover_on': 34.62113859021244, 'average_turnover_on': 10.105738046802282}
Model: {'start_bn': 4.880492776786426, 'kernels': 2.003447537341679, 'sub_blocks': 1.0364606635047944, 'gate_channels': 2.460099421116139, 'residual_channels': 15.16052058481371, 'skip_channels': 4.301512677524137, 'end_channels': 9.311633520507543}
Optimizer: {'betas': 0.9982990880714625, 'eps': 1.0243818093682281e-08, 'weight_decay': 0.019987539992897638, 'amsgrad': 19.496689755005246}
Scheduler: {'max_lr': 0.0022085446585436604, 'epochs': 1.0244479512664655, 'pct_start': 0.3345462730031154, 'anneal_strategy': -1.3506650286263788, 'base_momentum': 0.8439443332526304, 'max_momentum': 0.9448479477272338, 'div_factor': 27.007370575496044, 'final_div_factor': 7894.043358954786}
Количество слоев - 102
Количество параметров - 10677
Epochs - 1.02
Train size - 128412
~~> Train: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1037/1037 [01:00<00:00, 17.00it/s, 1.50460]
Тестовых дней: 1
Тестовых примеров: 115
~~> Test: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 29.90it/s, 1.81867]
STD: 0.0348
LLH: 1.8187
Timer: 62
Более слабый и наиболее медленный - удаляю:
Побед - 1
Data: {'batch_size': 127.45135180958202, 'history_days': 280.89739122307157, 'ticker_on': -22.0823140380744, 'day_of_year_on': -0.4995328699316497, 'day_of_period_on': 6.173325860767405, 'prices_on': 2.73561726442144, 'dividends_on': -67.65783995739321, 'turnover_on': 34.62113859021244, 'average_turnover_on': 10.105738046802282}
Model: {'start_bn': 4.880492776786426, 'kernels': 2.003447537341679, 'sub_blocks': 1.0364606635047944, 'gate_channels': 2.460099421116139, 'residual_channels': 15.16052058481371, 'skip_channels': 4.301512677524137, 'end_channels': 9.311633520507543}
Optimizer: {'betas': 0.9982990880714625, 'eps': 1.0243818093682281e-08, 'weight_decay': 0.019987539992897638, 'amsgrad': 19.496689755005246}
Scheduler: {'max_lr': 0.0022085446585436604, 'epochs': 1.0244479512664655, 'pct_start': 0.3345462730031154, 'anneal_strategy': -1.3506650286263788, 'base_momentum': 0.8439443332526304, 'max_momentum': 0.9448479477272338, 'div_factor': 27.007370575496044, 'final_div_factor': 7894.043358954786}
LLH: 1.8187
Timer: 62
Получил такую ошибку. Стал смотреть, пришел к этой строчке:
Не запускается evolve после ночного обновления данных. Перезагрузка компьютера не помогла.
(base) usr@macbookpro poptimizer % python3 -m poptimizer evolve
{'P_VALUE': 0.05, 'FORECAST_DAYS': 21, 'HISTORY_DAYS_MIN': 63, 'TARGET_POPULATION': 100, 'START_EVOLVE_HOUR': 1, 'STOP_EVOLVE_HOUR': 1, 'COSTS': 0.025, 'MARKET_IMPACT_FACTOR': 1, 'OPTIMIZER': 'hmean'}
DateCheckRequired(timestamp=datetime.datetime(2022, 1, 21, 7, 42, 54, 441795))
DateCheckRequired(timestamp=datetime.datetime(2022, 1, 21, 7, 42, 54, 857282))
2022-01-20: Шаг эволюции — 1
LLH - (2.2584, 2.5542, 2.6298)
RET - (-0.3664, 0.1511, 1.4129)
Организмов - 315 / Максимум оценок - 47 / Поколений - 46
Scale - 100.00% / Jump - 100.00%
Родитель:
LLH — 2.3963: 2.3901, 2.6493, 2.6055, 2.5613, 2.5048, 2.7029, 2.7147, 2.6704, 2.6321, 2.5576, 2.6209, 2.5343, 2.4712, 2.5926, 2.6181, 2.3990, 2.5223, 2.5368, 2.5439, 2.2595, 2.3036, 2.2998, 1.8157, 1.6775, 1.6825, 2.2263, 2.0741, 2.2370, 2.3936, 2.1730, 2.2580, 2.4198, 2.2640, 2.2981, 2.3188, 2.0094, 2.2812, 2.3129, 2.4277, 2.1300, 2.2505, 2.3495, 2.5247, 2.4868
RET — 0.2492: -0.7723, -0.4505, -0.4916, 0.6591, 0.9796, -0.3230, 0.1437, -0.6328, 1.3425, 0.4985, 0.8764, -0.2682, -0.5779, 0.2591, 0.4496, -0.3588, -0.1654, 0.0780, 0.2898, -0.2750, 0.1712, -0.6531, -0.1056, -1.0902, -0.2169, -0.3889, -0.5889, 0.1422, 0.8783, 0.0086, 1.3413, 1.4211, 1.2323, 1.1356, 1.2360, 0.2829, 0.2394, 1.7225, 1.2317, 1.1361, 2.2526, 1.4470, 2.1715, 1.9925
Timer — 0:00:49 / Scores - 44
Data: {'batch_size': 134.93720982008335, 'history_days': 95.15098732309566, 'ticker_on': 0.278420801448794, 'day_of_year_on': 1.1551116675663693, 'day_of_period_on': 1.407492925322691, 'prices_on': -1.773827611186878, 'dividends_on': 0.014815914938429209, 'turnover_on': -5.916029228275023, 'average_turnover_on': 2.5581540279669333, 'rvi_on': 4.121499840929927, 'mcftrr_on': 0.5684369411017152, 'imoex_on': 0.2905145279785871, 'ticker_type_on': 0.6350888402315884, 'usd_on': -0.49814312990177595, 'open_on': 2.186429049519103, 'high_on': 7.9216789403768875, 'low_on': 923.4768231918068, 'meogtrr_on': -0.863917323297819}
Model: {'start_bn': 0.9910986626592879, 'kernels': 6.515293278016405, 'sub_blocks': 1.824447247226041, 'gate_channels': 4.39506695664396, 'residual_channels': 5.4473180230707285, 'skip_channels': 13.305460075664165, 'end_channels': 15.178670905203244, 'mixture_size': 3.878975632979092}
Optimizer: {'betas': 0.9989941493992441, 'eps': 1.9611817594831026e-08, 'weight_decay': 0.00041400063415165337}
Scheduler: {'max_lr': 0.0037451630007245304, 'epochs': 0.2425709289463799, 'pct_start': 0.30574198650894757, 'anneal_strategy': 0.6214439018017496, 'base_momentum': 0.8484705644342704, 'max_momentum': 0.9500814644420272, 'div_factor': 136.75930600312077, 'final_div_factor': 76869.10327725377}
Utility: {'risk_aversion': 0.9999993419164337, 'error_tolerance': 7.944409623215222e-07}
~~> Test: 100%|█████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 21.61it/s, 2.48251]
RET = -24.25% / MEAN = -12.61% / PLAN = 33.23% / STD = 17.41% / DD = 9.13% / POS = 5 / MAX = 34.61%
LLH worst difference: median - -0.0382, upper - -0.0049, max - 0.0656
RET worst difference: median - -0.4175, upper - 0.2462, max - 0.9418
Traceback (most recent call last):
File "/Users/usr/miniconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/usr/miniconda3/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/usr/poptimizer/poptimizer/main.py", line 46, in
app(prog_name="poptimizer")
File "/Users/usr/.local/lib/python3.9/site-packages/typer/main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "/Users/usr/.local/lib/python3.9/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/Users/usr/.local/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/Users/usr/.local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/usr/.local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/usr/.local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/Users/usr/.local/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
return callback(**use_params) # type: ignore
File "/Users/usr/poptimizer/poptimizer/main.py", line 17, in evolve
ev.evolve()
File "/Users/usr/poptimizer/poptimizer/evolve/evolve.py", line 64, in evolve
current = self._step(current)
File "/Users/usr/poptimizer/poptimizer/evolve/evolve.py", line 117, in _step
if (hunter_margin := self._eval_organism(hunter)) is None:
File "/Users/usr/poptimizer/poptimizer/evolve/evolve.py", line 183, in _eval_organism
return self._get_margin(organism)
File "/Users/usr/poptimizer/poptimizer/evolve/evolve.py", line 215, in _get_margin
time_delta = _time_delta(org)
File "/Users/usr/poptimizer/poptimizer/evolve/evolve.py", line 228, in _time_delta
max_timer = max(doc["timer"] for doc in population.base_pop_metrics())
File "/Users/usr/poptimizer/poptimizer/evolve/evolve.py", line 228, in
max_timer = max(doc["timer"] for doc in population.base_pop_metrics())
KeyError: 'timer'
Почему доверительный интервал считается по одному числу? Точнее по Series, в котором всегда, как я понял, один элемент.
Trace:
https://gist.github.com/RomaKoks/1127898bdde3c159ed6afe13a4cb30a4
Добрый день.
Дано:
есть свой файлик yaml, в который решил добавить те тикеры, которые есть в моем портфеле
Основной файлик Base,yaml не трогал, лежит там же.
и последующем запуске эволюции сразу лезут ошибки вида
DateCheckRequired(timestamp=datetime.datetime(2022, 1, 18, 19, 39, 56, 71760))
2022-01-17: Шаг эволюции — 2
LLH - (2.4163, 2.5501, 2.6234)
RET - (-0.4540, -0.0710, 0.4886)
Организмов - 85 / Максимум оценок - 43 / Поколений - 28
Scale - 100.00% / Jump - 100.00%
Родитель:
LLH — 2.4163: 2.7493, 2.6579, 2.4699, 2.4052, 2.7301, 2.7596, 2.7478, 2.7000, 2.6327, 2.6701, 2.6168, 2.5104, 2.5461, 2.5803, 2.5389, 2.5670, 2.6316, 2.6467, 2.3963, 2.4169, 2.4133, 2.1570, 2.0501, 1.8055, 2.1613, 2.2694, 2.2799, 2.3627, 2.3293, 2.3727, 2.4316, 2.2940, 2.3144, 2.3029, 2.1684, 2.3228, 2.2833, 2.4085, 2.1471, 2.2747, 2.4163, 2.5512, 2.5370
RET — -0.0903: 0.7476, 0.9777, 0.8330, 0.9740, 0.8815, -0.5283, -0.8624, 0.2988, 0.1168, -0.0903, -0.6548, -0.4354, -0.5218, -0.3202, -0.2633, -0.2295, 0.1301, -0.2642, -0.2813, -0.1566, -0.3569, -0.5271, -0.6825, -0.5586, -0.8170, -0.9328, 0.1180, 0.8336, 0.0467, 0.5420, 0.9018, 0.6491, 0.0775, 0.1866, 0.0258, 0.3128, 0.1574, -0.1091, -0.1108, -0.4889, -0.1940, 0.7710, 1.3757
Timer — 0:03:22 / Scores - 43
Data: {'batch_size': 544.1814490060337, 'history_days': 126.37214308665581, 'ticker_on': -13491.846639888232, 'day_of_year_on': -0.601958378825853, 'day_of_period_on': 29.320250898279127, 'prices_on': 219.3078601523154, 'dividends_on': -94.03392082356538, 'turnover_on': -103.31561921851228, 'average_turnover_on': 37.584941661289825, 'rvi_on': 8.402798891568866, 'mcftrr_on': -0.2789684357328445, 'imoex_on': -24.26899966300151, 'ticker_type_on': 694.8571667119032, 'usd_on': 47.56922801593863, 'open_on': -132.18096255162706, 'high_on': 2.4895491290099665, 'low_on': 114.84474608037817, 'meogtrr_on': 2.095715318698276}
Model: {'start_bn': -421.3330993824715, 'kernels': 3.719762867174905, 'sub_blocks': 1.9520882594366002, 'gate_channels': 8.440372214039686, 'residual_channels': 5.875438269756684, 'skip_channels': 7.40759706459868, 'end_channels': 7.62571004461237, 'mixture_size': 10.523801895289735}
Optimizer: {'betas': 0.9990115004885081, 'eps': 7.955905503913788e-05, 'weight_decay': 0.005508104412380203}
Scheduler: {'max_lr': 0.009869463663043398, 'epochs': 2.2468356409415797, 'pct_start': 0.35004594173121356, 'anneal_strategy': -0.9423892531093931, 'base_momentum': 0.8445112716437683, 'max_momentum': 0.9234711039553989, 'div_factor': 295.2147414200126, 'final_div_factor': 44610292.672476746}
Utility: {'risk_aversion': 0.9999991339442165, 'error_tolerance': 5.0221981632873055e-09}
LLH worst difference: median - -0.0598, upper - 0.0321, max - 0.0907
RET worst difference: median - -0.1266, upper - 0.3109, max - 0.5497
Margin - 34.95%, Time excess - 0.00%
Претендент - новый организм:
LLH — -
RET — -
Timer — 0:00:00 / Scores - 0
Data: {'batch_size': 546.0288263569632, 'history_days': 70.69244290473151, 'ticker_on': -539390685.840478, 'day_of_year_on': -607495.4801111785, 'day_of_period_on': -8929092.988403583, 'prices_on': 860.7470191170048, 'dividends_on': -9067.985229056168, 'turnover_on': -1622023639.9564042, 'average_turnover_on': -255887899.8858772, 'rvi_on': -63315.11569619761, 'mcftrr_on': 6306.1632638281135, 'imoex_on': 1343119.0439553764, 'ticker_type_on': -3195745.751872108, 'usd_on': 9714512.01830391, 'open_on': -517211.8643529856, 'high_on': -207.54076891507916, 'low_on': -4808746.602419375, 'meogtrr_on': -351584.44049646135}
Model: {'start_bn': 949674.8117367319, 'kernels': 52.600334602229125, 'sub_blocks': 2.153565411706507, 'gate_channels': 8.347306285275819, 'residual_channels': 10.527285597153835, 'skip_channels': 2484.763325765774, 'end_channels': 4.157612469882803, 'mixture_size': 8.498520495937893}
Optimizer: {'betas': 0.9978341459548834, 'eps': 1.1944942001621881e-05, 'weight_decay': 0.13978151294114266}
Scheduler: {'max_lr': 0.006932698487612888, 'epochs': 1.63914976243321, 'pct_start': 0.38025769596284015, 'anneal_strategy': -423319.59884408774, 'base_momentum': 0.6350220928447462, 'max_momentum': 0.886668315656381, 'div_factor': 499938.9681106344, 'final_div_factor': 5948204855146.895}
Utility: {'risk_aversion': 1.000022100491393, 'error_tolerance': 9.250173338794867e-06}
DateCheckRequired(timestamp=datetime.datetime(2022, 1, 18, 19, 39, 58, 394184))
Удаляю - Слишком большая длинна истории - 70
И такое сообщение постоянно выходит
Причем если удалить данные тикеры - сразу становится все ок - при длине истории больше 130 модель эволюционирует без проблем
И вопрос про удаление тикеров - оптимизация мне настойчиво пишет купить тикер STSBP, которым мой брокер не торгует, я хотел бы исключить данный тикер из эволюции, но в этом случае при запуске эволюции в пятницу днем получал ошибку, что такой-то тикер не найден.
UPD: Хотя сейчас запустилось после удаления данного тикера. Значит, удалять тикеры можно делать только вечером ?
Есть ли смысл обмениваться моделями ? У меня в один момент численность популяции снизилась до 1 модели, я попросил товарища скинуть бэкап своих моделей, и загрузил их к себе, и уже на основе их продолжаю эволюцию
Добрый день
Подскажите пожалуйста: метод optimize, выдающий рекомендации - его следует запускать несколько раз в течение дня до тех пор пока он не покажет что оптимизировать больше нечего? или один раз в день?
т.е.:
Попробовал обновиться
2021-05-07: Шаг эволюции — 1
LLH - (-, -, -)
Максимум побед - None
Фактор - 100.00%
Переоцениваю родителя:
Побед — 0
Data: {'batch_size': 128.2320698209353, 'history_days': 77.63810095882658, 'ticker_on': 0.7109999601234973, 'day_of_year_on': 0.8938235704831489, 'day_of_period_on': -0.7415209704374337, 'prices_on': 0.3792032610427596, 'dividends_on': 0.7597968132102717, 'turnover_on': 0.5860555137690139, 'average_turnover_on': 0.6750234817940721, 'rvi_on': -0.758365200271929, 'mcftrr_on': -0.7678090475533608, 'imoex_on': -0.5441796726478852, 'ticker_type_on': -0.09997730965326945, 'usd_on': -0.334036922316718, 'open_on': -0.27818249383845906}
Model: {}
Optimizer: {'betas': 0.9990053414863606, 'eps': 9.991434975458438e-09, 'weight_decay': 0.009998127788332577}
Scheduler: {'max_lr': 0.009314820871952526, 'epochs': 1.0009126725233755, 'pct_start': 0.30099817073495205, 'anneal_strategy': 0.9825538484814915, 'base_momentum': 0.8492757678903675, 'max_momentum': 0.9508768810821049, 'div_factor': 24.975410927815844, 'final_div_factor': 10006.218260928737}
Traceback (most recent call last):
File "path_to\poptimizer\poptimizer_main_.py", line 19, in evolve
ev.evolve(port)
File "path_to\poptimizer\poptimizer\evolve\evolve.py", line 56, in evolve
if _eval_and_print(parent, tickers, end) is None:
File "path_to\poptimizer\poptimizer\evolve\evolve.py", line 98, in _eval_and_print
fitness = organism.evaluate_fitness(tickers, end)
File "path_to\poptimizer\poptimizer\evolve\population.py", line 78, in evaluate_fitness
llh = model.llh
File "path_to\poptimizer\poptimizer\dl\model.py", line 111, in llh
self._llh = self._eval_llh()
File "path_to\poptimizer\poptimizer\dl\model.py", line 145, in _eval_llh
model = self.prepare_model(loader)
File "path_to\poptimizer\poptimizer\dl\model.py", line 123, in prepare_model
self._model = self._train_model()
File "path_to\poptimizer\poptimizer\dl\model.py", line 218, in _train_model
model = self._make_untrained_model(loader)
File "path_to\poptimizer\poptimizer\dl\model.py", line 192, in _make_untrained_model
model = model_type(loader.history_days, loader.features_description, **self._phenotype["model"])
KeyError: 'model'
poptimizer/poptimizer/dl/model.py
Line 192 in 29d9010
Удаление моделей не изменило ситуацию.
В self._phenotype лежит вот что:
{
'type': 'WaveNet',
'data': {
'features': {
'Label': {
'on': True
},
'Ticker': {
'on': True
},
'DayOfYear': {
'on': True
},
'DayOfPeriod': {
'on': False
},
'Prices': {
'on': True
},
'Dividends': {
'on': True
},
'Turnover': {
'on': True
},
'AverageTurnover': {
'on': True
},
'RVI': {
'on': False
},
'MCFTRR': {
'on': False
},
'IMOEX': {
'on': False
},
'TickerType': {
'on': False
},
'USD': {
'on': False
},
'Open': {
'on': False
}
},
'batch_size': 128,
'history_days': 77
},
'optimizer': {
'betas': (0.9,
0.9990053414863606),
'eps': 9.991434975458438e-09,
'weight_decay': 0.009998127788332577
},
'scheduler': {
'max_lr': 0.009314820871952526,
'epochs': 1.0009126725233755,
'pct_start': 0.30099817073495205,
'anneal_strategy': 'cos',
'base_momentum': 0.8492757678903675,
'max_momentum': 0.9508768810821049,
'div_factor': 24.975410927815844,
'final_div_factor': 10006.218260928737
}
}
Доброго времени суток!
Столкнулся с ошибкой при запуске evolve()
Начинается "Загрузка данных по торговым дням" , затем сохранение и ошибка (в приложенном txt файле)
Прочитал в одном из Issue, что данную таблицу можно удалять, насколько раз её удаление не мешало нормальному запуску программы.
Запуская повторно Evolve() запускается и обучает модели, но optimize() валится с ошибкой
***2021-11-12: Шаг эволюции — 265***
LLH - (0.8711, 2.7421, 2.8299)
RET - (0.0533, 0.2857, 0.9786)
Организмов - 109 / Максимум оценок - 9
Доля принятых - 24.38%
Родитель:
LLH — 1.0599: 1.0878, 1.0499, 1.0321, 1.0360, 1.0724, 1.0739, 1.0629, 1.0702, 1.0543
RET — 0.3209: 0.2983, 0.3431, 0.5413, 0.4155, 0.3248, 0.2233, 0.2299, 0.2680, 0.2444
Timer — 0:01:35
Data: {'batch_size': 12538.909770179394, 'history_days': 206.6340404114518, 'ticker_on': -27.851100142615003, 'day_of_year_on': 3.003903146391359, 'day_of_period_on': 0.09298512564162972, 'prices_on': -1.2356011287782966, 'dividends_on': 5.281168692071963, 'turnover_on': 1097.6412588906178, 'average_turnover_on': -137.31812095042886, 'rvi_on': 1.1394448984646637, 'mcftrr_on': -1035.7671194581624, 'imoex_on': 16.451081354972647, 'ticker_type_on': -18.88785381132009, 'usd_on': 5.053386319673658, 'open_on': 18.628071609078365, 'high_on': -1.2838003948298191, 'low_on': 2.6327172181399794, 'meogtrr_on': 0.7555408299677332}
Model: {'start_bn': -6.106530253047609, 'kernels': 7.849879889206337, 'sub_blocks': 32.269481480168096, 'gate_channels': 13.361194742689179, 'residual_channels': 29.013755026884443, 'skip_channels': 4.485923773196008, 'end_channels': 3.6420667744873283, 'mixture_size': 23.769801352687686}
Optimizer: {'betas': 0.9989989792474319, 'eps': 2.69744865890668e-07, 'weight_decay': 0.37832394878427655}
Scheduler: {'max_lr': 0.5123636051185311, 'epochs': 0.034064066207938615, 'pct_start': 0.2992583930527831, 'anneal_strategy': 45.911779177450235, 'base_momentum': 0.7847317605834427, 'max_momentum': 0.949193175241471, 'div_factor': 1131.6321634466094, 'final_div_factor': 72315.14651363067}
Претендент - новый организм:
LLH — -
RET — -
Timer — 0:00:00
Data: {'batch_size': 12536.307453788919, 'history_days': 3768.612053130967, 'ticker_on': -27.8251336605893, 'day_of_year_on': 2.679672921571562, 'day_of_period_on': 0.07587551758791003, 'prices_on': -1.4444338554680354, 'dividends_on': 5.196017516980653, 'turnover_on': 1097.9431317010308, 'average_turnover_on': -141.3313656829068, 'rvi_on': 0.2803152984240963, 'mcftrr_on': -1035.737567150942, 'imoex_on': 16.464444571137232, 'ticker_type_on': -18.918861238789756, 'usd_on': 5.680644606399607, 'open_on': 18.587368984726588, 'high_on': -2.4609938002982825, 'low_on': 2.640818152018767, 'meogtrr_on': 0.8090094396211781}
Model: {'start_bn': -6.124026519955858, 'kernels': 7.779952469628877, 'sub_blocks': 32.31421098039613, 'gate_channels': 13.008014694168885, 'residual_channels': 28.991498992916597, 'skip_channels': 4.3918872007842875, 'end_channels': 3.6027284114052383, 'mixture_size': 23.806811571271666}
Optimizer: {'betas': 0.9989989192911831, 'eps': 2.595997674257352e-07, 'weight_decay': 0.3915807080445315}
Scheduler: {'max_lr': 0.5120581350534901, 'epochs': 0.01341430924179389, 'pct_start': 0.2990725920166547, 'anneal_strategy': 45.39829535425662, 'base_momentum': 0.7845912285044897, 'max_momentum': 0.9492807641264684, 'div_factor': 1150.6293052367987, 'final_div_factor': 77950.93505920934}
INFO:EventBus:DateCheckRequired(timestamp=datetime.datetime(2021, 11, 14, 5, 18, 45, 827478))
num_samples should be a positive integer value, but got num_samples=0
Traceback (most recent call last):
File "PATH_TO/poptimizer/loop.py", line 10, in <module>
evolve()
File "PATH_TO/poptimizer/poptimizer/__main__.py", line 12, in evolve
ev.evolve()
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 58, in evolve
next_, new = self._step(current)
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 130, in _step
if self._eval_organism(prey) is None:
File "PATH_TO/poptimizer/poptimizer/evolve/evolve.py", line 178, in _eval_organism
organism.evaluate_fitness(self._tickers, date)
File "PATH_TO/poptimizer/poptimizer/evolve/population.py", line 117, in evaluate_fitness
llh, ir = model.quality_metrics
File "PATH_TO/poptimizer/poptimizer/dl/model.py", line 106, in quality_metrics
self._llh = self._eval_llh()
File "PATH_TO/poptimizer/poptimizer/dl/model.py", line 140, in _eval_llh
model = self.prepare_model(loader)
File "PATH_TO/poptimizer/poptimizer/dl/model.py", line 118, in prepare_model
self._model = self._train_model()
File "PATH_TO/poptimizer/poptimizer/dl/model.py", line 202, in _train_model
loader = data_loader.DescribedDataLoader(
File "PATH_TO/poptimizer/poptimizer/dl/data_loader.py", line 68, in __init__
super().__init__(
File "PATH_TO/anaconda3/envs/finan/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 268, in __init__
sampler = RandomSampler(dataset, generator=generator)
File "PATH_TO/anaconda3/envs/finan/lib/python3.9/site-packages/torch/utils/data/sampler.py", line 102, in __init__
raise ValueError("num_samples should be a positive integer "
ValueError: num_samples should be a positive integer value, but got num_samples=0
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.