yumoxu / stocknet-code Goto Github PK

Hi, yumoxu.

great work about your paper, but i tried to reproduce the results you mentioned in your paper.

only 0.53 i can get.

Is there any tricks i should know to get 0.58?

Thanks

Hi,

I set up data, and run with default config,
but below err happened,
could you please help me how to fix it ?
Thanks !

F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
F:\Anaconda3\envs\py36\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2023-04-22 10:11:33,037 INFO INIT: #stock: 88, #vocab+1: 29867
WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:44: calling Constant.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
2023-04-22 10:11:33,089 INFO Start graph assembling...
WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:110: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:128: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:136: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:137: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:155: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob.
WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:192: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:197: GRUCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.GRUCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py:171: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell)), which is equivalent to this API
WARNING:tensorflow:From F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\rnn.py:464: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.RNN(cell), which is equivalent to this API
WARNING:tensorflow:From F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\rnn_cell_impl.py:574: calling Zeros.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method GRUCell.call of <tensorflow.python.ops.rnn_cell_impl.GRUCell object at 0x000001E7F239EEB8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method GRUCell.call of <tensorflow.python.ops.rnn_cell_impl.GRUCell object at 0x000001E7F239EEB8>>: AttributeError: module 'gast' has no attribute 'Str'
WARNING:tensorflow:From F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\rnn.py:244: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:Entity <bound method GRUCell.call of <tensorflow.python.ops.rnn_cell_impl.GRUCell object at 0x000001E7F23CCEF0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method GRUCell.call of <tensorflow.python.ops.rnn_cell_impl.GRUCell object at 0x000001E7F23CCEF0>>: AttributeError: module 'gast' has no attribute 'Str'
Traceback (most recent call last):
File "Main.py", line 8, in
model.assemble_graph()
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py", line 697, in assemble_graph
self._build_mie()
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py", line 253, in _build_mie
self._create_msg_embed_layer()
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py", line 210, in _create_msg_embed_layer
msg_embed = tf.reshape(_for_one_batch(), shape=msg_embed_shape)
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py", line 189, in _for_one_batch
iter_arg=self.mel_in, iter_arg2=self.ss_index_ph, iter_arg3=self.n_words_ph)
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\neural.py", line 39, in iter
_, ta_final = tf.while_loop(cond, body, loop_init)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3501, in while_loop
return_same_structure)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3012, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2937, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\neural.py", line 36, in
body = lambda i, ta: (i + 1, ta.write(i, func(iter_arg[i], iter_arg2[i], iter_arg3[i], **kwargs)))
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py", line 185, in _for_one_sample
iter_arg=sample, iter_arg2=sample_ss_index, iter_arg3=sample_mask)
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\neural.py", line 39, in iter
_, ta_final = tf.while_loop(cond, body, loop_init)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3501, in while_loop
return_same_structure)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3012, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2937, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\neural.py", line 36, in
body = lambda i, ta: (i + 1, ta.write(i, func(iter_arg[i], iter_arg2[i], iter_arg3[i], **kwargs)))
File "E:\CSDN\毕业论文\stocknet-code-master\stocknet-code-master\src\Model.py", line 175, in _for_one_trading_day
msg_ids = tf.constant(range(0, self.max_n_msgs), dtype=tf.int32, shape=[self.max_n_msgs, 1]) # [0, 1, 2, ...]
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\constant_op.py", line 180, in constant_v1
allow_broadcast=False)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\constant_op.py", line 284, in _constant_impl
allow_broadcast=allow_broadcast))
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 466, in make_tensor_proto
_AssertCompatible(values, dtype)
File "F:\Anaconda3\envs\py36\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 371, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).name))
TypeError: Expected int32, got range(0, 30) of type 'range' instead.

Hi,

I set up data, and run with default config,
but below err happened,
could you please help me how to fix it ?
Thanks !

Instructions for updating:
Use standard file APIs to delete files with this prefix.
2019-06-01 14:08:43,037 INFO Eval, eval loss: 1.6962803602218628, acc: 0.518293
2019-06-01 14:10:38,515 INFO iter: 280, batch loss: 1.7300004959106445, batch acc: 0.531250
2019-06-01 14:12:10,435 INFO Eval, eval loss: 1.60964834690094, acc: 0.507622
Traceback (most recent call last):
File "/Users/liran/AI/stocknet-code/src/DataPipe.py", line 396, in batch_gen
sample_dict = next(generators[gen_id])
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/liran/AI/stocknet-code/src/DataPipe.py", line 418, in batch_gen
raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "Main.py", line 15, in
exe.train_and_dev()
File "/Users/liran/AI/stocknet-code/src/Executor.py", line 148, in train_and_dev
for train_batch_dict in train_batch_gen:
RuntimeError: generator raised StopIteration

My name is Luis, I'm a big-data machine-learning developer, I'm a fan of your work, and I usually check your updates.

I was afraid that my savings would be eaten by inflation. I have created a powerful tool that based on past technical patterns (volatility, moving averages, statistics, trends, candlesticks, support and resistance, stock index indicators).
All the ones you know (RSI, MACD, STOCH, Bolinger Bands, SMA, DEMARK, Japanese candlesticks, ichimoku, fibonacci, williansR, balance of power, murrey math, etc) and more than 200 others.

The tool creates prediction models of correct trading points (buy signal and sell signal, every stock is good traded in time and direction).
For this I have used big data tools like pandas python, stock market libraries like: tablib, TAcharts ,pandas_ta... For data collection and calculation.
And powerful machine-learning libraries such as: Sklearn.RandomForest , Sklearn.GradientBoosting, XGBoost, Google TensorFlow and Google TensorFlow LSTM.

With the models trained with the selection of the best technical indicators, the tool is able to predict trading points (where to buy, where to sell) and send real-time alerts to Telegram or Mail. The points are calculated based on the learning of the correct trading points of the last 2 years (including the change to bear market after the rate hike).

I think it could be useful to you, to improve, I would like to share it with you, and if you are interested in improving and collaborating I am also willing, and if not file it in the box.

I think there is a risk that model see the true main target value of y when it calculates y_T, which is a predicted value.

In the function _create_vmd_with_zh_rec at Model.py, y_ values from self.y_ph (placeholder for y) are used for calculating h_z_post_t, z_post_t, and z_post_t_pdf in the tf.while loop.
Since its condition "cond = lambda t, *args: t < self.max_n_days", model use every y_ values. These values are needed to get self.g_T and self.y_T (the prediction value of the main target y).

y_ = tf.transpose(self.y_ph, [1, 0, 2])  # max_n_days * batch_size * y_size
        def _loop_body(t, ta_z_prior, ta_z_post, ta_kl):
            """
            iter body. iter over trading days.
            """
            with tf.variable_scope('iter_body', reuse=tf.AUTO_REUSE):
                ......
                with tf.variable_scope('h_z_post'):
                    h_z_post_t = self._linear([x[t], h_s[t], y_[t], z_post_t_1], self.z_size, 'tanh')
                with tf.variable_scope('z_post'):
                    z_post_t, z_post_t_pdf = self._z(h_z_post_t, is_prior=False)

                kl_t = ds.kl_divergence(z_post_t_pdf, z_prior_t_pdf)  # batch_size * z_size

                ta_z_prior = ta_z_prior.write(t, z_prior_t)  # write: batch_size * z_size
                ta_z_post = ta_z_post.write(t, z_post_t)  # write: batch_size * z_size
                ta_kl = ta_kl.write(t, kl_t)  # write: batch_size * 1

                return t + 1, ta_z_prior, ta_z_post, ta_kl

The problem is, it looks like that DataPipe._get_prices_and_ts( ) is the method to bring “y_batch” values to batch_dict (input batch), and the _get_prices_and_ts function put a y value of main target date into the "ys" list at the last position.

with io.open(stock_movement_path, 'r', encoding='utf8') as movement_f:
        for line in movement_f:  # descend
            data = line.split('\t')
            t = datetime.strptime(data[0], '%Y-%m-%d').date()
            # logger.info(t)
            if t == main_target_date:
                # logger.info(t)
                ts.append(t)
                ys.append(_get_y(data))
                main_mv_percent = data[1]
                if -0.005 <= float(main_mv_percent) < 0.0055:  # discard sample with low movement percent
                    return None

It means the true value y of main target date can be used for predicting self.y_T, its prediction value, so model can cheat. I have tried to find any masking concept for this y value in Model.py, Executor.py, and DataPipe.py, but I could not find.

In fact, I have modified stocknet-code to apply Korean stock market. I refactored Executor.py and DataPipe.py to suit my purpose (I do not touch Model.py), and I am quite sure that my new codes follow your intention well (I checked my refactored code twice and compared batch results from my code to the results from yours). However, I found the model got 100% accuracy in the training phase, it means the model cheats. But it looks like stocknet codes run well with your own codes and data.

If you had handled this issue in advance, would you tell me how you handled it? It would be helpful for me.

Thank you for reading.

About generating g:

I am really confused about whether use posterior z or prior z. Can you help to explain?

"""

def _get_unaligned_corpora(self, ss, main_target_date, vocab_id_dict):
    def get_ss_index(word_seq, ss):
        ss = ss.lower()
        ss_index = len(word_seq) - 1  # init
        if ss in word_seq:
            ss_index = word_seq.index(ss)
        else:
            if '$' in word_seq:
                dollar_index = word_seq.index('$')
                if dollar_index is not len(word_seq) - 1 and ss in word_seq[dollar_index + 1]:
                    ss_index = dollar_index + 1
                else:
                    for index in range(dollar_index + 1, len(word_seq)):
                        if ss in word_seq[index]:
                            ss_index = index
                            break
        return ss_index

"""
This function is from def _get_unaligned_corpora in DataPipe.py file
I am not sure what we are doing if stock symbol ss is not in the word_Seq?

Hi, how can I obtain the vocab.txt file? Could you please guide me on how to get it?