nicolay-r / arenets Goto Github PK

For NetworkSampleRowProvider there is a need to perform refactoring, such that allows to manually setup the required features in output

This parameter might be utilized to embed self-attention on top of it

• P_Zhou attention
• Z_Yang attention

Copied from AREkit

nicolay-r/AREkit#376

• remove sample_id
• remove opinion_id parsing; use a separated column instead.

• replace with original and complex LSTM by default

#18 related

Here, obviously it is expected to be bw:

The implementation should be frustrated in terms of shift operation. The pipeline is as follows:

1. Calculating result regions of terms
2. Perform everything else using the precalculated regions.

(https://github.com/nicolay-r/AREkit/blob/master/contrib/networks/sample.py, from_tsv_row function)

Reasons: tensorflow 1.14.0 is pretty outdated. The main contribution of this framework is a data preparation rather than model training.

• move dependency into network contrib module
• make it a particular kernel. (Move all the tf dependencies into the related subfolder)
• xw_plus_b
• Adadelta optimizer
• Dropout wrapper

#9 related

Solution:
Use as IDE as follows:

• download links

By default it is 100.

• returing None instead of the [] in some cases lead to the incorrect behavior during sampling. This is caused by subj and obj inds, which are expected to be declared anyway, with non empty its synonyms lists.

• Remove DataFolding
• f639f0d
• Remove BaseDataFolding; use only pre-separated samples. (#7 related)
• Remove duplicated Label instance for test; the latter causes an extra errors.
• Storages: rows modification and blanking.
• get_row might be refactored onto usage of __iter__ for BaseRowsStorage (d5e438c)
• remove model name suffix model (train/infer)

• rewrite code into function
• calling that function

http://vectors.nlpl.eu/repository/

Since we only need a labels count to use it in a form a list of labels [0, c) where c -- amount of labels

• remove labels declaration from tutorial

Proceeds nicolay-r/AREkit#284 as the latter address on the exact same problem.

Such parameters should be removed and provided via dictionary
https://github.com/nicolay-r/AREkit/blob/2f0f69e14afe070ca8f91a507150dd7fe4aade66/arekit/contrib/networks/core/input/providers/sample.py#L14-L19

Pandas utilized only for csv reading. It might be replaced with something simplier instead.

It is crutial to set comments = None instead of # by default!

vocab = np.loadtxt(source, dtype=str, comments=None)

#10 related

• By default, keep all logging information in file
• Keep model details in log as well

from logging.handlers import TimedRotatingFileHandler
h = TimedRotatingFileHandler('/home/user/Desktop/myLogFile.log')

Problem: in most cases there is a need to initialize the following parameters once you encounter with an exception.
Reason: some of the parameters below are None by default, which means it is necessary to initialize them.

This might be wrapped for a better usage:

config = network_config_func()
config.modify_classes_count(value=labels_count)
config.modify_learning_rate(0.01)
config.modify_use_class_weights(True)
config.modify_dropout_keep_prob(0.9)
config.modify_bag_size(1)
config.modify_bags_per_minibatch(1)
config.modify_embedding_dropout_keep_prob(1.0)
config.modify_terms_per_context(50)
config.modify_use_entity_types_in_embedding(False)
config.set_pos_count(PartOfSpeechTypesService.get_mystem_pos_count())

Reason: necessary for further visualization and analysis, including attention-related data especially.
Continues #18

for now we consider the incomplete embedding only in case when model fine-tuned from scratch

• TermsPerContext provide as a custom config callback modifier.
• Batch size -- required for not to exceed the memory limit;
• Learning Rate -- to modify the speed of training;
• model_input_type -- (single, ctx)
• Disable logging hidden states logging on training by default
• Train acc limit move outside
• Support input as the list of terms (#23)
• text_a -> text
• dropout
• remove -0 (index) suffix for input filenames\
• switch to English examples instead of Russian (due to internalization)
• rename rnn to lstm
• rename att-bilstm to att-bilstm-pzhou
• data description tutorial!
• avg_acc and avg_loss

Since this is a crucial parameter of the every task in particular.

#15

reason -- simplify SamplesIO API

Reason: these functions are only applicable for RNN networks:

and

It may results in the following exeption:

  File "/home/nicolay/proj/AREnets/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[23,12] = -1 is not in [0, 302866)
	 [[node embedding_lookup (defined at /arenets/context/architectures/base/base.py:352) ]]

• need to update to tensorflow==2.0+ API