Author: Han Xiao https://hanxiao.github.io

A collection of frequently-used deep learning blocks I have implemented in Tensorflow. It covers the core tasks in NLP such as embedding, encoding, matching and pooling. All implementations follow a modularized design pattern which I called the "block-design". More details can be found in my blog post.

• Requirements
• Contents
  • encode_blocks.py
  • match_blocks.py
  • pool_blocks.py
  • embed_blocks.py
  • mulitask_blocks.py
  • nn.py
• Run

• Python >= 3.6
• Tensorflow >= 1.6

A collection of sequence encoding blocks. Input is a sequence with shape of [B, L, D], output is another sequence in [B, L, D'], where B is batch size, L is the length of the sequence and D and D' are the dimensions.

A collection of sequence matching blocks, aka. attention. Input are two sequnces: context in the shape of [B, L_c, D], and query in the shape of [B, L_q, D]. The output is a sequence has the same length as context, i.e. with shape of [B, L_c, D]. Each position in the output should encodes the relevance of that position in context to the complete query.

A collection of pooling blocks. It fuses/reduces on the time axis L. Input is a sequence with shape of [B, L, D], output is in [B, D].

There are also some convolution-based pooling blocks build on SWEM_pool, but they are for experimental purpose. Thus, I will not list them here.

A collection of positional encoding on the sequence.

A collection of multi-task learning blocks. So far only the "cross-stitch block" is available.

A collection of auxiliary functions, e.g. masking, normalizing, slicing.

Run app.py for a simple test on toy data.