Code Monkey home page Code Monkey logo

sbert-chineseexample's Introduction


Sbert-ChineseExample

Sentence-Transformers Information Retrieval example on Chinese


Table of Contents

About The Project

Sentence Transformers is a multilingual sentence embedding generate framework, which provides an easy method to compute dense vector representations for sentences and paragraphs (based on HuggingFace Transformers)

This repository target at ms_macro like task on a Chinese dataset, train bi_encoder and cross_encoder, with the help of elasticsearch easy interface on pandas to build serlizable conclusion.

Built With

Getting Started

Installation

  • pip
pip install -r requirements.txt
  • install Elasticsearch and start service

Usage

1. Download Data from google drive

2. bi_encoder data prepare

3. train bi_encoder

4. cross_encoder train data prepare

5. cross_encoder valid data prepare

6. train cross_encoder

7. show bi_encoder cross_encoder inference

Roadmap


* 1 This repository use edited es-pandas interface (support vector serlized) to have a simple manipulate on elasticsearch by pandas.
* 2 try_sbert_neg_sampler.py sample hard negative samples drived from class provide by https://guzpenha.github.io/transformer_rankers/ can also use elastic search to generate hard samples , relate functions have defined in valid_cross_encoder_on_bi_encoder.py
* 3 Before training your dataset on cross_encoder, should take a look at the semantic similarity between different questions. Combine some samples with similar semantic may give help.
* 4 Add some toolkit to Sbert to support multi-class-evaluation (as dictionary) ## Contributing

License

Distributed under the MIT License. See LICENSE for more information.

Contact

svjack - [email protected]

Project Link: https://github.com/svjack/Sbert-ChineseExample

Acknowledgements

sbert-chineseexample's People

Contributors

svjack avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.