zimix0 Goto Github PK

antispam

Dead simple bayesian anti-spam classifier written in Python.

cc_net

Tools to download and cleanup Common Crawl data

colossalai

Making large AI models cheaper, faster and more accessible

data-preparation

Code used for sourcing and cleaning the BigScience ROOTS corpus

deduplicate-text-datasets

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

doccano

Open source annotation tool for machine learning practitioners.

fastertransformer

Transformer related optimization, including BERT, GPT

fastertransformer_backend

gitflow_test

git工作流

gpt-neox

An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

inference

Xorbits Inference (Xinference) is a powerful and versatile library designed to serve LLMs, speech recognition models, and multimodal models, even on your laptop. It supports a variety of models compatible with GGML, such as llama, chatglm, baichuan, whisper, vicuna, orac, and many others.

justext

Heuristic based boilerplate removal tool

llama.cpp

Port of Facebook's LLaMA model in C/C++

llava

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLM

megatron-deepspeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

megatron-lm

Ongoing research training transformer language models at scale, including: BERT & GPT-2

mnbvc

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。

nlp-beginner

NLP上手教程

opencompass

OpenCompass is an LLM evaluation platform, supporting evaluation of 20+ models over 50+ datasets, that enables fast, comprehensive benchmarking of large models using efficient distributed evaluation techniques.

pyobjtemplate

python项目模版

redpajama-data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

st-chat

Streamlit Component, for a Chatbot UI

test4circleci

uer-py

Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo