Light

zake7749 / word2vec-tutorial Goto Github PK

View Code? Open in Web Editor NEW

507.0 19.0 167.0 2.91 MB

中文詞向量訓練教學

Home Page: http://zake7749.github.io/2016/08/28/word2vec-with-gensim/

License: MIT License

Python 100.00%

gensim word2vec

word2vec-tutorial's Introduction

使用 gensim 訓練中文詞向量

教學文件

套件需求

jieba

pip3 install jieba

gensim

pip3 install -U gensim

OpenCC (可更換為任何繁簡轉換套件)

訓練流程

1.取得中文維基數據，本次實驗是採用 2016/8/20 的資料。

目前 8 月 20 號的備份已經被汰換掉囉，請前往維基百科:資料庫下載按日期來挑選更新的訓練資料。( 請挑選以pages-articles.xml.bz2為結尾的檔案 )

2.將下載後的維基數據置於與專案同個目錄，再使用wiki_to_txt.py從 xml 中提取出維基文章

python3 wiki_to_txt.py zhwiki-20160820-pages-articles.xml.bz2

若您採用的不是 8 月 20 號的備份，請更換 zhwiki-20160820-pages-articles.xml.bz2 為您採用的備份的檔名。

3.使用 OpenCC 將維基文章統一轉換為繁體中文

opencc -i wiki_texts.txt -o wiki_zh_tw.txt -c s2tw.json

4.使用jieba 對文本斷詞，並去除停用詞

python3 segment.py

5.使用gensim 的 word2vec 模型進行訓練

python3 train.py

6.測試我們訓練出的模型

python3 demo.py

word2vec-tutorial's People

Contributors

Stargazers

Watchers

Forkers

xyz8 davidlanz kenyeung128 jimmy-walker cdyangbo binkmust acc8518 jaric elfscript pengtedshen randy-ran tifoit afcentry nanfengpo wangerniuniu zhangguanghui1 liesenf sophiealex maureenlin arioscen hwade skydream37 jamesyeh2017 noahcse o-github-o yumiao1203 chilunhuang ivanvera kingpeterson kmkmkkball llamaslama xxyou00 zwytop yuanyuanzwj sheephow hanksantford forget095 savourylie youngmihuang woniuhu louisliaoxh1989 bigmolemushroomntu xrick caichun tarogege r06522611 fandongwan jinligen shuochec zhaoshiling1017 hdy1995 yangzilong1986 dchaochao cclljj meganhsu chia313339 aiedward alex821226 nellymilk ghldun pinglunliao georgetsui hyc256 latin1593 awfssv qianqq jackmyth lzbgt kaiyiuw jjwangnlp johnhawkins1991 rnnnnn lucklamb yehuangcn xqk tutty427 1mrliu xuezhizeng louishuang1002 aurora314156 alexis1437 yw1991 suzhidong wuchirat gujingjing sportzhang renjiehan guozhilong wangenbao1001 tusonggao nn-tony lcyuanjiang tianyikenan tianchi03 wuyunxiangwyx edwardxycai cutejaneii yuebansangeng dengziheng jwen-wj

word2vec-tutorial's Issues

有简体中文的字典吗

有简体中文的字典吗

FileNotFoundError: [Errno 2] No such file or directory: 'wiki_zh_tw.txt'

请问没有wiki_zh_tw.txt怎么办呢

请问我能拿word2vec.model自己做一個閒聊的聊天機器人嗎

大神请问我这样traing是对的吗

1各小时才出来0.10%

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.