biangbiang

This package provides methods for Chinese language analysis and exploration in Node.js. In particular, it provides three broad functions:

Dictionary definitions of words
Retrieval and calculation of character and word frequency statistics
Hierarchical decomposition of characters into components

Installation

For npm:

npm install biangbiang

For Yarn:

yarn add biangbiang

Getting started

With import:

import biangbiang from "biangbiang";

With require:

var biangbiang = require('biangbiang');

Methods

Dictionary

`define(word, dictionary)`

Get the pinyin and definition of a word, where dictionary is "simplified", "traditional", or "merged". Also returns the frequency index (rank).

define('面条', 'simplified');

{
    simplified: '面条',
    traditional: '麵條',
    pinyin: 'mian4 tiao2',
    definition: 'noodles',
    index: 6029
}

`kind(character)`

Check if a character is a traditional or simplified one. If so, returns the other form. type is 1 for simplified, 2 for traditional, and 3 for both.

kind("面");

{ type: 1, other: '麵'}

`wordsContaining(character)`

Get a list of all dictionary words containing a character, sorted in order of decreasing frequency.

wordsContaining('面');

[
	{
		word: '面',
		index: 322,
	},
	{
		word: '里面',
		index: 706,
	},
	{
		word: '面对',
		index: 930,
	},
	{
		word: '外面',
		index: 1234,
	},
	{
		word: '后面',
		index: 1270,
	},
  ...
]

Frequency

`characterFrequency(character)`

Get frequency statistics for a character.

characterFrequency('面');

{
	symbol: '面',
	index: 211,
	frequency: 1631866,
	percentage: 0.0006532897206780486,
	cumulativePercentage: 0.7101332080329651,
}

`wordFrequency(word)`

Get frequency statistics for a word.

wordFrequency('面条');

{
	symbol: '面条',
	index: 6029,
	frequency: 66879,
	percentage: 0.000015823013308250793,
	cumulativePercentage: 0.8864603725508198,
}

`multiFrequency(sentence)`

Get frequency statistics for a body of text.

multiFrequency('我喜欢吃面条。')；

{
	byCharacter: [
		{
			symbol: '我',
			index: 1,
			frequency: 107133693,
			percentage: 0.042889146765223256,
			cumulativePercentage: 0.12608816399204145,
		},
		{
			symbol: '喜',
			index: 479,
			frequency: 681772,
			percentage: 0.0002729357921827617,
			cumulativePercentage: 0.8216732504061582,
		},
		{
			symbol: '欢',
			index: 1490,
			frequency: 140530,
			percentage: 0.000056258788679270345,
			cumulativePercentage: 0.9496496712024702,
		},
		{
			symbol: '吃',
			index: 42,
			frequency: 9348265,
			percentage: 0.0037424184526636244,
			cumulativePercentage: 0.46991986609112824,
		},
		{
			symbol: '面',
			index: 211,
			frequency: 1631866,
			percentage: 0.0006532897206780486,
			cumulativePercentage: 0.7101332080329651,
		},
		{
			symbol: '条',
			index: 169,
			frequency: 2102653,
			percentage: 0.0008417612665824651,
			cumulativePercentage: 0.6785621013285376,
		},
		{
			symbol: '。',
			index: -1,
			frequency: -1,
			percentage: -1,
			cumulativePercentage: -1,
		},
	],
	indices: [1, 479, 1490, 42, 211, 169],
	percentages: [
		0.042889146765223256,
		0.0002729357921827617,
		0.000056258788679270345,
		0.0037424184526636244,
		0.0006532897206780486,
		0.0008417612665824651,
	],
	cumulativePercentages: [
		0.12608816399204145,
		0.8216732504061582,
		0.9496496712024702,
		0.46991986609112824,
		0.7101332080329651,
		0.6785621013285376,
	],
}

Components

`decompose(character, depth)`

Decompose a character into its components up to a specified depth. If depth is undefined, then the full component tree is returned.

decompose('面')；

{
	丆: {
		'㇐': '㇐',
		'㇓': '㇓',
	},
	囬: {
		'55103': {
			'10001': {
				'10001': '㇑',
			},
			二: {
				二: '㇐',
			},
		},
		囗: {
			'⺆': {
				'㇑': '㇑',
				'㇆': '㇆',
			},
			'㇐': '㇐',
		},
	},
}

`charactersWithComponent(component)`

Get a list of characters containing a component, sorted in order of decreasing frequency.

charactersWithComponent('囗')；

[
	{ character: '回', index: 139 },
	{ character: '图', index: 166 },
	{ character: '口', index: 307 },
	{ character: '因', index: 381 },
	{ character: '西', index: 382 },
	{ character: '团', index: 388 },
	{ character: '困', index: 413 },
	{ character: '国', index: 544 },
	{ character: '围', index: 644 },
	{ character: '圈', index: 717 },
  ...
]

How it works

JSON files containing character/word/component information are generated by /src/prepare.js from raw files contained in /data/raw, with outputs saved to /data/processed.

The preparation script can also be run with npm run prepare or yarn prepare.

Sources

Dictionary entries are entirely from CEDICT
Frequency statistics are from BCC_LEX_Zh
Character composition entries are from CJK-decomp

This project was inspired by HanziJS and offers many of the same functionalities.

Etymology

Biangbiang noodles are a common cuisine in China's Shaanxi province. The character for 'biáng' is one of the most complicated in modern usage. Ironically, is not (yet) included in any of our datasets, as the character was only added to Unicode in March of 2020.

kevinhu / biangbiang Goto Github PK

biangbiang's Introduction

biangbiang

Installation

Getting started

Methods

Dictionary

define(word, dictionary)

kind(character)

wordsContaining(character)

Frequency

characterFrequency(character)

wordFrequency(word)

multiFrequency(sentence)

Components

decompose(character, depth)

charactersWithComponent(component)

How it works

Sources

Etymology

biangbiang's People

Contributors

Stargazers

Watchers

Forkers

biangbiang's Issues

Recommend Projects

Recommend Topics

Recommend Org