bestlake / sensitive_word_filter Goto Github PK

1. Use the trie tree, based on datrie lib, which is a famous trie library coded in C. We invoke the c lib via cgo.
2. Use the gin framework for network service, gin is a simple golang framework, you could use other network framerork for your convenience.
3. This filter support Chinese word and other complex script as I have done some pretreatment for the words: split all characters map its bytes to the 0~255, such as "换" will be map to 63 62.

post with form-data:

1. http://127.0.0.1:8099/filter/upload

file: upload a text file with keys line by line level: all the keys restrict level in the file, a int value

2. http://127.0.0.1:8099/filter/delete

key: the key you want to delete

3. http://127.0.0.1:8099/filter/search_all

content Return

[
    {
        "head": 0, // The dirty word start position
        "length": 3, // Dirty word length
        "level": 1 // Dirty level
    },
    {
        "head": 4,
        "length": 3,
        "level": 1
    }
]

I have already build the libdatrie and put files need into the project on Mac OSX. If you put the project on Linux, you should build the trie tree library on linux, download from the web and follow its ReadMe: $ LIB_PATH = XXX/lib_path $ tar zxvf libdatrie-0.2.5.tar.gz $ cd libdatrie-0.2.5 $ make clean $ ./configure --prefix=$LIB_PATH $ make $ make install Then, you will see four folder generated in your lib_path, bin include lib share.

1. Using KMP algorithm optimize the content filter operation.
2. Set up a testing data used as benchmark