ChatGPT Tokenizer

This is a Go program that performs natural language processing tasks using the OpenAI API, including sentiment analysis, word frequency counting, part-of-speech tagging, and translation. It receives input text through an HTTP endpoint /api/tokenizer and returns the processing results in specified JSON format.

Configuration

The program requires a valid OpenAI API key to work correctly. Please store your key in the environment variable.

Linux/macOS:

export OPENAI_API_KEY=your_api_key_here

Windows:

setx OPENAI_API_KEY "your_api_key_here"

Docker Deployment

Build

Clone or download this project to your local.

git clone https://github.com/gopkg-dev/chatgpt-tokenizer.git

Run the following command in the project root directory to build a Docker image:
```
docker build -t chatgpt-tokenizer .
```

Run

With `docker run`

Run the following command to start a container:
```
docker run -p 3000:3000 -e OPENAI_API_KEY=your_api_key_here chatgpt-tokenizer
```
Replace your_api_key_here with your OpenAI API key.

With `docker-compose`

Save the following content to a file named docker-compose.yml:

version: '3'
services:
  chatgpt-tokenizer:
    image: chatgpt-tokenizer
    ports:
      - "3000:3000"
    environment:
      OPENAI_API_KEY: your_api_key_here

Run the following command to start a container:
```
docker-compose up -d
```
Replace your_api_key_here with your OpenAI API key.

Local Deployment (non-Docker)

Ensure you have installed Go 1.16 or later.
Clone or download this project to your local.
Run the following command in the project root directory to build and run the program:
```
go build -o chatgpt-tokenizer .
./chatgpt-tokenizer
```

Usage

Sending Requests

Send a POST request to the /api/tokenizer endpoint with the input_text parameter, which contains the text content to be processed.

Response Format

The program returns the processing results in the following JSON format:

{
    "text":"The original text",
    "translate_cn":"Translated text",
    "language":"English",
    "words":[
        {
            "word":"Word1",
            "count":1,
            "translation_cn":"Translation1"
        },
        {
            "word":"Word2",
            "count":2,
            "translation_cn":"Translation2"
        }
    ],
    "sentiment":"Sentiment analysis result",
    "aspects":[
        {
            "aspect":"Aspect1",
            "polarity":"Polarity1",
            "translation_cn":"Translation1 of Aspect1"
        },
        {
            "aspect":"Aspect2",
            "polarity":"Polarity2",
            "translation_cn":"Translation of Aspect2"
        }
    ]
}

Where:

text: The original input text.
translate_cn: The translated text of the original text, if any translation is performed. Otherwise, it is an empty string.
language: The language type of the original text.
words: An array that includes detailed information about each word, including its count, translation, etc.
sentiment: The sentiment analysis result of this text.
aspects: An array that includes detailed information about aspects, polarity, translations, etc. related to the text.

gopkg-dev / chatgpt-tokenizer Goto Github PK