This is the official repository for "SeqXGPT: Sentence-Level AI-Generated Text Detection".

Widely applied large language models (LLMs) can generate human-like content, raising concerns about the abuse of LLMs. Therefore, it is important to build strong AI-generated text (AIGT) detectors. Current works only consider document-level AIGT detection, therefore, we first introduce a sentence-level detection challenge by synthesizing a dataset that contains documents that are polished with LLMs, that is, the documents contain sentences written by humans and sentences modified by LLMs. Then we propose Sequence X (Check) GPT, a novel method that utilizes log probability lists from white-box LLMs as features for sentence-level AIGT detection. These features are composed like waves in speech processing and cannot be studied by LLMs. Therefore, we build SeqXGPT based on convolution and self-attention networks. We test it in both sentence and document-level detection challenges. Experimental results show that previous methods struggle in solving sentence-level AIGT detection, while our method not only significantly surpasses baseline methods in both sentence and document-level detection challenges but also exhibits strong generalization capabilities.

Process of Sentence-Level AIGT Detection Challenge.

SeqXGPT is an advance method for sentence-level AI-generated text detection, , which consists of the following three parts:

1. Perplexity Extraction and Alignment;
2. Feature Encoder;
3. Linear Classification Layer.

SeqXGPT Framework.

Please refer to our paper for more details.

We list all the datasets, models, training and testing codes related to SeqXGPT here.

All the values listed in our table are F1 scores, and Macro-F1 scores to consider the overall performance. For detailed precision and recall scores, please refer to our paper.

SeqXGPT performs much better than two zero-shot methods and supervised method Sent-RoBERTa.

SeqXGPT shows the best performance among these four methods. In contrast, the performance of Sniffer is noticeably inferior, which emphasizes that document-level AIGT detection methods cannot be effectively modified for sentence-level AIGT detection. Interestingly, we find that the performance of both RoBERTa-based methods is slightly inferior to SeqXGPT in overall performance. This suggests that the semantic features of RoBERTa might be helpful to discriminate human-created sentences.

SeqXGPT can accurately discriminate sentences generated by various models and those authored by humans, demonstrating its strength in multi-class detection. It is noteworthy that RoBERTa-based methods perform significantly worse than binary AIGT detection.

sentence-level detection methods can be transformed and directly applied to document-level detection, and the performance is positively correlated with their performance on sentence-level detection. Overall, SeqXGPT exhibits excellent performance in document-level detection.

The great performance of SeqXGPT on OOD data reflects the strong generalization capabilities of SeqXGPT.

In this paper, we first introduce the challenge of sentence-level AIGT detection and propose three tasks based on existing research in AIGT detection. Further, we introduce a strong approach, SeqXGPT, as well as a benchmark to solve this challenge. Through extensive experiments, our proposed SeqXGPT can obtain promising results in both sentence and document-level AIGT detection. On the OOD testset, SeqXGPT also exhibits strong generalization. We hope that SeqXGPT will inspire future research in AIGT detection, and may also provide insightful references for the detection of content generated by models in other fields.

Despite SeqXGPT exhibits excellent performance in both sentence and document-level AIGT detection challenge and exhibits strong generalization, it still present certain limitations:

1. We did not incorporate semantic features, which could potentially assist our model further in the sentence recognition process, particularly in cases involving human-like sentence generation. We leave this exploration for future work.
2. During the construction of GPT-3.5-turbo data, we did not extensively investigate the impact of more diversified instructions. Future research will dive into exploring the influence of instructions on AIGT detection.
3. Due to limitations imposed by the model's context length and generation patterns, our samples only consist of two distinct sources of sentences. In future studies, we aim to explore more complex scenarios where a document contains sentences from multiple sources.

• We sincerely thank Qipeng Guo for his valuable discussions and insightful suggestions.
• This work was supported by the National Natural Science Foundation of China (No. 62236004 and No. 62022027).
• This work's logo was initially generated using OpenAI's DALL-E 3 and further refined by Youqi Sun. Special thanks to both.

If you find SeqXGPT useful for your research and applications, please cite using the Bibtex:

@misc{wang2023seqxgpt,
      title={SeqXGPT: Sentence-Level AI-Generated Text Detection}, 
      author={Pengyu Wang and Linyang Li and Ke Ren and Botian Jiang and Dong Zhang and Xipeng Qiu},
      year={2023},
      eprint={2310.08903},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}