LLM Detective: A Comparative Study of Supervised and Zero-Shot Methods for Detecting Texts Generated by Large Language Models

Abstract

The widespread adoption of large language models (LLMs) has raised concerns about the potential for generating misleading or harmful content, necessitating reliable methods to distinguish between human-written and LLM-generated texts. This study compares supervised and zero-shot approaches for detecting LLM-generated texts across English, Chinese, and multilingual datasets. We fine-tune BERT models for supervised classification and implement the FourierGPT zero-shot method, evaluating both on accuracy, precision, recall, F1 score, and AUROC. Our results demonstrate that supervised methods significantly outperform zero-shot approaches, achieving higher accuracy and AUROC across all datasets, particularly in English and Chinese contexts. However, the zero-shot method offers flexibility in scenarios with limited labeled data, showing competitive precision in Chinese and multilingual settings. These findings underscore the trade-offs between the two approaches and highlight the need for further research into robust, language-agnostic detection methods.

Introduction

Large language models (LLMs) have become integral to various applications, from content generation to conversational agents. However, their widespread use has raised concerns about the potential for generating misleading or harmful content. This has necessitated the development of reliable methods to detect whether a given text was generated by an LLM or written by a human. Existing research has primarily focused on supervised learning approaches, where models are trained on labeled datasets to distinguish between human-written and LLM-generated texts. However, supervised methods require large labeled datasets, which may not always be available, especially for languages other than English. To address this limitation, zero-shot detection methods have been proposed, leveraging the inherent properties of LLMs without requiring labeled training data. In this paper, we conduct a comparative study of supervised and zero-shot methods for detecting LLM-generated texts across English, Chinese, and multilingual datasets, evaluating the performance of fine-tuned BERT models against the FourierGPT zero-shot approach.

Methodology

Datasets

We use three datasets to evaluate our methods:

• English Dataset: Derived from the Ghostbuster dataset, with the 'essay' domain used for training and validation, and the 'wp' domain as an out-of-distribution (OOD) test set.
• Chinese Dataset: Sourced from the Face2 dataset, using the 'news' domain for training and validation, and the 'wiki' domain for OOD testing.
• Multilingual Dataset: Combines English 'essay' and Chinese 'news' domains for training and validation, with English 'wp' and Chinese 'wiki' domains for testing.

Preprocessing

For supervised methods, texts are tokenized using language-specific BERT tokenizers and standardized to a maximum length of 512 tokens. To address class imbalance (LLM-generated texts were approximately six times more frequent than human-written texts), we employed undersampling by randomly selecting a subset of the majority class to match the minority class. For zero-shot methods, we follow the FourierGPT approach, extracting texts from JSONL files, computing negative log-likelihood (NLL) scores with pre-trained models, and applying Fourier transforms to generate spectrum data.

Supervised Learning Approach

We fine-tune BERT models for classification:

• English: BERT-base-uncased
• Chinese: BERT-base-chinese
• Multilingual: BERT-base-multilingual-cased

Training uses the Hugging Face Transformers library with a batch size of 16, 10 epochs, and model selection based on the validation F1 score.

Zero-Shot Detection Approach

The zero-shot method replicates FourierGPT:

1. Compute NLL scores using Mistral-7B (English) and Qwen-7B (Chinese, multilingual).
2. Normalize NLL scores with z-score.
3. Apply Fourier transforms to obtain spectrum data.
4. Classify texts using a heuristic based on the power sum of the first ( k ) frequencies, with ( k ) optimized on validation data.

Evaluation Metrics

Performance is assessed using:

• Accuracy
• Precision
• Recall
• F1 Score
• Area Under the ROC Curve (AUROC)

Results

The table below summarizes the performance of both methods across the datasets.

Metric	English	Chinese	Multilingual
Supervised
Accuracy	0.8539	0.7665	0.5409
Precision	0.8582	0.6976	0.5215
Recall	0.8475	0.9410	0.9871
F1 Score	0.8528	0.8012	0.6825
AUROC	0.9297	0.8715	0.5509
Zero-Shot
Accuracy	0.5317	0.5559	0.5443
Precision	0.5434	0.7182	0.6916
Recall	0.3971	0.1840	0.1598
F1 Score	0.4589	0.2929	0.2596
AUROC	0.5440	0.6502	0.6055

Supervised methods consistently outperformed zero-shot approaches. For English, the supervised model achieved an accuracy of 0.8539 and an AUROC of 0.9297. For Chinese, it recorded an accuracy of 0.7665 and a high recall of 0.9410. The multilingual dataset showed an accuracy of 0.5409 but a remarkable recall of 0.9871, though with lower precision (0.5215). The zero-shot method struggled with lower recall (e.g., 0.1598 for multilingual) but showed competitive precision in Chinese (0.7182) and multilingual (0.6916) settings.

Future Work

Future research could explore alternative zero-shot techniques (e.g., linguistic features or stylometry) to improve performance with limited labeled data. Enhancing the multilingual model's generalization through cross-lingual transfer learning or language-agnostic features is another avenue. Additionally, advanced preprocessing strategies like data augmentation and extending the study to other languages and domains could further refine and generalize the findings.

Visualizations

For detailed visualizations, including dataset distributions and performance metrics, refer to the following directories:

• supervised/viz/train
• supervised/viz/test
• supervised/viz/dataset