CommentScore

CommentScore is a multi-platform sentiment analysis application for evaluating community feedback from YouTube, GitHub, and Hacker News. It collects public comments and discussions, classifies their sentiment, and turns the results into practical signals such as tutorial usefulness, repository friction, and product reception.

The project combines external platform APIs, text preprocessing, locally stored machine-learning artifacts, and a Streamlit interface. It supports a fast local inference path for interactive analysis and a full transformer-based path through a configured remote inference API.

Abstract

Online comments contain useful evidence about whether a tutorial is understandable, whether an open-source project is creating user friction, and how a technical product is being received. CommentScore provides a common analysis workflow across three different discussion platforms:

1. Retrieve public content from YouTube, GitHub, or Hacker News.
2. Normalize comment text for the selected inference engine.
3. Classify comments as POSITIVE, NEUTRAL, or NEGATIVE.
4. Aggregate the classifications into platform-specific summaries.
5. Present sentiment distributions, representative comments, and diagnostic signals in Streamlit.

Core Use Cases

Platform	Analysis goal	Main output
YouTube	Assess tutorial quality and audience friction	Usefulness verdict, sentiment ratio, timestamps, and diagnostic badges
GitHub	Prioritize open issues by community frustration	Issue-level sentiment counts and positive or negative feedback
Hacker News	Measure discussion reception around technical topics	Story-level sentiment, praise, skepticism, points, and discussion links

System Architecture

Platform APIs
	├─ YouTube Data API
	├─ GitHub REST API
	└─ Hacker News Algolia API
			↓
Data Collection Layer
	├─ Video search and metadata
	├─ Video comments
	├─ Open GitHub issues and comments
	└─ Hacker News stories and nested comments
			↓
Text Processing Layer
	├─ Lowercasing
	├─ URL removal
	├─ Non-letter filtering
	├─ Stopword removal
	└─ WordNet lemmatization
			↓
Inference Layer
	├─ Fast: TF-IDF + SGD classifier
	└─ Full: remote DistilBERT inference API
			↓
Aggregation and Presentation
	├─ Sentiment counts and proportions
	├─ Platform-specific ranking
	├─ Representative comments
	└─ Streamlit dashboards

Analysis Modes

Fast Analysis

Fast mode is designed for interactive exploration. It preprocesses comments with NLTK, transforms them using the bundled TF-IDF vectorizer, and predicts labels with the bundled SGD classifier.

The local model artifacts are stored in:

models/sgd/tfidf_vectorizer.pkl
models/sgd/SGDClassifier_model.pkl

The fast classifier maps model outputs as follows:

0 → NEGATIVE
1 → NEUTRAL
2 → POSITIVE

Full Analysis

Full mode sends the original comments to the endpoint configured through MODAL_API_URL. The endpoint is expected to accept a JSON payload in this form:

{
  "comments": ["First comment", "Second comment"]
}

It should return a JSON object containing a results list of sentiment labels. The repository also includes a DistilBERT model directory under models/distilbert, while the current application uses the configured remote API for full-mode inference.

Platform Workflows

YouTube

The YouTube page searches for videos, retrieves video metadata and comments, classifies the comments, and ranks the results. It provides:

• Sentiment counts for each video
• A usefulness verdict: Useful, Partially Useful, Not Useful, or Controversial
• Praise and friction examples
• Possible confusion timestamps extracted from negative comments
• Diagnostic badges for outdated content, rushed explanations, or broken code
• Sorting by sentiment rank, praise ratio, or video duration

YouTube searches use the YouTube Data API and require an API key.

GitHub

The GitHub page accepts a repository path such as streamlit/streamlit. It retrieves open issues, skips pull requests, fetches issue comments, and scores the available discussions.

Results include:

• Number of issues analyzed
• Number of comments scored
• High-friction issue count
• Sentiment distribution for each issue
• A representative positive comment
• A representative negative comment

The optional GITHUB_TOKEN environment variable can be used for authenticated GitHub API requests.

Hacker News

The Hacker News page searches stories through the Algolia Hacker News API and recursively retrieves nested comments for each matching story.

Results include:

• Stories evaluated
• Comments scored
• Highest story point count
• Sentiment distribution per story
• Community praise and skepticism examples
• Links to the original story and its Hacker News discussion

Project Structure

CommentScore2/
├── app/
│   ├── main.py                 # Streamlit landing page
│   └── pages/
│       ├── github.py           # GitHub issue analysis
│       ├── hackernews.py       # Hacker News discussion analysis
│       └── youtube.py          # YouTube tutorial analysis
├── models/
│   ├── distilbert/             # DistilBERT configuration and tokenizer files
│   └── sgd/                    # Local TF-IDF vectorizer and SGD model
├── src/commentscore/
│   ├── data/
│   │   ├── github_fetch.py     # GitHub API integration
│   │   ├── hn_fetch.py         # Hacker News API integration
│   │   ├── yt_fetch.py         # YouTube comment retrieval
│   │   └── yt_search.py        # YouTube search and metadata retrieval
│   ├── features/
│   │   └── preprocess.py       # Comment cleaning and normalization
│   ├── models/
│   │   ├── aggregate.py        # Video-level sentiment aggregation
│   │   ├── classify.py         # Fast and full inference
│   │   └── evaluate.py         # Classification metrics and confusion matrix
│   └── visualization/
│       └── visualize.py        # Sentiment charts and word clouds
├── Dataset/
│   └── youtube-comments-sentiment.csv
├── pyproject.toml
├── requirements.txt
└── README.md

Installation

The project requires Python 3.11.

Using a virtual environment

git clone <repository-url>
cd CommentScore2

python -m venv .venv

Activate the environment on Windows:

.\.venv\Scripts\Activate.ps1

Install the project dependencies:

python -m pip install --upgrade pip
pip install -r requirements.txt

Alternatively, install the package from the project metadata:

pip install -e .

Configuration

Create a .env file in the project root. Add the credentials and endpoint required by the platform workflows you plan to use:

YOUTUBE_API_KEY=your_youtube_data_api_key
GITHUB_TOKEN=your_github_token
MODAL_API_URL=https://your-inference-endpoint.example.com

Configuration details:

• YOUTUBE_API_KEY is required for YouTube search and comment retrieval.
• GITHUB_TOKEN is optional, but helps GitHub API requests operate with authenticated rate limits.
• MODAL_API_URL is required for Full analysis mode.

Do not commit .env or API credentials to source control.

Running the Application

Start the Streamlit application from the project root:

streamlit run app/main.py

Open the local URL shown by Streamlit, usually:

http://localhost:8501

Choose a platform from the home page, select the inference engine in the sidebar, enter a query or repository path, and run the analysis.

Example Inputs

YouTube topic:

Docker for beginners

GitHub repository:

streamlit/streamlit

Hacker News topic:

Show HN: FastAPI

Text Preprocessing

Fast mode applies the following transformations before classification:

1. Convert text to lowercase.
2. Remove HTTP and WWW URLs.
3. Remove characters other than letters and whitespace.
4. Tokenize on whitespace.
5. Remove English stopwords while preserving not for sentiment context.
6. Lemmatize tokens with WordNet.

Full mode sends the original comment text to the remote inference endpoint, allowing the endpoint to apply its own model-specific processing.

Aggregation Logic

For YouTube videos, the application converts comment-level labels into a usefulness verdict:

• Useful when the positive proportion is greater than $1.25$ times the negative proportion.
• Not Useful when more than half of the comments are negative.
• Partially Useful otherwise.
• Controversial when positive and negative comments are sufficiently balanced and their combined count exceeds five.

GitHub and Hacker News use the same comment-level labels but present platform-specific summaries rather than a single usefulness verdict.

Evaluation and Visualization Utilities

The source package also contains reusable utilities for offline analysis:

• evaluate_comments prints accuracy, a classification report, and a confusion matrix.
• plot_distribution creates an ordered sentiment count plot.
• generate_wordcloud creates a word cloud for a group of comments.

These utilities are separate from the Streamlit pages and can be used in experiments or evaluation scripts.

Data and External Services

The application depends on live external services:

• YouTube Data API v3 for video search, metadata, and comments
• GitHub REST API for open issues and issue comments
• Hacker News Algolia API for story search and comment trees
• A configured remote inference service for Full analysis mode

API responses can be incomplete. Videos may have comments disabled, repositories may have no commented issues, and external services may return errors or rate-limit requests. The fetchers generally return empty results when a request fails so the dashboard can continue rendering.

Limitations

• Sentiment quality depends on the training data and the selected inference engine.
• The local fast model requires the bundled model artifacts to remain available under models/sgd.
• Full analysis depends on the availability and response format of MODAL_API_URL.
• Platform APIs impose quotas, authentication requirements, and rate limits.
• Only a limited number of results and comments are retrieved per request for responsiveness.
• Sentiment is an opinion signal, not a definitive measure of tutorial quality, issue severity, or product success.
• Timestamp extraction and diagnostic badges use simple pattern matching and may miss context.

Future Work

Potential extensions include:

• Add pagination and retry handling for all external APIs.
• Move shared dashboard styling into reusable Streamlit components.
• Add confidence scores and calibrated sentiment probabilities.
• Store historical analyses to track sentiment over time.
• Add multilingual preprocessing and multilingual models.
• Expand evaluation with platform-specific validation sets.
• Add richer visualizations, including distributions and word clouds in the dashboard.
• Improve issue and discussion ranking with engagement and recency signals.

License

No license file is currently included in the repository. Add an appropriate license before distributing the project publicly.

Author

Shivam Kumar