An end-to-end data science project that scrapes live game data from Steam250, cleans it, subjects it to a full regression and inference pipeline with model diagnostics, and exposes both the data and the statistical results through a multi-tab interactive Shiny dashboard.
Course: MTH208 – Data Science Lab I | Institute: IIT Kanpur Team: Mainak Sarkar · Neelaksha Malik · Tenzin Tsomu · Harsh Deep
- What this project does
- Research questions
- Project Structure
- Tech Stack
- Getting Started
- Data Dictionary
- Statistical Methods
- Key Findings
- Limitations
- Team
web scraping → data cleaning → exploratory analysis → statistical modelling
→ diagnostics → careful interpretation → interactive visualisation
The Shiny app has four tabs:
| Tab | What it does |
|---|---|
| Overview | Summary stats (total games, avg score, avg votes) + score box plot |
| Search & Filter | Filter by price, votes, release date, rating, title; scatter of price vs votes |
| Top Games | Adjustable top-N leaderboard, switchable between Score and Price |
| Statistical Analysis | Distributions, correlation with both measures and CIs, all fitted regression models with HC3 standard errors, model diagnostics with written verdicts, the Free-to-Play comparison, and an interactive selection-bias simulation |
- How is popularity (review volume) associated with game age, price and player rating?
- Which observable variables are associated with the Steam250 Score?
- Is there evidence of a non-linear relationship between price and popularity?
- Do Free-to-Play and Paid games differ in score, rating or popularity?
Game-Scraping-with-R/
│
├── Data/
│ ├── Submission_Cleaning.R # Web scraper — hits steam250.com, writes the CSV
│ └── game_dataset.csv # Scraped dataset (250 games × 7 columns)
│
├── Analysis/ # ← the statistical layer
│ ├── R/
│ │ ├── 00_helpers.R # HC3 SEs, VIF, Breusch-Pagan, effect sizes, correlation CIs
│ │ ├── 01_data_preparation.R # canonical cleaning + audit + outlier screening
│ │ ├── 02_exploratory_correlation.R
│ │ ├── 03_diagnostics.R # one diagnostic battery for every model
│ │ ├── 04_age_popularity.R # Q1
│ │ ├── 05_score_model.R # Q2
│ │ ├── 06_price_popularity.R # Q3
│ │ ├── 07_free_vs_paid.R # Q4
│ │ └── 08_selection_bias.R # range restriction + selection simulation
│ ├── run_all.R # reproduces everything
│ ├── output/ # tables, figures, results.rds, analysis_log.txt
│ └── README.md
│
├── ShinyApp/
│ └── app.R # dashboard, sources Analysis/R/
│
├── Report/
│ ├── GameScape_report.Rmd # full report, sources Analysis/R/
│ └── render_report.R # builds HTML, or Markdown if pandoc is absent
│
├── Presentation/
└── README.md
Cleaning lives in exactly one place. The app and the report both call
gamescape_prepare(), so they cannot disagree.
| Library | Purpose |
|---|---|
rvest, httr |
HTML parsing and CSS-selector web scraping |
tidyverse / dplyr |
Data wrangling |
shiny, DT, bslib |
Reactive dashboard, tables, dark theme |
ggplot2 |
All visualisations |
rmarkdown |
Reproducible report |
| base R | All inference and diagnostics — HC3, VIF, Breusch-Pagan, RESET, Cliff's delta and the bootstrap are implemented directly, so the analysis runs without car, lmtest or sandwich |
install.packages(c("tidyverse", "rvest", "httr", "shiny", "DT",
"ggplot2", "bslib", "rmarkdown"))# 1. (optional) re-scrape
source("Data/Submission_Cleaning.R")
# 2. run the full statistical analysis
# -> Analysis/output/{tables,figures,results.rds,analysis_log.txt}
system("Rscript Analysis/run_all.R")
# 3. launch the dashboard
shiny::runApp("ShinyApp")
# 4. build the report (HTML, or Markdown if pandoc is unavailable)
source("Report/render_report.R")Raw columns (as scraped)
| Column | Type | Description | Example |
|---|---|---|---|
Ranks |
Character | Steam250 rank (1–250) | " 1. " |
Titles |
Character | Game name | "Stardew Valley" |
Release.Date |
Character | Launch date | "26 Feb 2016" |
Price |
Character | USD price or "Free" |
"$14.99" |
Score |
Numeric | Steam250 weighted score | 8.85 |
Player.Ratings |
Character | Percentage of positive reviews | "98%" |
Number.of.Votes |
Character | Total review count | "777,890" |
Cleaned and derived (from gamescape_prepare())
| Variable | Description |
|---|---|
Rank, Title, Release_Date |
parsed rank, title, date |
Price |
numeric USD; "Free" → 0 (a real price, not a missing value) |
Score |
Steam250 score, 0–10 scale |
Player_Rating |
positive-review share in percentage points |
Number_of_Votes |
integer review count |
Log_Votes |
log(Number_of_Votes) — justified by skewness 3.46 → 0.22 |
Game_Age |
years between release and the reference date |
Game_Type |
Free-to-Play (n = 32) or Paid (n = 218) |
Reference date. The scrape timestamp is not recorded anywhere in the repo, so
Game_Age uses a documented assumption of 2024-10-15 (the latest release in
the data is 2024-10-11, and the course timeline places collection in Oct 2024).
Shifting it ±90 days leaves the age coefficient numerically unchanged.
| Area | What is used |
|---|---|
| Transformation | Skewness, excess kurtosis, Hill tail index, Shapiro–Wilk — used to justify Log_Votes, not assume it |
| Correlation | Pearson and Spearman with 95% CIs (Fisher z; Bonett–Wright for Spearman), with a stated reason for the preferred measure per pair |
| Regression | OLS with HC3 heteroscedasticity-robust standard errors |
| Model selection | Nested F tests, HC3 Wald tests, adjusted R², AIC, BIC |
| Diagnostics | Residual-vs-fitted, Q-Q, scale-location, RESET, Breusch–Pagan, VIF, Cook's distance and leverage, plus a refit excluding all screened points |
| Group comparison | Welch's t-test and Wilcoxon rank-sum, with Hedges' g and Cliff's delta (bootstrap CI) |
| Selection bias | Admission-boundary reconstruction and a Monte Carlo demonstration of collider/Berkson selection |
Corrections follow evidence: HC3 is applied because Breusch–Pagan rejects constant variance, and Wilcoxon is primary because Shapiro–Wilk rejects normality in both groups.
All findings are scoped to games already inside the Steam250 Top 250. None is a causal claim.
Age and popularity. Each additional year of age is associated with about 8.1% more reviews (95% CI 4.6%–11.8%) after controlling for price and rating. Accumulation is strongly sub-proportional to age: the log–log slope is 0.41, significantly below 1 (p < 0.0001), so reviews arrive fastest early and taper.
The Score is a formula, not a market signal. Player rating and log-votes alone explain 93.2% of the variation in Score; adding price and game age changes nothing (p = 0.183). Player rating is published rounded to whole percent, and that rounding alone accounts for 65% of the residual variance — allowing each true rating to sit anywhere in its rounding interval reproduces 83.6% of scores exactly. Score is effectively a deterministic function of its two inputs, so this regression recovers Steam250's ranking formula rather than a fact about games. A rating × log-votes interaction is significant and improves AIC and BIC, consistent with a confidence-weighted rating rule.
Price and popularity. Positive, modest and linear: about 2.6% more reviews per dollar (95% CI 1.3%–3.8%) after controls. The quadratic term is unsupported in every specification (p = 0.97 with controls; p = 0.75 among paid games only), and its implied turning point falls outside the observed price range.
Free-to-Play vs Paid. F2P titles are rated higher (Cliff's delta 0.56) and have roughly 75% fewer reviews (delta −0.61). There is no distinguishable Score difference once rating and votes are held fixed (p = 0.49).
The selection rule explains much of the above. Because Score increases in both rating and votes, admission to the Top 250 forces the two to trade off. The in-sample correlation between them is −0.57, but a Monte Carlo simulation applying the same top-250 cut-off to populations with known correlation produces comparable negative values even when the population correlation is zero or positive. That correlation therefore says nothing about Steam as a whole.
Three claims previously stated here are not supported by the analysis:
| Earlier claim | What the data show |
|---|---|
| "Sweet-spot pricing: the $14.99–$19.99 range dominates both vote counts and score rankings." | No non-linearity is supported (p = 0.97 with controls). Mean log-votes rises roughly monotonically with price. The apparent sweet spot is a density artefact of charm pricing — 210 of 218 paid games end in .99, and 48 sit at exactly $19.99. |
| "Free-to-Play anomaly … possibly because F2P players review less frequently." | The vote gap is real, but the rate claim is unmeasurable here: the dataset records review counts and never player counts. The observed pattern (higher rating and fewer votes) is what the admission boundary mechanically requires. |
| "Recent titles underrepresented: older games accumulate more votes over time, giving them a structural advantage." | Age is positively associated with votes, but accumulation is strongly sub-proportional (log–log slope 0.41, not 1), and the Top 250 is in fact weighted towards recent releases — 123 of 250 games were released in 2020 or later. |
- Sample selection. This is the top of a ranking, not a sample of Steam.
Every relationship involving
Scoreis conditional on inclusion in the Top 250 and must not be generalised to the marketplace. - Restricted range. Score spans 0.39 points of a nominal 0–10 scale;
Player_Ratingtakes only 7 distinct values, all ≥ 94%; the least-reviewed game still has 3,495 reviews. Correlations are attenuated accordingly. - n = 250, with only 32 Free-to-Play games — those comparisons rest on a small group, which is why effect-size confidence intervals are reported.
- Observational data. No randomisation, no instruments, no causal claims.
- Omitted variables. Genre, developer, marketing, bundles, sale history and concurrent players are all absent and plausibly related to price and volume.
- Measurement. Prices are listed prices at scrape time and several are visibly discounted; ratings are rounded to whole percent; one release date is imputed by the scraper rather than scraped.
- A single snapshot — nothing here supports a claim about trends over time.
| Name | Contributions |
|---|---|
| Mainak Sarkar | Web scraping, Data analysis ,data cleaning, Shiny app development |
| Neelaksha Malik | Data cleanings, R Markdown report, visualizations |
| Tenzin Tsomu | Report writing, visualizations, presentation |
| Harsh Deep | Report writing, visualizations, presentation |
Released under the MIT License. Data sourced from Steam250 — all game data belongs to their respective publishers.