SEC HTML Data Miner

Automated keyword extraction and visual highlighting from SEC EDGAR HTML filings

📋 Overview

SEC HTML Data Miner is a specialized data extraction and visualization tool for SEC EDGAR filings. It automatically:

• 📊 Loads keyword data from Parquet files (financial indicators, executive names, stock transactions)
• 🔍 Matches keywords in HTML filings using smart fuzzy/strict matching strategies
• 🎨 Highlights results directly in HTML for easy visualization
• ⚡ Processes at scale with support for batch processing and ZIP archives

Perfect for:

• Financial research and due diligence automation
• Extracting officer/insider trading information from Form 4 filings
• Automating SEC document analysis workflows
• Building financial data pipelines

🚀 Quick Start

Installation

# Clone repository
git clone https://github.com/Alex-Wang66/SEC-HTML-Data-Miner.git
cd SEC-HTML-Data-Miner

# Install dependencies
pip install -r requirements.txt

# Install as package
pip install -e .

Basic Usage

# Process a directory of HTML filings
python -m src.cli \
  --parquet-dir ./data/keywords_parquet \
  --html-dir ./data/sec_filings \
  --output-dir ./output_highlighted

# With custom highlight color and debug mode
python -m src.cli \
  --parquet-dir ./data/keywords_parquet \
  --html-dir ./data/sec_filings \
  --output-dir ./output \
  --color lime \
  --debug

Python API

from src.pipeline import SECDataMiningPipeline

pipeline = SECDataMiningPipeline(
    parquet_dir='./data/keywords',
    html_dir='./data/filings',
    output_dir='./output',
    highlight_color='yellow',
    debug_border=False,
    match_strategy='smart'
)

stats = pipeline.run()
print(f"Processed {stats['processed']} files, found {stats['matched']} matches")

🏗️ Architecture

Core Components

Parser (parser.py)

• ParquetReader: Loads keyword data indexed by SEC Accession ID
• HTMLParser: Parses SEC EDGAR HTML files while preserving DOM structure
• AccessionIDExtractor: Extracts SEC identifiers from filenames

Matcher (matcher.py)

• StrictMatcher: Matches table data (ignores punctuation/spacing)
• LooseMatcher: Matches natural text with flexible spacing
• SmartMatcher: Combines strategies and extracts sub-keywords (dates, numbers)
• MatchMerger: Merges overlapping match intervals

Highlighter (highlighter.py)

• HTMLHighlighter: Applies CSS highlighting to matched DOM nodes
• Cross-node matching support (keywords spanning multiple elements)

Pipeline (pipeline.py)

• SECDataMiningPipeline: Orchestrates the complete workflow

Data Flow

Parquet Files (Accession ID → Keywords)
           ↓
   ParquetReader (builds lookup table)
           ↓
    HTML Files ──→ HTMLParser (extracts DOM)
           ↓
    Full Text + Node Map
           ↓
    Matcher (finds keyword positions)
           ↓
    Merge Overlapping Matches
           ↓
    HTMLHighlighter (modifies DOM)
           ↓
    Output HTML (with highlights)

💡 Use Cases

Case 1: Extract Insider Trading Information

Process Form 4 (insider transaction) filings to automatically highlight:

• Transaction dates (MM/DD/YYYY)
• Share quantities
• Stock prices
• Executive names

Input Parquet: fraud_dedup_df_0000038725.parquet

file_name: "0000038725-25-000108_FRANKLIN...8-K_2025-05-08.html"
source_contents: ["05/20/2024", "$42.84", "718,848 shares", "Executive Name"]

Output: Same HTML with all matching text highlighted in yellow

Case 2: Financial Fraud Detection

Highlight specific financial metrics and key statements from 8-K, 10-K filings:

• Revenue figures
• Merger/acquisition announcements
• Restatements
• Officer changes

Case 3: Competitive Intelligence

Extract and highlight competitive references and product mentions from 10-K filings

🔧 Matching Strategies

Smart Matching (Default)

Combines multiple strategies:

• Extracts sub-keywords (dates, numbers) from complex strings
• Applies both strict and loose matching
• Best for table data with mixed formatting

Strict Matching

Removes all punctuation/spacing before matching:

• Input: "05/20/2024S34,960D$42.84(1)718,848D"
• Cleaned: "052020243496042.84171884"
• Matches fragments like dates, amounts, quantities

Loose Matching

Allows flexible whitespace but preserves punctuation:

• "Russell D. Fleeger" matches "Russell D. Fleeger"
• Useful for names and narrative text

📊 Performance

On typical SEC filing batch (100+ HTML files with 10-20 keywords each):

• Speed: ~0.5-1.0 seconds per file
• Memory: ~50-100MB per 100 files
• Accuracy: 95%+ match accuracy with smart strategy

📖 Examples

See examples/ directory for complete workflows:

• basic_highlighting.py - Simple HTML highlight workflow
• batch_processing.py - Process multiple SEC filings
• custom_matching.py - Use custom matcher strategies

🛠️ Development

Project Structure

SEC-HTML-Data-Miner/
├── src/
│   ├── __init__.py
│   ├── parser.py          # HTML/Parquet parsing
│   ├── matcher.py         # Keyword matching strategies
│   ├── highlighter.py     # DOM highlighting
│   ├── pipeline.py        # Main workflow
│   └── cli.py            # Command-line interface
├── tests/                 # Unit tests
├── examples/             # Usage examples
├── data/
│   ├── sample_html/      # Sample SEC filings
│   └── sample_parquet/   # Sample keyword data
├── docs/                 # Documentation
├── pyproject.toml        # Project config
├── requirements.txt      # Dependencies
└── README.md

Running Tests

pytest tests/ -v

Code Style

# Format code
black src/

# Lint
flake8 src/

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/improvement)
3. Commit changes (git commit -am 'Add feature')
4. Push to branch (git push origin feature/improvement)
5. Open a Pull Request

📝 License

MIT License - see LICENSE file for details

👤 Author

Alex Wang - GitHub

🙏 Acknowledgments

• Built during internship at HSBC for SEC EDGAR data automation
• Inspired by need to scale financial document processing
• Dedicated to researchers and practitioners in financial analysis

---

Star the repo if you find this useful! ⭐