Skip to content
BranchesTags

Repository files navigation

🔄 Multi-Source Candidate Data Transformer

An enterprise-grade, deterministic pipeline designed to ingest candidate information from multiple heterogeneous sources (structured recruiter CSVs, proprietary ATS JSON databases, unstructured PDF/TXT resumes, and handwritten recruiter notes), resolve field conflicts, clean and normalize formats, score data confidence, validate profile health, and project output structures dynamically.

🚀 Key Architectural Goals

  • Determinism: The pipeline avoids probabilistic black boxes. Given the same inputs, it consistently produces the exact same merged outputs.
  • Traceability (Provenance): Every single field value in the merged canonical profile is tracked back to its originating source and the resolution rule applied.
  • Robustness: The pipeline handles partial, malformed, or missing data gracefully, utilizing warning/error levels without ever crashing.
  • Runtime Adaptability (The Twist): The output structure is fully decoupled from the processing layers. A runtime JSON config reshapes, renames, and filters outputs without modifying the Python source code.

🛠️ Pipeline Architecture & Data Flow

The system operates as a layered data processing pipeline. Each layer performs a single task and passes the formatted object to the next stage:

📁 Input Folder (CSV, JSON, PDF, TXT)
   │
   ▼
1. [INGESTION]      ──► Parsers ingest files and output unified RawRecord objects.
   │
   ▼
2. [MATCHING]       ──► Groups RawRecords belonging to the same candidate via exact email or fuzzy name matching.
   │
   ▼
3. [MERGING]        ──► Resolves conflicts using source-priority rules and generates provenance tracking metadata.
   │
   ▼
4. [NORMALIZATION]  ──► Normalizes phones (E.164), countries (ISO-3166), dates (YYYY-MM), and skills (TF-IDF mapping).
   │
   ▼
5. [CONFIDENCE]     ──► Calculates transparent, feature-weighted profile and skill confidence scores.
   │
   ▼
6. [VALIDATION]     ──► Audits the profile for data completeness, syntax correctness, and structural warnings.
   │
   ▼
7. [PROJECTION]     ──► Reshapes and filters the final JSON structure dynamically using a custom runtime config.
   │
   ▼
📄 Custom JSON Output

🔬 NLP/ML Techniques Applied

The pipeline integrates light, fast, and deterministic NLP and ML heuristics optimized for performance and reliability:

Technique Core Library Implementation Target Rationale & Mechanics
Named Entity Recognition (NER) spaCy en_core_web_sm Unstructured Resume & Recruiter Notes parsing Identifies names (PERSON), companies (ORG), locations (GPE), and date ranges (DATE) in raw blocks of text. More context-aware than pure regular expressions.
Fuzzy String Matching thefuzz (Levenshtein) Candidate deduplication & grouping Compares candidate names using the Levenshtein edit distance. Uses token_sort_ratio (with an 85% threshold) to group name variations (e.g., "Jane Smith" and "Smith Jane") while separating different individuals.
TF-IDF + Cosine Similarity scikit-learn Skill normalization layer Converts messy raw skill names into char n-gram vectors (2-4 character chunks) and matches them against a vocabulary of standard canonical skills. Matches abbreviations (e.g., k8s ➔ Kubernetes) and variations (e.g., React.js Development ➔ React) with a 0.7 similarity threshold.
Feature-Weighted Heuristics Pure Python Profile and skill confidence scoring Scores profile reliability based on 5 features: source count, source agreement, source quality, normalization success, and completeness. Provides clear, explainable metrics rather than black-box scores.

📁 Repository Structure & Module Responsibilities

candidate-data-transformer/
├── app.py                      # Main CLI entry point. Parses command line arguments.
├── requirements.txt            # Python dependencies list.
├── README.md                   # System documentation.
├── configs/
│   ├── default_config.json     # Standard configuration containing all output fields.
│   └── custom_config.json      # Dynamic layout configuration (custom renames/filters).
├── data/
│   ├── input/                  # Sample source files (CSV, JSON, PDF, TXT).
│   └── output/                 # Destination for generated JSON profiles.
├── models/
│   ├── raw_record.py           # Standard internal format output by ingestion parsers.
│   └── canonical.py            # Unified candidate model containing merged profiles and provenance.
├── parsers/
│   ├── base_parser.py          # Base parser class defining the standard parsing interface.
│   ├── csv_parser.py           # Ingests recruiter CSV exports.
│   ├── json_parser.py          # Ingests ATS JSON files and maps proprietary columns.
│   ├── resume_parser.py        # Extracts sections from resumes (PDF/TXT) via spaCy NER and Regex.
│   └── notes_parser.py         # Parses free-text recruiter notes using NER and keyword mappings.
├── resolution/
│   ├── matcher.py              # Fuzzy matching matcher for candidate grouping.
│   ├── merger.py               # Combines RawRecords into a single CanonicalRecord.
│   └── conflict_resolver.py    # Priority-based logic resolver for conflicting fields.
├── normalizers/
│   ├── phone_normalizer.py     # Phone E.164 formatter using the phonenumbers library.
│   ├── date_normalizer.py      # Date formatter mapping to YYYY-MM or "Present".
│   ├── skill_normalizer.py     # Skill canonicalizer using scikit-learn TF-IDF matcher.
│   └── location_normalizer.py  # Country translator mapping to ISO-3166 alpha-2.
├── scoring/
│   └── confidence.py           # Computes confidence scores using feature-weighted heuristics.
├── validators/
│   ├── validator.py            # Field-level profile health validator.
│   └── report_generator.py     # Compiles validation issues into clean scorecard reports.
├── projection/
│   ├── projector.py            # Config-driven output projector supporting custom structures.
│   └── config_parser.py        # Config validator for projection files.
├── pipeline/
│   └── pipeline.py             # Pipeline orchestrator wiring all stages together.
├── utils/
│   ├── ner_extractor.py        # Shared singleton loader for the spaCy NER model.
│   ├── file_utils.py           # Scanning and writing I/O utilities.
│   └── logger.py               # Standardized console logging setup.
├── ui/
│   └── streamlit_app.py        # Streamlit web interface.
└── tests/                      # Automated test suite (62 total unit and integration tests).

⚙️ Environment Setup & Installation

# 1. Create and activate a Python virtual environment
python -m venv venv
venv\Scripts\activate           # On Windows
source venv/bin/activate        # On macOS/Linux

# 2. Install all library dependencies
pip install -r requirements.txt

# 3. Download the spaCy English NLP model
python -m spacy download en_core_web_sm

🏃 Run Guide

1. Command Line Interface (CLI)

# Run the pipeline with the default output configuration (includes all fields)
python app.py --input data/input/ --output data/output/results.json

# Run the pipeline with a custom config (custom renaming and field selection)
python app.py --input data/input/ --config configs/custom_config.json --output data/output/custom_results.json

2. Web Application UI

# Start the interactive Streamlit UI
streamlit run ui/streamlit_app.py

🧪 Automated Testing & Gold-Profile Edge Case

The test suite contains 62 tests covering parser logic, normalizations, merging, validation, and projection.

Run Tests:

pytest tests/ -v

Gold-Profile Integration Test:

The test test_gold_profile_comparison_jane_smith in tests/test_pipeline.py serves as a "gold standard" comparison. It runs the pipeline end-to-end against the sample data to verify the complex edge cases of merging 5 different records for the same candidate. It asserts:

  1. Fuzzy Deduplication: Matcher correctly groups "Jane Smith" and "Jane A. Smith" as a single candidate based on name similarity and overlapping emails.
  2. Conflict Resolution: The name "Jane A. Smith" (JSON longest name) wins over "Jane Smith" (CSV name) because of source length priority.
  3. Email Union: Combines jane.smith@email.com and jane.s@techcorp.com into a single deduplicated list.
  4. Phone Standardization: Normalizes raw inputs (555) 123-4567 and 5551234567 into E.164 format +15551234567.
  5. Location Normalization: Translates country string "United States" to its ISO code "US".
  6. Skill Mapping (TF-IDF): Standardizes raw skill text "communication" into "Communication" and "k8s" into "Kubernetes".
  7. Experience Grouping: Deduplicates multiple overlapping TechCorp job logs into exactly two unique job entries.
  8. Unstructured Parsing: Correctly extracts 7 years of experience from conversational recruiter notes.

📊 Sample Inputs vs. Generated Outputs

The Input Sources

  • data/input/recruiter_export.csv: Contains structured columns with standard name, email, and phone info.
  • data/input/ats_data.json: An ATS database dump with distinct labels (e.g. applicant_name, contact_phone, skill_tags).
  • data/input/resume_jane_smith.txt: Unstructured document layout containing a resume summary, section headers, and lists.
  • data/input/recruiter_notes.txt: Conversational, highly unstructured text call log mentioning candidate details.

The Output Results

1. Default Output (results.json)

Generated using the standard configuration file configs/default_config.json. This outputs the full canonical profile including the computed field provenance list:

{
  "candidates": [
    {
      "candidate_id": "3a98e91b-b17b-476b-9c1f-88dbc2e63cc8",
      "full_name": "Jane A. Smith",
      "emails": [
        "jane.smith@email.com",
        "jane.s@techcorp.com"
      ],
      "phones": [
        "+15551234567"
      ],
      "location": {
        "city": "San Francisco",
        "region": "CA",
        "country": "US"
      },
      "links": {
        "linkedin": "linkedin.com/in/janesmith",
        "github": "github.com/janesmith",
        "portfolio": null,
        "other": []
      },
      "headline": "Lead Software Engineer",
      "years_experience": 7,
      "skills": [
        { "name": "Python", "confidence": 0.9, "sources": [ "ats_json", "resume_pdf", "notes_txt" ] },
        { "name": "AWS", "confidence": 0.8, "sources": [ "ats_json", "resume_pdf", "notes_txt" ] },
        { "name": "System Design", "confidence": 0.8, "sources": [ "ats_json", "resume_pdf", "notes_txt" ] },
        { "name": "Leadership", "confidence": 0.8, "sources": [ "ats_json", "resume_pdf" ] },
        { "name": "Java", "confidence": 0.6, "sources": [ "resume_pdf" ] },
        { "name": "Docker", "confidence": 0.8, "sources": [ "resume_pdf", "notes_txt" ] },
        { "name": "Kubernetes", "confidence": 0.6, "sources": [ "resume_pdf" ] },
        { "name": "PostgreSQL", "confidence": 0.6, "sources": [ "resume_pdf" ] },
        { "name": "Redis", "confidence": 0.6, "sources": [ "resume_pdf" ] },
        { "name": "CI/CD", "confidence": 0.6, "sources": [ "resume_pdf" ] },
        { "name": "Agile", "confidence": 0.6, "sources": [ "resume_pdf" ] },
        { "name": "REST APIs", "confidence": 0.6, "sources": [ "resume_pdf" ] },
        { "name": "SQL", "confidence": 0.5, "sources": [ "csv" ] },
        { "name": "Communication", "confidence": 0.5, "sources": [ "notes_txt" ] }
      ],
      "experience": [
        {
          "company": "TechCorp",
          "title": "Senior Software Engineer",
          "start": "2020-01",
          "end": "Present",
          "summary": "Led backend team of 5 engineers building microservices Reduced API response time by 40% through caching strategy Designed and implemented event-driven architecture"
        },
        {
          "company": "StartupXYZ",
          "title": "Software Engineer",
          "start": "2017-06",
          "end": "2019-12",
          "summary": "Full-stack development"
        }
      ],
      "education": [
        {
          "institution": "MIT",
          "degree": "B.S.",
          "field": "Computer Science",
          "end_year": 2017
        }
      ],
      "provenance": [
        { "field": "full_name", "source": "ats_json", "method": "longest_value" },
        { "field": "headline", "source": "ats_json", "method": "only_source" },
        { "field": "years_experience", "source": "notes_txt", "method": "longest_value" },
        { "field": "emails", "source": "multiple", "method": "union" },
        { "field": "phones", "source": "multiple", "method": "union" },
        { "field": "skills", "source": "multiple", "method": "union" },
        { "field": "location", "source": "multiple", "method": "union" },
        { "field": "experience", "source": "multiple", "method": "union" },
        { "field": "education", "source": "multiple", "method": "union" }
      ],
      "overall_confidence": 0.58
    },
    ...
  ]
}

2. Custom Projected Output (custom_results.json)

Generated using the custom configuration file configs/custom_config.json. This demonstrates the twist by flattening list paths, renaming fields (e.g. emails[0] ➔ primary_email), and omitting metadata:

{
  "candidates": [
    {
      "full_name": "Jane A. Smith",
      "primary_email": "jane.smith@email.com",
      "phone": "+15551234567",
      "city": "San Francisco",
      "country": "US",
      "skills": [
        "Python", "AWS", "System Design", "Leadership", "Java", "Docker", 
        "Kubernetes", "PostgreSQL", "Redis", "CI/CD", "Agile", "REST APIs", 
        "SQL", "Communication"
      ],
      "overall_confidence": 0.58
    },
    {
      "full_name": "John Doe",
      "primary_email": "john.doe@gmail.com",
      "phone": "+15559876543",
      "city": "New York",
      "country": "US",
      "skills": [
        "SQL", "Python", "Tableau", "Excel"
      ],
      "overall_confidence": 0.41
    }
  ]
}

📈 Scalability Analysis

The current single-threaded local implementation runs successfully in memory. If deploying this system at scale to handle large files, bulk candidates, or massive folder uploads, we should focus on the following bottlenecks:

1. Ingestion Scaling (IO Bound & CPU Bound)

  • Memory Footprint: Currently, standard Python libraries parse files fully into memory. If processing large multi-gigabyte CSV/JSON files, we should utilize stream parsing generators (like ijson for JSON stream decoding or pandas chunking for CSV processing) to keep memory consumption low.
  • NER Parsing CPU Bottleneck: Extracting named entities via spaCy is CPU-bound. If parsing thousands of resumes in a batch, we can utilize Python's multiprocessing package to distribute documents across multiple processor cores, or switch to spaCy's native batched pipeline (nlp.pipe()) to optimize GPU/CPU execution.

2. Deduplication & Matching Scaling

  • Match Complexity: The current matcher compares candidate pairs using a nested loop, resulting in a complexity of $O(N^2)$ where $N$ is the number of candidates. For hundreds of thousands of candidate profiles, this loop becomes a bottleneck.
  • Scaling Solution:
    1. Blocking/Indexing (Canopy Clustering): Pre-filter candidate lists into smaller buckets (e.g. by grouping by the first letter of their last names, their country, or area codes) so fuzzy calculations are only performed on candidates within the same block.
    2. Vector Encodings: Convert names and profiles into vectors and run approximate nearest neighbors (ANN) searches using vector indexing databases (like Milvus, Pinecone, or FAISS) to retrieve matched duplicates in $O(\log N)$ time.

🔮 Future Roadmap & Production Upgrades

To scale this codebase into an enterprise-grade production platform, the following features are planned:

📁 Uploads ──► [Kafka Queue] ──► [Docker Workers] ──► [OCR / LLM Parser] ──► [PostgreSQL Cache]

1. Advanced Ingestion & OCR Parsers

  • Document OCR Layouts: Integrate Tesseract or easyOCR to parse scanned images and PDF resumes that do not contain raw text layers.
  • LLM Structured Parser: Integrate local LLMs (e.g., Llama-3-8B) or hosted models using structured outputs (via JSON Schema schema validations) to parse unstructured recruiter notes and resume work histories with near-perfect conceptual accuracy, replacing fragile regex boundaries.

2. Event-Driven Distributed Scale

  • Message Broker Queues: Deploy a broker system (like Apache Kafka or RabbitMQ) in front of the parsers. When files are uploaded, ingestion tasks are queued.
  • Containerized Workers: Run parser workers inside auto-scaling Docker containers managed by Kubernetes (EKS) or serverless triggers (AWS Lambda / Google Cloud Run), allowing the system to scale horizontally to parse tens of thousands of resumes in parallel.

3. Distributed Resolution Cache

  • Database Storage: Move the canonical record storage from local JSON files to a relational database (like PostgreSQL with JSONB support).
  • Caching Layer: Cache TF-IDF vectors and lookup matrices in Redis to speed up normalizations and fuzzy checks.

About

An enterprise-grade Python pipeline that ingests heterogeneous candidate data (CSV, JSON, PDF/TXT resumes, notes), resolves conflicts, normalizes formats, and projects structured profiles dynamically via runtime configuration.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages