Hydra 🌊

High-Throughput Log Router & API Ingestion Gateway

Hydra is an event logging ingestion framework built using TypeScript (for the edge filtering and rate limiting) and Java (for the concurrent log buffer and batch storage writer). It implements edge security, token-bucket rate limiting, a lock-free Ring Buffer queue, and asynchronous batch storage flushes.

---

📖 Hydra in Plain English (Layman's Terms)

When thousands of applications, servers, and devices generate logs (telemetry, security audits, checkout events) at the same time, writing each log to a database instantly will crash the database due to the massive volume of individual writes.

Hydra solves this using a two-tier architecture:

1. Edge Filter Gateway (TypeScript): Inspects incoming requests to make sure they have a valid security API key, and enforces a rate limit (Token Bucket). If a client floods the system beyond limits, they get blocked immediately before consuming server resources.
2. Buffer Ingestor (Java): Instead of writing to the database immediately, allowed logs are added to a fast, in-memory circular queue (Ring Buffer). The ingestor replies 202 Accepted instantly so the client doesn't wait.
3. Lock-Free Batching: A background worker thread drains logs from the Ring Buffer in batches (e.g. 128 logs at a time) and flushes them to disk or storage in one write operation, drastically reducing database overhead.
4. Backpressure: If the queue fills up (database is down or slow), the Ring Buffer blocks new writes and returns 429 Ingestor Queue Full, informing the edge gateway to apply backpressure.

---

📂 Project Structure

hydra/
├── gateway/                 <-- TypeScript Edge API Gateway / Rate Limiter
│   ├── src/
│   │   ├── gateway.ts       <-- Token Bucket filter & proxy route server
│   │   └── test-client.ts   <-- Auth & Rate-limit flood verification suite
│   ├── package.json
│   └── tsconfig.json
│
├── ingestor/                <-- Java High-Performance Log Buffer Ingestor
│   ├── src/
│   │   └── main/
│   │       └── java/
│   │           └── com/
│   │               └── hydra/
│   │                   ├── IngestServer.java <-- Native HTTP server & batch flusher
│   │                   ├── LogEvent.java     <-- Pre-allocated reusable log entity
│   │                   └── RingBuffer.java   <-- Lock-free sequence queue
│   └── pom.xml              <-- Maven compilation properties
│
├── package.json            <-- Monorepo workspaces coordinator
└── README.md               <-- You are here!

---

⚙️ Requirements

• Java JDK 17 (Ensure java and javac are in your PATH)
• Node.js (v20+ or Bun runtime)
• Git

---

🚀 How to Run the Verification Suite

To start the gateway, ingestor, and execute the test client validation suite, follow these steps:

1. Compile & Start the Java Ingestor (Terminal 1)

Compile the Java source files, and start the HTTP ingestion backend:

cd ingestor
mkdir -p target/classes
javac -d target/classes src/main/java/com/hydra/*.java
java -cp target/classes com.hydra.IngestServer

2. Start the TypeScript Edge Gateway (Terminal 2)

In a second window, start the gateway filter:

cd gateway
npm run dev

3. Run the Test Client Verification Suite (Terminal 3)

In a third window, execute the automated test suite:

cd gateway
npx tsx src/test-client.ts

📊 Expected Test Output

When you run the test suite, the output will show:

• Test 1 & 2: Requests without keys or with wrong keys get blocked with HTTP 401 Unauthorized.
• Test 3: A valid authorized request successfully queues and returns HTTP 202 Accepted.
• Test 4 (Flood Test): A burst of 150 requests will consume the rate limit. You will see a count of successful queueings and a count of rate-limited blockings with HTTP 429 Too Many Requests, proving the rate limiter works!
• Ingested Logs: Check ingestor/target/ingested_logs.log to see the batched logs flushed by the Java background thread.