Nerva - Deep Learning Framework from Scratch

A complete deep learning framework implemented from scratch in Rust and C, with zero external machine learning dependencies.

Overview

Nerva is both a custom programming language and a deep learning framework built entirely from the ground up. It does not rely on PyTorch, TensorFlow, or any external ML libraries. Every component is implemented by hand:

• Custom compiler (Rust)
• Tensor runtime (C)
• Complete autograd engine (automatic backpropagation)
• Functional CNN achieving 93.2% accuracy on MNIST
• AdamW optimizer
• Model persistence (save/load)

Results

MNIST Digit Classification

Architecture: CNN (simplified LeNet-5)
- Conv2D: 16 filters 5x5 -> ReLU -> MaxPool 2x2
- Conv2D: 32 filters 5x5 -> ReLU -> MaxPool 2x2
- Flatten -> Linear 512x64 -> ReLU -> Linear 64x10

Training:
- 10,000 images
- 20 epochs
- Batch size: 128
- Learning rate: 0.003
- Optimizer: AdamW

Results:
- Final loss: 1.19
- Test set accuracy: 93.20%

Architecture

+-----------------------------------------+
|         Nerva Compiler (Rust)           |
|  Lexer -> Parser -> Semantic -> Codegen |
+----------------+------------------------+
                 |
                 v
+-----------------------------------------+
|         Tensor Runtime (C)              |
|  - 2D and 4D Tensors                    |
|  - Autograd (backpropagation)           |
|  - Operations: matmul, conv2d, relu     |
|  - Optimizers: AdamW                    |
|  - Loss: Softmax + Cross-Entropy        |
+-----------------------------------------+

Usage Examples

Simple Neural Network (MLP)

fn main() -> int {
    let X = Tensor::load_mnist_images("data/train-images-idx3-ubyte", 1000);
    let y = Tensor::load_mnist_labels("data/train-labels-idx1-ubyte", 1000).one_hot(10);
    
    let mut W1 = Tensor::rand([784, 128], 0).trainable();
    let mut b1 = Tensor::zeros([1, 128], 0).trainable();
    let mut W2 = Tensor::rand([128, 10], 0).trainable();
    let mut b2 = Tensor::zeros([1, 10], 0).trainable();
    
    for epoch in 0..10 {
        let z1 = X.matmul(W1) + b1;
        let h1 = z1.relu();
        let logits = h1.matmul(W2) + b2;
        
        let loss = logits.softmax_cross_entropy(y);
        loss.backward();
        
        W1.adam_step(0.001, 0.9, 0.999, 0.000001, 0.0);
        b1.adam_step(0.001, 0.9, 0.999, 0.000001, 0.0);
        W2.adam_step(0.001, 0.9, 0.999, 0.000001, 0.0);
        b2.adam_step(0.001, 0.9, 0.999, 0.000001, 0.0);
        
        W1.zero_grad();
        b1.zero_grad();
        W2.zero_grad();
        b2.zero_grad();
    }
    
    return 0;
}

Convolutional Neural Network (CNN)

fn main() -> int {
    let X = Tensor::load_mnist_images("data/train-images-idx3-ubyte", 10000);
    let y = Tensor::load_mnist_labels("data/train-labels-idx1-ubyte", 10000).one_hot(10);
    
    let mut W_conv1 = Tensor::rand([16, 1, 5, 5], 0).trainable();
    let mut b_conv1 = Tensor::zeros([16], 0).trainable();
    
    let mut W_conv2 = Tensor::rand([32, 16, 5, 5], 0).trainable();
    let mut b_conv2 = Tensor::zeros([32], 0).trainable();
    
    let mut W_fc1 = Tensor::rand([512, 64], 0).trainable();
    let mut b_fc1 = Tensor::zeros([1, 64], 0).trainable();
    
    let mut W_fc2 = Tensor::rand([64, 10], 0).trainable();
    let mut b_fc2 = Tensor::zeros([1, 10], 0).trainable();
    
    for epoch in 0..20 {
        let c1 = X.conv2d(W_conv1, b_conv1);
        let p1 = c1.max_pool2d(2, 2);
        let c2 = p1.conv2d(W_conv2, b_conv2);
        let p2 = c2.max_pool2d(2, 2);
        let flat = p2.flatten();
        
        let z1 = flat.matmul(W_fc1) + b_fc1;
        let h1 = z1.relu();
        let logits = h1.matmul(W_fc2) + b_fc2;
        
        let loss = logits.softmax_cross_entropy(y);
        loss.backward();
        
        W_conv1.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        b_conv1.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        W_conv2.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        b_conv2.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        W_fc1.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        b_fc1.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        W_fc2.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        b_fc2.adam_step(0.003, 0.9, 0.999, 0.000001, 0.0);
        
        W_conv1.zero_grad();
        b_conv1.zero_grad();
        W_conv2.zero_grad();
        b_conv2.zero_grad();
        W_fc1.zero_grad();
        b_fc1.zero_grad();
        W_fc2.zero_grad();
        b_fc2.zero_grad();
    }
    
    return 0;
}

Model Persistence

// Train and save
W1.save("models/W1.bin");
b1.save("models/b1.bin");

// Load and predict
let W1 = Tensor::load("models/W1.bin");
let b1 = Tensor::load("models/b1.bin");

Installation

Requirements

• Rust (stable)
• GCC or Clang

Build

cargo build --release

Run a Nerva script

cargo run --bin nerva examples/mnist.nv

Project Structure

nerva/
|-- src/
|   |-- main.rs          # Compiler entry point
|   |-- lexer.rs         # Lexical analysis
|   |-- parser.rs        # Syntax analysis
|   |-- semantic.rs      # Semantic analysis
|   +-- codegen.rs       # C code generation
|-- runtime/
|   |-- tensor.h         # Tensor definitions
|   +-- tensor.c         # Operation implementations

Implemented Features

• Basic compiler (lexer, parser, codegen)
• 2D and 4D tensor runtime
• Complete autograd engine
• AdamW optimizer
• CNN operations (Conv2D, MaxPool, Flatten)
• Model persistence
• Fused Softmax + Cross-Entropy
• Gradient clipping
• Xavier/Glorot initialization

Key Learnings

This project provided deep understanding of:

• Compiler internals and implementation
• Autograd and backpropagation mechanics
• Tensor operations and broadcasting
• Advanced optimizers (AdamW)
• Convolutional neural networks from first principles
• Memory management in C

License

MIT License