encoder-stack — Transformer Encoder Stack (Pure C)

Part 11 of: Transformer from Scratch in Pure C

Goal: Implement the full "Attention is All You Need" paper in pure C, building up from primitives to a complete machine translation pipeline.

encoder-stack marks the transition from building a block to building a model. It sequentially composes $N$ independent EncoderLayer instances into a full deep representation pipeline, rigorously managing memory lifecycles along the way.

---

Architecture Context

                Input
                  │
                  ▼
        ┌────────────────┐
        │ Encoder Layer 1│
        └────────────────┘
                  │
                  ▼
        ┌────────────────┐
        │ Encoder Layer 2│
        └────────────────┘
                  │
                  ▼
        ┌────────────────┐
        │ Encoder Layer 3│
        └────────────────┘
                  │
                 ...
                  │
                  ▼
        ┌────────────────┐
        │ Encoder Layer N│
        └────────────────┘
                  │
                  ▼
             Encoder Output

---

Highlights

• Pure C (C99), orchestrating 7 previous standalone primitives into a deep architecture.
• Strict Layer Independence — $N$ layers are dynamically allocated, ensuring no weights are accidentally shared and each layer learns distinct representations.
• $O(1)$ Memory Overhead — A deterministic pointer-swapping ownership model ensures a maximum of 2 intermediate sequence matrices are alive in RAM at any time, regardless of depth $N$.
• 11-Test Suite — Verifying structural integrity, dimensional preservation, and exact element-by-element equivalence between a 1-layer stack and a bare Encoder Layer.
• Built on encoder-layer ← | Foundation for decoder-layer →

---

What encoder-stack provides

A single public header (include/encoder_stack.h) exposes:

Encoder Stack

typedef struct {
    EncoderLayer **layers;    /* Array of N independent layers */
    size_t num_layers;        /* Stack depth */
    size_t d_model;
    size_t num_heads;
    size_t d_ff;
} EncoderStack;

EncoderStack* encoder_stack_create(size_t num_layers, size_t d_model, size_t num_heads, size_t d_ff);
Matrix*       encoder_stack_forward(EncoderStack *stack, const Matrix *input);
EncoderLayer* encoder_stack_get_layer(const EncoderStack *stack, size_t index);
size_t        encoder_stack_depth(const EncoderStack *stack);
size_t        encoder_stack_num_parameters(const EncoderStack *stack);
void          encoder_stack_free(EncoderStack *stack);

---

Example usage

#include "encoder_stack.h"

int main(void) {
    size_t d_model = 512;
    size_t heads = 8;
    size_t d_ff = 2048; 
    size_t num_layers = 6;
    size_t seq_len = 10;

    /* Create the full N=6 Encoder Stack */
    EncoderStack *stack = encoder_stack_create(num_layers, d_model, heads, d_ff);

    /* Embeddings with positional encodings */
    Matrix *input = matrix_create(seq_len, d_model);

    /* Run the complete self-attention + FFN pipeline across all 6 layers */
    Matrix *output = encoder_stack_forward(stack, input);

    matrix_free(output);
    matrix_free(input);
    encoder_stack_free(stack);
    
    return 0;
}

---

Directory structure

encoder-stack/
├── include/
│   └── encoder_stack.h        # Public API
├── src/
│   ├── encoder_stack.c        # Orchestrator and memory ownership logic
│   └── demo.c                 # Visual pipeline checkpoint tracer
├── tests/
│   └── test_encoder_stack.c   # 11-test suite
├── Makefile                   # Links 7 previous upstream projects
├── README.md
└── Analysis.md

---

Build & Run

Requirements

• GCC (C99 compatible)
• GNU Make
• libm (math library)
• Seven previous projects: mini-tensor, neural-net, attention, multi-head-attention, layernorm, feed-forward, and encoder-layer must be adjacent in the directory tree.

Build

make

Run tests

make test

Run visual demo

make demo

---

Pipeline Demonstration Output

=========================================
  Encoder Stack — Architecture Pipeline
=========================================

--- Model Details ---
d_model    : 8
heads      : 2
d_ff       : 32
num_layers : 6
--------------------------------
Layer 1 : 840
Layer 2 : 840
Layer 3 : 840
Layer 4 : 840
Layer 5 : 840
Layer 6 : 840
--------------------------------
Total Parameters : 5040

--- Forward Pass Tracing ---

Input Shape: 3x8

Encoder Layer 1/6 complete
  Shape      : 3x8
  Parameters : 840

Encoder Layer 2/6 complete
  Shape      : 3x8
  Parameters : 840

Encoder Layer 3/6 complete
  Shape      : 3x8
  Parameters : 840

Encoder Layer 4/6 complete
  Shape      : 3x8
  Parameters : 840

Encoder Layer 5/6 complete
  Shape      : 3x8
  Parameters : 840

Encoder Layer 6/6 complete
  Shape      : 3x8
  Parameters : 840

Encoder Stack output successfully generated!

---

Test suite

=== Encoder Stack Test Suite ===
Running test_stack_create...
  Pass.
Running test_layer_count...
  Pass.
Running test_forward_shape...
  Pass.
Running test_parameter_count...
  Pass.
Running test_invalid_configuration...
[encoder-stack] ERROR: All configuration parameters must be > 0.
[encoder-stack] ERROR: All configuration parameters must be > 0.
[encoder-stack] ERROR: All configuration parameters must be > 0.
[encoder-stack] ERROR: All configuration parameters must be > 0.
[encoder-stack] ERROR: d_model (9) must be divisible by num_heads (2).
  Pass.
Running test_unique_layers...
  Pass.
Running test_independent_parameters...
  Pass.
Running test_component_reuse...
  Pass.
Running test_sequence_preserved...
  Pass.
Running test_model_dimension_preserved...
  Pass.
Running test_single_layer_equivalence...
  Pass.
All 67 tests passed!

---

What's explicitly NOT in this project

Missing feature	Why	Where it belongs
Decoder	Focus is solely on completing the Encoder branch	Part 12 — Decoder Layer
Embeddings	Input to the stack must already be embedded	Part 04 / Part 05
Masking	Only needed for the Decoder branch to prevent looking into the future	Part 12 — Decoder Layer

---

Engineering Design © 2026 Shahid Ul Islam.
Built with passion for Mathematical Rigor and Technical Excellence.