Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/architecture/core-layering.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ src/core/agent

Layer 1: Infrastructure and domain primitives
src/core/llm, src/core/tools, src/core/trace, src/core/auth, src/core/approvals,
src/core/commands
src/core/commands, src/core/checkpoint

Layer 0: Shared types and utilities
src/core/types, src/core/utils, src/core/config
Expand Down Expand Up @@ -84,6 +84,13 @@ service behavior. If a utility clearly belongs to one domain, place it in that
domain instead; for example, step-budget behavior lives under `src/core/agent`
rather than generic `src/core/utils`.

`src/core/checkpoint/portable-directory` is the lower shared filesystem
primitive for domain-owned checkpoints. It owns safe relative paths, selected
regular-file integrity, explicit resource policies, and staged local restore.
Memory and future directory-backed domains retain separate scope, manifest,
store, and lifecycle contracts above it; do not turn this primitive into a
universal persistence provider.

## Current Violations And Improvement Areas

These are known cleanup directions, not blockers for every feature:
Expand Down
10 changes: 10 additions & 0 deletions docs/architecture/durable-state.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,16 @@ boundaries. A shutdown checkpoint may reduce loss, but must not be the sole
commit path. An official object-store adapter, periodic long-session trigger,
and Execution Host recovery integration are not implemented yet.

Memory's local capture and restore delegate to the reusable
[`PortableDirectoryCheckpointService`](../../src/core/checkpoint/portable-directory/README.md).
That lower primitive requires an explicit selection and resource policy,
rejects unsafe paths and symlinks, validates per-file integrity, and stages a
complete directory before rename. It does not define durable scope, generation
identity, manifest compare-and-swap, or execution settlement. Memory v1 uses
compatibility ceilings so this extraction does not silently reject previously
accepted memory checkpoints; a future working-directory specialization must
choose narrower operational limits in its own public contract.

### Approval policy and project configuration

Remembered project approvals live in `command-approvals.json` through
Expand Down
4 changes: 4 additions & 0 deletions docs/project-posture.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,10 @@ See `docs/guides/programmatic/component-model.md` for the user-facing model.
- `src/core/runtime/` owns host-facing runtime boundaries, default tool
assembly, credentials, workspace catalogs, daemon discovery, and evented
single-run execution over `src/core/agent/`.
- `src/core/checkpoint/` owns lower shared checkpoint primitives such as safe,
bounded portable-directory capture and staged restore. Domain scopes,
manifests, stores, and lifecycle timing remain in memory, heartbeat, or the
future domain that gives those files meaning.
- `src/core/heartbeat/` owns autonomous runner cycles, heartbeat scheduling,
checkpoint reuse, and heartbeat task/run views.
- `src/core/chat/engine/` owns persisted conversation sessions, turns,
Expand Down
42 changes: 42 additions & 0 deletions src/__tests__/unit/core/memory-checkpoint.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { afterEach, describe, expect, it } from 'vitest';
import {
MemoryCheckpointCodec,
MemoryCheckpointConflictError,
MemoryCheckpointCorruptionError,
MemoryCheckpointGenerationIdSchema,
Expand Down Expand Up @@ -100,6 +101,47 @@ describe('MemoryCheckpointService', () => {
await expect(stat(join(fixture.memoryRoot, 'credentials.json'))).rejects.toMatchObject({ code: 'ENOENT' });
});

it('preserves the exact released memory-v1 generation and manifest bytes', () => {
const generation = MemoryCheckpointCodec.createGeneration({
scopeId,
generationId: MemoryCheckpointGenerationIdSchema.parse('generation-1'),
createdAt: '2026-08-26T00:00:00.000Z',
files: [MemoryCheckpointCodec.createFile('README.md', Buffer.from('# Memory\n'))],
});
const manifest = MemoryCheckpointCodec.createManifest({
generation,
committedAt: '2026-08-26T00:00:00.000Z',
});

expect(MemoryCheckpointCodec.serializeGeneration(generation)).toBe(`{
"kind": "heddle-memory-checkpoint-generation",
"schemaVersion": 1,
"scopeId": "memory-v1-f6c317c8362c7f6a261782e0c296239075d478fdbf9e861bc6a7a5c3ffc50a97",
"generationId": "generation-1",
"createdAt": "2026-08-26T00:00:00.000Z",
"files": [
{
"path": "README.md",
"contentBase64": "IyBNZW1vcnkK",
"byteLength": 9,
"sha256": "d7870cdadd1ac3b46461cce0776275aeb54f15f19338e597fafd0f277b1f0070"
}
]
}
`);
expect(MemoryCheckpointCodec.serializeManifest(manifest)).toBe(`{
"kind": "heddle-memory-checkpoint-manifest",
"schemaVersion": 1,
"scopeId": "memory-v1-f6c317c8362c7f6a261782e0c296239075d478fdbf9e861bc6a7a5c3ffc50a97",
"generationId": "generation-1",
"generationSha256": "e114fc81376990386a2bac155bd20dd7875c4595b056f7f3fa6a0790ef0d0a3b",
"fileCount": 1,
"totalBytes": 9,
"committedAt": "2026-08-26T00:00:00.000Z"
}
`);
});

it('rejects corrupt durable content before creating a local working copy', async () => {
const fixture = await createFixture();
const store = new InMemoryCheckpointStore();
Expand Down
225 changes: 225 additions & 0 deletions src/__tests__/unit/core/portable-directory-checkpoint.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
import {
lstat,
mkdir,
mkdtemp,
readFile,
rm,
symlink,
writeFile,
} from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, describe, expect, it } from 'vitest';
import {
PortableDirectoryCheckpointCodec,
PortableDirectoryCheckpointCorruptionError,
PortableDirectoryCheckpointPolicy,
PortableDirectoryCheckpointPolicyError,
PortableDirectoryCheckpointRestoreTargetError,
PortableDirectoryCheckpointService,
type PortableDirectoryCheckpointFile,
type PortableDirectoryCheckpointLimits,
} from '@/core/checkpoint/portable-directory/index.js';

const GENEROUS_LIMITS: PortableDirectoryCheckpointLimits = {
maxFileCount: 20,
maxFileBytes: 1024,
maxTotalBytes: 4096,
};

describe('PortableDirectoryCheckpointService', () => {
const temporaryRoots: string[] = [];

afterEach(async () => {
await Promise.all(temporaryRoots.splice(0).map(path => rm(path, { recursive: true, force: true })));
});

it('round-trips selected binary files in canonical path order', async () => {
const directoryRoot = await createDirectoryRoot();
await mkdir(join(directoryRoot, 'a'), { recursive: true });
await mkdir(join(directoryRoot, 'excluded'), { recursive: true });
await writeFile(join(directoryRoot, 'a.md'), Buffer.from('top-level'));
await writeFile(join(directoryRoot, 'a', 'binary.dat'), Buffer.from([0x00, 0xff, 0x10]));
await writeFile(join(directoryRoot, 'z.txt'), Buffer.from('last'));
await writeFile(join(directoryRoot, 'excluded', 'secret.txt'), Buffer.from('not portable'));
const service = createService(directoryRoot, {
includeFile: path => !path.startsWith('excluded/'),
});

const files = await service.capture();

expect(files.map(file => file.path)).toEqual(['a.md', 'a/binary.dat', 'z.txt']);
await rm(directoryRoot, { recursive: true });
await expect(service.restore(files)).resolves.toBe(directoryRoot);
await expect(readFile(join(directoryRoot, 'a', 'binary.dat')))
.resolves.toEqual(Buffer.from([0x00, 0xff, 0x10]));
await expect(lstat(join(directoryRoot, 'excluded'))).rejects.toMatchObject({ code: 'ENOENT' });
});

it('recognizes canonical relative paths across POSIX and Windows syntax', () => {
expect(PortableDirectoryCheckpointCodec.isSafeRelativePath('notes/current.md')).toBe(true);
expect([
'',
'.',
'../escape.txt',
'notes/../escape.txt',
'/absolute.txt',
'C:/absolute.txt',
'C:drive-relative.txt',
'notes\\windows.txt',
'notes//duplicate-separator.txt',
].map(path => PortableDirectoryCheckpointCodec.isSafeRelativePath(path)))
.toEqual([false, false, false, false, false, false, false, false, false]);
});

it.runIf(process.platform !== 'win32')('rejects unsafe relative names and symbolic links during capture', async () => {
const unsafeRoot = await createDirectoryRoot();
await writeFile(join(unsafeRoot, 'unsafe\\name.txt'), 'unsafe');
await expect(createService(unsafeRoot).capture()).rejects.toMatchObject({
code: 'PORTABLE_DIRECTORY_CHECKPOINT_CAPTURE_ERROR',
detail: expect.stringContaining('unsafe relative file path'),
});

const symlinkRoot = await createDirectoryRoot();
await writeFile(join(symlinkRoot, 'source.txt'), 'source');
await symlink(join(symlinkRoot, 'source.txt'), join(symlinkRoot, 'link.txt'));
await expect(createService(symlinkRoot).capture()).rejects.toMatchObject({
code: 'PORTABLE_DIRECTORY_CHECKPOINT_CAPTURE_ERROR',
detail: expect.stringContaining('symbolic links'),
});
});

it.each([
{
name: 'file count',
limits: { maxFileCount: 1, maxFileBytes: 10, maxTotalBytes: 20 },
files: [['one.txt', '1'], ['two.txt', '2']],
detail: 'file count 2 exceeds limit 1',
},
{
name: 'per-file bytes',
limits: { maxFileCount: 2, maxFileBytes: 2, maxTotalBytes: 20 },
files: [['large.txt', '123']],
detail: 'file large.txt has 3 bytes, exceeding limit 2',
},
{
name: 'total bytes',
limits: { maxFileCount: 2, maxFileBytes: 4, maxTotalBytes: 5 },
files: [['one.txt', '123'], ['two.txt', '456']],
detail: 'total bytes 6 exceed limit 5',
},
])('fails closed when capture exceeds the $name limit', async ({ limits, files, detail }) => {
const directoryRoot = await createDirectoryRoot();
await Promise.all(files.map(([path, content]) => writeFile(join(directoryRoot, path), content)));

await expect(createService(directoryRoot, { limits }).capture()).rejects.toMatchObject({
code: 'PORTABLE_DIRECTORY_CHECKPOINT_CAPTURE_ERROR',
detail,
});
});

it.each([
{
name: 'file count',
limits: { maxFileCount: 1, maxFileBytes: 10, maxTotalBytes: 20 },
files: [createFile('one.txt', '1'), createFile('two.txt', '2')],
detail: 'file count 2 exceeds limit 1',
},
{
name: 'per-file bytes',
limits: { maxFileCount: 2, maxFileBytes: 2, maxTotalBytes: 20 },
files: [createFile('large.txt', '123')],
detail: 'file large.txt has 3 bytes, exceeding limit 2',
},
{
name: 'total bytes',
limits: { maxFileCount: 2, maxFileBytes: 4, maxTotalBytes: 5 },
files: [createFile('one.txt', '123'), createFile('two.txt', '456')],
detail: 'total bytes 6 exceed limit 5',
},
])('fails closed when restore exceeds the $name limit', async ({ limits, files, detail }) => {
const directoryRoot = await createAbsentDirectoryRoot();

await expect(createService(directoryRoot, { limits }).restore(files)).rejects.toMatchObject({
code: 'PORTABLE_DIRECTORY_CHECKPOINT_CORRUPTION',
detail,
});
await expect(lstat(directoryRoot)).rejects.toMatchObject({ code: 'ENOENT' });
});

it('validates integrity, policy, and path conflicts before creating the target', async () => {
const corruptRoot = await createAbsentDirectoryRoot();
const valid = createFile('valid.txt', 'valid');
await expect(createService(corruptRoot).restore([
{ ...valid, contentBase64: Buffer.from('corrupt').toString('base64') },
])).rejects.toBeInstanceOf(PortableDirectoryCheckpointCorruptionError);
await expect(lstat(corruptRoot)).rejects.toMatchObject({ code: 'ENOENT' });

const unsafeRoot = await createAbsentDirectoryRoot();
await expect(createService(unsafeRoot).restore([{ ...valid, path: '../escape.txt' }]))
.rejects.toBeInstanceOf(PortableDirectoryCheckpointCorruptionError);
await expect(lstat(unsafeRoot)).rejects.toMatchObject({ code: 'ENOENT' });

const conflictRoot = await createAbsentDirectoryRoot();
await expect(createService(conflictRoot).restore([
createFile('entry', 'file'),
createFile('entry/child.txt', 'child'),
])).rejects.toMatchObject({
code: 'PORTABLE_DIRECTORY_CHECKPOINT_CORRUPTION',
detail: 'file path conflicts with descendant: entry',
});
await expect(lstat(conflictRoot)).rejects.toMatchObject({ code: 'ENOENT' });
});

it('restores through an empty target but does not merge with a populated target', async () => {
const emptyRoot = await createDirectoryRoot();
await expect(createService(emptyRoot).restore([createFile('restored.txt', 'restored')]))
.resolves.toBe(emptyRoot);
await expect(readFile(join(emptyRoot, 'restored.txt'), 'utf8')).resolves.toBe('restored');

const populatedRoot = await createDirectoryRoot();
await writeFile(join(populatedRoot, 'existing.txt'), 'keep');

await expect(createService(populatedRoot).restore([createFile('next.txt', 'next')]))
.rejects.toBeInstanceOf(PortableDirectoryCheckpointRestoreTargetError);
await expect(readFile(join(populatedRoot, 'existing.txt'), 'utf8')).resolves.toBe('keep');
});

it('requires every policy limit to be an explicit non-negative safe integer', () => {
expect(() => new PortableDirectoryCheckpointPolicy({
includeFile: () => true,
limits: { maxFileCount: Number.POSITIVE_INFINITY, maxFileBytes: 1, maxTotalBytes: 1 },
})).toThrow(PortableDirectoryCheckpointPolicyError);
});

async function createDirectoryRoot(): Promise<string> {
const root = await mkdtemp(join(tmpdir(), 'heddle-portable-directory-'));
temporaryRoots.push(root);
return root;
}

async function createAbsentDirectoryRoot(): Promise<string> {
const parent = await createDirectoryRoot();
return join(parent, 'restored');
}
});

function createService(
directoryRoot: string,
options: {
limits?: PortableDirectoryCheckpointLimits;
includeFile?: (path: string) => boolean;
} = {},
): PortableDirectoryCheckpointService {
return new PortableDirectoryCheckpointService(
directoryRoot,
new PortableDirectoryCheckpointPolicy({
limits: options.limits ?? GENEROUS_LIMITS,
includeFile: options.includeFile ?? (() => true),
}),
);
}

function createFile(path: string, content: string): PortableDirectoryCheckpointFile {
return PortableDirectoryCheckpointCodec.createFile(path, Buffer.from(content));
}
17 changes: 17 additions & 0 deletions src/advanced.ts
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,23 @@ export {
} from './core/observability/index.js';
export { buildSystemPrompt } from './core/prompts/system-prompt.js';

// --- Building blocks: portable directory checkpoints -----------------------
export {
PortableDirectoryCheckpointCaptureError,
PortableDirectoryCheckpointCodec,
PortableDirectoryCheckpointCorruptionError,
PortableDirectoryCheckpointFileSchema,
PortableDirectoryCheckpointPolicy,
PortableDirectoryCheckpointPolicyError,
PortableDirectoryCheckpointRestoreTargetError,
PortableDirectoryCheckpointService,
} from './core/checkpoint/index.js';
export type {
PortableDirectoryCheckpointFile,
PortableDirectoryCheckpointLimits,
PortableDirectoryCheckpointPolicyOptions,
} from './core/checkpoint/index.js';

// --- Building blocks: memory & knowledge -----------------------------------
export { buildMemoryDomainSystemContext } from './core/memory/domain-prompt.js';
export {
Expand Down
1 change: 1 addition & 0 deletions src/core/checkpoint/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from './portable-directory/index.js';
Loading
Loading