Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/dual-runtime-layers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
"@effectmq/core": minor
---

Document Bun plus node-redis as a supported composition.

`TaskEngine.layer` stays the live graph. `TaskEngine.layerNoDeps` is the
compose path for a custom `RedisPool` or `BunCrypto`. Bun's built-in
`RedisClient` is not the supported adapter yet.
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Flat `src/` with a strict layering, top to bottom:
- **`TaskEvent.ts`** — public queue lifecycle event schemas.
- **`EngineRecord.ts` / `MessagePack.ts` / `RetrySchedule.ts`** — internal Redis record, binary codec, and retry-schedule concepts.

Wiring: `TaskEngine.layer()` is the complete zero-requirement Node live graph and intentionally retains Redis operational services. `TaskEngine.layerNoDeps()` is the custom-client layer that requires `RedisPool`.
Wiring: `TaskEngine.layer()` is the live graph (`NodeRedisPool` + `NodeCrypto`). `TaskEngine.layerNoDeps()` requires `RedisPool` for custom clients and for Bun plus `BunCrypto`. Run the program with `NodeRuntime` or `BunRuntime`.

`Worker` provides built-in bounded local concurrency, lease supervision,
maintenance, and graceful draining. It does not provide distributed/global
Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ Production modules import supported narrow `effect/*` subpaths. Public
Effect-returning functions pin exact success, error, and service channels and
use `Effect.fnUntraced` for reusable generator implementations. Services are
`Context.Service` classes with `@effectmq/core/<Service>` identifiers; optional
fiber-local values are `Context.Reference`s. Use `TaskEngine.layerNoDeps()` for
custom Redis composition and reserve `TaskEngine.layer()` for the complete Node
live graph.
fiber-local values are `Context.Reference`s. Use `TaskEngine.layer()` for the
live graph. Use `TaskEngine.layerNoDeps()` when the application supplies
`RedisPool`.

Edit `src/lua/taskEngine.lua`, then run `pnpm gen:lua`; never hand-edit the
generated TypeScript module. CI rejects generated drift. Add committed golden
Expand Down
42 changes: 33 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,18 @@ a scheduled report. It provides:
## Install

```bash
# Node
pnpm add @effectmq/core@0.3.0-rc.0 effect@4.0.0-beta.107 @effect/platform-node@4.0.0-beta.107

# Bun (node-redis is bundled; add platform-bun for BunRuntime and BunCrypto)
bun add @effectmq/core@0.3.0-rc.0 effect@4.0.0-beta.107 @effect/platform-bun@4.0.0-beta.107
```

> [!IMPORTANT]
> effectmq currently targets the Effect 4 beta and is not compatible with the
> stable Effect 3 release. Pin the versions shown above. Node.js 22.19 or newer
> is required; CI verifies Node.js 22 and 24.

The package includes its pooled `NodeRedisPool` implementation.
`@effect/platform-node` is only needed by these examples for `NodeRuntime`.
> is required; CI verifies Node.js 22 and 24. Bun plus node-redis is a
> supported composition. It is not yet a CI platform.

---

Expand Down Expand Up @@ -78,9 +80,9 @@ Redis startup and expected output, follow the

## Runtime setup

`TaskEngine.layer()` is the complete Node live graph: it provides the engine,
cryptographic identity generation, and the retained Redis pool, role, and
health services:
`TaskEngine.layer()` is the live graph: engine, cryptographic identity, and
the Redis pool, role, and health services. Run it with `NodeRuntime` or
`BunRuntime`. The Redis adapter stays `NodeRedisPool` on both.

```ts
import { TaskEngine } from "@effectmq/core";
Expand All @@ -90,8 +92,30 @@ const AppLayer = TaskEngine.layer({
});
```

Use `TaskEngine.layerNoDeps()` when composing a custom `RedisPool`
implementation. `NodeRedisPool.layer()` remains available independently and
Bun plus node-redis uses the same `TaskEngine.layer` call and
`BunRuntime.runMain`. Swap `NodeCrypto` for `BunCrypto` when you compose
yourself through `TaskEngine.layerNoDeps()`. Bun's built-in `RedisClient`
is not the supported adapter yet. It has no binary `send`.

```ts
import { Effect, Layer } from "effect";
import { BunCrypto, BunRuntime } from "@effect/platform-bun";
import { NodeRedisPool, TaskEngine } from "@effectmq/core";

const AppLayer = TaskEngine.layerNoDeps().pipe(
Layer.provideMerge(
Layer.merge(
NodeRedisPool.layer({ url: "redis://localhost:6379" }),
BunCrypto.layer,
),
),
);

Effect.void.pipe(Effect.provide(AppLayer), BunRuntime.runMain);
```

Use `TaskEngine.layerNoDeps()` when you bring your own `RedisPool`.
`NodeRedisPool.layer()` remains available independently and
accepts node-redis client options. It establishes separate producer, worker,
and maintenance pools when the Layer starts. It supports standalone Redis and
Sentinel; Redis Cluster fails startup because queue transitions use multi-key
Expand Down
25 changes: 21 additions & 4 deletions apps/docs/content/docs/reference/redis-and-runtime.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ TaskEngine.layer({ engine?, redis? }): Layer<LiveServices, LiveError>
```

Provides `TaskEngine`, `RedisPool`, `RedisConnectionRoles`,
`RedisConnectionHealth`, Effect Redis, and Crypto. It is the standard Node.js
live graph.
`RedisConnectionHealth`, Effect Redis, and Crypto. Run it with `NodeRuntime`
or `BunRuntime`.

Engine configuration:

Expand All @@ -27,8 +27,25 @@ Engine configuration:
TaskEngine.layerNoDeps(config?): Layer<TaskEngine, TaskEngineConfigurationError, RedisPool>
```

Requires an ambient custom `RedisPool`. It does not provide connection roles,
health, Effect Redis, or Crypto.
Requires an ambient `RedisPool`. It does not provide connection roles, health,
Effect Redis, or Crypto.

Bun plus `BunCrypto` is this composition:

```ts
import { Layer } from "effect"
import { BunCrypto } from "@effect/platform-bun"
import { NodeRedisPool, TaskEngine } from "@effectmq/core"

const AppLayer = TaskEngine.layerNoDeps().pipe(
Layer.provideMerge(
Layer.merge(
NodeRedisPool.layer({ url: "redis://127.0.0.1:6379" }),
BunCrypto.layer
)
)
)
```

## `NodeRedisPool.layer`

Expand Down
18 changes: 5 additions & 13 deletions apps/docs/content/docs/tutorials/getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,8 @@ Create `src/main.ts`:

```ts
import { NodeRuntime } from "@effect/platform-node"
import {
NodeRedisPool,
Task,
TaskEngine,
TaskQueue,
Worker
} from "@effectmq/core"
import { Console, Effect, Layer, Schema } from "effect"
import { Task, TaskEngine, TaskQueue, Worker } from "@effectmq/core"
import { Console, Effect, Schema } from "effect"

const Greet = Task.make({
name: "greet",
Expand All @@ -93,11 +87,9 @@ const Greet = Task.make({

const greetings = TaskQueue.make("tutorial-greetings", Greet)

const EngineLive = TaskEngine.layer().pipe(
Layer.provideMerge(
NodeRedisPool.layer({ url: "redis://127.0.0.1:6379" })
)
)
const EngineLive = TaskEngine.layer({
redis: { url: "redis://127.0.0.1:6379" }
})

const worker = Worker.make(
greetings,
Expand Down
17 changes: 10 additions & 7 deletions docs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,13 @@ coordination, ordinary removal, and administrative force removal. Prefer
`TaskQueue`, `Worker`, and `Scheduler` unless building tooling or an alternate
runtime.

- `TaskEngine.layer(config?)` is the zero-requirement Node live graph and retains
Redis operational services in its output.
- `TaskEngine.layerNoDeps(config?)` requires an ambient `RedisPool` for custom
client compositions.
- `TaskEngine.layer({ engine?, redis? })` is the live graph and retains Redis
operational services plus Crypto in its output.
- `TaskEngine.layerNoDeps(config?)` requires an ambient `RedisPool`. Use it for
a custom client or for Bun plus `BunCrypto`.
- Bun plus node-redis is `TaskEngine.layer` plus `BunRuntime`, or
`TaskEngine.layerNoDeps` composed with `NodeRedisPool.layer` and
`BunCrypto.layer`.
- Invalid configuration and Redis reply shapes use structured typed errors;
diagnostic strings are retained only as causes.

Expand All @@ -93,6 +96,6 @@ version, schema, value, size, and count errors. `Observability` exports Effect
metrics for depth/age/backlogs, Redis errors/reconnects/script reloads,
ownership loss, and retention failure.

Stable subpaths are `./NodeRedisPool`, `./Observability`, `./RedisPool`,
`./Scheduler`, `./StorageProtocol`, `./Task`, `./TaskEngine`, `./TaskEvent`,
`./TaskQueue`, `./TaskRecord`, and `./Worker`.
Stable subpaths are `./NodeRedisPool`, `./Observability`,
`./RedisPool`, `./Scheduler`, `./StorageProtocol`, `./Task`, `./TaskEngine`,
`./TaskEvent`, `./TaskQueue`, `./TaskRecord`, and `./Worker`.
18 changes: 11 additions & 7 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ package subpaths:
processing, and scheduling APIs;
- `TaskRecord` owns durable typed task records and `TaskEvent` owns lifecycle
events;
- `TaskEngine` owns atomic queue storage behavior;
- `RedisPool`, `NodeRedisPool`, `StorageProtocol`, and `Observability` own the
external client, live Node adapter, value protocol, and metrics boundaries.
- `TaskEngine` owns atomic queue storage behavior and the live graph;
- `RedisPool`, `NodeRedisPool`, `StorageProtocol`, and
`Observability` own the external client, node-redis adapter, value
protocol, and metrics boundaries.

Internal modules are deliberately not package subpaths. `MessagePack` owns the
binary transform, `EngineRecord` owns Redis-facing record schemas,
Expand All @@ -27,11 +28,14 @@ TaskEvent -> TaskRecord + EngineRecord + MessagePack
NodeRedisPool -> RedisPool + RedisReadiness + Observability
```

`TaskEngine.layer()` is the standard zero-requirement Node live graph. It
retains `TaskEngine`, `RedisPool`, `RedisConnectionRoles`,
`RedisConnectionHealth`, Effect Redis, and Crypto services. Custom Redis
`TaskEngine.layer()` is the live graph. It retains `TaskEngine`, `RedisPool`,
`RedisConnectionRoles`, `RedisConnectionHealth`, Effect Redis, and Crypto.
`TaskEngine.layerNoDeps()` requires an ambient `RedisPool`. Bun plus
node-redis uses `TaskEngine.layer` with `BunRuntime`, or `layerNoDeps`
composed with `NodeRedisPool.layer` and `BunCrypto.layer`. Custom Redis
integrations provide `RedisPool` to `TaskEngine.layerNoDeps()`.

Public queue declarations use named exact aliases for success, typed failure,
and required services. The strict test compiler pins `complete`, `completeOne`,
`decodeTask`, `wait`, `execute`, and both TaskEngine layer modes.
`decodeTask`, `wait`, `execute`, both TaskEngine layer modes, and the Bun plus
node-redis composition.
12 changes: 8 additions & 4 deletions openspec/specs/effect-module-architecture/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,16 +41,20 @@ Project runtime services SHALL use class-based service declarations with stable
- **THEN** it receives the documented absent default without requiring an extra layer

### Requirement: Service layers communicate dependency ownership
A service module SHALL expose `layerNoDeps` for construction that still requires upstream services and `layer` for the standard live composition with those dependencies supplied. The live layer SHALL retain upstream outputs only when they are intentionally part of its documented public service graph.
`TaskEngine.layer` SHALL provide the live graph and SHALL retain Redis operational services plus Crypto. `TaskEngine.layerNoDeps` SHALL require an ambient `RedisPool` and SHALL NOT select a process runtime or Redis client. A Bun plus node-redis application SHALL run `TaskEngine.layer` under `BunRuntime`, or compose `TaskEngine.layerNoDeps` with `NodeRedisPool.layer` and `BunCrypto.layer`.

#### Scenario: Application supplies a custom Redis pool
- **WHEN** an application uses the dependency-free engine layer constructor
- **WHEN** an application uses `TaskEngine.layer` or `TaskEngine.layerNoDeps`
- **THEN** the type system requires the Redis pool service from the application

#### Scenario: Application uses the standard live layer
- **WHEN** an application uses the fully wired engine layer
#### Scenario: Application uses the live layer
- **WHEN** an application uses `TaskEngine.layer`
- **THEN** it receives the documented engine and Redis operational services with no unresolved requirements

#### Scenario: Application uses Bun with node-redis
- **WHEN** an application composes `TaskEngine.layerNoDeps` with `NodeRedisPool.layer` and `BunCrypto.layer`
- **THEN** the composed layer has no unresolved requirements

### Requirement: Effect implementation style preserves contracts
Reusable Effectful functions SHALL use the project's traceable function wrapper convention and public or recursive functions SHALL declare exact return contracts. Production modules SHALL import Effect APIs through stable narrow subpaths.

Expand Down
7 changes: 7 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,18 @@
"author": "",
"license": "MIT",
"peerDependencies": {
"@effect/platform-bun": ">=4.0.0-beta.107",
"effect": ">=4.0.0-beta.107"
},
"peerDependenciesMeta": {
"@effect/platform-bun": {
"optional": true
}
},
"devDependencies": {
"@biomejs/biome": "2.5.1",
"@changesets/cli": "^2.31.0",
"@effect/platform-bun": "4.0.0-beta.107",
"@effect/vitest": "4.0.0-beta.107",
"@testcontainers/redis": "^12.0.3",
"@types/node": "^22.0.0",
Expand Down
16 changes: 16 additions & 0 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions scripts/check-architecture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,17 @@ const visit = (directory: string) => {
if (/\bDate\.now\(\)|\bcrypto\.randomUUID\(/.test(text)) {
failures.push(`${relative(root, path)} reads ambient time or randomness`);
}
const rel = relative(root, path);
const mayImportRuntime =
rel === "src/TaskEngine.ts" ||
rel === "src/NodeRedisPool.ts" ||
rel.startsWith("src/cli/");
if (
!mayImportRuntime &&
/from ["'](?:redis|@effect\/platform-node(?:\/[^"']*)?)["']/.test(text)
) {
failures.push(`${rel} imports redis or @effect/platform-node`);
}
}
};

Expand Down
27 changes: 25 additions & 2 deletions src/PublicContracts.test.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
import * as BunCrypto from "@effect/platform-bun/BunCrypto";
import * as Context from "effect/Context";
import type * as Crypto from "effect/Crypto";
import type * as Effect from "effect/Effect";
import type * as Layer from "effect/Layer";
import * as Layer from "effect/Layer";
import * as Schema from "effect/Schema";
import type * as Stream from "effect/Stream";
import type * as Redis from "effect/unstable/persistence/Redis";
import { expect, it } from "vitest";
import * as NodeRedisPool from "./NodeRedisPool.js";
import type * as RedisPool from "./RedisPool.js";
import * as TaskEngine from "./TaskEngine.js";
import * as TaskQueue from "./TaskQueue.js";
Expand Down Expand Up @@ -150,12 +154,29 @@ const compilePublicContracts = () => {

const layerNoDeps = TaskEngine.layerNoDeps();
const liveLayer = TaskEngine.layer();
const bunNodeRedisLayer = TaskEngine.layerNoDeps().pipe(
Layer.provideMerge(Layer.merge(NodeRedisPool.layer(), BunCrypto.layer)),
);
type LayerNoDepsRequirement = Expect<
Equal<Layer.Services<typeof layerNoDeps>, RedisPool.RedisPool>
>;
type LiveLayerRequirement = Expect<
Equal<Layer.Services<typeof liveLayer>, never>
>;
type LiveLayerSuccess = Expect<
Equal<
Layer.Success<typeof liveLayer>,
| TaskEngine.TaskEngine
| RedisPool.RedisPool
| RedisPool.RedisConnectionRoles
| NodeRedisPool.RedisConnectionHealth
| Redis.Redis
| Crypto.Crypto
>
>;
type BunNodeRedisRequirement = Expect<
Equal<Layer.Services<typeof bunNodeRedisLayer>, never>
>;

return undefined as unknown as
| CompleteSuccess
Expand All @@ -179,7 +200,9 @@ const compilePublicContracts = () => {
| RejectAnyServices
| FailedEventError
| LayerNoDepsRequirement
| LiveLayerRequirement;
| LiveLayerRequirement
| LiveLayerSuccess
| BunNodeRedisRequirement;
};

it("pins public Effect and Layer channels at compile time", () => {
Expand Down
Loading