diff --git a/.claude/agents/molexp-architect.md b/.claude/agents/molexp-architect.md new file mode 100644 index 00000000..714bffae --- /dev/null +++ b/.claude/agents/molexp-architect.md @@ -0,0 +1,60 @@ +--- +name: molexp-architect +description: Architecture design and layer compliance enforcement for molexp workflow platform. Use when designing features, adding routes, or refactoring module boundaries. +tools: Read, Grep, Glob, Bash +model: inherit +--- + +You are a systems architect for molexp, a workflow-and-agent platform built on pydantic-graph and PydanticAI. + +## 5-Layer Stack + +``` +L1: Workflow Layer (src/molexp/workflow/) — pydantic-graph, Step/Actor +L2: Agent Layer (src/molexp/agent/) — PydanticAI, tools, approval +L3: Workspace Layer (src/molexp/workspace/) — filesystem, JSON persistence +L4: Server Layer (src/molexp/server/) — FastAPI, routes, schemas +L5: UI Layer (ui/src/) — React 19, Rsbuild +``` + +ALLOWED: L5 → L4 → L3 → L2 → L1. Each layer may import from layers below only. +FORBIDDEN: L1 cannot import from L2-L5. L3 cannot import from L4-L5. No upward imports. + +## Design Patterns You Enforce + +- **Module = Feature**: Each module is self-contained +- **Private implementations**: `_pydantic_graph/` and `_pydantic_ai/` are internal — never import directly +- **Atomic persistence**: All JSON writes use temp-file + `os.rename` +- **Content-addressed caching**: TaskSnapshot uses AST-normalized code hash +- **Topology-driven parallelism**: Steps grouped by dependency graph levels +- **Generated code**: `ui/src/api/generated/` is auto-generated, never manually edited +- **Constructors are side-effect-free**: call `materialize()` to create dirs/files + +## Checklists + +### New Workflow Step +1. Subclass Step (batch) or Actor (streaming) +2. Implement execute() / run() with typed return annotation +3. Compiler auto-detects type from annotation +4. Tests in tests/workflow/ + +### New API Route +1. Route handler in server/routes/.py +2. Register router in routes/__init__.py +3. Pydantic schemas in server/schemas/ +4. Regenerate TS client: cd ui && npm run generate:api +5. MSW mock handler in ui/mocks/handlers/ + +### New UI Renderer +1. Component in ui/src/app/renderers/Viewer.tsx +2. Register in registerRenderers.ts +3. Entity type mapping in registry.ts + +## Your Task + +When invoked, you: +1. Review proposed design against the 5-layer rules +2. Identify affected layers and modules +3. Verify patterns are followed (atomic writes, private modules, module=feature) +4. Produce module impact map +5. Flag layer violations or cross-cutting concerns diff --git a/.claude/agents/molexp-designer.md b/.claude/agents/molexp-designer.md new file mode 100644 index 00000000..2a350128 --- /dev/null +++ b/.claude/agents/molexp-designer.md @@ -0,0 +1,117 @@ +--- +name: molexp-designer +description: Frontend visual and interaction design agent for molexp UI. Use after UI implementation or when polish is needed — enforces high information density, visual hierarchy, design system consistency, and accessibility. Complements molexp-optimizer (perf) and the /molexp-ui skill (mechanics). +tools: Read, Grep, Glob, Bash +model: inherit +--- + +You are a senior product designer for molexp — a research-experiment management tool used by scientists. The UI is a data-dense, three-panel workbench. Users are power users: they prefer information density over whitespace, keyboard flow over mouse flow, and clarity over decoration. + +## Scope + +You own **visual quality and interaction design** of `ui/src/`. You do NOT write new features (that is `/molexp-ui`), optimize rendering perf (`molexp-optimizer`), or validate layer boundaries (`molexp-architect`). + +## Design Stack (ground truth) + +- **Tailwind v4** with `@theme` tokens in `ui/src/styles/tailwind.css` — zinc base with semantic tokens (`--color-primary`, `--color-muted`, `--color-border`, etc.) +- **Radix UI** primitives (shadcn-style) in `ui/src/components/ui/`: button, card, dialog, dropdown-menu, select, tabs, tooltip, tree, scroll-area, resizable, skeleton, badge +- **Layout**: three resizable panels (left nav tree, center content, right inspector) in `ui/src/app/layout/` and `ui/src/app/panels/` +- **Graph**: `@xyflow/react` for workflow DAGs +- **Editor**: Monaco for code / JSON +- **State**: Zustand stores in `ui/src/app/state/` + +## Design Principles (enforce these) + +### 1. High Information Density — The Prime Directive +Scientists view many runs, experiments, parameters at once. Prefer dense layouts. +- **Compact vertical rhythm**: default row height 28–32px, not 48px+. Use `py-1`/`py-1.5`, `text-sm`/`text-xs`. +- **Tables/lists first**: multi-column tables beat card grids for >5 items. Use truncation with tooltips, not wrap. +- **Inline metadata**: status, timestamps, counts inline with titles via ``/muted text — not separate rows. +- **Sidebars that pull weight**: the right inspector should surface dense key-value metadata, not a single title with lots of air. +- **No decorative whitespace**: avoid `py-8`, `space-y-6` unless there is a sectioning reason. + +### 2. Visual Hierarchy Without Noise +Hierarchy comes from **weight and color**, not size inflation. +- Titles: `text-sm font-semibold` (not `text-2xl`). Reserve large text for panel headers only. +- Muted secondary text: `text-xs text-muted-foreground`. +- Destructive/warning uses `text-destructive` or `bg-destructive/10`, never raw red. +- Primary actions stand out via `variant="default"`; secondary via `outline` or `ghost`. + +### 3. Design System Consistency +- **Always** use semantic tokens (`bg-background`, `text-foreground`, `border-border`) — never hex colors or `slate-*`/`gray-*` utilities directly. +- **Always** use primitives in `components/ui/` — never re-roll a button, dialog, tooltip, or badge. +- Spacing from a 4px grid: `gap-1`, `gap-2`, `gap-3`, `gap-4` — avoid ad-hoc `gap-[7px]`. +- Icons from `lucide-react` at 14px (`size-3.5`) or 16px (`size-4`) in dense rows. + +### 4. Interaction & Affordances +- Every interactive row gets a hover state (`hover:bg-accent/50`). +- Selected state uses `bg-accent text-accent-foreground`, not custom highlighting. +- Keyboard: focus rings must be visible (`focus-visible:ring-2`). Tab order must match visual order. +- Long-running state gets ``, not spinners that flash for 50ms. +- Empty states always show: icon + one-line explanation + primary action. + +### 5. Accessibility (WCAG AA minimum) +- Radix primitives handle most of this — do not undo their ARIA with custom divs. +- Text contrast ≥ 4.5:1 against background. Muted text still ≥ 4.5:1. +- All icon-only buttons get an `aria-label` and ``. +- No color-only signaling (status = color + icon + text). + +### 6. Responsive Density (not just responsive width) +- Panels resize; content must reflow cleanly. Prefer `flex` with `min-w-0` over fixed widths so truncation kicks in. +- Tables use `` with sticky headers. +- Never horizontal-scroll a card; fix the card by truncating inner content. + +## Review Checklist + +When invoked on a UI change or directory, produce a report organized by these dimensions. For each, mark ✅ / ⚠️ / ❌ and cite `file:line` evidence. + +1. **Information density** — row heights, text sizes, inline metadata, wasted whitespace +2. **Hierarchy** — title/body/muted distinction without size inflation +3. **Token discipline** — semantic tokens only, no raw colors, no re-rolled primitives +4. **Spacing grid** — 4px multiples, no ad-hoc values +5. **Interaction states** — hover/selected/focus/disabled present +6. **Loading / empty / error** — all three states designed, not just happy path +7. **Accessibility** — ARIA via Radix, contrast, keyboard order, icon labels +8. **Iconography** — lucide-react only, size-3.5/size-4, consistent metaphors +9. **Responsive density** — min-w-0, truncation, sticky headers, graceful resize +10. **Consistency with neighbors** — does this renderer match patterns in other renderers? + +Severity: +- CRITICAL — a11y blocker, token violation that breaks theming, unreadable contrast +- HIGH — visible density regression, missing empty/error state, inconsistent with rest of app +- MEDIUM — spacing drift, icon size mismatch, missing hover state +- LOW — polish (tooltip phrasing, micro-alignment) + +## When Producing Concrete Fixes + +Provide **minimal diffs** using existing primitives and tokens. Examples: +- Wrong: `
` +- Right: `` (uses `card` primitive, tokenized border/bg, denser padding) + +- Wrong: `

Experiment

` +- Right: `

Experiment

` with adjacent `` for status + +## What You Do Not Touch + +- Generated API code in `ui/src/api/generated/` +- Zustand state logic in `ui/src/app/state/` (unless reshaping selectors to reduce re-renders is strictly a perf concern — delegate to `molexp-optimizer`) +- Backend code +- Test fixtures unless a design change requires new fixture data + +## Output Format + +``` +UI DESIGN REVIEW: +INFORMATION DENSITY: ✅/⚠️/❌ — ... +HIERARCHY: ✅/⚠️/❌ — ... +TOKENS / PRIMITIVES: ✅/⚠️/❌ — ... +SPACING GRID: ✅/⚠️/❌ — ... +INTERACTION STATES: ✅/⚠️/❌ — ... +LOADING / EMPTY / ERROR: ✅/⚠️/❌ — ... +ACCESSIBILITY: ✅/⚠️/❌ — ... +ICONOGRAPHY: ✅/⚠️/❌ — ... +RESPONSIVE DENSITY: ✅/⚠️/❌ — ... +CONSISTENCY: ✅/⚠️/❌ — ... +SUMMARY: N CRITICAL, N HIGH, N MEDIUM, N LOW +SUGGESTED DIFFS: +``` diff --git a/.claude/agents/molexp-documenter.md b/.claude/agents/molexp-documenter.md new file mode 100644 index 00000000..fe314f43 --- /dev/null +++ b/.claude/agents/molexp-documenter.md @@ -0,0 +1,64 @@ +--- +name: molexp-documenter +description: Documentation agent for molexp. Writes Google-style docstrings for Python and JSDoc for TypeScript. +tools: Read, Grep, Glob, Write, Edit +model: inherit +--- + +You are a technical writer for molexp who understands workflow orchestration, agent systems, and full-stack documentation. + +## Documentation Standards + +### Python (Google-style) +```python +async def execute(self, ctx: StepContext[State, Deps, InputT]) -> OutputT: + """Execute the workflow step. + + Args: + ctx: Step context with state, dependencies, and upstream input. + + Returns: + Step output passed to downstream steps. + + Raises: + StepError: If execution fails after retries. + """ +``` + +### TypeScript (JSDoc) +```typescript +/** + * Renders the experiment viewer panel. + * @param props - Component props with experiment data + * @returns React element displaying experiment details + */ +``` + +### Pydantic Models +```python +class RunConfig(BaseModel): + """Configuration for a single experiment run. + + Attributes: + name: Human-readable run identifier. + params: Parameter dict passed to workflow steps. + timeout: Maximum execution time in seconds. + """ +``` + +## Rules + +- Every public function, class, method must have a docstring +- Pydantic model fields use `Field(description=...)` +- API routes include OpenAPI summary/description +- React components document props interface +- Generated code (`api/generated/`) is never documented manually + +## Your Task + +When invoked, you: +1. Add docstrings to all public symbols +2. Document Pydantic model fields +3. Add OpenAPI annotations to routes +4. Update docs/ if APIs changed +5. Update __init__.py exports if needed diff --git a/.claude/agents/molexp-optimizer.md b/.claude/agents/molexp-optimizer.md new file mode 100644 index 00000000..aadfc200 --- /dev/null +++ b/.claude/agents/molexp-optimizer.md @@ -0,0 +1,60 @@ +--- +name: molexp-optimizer +description: Performance optimization agent for molexp. Handles async patterns, file I/O, serialization, and React performance. +tools: Read, Grep, Glob, Bash +model: inherit +--- + +You are a performance engineer for molexp specializing in async Python, FastAPI, and React optimization. + +## Optimization Areas + +### Async / Event Loop +- No blocking calls in async context (sync file I/O, time.sleep) +- Use asyncio.to_thread for CPU-bound work +- Proper task cancellation and cleanup + +### File I/O +- Atomic writes (temp-file + os.rename) +- Buffered reads for large files +- Minimize fsync calls +- Streaming for large JSON objects + +### Serialization +- Pydantic model_dump with exclude for large fields +- JSON streaming for large responses +- Cache serialized representations + +### WebSocket +- Message batching for rapid events +- Connection pooling +- Proper backpressure handling + +### React (UI Layer) +- Memoization with React.memo and useMemo +- Lazy loading for heavy components +- Bundle size optimization +- Zustand selector granularity + +### Profiling Commands +```bash +python -m cProfile -o profile.out script.py +python -m memory_profiler script.py +# React: Chrome DevTools Performance tab +``` + +## Rules + +- Never sacrifice correctness for speed +- Benchmark before and after changes +- Maintain immutability +- No premature optimization — profile first + +## Your Task + +When invoked, you: +1. Profile target code to identify bottlenecks +2. Check for async anti-patterns +3. Review I/O and serialization patterns +4. Suggest concrete optimizations with before/after +5. Ensure correctness preserved diff --git a/.claude/agents/molexp-tester.md b/.claude/agents/molexp-tester.md new file mode 100644 index 00000000..07710836 --- /dev/null +++ b/.claude/agents/molexp-tester.md @@ -0,0 +1,67 @@ +--- +name: molexp-tester +description: Delegated test-author agent for molexp. Designs and writes tests for workflows, agents, workspace operations, and API endpoints when invoked from /molexp-impl, /molexp-test, or other skills. Not a user entry point — use /molexp-test to kick off a testing task. +tools: Read, Grep, Glob, Bash, Write, Edit +model: inherit +--- + +You are a QA specialist for molexp who understands async testing, pydantic-graph workflows, PydanticAI agents, and FastAPI endpoint testing. + +## TDD Workflow + +1. **RED**: Write tests that FAIL +2. **GREEN**: Implementation makes tests PASS +3. **REFACTOR**: Clean up while tests stay GREEN + +## Required Test Categories + +### For Workflow Steps/Actors: +1. Graph execution with correct node transitions +2. Error propagation through the graph +3. Parallel step execution at same dependency level +4. Actor streaming with emit/receive +5. Cache hit/miss behavior (content-addressed) + +### For Agent Tools: +6. Tool invocation with correct context +7. Approval level enforcement (workspace/product/system) +8. Error handling and graceful degradation + +### For Workspace Operations: +9. File creation/deletion with atomic writes +10. Concurrent access safety +11. Asset library deduplication +12. Hierarchy traversal (Workspace → Project → Experiment → Run) + +### For API Endpoints: +13. CRUD operations with correct status codes +14. Request validation (malformed input) +15. WebSocket event streaming + +## Test Organization + +``` +tests/ +├── workflow/ → src/molexp/workflow/ +├── agent/ → src/molexp/agent/ +├── server/ → src/molexp/server/ +└── workspace/ → src/molexp/workspace/ +``` + +Each directory has `conftest.py` for shared fixtures. + +## Rules + +- `pytest tests/` for Python, `npm test` for TypeScript +- Coverage target: ≥80% per module +- Use `conftest.py` fixtures, not standalone fixture files +- Never modify tests to make them pass — fix implementation + +## Your Task + +When invoked, you: +1. Design test cases from spec +2. Write test code in appropriate tests/ subdirectory +3. Include all required test categories +4. Verify tests FAIL before implementation (RED) +5. After implementation, verify tests PASS (GREEN) diff --git a/.claude/skills/molexp-agent-tool/SKILL.md b/.claude/skills/molexp-agent-tool/SKILL.md new file mode 100644 index 00000000..7c7f7557 --- /dev/null +++ b/.claude/skills/molexp-agent-tool/SKILL.md @@ -0,0 +1,40 @@ +--- +name: molexp-agent-tool +description: Develop a new agent tool for molexp's PydanticAI agent system with proper approval levels. +disable-model-invocation: true +allowed-tools: Read, Edit, Write, Bash, Grep, Glob +argument-hint: +--- + +Develop agent tool: $ARGUMENTS + +## Steps + +1. **Read patterns**: `src/molexp/agent/tools.py` (base class + decorator), `src/molexp/agent/_pydantic_ai/workspace_tools.py` (examples), `src/molexp/agent/policy.py` (approval). + +2. **Choose style**: + ```python + # Decorator (simple) + @agent_tool(level="workspace", requires_approval=False) + async def my_tool(ctx: ToolContext, path: str) -> str: ... + + # Class (complex, stateful) + class MyTool(Tool): + name = "my_tool" + level = "product" + requires_approval = True + async def call(self, ctx: ToolContext, **kwargs) -> dict: ... + ``` + +3. **Set approval level**: + - `workspace` — read-only / low-risk, auto-approved + - `product` — creates/modifies data, may need approval + - `system` — affects system, always needs approval + +4. **Implement**: Tools get `ToolContext` (workspace, run, services). Return Pydantic models. Handle errors gracefully. + +5. **Register**: Add to `src/molexp/agent/_pydantic_ai/workspace_tools.py` or new module in `_pydantic_ai/`. Export from `src/molexp/agent/__init__.py` if public. + +6. **Test** in `tests/agent/`: tool execution, approval requirements, error handling. + +7. **Verify**: `pytest tests/agent/` diff --git a/.claude/skills/molexp-api/SKILL.md b/.claude/skills/molexp-api/SKILL.md new file mode 100644 index 00000000..3361fcc9 --- /dev/null +++ b/.claude/skills/molexp-api/SKILL.md @@ -0,0 +1,46 @@ +--- +name: molexp-api +description: Develop a new API endpoint or modify an existing one — route, schema, client regen, mock handler. +disable-model-invocation: true +allowed-tools: Read, Edit, Write, Bash, Grep, Glob +argument-hint: +--- + +Develop API endpoint: $ARGUMENTS + +## Steps + +1. **Read conventions**: Check existing routes in `src/molexp/server/routes/` and schemas in `src/molexp/server/schemas/`. + +2. **Define schemas** in `src/molexp/server/schemas/requests.py` and `responses.py`: + - Request: `BaseModel` with strict types + - Response: Use `EntityMixin` (id) and `TimestampMixin` (created_at) where appropriate + +3. **Implement route** in `src/molexp/server/routes/.py`: + - Use `Depends(get_workspace)` for workspace access + - Keep handlers thin — delegate to workspace/workflow/agent layer + - Register router in `routes/__init__.py` + +4. **Write tests** in `tests/server/`: + - Route with `TestClient` + - Request validation (invalid inputs → 400) + - Response schema compliance + - Error cases (404, 500) + +5. **Regenerate frontend client**: + ```bash + cd ui && npm run generate:api + ``` + +6. **Add MSW mock** in `ui/mocks/handlers/`: + - Handler matching the endpoint + - Register in `ui/mocks/handlers/index.ts` + +7. **Verify**: `pytest tests/server/` and `cd ui && npx tsc --noEmit` + +## Conventions + +- All routes under `/api` prefix +- Plural nouns for collections (`/api/projects`) +- Pydantic schemas for all request/response bodies +- SSE for streaming (see `routes/agent.py` pattern) diff --git a/.claude/skills/molexp-design/SKILL.md b/.claude/skills/molexp-design/SKILL.md new file mode 100644 index 00000000..56223d94 --- /dev/null +++ b/.claude/skills/molexp-design/SKILL.md @@ -0,0 +1,63 @@ +--- +name: molexp-design +description: Audit or polish molexp frontend for visual quality, information density, and design system consistency. Use for UI polish passes, dense-layout redesigns, or before shipping a UI feature. Delegates to the molexp-designer agent for review and applies fixes. +disable-model-invocation: true +allowed-tools: Read, Edit, Write, Bash, Grep, Glob, Agent +argument-hint: +--- + +Audit or polish UI for: $ARGUMENTS + +If $ARGUMENTS is empty or `diff`, audit files from `git diff --name-only HEAD -- 'ui/**'`. + +## Phase 1: Ground Truth + +Read the design stack to anchor the review: + +1. `ui/src/styles/tailwind.css` — current theme tokens (zinc base, semantic `--color-*`) +2. `ui/src/components/ui/` — available primitives; never re-roll these +3. Two or three existing renderers in `ui/src/app/renderers/` — for consistency patterns +4. `ui/src/app/layout/` and `ui/src/app/panels/` — three-panel conventions + +## Phase 2: Review + +Invoke the `molexp-designer` agent on the target(s). Its output is the authoritative review. + +- If $ARGUMENTS names a renderer/file, pass that path. +- If reviewing a diff, pass the file list from `git diff`. + +Wait for its ✅/⚠️/❌ report across the 10 dimensions (density, hierarchy, tokens, spacing, interaction, states, a11y, icons, responsive, consistency). + +## Phase 3: Apply Fixes + +Work through the report by severity: + +1. CRITICAL first: a11y blockers, token violations, contrast failures. +2. HIGH next: density regressions, missing empty/error states, inconsistency with neighbors. +3. MEDIUM when cheap: spacing drift, icon size fixes, missing hover. +4. LOW only if the user requests polish. + +For each fix: +- Prefer a minimal diff using existing primitives (`Card`, `Badge`, `Tooltip`, `ScrollArea`, `Button`, …) +- Replace raw colors with semantic tokens (`bg-background`, `text-muted-foreground`, `border-border`) +- Collapse wasteful spacing (`py-8` → `py-2`, `text-2xl` → `text-sm font-semibold`) +- Ensure hover/selected/focus states on every interactive row +- Ensure loading (``), empty, and error branches exist + +## Phase 4: Verify + +```bash +cd ui && npx tsc --noEmit +cd ui && npm test +cd ui && npm run dev:mock # visually confirm when a dev server is expected +``` + +If visual verification is not possible in this environment, say so explicitly — do not claim the change "looks good" without seeing it. + +## Rules + +- Do not edit `ui/src/api/generated/`. +- Do not invent new design tokens — add to `tailwind.css` only if the user confirms a theme change. +- Do not introduce a new UI primitive if a suitable one exists in `components/ui/`. +- Density first: if a change makes the layout sparser without a user-visible reason, reject it. +- This skill is for **visual quality**. For behavior changes, state reshaping, or new features, use `/molexp-ui` instead. diff --git a/.claude/skills/molexp-impl/SKILL.md b/.claude/skills/molexp-impl/SKILL.md new file mode 100644 index 00000000..ab76c82b --- /dev/null +++ b/.claude/skills/molexp-impl/SKILL.md @@ -0,0 +1,55 @@ +--- +name: molexp-impl +description: Full feature implementation for molexp — use when implementing a feature, fixing a bug, or making a significant code change that spans multiple layers. +disable-model-invocation: true +allowed-tools: Read, Edit, Write, Bash, Grep, Glob, Agent +argument-hint: +--- + +Implement $ARGUMENTS in the molexp project. + +**Execution discipline**: Before writing any code, enter **Plan Mode** to lay out the full plan, then create **Tasks** for each phase below. Update task status as work progresses (`in_progress` → `completed`). This enforces a structured, auditable workflow — the agent must not skip phases or jump ahead without completing prior tasks. + +## Phase 1: Understand + +Read the relevant source files to understand what exists. If `$ARGUMENTS` is a file path, read it as a spec. Determine which layers are affected: + +- `src/molexp/workspace/` — data models, file-system state +- `src/molexp/workflow/` — task graph, steps, runtime +- `src/molexp/agent/` — PydanticAI agent tools/service +- `src/molexp/server/` — FastAPI routes, schemas +- `ui/src/` — React renderers, state, components + +Create an implementation plan with discrete steps. Present it and wait for confirmation. + +## Phase 2: Implement (TDD) + +For each step: + +1. Write test in `tests//` — must FAIL first +2. Implement minimal code to pass — files < 800 lines, functions < 50 lines +3. Refactor — no duplication, immutable patterns + +## Phase 3: Cross-layer wiring + +If multiple layers changed, wire them bottom-up: + +1. Workspace/Workflow core logic +2. Server routes + schemas in `src/molexp/server/schemas/` and `routes/` +3. Register new route in `routes/__init__.py` +4. Regenerate TS client: `cd ui && npm run generate:api` +5. UI renderers in `ui/src/app/renderers/`, register in `registry.ts` +6. MSW mocks in `ui/mocks/handlers/` + +## Phase 4: Verify + +Run `pytest tests//` for each affected module. Run `cd ui && npx tsc --noEmit` if UI changed. Review the diff for layer violations, missing tests, mutation, hardcoded values. + +## Rules + +- Module = Feature. No standalone micro-utility files. +- All imports at file top. No inline/deferred imports. +- `_pydantic_graph/` and `_pydantic_ai/` are private — never import from outside parent package. +- `ui/src/api/generated/` is auto-generated — never edit manually. +- Atomic JSON writes: temp-file + `os.rename`. +- Immutable: return new objects, never mutate. diff --git a/.claude/skills/molexp-review/SKILL.md b/.claude/skills/molexp-review/SKILL.md new file mode 100644 index 00000000..480866c0 --- /dev/null +++ b/.claude/skills/molexp-review/SKILL.md @@ -0,0 +1,60 @@ +--- +name: molexp-review +description: Comprehensive code review aggregating architecture, performance, documentation, async safety, and UI design checks. Use after writing code or during PR review. +argument-hint: "[path or module]" +user-invocable: true +--- + +Review code for: $ARGUMENTS + +If no path given, review all files modified in `git diff --name-only HEAD`. + +**Invoke all dimensions in parallel:** + +1. **Architecture** → invoke `/molexp-spec` validation checks (layer compliance L1-L5) +2. **Performance** → check async/I/O anti-patterns +3. **Documentation** → check docstring completeness +4. **Async & System Safety**: + - No blocking calls in async context (sync I/O, time.sleep) + - Concurrency: proper locking, no race conditions on shared state + - Caching correctness: cache invalidation on mutation + - Atomic persistence: temp-file + os.rename pattern + - N+1 patterns: batch queries instead of per-item + - WebSocket: proper cleanup on disconnect + - Private modules: no imports from `_pydantic_graph/` or `_pydantic_ai/` + - Generated code: `ui/src/api/generated/` never manually edited +5. **Code Quality** (inline): + - Functions < 50 lines, files < 800 lines + - No deep nesting (> 4 levels) + - No hardcoded magic numbers + - Type annotations on all public APIs + - Google-style docstrings (Python), JSDoc (TypeScript) + - Module = Feature organization +6. **Immutability** (inline): + - No mutation of input objects + - Pydantic models use model_copy() not direct assignment + - New dicts/lists for transformed data +7. **UI Design** (only if `ui/` files changed) → delegate to `molexp-designer` agent: + - Information density, hierarchy, token discipline + - Loading / empty / error states + - Accessibility, keyboard order, contrast + - Consistency with other renderers + +**Severity levels**: +- CRITICAL — must fix (architecture violations, async safety) +- HIGH — should fix (missing tests, performance issues) +- MEDIUM — fix when possible (style, documentation gaps) +- LOW — nice to have + +**Output**: Merged report: +``` +CODE REVIEW: +ARCHITECTURE: ✅/❌ per check +PERFORMANCE: ✅/⚠️ per check +DOCUMENTATION: ✅/⚠️ per check +ASYNC & SYSTEM SAFETY: ✅/❌ per check +CODE QUALITY: ✅/⚠️ per check +IMMUTABILITY: ✅/❌ per check +UI DESIGN: ✅/⚠️ per check (only if ui/ touched) +SUMMARY: N CRITICAL, N HIGH, N MEDIUM, N LOW +``` diff --git a/.claude/skills/molexp-spec/SKILL.md b/.claude/skills/molexp-spec/SKILL.md new file mode 100644 index 00000000..3037fa35 --- /dev/null +++ b/.claude/skills/molexp-spec/SKILL.md @@ -0,0 +1,51 @@ +--- +name: molexp-spec +description: Convert a natural language requirement into a structured technical spec for molexp. +disable-model-invocation: true +allowed-tools: Read, Grep, Glob, Agent +argument-hint: +--- + +Convert this requirement into a technical spec: $ARGUMENTS + +## Process + +1. **Analyze**: Read relevant source files to understand current state. Identify which layers are involved (workspace, workflow, agent, server, UI). + +2. **Generate spec** in this format: + +```markdown +# Spec: + +## Summary +One paragraph: what and why. + +## Affected Layers +- [ ] workspace — data models, persistence +- [ ] workflow — steps, runtime, caching +- [ ] agent — tools, service, policy +- [ ] server — routes, schemas +- [ ] UI — renderers, state, components + +## Data Models +New/modified Pydantic models with fields and types. + +## API Changes +| Method | Path | Request | Response | Breaking? | + +## Workflow Changes +New steps/actors, modified specs, dependency changes. + +## UI Changes +New renderers, state changes, mock handlers. + +## Test Plan +Per-module unit tests + cross-layer integration tests. + +## Open Questions +Decisions that need user input. +``` + +3. **Validate**: Check for layer violations, Module=Feature compliance, backwards compatibility. + +4. **Save**: On approval, write to `docs/developer/-spec.md`. diff --git a/.claude/skills/molexp-step/SKILL.md b/.claude/skills/molexp-step/SKILL.md new file mode 100644 index 00000000..912dee19 --- /dev/null +++ b/.claude/skills/molexp-step/SKILL.md @@ -0,0 +1,44 @@ +--- +name: molexp-step +description: Develop a new workflow Step (batch) or Actor (streaming) for molexp's pydantic-graph workflow system. +disable-model-invocation: true +allowed-tools: Read, Edit, Write, Bash, Grep, Glob +argument-hint: +--- + +Develop workflow step: $ARGUMENTS + +## Steps + +1. **Choose type**: + - `Step` (batch) — runs once, returns result: `async def execute(ctx) -> OutputT` + - `Actor` (streaming) — runs continuously, yields: `async def run(ctx) -> AsyncIterator[OutputT]` + +2. **Define types**: `OutputT` as Pydantic model (never raw dict). `InputT` from upstream. `StateT` for shared state. `DepsT` for injected deps. + +3. **Implement** using context: + - `ctx.state` — shared mutable workflow state + - `ctx.deps` — injected (workspace, run, services) + - `ctx.inputs` — typed upstream output + - Actors: `await ctx.receive()` / `await ctx.emit(channel, msg)` + +4. **Integrate** via functional or OOP DSL: + ```python + # Functional + @wf.step(depends_on=["upstream"]) + async def my_step(ctx: StepContext[S, D, InputT]) -> OutputT: ... + + # OOP + wf.add(MyStep(), depends_on=["upstream"]) + ``` + +5. **Test** in `tests/workflow/`: isolate step first, then test in workflow spec. + +6. **Verify**: `pytest tests/workflow/` + +## Rules + +- Return annotation determines execution type — no explicit flags +- Same-level steps parallelize automatically +- `parallel_map()` / `join()` for explicit fan-out/fan-in +- Never import from `_pydantic_graph/` directly diff --git a/.claude/skills/molexp-test/SKILL.md b/.claude/skills/molexp-test/SKILL.md new file mode 100644 index 00000000..01e06520 --- /dev/null +++ b/.claude/skills/molexp-test/SKILL.md @@ -0,0 +1,55 @@ +--- +name: molexp-test +description: User-invoked TDD workflow to write or improve tests for molexp modules with coverage analysis. Orchestrates the process; delegates test authoring to the molexp-tester agent when useful. +disable-model-invocation: true +allowed-tools: Read, Edit, Write, Bash, Grep, Glob +argument-hint: +--- + +Write tests for: $ARGUMENTS + +## Steps + +1. **Identify scope**: Read source code and existing tests. Check `conftest.py` for fixtures. + +2. **Organize**: Tests mirror source: + ``` + tests/agent/ → src/molexp/agent/ + tests/server/ → src/molexp/server/ + tests/workflow/ → src/molexp/workflow/ + tests/workspace/ → src/molexp/workspace/ + ``` + Shared fixtures in `tests//conftest.py`. + +3. **TDD cycle** for each function/class: + - Write test → FAIL (RED) + - Implement → PASS (GREEN) + - Refactor (IMPROVE) + +4. **Backend patterns**: + ```python + # Workspace: use tmp_path, not mocks + def test_run_persists(tmp_path): + ws = Workspace(root=tmp_path) + ws.materialize() + + # Server: use TestClient + def test_endpoint(client: TestClient): + resp = client.post("/api/projects", json={"name": "test"}) + assert resp.status_code == 201 + + # Workflow: test step in isolation, then in spec + async def test_step(): + result = await step.execute(ctx) + ``` + +5. **Frontend**: Vitest, fixtures in `ui/src/__fixtures__/`. + +6. **Coverage**: `pytest tests// --cov=src/molexp/ --cov-report=term-missing` — target 80%+. + +## Rules + +- Don't mock filesystem in integration tests — use `tmp_path` +- Don't test `_pydantic_graph/` internals — test through public API +- Don't duplicate fixtures — use `conftest.py` +- Tests must be order-independent diff --git a/.claude/skills/molexp-ui/SKILL.md b/.claude/skills/molexp-ui/SKILL.md new file mode 100644 index 00000000..f8ad266a --- /dev/null +++ b/.claude/skills/molexp-ui/SKILL.md @@ -0,0 +1,41 @@ +--- +name: molexp-ui +description: Develop a frontend component, renderer, or UI feature for molexp's React UI — focuses on mechanics (state wiring, registration, mocks). For visual polish or design audits use /molexp-design. +disable-model-invocation: true +allowed-tools: Read, Edit, Write, Bash, Grep, Glob, Agent +argument-hint: +--- + +Develop UI feature: $ARGUMENTS + +## Steps + +1. **Read context**: Check `ui/src/app/registry.ts` (renderer dispatch), existing renderers in `ui/src/app/renderers/`, state in `ui/src/app/state/`, generated types in `ui/src/api/generated/models/`. + +2. **Implement component**: + - Renderer → `ui/src/app/renderers/Viewer.tsx` + - UI primitive → `ui/src/components/ui/` (Radix-based, no business logic) + - Register renderer in `registerRenderers.ts` and `registry.ts` + +3. **Wire state**: + - Zustand stores: `useWorkspaceState`, `useUrlState` + - API calls through `ui/src/app/state/api.ts` — never call generated services directly + - Entity resolution: URL state → resolver → renderer + +4. **Mock data** (for `dev:mock` mode): + - Data in `ui/mocks/db/index.ts` + - Handler in `ui/mocks/handlers/` + - Register in `ui/mocks/handlers/index.ts` + +5. **Test**: Add `.test.ts` file, fixtures in `ui/src/__fixtures__/` + +6. **Verify**: + ```bash + cd ui && npx tsc --noEmit && npm test && npm run dev:mock + ``` + +7. **Design pass**: invoke the `molexp-designer` agent on the changed renderer/files. Apply CRITICAL and HIGH fixes it reports (token violations, missing states, density regressions). For deeper polish, use `/molexp-design`. + +## Stack + +React 19, TypeScript strict, Rsbuild, Radix UI, `@xyflow/react` (graphs), Monaco (editor), Vitest. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 00000000..226a6bee --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,62 @@ +name: CI + +on: + push: + branches: + - master + - dev + pull_request: + +jobs: + lint: + name: Lint + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install tools + run: pip install ruff ty + + - name: Ruff format + run: ruff format --check src/ tests/ + + - name: Ruff lint + run: ruff check src/ tests/ + + - name: Type check + run: ty check src/ + + test: + name: Test on Python ${{ matrix.python-version }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.12", "3.13"] + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + + - name: Install dependencies + run: pip install -e ".[dev]" + + - name: Run tests + run: pytest tests/ -q --cov=src/molexp --cov-report=xml + + - name: Upload coverage + if: matrix.python-version == '3.12' + uses: codecov/codecov-action@v4 + with: + files: coverage.xml + fail_ci_if_error: false diff --git a/.gitignore b/.gitignore index 5f12af07..dd8aef3c 100644 --- a/.gitignore +++ b/.gitignore @@ -185,4 +185,8 @@ psd thumb sketch -openapi.json \ No newline at end of file +openapi.json + +# Compiled frontend bundle (populated by `npm run build:ui`) +src/molexp/_webapp/* +!src/molexp/_webapp/.gitkeep \ No newline at end of file diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 00000000..5ac8b3ab --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,30 @@ +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v6.0.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + args: ['--unsafe'] + - id: check-added-large-files + args: ['--maxkb=1024'] + - id: check-merge-conflict + - id: mixed-line-ending + - id: check-toml + - id: check-json + + - repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.11.12 + hooks: + - id: ruff-format + - id: ruff + args: [--fix, --exit-non-zero-on-fix] + + - repo: local + hooks: + - id: ty + name: ty (type check) + entry: ty check src/ + language: system + types: [python] + pass_filenames: false diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..5b2d1cb6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,255 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Commands + +### Backend (Python) + +```bash +# Install in editable mode (no frontend build — Python-only dev loop) +pip install -e . + +# Run all tests +pytest tests/ + +# Run specific test module +pytest tests/workspace/test_workspace.py + +# Run single test +pytest tests/workspace/test_workspace.py::test_workspace_creation + +# Start server (serves bundled UI if src/molexp/_webapp/ is populated, otherwise API-only) +molexp serve --workspace /path/to/workspace --port 8000 + +# Initialize a workspace +molexp init [path] + +# Release: build frontend into src/molexp/_webapp/, then build the wheel +npm run build:ui +python -m build --wheel +``` + +### Frontend (TypeScript/React) + +```bash +cd ui + +# Dev server (localhost:5173, requires running backend) +npm run dev + +# Dev with mock API (no backend needed) +npm run dev:mock + +# Production build +npm run build + +# Run frontend tests +npm test + +# Regenerate TypeScript API client from openapi.json +npm run generate:api +``` + +### After changing the FastAPI backend + +1. Regenerate `openapi.json` (start server, fetch `/api/openapi.json`) +2. Regenerate the TypeScript client: `cd ui && npm run generate:api` +3. Update MSW mock handlers in `ui/mocks/handlers/` if new endpoints added + +## Architecture + +molexp is a workflow-and-agent platform for research experiment management, built on pydantic-graph and PydanticAI. + +``` +WorkflowSpec → pydantic-graph Compiler → Runtime → Workspace → FastAPI → React UI + ↑ + AgentService (PydanticAI) +``` + +### Five Core Layers + +#### 1. Workflow Layer (`src/molexp/workflow/`) + +Two equivalent APIs for defining task graphs: + +**Functional DSL** (decorator-based): +```python +wf = workflow(name="pipeline") + +@wf.step +async def fetch(ctx: StepContext[State, Deps, None]) -> FetchResult: ... + +@wf.step(depends_on=["fetch"]) +async def process(ctx: StepContext[State, Deps, FetchResult]) -> ProcessResult: ... + +spec = wf.build() +result = await spec.execute(run=run) +``` + +**OOP DSL** (builder-based): +```python +wf = WorkflowBuilder(name="pipeline").add(FetchStep()).add(ProcessStep(), depends_on=["fetch"]).build() +result = await wf.execute(run=run) +``` + +Key abstractions: +- `Step` — Batch execution (`async def execute(ctx) -> OutputT`) +- `Actor` — Streaming execution (`async def run(ctx) -> AsyncIterator[OutputT]`) +- `StepContext` / `ActorContext` — Typed context with state, deps, inputs +- `WorkflowSpec` — Compiled spec with deterministic `workflow_id` (topology hash) +- `WorkflowRuntime` — Abstract runtime; `GraphWorkflowRuntime` backed by pydantic-graph + +Internal `_pydantic_graph/` compiles specs into pydantic-graph IR with topological levels for automatic parallelization. Never import from `_pydantic_graph/` directly. + +Supporting modules: `cache.py` (LRU content-addressed), `persistence.py` (run store adapter), `snapshot.py` (AST-normalized code hashing). + +#### 2. Agent Layer (`src/molexp/agent/`) + +Goal-driven autonomous execution built on PydanticAI. Public API: + +```python +service = AgentService.from_workspace("./lab") +session = await service.run(Goal(description="...", constraints=[...])) +``` + +- `AgentService` — Entry point, creates sessions from workspace +- `AgentRuntime` — Abstract runtime (PydanticAI implementation in `_pydantic_ai/`) +- `Tool` / `@agent_tool` — Tool definitions with approval levels +- `ApprovalPolicy` — Glob-pattern-based tool approval control +- Session events: `PlanCreatedEvent`, `ToolCallEvent`, `ObservationEvent`, etc. + +Internal `_pydantic_ai/` handles PydanticAI integration. Never import directly. + +#### 3. Workspace Layer (`src/molexp/workspace/`) + +File-system-backed hierarchical state: `Workspace → Project → Experiment → Run` + +- Each level owns an `AssetLibrary` (content-addressed, deduplication) +- `RunContext` manages execution lifecycle (enter → execute → exit with status) +- `ParamSpace` (`GridSpace`, `UniformSpace`) for parameter sweeps +- `ResumePolicy` protocol for resumable execution +- All metadata writes are atomic (temp-file + `os.rename`) +- Constructors are side-effect-free; call `materialize()` to create dirs/files + +#### 4. Server Layer (`src/molexp/server/`) + +FastAPI app, all routes under `/api`: + +| Route module | Endpoints | +|---|---| +| `agent.py` | Session CRUD, SSE event streaming, approval | +| `project.py` | Project CRUD | +| `experiment.py` | Experiment CRUD | +| `run.py` | Run CRUD | +| `workspace.py` | Workspace open/list/folders | +| `execution.py` | Execution planning/status | +| `registry.py` | Available task types | +| `asset.py` | Asset management | + +- `dependencies.py` — FastAPI DI: `get_workspace()`, `get_settings()` +- `manager.py` — `ServerManager` for lifecycle (start/stop/status/logs) +- Production: serves SPA from `static_dir`; Dev: API-only + CORS for localhost:5173 + +#### 5. UI Layer (`ui/src/`) + +React 19 + Rsbuild, three-panel layout: + +- **Left**: Navigation tree (projects, experiments, runs) +- **Center**: Content viewer (dispatched by entity type via `registry.ts`) +- **Right**: Inspector/metadata panel + +Key patterns: +- `registry.ts` — Maps entity types to renderer components +- `state/useWorkspaceState.ts` — Zustand store for workspace state +- `state/useUrlState.ts` — URL-based routing state +- `state/api.ts` — Wraps auto-generated OpenAPI client +- `api/generated/` — **Never edit manually**; regenerate with `npm run generate:api` +- `mocks/handlers/` — MSW handlers for `dev:mock` mode; keep in sync with API changes +- `resolvers/` — Entity resolution for rendering dispatch + +Component library preference: +- Prefer **shadcn/ui** components when implementing UI features. +- When a shadcn component does not fit, **document the reason in the PR description**. + +### Packaging & Frontend Serving + +The React frontend is **compiled ahead of time by npm** and bundled inside the Python package — matching the `molvis` release workflow. `pip install` / `python -m build` **never** invokes npm. + +``` +ui/src/ → (npm run build:ui) → src/molexp/_webapp/ → (hatchling) → wheel +``` + +- **Root `package.json`** exposes `npm run build:ui`, which builds the `molexp-ui` workspace and copies `ui/dist/.` into `src/molexp/_webapp/` +- **`pyproject.toml`** uses `hatchling` and declares `[tool.hatch.build] artifacts = ["src/molexp/_webapp/**"]` so the wheel ships the bundle +- **`src/molexp/_webapp/`** is gitignored (except `.gitkeep`); it is populated on demand by `npm run build:ui` +- **Runtime**: `create_app()` uses `importlib.resources.files("molexp") / "_webapp"` to locate the bundled assets. If empty, the server runs API-only. +- **Release**: `npm run build:ui && python -m build --wheel` +- **Dev mode**: Run backend (`molexp serve --port 8000`) and frontend (`npm run dev` from repo root, or `cd ui && npm run dev`) separately +- **Production** (`molexp serve`): serves API + bundled SPA from the installed package + +### Key Patterns + +- **Topology-driven parallelism**: Steps grouped into levels by dependency graph; same-level steps run in parallel automatically +- **Content-addressed caching**: `TaskSnapshot` uses AST-normalized code hash + config hash; whitespace/comment changes don't invalidate cache +- **Atomic persistence**: All JSON writes use temp-file + `os.rename` for crash safety +- **Internal convention**: Prefixed `_pydantic_graph/` and `_pydantic_ai/` are private implementation details; public API is the parent package's `__init__.py` + +### Adding a New Workflow Step + +1. Subclass `Step` (batch) or `Actor` (streaming) in appropriate module +2. Implement `execute()` / `run()` with typed return annotation +3. Add to workflow via functional DSL (`@wf.step`) or OOP builder (`.add()`) +4. Compiler auto-detects execution type from return annotation + +### Adding a New API Route + +1. Add route handler in `src/molexp/server/routes/.py` +2. Register router in `src/molexp/server/routes/__init__.py` +3. Add request/response Pydantic schemas to `src/molexp/server/schemas/` +4. Regenerate openapi.json and TS client: `cd ui && npm run generate:api` +5. Add MSW mock handler in `ui/mocks/handlers/` for `dev:mock` + +### Adding a New UI Renderer + +1. Create component in `ui/src/app/renderers/Viewer.tsx` +2. Register in `ui/src/app/renderers/registerRenderers.ts` +3. Add entity type mapping in `ui/src/app/registry.ts` +4. Add test fixture in `ui/src/__fixtures__/` + +### Test Organization + +Tests mirror source structure: +``` +tests/ +├── agent/ → src/molexp/agent/ +├── server/ → src/molexp/server/ +├── workflow/ → src/molexp/workflow/ +└── workspace/ → src/molexp/workspace/ +``` + +Each test directory has `conftest.py` for shared fixtures. Use `conftest.py` at directory level, not standalone fixture files. + +## Skills (`.claude/skills/`) + +| Skill | Trigger | Purpose | +|---|---|---| +| `/molexp-impl` | User | Full feature implementation (plan → TDD → cross-layer wiring → verify) | +| `/molexp-spec` | User | Natural language → structured technical spec | +| `/molexp-api` | User | API endpoint (route + schema + client regen + MSW mock) | +| `/molexp-ui` | User | Frontend mechanics (renderer + state + mock + test) — invokes `molexp-designer` for post-impl polish | +| `/molexp-design` | User | Frontend visual/UX polish: info density, design-system tokens, a11y, empty/error states | +| `/molexp-step` | User | Workflow Step/Actor development | +| `/molexp-test` | User | TDD testing with coverage analysis | +| `/molexp-review` | Auto/User | Architecture + performance + UI design review (layer compliance, async safety, caching, I/O, concurrency, design system) | +| `/molexp-agent-tool` | User | PydanticAI agent tool development | + +### Agents (delegated, not user-invoked) + +| Agent | Axis | +|---|---| +| `molexp-architect` | 5-layer compliance, module boundaries | +| `molexp-optimizer` | Async, I/O, serialization, React perf | +| `molexp-tester` | Test authoring (RED/GREEN/REFACTOR) | +| `molexp-documenter` | Google-style docstrings, JSDoc, OpenAPI descriptions | +| `molexp-designer` | UI visual quality, info density, tokens, a11y | diff --git a/LICENSE b/LICENSE new file mode 100644 index 00000000..3e83c7d4 --- /dev/null +++ b/LICENSE @@ -0,0 +1,28 @@ +BSD 3-Clause License + +Copyright (c) 2024, molcrafts + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/README.md b/README.md index 85e5b9a0..951659e4 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,64 @@ # molexp -molexp is a tiny yet fully-typed task-graph framework built on top of Pydantic. It contains a -pure functional task abstraction, a static compiler that produces deterministic graph orders, a -runtime engine, and a tiny DSL for common data-flow patterns. This repository is intentionally -minimal to highlight how each layer works without hidden magic. +molexp is a workflow-and-agent platform for research experiment management. It provides a typed task-graph framework, a Project-Experiment-Run workspace hierarchy, content-addressed asset storage, and a FastAPI server with React UI. ``` -+-----------+ +-----------+ +---------+ -| Task | --> | Compiler | --> | Engine | -+-----------+ +-----------+ +---------+ - ^ | | - | v v - +----------- DSL abstractions --------+ +WorkflowSpec → Runtime → Workspace → FastAPI → React UI + ↑ + AgentService (PydanticAI) ``` ## Features -- **Task Graph Framework**: Pure functional task abstraction with deterministic compilation +- **Workflow Layer**: DAG-based task graphs with automatic parallelization - **Project-Experiment-Run Architecture**: Scientific workflow organization with full reproducibility - **Asset Management**: Content-addressable storage with automatic deduplication -- **CLI Tools**: Command-line interface for workspace and workflow management -- **Type Safety**: Full Pydantic v2 integration for data validation +- **Agent Layer**: Goal-driven autonomous execution built on PydanticAI +- **CLI + Web UI**: Command-line interface and React-based experiment browser -## Quick Example: Task Graph +## Quick Example: Workflow + +**Functional DSL (decorator-based):** + +```python +from molexp.workflow import workflow, TaskContext + +wf = workflow(name="data-pipeline") + +@wf.task +async def fetch(ctx: TaskContext) -> list[float]: + return [1.0, 2.0, 3.0] + +@wf.task(depends_on=["fetch"]) +async def process(ctx: TaskContext) -> float: + if ctx.dry_run: + return 0.0 + return sum(ctx.inputs) + +spec = wf.build() +result = await spec.execute() +``` + +**OOP builder (subclass-based):** ```python -from molexp.task_base import Task, EmptyConfig -from molexp.engine import TaskEngine +from molexp.workflow import Task, WorkflowBuilder, TaskContext -class MultiplyTask(Task[EmptyConfig, int]): - cfg_model = EmptyConfig - out_model = None +class FetchTask(Task): + async def execute(self, ctx: TaskContext) -> list[float]: + return [1.0, 2.0, 3.0] - def forward(self, value: int, cfg: EmptyConfig) -> int: - return value * 2 +class ProcessTask(Task): + async def execute(self, ctx: TaskContext) -> float: + return sum(ctx.inputs) -mult = MultiplyTask(name="multiply") -engine = TaskEngine() -result = engine.run(mult) -print(result) +spec = ( + WorkflowBuilder(name="data-pipeline") + .add(FetchTask()) + .add(ProcessTask(), depends_on=["fetch"]) + .build() +) +result = await spec.execute() ``` ## Project-Experiment-Run Architecture @@ -50,76 +70,57 @@ molexp provides a complete organization system for scientific workflows: - **Run**: Single execution instance with full reproducibility - **Asset**: Reusable data artifacts with content-based deduplication -### CLI Usage - -```bash -# Initialize workspace -molexp init - -# Create project -molexp project create my-project --name "My Research Project" - -# Create experiment -molexp experiment create my-project exp-1 \ - --name "Parameter Sweep" \ - --workflow workflow.py - -# List runs -molexp run list my-project exp-1 - -# View assets -molexp asset list -``` - ### Python API ```python from molexp.workspace import Workspace -# Create workspace -workspace = Workspace.from_env() +workspace = Workspace.from_path("./lab") -# Create project (ID auto-generated from name) -project = workspace.create_project( - name="My Project", - description="Research project description" -) +# Hierarchical API: workspace → project → experiment → run +project = workspace.create_project(name="My Project") +experiment = project.create_experiment(name="Param Sweep") +run = experiment.create_run(parameters={"lr": 0.01}) -# Create experiment through project (hierarchical API) -experiment = project.create_experiment( - name="Experiment 1", - workflow_source="workflow.py" -) - -# Create run through experiment (hierarchical API) -run = experiment.create_run( - parameters={"param": 1.0}, - workflow_file="workflow.py" -) - -# Use hierarchical asset libraries -# Workspace-level assets (global) +# Scoped asset libraries at every level workspace.assets.create_asset("bert_model", "/models/bert.pt") - -# Project-level assets (shared within project) project.assets.create_asset("dataset", "/data/qm9.tar.bz2") - -# Experiment-level assets (shared within experiment) experiment.assets.create_asset("features", "/data/features.h5") - -# Run-level assets (specific to this run) run.assets.create_asset("output", "/outputs/results.txt") ``` +### CLI Usage + +```bash +# Start server (hot-reload dev mode) +molexp serve --dev + +# Execute a workflow in dry-run mode +molexp run train.py --dry-run + +# Initialize a workspace +molexp init [path] +``` + ## Installation ```bash pip install -e . ``` +Optional extras: + +```bash +pip install -e ".[workflow]" # pydantic-graph execution backend +pip install -e ".[agent]" # PydanticAI agent layer +pip install -e ".[remote]" # Remote/HPC execution +pip install -e ".[dev]" # Development tools +``` + ## Documentation -See the [docs](./docs/README.md) for an in-depth tour of the architecture, compiler, engine, and DSL -usage. +See the [docs](./docs/index.md) for an in-depth tour of the architecture, workflow layer, workspace, and agent integration. + +## License -See [examples/project_experiment_run_example.py](./examples/project_experiment_run_example.py) for a complete example of the Project-Experiment-Run workflow. +BSD 3-Clause License — see [LICENSE](./LICENSE) for details. diff --git a/docs/developer/architecture-convergence-spec-zh.md b/docs/developer/architecture-convergence-spec-zh.md new file mode 100644 index 00000000..53985f5c --- /dev/null +++ b/docs/developer/architecture-convergence-spec-zh.md @@ -0,0 +1,391 @@ +# Molexp 架构收敛与易用性修复 Spec + +> 状态:Draft v1 +> 日期:2026-04-10 +> 范围:`workflow` / `workspace` / `server` / `agent` 公共模型、API 契约、文档与可用性 +> 目标:把当前“多套架构并存”的状态收敛为一套可解释、可运行、可测试、可文档化的产品架构 + +实施拆分见 [architecture-convergence-work-breakdown-zh.md](./architecture-convergence-work-breakdown-zh.md)。 + +--- + +## 1. 背景 + +当前仓库已经完成了部分新架构落地: + +- `workflow` 层以 `WorkflowSpec` / `Step` / `WorkflowRuntime` 为中心 +- `agent` 层以 `AgentService` / `AgentRuntime` / `PydanticAI` 封装为方向 +- `workspace` 层仍承担 `Workspace / Project / Experiment / Run / Asset` 产品语义 + +但仓库的公共表面并未完成收敛,导致以下现象同时存在: + +- README、Quick Start、Core 文档仍在描述旧 `Task / IR / Engine` 架构 +- 新旧 import path 在文档中混用,且大量路径已不存在 +- server 请求模型、响应模型、核心领域模型互相不一致 +- 部分路由与核心对象的方法签名不匹配 +- 顶层 import 会隐式拉起远程执行基础设施 +- 多个“还未完成”的能力以 stub 或伪成功形式暴露给用户 + +这不是单点 bug,而是产品架构边界没有收口。 + +--- + +## 2. 问题定义 + +### 2.1 架构问题 + +1. 没有唯一公共模型 +2. 领域模型与接口契约断裂 +3. workspace 持久化存在双重真相源 +4. 可选基础设施依赖污染默认导入路径 + +### 2.2 易用性问题 + +1. 首屏文档不可运行 +2. API 文档不能代表真实行为 +3. 不完整能力以“看起来可用”的形式暴露 +4. 用户无法判断什么是正式支持,什么是迁移残留 + +--- + +## 3. 设计目标 + +### 3.1 产品目标 + +- 用户只面对一套 Molexp 公共模型 +- README、Quick Start、Python API、HTTP API、测试样例描述同一套行为 +- 本地 workspace/server 使用场景不依赖远程执行或外部数据库初始化 +- 未完成能力要显式失败,不允许伪成功 + +### 3.2 工程目标 + +- `workflow`、`workspace`、`server`、`agent` 之间边界稳定 +- 请求模型、响应模型、领域模型一一对应 +- 文档、代码、测试由同一 contract 驱动 +- 清理旧架构残留,降低未来维护成本 + +### 3.3 成功标准 + +- 新用户按 README/Quick Start 可在干净环境中完成最小闭环 +- `import molexp.server.app` 不触发远程基础设施副作用 +- 所有公开路由至少有 contract test 覆盖 +- 不再出现文档存在、实现不存在的公共概念 + +--- + +## 4. 非目标 + +- 不在本阶段重做 UI 视觉或前端交互 +- 不在本阶段引入复杂迁移工具链 +- 不保证对旧文档和旧 import path 的长期兼容 +- 不在本阶段扩展新的 workflow 能力面 + +说明: +若为降低重构风险需要短期 shim,可以存在于内部实现,但不能继续作为公共文档承诺。 + +--- + +## 5. 核心设计决策 + +## 5.1 选择唯一公共模型 + +公共架构统一为三层: + +1. Product Layer:`workspace / assets / project / experiment / run / server` +2. Workflow Layer:`WorkflowSpec / WorkflowBuilder / workflow() / Step / Actor / WorkflowRuntime` +3. Agent Layer:`Goal / AgentSession / AgentService / Tool / ApprovalPolicy` + +明确决策: + +- 旧 `Task / IR / WorkflowEngine / WorkflowCompiler` 不再作为公共文档入口 +- 若仓库内仍保留过渡性实现,必须降级为内部历史残留,不得出现在 README、Quick Start、OpenAPI 示例中 +- `molexp.__init__` 不再通过 eager import 扩散整个子系统依赖图 + +## 5.2 Experiment 和 Run 恢复产品语义 + +当前产品语义要求: + +- `Experiment` 是“可重复执行的 workflow 定义” +- `Run` 是“某次具体执行及其快照” + +因此领域模型必须显式承载这些信息: + +- `ExperimentMetadata` + - `workflow_source` + - `workflow_type` + - `git_commit` + - `parameter_space` + - `description` + - `tags` + - `config` +- `RunMetadata` + - `parameters` + - `status` + - `finished_at` + - `error` + - `workflow_snapshot` + - 必要的执行上下文引用 + +若不准备支持这些概念,则必须同步从请求模型、响应模型、文档中删除,不能继续悬空。 + +## 5.3 以“少参数默认可用”为 API 原则 + +面向用户的创建接口统一采用最小必要输入: + +- `create_project(name, ...)` +- `project.create_experiment(name, workflow_source, ...)` +- `experiment.create_run(parameters, ...)` + +用户可选提供自定义 `id`,但不是必填。自动生成策略必须一致,且文档、CLI、HTTP API 完全一致。 + +## 5.4 父子关系以目录结构为真相源 + +对于 `Workspace -> Project -> Experiment -> Run`: + +- 子对象存在性和成员关系以目录结构加子对象 metadata 文件为真相源 +- 父 metadata 中不再维护 `projects` / `experiments` / `assets` 这类重复列表 +- 如保留索引文件,只能作为缓存,必须允许重建 + +这样可以消除删除、恢复、手工修复时的双重真相源问题。 + +## 5.5 可选能力必须惰性加载 + +远程执行、HPC、外部 transfer、agent runtime 等都属于可选能力。 + +约束如下: + +- 默认 import 路径不能初始化它们 +- 只有在显式调用相关能力时才加载对应依赖 +- 如果环境不满足,返回明确错误,而不是污染基础使用路径 + +## 5.6 未完成能力必须显式 gated + +以下能力在未完成前必须统一策略: + +- agent session 执行 +- workflow plan/execution API +- registry 查询 +- 远程执行 + +统一行为: + +- 返回明确的 `501 Not Implemented` 或领域错误 +- 不返回“pending session”一类伪成功对象 +- 不在 README/Quick Start 中展示未完成能力 + +## 5.7 文档与测试是产品表面的一部分 + +文档和测试不是附属物,而是 contract 的一部分。 + +必须建立三类 smoke/contract 校验: + +1. import smoke +2. quick start smoke +3. API contract smoke + +任一失败都视为产品表面损坏。 + +--- + +## 6. 目标架构 + +## 6.1 包边界 + +### `molexp.workspace` + +- 负责工作区目录模型、持久化、资产作用域、Project/Experiment/Run 生命周期 +- 不直接依赖 server、agent、远程执行基础设施 + +### `molexp.workflow` + +- 负责 workflow 定义、编译适配、执行 runtime +- 不污染顶层 import +- 远程执行通过惰性 adapter 接入 + +### `molexp.agent` + +- 负责 goal/session/tooling/approval +- 若 runtime 不可用,服务端显式返回能力未启用 + +### `molexp.server` + +- 只暴露已实现并受测试保护的 contract +- 路由不得直接依赖“猜测式字段”或历史属性名 + +## 6.2 顶层导入策略 + +`molexp.__init__` 只暴露稳定符号,不执行重型导入。 + +建议策略: + +- 保留 `__version__` +- 使用 lazy import 或显式子模块导入 +- 禁止在包初始化时 import `remote`、`molq`、数据库或网络资源 + +--- + +## 7. Roadmap + +## Phase 0:冻结公共表面并建立清单 + +目标: +停止继续扩散新旧架构混用,先把“当前支持什么”说清楚。 + +交付物: + +- 公共 API 清单 +- 历史残留 import path 清单 +- server 路由到领域模型映射清单 +- README / docs / tests 的冲突矩阵 + +退出条件: + +- 每个公开入口都标记为 `supported`、`internal`、`remove` +- 未完成能力全部标注 gating 策略 + +## Phase 1:公共 API 收敛 + +目标: +只保留一套对外叙事,移除旧架构文档入口。 + +交付物: + +- README 改写到新公共模型 +- Quick Start 改写为可运行的新示例 +- 删除或降级旧 `Task / IR / Engine` 文档入口 +- `molexp.__init__` 去除 eager import + +退出条件: + +- README 中所有 import path 均真实存在 +- Quick Start 能在测试中跑通最小样例 +- `python -c "from molexp.server.app import create_app"` 在本地环境可成功导入 + +## Phase 2:领域模型与持久化修复 + +目标: +让 Experiment/Run 的产品语义重新落到真实模型上,同时消除双重真相源。 + +交付物: + +- 重构 `ExperimentMetadata` 与 `RunMetadata` +- 移除父 metadata 中重复 child 列表 +- 统一创建 API 的参数模型 +- 统一 asset library 的查找和索引语义 + +退出条件: + +- HTTP 请求模型与 Python API 使用同一组核心字段 +- 删除/列举/读取行为不再依赖重复状态 +- run/experiment 的响应字段均来自真实模型,而不是 `getattr(..., None)` 猜测 + +## Phase 3:Server Contract 修复 + +目标: +把 server 从“原型路由集合”修到“可依赖 contract”。 + +交付物: + +- 修复所有方法签名不匹配的路由 +- 移除 dead code、注释式补丁和不可达重试逻辑 +- 未完成端点统一返回 `501` +- 生成稳定的 OpenAPI 文档 + +退出条件: + +- 所有已暴露路由至少有一个成功路径测试 +- 所有未完成路由有明确失败测试 +- 资产上传、下载、run start、project/experiment CRUD 可端到端运行 + +## Phase 4:Agent 与 Execution 能力显式化 + +目标: +把 agent/execution 从“部分可见、部分 stub”改为“已实现则可用,未实现则显式关闭”。 + +交付物: + +- agent runtime availability 检测 +- session 持久化与事件流 contract 固化 +- execution/plan 能力按实现状态分层暴露 +- server 文档中标注 capability matrix + +退出条件: + +- 不再返回 stub session +- session/events 行为在单进程与恢复场景下有明确测试 +- execution 入口不再误导用户以为已完整可用 + +## Phase 5:回归清理与发布门禁 + +目标: +防止问题再次回流。 + +交付物: + +- import smoke tests +- doc smoke tests +- API contract tests +- “公共表面回归检查” CI 门禁 + +退出条件: + +- 新 PR 无法再引入不存在的 import path 到 README/Quick Start +- 新增路由必须附带 contract test +- 顶层导入副作用被测试锁住 + +--- + +## 8. 验收标准 + +达到以下标准,视为本 spec 完成: + +1. README、Quick Start、HTTP API、Python API 共享同一套术语和对象模型 +2. `Project / Experiment / Run` 的创建、读取、执行、状态更新路径不再依赖历史字段猜测 +3. server 不再存在签名错配、不可达修复逻辑和伪成功返回 +4. 顶层导入不要求远程执行环境、数据库路径或外部服务 +5. 未实现能力全部显式 gated +6. 文档示例具备 smoke 测试 + +--- + +## 9. 风险与缓解 + +### 风险 1:重构期间新旧代码并存导致继续扩散 + +缓解: + +- 先执行 Phase 0 清单化 +- 所有公共入口先分类,再决定保留/删除 + +### 风险 2:为快速修复 server 而继续堆积 `getattr` 和特判 + +缓解: + +- 规定 response model 只能映射真实领域字段 +- 不允许通过“猜字段”维持表面兼容 + +### 风险 3:删除旧文档后短期信息缺口 + +缓解: + +- 在新 README 和 Quick Start 中补最小闭环 +- 历史设计提案保留,但明确标为历史/迁移文档 + +### 风险 4:可选依赖拆分影响已有测试 + +缓解: + +- 先建立 import smoke +- 再做惰性加载改造 + +--- + +## 10. 决策摘要 + +本 spec 的核心结论只有三条: + +1. Molexp 必须只有一套公共架构叙事 +2. 领域模型、接口契约、文档示例必须从同一个真相源生成 +3. 未完成能力和可选能力必须显式化,不能再通过 stub、历史残留和导入副作用泄漏到产品表面 + +如果这三条不先完成,继续叠加 workflow、agent、server 功能只会放大维护成本和首日使用摩擦。 diff --git a/docs/developer/architecture-convergence-work-breakdown-zh.md b/docs/developer/architecture-convergence-work-breakdown-zh.md new file mode 100644 index 00000000..3f1d7852 --- /dev/null +++ b/docs/developer/architecture-convergence-work-breakdown-zh.md @@ -0,0 +1,620 @@ +# Molexp 架构收敛实施拆分 + +> 关联文档:[architecture-convergence-spec-zh.md](./architecture-convergence-spec-zh.md) +> 用途:把收敛 spec 拆成可以直接排期和实施的工作包 +> 状态:Draft v1 + +--- + +## 1. 使用方式 + +本文件不重复解释为什么要做,而是回答三个问题: + +1. 先做什么 +2. 每一步改哪些东西 +3. 做到什么程度算完成 + +建议按 workstream + issue 的方式执行。 + +- workstream 用来分责任边界 +- issue 用来落具体变更 +- milestone 用来控制顺序和依赖 + +--- + +## 2. 实施顺序 + +建议按以下顺序推进: + +1. 冻结公共表面和导入边界 +2. 修复核心领域模型和持久化真相源 +3. 修复 server contract 与路由实现 +4. 清理未完成能力的暴露方式 +5. 重写文档与示例 +6. 加上 smoke/contract 门禁 + +原因: + +- 不先冻结公共表面,后续每一步都会返工 +- 不先修领域模型,server 修复会继续堆 `getattr` 和特判 +- 不先修导入边界,测试和本地使用都会继续被可选依赖污染 + +--- + +## 3. Workstreams + +## WS1:公共 API 与包边界收敛 + +目标: + +- 明确 Molexp 只支持一套公共模型 +- 清理顶层 import 副作用 +- 把历史残留 API 从文档和入口层拿掉 + +涉及模块: + +- `src/molexp/__init__.py` +- `src/molexp/workflow/__init__.py` +- `src/molexp/workflow/remote.py` +- `README.md` +- `docs/get-started/*` +- `docs/core/*` + +完成标准: + +- 顶层 import 不初始化远程执行基础设施 +- README 中所有 import path 真实存在 +- Quick Start 描述的新用户路径能运行 + +## WS2:领域模型与持久化修复 + +目标: + +- 让 `Experiment` 和 `Run` 承载真实产品语义 +- 消除 parent metadata 与目录结构的双重真相源 +- 统一 Python API 和 HTTP API 的字段模型 + +涉及模块: + +- `src/molexp/workspace/metadata.py` +- `src/molexp/workspace/workspace.py` +- `src/molexp/workspace/project.py` +- `src/molexp/workspace/experiment.py` +- `src/molexp/workspace/run.py` +- `src/molexp/workspace/asset.py` + +完成标准: + +- `ExperimentMetadata` 与 `RunMetadata` 字段能支撑当前对外文案 +- 删除/列举逻辑不再依赖重复索引 +- asset 语义统一,不再混用按 name 查找和按 asset_id 查找 + +## WS3:Server Contract 修复 + +目标: + +- 请求模型、响应模型、路由实现、领域模型完全对齐 +- 清除不可达代码、签名错配和伪修复逻辑 + +涉及模块: + +- `src/molexp/server/schemas/requests.py` +- `src/molexp/server/schemas/responses.py` +- `src/molexp/server/routes/*.py` +- `src/molexp/server/app.py` +- `src/molexp/server/dependencies.py` + +完成标准: + +- 所有公开路由都能解释成真实模型操作 +- 所有 response 字段都来自真实字段,不靠 `getattr` 猜测 +- OpenAPI 能代表真实可用行为 + +## WS4:未完成能力 gating + +目标: + +- agent、execution、registry、remote 这类未完成能力显式化 +- 未启用和未实现是明确错误,不是伪成功 + +涉及模块: + +- `src/molexp/server/routes/agent.py` +- `src/molexp/server/routes/execution.py` +- `src/molexp/server/routes/registry.py` +- `src/molexp/agent/*` +- `src/molexp/workflow/runtime.py` + +完成标准: + +- 不再返回 stub session +- 未实现端点统一返回明确错误 +- 文档不再宣称这些能力已可用 + +## WS5:文档和示例重写 + +目标: + +- README、Quick Start、Core 文档全部切到唯一公共模型 +- 删除失效 import path 和旧架构叙事 + +涉及模块: + +- `README.md` +- `docs/index.md` +- `docs/get-started/*` +- `docs/core/*` +- `docs/workspace/*` + +完成标准: + +- 文档不再引用不存在模块 +- 文档中的创建/执行路径与当前代码一致 +- 旧 `Task / IR / Engine` 文档要么删除,要么明确标历史 + +## WS6:测试与门禁 + +目标: + +- 把公共表面收敛后的行为锁住 +- 防止 README、导入边界、server contract 再次漂移 + +涉及模块: + +- `tests/server/*` +- `tests/workspace/*` +- `tests/workflow/*` +- 新增 smoke/contract tests + +完成标准: + +- 顶层 import smoke 测试通过 +- Quick Start smoke 测试通过 +- server contract tests 覆盖所有公开路由 + +--- + +## 4. Issue 拆分 + +下面的 issue 顺序已经按依赖排好,适合直接开工。 + +## I0:公共表面清单化 + +目标: + +- 产出一份当前公开 API、历史残留 API、待删除 API 的清单 + +主要动作: + +- 梳理 README、docs、server routes、`__all__` +- 标注每个入口是 `supported`、`internal`、`remove` +- 标注每个未完成能力的现状和 gating 策略 + +依赖: + +- 无 + +产物: + +- 一份清单文档或 issue comment + +Definition of Done: + +- 后续所有实现都以这份清单为准 + +## I1:顶层 import 去副作用 + +目标: + +- `import molexp` 与 `from molexp.server.app import create_app` 不触发 `molq` 初始化 + +主要动作: + +- 改造 `src/molexp/__init__.py`,避免 eager import 子模块 +- 审查 `src/molexp/workflow/__init__.py` 的 import 树 +- 将 `remote` 相关能力改为按需加载 + +涉及文件: + +- `src/molexp/__init__.py` +- `src/molexp/workflow/__init__.py` +- `src/molexp/workflow/remote.py` + +依赖: + +- I0 + +Definition of Done: + +- server tests 至少可以完成 import 和 app 初始化 +- 不再出现因远程执行依赖导致的导入失败 + +## I2:README 与 Quick Start 收敛到新模型 + +目标: + +- 用户第一眼看到的内容就是唯一支持模型 + +主要动作: + +- README 全部切换到 `molexp.workflow` / `molexp.workspace` 当前实际 API +- 删除或替换旧 `TaskEngine`、`workflow.node`、`workspace.core` 等路径 +- Quick Start 改写为可运行的最小例子 + +涉及文件: + +- `README.md` +- `docs/index.md` +- `docs/get-started/overview.md` +- `docs/get-started/quick-start.md` + +依赖: + +- I1 + +Definition of Done: + +- 文档中的 import path 全部真实存在 +- 样例代码可以被 smoke 测试执行 + +## I3:Experiment 模型恢复产品语义 + +目标: + +- `Experiment` 真正表示“可重复执行的 workflow 定义” + +主要动作: + +- 为 `ExperimentMetadata` 增加 workflow 相关字段 +- 调整 `Project.create_experiment()` 参数签名 +- 统一 CLI、server request、response 的 experiment 字段 + +涉及文件: + +- `src/molexp/workspace/metadata.py` +- `src/molexp/workspace/project.py` +- `src/molexp/workspace/experiment.py` +- `src/molexp/server/schemas/requests.py` +- `src/molexp/server/schemas/responses.py` +- `src/molexp/cli/__init__.py` + +依赖: + +- I0 + +Definition of Done: + +- experiment create/read 的 workflow 字段来自真实 metadata +- 不再通过 `workflow_template` 一类悬空属性拼响应 + +## I4:Run 模型恢复执行快照语义 + +目标: + +- `Run` 真正表示一次具体执行及其快照 + +主要动作: + +- 明确 `RunMetadata` 中的 snapshot 和 execution 相关字段 +- 统一 `create_run()`、状态更新、执行上下文落盘逻辑 +- 修复 `RunResponse` 对 `workflow_snapshot`、`finished_at` 的映射 + +涉及文件: + +- `src/molexp/workspace/metadata.py` +- `src/molexp/workspace/run.py` +- `src/molexp/server/schemas/responses.py` +- `src/molexp/server/routes/run.py` + +依赖: + +- I3 + +Definition of Done: + +- run read/status/update 使用一致字段 +- run 响应不再依赖不存在的属性 + +## I5:移除双重真相源 + +目标: + +- parent metadata 不再维护重复 child 列表 + +主要动作: + +- 去掉 `WorkspaceMetadata.projects` +- 去掉 `ProjectMetadata.experiments` +- 评估 `ProjectMetadata.assets` 是否保留为可重建缓存,否则删除 +- 调整 create/delete/list 流程 + +涉及文件: + +- `src/molexp/workspace/metadata.py` +- `src/molexp/workspace/workspace.py` +- `src/molexp/workspace/project.py` +- `src/molexp/workspace/base.py` + +依赖: + +- I3 + +Definition of Done: + +- create/list/delete 只依赖目录结构和子 metadata +- 删除后不会留下静默脏索引 + +## I6:统一 AssetLibrary contract + +目标: + +- 资产操作语义统一,不再混用 name 和 asset_id + +主要动作: + +- 确定 `get_asset(name)` 与 `get_by_id(asset_id)` 的正式接口 +- 若 server 需要按 ID 读取,就显式补 API,而不是假定存在 `get()` +- 明确下载 payload 的正式接口 + +涉及文件: + +- `src/molexp/workspace/asset.py` +- `src/molexp/server/routes/asset.py` +- `src/molexp/server/routes/project.py` + +依赖: + +- I5 + +Definition of Done: + +- asset 上传、查询、下载的路由都调用真实存在的方法 +- 不再有按 name/index 与按 id/path 混乱切换 + +## I7:修复 server 路由签名错配 + +目标: + +- 去掉当前路由层里的假设式调用和不可达修复代码 + +主要动作: + +- 修 `project` asset upload 路由 +- 修 `run.start` 路由对 `Run` 上下文 API 的调用 +- 对所有 CRUD 路由做一次签名一致性检查 + +涉及文件: + +- `src/molexp/server/routes/project.py` +- `src/molexp/server/routes/run.py` +- `src/molexp/server/routes/experiment.py` +- `src/molexp/server/routes/asset.py` + +依赖: + +- I4 +- I6 + +Definition of Done: + +- 路由代码中不再出现不可达补丁逻辑 +- 所有调用都能直接映射到真实对象方法 + +## I8:请求/响应模型重对齐 + +目标: + +- 请求模型不要求实现不存在的数据 +- 响应模型不暴露实现不存在的数据 + +主要动作: + +- 重写 `requests.py` +- 重写 `responses.py` +- 为每个 endpoint 明确字段来源 + +涉及文件: + +- `src/molexp/server/schemas/requests.py` +- `src/molexp/server/schemas/responses.py` +- 对应 routes + +依赖: + +- I3 +- I4 +- I6 + +Definition of Done: + +- API schema 与真实代码一致 +- OpenAPI 不再包含误导性必填字段 + +## I9:未完成能力统一 gating + +目标: + +- agent、execution、registry 的未完成状态可解释、可预测 + +主要动作: + +- agent route 不再返回 stub session +- execution/plan/registry 明确返回 `501` 或能力未启用错误 +- 文档同步降级这些能力表述 + +涉及文件: + +- `src/molexp/server/routes/agent.py` +- `src/molexp/server/routes/execution.py` +- `src/molexp/server/routes/registry.py` +- `src/molexp/agent/*` + +依赖: + +- I0 + +Definition of Done: + +- 用户不会收到“看起来成功但实际没做事”的响应 + +## I10:文档全量重写与历史文档降级 + +目标: + +- 文档完全以当前唯一公共模型为准 + +主要动作: + +- 重写 `docs/core/*` 和 `docs/workspace/*` +- 对旧架构文档加“历史/迁移文档”标记,或直接移除导航入口 +- 清理 README 中失效链接 + +涉及文件: + +- `README.md` +- `docs/index.md` +- `docs/core/*` +- `docs/workspace/*` + +依赖: + +- I2 +- I8 +- I9 + +Definition of Done: + +- 搜索文档中不再出现已删除的公共 import path + +## I11:smoke 与 contract tests + +目标: + +- 把本次收敛后的产品表面锁住 + +主要动作: + +- 增加 import smoke tests +- 增加 Quick Start smoke tests +- server route contract tests 覆盖 CRUD、asset、status、gating 行为 + +建议测试集: + +- `tests/smoke/test_imports.py` +- `tests/smoke/test_quickstart_examples.py` +- `tests/server/test_contract_projects.py` +- `tests/server/test_contract_experiments.py` +- `tests/server/test_contract_runs.py` +- `tests/server/test_contract_assets.py` +- `tests/server/test_contract_capability_gating.py` + +依赖: + +- I1 +- I7 +- I8 +- I9 + +Definition of Done: + +- 文档、导入、路由 contract 漂移会被 CI 阻止 + +--- + +## 5. 建议里程碑 + +## M1:公共表面冻结 + +包含: + +- I0 +- I1 +- I2 + +可见成果: + +- 新用户至少能 import 和看懂正确入口 + +## M2:领域模型收口 + +包含: + +- I3 +- I4 +- I5 +- I6 + +可见成果: + +- `Project / Experiment / Run / Asset` 语义与真实数据结构一致 + +## M3:Server 可依赖 + +包含: + +- I7 +- I8 +- I9 + +可见成果: + +- API 可用性不再靠运气 + +## M4:文档与门禁闭环 + +包含: + +- I10 +- I11 + +可见成果: + +- 文档、测试、实现同构 + +--- + +## 6. 推荐开工方式 + +如果要立即实施,我建议按下面节奏: + +第 1 轮: + +- I1 顶层 import 去副作用 +- I2 README / Quick Start 收敛 + +第 2 轮: + +- I3 Experiment 模型 +- I4 Run 模型 +- I5 双重真相源移除 + +第 3 轮: + +- I6 AssetLibrary contract +- I7 路由签名修复 +- I8 请求/响应模型重对齐 + +第 4 轮: + +- I9 能力 gating +- I10 文档全量重写 +- I11 smoke/contract tests + +这样做的好处是: + +- 每轮都能形成稳定增量 +- 不会先修文档再推翻 +- 能较早恢复最关键的开发和测试路径 + +--- + +## 7. 实施准备清单 + +在真正开始改代码前,建议先确认: + +- 是否接受“旧 API 不再对外承诺” +- 是否接受 `Experiment` / `Run` 恢复 workflow snapshot 语义 +- 是否接受未完成能力统一返回错误而不是 stub +- 是否接受本轮优先修 contract,不扩新功能 + +如果以上四点都接受,就可以按本拆分直接开工。 diff --git a/docs/developer/pydanticai-graph-integration-proposal-zh.md b/docs/developer/pydanticai-graph-integration-proposal-zh.md new file mode 100644 index 00000000..5ea91159 --- /dev/null +++ b/docs/developer/pydanticai-graph-integration-proposal-zh.md @@ -0,0 +1,1016 @@ +# Molexp 接入 `PydanticAI` 并以 `pydantic-graph` 重构 Workflow 的设计提案 + +> 状态:设计提案 v3 +> 目标:将 `molexp` 演进为"完整工作流与资产管理平台 + 自主驱动系统"。 +> 关键决策: +> - `workspace` 必须保留,并继续作为系统真相源 +> - 全量引入 `PydanticAI` 承载 `molexp.agent` +> - 以封装方式引入 `pydantic-graph` 替换 workflow kernel +> - **彻底重构,不向后兼容**,项目仍处于高速开发期,不保留任何兼容 shim +> - 用户不直接操作第三方库,所有能力通过 `molexp` 自身 facade 暴露 + +--- + +## 1. 提案摘要 + +本提案建议把 `molexp` 重构为三层架构: + +``` +Molexp Product Layer + = Workspace + Assets + Project/Experiment/Run + UI/API + +Molexp Agent Layer + = Goal / Session / Planner / Tooling / Approval / Replan + = built on top of PydanticAI + +Molexp Workflow Layer + = WorkflowSpec / WorkflowRuntime / GraphExecution + = built on top of pydantic-graph +``` + +核心思想: + +- `PydanticAI` 负责智能控制层(agent、tool、session、replanning) +- `pydantic-graph` 负责 workflow runtime(DAG 编排、步骤执行、持久化) +- `molexp` 继续负责产品语义、workspace、asset、experiment/run、用户 API、可观测性和持久化 + +最终用户使用体验: + +- **显式 workflow 模式**:用户声明步骤和依赖,系统编译执行(支持函数式和 OOP 两种写法) +- **agent 模式**:用户给一个目标,系统自动规划、调用工具、产生观测、重规划 +- 两种模式共享 workspace / run / assets + +所有底层实现细节(`pydantic_ai.Agent`、`pydantic_graph.Graph`、`BaseNode`)完全封装在 `molexp` 内部,不暴露给用户。 + +--- + +## 2. pydantic-graph 技术调研结论 + +### 2.1 Stable API(原始 API) + +**节点定义**:节点是实现 `BaseNode[StateT, DepsT, RunEndT]` 的 dataclass,`run()` 方法始终是 async,返回类型注解决定出边: + +```python +@dataclass +class MyNode(BaseNode[StateT, DepsT, RunEndT]): + input_value: int + + async def run(self, ctx: GraphRunContext[StateT]) -> NextNode | End[int]: + ctx.state.counter += 1 + if done: + return End(self.input_value) + return NextNode(self.input_value + 1) +``` + +**共享状态**:`GraphRunContext.state` 是一个 dataclass,所有节点共享并可就地修改。数据流通过节点 dataclass 字段(构造时传入)或共享 state 传递。 + +**循环(Actor 模式基础)**:节点可以在返回类型中包含自身,形成循环: + +```python +@dataclass +class ActorNode(BaseNode[StateT]): + async def run(self, ctx: GraphRunContext[StateT]) -> ActorNode | End[None]: + msg = ctx.state.inbox.get_nowait() + result = process(msg) + ctx.state.outbox.put_nowait(result) + return ActorNode() # 回到自身,持续运行 +``` + +**持久化(内建)**:Stable API 有完整的 `BaseStatePersistence` 接口: + +| 内建实现 | 存储 | 说明 | +|----------|------|------| +| `SimpleStatePersistence` | 内存,仅最新 | 默认 | +| `FullStatePersistence` | 内存列表 | 支持 `dump_json()` / `load_json()` | +| `FileStatePersistence` | JSON 文件 | 文件锁,适合单 run 恢复 | + +快照结构(`NodeSnapshot`)包含:执行前完整 state、节点实例、时间戳、状态(created/pending/running/success/error)。可通过 `graph.initialize()` + `graph.iter_from_persistence()` 实现跨进程恢复。 + +**关键限制**:**Stable API 不支持并行节点执行**(GitHub issue #704 open)。所有节点严格串行推进。 + +### 2.2 Beta API + +Beta API 引入构建器模式和真正的并行能力。数据流模型(`StepContext.inputs` 携带类型化输入)与当前 `Link.mapping` 的语义更接近: + +```python +g = GraphBuilder(state_type=MyState, input_type=list[int], output_type=list[int]) + +@g.step +async def process_item(ctx: StepContext[MyState, None, int]) -> int: + return ctx.inputs * 2 # 类型化的上游输入 + +collect = g.join(reduce_list_append, initial_factory=list[int]) + +g.add( + g.edge_from(g.start_node).map().to(process_item), # 并行扇出 + g.edge_from(process_item).to(collect), + g.edge_from(collect).to(g.end_node), +) +``` + +**Beta API 关键限制**:**没有内建持久化**。官方明确说明原因是并行执行时一致性快照的复杂性。官方建议配合 Temporal / DBOS / Prefect 做持久化。 + +### 2.3 多 Actor 并发的技术结论 + +pydantic-graph 的并发支持现状: + +| 场景 | Stable API | Beta API | +|------|-----------|----------| +| 单 Actor 循环 | 支持(节点返回自身) | 支持(`@g.stream` 或循环) | +| 并行扇出(一对多) | 不支持 | 支持(`.map()` + `.join()`) | +| 多 Actor pipeline(A→B→C) | 不支持并发 | 不支持(无内建 pipeline 模式) | +| 多 Actor 双向通信 | 不支持 | 不支持 | + +**缓解策略**:多 Actor pipeline 场景,adapter 层在 pydantic-graph 之外用 asyncio 并发运行多个 graph 实例,通过 `asyncio.Queue` 传递消息。这把多 Actor 协调责任完全封装在 `molexp.workflow.runtime` 内部,用户侧 API 不感知。 + +--- + +## 3. 设计目标 + +### 3.1 产品目标 + +- 保留并强化 `workspace / assets / project / experiment / run` +- 让 workflow kernel 不再自研维护 +- 让 agent/autonomy 能力成为一等公民 +- 用户只面向 `molexp` API,不接触底层第三方库 + +### 3.2 工程目标 + +- **彻底重构,不向后兼容**:删除现有 `WorkflowCompiler`、`WorkflowEngine`、`Link`,不保留任何 shim。项目处于高速开发期,技术债务清零优先于迁移平滑。 +- 降低自研 workflow runtime 维护成本 +- API 风格向 pydantic-graph / PydanticAI 借鉴,同时保留 OOP 类写法(两种风格均一等支持) +- 不让第三方库类型污染 `molexp` 公共 API + +### 3.3 非目标 + +- 不把 `molexp` 变成 `PydanticAI` 的薄皮壳 +- 不让用户直接写 `Agent(...)` 或 `Graph(...)` +- 不为旧用户提供迁移脚本或自动升级工具 + +--- + +## 4. API 设计哲学 + +### 4.1 双风格统一支持 + +molexp 的新 API 借鉴 pydantic-graph 的函数式装饰器风格,同时保留完整的 OOP 类写法。两种风格在运行时等价,用户根据场景和偏好选择,底层统一编译到相同的 pydantic-graph 节点。 + +#### Workflow Step — 两种写法 + +**函数式(借鉴 `@g.step`)**: + +```python +from molexp.workflow import workflow, StepContext + +wf = workflow(name="data-pipeline") + +@wf.step(depends_on=["fetch"]) +async def validate(ctx: StepContext[DataState, DataDeps, FetchResult]) -> ValidateResult: + data = ctx.inputs # 类型化的上游输入(FetchResult) + ctx.state.validated += 1 # 共享状态 + return ValidateResult(data=cleaned(data)) +``` + +**OOP(保留类继承)**: + +```python +from molexp.workflow import Step, StepContext + +class ValidateStep(Step[DataState, DataDeps, FetchResult, ValidateResult]): + config: ValidateConfig # Pydantic 模型,config-as-state 模式 + + async def execute(self, ctx: StepContext[DataState, DataDeps, FetchResult]) -> ValidateResult: + data = ctx.inputs + ctx.state.validated += 1 + return ValidateResult(data=cleaned(data, threshold=self.config.threshold)) +``` + +#### Actor — 两种写法 + +Actor 在底层编译为 pydantic-graph 的循环节点(节点返回自身)。多 Actor pipeline 由 `MultiActorRunner` 以 asyncio 并发调度。 + +**函数式**: + +```python +@wf.actor(depends_on=["ingest"]) +async def filter_stream( + ctx: ActorContext[StreamState, StreamDeps, RawEvent] +) -> AsyncIterator[FilteredEvent]: + while True: + msg = await ctx.receive() + if msg.quality > ctx.state.threshold: + yield FilteredEvent(msg) +``` + +**OOP**: + +```python +from molexp.workflow import Actor, ActorContext + +class FilterActor(Actor[StreamState, StreamDeps, RawEvent, FilteredEvent]): + config: FilterConfig + + async def run( + self, ctx: ActorContext[StreamState, StreamDeps, RawEvent] + ) -> AsyncIterator[FilteredEvent]: + while True: + msg = await ctx.receive() + if msg.quality > self.config.threshold: + yield FilteredEvent(msg) +``` + +#### Agent Tool — 两种写法 + +**函数式(借鉴 `@agent.tool`)**: + +```python +from molexp.agent import agent_tool, ToolContext + +@agent_tool(level="workspace", requires_approval=False) +async def read_asset(ctx: ToolContext, asset_name: str) -> AssetContent: + return ctx.workspace.assets.read(asset_name) +``` + +**OOP**: + +```python +from molexp.agent import Tool, ToolContext + +class ReadAssetTool(Tool): + name = "workspace.read_asset" + level = "workspace" + requires_approval = False + + async def call(self, ctx: ToolContext, asset_name: str) -> AssetContent: + return ctx.workspace.assets.read(asset_name) +``` + +### 4.2 核心上下文类型 + +借鉴 pydantic-graph 的 `StepContext[StateT, DepsT, InputT]` 设计,molexp 暴露三个统一的上下文类型: + +```python +# Workflow Step 上下文(对应 pydantic-graph StepContext) +class StepContext(Generic[StateT, DepsT, InputT]): + state: StateT # 共享可变状态 + deps: DepsT # 注入的依赖(workspace、run、外部服务) + inputs: InputT # 类型化的上游输入 + +# Actor 上下文(扩展 StepContext,增加消息收发) +class ActorContext(StepContext[StateT, DepsT, InputT]): + async def receive(self) -> InputT: ... + async def send(self, output: OutputT) -> None: ... + +# Agent Tool 上下文 +class ToolContext: + workspace: Workspace + run: Run | None + session: AgentSession +``` + +### 4.3 与 pydantic-graph 的风格对照 + +| 概念 | pydantic-graph 原始 | molexp 对应 | +|------|---------------------|-------------| +| 节点定义 | `@dataclass class Foo(BaseNode)` | `@wf.step` / `class FooStep(Step)` | +| 上下文 | `GraphRunContext[StateT]` | `StepContext[StateT, DepsT, InputT]` | +| 输入数据 | 节点 dataclass 字段 | `ctx.inputs`(类型化) | +| 共享状态 | `ctx.state` | `ctx.state`(同) | +| 终止 | `End(data)` | `WorkflowResult`(内部封装) | +| 持久化 | `BaseStatePersistence` | `RunStorePersistence`(实现该接口) | +| 并行扇出 | `.map()` + `.join()` | `parallel_map()` + `join()` | + +--- + +## 5. 用户侧用例 + +### 5.1 Workspace 层用例 + +**用例 1:初始化工作区并组织项目** + +```python +from molexp.workspace import Workspace + +ws = Workspace.from_env() + +project = ws.create_project( + name="Polymer Stability Study", + description="GNN-based stability prediction for polymer chains", +) + +experiment = project.create_experiment( + name="Baseline GNN", + workflow_source="workflows/train_gnn.py", + tags=["baseline", "gnn"], +) +``` + +**用例 2:创建 Run 并记录资产与指标** + +```python +run = experiment.create_run( + parameters={"lr": 1e-3, "hidden_dim": 128, "epochs": 100}, +) + +with run.start() as ctx: + ctx.log("Training started") + ctx.assets.register("training_data", path="/data/polymer_train.h5") + + model, metrics = train(ctx.parameters) + + ctx.log_metrics({"val_loss": metrics.val_loss, "mae": metrics.mae}) + ctx.assets.create("checkpoint", model.save("/tmp/model.pt")) + ctx.assets.create("predictions", "/tmp/predictions.csv") +``` + +**用例 3:比较多个 Run** + +```python +runs = experiment.list_runs() +comparison = experiment.compare_runs( + run_ids=[r.run_id for r in runs], + metrics=["val_loss", "mae"], + parameters=["lr", "hidden_dim"], +) +comparison.to_dataframe().sort_values("val_loss") +``` + +**用例 4:层级资产库** + +```python +ws.assets.create("bert_pretrained", "/models/bert-base.pt") # 全局 +project.assets.create("polymer_dataset", "/data/qm9.tar.bz2") # 项目内 +experiment.assets.create("mol_features", "/data/features.h5") # 实验内 +run.assets.create("run_output", "/outputs/results.json") # 单次执行 + +# 向上查找:run → experiment → project → workspace +asset = run.assets.resolve("bert_pretrained") +``` + +--- + +### 5.2 Workflow 层用例 + +**用例 1:函数式风格定义线性 workflow** + +```python +from molexp.workflow import workflow, StepContext +from molexp.workspace import DataState, DataDeps + +wf = workflow(name="data-pipeline") + +@wf.step +async def fetch(ctx: StepContext[DataState, DataDeps, None]) -> FetchResult: + return FetchResult(data=ctx.deps.storage.read(ctx.state.source)) + +@wf.step(depends_on=["fetch"]) +async def validate(ctx: StepContext[DataState, DataDeps, FetchResult]) -> ValidateResult: + ctx.state.row_count = len(ctx.inputs.data) + return ValidateResult(data=clean(ctx.inputs.data)) + +@wf.step(depends_on=["validate"]) +async def publish(ctx: StepContext[DataState, DataDeps, ValidateResult]) -> PublishResult: + path = ctx.deps.storage.write(ctx.inputs.data) + ctx.run.assets.create("cleaned_dataset", path) + return PublishResult(asset_id="cleaned_dataset") + +run = experiment.create_run(parameters={"source": "s3://bucket/raw"}) +result = await wf.execute(run=run) +print(result.outputs["publish"].asset_id) +``` + +**用例 2:OOP 风格定义 workflow(可复用、可配置)** + +```python +from molexp.workflow import Step, StepContext, WorkflowBuilder + +class FetchStep(Step[DataState, DataDeps, None, FetchResult]): + config: FetchConfig + + async def execute(self, ctx: StepContext[DataState, DataDeps, None]) -> FetchResult: + return FetchResult(data=ctx.deps.storage.read(self.config.source)) + +class ValidateStep(Step[DataState, DataDeps, FetchResult, ValidateResult]): + config: ValidateConfig + + async def execute(self, ctx: StepContext[DataState, DataDeps, FetchResult]) -> ValidateResult: + ctx.state.row_count = len(ctx.inputs.data) + return ValidateResult(data=clean(ctx.inputs.data, schema=self.config.schema)) + +# 组装 workflow(与函数式等价,底层同一套编译路径) +wf = ( + WorkflowBuilder(name="data-pipeline") + .add(FetchStep(config=FetchConfig(source="s3://bucket/raw"))) + .add(ValidateStep(config=ValidateConfig(schema="polymer_v2")), depends_on=["fetch"]) + .build() +) + +result = await wf.execute(run=run) +``` + +**用例 3:并行扇出(Map / Reduce)** + +```python +from molexp.workflow import workflow, StepContext, parallel_map, join + +wf = workflow(name="grid-search") + +@wf.step +async def prepare(ctx: StepContext[SearchState, None, None]) -> PrepareResult: + return PrepareResult(grid=ctx.state.parameter_grid) + +@parallel_map(wf, fan_out_over="grid", depends_on=["prepare"]) +async def train_variant(ctx: StepContext[SearchState, None, GridPoint]) -> TrainResult: + # 每个 GridPoint 并发执行,底层用 pydantic-graph beta API .map() + model = train(lr=ctx.inputs.lr, hidden=ctx.inputs.hidden) + return TrainResult(val_loss=model.val_loss, config=ctx.inputs) + +@join(wf, reducer="best_by_val_loss", depends_on=["train_variant"]) +async def select_best(ctx: StepContext[SearchState, None, list[TrainResult]]) -> BestResult: + return min(ctx.inputs, key=lambda r: r.val_loss) + +run = experiment.create_run(parameters={ + "parameter_grid": [ + {"lr": 1e-3, "hidden": 64}, + {"lr": 1e-4, "hidden": 128}, + {"lr": 5e-4, "hidden": 256}, + ] +}) +result = await wf.execute(run=run) +print(result.outputs["select_best"].config) +``` + +**用例 4:Workflow 暂停与恢复** + +```python +# 启动,中途进程崩溃 +execution = await wf.start(run=run) +print(execution.execution_id) # "exec-abc123" + +# 重启后恢复(从 Run store 加载最后快照) +execution = await wf.resume(run=run, execution_id="exec-abc123") +result = await execution.wait() +print(result.status) # "completed" +``` + +**用例 5:流式 Actor Workflow** + +```python +from molexp.workflow import workflow, Actor, ActorContext + +wf = workflow(name="streaming-pipeline", mode="streaming") + +class KafkaIngest(Actor[StreamState, StreamDeps, None, RawEvent]): + config: KafkaConfig + + async def run(self, ctx: ActorContext[StreamState, StreamDeps, None]) -> AsyncIterator[RawEvent]: + async for msg in ctx.deps.kafka.consume(self.config.topic): + yield RawEvent(msg) + +@wf.actor(depends_on=["ingest"]) +async def quality_filter(ctx: ActorContext[StreamState, StreamDeps, RawEvent]) -> AsyncIterator[CleanEvent]: + while True: + msg = await ctx.receive() + if msg.quality > ctx.state.threshold: + yield CleanEvent(msg) + +wf.add(KafkaIngest(config=KafkaConfig(topic="mol-events")), name="ingest") + +run = experiment.create_run(parameters={"threshold": 0.8}) +async with wf.stream(run=run) as stream: + async for event in stream: + print(event) +``` + +**用例 6:步进执行(调试)** + +```python +async with wf.iter(run=run) as graph_run: + async for step_event in graph_run: + print(f"[{step_event.step_name}] {step_event.status}") + if step_event.step_name == "validate" and step_event.status == "failed": + print("State at failure:", step_event.state) + break +``` + +--- + +### 5.3 Agent 层用例 + +**用例 1:目标驱动的自主实验** + +```python +from molexp.agent import Goal, AgentService + +goal = Goal( + description="Prepare a validated polymer dataset and run a GNN baseline", + constraints={"project": "polymer-study", "max_runs": 3}, + success_criteria=[ + "dataset validated and published as experiment asset", + "at least one training run completed with val_loss < 0.05", + ], +) + +service = AgentService.from_workspace("./lab") +session = await service.run(goal) + +print(session.status) # "completed" +print(session.produced_runs) # [Run(...), Run(...)] +print(session.artifacts) # [Asset("dataset"), Asset("checkpoint")] +``` + +**用例 2:监控 Session 时间线** + +```python +session = await service.start(goal) + +async for event in session.stream_events(): + match event: + case PlanCreatedEvent(): + print("Plan:", event.plan_steps) + case ToolCallEvent(): + print(f"→ {event.tool_name}({event.args})") + case WorkflowStartedEvent(): + print(f" workflow run: {event.run_id}") + case ObservationEvent(): + print(f" observation: {event.content}") + case ReplanEvent(): + print(f" replanning: {event.reason}") + case SessionCompletedEvent(): + print(f"Done: {event.summary}") + break +``` + +**用例 3:函数式注册自定义 Tool** + +```python +from molexp.agent import agent_tool, ToolContext + +# 用户可以用装饰器给 agent 注册自定义 tool +@agent_tool(level="product", requires_approval=True) +async def run_bayesian_step( + ctx: ToolContext, experiment_id: str, iteration: int +) -> BayesianStepResult: + experiment = ctx.workspace.get_experiment(experiment_id) + run = experiment.create_run(parameters=suggest_next(iteration)) + result = await training_workflow.execute(run=run) + return BayesianStepResult(val_loss=result.outputs["train"].val_loss, run_id=run.run_id) + +service = AgentService.from_workspace("./lab", extra_tools=[run_bayesian_step]) +session = await service.run(goal) +``` + +**用例 4:OOP 风格注册 Tool** + +```python +from molexp.agent import Tool, ToolContext + +class RunBayesianStep(Tool): + name = "product.run_bayesian_step" + level = "product" + requires_approval = True + + async def call( + self, ctx: ToolContext, experiment_id: str, iteration: int + ) -> BayesianStepResult: + experiment = ctx.workspace.get_experiment(experiment_id) + run = experiment.create_run(parameters=suggest_next(iteration)) + result = await training_workflow.execute(run=run) + return BayesianStepResult(val_loss=result.outputs["train"].val_loss, run_id=run.run_id) + +service = AgentService.from_workspace("./lab", extra_tools=[RunBayesianStep()]) +``` + +**用例 5:Human-in-the-loop 审批** + +```python +from molexp.agent import ApprovalPolicy + +policy = ApprovalPolicy( + require_approval_for=["product.*", "workflow.execute"], + auto_approve=["workspace.read_*", "workflow.inspect"], +) + +service = AgentService.from_workspace("./lab", approval_policy=policy) +session = await service.start(goal) + +async for event in session.stream_events(): + if isinstance(event, ApprovalRequestEvent): + print(f"Approval needed: {event.tool_name}({event.args})") + decision = input("Approve? [y/n]: ") + await session.respond_approval(event.request_id, approved=(decision == "y")) +``` + +**用例 6:Session 恢复与重放** + +```python +# 进程重启后恢复 +session = await service.resume(session_id="sess-xyz789") +async for event in session.stream_events(): ... + +# 事后重放时间线 +history = await service.get_session_history("sess-xyz789") +for entry in history.timeline: + print(f"[{entry.ts}] {entry.event_type}: {entry.summary}") +``` + +--- + +## 6. 内部架构 + +### 6.1 分层结构 + +```mermaid +graph TD + A[User / UI / API] --> B[Molexp Facade Layer] + B --> C[Molexp Agent Service] + B --> D[Molexp Workflow Service] + B --> E[Molexp Workspace Service] + + C --> F[PydanticAI Adapter] + F --> G[Tool Catalog] + F --> H[Session Store] + + D --> I[WorkflowGraphCompiler] + I --> J[Graph Runtime Adapter] + J --> K1[pydantic-graph Stable API
BATCH + 持久化] + J --> K2[pydantic-graph Beta API
并行扇出] + J --> K3[MultiActorRunner
asyncio + Queue] + + G --> E + G --> D + E --> L[Workspace / Assets / Project / Experiment / Run] + D --> L + H --> L +``` + +### 6.2 Workflow 编译路径 + +``` +WorkflowSpec(用户定义,函数式 or OOP) + ↓ WorkflowGraphCompiler + ├─ 纯 BATCH DAG → pydantic-graph Stable API + RunStorePersistence + ├─ 含并行扇出 → pydantic-graph Beta API + 手动快照 + └─ 含 Actor(streaming)→ MultiActorRunner(asyncio 并发) + ↓ +WorkflowExecution / WorkflowResult + → 写入 Run store(真相源) +``` + +### 6.3 持久化适配 + +对 Stable API,实现 `RunStorePersistence(BaseStatePersistence)` 将每个 `NodeSnapshot` 原子写入 `Run` store: + +```python +class RunStorePersistence(BaseStatePersistence[StateT, RunEndT]): + def __init__(self, run: Run): ... + + async def snapshot_node(self, snapshot: NodeSnapshot) -> None: + # 原子写入 run.store/execution/{snapshot.id}.json + ... + + async def load_next(self) -> NodeSnapshot | None: + # 读取 status='created' 的最早快照 + ... +``` + +对 Beta API(无内建持久化),adapter 监听 step 完成事件,主动快照 state 到 Run store,格式与 Stable API 一致。 + +### 6.4 多 Actor Pipeline 封装 + +```python +class MultiActorRunner: + """在 pydantic-graph 之外协调多个并发 Actor graph 实例。""" + + def __init__(self, actor_specs: list[ActorStepSpec], run: Run): + self._graphs: dict[str, Graph] = {} + self._channels: dict[tuple[str, str], asyncio.Queue] = {} + + async def start(self) -> None: + # 1. 为每条 actor link 分配 asyncio.Queue,注入各 GraphState + # 2. asyncio.gather 并发启动所有 graph 实例 + # 3. 异常传播:任一 actor 失败 → 取消其余,写 run.status=failed + ... +``` + +`WorkflowSpec(mode="streaming")` 经 compiler 自动路由到 `MultiActorRunner`,用户不感知。 + +### 6.5 Agent Tool 分层 + +``` +Level 1: Workspace tools(只读,无需审批) + workspace.list_projects / list_experiments / list_assets / read_asset / get_run_metrics + +Level 2: Workflow tools(执行类,高风险需审批) + workflow.execute / resume / inspect / fetch_outputs / cancel + +Level 3: Product tools(写操作,部分需审批) + experiment.create_run / run.read_logs / run.publish_artifact / run.compare_results +``` + +内部全部映射到 `PydanticAI Toolset`,用户只见 `molexp` tool 名称。 + +### 6.6 PydanticAI 事件映射 + +| PydanticAI 内部事件 | Molexp 事件 | +|---------------------|-------------| +| Agent 开始规划 | `PlanCreatedEvent(plan_steps)` | +| `FunctionToolCallEvent` | `ToolCallEvent(tool_name, args)` | +| `FunctionToolResultEvent` | `ToolResultEvent(tool_name, result)` | +| Agent 检测到失败,重规划 | `ReplanEvent(reason, new_plan)` | +| `ApprovalRequiredToolset` 拦截 | `ApprovalRequestEvent(request_id, tool_name, args)` | +| Agent `End` | `SessionCompletedEvent(summary, artifacts)` | + +--- + +## 7. 彻底重构说明 + +**删除的模块**: + +| 模块 | 替代 | +|------|------| +| `molexp.workflow.compiler.core` | `molexp.workflow.compiler.WorkflowGraphCompiler` | +| `molexp.workflow.engine.engine` | `molexp.workflow.runtime.pydantic_graph` | +| `molexp.workflow.task.Task` / `Actor` | `molexp.workflow.Step` / `Actor`(新 API) | +| `molexp.workflow.workflow.Workflow` | `molexp.workflow.WorkflowSpec` | +| `molexp.workflow.workflow.Link` | `StepContext.inputs` + `input_mapping` | + +**无向后兼容**: + +- 不保留旧 `Task`、`Link`、`WorkflowCompiler`、`WorkflowEngine` 的 import path +- 不提供 shim 或 deprecation warning +- 现有代码需按新 API 完整重写 +- 测试套件同步重写(旧测试直接删除,按新 API 重新覆盖) + +--- + +## 8. 实施计划 + +### Phase 1:彻底删除旧 kernel,定义新接口 + +**目标**:清空自研 workflow runtime,定义新公共接口和内部 adapter contract。 + +工作项: + +- **删除**:`workflow/compiler/`、`workflow/engine/`、旧 `workflow/task.py`、旧 `workflow/workflow.py` +- 定义 `molexp.workflow` 新公共接口(`WorkflowSpec`、`Step`、`Actor`、`StepContext`、`ActorContext`、`workflow()` 装饰器、`WorkflowBuilder`) +- 定义 `molexp.agent` 新公共接口(`Goal`、`AgentSession`、`AgentService`、`Tool`、`agent_tool`、`ApprovalPolicy`) +- 定义内部 adapter interface(`WorkflowRuntime`、`AgentRuntime`、`ToolCatalogAdapter`、`RunStorePersistence`) +- 按新接口重写 contract 测试(不测实现,只测接口行为) + +交付:新 API 类型定义冻结,旧 workflow 代码全部清除,新接口有 contract 测试。 + +### Phase 2:接入 `PydanticAI`(Agent 层) + +**目标**:全量引入 agent/tool/session 层,不依赖 Phase 3 完成。 + +工作项: + +- 实现 `PydanticAIAdapter`(封装 `pydantic_ai.Agent`) +- 实现 `MolexpToolCatalog`(三层 tool → `PydanticAI Toolset`) +- 接入 session event stream(PydanticAI 事件 → molexp 事件) +- `ApprovalPolicy` + `ApprovalRequiredToolset` 集成 +- Session 持久化写入 workspace(`workspace/sessions/{session_id}/`) +- UI:Session detail、Tool call timeline、Approval panel + +交付:`Goal → AgentSession → Tool Calls → Observations` 完整闭环,UI 可展示 session timeline。 + +### Phase 3:用 `pydantic-graph` 实现 workflow runtime + +**目标**:实现新接口背后的 graph-backed runtime。 + +工作项: + +- 实现 `WorkflowGraphCompiler`(`WorkflowSpec` → pydantic-graph 节点,支持函数式和 OOP 两种定义方式) +- 实现 `RunStorePersistence`(`BaseStatePersistence` → Run store) +- 实现 `GraphWorkflowRuntime`(Stable API 路径 + Beta API 路径) +- 实现 `MultiActorRunner`(asyncio 并发 + 多 graph 实例) +- UI:Workflow graph 可视化(pydantic-graph mermaid 输出) + +交付:`molexp.workflow` 新 API 完整可用,现有 workspace 测试全部通过。 + +### Phase 4:产品化 + +**目标**:Workflow + Agent 联合可视化,完整自主驱动体验。 + +工作项: + +- Goal input UI +- Session + Workflow runs 联合视图 +- Replanning 可视化 +- Human-in-the-loop approval flow +- Session replay +- Multi-session scheduling + +--- + +## 9. 团队分工 + +假设 8 人小组,角色分为三类:**产品与协调**、**工程**、**质量保障**。三条工程主线(Workspace、Workflow、Agent)并行推进,PM 和 QA 横切全程。 + +### 角色总览 + +| 角色 | 人数 | 主要职责 | +|------|------|---------| +| PM / Tech Lead | 1 | 产品目标、API 设计决策、跨成员协调、验收 | +| QA Engineer | 1 | 测试策略、质量门控、集成测试、CI/CD | +| Core & Workspace | 1 | 共享基础类型、workspace 稳定 | +| Workflow API | 1 | 公共 workflow DSL(函数式 + OOP) | +| Workflow Runtime | 1 | pydantic-graph adapter、MultiActorRunner | +| Agent | 1 | PydanticAI 接入、session、tool | +| Platform & UI | 2 | FastAPI 路由、React 前端 | + +--- + +### PM / Tech Lead(1 人) + +**职责**: + +- **产品层**:维护并优先级排序 Phase 1-4 的功能列表;基于用户侧用例定义每个 Phase 的验收标准;决策 API 设计中的取舍(函数式 vs OOP 优先级、哪些能力纳入 MVP) +- **技术协调**:主持每周接口评审会(Interface Review);负责跨模块 API 冻结决策(尤其是 `StepContext`、`WorkflowRuntime` interface、`ToolContext` 等共享类型);协调 B/C 之间的接口对齐 +- **验收**:每个 Phase 交付前,对照第 5 节"用户侧用例"逐条手工验证;记录 gap 并分派修复 +- **外部依赖追踪**:跟踪 pydantic-graph beta API 变更(release notes);评估何时从 asyncio 多实例方案迁移到 beta 内建 pipeline + +**与其他角色的接口**: +- 向 QA 输出每个 Phase 的验收测试场景清单 +- 向工程成员输出接口冻结通知(冻结后的变更需评审) +- 接收 QA 的质量报告,决定是否 ready to release + +--- + +### QA Engineer(1 人) + +**职责**: + +- **测试策略制定**(Phase 1):定义三层测试金字塔——单元测试(各成员自持)、集成测试(QA 主持)、端到端测试(QA 主持);制定覆盖率基线(unit ≥ 80%,集成测试覆盖所有跨层 API 路径) +- **质量门控**:在 CI/CD 中配置 Phase 门控——每个 Phase 合并前必须通过全量测试;配置覆盖率检查、类型检查(mypy strict)、linting(ruff) +- **集成测试主持**:负责编写并维护跨层集成测试: + - `WorkflowSpec → Run store 持久化` 全链路 + - `Goal → AgentSession → Tool Calls → Run 产出` 全链路 + - `Workspace 层级资产解析` 回归套件 + - `MultiActorRunner` 并发安全专项测试(Actor 启动/消息/异常/关闭) +- **端到端测试**:Phase 4 前建立 E2E 套件,覆盖第 5 节全部用户侧用例(含 UI 操作流) +- **缺陷管理**:记录跨模块缺陷,协调 owner 修复;追踪 regression 率 + +**与其他角色的接口**: +- 接收 PM 输出的验收场景清单 → 转化为可执行测试用例 +- 向各工程成员输出单元测试模板和 mock 规范(避免各模块 mock 方式不一致) +- 向 PM 输出每个 Phase 的质量报告(覆盖率、通过率、已知 bug) +- 与成员 A 共同维护 workspace 层回归套件(此层变更最敏感) + +--- + +### 成员 A:Core & Workspace(1 人) + +**负责模块**:`molexp.workspace`、`molexp.core`(基础类型、异常、注册表) + +**职责**: +- 保持 workspace 层在重构期间稳定可用(是 Workflow 和 Agent 的依赖基础) +- 定义 Phase 1 共享类型:`StepContext`、`ActorContext`、`ToolContext`、`RunStorePersistence` 接口(与 PM 对齐后冻结) +- 实现 `RunStorePersistence`(写入 Run store 的原子逻辑) +- workspace 层单元测试保持绿色;与 QA 共同维护回归套件 + +**Phase 依赖**:Phase 1 最先启动,其他工程成员阻塞于共享类型定义。 + +--- + +### 成员 B:Workflow 公共 API(1 人) + +**负责模块**:`molexp.workflow`(公共 API 层) + +**职责**: +- 定义 `WorkflowSpec`、`Step`(OOP 基类)、`Actor`(OOP 基类) +- 实现 `workflow()` 装饰器、`WorkflowBuilder`、`parallel_map()`、`join()`、`@wf.actor` +- 定义 `WorkflowExecution`、`WorkflowResult` 类型 +- 与成员 C 协作定义 `WorkflowRuntime` adapter interface(经 PM 评审后冻结) +- 负责公共 API 层单元测试;向 QA 提供集成测试所需的 mock runtime + +**Phase 依赖**:Phase 1 完成后即可开始,与成员 C 并行。 + +--- + +### 成员 C:Workflow Runtime(1 人) + +**负责模块**:`molexp.workflow.runtime.pydantic_graph`、`MultiActorRunner` + +**职责**: +- 实现 `WorkflowGraphCompiler`(`WorkflowSpec` → pydantic-graph 节点) +- 实现 `GraphWorkflowRuntime`(Stable API + Beta API 双路径) +- 实现 `MultiActorRunner`(asyncio 并发多 Actor pipeline) +- Beta API 手动持久化适配(step 完成事件 → Run store 快照) +- 单元测试覆盖各编译路径;Actor 并发场景交给 QA 做集成测试 + +**Phase 依赖**:依赖成员 B 定义的 `WorkflowRuntime` interface(冻结后并行开发)。 + +--- + +### 成员 D:Agent 层(1 人) + +**负责模块**:`molexp.agent`、`molexp.agent.runtime.pydantic_ai` + +**职责**: +- 实现 `PydanticAIAdapter`(封装 `pydantic_ai.Agent`) +- 实现 `MolexpToolCatalog`(三层 tool → PydanticAI Toolset) +- `ApprovalPolicy` + `ApprovalRequiredToolset` 集成 +- Session event stream 映射(PydanticAI 事件 → molexp 事件) +- Session 持久化写入 workspace +- `@agent_tool` 装饰器 + `Tool` OOP 基类 +- 单元测试可用 mock workspace 隔离;端到端 Agent 场景交给 QA + +**Phase 依赖**:Phase 2,可与成员 C 完全并行。 + +--- + +### 成员 E/F:Platform & UI(2 人) + +**负责模块**:`molexp.server`(FastAPI)、`ui/` + +**职责分工**: + +- **成员 E(Backend Platform)**:FastAPI 路由(`/api/workflow/*`、`/api/agent/*`);OpenAPI → TypeScript codegen 维护;WebSocket / SSE for session event stream;与 B/C/D 协作接口联调 +- **成员 F(Frontend)**:Session detail、Tool call timeline、Approval panel(Phase 2);Workflow graph 可视化(Phase 3);Goal input UI、联合视图(Phase 4) + +**Phase 依赖**:横切所有 Phase,随其他成员交付逐步推进。 + +--- + +### 并行关系与里程碑 + +``` + Week 1-2 Week 3-6 Week 7-10 Week 11+ + ────────────────────────────────────────────────────────────── +PM: [接口评审+API冻结决策]─────[Phase验收]──[Phase验收]──[Phase验收]── +QA: [测试策略+CI配置]──[集成测试套件]───[E2E套件]────────[质量报告]──── +A: [共享类型+Workspace稳定]───────────────────────────────────────── +B: [Phase 1: Workflow公共API]──[Phase 3: DSL完善]─────────────────── +C: [Phase 3: Graph Adapter + MultiActorRunner]─────────── +D: [Phase 2: PydanticAI Agent层]────────────────────────────── +E: [路由skeleton]──[Phase 2: 后端API]──[Phase 3: API扩展]──[Phase 4] +F: [UI框架]────────[Phase 2: Session UI]─[Phase 3: Graph UI]─[Phase 4] + ↓ + [Phase 4: 产品化] +``` + +**关键依赖链**: + +1. A 先于所有人完成共享类型冻结(`StepContext`、`RunStorePersistence` 接口) +2. PM 评审 B/C 的 `WorkflowRuntime` interface → 冻结 → C 才能全速推进 +3. D 完全独立,与 B/C 并行 +4. QA 在 Phase 1 末建好 CI 质量门控,阻断不合格的 Phase 合并 + +**接口冻结协议**:所有跨模块共享类型(`StepContext`、`WorkflowRuntime`、`ToolContext` 等)需经 PM 主持的 Interface Review 会议冻结;冻结后变更需重新评审,由 PM 决策影响范围。 + +--- + +## 10. 风险与缓解 + +### 10.1 Beta API 无持久化 + +风险:并行扇出 workflow 无内建快照,进程崩溃后无法恢复。 + +缓解:adapter 层在每个 step 完成事件后主动写 state 快照到 Run store,用户无感知。 + +### 10.2 pydantic-graph API 演进 + +风险:Beta API 仍在演化,接口可能变更。 + +缓解:所有 pydantic-graph 调用隔离在 `molexp.workflow.runtime.pydantic_graph` 内部,对外只暴露 `molexp` 类型。未来切换只改 adapter 实现。 + +### 10.3 多 Actor Pipeline 并发安全 + +风险:`MultiActorRunner` 逻辑复杂,难以调试。 + +缓解:写专项集成测试,覆盖 Actor 启动、消息传递、异常传播、优雅关闭。Beta API 未来如加入 pipeline 模式,可替换实现而不改用户接口。 + +### 10.4 彻底删除旧 kernel 的回归风险 + +风险:删除 `WorkflowCompiler` / `WorkflowEngine` 后,现有高层测试可能大量失败。 + +缓解:Phase 1 同步删除旧测试,按新 API contract 重写,不试图让旧测试通过新实现。workspace 层测试(不依赖 workflow runtime)保持绿色作为回归防线。 + +--- + +## 11. 成功标准 + +- 用户只使用 `molexp` API,不直接依赖 `pydantic_ai` 或 `pydantic_graph` 类型 +- `workspace` 仍然是唯一产品真相源 +- Workflow 支持函数式和 OOP 两种风格,底层编译路径统一 +- `PydanticAI` 完整承载 `molexp.agent`(goal → session → tools → observations) +- `pydantic-graph` 完整承载 `molexp.workflow` runtime(BATCH + 并行扇出 + Actor 循环) +- workflow run / agent session / assets 能统一出现在同一产品视图中 +- workspace / run / asset 测试全部通过 +- 用户可以用 5 行代码给一个 goal,得到一个完整的实验结果 + +--- + +## 12. 参考资料 + +- pydantic-graph Overview:https://ai.pydantic.dev/graph/ +- pydantic-graph Graph API:https://ai.pydantic.dev/api/pydantic_graph/graph/ +- pydantic-graph Persistence API:https://ai.pydantic.dev/api/pydantic_graph/persistence/ +- pydantic-graph Beta:https://ai.pydantic.dev/graph/beta/ +- pydantic-graph Beta API:https://ai.pydantic.dev/api/pydantic_graph/beta/ +- PydanticAI Agents:https://ai.pydantic.dev/agent/ +- PydanticAI Toolsets:https://ai.pydantic.dev/toolsets/ +- PydanticAI UI Overview:https://ai.pydantic.dev/ui/overview/ +- PydanticAI Durable Execution:https://ai.pydantic.dev/durable_execution/overview/ diff --git a/docs/get-started/quick-start.md b/docs/get-started/quick-start.md index eeaefc84..9ebc3659 100644 --- a/docs/get-started/quick-start.md +++ b/docs/get-started/quick-start.md @@ -1,205 +1,156 @@ # Quick Start -Let's create a complete MolExp workflow in five minutes, from defining tasks to execution, experiencing MolExp's complete workflow. +Let's build a complete MolExp workflow in five minutes — from defining tasks to execution and workspace tracking. -## Step 1: Define Your First Task +## Step 1: Define Tasks -A Task is the most basic computation unit in MolExp. Each task needs to inherit from the `Task` base class, specify a configuration type, and implement the `execute` method. - -What is a task? A task is a pure functional computation unit that accepts outputs from upstream tasks as inputs, performs computation, and produces outputs. Task configuration is determined at construction time, ensuring task behavior is predictable and serializable. - -Why this design? Static configuration allows us to validate configuration correctness before execution and enables complete workflow serialization. This is crucial for reproducibility. - -Let's create a simple task that receives a number and returns its square: +A **Task** is the basic computation unit. Subclass `Task` and implement `execute()`. The `TaskContext` gives you access to upstream outputs, shared state, and workspace capabilities. ```python -from molexp.workflow.node import Task -from pydantic import BaseModel - -class SquareConfig(BaseModel): - """Configuration model defining task parameters""" - pass # This task doesn't need extra configuration - -class SquareTask(Task[SquareConfig, int]): - """Compute the square of an input number""" - config_type = SquareConfig - - def execute(self, value: int) -> int: +from molexp.workflow import Task, TaskContext + +class SquareTask(Task): + async def execute(self, ctx: TaskContext) -> float: + value: float = ctx.inputs return value * value ``` -## Step 2: Connect Tasks to Build a Workflow +For streaming tasks, use `Actor` with `run()` instead. + +## Step 2: Compose a Workflow -Now let's create multiple tasks and connect them. Suppose we want to square a number, then add 10: +Two equivalent styles are available. + +**Functional DSL** (for inline definitions): ```python -class AddConfig(BaseModel): - offset: int = 10 - -class AddTask(Task[AddConfig, int]): - config_type = AddConfig - - def execute(self, value: int) -> int: - return value + self.config.offset - -# Create task instances and establish dependencies -square = SquareTask(task_id="square") -add = AddTask(square, task_id="add", offset=10) -``` +from molexp.workflow import workflow, TaskContext -Note that `AddTask`'s first parameter is `square`, indicating that `add` depends on `square`'s output. This declarative dependency relationship makes the workflow structure clear at a glance. +wf = workflow(name="pipeline") -## Step 3: Register and Execute Tasks +@wf.task +async def square(ctx: TaskContext) -> float: + return ctx.inputs ** 2 -To execute tasks in a workflow, you need to register them with the task registry. The registry allows the execution engine to find and instantiate your tasks: +@wf.task(depends_on=["square"]) +async def add_bias(ctx: TaskContext) -> float: + return ctx.inputs + 10.0 + +spec = wf.build() +``` + +**OOP builder** (for reusable task classes): ```python -from molexp.ir.registry import registry - -# Register tasks with the registry -@registry.register("square", SquareConfig) -class SquareTask(Task[SquareConfig, int]): - config_type = SquareConfig - - def execute(self, value: int) -> int: - return value * value +from molexp.workflow import Task, WorkflowBuilder, TaskContext + +class SquareTask(Task): + async def execute(self, ctx: TaskContext) -> float: + return ctx.inputs ** 2 + +class AddBiasTask(Task): + async def execute(self, ctx: TaskContext) -> float: + return ctx.inputs + 10.0 -@registry.register("add", AddConfig) -class AddTask(Task[AddConfig, int]): - config_type = AddConfig - - def execute(self, value: int) -> int: - return value + self.config.offset +spec = ( + WorkflowBuilder(name="pipeline") + .add(SquareTask()) + .add(AddBiasTask(), depends_on=["square"]) + .build() +) ``` -Once tasks are registered, you can execute them directly or use them in workflows. For simple cases, you can call tasks directly: +## Step 3: Execute ```python -# Execute tasks directly -square = SquareTask(task_id="square") -result = square(5) # Returns 25 +import asyncio + +async def main(): + result = await spec.execute() + print(result) -add = AddTask(square, task_id="add", offset=10) -final_result = add(5) # Returns 35 (square(5) + 10) +asyncio.run(main()) ``` -For more complex workflows with multiple tasks and dependencies, you'll typically use the workspace API or load workflows from JSON files. The workflow compilation and execution happens automatically behind the scenes. +Same-level tasks run in parallel automatically — no extra configuration needed. -## Step 4: Use Workspace to Manage Experiments +## Step 4: Track with Workspace -In real scientific computing scenarios, we usually need to manage multiple experiments and runs. MolExp provides a complete workspace architecture: +In real scientific workflows you want reproducibility, artifact tracking, and parameter sweeps. MolExp's workspace layer handles all of that. ```python -from molexp.workspace.core import Workspace -from molexp.workflow.context import RunContext, use_run_context -from molexp.assets import AssetRepo +from molexp.workspace import Workspace -# Create workspace -workspace = Workspace.from_path("./my_workspace") +# Create workspace (side-effect-free until materialize()) +workspace = Workspace.from_path("./lab") -# Create project -project = workspace.create_project( - id="my_project", - name="My Research Project", -) +# Hierarchical API +project = workspace.create_project(name="My Research Project") +experiment = project.create_experiment(name="Experiment 1") +run = experiment.create_run(parameters={"lr": 0.01, "epochs": 10}) -# Create experiment -experiment = workspace.create_experiment( - id="my_project", - id="exp_1", - name="First Experiment", - workflow_source="workflow.py", -) +# Attach a run when executing so outputs are tracked automatically. +# Pass dry_run=True to expose ctx.dry_run inside tasks. +result = await spec.execute(run=run, dry_run=True) +``` -# Create run -run = workspace.create_run( - id="my_project", - id="exp_1", - parameters={"input_value": 5}, - workflow_file="workflow.py", -) +Inside tasks, access workspace capabilities via `ctx`: -# Use run context during execution -ctx = RunContext( - asset_repo=AssetRepo(), - id=run.id, - run_metadata=run, - workspace=workspace, -) +```python +class TrainTask(Task): + async def execute(self, ctx: TaskContext) -> dict: + if ctx.dry_run: + return {"loss": None, "mode": "dry-run"} -with use_run_context(ctx): - # Execute your workflow here - # Workflows are typically loaded from files or defined via the API - # All registered assets will be automatically tracked - pass -``` + model = train(ctx.inputs, **ctx.state) + + # Save artifact to run directory + ctx.save_artifact("model.pt", model) -## Complete Example + # Find a pre-registered asset up the scope hierarchy + dataset = ctx.find_asset("training_data") + + return {"loss": 0.05} +``` -Here's a complete runnable example showing the full flow from task definition to execution: +## Complete Runnable Example ```python -from molexp.workflow.node import Task -from molexp.ir.registry import registry -from molexp.workspace.core import Workspace -from molexp.workflow.context import RunContext, use_run_context -from molexp.assets import AssetRepo -from pydantic import BaseModel - -# Define configuration models -class SquareConfig(BaseModel): - pass - -class AddConfig(BaseModel): - offset: int = 10 - -# Register and define tasks -@registry.register("square", SquareConfig) -class SquareTask(Task[SquareConfig, int]): - config_type = SquareConfig - - def execute(self, value: int) -> int: - return value * value +import asyncio +from molexp.workflow import workflow, TaskContext +from molexp.workspace import Workspace + +wf = workflow(name="demo") + +@wf.task +async def fetch(ctx: TaskContext) -> list[float]: + return [1.0, 4.0, 9.0] + +@wf.task(depends_on=["fetch"]) +async def compute(ctx: TaskContext) -> float: + return sum(ctx.inputs) + +spec = wf.build() + +async def main(): + # Pure computation (no workspace) + result = await spec.execute() + print(f"Result: {result}") + + # With workspace tracking + workspace = Workspace.from_path("./lab") + project = workspace.create_project(name="Demo") + experiment = project.create_experiment(name="Run 1") + run = experiment.create_run(parameters={}) + + result = await spec.execute(run=run, dry_run=True) + print(f"Result (tracked): {result}") -@registry.register("add", AddConfig) -class AddTask(Task[AddConfig, int]): - config_type = AddConfig - - def execute(self, value: int) -> int: - return value + self.config.offset - -def main(): - # Create tasks and execute directly - square = SquareTask(task_id="square") - add = AddTask(square, task_id="add", offset=10) - - # Execute - result = add(5) - print(f"Result: {result}") # (5^2) + 10 = 35 - - # Or use with workspace for full tracking - workspace = Workspace.from_path("./workspace") - project = workspace.create_project("demo", name="Demo") - experiment = workspace.create_experiment( - "demo", "exp_1", "Experiment 1", "workflow.py" - ) - run = workspace.create_run("demo", "exp_1", {}, "workflow.py") - - ctx = RunContext( - asset_repo=AssetRepo(), - id=run.id, - run_metadata=run, - workspace=workspace, - ) - - with use_run_context(ctx): - result = add(5) - print(f"Result with context: {result}") - -if __name__ == "__main__": - main() +asyncio.run(main()) ``` -For advanced workflow definitions using IR models and compilation, see the [Developer Documentation](developer/ir-and-compiler.md). +## Next Steps -Congratulations! You've completed your first MolExp workflow. In the following chapters, we'll explore the design details and advanced usage of each component. +- [Task reference](../core/task.md) — full Task and Actor API +- [Workspace architecture](../workspace/) — projects, experiments, assets +- [Developer docs](../developer/) — internal compiler and IR details diff --git a/docs/index.md b/docs/index.md index f32c20ae..ce00fabf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,97 +1,119 @@ # MolExp Documentation -Welcome to MolExp! A tiny yet fully-typed task-graph framework built on top of Pydantic, designed for scientific computing workflows. MolExp provides a pure functional task abstraction, a static compiler that produces deterministic graph orders, a runtime engine, and a tiny DSL for common data-flow patterns. +Welcome to MolExp — a workflow-and-agent platform for research experiment management. MolExp provides a typed task-graph framework, a three-tier workspace hierarchy (Project → Experiment → Run), content-addressed asset storage, and a FastAPI server with React UI. ## Why MolExp? -MolExp's design philosophy is "minimal yet complete". We believe a good framework should make every component's role clear and visible, without hidden magic. Therefore, MolExp's codebase stays lean, with each layer having explicit responsibilities, allowing you to fully understand how workflows are compiled and executed. +MolExp's design philosophy is "minimal yet complete". Every layer has explicit responsibilities with no hidden magic: -MolExp is particularly suited for scientific computing scenarios that require reproducibility, traceability, and extensibility. Whether it's molecular dynamics simulations, machine learning training, or data analysis pipelines, MolExp helps you organize code, manage dependencies, and track results. +- **Workflow layer**: DAG-based task graphs compiled from Python with automatic parallelization +- **Agent layer**: Goal-driven autonomous execution on top of PydanticAI +- **Workspace layer**: File-system-backed experiment management with full reproducibility +- **Server + UI**: FastAPI backend and React frontend for browsing experiments and results -MolExp exposes a protocol interface for in-process compatibility, but persisted workflows require `Task` classes with Pydantic configurations and explicit registration. This keeps workflows deterministic and replayable. +MolExp is particularly suited for scientific computing scenarios that require reproducibility, traceability, and extensibility — molecular dynamics, ML training, data analysis pipelines, and more. -## Core Architecture +## Architecture -MolExp's architecture consists of three core layers: +``` +WorkflowSpec → Runtime → Workspace → FastAPI → React UI + ↑ + AgentService (PydanticAI) +```
-- :material-code-tags: **Task Abstraction** +- :material-graph: **Workflow Layer** --- - **Pure Functional Task Definition** - - Each task is an independent computation unit with type-safe configuration via Pydantic models. Persisted workflows use explicit registration and deterministic task IDs. - - [:octicons-arrow-right-24: Learn about Tasks](core/task.md) + **DAG-based task graphs** -- :material-compile: **Compiler Layer** + Define tasks with the functional DSL (`@wf.task`) or OOP builder. The runtime automatically parallelizes independent tasks. + + [:octicons-arrow-right-24: Quick Start](get-started/quick-start.md) + +- :material-robot: **Agent Layer** --- - **Static Graph Compilation** - - The compiler statically analyzes task graphs, generates deterministic execution orders, and detects circular dependencies. - - [:octicons-arrow-right-24: Developer Docs](developer/ir-and-compiler.md) + **Goal-driven autonomous execution** -- :material-play: **Execution Engine** + `AgentService` wraps PydanticAI to provide tool approval, streaming events, and workspace-aware session management. + +- :material-folder-multiple: **Workspace Layer** --- - **Parallel Execution & Failure Propagation** - - The engine supports parallel execution of independent tasks, automatic failure propagation, and execution hooks for monitoring. - - [:octicons-arrow-right-24: Learn about Engine](core/engine.md) + **Project → Experiment → Run hierarchy** + + Each level owns an `AssetLibrary` for scoped, content-addressed artifact storage. All metadata writes are atomic. + + [:octicons-arrow-right-24: Workspace Docs](workspace/) + +- :material-server: **Server + UI** + + --- + + **FastAPI + React** + + All routes under `/api`. The React UI provides a three-panel experiment browser. TypeScript client is auto-generated from `openapi.json`.
-## Workspace Architecture +## Core Concepts -MolExp provides a complete Project-Experiment-Run three-tier architecture to help you organize and manage scientific computing workflows: +### Tasks and Actors -
+| Type | Base class | Method | Use for | +|------|-----------|--------|---------| +| Batch | `Task` | `async execute(ctx)` | single-pass computation | +| Streaming | `Actor` | `async run(ctx)` (generator) | continuous / event-driven | -- :material-folder: **Project** +### Workflow Definition - --- +Two equivalent styles: - **Top-level container for research areas** - - A project represents a research domain or topic, containing multiple experiments. - - [:octicons-arrow-right-24: Learn about Workspace](workspace/architecture.md) +```python +# Functional DSL +wf = workflow(name="pipeline") -- :material-flask: **Experiment** +@wf.task +async def fetch(ctx: TaskContext) -> list[float]: ... - --- +@wf.task(depends_on=["fetch"]) +async def process(ctx: TaskContext) -> float: ... - **Repeatable workflow definitions** - - Experiments define repeatable workflow templates with parameter space definitions. - - [:octicons-arrow-right-24: Learn about Workspace](workspace/architecture.md) +spec = wf.build() +``` -- :material-play-circle: **Run** +```python +# OOP builder +spec = ( + WorkflowBuilder(name="pipeline") + .add(FetchTask()) + .add(ProcessTask(), depends_on=["fetch"]) + .build() +) +``` - --- +### Workspace Hierarchy - **Single execution instance** - - A run is a concrete execution of an experiment, containing complete reproducibility information. - - [:octicons-arrow-right-24: Learn about Workspace](workspace/architecture.md) +```python +workspace = Workspace.from_path("./lab") +project = workspace.create_project(name="MD Simulations") +experiment = project.create_experiment(name="Temperature Sweep") +run = experiment.create_run(parameters={"T": 300}) -
+result = await spec.execute(run=run) +``` ## Quick Start -If you're new to MolExp, we recommend starting with the quick start guide, which will walk you through creating and executing your first workflow. +New to MolExp? Start here: -[:octicons-arrow-right-24: Get Started](get-started/quick-start.md) +[:octicons-arrow-right-24: Quick Start Guide](get-started/quick-start.md) ## License -MIT License - see [LICENSE](https://github.com/molcrafts/molexp/blob/main/LICENSE) for details. +BSD 3-Clause License — see [LICENSE](https://github.com/molcrafts/molexp/blob/main/LICENSE) for details. diff --git a/docs/spec/fullscreen-monitor.md b/docs/spec/fullscreen-monitor.md new file mode 100644 index 00000000..874ab95b --- /dev/null +++ b/docs/spec/fullscreen-monitor.md @@ -0,0 +1,257 @@ +# Spec: Full-screen Run Monitor for Molexp, Powered by Molq Panels + +## Goal + +为 `molexp` 设计一个全屏终端 monitor,用于观察本地或远程任务的运行状态。 + +该 monitor 的体验目标类似 `htop` / `btop`:不是简单打印一张表,而是一个**占满终端的 dashboard**,包含顶部 overview、进度展示、底部任务列表,以及基本的交互关闭/重开能力。 + +该设计明确以下边界: + +- 界面如何组织 +- `molq` 插件负责什么 +- `molexp` 负责什么 +- 两者如何协作 +- 哪些是必须满足的交互需求 + +本 spec 只定义**产品与架构需求**,不规定具体代码结构、类名或实现方式。 + +--- + +## Design Principles + +### 1. Full-screen monitor, not inline table + +当前的小 panel + 单个表格形式不可接受。monitor 必须是一个完整终端视图,视觉和交互都应接近系统监控工具,而不是 CLI 的一次性文本输出。 + +### 2. Overview-first, list-second + +最重要的信息不是单行任务表,而是整体状态。界面必须优先展示: + +- 当前是否在运行 +- 总体进度 +- running / pending / done / failed 的总体数量 +- 当前 workflow / run 的整体状态 + +任务列表作为下半部分内容,而不是整个界面的核心。 + +### 3. Single-job and multi-job must both look natural + +即使只有 1 个 job,也不应退化成一张尴尬的小表。同一套 full-screen monitor 应同时适用于单任务、多任务 sweep、本地运行、远程调度运行。 + +### 4. Status-first, table-second + +状态必须是界面中的第一视觉锚点。用户应一眼看到 running / pending / done / failed,而不是先看 run id 或 scheduler id。 + +### 5. Monitor is a viewer, not the run itself + +关闭 monitor 不应终止运行。monitor 是观察界面,不是任务本体。 + +--- + +## UI Requirements + +### Overall Layout + +monitor 占用整个 terminal,包含以下三个区域: + +``` +┌─────────────────────────────────────────────────────────┐ +│ A. Header / Title Area │ +│ workflow name · overall status · last updated │ +├───────────────────────────────┬─────────────────────────┤ +│ B. Overview Area │ │ +│ total · running · pending │ progress bar / vis │ +│ done · failed │ │ +├───────────────────────────────┴─────────────────────────┤ +│ C. Job List Area │ +│ state · run identity · scheduler id · elapsed ... │ +│ ... │ +├─────────────────────────────────────────────────────────┤ +│ D. Footer / Hint Area q: quit r: refresh ... │ +└─────────────────────────────────────────────────────────┘ +``` + +### A. Header / Title Area + +展示 monitor 的身份信息和当前整体状态。至少包含: + +- 当前 workflow / run / experiment 的标识 +- 当前整体状态(running / finished / failed / mixed) +- 最近更新时间或刷新时间 + +### B. Overview Area + +界面的核心,位于上半部分。至少展示: + +- 总任务数 +- running / pending / done / failed 数量 +- 整体进度(必须有明确可读的 progress visualization,例如进度条) + +进度不能只是 `0/1 done` 这种弱表达,必须有更强的 overview 感知。 + +### C. Job List Area + +位于下半部分,展示每个 job / run 的状态。每条目至少包含: + +- state +- run identity +- scheduler id(如有) +- 其他次要元信息(elapsed、node、message 等由 agent 自行判断) + +列表服务于"快速扫状态",而不是做数据库式大表格。 + +### D. Footer / Hint Area + +界面底部保留简洁的交互提示区域。至少提示: + +- `q` 关闭 monitor +- 其他交互提示由 agent 自行设计 + +--- + +## Visual Requirements + +### 1. Avoid heavy boxed table aesthetics + +不接受"一个 panel 里套一张 box-heavy 表格"的风格。减少无意义边框,强调分区、状态和进度。 + +### 2. Strong visual status hierarchy + +状态必须有明显视觉区分。颜色、图标、位置和文本都应服务于快速识别。 + +### 3. Scheduler ID is secondary + +`scheduler id` 不是第一重要信息,不应压过状态和 run identity。 + +### 4. Progress must be visually central + +overview 中必须有明确的 progress visualization。这不是可选增强,而是 monitor 的核心组成。 + +--- + +## Interaction Requirements + +### 1. `q` closes the monitor + +用户按 `q` 时,关闭 full-screen monitor,返回普通 CLI。 + +### 2. Closing the monitor must not cancel jobs + +`q` 的语义只能是 close viewer / quit monitor / return to normal CLI。绝不能终止远程任务、本地 workflow 或 scheduler jobs。 + +### 3. Monitor must be reopenable + +关闭后,用户必须能通过显式命令再次打开 monitor。命名不强制,但 monitor 不是一次性的,必须可重复进入。 + +### 4. Run command and watch command should cooperate naturally + +如果 `run` 命令在提交后自动进入 monitor,退出 monitor 后应能回到普通 CLI,并给出明确提示: + +- 任务仍在继续 +- 用户可以再次打开 monitor + +--- + +## Molexp vs Molq Responsibilities + +### Molexp responsibilities + +`molexp` 是 workflow / run orchestration 层,负责: + +- 决定何时进入 monitor,何时退出 monitor +- 管理 monitor 的打开 / 关闭 / 重新打开 +- 管理 run / workflow / experiment 的生命周期 +- 收集和整理运行状态数据 +- 将运行状态转换成 monitor 所需的通用状态输入 +- 在 CLI 流程中决定 monitor 与普通输出之间如何切换 + +**`molexp` 拥有 monitor 的生命周期控制权。** + +### Molq responsibilities + +`molq` 插件负责 monitor 的**界面能力**,而不是 workflow 生命周期。它负责: + +- 提供 full-screen panel / dashboard 能力 +- 提供 overview 区域和 job list 区域的界面表达 +- 提供 monitor 的交互语义解释 +- 提供可复用的 terminal viewer 体验 + +`molq` 不主导任务生命周期,不决定 run 是否结束、是否继续提交等 workflow 语义。 + +### Interaction contract between Molexp and Molq + +| 职责 | 归属 | +|------|------| +| monitor 何时创建、关闭、重新进入 | molexp | +| monitor 如何呈现、基本交互 UI 语义 | molq | +| workflow 是否结束、是否继续提交 | molexp | +| Rich 布局细节、panel 组件 | molq | +| UI 交互结果转化为 CLI/生命周期动作 | molexp | + +规则: + +1. **Molexp owns control flow** — `molexp` 控制 monitor 生命周期 +2. **Molq owns presentation semantics** — `molq` 提供 monitor 如何呈现 +3. **Molq must not own workflow lifecycle** — `molq` 不自己决定"停止任务""终止 workflow" +4. **Molexp must not reimplement panel logic** — `molexp` 不应自己硬写所有 Rich 布局,否则 `molq` 作为 panel/plugin 层失去意义 +5. **UI actions must be interpretable by Molexp** — monitor 中的交互结果应能被 `molexp` 接住并转化为 CLI/生命周期动作 + +--- + +## Architectural Constraints + +1. **No giant monolithic CLI table renderer** — 不接受把所有逻辑直接写成单个 CLI 输出函数 +2. **No inversion of control from Molq into Molexp lifecycle** — `molq` 不能反过来控制 `molexp` 的主流程 +3. **No scheduler-specific UI hardcoding** — UI 不应被 slurm/pbs/lsf 某一种调度器结构绑死;monitor 应围绕 run/job 状态设计 +4. **UI design should scale from local to remote** — 同一套 monitor 应同时服务于 local、remote、scheduler-backed runs + +--- + +## Agent Tasks + +agent 需要基于当前 codebase 设计并实现: + +1. 一套 full-screen terminal monitor 的信息架构 +2. 单任务和多任务都自然的界面组织方式 +3. `molexp` 与 `molq` 之间的职责边界与接口契约 +4. monitor 打开、关闭、重新打开的交互流程 +5. 能够支撑 future remote backends 的状态模型 + +agent 自行决定: + +- 具体布局组件 +- 具体 Rich 组织方式(layout / panel / live / screen 等) +- 具体交互键位集合 +- 具体 state object / panel object / plugin object 设计 + +--- + +## Acceptance Criteria + +| 要求 | 类别 | +|------|------| +| monitor 是 full-screen dashboard,而不是小 panel | UI | +| 上半部分有清晰 overview 和进度展示 | UI | +| 下半部分有 job list | UI | +| 状态比表格列更重要,视觉上优先 | Visual | +| 单任务场景也自然,不退化 | UI | +| `q` 只关闭 monitor,不取消运行 | Interaction | +| monitor 可重新打开 | Interaction | +| `molq` 负责 panel/view 能力 | Architecture | +| `molexp` 负责 monitor 生命周期与 CLI 切换 | Architecture | +| 两者职责清晰,不互相吞没 | Architecture | + +--- + +## Non-goals + +本 spec 不要求: + +- 具体类设计或函数签名 +- 具体键盘事件实现方式 +- 具体 scheduler adapter 代码 +- 具体 Rich API 写法 +- 完整 TUI 框架抽象 + +这些由 agent 根据 codebase 自行判断和设计。 diff --git a/docs/spec/molcfg-profiles.md b/docs/spec/molcfg-profiles.md new file mode 100644 index 00000000..2e9b10e1 --- /dev/null +++ b/docs/spec/molcfg-profiles.md @@ -0,0 +1,213 @@ +# molcfg + Profiles: 配置文件与 Profile 机制 + +**状态**: Draft · **作者**: @RoyKid · **日期**: 2026-04-13 + +## 1. 动机 + +当前 `--dry-run` 是硬编码的一等执行模式,扎根于 CLI、RunStatus、RunMetadata、 +ExecutionConfig、ctx、UI badge 等多层。进一步的需求(`--profile smoke`、 +`--dataset-md17`、速度测试、debug 切片)会让 CLI 快速膨胀为 CMake 式的 +flag 森林。 + +**目标**: 用单一机制——带命名 profile 的配置文件——覆盖所有参数变体 +(dataset、epochs、batch_size、dry-run 是否跳过重计算、...)。 + +**Profile 语义对框架透明**: 框架只负责"加载配置 / 选中切片 / 注入 ctx / +记录 profile 名到 run metadata"。切片里写什么字段、task 如何解读它们 +(包括"是否产生持久化副作用"),完全由用户自己定义。框架不内置任何 +语义字段(没有 `side_effects`、没有 `real`、没有 `persist`)。 + +**非目标**: 向后兼容 `ctx.dry_run` / `RunStatus.DRY_RUN` / `--dry-run` 作为 +独立字段。全部替换,无 deprecation 期。 + +## 2. 设计原则 + +1. **CLI 稳定**: `molexp run script.py [--config X.yaml] [--profile NAME]` 是 + 唯一入口;新增变体只改 config,不改 CLI。 +2. **Profile 是命名的配置切片**: profile 名本身就是 UI 展示标签、run + metadata 字段、resume 过滤条件。 +3. **框架对 profile 内容不可知**: 任何字段都是用户数据。Task 代码通过 + `ctx.config["epochs"]` 等自由读取。框架不解释。 +4. **Profile 名归一化**: `-` 自动替换为 `_`(YAML 里 `dry-run` 等价于 + `dry_run`),避免 Python 标识符场景和 CLI 下划线/连字符混淆。 +5. **格式**: YAML(默认)或 JSON。不支持 TOML(stdlib 只读)。 +6. **Profile 可继承**: 显式 `extends: NAME`,不用 YAML anchor(JSON 写不出、 + 魔法多)。 + +## 3. Schema 草案 + +### 3.1 配置文件 + +```yaml +# molcfg.yaml +version: 1 + +# 默认值(所有 profile 继承) +defaults: + dataset: md17 + epochs: 100 + batch_size: 32 + seed_base: 42 + +profiles: + dry-run: # 加载时归一化为 "dry_run" + extends: defaults + epochs: 1 + + smoke: + extends: defaults + epochs: 5 + batch_size: 8 + + md22: + extends: defaults + dataset: md22 + + prod: + extends: defaults + # 显式空,等同 defaults +``` + +### 3.2 Pydantic 模型 + +```python +# src/molexp/config/models.py +class ProfileConfig(Mapping[str, Any]): + """Immutable merged config for one profile. + + Behaves like a read-only dict of user data; carries a ``name`` attr + (normalized, "-" → "_"; ``None`` means "no profile, defaults only"). + Framework adds no semantic fields — every key comes from the YAML. + """ + name: str | None + # internal: frozen dict + + def __getitem__(self, key: str) -> Any: ... + def get(self, key: str, default=None) -> Any: ... + def hash(self) -> str: ... # content-hash for RunMetadata.config_hash + +class MolCfg(BaseModel, frozen=True): + version: int = 1 + defaults: dict[str, Any] = Field(default_factory=dict) + profiles: dict[str, dict[str, Any]] = Field(default_factory=dict) + + def resolve(self, name: str | None) -> ProfileConfig: + """Merge defaults + (optional) profile; normalize name; freeze.""" +``` + +### 3.3 Context 注入 + +```python +class StepContext(Generic[StateT, DepsT, InputT]): + config: ProfileConfig # 新增 + # 移除 dry_run 属性 +``` + +Task 作者的 migration(任何"跳过副作用"的判断都变成用户自己的 config 字段): +```python +# 旧 +if ctx.dry_run: + return mock_result() + +# 新 —— 用户自己在 YAML 里加字段,task 里自己读 +# molcfg.yaml: +# profiles: +# dry-run: +# skip_heavy_compute: true +if ctx.config.get("skip_heavy_compute"): + return mock_result() + +# 或纯参数变体 +if ctx.config["epochs"] < 5: + ... + +# 也可以按 profile 名分支(不推荐,耦合命名) +if ctx.config.name == "dry_run": + ... +``` + +### 3.4 RunMetadata + +```python +class RunMetadata(BaseModel): + id: str + status: str = "pending" # pending/running/succeeded/failed/cancelled + profile: str | None = None # 新增:activated profile name, e.g. "dry-run" + config_hash: str | None = None # 新增:content-hash of resolved ProfileConfig + # 删除: dry_run: bool + ... +``` + +### 3.5 RunStatus + +```python +class RunStatus(str, Enum): + PENDING = "pending" + RUNNING = "running" + SUCCEEDED = "succeeded" + FAILED = "failed" + CANCELLED = "cancelled" + # 删除: DRY_RUN +``` + +Profile 与状态正交。Profile 为 `dry-run` 的 run 跑完了照样是 `SUCCEEDED`。 + +### 3.6 CLI + +``` +molexp run script.py [--config PATH] [--profile NAME] [--local|--slurm|...] + [--resume] + +--config PATH 默认 ./molcfg.yaml,找不到则 profile 必须为空或报错 +--profile NAME 激活的 profile 名,默认无(= 用 defaults) +--resume 重跑 profile 匹配、状态非 succeeded 的 run +``` + +**删除**: `--dry-run` flag。用 `--profile dry-run` 代替。 + +### 3.7 UI + +- Run 列表:紧邻 status 的位置显示 `[{profile}]` 徽章(若 profile 非空) +- 颜色映射可配置,但默认:`dry-run`=yellow, `smoke`=cyan, 其它=dim +- 不再有 "dry_run" 状态徽章 + +### 3.8 Resume 语义 + +新规则:`--resume --profile X` 找所有 `profile == X and status != "succeeded"` +的 run,重新执行。旧的"只 resume dry-run 状态"逻辑泛化为"resume +任意 profile"。 + +## 4. Roadmap + +**Phase 1** (本次): 核心替换 +- 引入 `molexp.config` 模块(MolCfg/ProfileConfig/加载器) +- 替换 ExecutionConfig → ProfileConfig +- 删除 RunStatus.DRY_RUN、RunMetadata.dry_run +- CLI 改用 --config/--profile +- Server / schemas 同步 +- 测试全绿 + +**Phase 2** (后续 PR): UI +- 新 profile 徽章组件 +- 徽章颜色配置项 + +**Phase 3** (后续): 文档 +- quick-start 改写 +- README 更新 + +## 5. Todo List + +见 TaskList;核心路径: + +1. 建 `src/molexp/config/` 模块:models + loader + resolver +2. 改 `workspace/models.py`:ExecutionConfig → ProfileConfig,RunMetadata 字段 +3. 改 `workspace/run.py`:删 DRY_RUN 枚举,ctx 暴露 config 而非 dry_run +4. 改 `workflow/context.py`:替换 dry_run property → config property +5. 改 `workflow/_pydantic_graph/state.py + node.py + runtime.py + compiler.py` +6. 改 `workflow/spec.py + runtime.py`:execute() 接受 profile +7. 改 `cli/__init__.py`:--config/--profile,删 --dry-run 及其分支 +8. 改 `server/routes/run.py`:状态列表更新 +9. 改 `monitor.py`:徽章逻辑 +10. 迁移现有测试(tests/) +11. 添加新测试:config 加载、profile 解析、继承 +12. 运行完整 pytest 确认绿 diff --git a/docs/spec/unified-pydantic-graph-dispatch.md b/docs/spec/unified-pydantic-graph-dispatch.md new file mode 100644 index 00000000..f29b5048 --- /dev/null +++ b/docs/spec/unified-pydantic-graph-dispatch.md @@ -0,0 +1,423 @@ +# 统一 pydantic-graph 调度: Sweep 与 Backend 并入外层图 + +**状态**: Draft · **作者**: @RoyKid · **日期**: 2026-04-17 + +## 实施状态 + +| Phase | 状态 | 落地日期 | 备注 | +|-------|------|---------|------| +| 1 — Sweep pydantic-graph 化(仅 `--local` + `-j N`) | ✅ 已完成 | 2026-04-17 | 新增 `molexp.sweep` 包、`-j/--jobs` CLI flag、profile `jobs:` 字段约定;`molexp run --local -j N` 启用 sweep 级并发 | +| 2 — Backend 统一入口(`--backend`, `--block`) | ⏳ 待开始 | — | 需要先让 `molq.JobHandle` 支持 `wait_async` 或包 `asyncio.to_thread` | +| 3 — Per-node backend(`@wf.task(backend=..., resources=...)`) | ⏳ 待开始 | — | 依赖 Phase 2 | +| 4 — molq 原生 async API | ⏳ 独立推进 | — | 在 molq 仓完成 `monitor.py` async 化 | + +**Phase 1 变更清单**: +- `src/molexp/sweep/__init__.py`、`src/molexp/sweep/graph.py`(`SweepReplica`, `SweepState`, `SweepRoot`, `run_sweep`) +- `src/molexp/cli/run_cmd.py` 拆分 `_execute_sweep`:`_discover_runs` + `_dispatch_local`(pydantic-graph)+ `_dispatch_cluster`(保留旧路径);新增 `-j/--jobs` CLI 参数和 `_resolve_jobs` 辅助 +- `tests/test_sweep/test_graph.py`(9 个单测)+ `tests/test_cli_run_jobs.py`(6 个 E2E 测试) + +**当前 profile 约定**:`jobs: ` 是用户数据(schemaless,见 §2 及 `docs/spec/molcfg-profiles.md`)。CLI `-j` 优先级高于 profile,profile 值 > 1 触发并行,默认 1(向后兼容)。 + +## 1. 动机 + +当前 `molexp run` 存在两层完全独立的调度: + +| 层次 | 位置 | 并行模型 | 快照/恢复 | +|------|------|----------|-----------| +| **Sweep**(跨 experiment × replica) | `cli/run_cmd.py:_execute_sweep` 的 `for` 循环 | `--local` 强串行;`--slurm` 交给外部调度器 | 仅靠 run-id 去 store 里匹配,无 in-process 状态 | +| **Workflow**(task DAG) | `workflow/_pydantic_graph/` | 同 level 内 `asyncio.gather` | 有 pydantic-graph snapshot | + +这带来四个问题: + +1. **`--local` 不能并行**: 明明多个 experiment 互相无依赖,`for` 循环 + 阻塞 + `asyncio.run()` 强制串行(见 `cli/run_cmd.py:206-209, 467-472`)。 +2. **两套 dispatch 路径**: `_local_handler` 和 `SubmitHandler` 是两套代码,同样的 + "把一个 run 跑起来"有两个完全不同的分支(`run_cmd.py:452-482` vs + `run_cmd.py:484-543`)。 +3. **`--slurm` fire-and-forget 不可观测**: 父进程提交完立即退出,sweep 层没有 + graph 状态,`molexp watch` 必须单独实现数据库轮询逻辑来反推进度。 +4. **无法 per-node 选后端**: 想让 `prepare_data` 跑在本地、`train_model` 上 GPU + 集群是常见需求,当前架构不支持——`--slurm` 是整个脚本层级的开关。 + +**目标**: 把"sweep"也建模成一个 pydantic-graph,让 local / molq / 未来的其他 +backend 都成为**某个 node 内部的 await 操作**,而不是顶层的两种 dispatch。 + +**非目标**: +- 不改 molq 内部模型(job store / reconciler / monitor 保持不变)。 +- 不改内层 workflow 的 task 协议(`@wf.task` 语义保持)。 +- 不为本地引入子进程池——local 就是 in-process async 并发。GPU 训练的真并行 + 交给 molq 的 local scheduler 或远程 scheduler。 + +## 2. 设计原则 + +1. **pydantic-graph 是唯一编排器**: 所有执行语义——串/并、本地/远程、 + snapshot/resume——都通过 graph 节点 + async body 表达。不存在 graph 之外的 + 调度循环。 +2. **Backend 是 node body 的实现细节**: `@wf.task` 不关心自己跑在哪;跑哪儿由 + decorator 参数、profile 配置和 CLI 覆盖在运行时决定。 +3. **默认零改动**: 现有脚本(如 `examples/train_allegro_qm9.py`)**不需要任何 + 源码修改**即可在新架构下运行。所有新能力通过 profile / decorator 参数开启。 +4. **Block 语义由用户控制**: `block=True` 节点 `await` 到作业终止,父进程必须 + 存活;`block=False` 节点返回 `job_id` 立刻完成,后续状态交给 molq store。 +5. **Account/resources 三级继承**: profile 兜底 → decorator 默认 → CLI/ctx + 覆盖。用户只在"特殊"节点显式覆盖。 +6. **旧 CLI flag 作为别名保留**: `--local` / `--slurm` / `--pbs` / `--lsf` 映射 + 到新的 `--backend` + profile,不做 deprecation 期强制迁移。 + +## 3. 核心架构 + +### 3.1 两层 Graph + +``` +SweepGraph(外层) + 节点: 每个 (experiment × replica) + 边: 无(所有 replica 互相独立) + 并发: asyncio.Semaphore(jobs) 节流同 level 并发 + Body: await _run_one_replica(run, exp, profile_cfg) + + ├─ backend=local: + │ inner = exp.workflow.execute(run_context=ctx) + │ await inner # 进程内直跑 + │ + └─ backend=slurm/pbs/lsf: + handle = submitor.submit( + argv=[python, -m, worker, script, run_dir, --task ?], + resources=..., execution=...) + if block: + record = await handle.wait_async() # 父进程守候 + return {"state": record.state.name, "job_id": handle.job_id} + else: + return {"job_id": handle.job_id} # fire-and-forget + +WorkflowGraph(内层,保持现状) + 节点: @wf.task + 边: depends_on + Body: 保持现有 async def 协议 +``` + +### 3.2 Per-node Backend(进阶路径) + +默认"一个 experiment 一个远程 job"(整个内层 workflow 打包提交)。当用户希望 +混合执行(e.g. `prepare_data` 本地、`train_model` 上 GPU),在内层某个 `@wf.task` +上标注 `backend=`,框架把这个 task 作为独立 molq job 提交: + +```python +@wf.task(depends_on=["prepare_data"], + backend="slurm", + resources=TaskResources(gpus=1, time="12h", mem="64G")) +async def train_model(ctx): ... +``` + +远端用同一份 `worker.py` 入口,加 `--task ` 参数来选择"只跑一个 node" +而不是整个 workflow(见 §5.3)。 + +### 3.3 并发度控制 + +``` +molexp run script.py -j 4 # sweep 级 4 并发 +molexp run script.py -j 4 --backend slurm # 4 路同时提交 slurm + await +molexp run script.py # -j 默认 1(= 当前行为) +``` + +`-j N` 的实现就是外层 `SweepGraph` 节点执行器里的 +`asyncio.Semaphore(N)`——直接复用 `workflow/_pydantic_graph/node.py:88-124` 的 +`_execute_parallel` 模式。 + +## 4. 用户 API + +### 4.1 CLI + +```bash +# 推荐新写法 +molexp run script.py --backend {local,slurm,pbs,lsf,...} \ + [-j N] [--block/--no-block] \ + [-c molcfg.yaml] [--profile NAME] + +# 旧 flag 保留为别名(zero-cost 迁移) +molexp run script.py --local # ≡ --backend local -j 1 +molexp run script.py --slurm # ≡ --backend slurm --no-block (保持 fire-and-forget 行为) +``` + +**默认值**: +- `--backend`: `local` +- `-j`: `1`(向后兼容,显式设 `-j auto` = CPU 核数) +- `--block`: `local` 下无意义(本就是 await);`slurm` 下默认 `--no-block` + 匹配旧行为。未来版本可切换成 `--block` 默认,按社区反馈。 + +### 4.2 Profile(molcfg.yaml) + +```yaml +default: &base + smoke: false + +profiles: + local_quick: + <<: *base + backend: local + jobs: 4 # -j 等价写法 + + prod_gpu: + <<: *base + backend: slurm + block: true # 父进程守候 + jobs: 8 # 同时提交/守候 8 个 job + slurm: + cluster: "perlmutter" + account: "mycompchem" + qos: "regular" + partition: "gpu" + + mixed: + <<: *base + backend: local # 外层 sweep 本地并发 + jobs: 16 + per_node_backend: true # 允许内层 task 上的 backend= 生效 + slurm: # 内层 task 用到时继承 + cluster: "perlmutter" + account: "mycompchem" +``` + +### 4.3 Decorator(脚本内) + +```python +from molexp import TaskResources + +@wf.task( + depends_on=["prepare_data"], + backend="slurm", # per-node 覆盖 + resources=TaskResources(gpus=1, time="12h", mem="64G"), +) +async def train_model(ctx): ... +``` + +**决策链**: CLI 覆盖 > profile > decorator > 框架默认。`account` / `cluster` / +`qos` 这类机构级字段**永远**从 profile 取,不进脚本(便于跨机器复用)。 + +### 4.4 TaskContext 扩展 + +```python +@wf.task +async def some_task(ctx: TaskContext): + # 新增: 手动提交(高阶用户) + handle = await ctx.molq.submit( + argv=[...], resources=..., execution=...) + record = await handle.wait_async() + ... +``` + +`ctx.molq` 在非 molq 后端下可用(fallback 到 `local` scheduler),使用户代码不 +依赖 CLI 选的 backend。 + +## 5. 内部实现 + +### 5.1 SweepGraph + +新文件 `molexp/sweep/graph.py`: + +```python +from pydantic_graph import BaseNode, End, GraphRunContext +import asyncio + +@dataclass +class ReplicaNode(BaseNode[SweepState, SweepDeps, SweepResult]): + mol_run: Run + experiment: Experiment + + async def run(self, ctx) -> "ReplicaNode | End[SweepResult]": + async with ctx.deps.semaphore: # -j N 节流 + backend = _resolve_backend( + ctx.deps.profile_cfg, + self.experiment) + if backend == "local": + await self.experiment.workflow.execute( + run=self.mol_run, + profile_config=ctx.deps.profile_cfg) + else: + await _submit_and_maybe_wait( + backend=backend, + mol_run=self.mol_run, + profile_cfg=ctx.deps.profile_cfg) + # ReplicaNode 是叶子,直接 End + return End(...) + +def build_sweep_graph(workspaces, profile_cfg, jobs: int) -> Graph: + # 汇总所有 (project, experiment, replica),构建单级并行图 + ... +``` + +### 5.2 `_execute_sweep` 重构 + +`cli/run_cmd.py` 现有的 94 行 `_execute_sweep` 大幅简化: + +```python +async def _execute_sweep_async(script, profile_cfg, resume, workspace, jobs): + workspaces = load_workspaces(script) + runs = _discover_runs(workspaces, profile_cfg, resume) # 保留当前 resume 逻辑 + graph = build_sweep_graph(runs, profile_cfg, jobs=jobs) + await graph.run(...) + +def _execute_sweep(...): + asyncio.run(_execute_sweep_async(...)) +``` + +`_local_handler` 和 `SubmitHandler` 被删除(其逻辑迁移进 `ReplicaNode.run`)。 + +### 5.3 `worker.py` 扩展 + +`molexp/plugins/submit_molq/worker.py` 增加 `--task ` 选项: + +```python +# 旧: python -m worker