Skip to content

operations.md

Latest commit

 

History

History
268 lines (195 loc) · 12.4 KB

operations.md

File metadata and controls

268 lines (195 loc) · 12.4 KB
Raw

Operations — day-2 runbook

After onboarding is done, these are the things you'll actually use day-to-day.

Quick reference

Task PowerShell Bash
Check on-chain status .\scripts\powershell\status.ps1 ./scripts/bash/status.sh
Tail all logs docker logs -f lightchain-worker same
Tail just job events .\scripts\powershell\tail-jobs.ps1 ./scripts/bash/tail-jobs.sh
Stop (registration stays) .\scripts\powershell\stop.ps1 ./scripts/bash/stop.sh
Restart .\scripts\powershell\08-run-worker.ps1 -NoTail ./scripts/bash/08-run-worker.sh --no-tail
Sweep rewards to cold wallet .\scripts\powershell\sweep-rewards.ps1 -To 0x... ./scripts/bash/sweep-rewards.sh 0x...
Deregister + unlock stake .\scripts\powershell\deregister.ps1 ./scripts/bash/deregister.sh
Show worker wallet balance cast balance $WORKER_ADDR --rpc-url $RPC_URL same

Monitoring

Per-job lifecycle

A successful job log block looks like this (timestamps and IDs vary):

INFO ws_job_received       jobId=543
INFO processing job        jobID=543 model="f4a414fa..." sessionID=336
INFO stage 1: sending ack tx           jobID=543 stage=ack         timeout=15s
INFO stage 1 complete                  jobID=543 stage=ack         durationMs=6269
INFO stage 2 starting                  jobID=543 stage=fetch_blob  promptBlobHash=0x013e69...
INFO stage 2 complete                  jobID=543 stage=fetch_blob  blobBytes=2224 durationMs=293
INFO stage 3: session key cache miss, deriving from chain   jobID=543 stage=session_key
INFO stage 3 complete                  jobID=543 stage=session_key durationMs=22
INFO stage 4 complete                  jobID=543 stage=decrypt     promptBytes=2196 durationMs=0
INFO stage 5 complete                  jobID=543 stage=resolve_model
INFO stage 6 complete                  jobID=543 stage=inference   tokens=512 durationMs=11400
INFO stage 7 complete                  jobID=543 stage=publish
INFO stage 8 complete                  jobID=543 stage=submit_blob
INFO job completed                     jobID=543 payoutWei=...

Tail filtered down to just these lines:

.\scripts\powershell\tail-jobs.ps1
./scripts/bash/tail-jobs.sh

Metrics

The sidecar exposes Prometheus metrics on 127.0.0.1:9101 inside the container. Scrape from your host with:

docker exec lightchain-worker wget -qO- http://127.0.0.1:9101/metrics
docker exec lightchain-worker wget -qO- http://127.0.0.1:9101/metrics

The metrics worth watching (see architecture.md § Metrics endpoint for the full list):

Metric What "healthy" looks like
worker_active_jobs 0 when idle, > 0 during inference
worker_release_pending 0 most of the time; brief spikes are normal
worker_release_released_total Monotonically increasing as jobs settle
worker_release_reconcile_last_block Within ~1 hour of current head
worker_redis_publish_failures_total 0 ideally, low non-zero is non-fatal
worker_stuck_nonce_tracked 0 (anything else means a tx is wedged in the mempool)
worker_subpool_orphans_total 0

Quirks — see troubleshooting.md:

  • worker_ollama_up often stays at 0 on idle workers — appears to be a lazy/probe-driven metric, not an actual health signal. Doesn't block job routing in practice.
  • worker_heartbeat_last_emit_timestamp_seconds stays at 0 unless Redis is configured (which the Lightchain docs don't currently mention).

External dashboards

URL What it shows
https://workers.lightchain.ai/worker/<YOUR_WORKER_ADDR> Public profile of your worker (status, total jobs, payout)
https://status.mainnet.lightchain.ai Mainnet health (RPC, Dispatcher, Relay)
https://lightscan.app/address/<YOUR_WORKER_ADDR> Mainnet block explorer view of your wallet
https://testnet.lightscan.app/address/<YOUR_WORKER_ADDR> Testnet equivalent

Stopping and restarting

Stop without losing registration

.\scripts\powershell\stop.ps1
./scripts/bash/stop.sh

This docker stops and docker rms the container. The keystore + ECDH key stay on disk under ~/lightchain-worker/keys, and your registration + stake stay on-chain. The worker will appear "offline" on the public dashboard.

Restart

.\scripts\powershell\08-run-worker.ps1 -NoTail
./scripts/bash/08-run-worker.sh --no-tail

Survive reboots automatically

The toolkit starts the container with --restart always, so as long as Docker Desktop / Docker Engine is running, the container will too. To make Docker itself auto-start at login:

  • Windows: Docker Desktop → Settings → General → ✓ "Start Docker Desktop when you sign in"
  • macOS: Docker Desktop → Settings → General → ✓ "Start Docker Desktop when you log in"
  • Linux (systemd): sudo systemctl enable docker — already enabled by the official install

For a fully unattended setup on Linux, see examples/systemd/lightchain-worker.service.

Sweeping rewards

Rewards land directly in the worker wallet's balance (which is the same address holding your stake). Nothing is auto-forwarded — you're responsible for moving the LCAI to a cold wallet on the cadence that matches your risk tolerance.

# Sweep everything except 1 LCAI gas buffer
.\scripts\powershell\sweep-rewards.ps1 -To 0xYourColdWallet

# Or keep a bigger buffer
.\scripts\powershell\sweep-rewards.ps1 -To 0xYourColdWallet -GasBuffer 2
./scripts/bash/sweep-rewards.sh 0xYourColdWallet
./scripts/bash/sweep-rewards.sh 0xYourColdWallet 2

The script:

  1. Reads your worker wallet balance.
  2. Subtracts the gas buffer (default 1 LCAI) to compute the sweep amount.
  3. Aborts if the result would be zero or negative.
  4. Asks you to type sweep to confirm (skip with FORCE=1).
  5. Sends the sweep via cast send.

Important: the stake is not part of your wallet balance — it's locked inside WorkerRegistry. So sweeping doesn't touch the stake. Don't worry about accidentally unstaking.

Scheduling sweeps

If you want to automate this, use the OS's scheduler. Example: a Linux cron that sweeps weekly on Sundays at 03:00:

crontab -e
0 3 * * 0 cd /home/you/lightchain-worker-toolkit && FORCE=1 ./scripts/bash/sweep-rewards.sh 0xYourColdWallet >> /var/log/lightchain-sweep.log 2>&1

The Windows equivalent is Task Scheduler → Create Basic Task → Weekly.

Deregistering and recovering your stake

When you want out:

.\scripts\powershell\deregister.ps1
# Type 'deregister' to confirm
./scripts/bash/deregister.sh
# Type 'deregister' to confirm

This calls the worker CLI's deregister subcommand, which executes WorkerRegistry.deregister() on-chain. The contract:

  1. Verifies you have no in-flight jobs.
  2. Releases your stake (50,000 LCAI minus any slashing penalties) back to the worker wallet.
  3. Removes your worker from the registry.

The sidecar container keeps running until you stop it. After deregister, do:

.\scripts\powershell\stop.ps1
.\scripts\powershell\sweep-rewards.ps1 -To 0xYourFinalWallet -GasBuffer 0
./scripts/bash/stop.sh
./scripts/bash/sweep-rewards.sh 0xYourFinalWallet 0

(GasBuffer 0 is OK here because you're done — no more gas needed.)

Switching networks (mainnet ↔ testnet)

Edit scripts/powershell/env.ps1 (or scripts/bash/env.sh) and change:

$env:NETWORK = "testnet"   # was "mainnet"
export NETWORK="testnet"

Then re-run from Phase 01:

.\scripts\powershell\01-resolve-addresses.ps1   # gets the new network's contract proxies
# Phase 02 is network-agnostic (Ollama only)
.\scripts\powershell\03-pull-image.ps1          # different image for testnet
# Phase 04 is fine - same keystore works on both
# Phase 05 is fine - same ECDH key works on both
# Phase 06 - use testnet faucet to fund (free)
.\scripts\powershell\07-register.ps1            # new on-chain registration
.\scripts\powershell\08-run-worker.ps1 -NoTail

Note: the same worker private key can be registered on both networks since they have different chain IDs and separate contract state. But the LCAI on each network is separate too — testnet LCAI from the faucet has zero value on mainnet.

Upgrading the worker image

Lightchain occasionally pushes new versions of the worker image. First, check whether there's actually a new image to pull (so you don't restart the container for no reason):

# Registry's current :latest digest (use the mainnet URL if you're on mainnet)
curl -sSI "https://us-central1-docker.pkg.dev/v2/lightchain/lightchain-testnet-public-docker/worker/manifests/latest" \
  -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  | awk 'tolower($1)=="docker-content-digest:" {print $2}' | tr -d '\r'

# Your running container's image digest
docker image inspect "$(docker inspect lightchain-worker --format '{{.Image}}')" \
  --format '{{range .RepoDigests}}{{.}}{{"\n"}}{{end}}' | head -1

If the digests match, you're already current — nothing to do. If they differ, upgrade:

$env:NETWORK = "testnet"    # or "mainnet" — required if this is a fresh terminal
.\scripts\powershell\01-resolve-addresses.ps1
.\scripts\powershell\stop.ps1
.\scripts\powershell\03-pull-image.ps1
.\scripts\powershell\08-run-worker.ps1 -NoTail
export NETWORK=testnet      # or mainnet — required if this is a fresh terminal
./scripts/bash/01-resolve-addresses.sh
./scripts/bash/stop.sh
./scripts/bash/03-pull-image.sh
./scripts/bash/08-run-worker.sh --no-tail

No re-registration needed — your on-chain state is independent of the binary version. No other onboarding phase (00, 02, 04-07) needs to be re-run.

Multi-worker on one machine

The toolkit assumes one worker per host. If you want multiple workers on the same machine:

  1. Each needs a separate worker key (run Phase 00 separately).
  2. Each needs a separate keys directory (set $env:KEYS_DIR / KEYS_DIR differently).
  3. Each needs a separate container name (set $env:CONTAINER_NAME differently).
  4. They share the same Ollama instance — but worker_max_jobs is 2 per worker by default, so 2 workers on one GPU is already 4 concurrent inferences. Watch VRAM.

The simplest layout: clone this repo twice into two folders, edit each env.ps1 / env.sh to use different KEYS_DIR and CONTAINER_NAME values, and run them independently.