diff --git a/content/build/create-a-view/examples/index.md b/content/build/create-a-view/examples/index.md deleted file mode 100644 index 89f73bc..0000000 --- a/content/build/create-a-view/examples/index.md +++ /dev/null @@ -1,623 +0,0 @@ -+++ -title = "View examples" -aliases = ["/views/examples"] -description = "Copy-pasteable Shinzo View examples: decode event logs, filter by contract, decode multiple event types, edit and update views, and query the results." -+++ - -Seven progressively more complex Views, from a basic event decode to a multi-event decoder with editing and updates. Each example shows the goal, the three View components (query, SDL, lens), the Viewkit commands to build it, and the GraphQL query you run against the result. - -## Primitive data - -Views query the primitive collections that Generator clients produce. All collection names are prefixed with `____`, derived from the Generator's `chain.name` and `chain.network` settings. Viewkit lets you use short names like `Log`, and the Host client auto-prefixes them at runtime. - -| Collection | Common fields | Typical use | -| --- | --- | --- | -| `Log` | `address`, `topics`, `data`, `transactionHash`, `blockNumber` | Event decoding (ERC-20, NFT, governance) | -| `Transaction` | `hash`, `from`, `to`, `value`, `blockNumber`, `status`, `gasUsed` | Transaction analytics | -| `Block` | `number`, `hash`, `timestamp`, `miner`, `gasUsed`, `gasLimit` | Block-level metadata | -| `AccessListEntry` | `address`, `storageKeys`, `blockNumber` | EIP-2930 access lists | - -There is no `Event` collection. Raw event data lives in `Log`, where `topics` holds indexed parameters and `data` holds non-indexed ones. A lens decodes those raw fields into structured output. For the full list of primitive collections, including `BlockSignature` and `SnapshotSignature`, see [Views for builders](/build/concepts/views-for-builders/#primitive-data-views-operate-on). - -## Decode event logs - -Decode all ERC-20 `Transfer` events into structured records. This is the simplest useful View that includes a lens: it decodes raw log `topics` and `data` into named fields using an ABI. - -### Query - -```graphql -Log { address topics data transactionHash blockNumber transaction { hash from to } } -``` - -The query selects raw log fields plus the nested `transaction` relation. The `decode_log` lens uses `transaction.hash`, `transaction.from`, and `transaction.to` to populate the output's `hash`, `from`, and `to` fields. - -### SDL - -```graphql -type EventView @materialized(if: true) { - hash: String - from: String - to: String - blockNumber: Int - logAddress: String - event: String - signature: String - arguments: [String] -} -``` - -The `decode_log` lens outputs these fields: - -- `hash`, `from`, `to`: from the parent transaction. -- `blockNumber`: block the log was emitted in. -- `logAddress`: the contract that emitted the log. -- `event`: decoded event name (e.g. `"Transfer"`). -- `signature`: event signature (e.g. `"Transfer(address,address,uint256)"`). -- `arguments`: array of decoded parameters as JSON strings. - -`@materialized(if: true)` tells DefraDB to pre-compute and store the output. See [Materialized versus on-query](#materialized-vs-on-query) for the tradeoff. - -### Lens - -| Lens | Purpose | Arguments | -| --- | --- | --- | -| `decode_log` | ABI-decode log events into named fields | `{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",...}]"}` | - -The `decode_log` lens takes an `abi` argument: a stringified JSON array of event definitions. For the ERC-20 `Transfer` event: - -```json -[{"type":"event","name":"Transfer","inputs":[{"type":"address","name":"from","indexed":true},{"type":"address","name":"to","indexed":true},{"type":"uint256","name":"value","indexed":false}]}] -``` - -### Commands - -1. Initialize the view: - - ```shell - viewkit view init event-view - ``` - -1. Add the query (raw log shape with transaction relation): - - ```shell - viewkit view add query \ - "Log { address topics data transactionHash blockNumber transaction { hash from to } }" \ - --name event-view - ``` - -1. Add the SDL (output schema matching decode_log output): - - ```shell - viewkit view add sdl \ - "type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ - --name event-view - ``` - -1. Attach the decode lens with the Transfer ABI: - - ```shell - viewkit view add lens \ - --label "decode-transfer" \ - --url "https://raw.githubusercontent.com/shinzonetwork/wasm-bucket/main/bucket/decode_log/decode_log.wasm" \ - --args '{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]}]"}' \ - --name event-view - ``` - -1. Inspect to confirm everything is attached: - -```shell -viewkit view inspect event-view -``` - -1. Test locally (optional but recommended): - -```shell -viewkit view test event-view -``` - -1. Deploy locally and explore in the playground: - -```shell -viewkit view deploy event-view --target local -``` - -### Querying the result - -Once deployed, open the DefraDB Playground (URL printed in the terminal) and run: - -```graphql -{ - EventView(limit: 10, order: { blockNumber: DESC }) { - hash - from - to - blockNumber - logAddress - event - signature - arguments - } -} -``` - -This returns all decoded `Transfer` events across all contracts. To narrow down to a specific token, see [Example 2](#filter-by-contract-address). - -## Filter by contract address - -Decode `Transfer` events from a specific contract only (e.g. USDC). Without a filter lens, `decode_log` processes every log on the chain. You filter the output using GraphQL queries against the `logAddress` field. - -### Query and SDL - -Same as the [Decode event logs example](#decode-event-logs). The query, SDL, and lens are identical. The filtering happens at query time, not at the lens level. - -### Commands - -Same as Example 1. Create a view named `usdc-event` with the same query, SDL, and lens. - -### USDC transfers only - -```graphql -{ - EventView( - filter: { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } - order: { blockNumber: DESC } - limit: 10 - ) { - hash - from - to - blockNumber - event - signature - arguments - } -} -``` - -The `logAddress` field contains the contract address that emitted the log. Filter on it to narrow results to one contract. - -### Filter by sender or receiver - -The `from` and `to` fields come from the parent transaction, not the event's indexed parameters. To filter by the event's `from`/`to` (the actual transfer sender and receiver), use the `arguments` field. With `decode_log_str` (which serializes `arguments` as a JSON string), you can use `_like`: - -```graphql -{ - EventView( - filter: { - _and: [ - { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } - { arguments: { _like: "%0x28C6c06298d514De0879A2640AB71F86b50Ce4E5%" } } - ] - } - ) { - hash - from - to - arguments - blockNumber - } -} -``` - -{% admonition(type="tip", title="decode_log vs decode_log_str") %} -`decode_log` outputs `arguments` as a JSON array (`[String]` in SDL). `decode_log_str` outputs it as a JSON string (`String` in SDL), which enables `_like` filtering in DefraDB queries. Use `decode_log_str` when you need to filter on decoded parameter values. The `_str` variant uses the same URL pattern but with `decode_log_str` in the path. -{% end %} - -## Decode multiple event types - -Decode both `Transfer` and `Approval` events from a single contract in one View. Pass both event definitions in the ABI argument to `decode_log`. - -### Query - -```graphql -Log { address topics data transactionHash blockNumber transaction { hash from to } } -``` - -### SDL - -```graphql -type EventView @materialized(if: true) { - hash: String - from: String - to: String - blockNumber: Int - logAddress: String - event: String - signature: String - arguments: [String] -} -``` - -### Lens - -The ABI argument includes both `Transfer` and `Approval` event definitions. The `decode_log` lens matches each log's `topics[0]` against the event signature hash and decodes accordingly. - -```json -[ - {"type":"event","name":"Transfer","inputs":[{"type":"address","name":"from","indexed":true},{"type":"address","name":"to","indexed":true},{"type":"uint256","name":"value","indexed":false}]}, - {"type":"event","name":"Approval","inputs":[{"type":"address","name":"owner","indexed":true},{"type":"address","name":"spender","indexed":true},{"type":"uint256","name":"value","indexed":false}]} -] -``` - -### Commands - -```shell -# 1) initialize the view -viewkit view init erc20-events - -# 2) add the query -viewkit view add query \ - "Log { address topics data transactionHash blockNumber transaction { hash from to } }" \ - --name erc20-events - -# 3) add the SDL -viewkit view add sdl \ - "type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ - --name erc20-events - -# 4) attach the decode lens with both Transfer and Approval in the ABI -viewkit view add lens \ - --label "decode-erc20" \ - --url "https://raw.githubusercontent.com/shinzonetwork/wasm-bucket/main/bucket/decode_log/decode_log.wasm" \ - --args '{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]},{\"type\":\"event\",\"name\":\"Approval\",\"inputs\":[{\"type\":\"address\",\"name\":\"owner\",\"indexed\":true},{\"type\":\"address\",\"name\":\"spender\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]}]"}' \ - --name erc20-events - -# 5) inspect, test, and deploy -viewkit view inspect erc20-events -viewkit view test erc20-events -viewkit view deploy erc20-events --target local -``` - -### Transfers only - -```graphql -{ - EventView( - filter: { - _and: [ - { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } - { event: { _eq: "Transfer" } } - ] - } - limit: 10 - ) { - hash - from - to - event - arguments - blockNumber - } -} -``` - -### Approvals only - -```graphql -{ - EventView( - filter: { - _and: [ - { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } - { event: { _eq: "Approval" } } - ] - } - limit: 10 - ) { - hash - from - to - event - arguments - blockNumber - } -} -``` - -The `event` field lets you distinguish between event types in the same View collection. - -## Transaction-based view (no lens) - -Expose all transactions sent to a specific contract. This View queries `Transaction` documents directly. No lens needed because we're not decoding events. - -### Query - -```graphql -Transaction { hash from to value blockNumber gasUsed gasPrice } -``` - -### SDL - -```graphql -type TransactionView @materialized(if: false) { - hash: String - from: String - to: String - value: String - blockNumber: Int - gasUsed: String - gasPrice: String -} -``` - -Here we use `@materialized(if: false)`: the view is computed on query, not pre-stored. This makes sense for transaction data, which is large and queried less frequently than decoded events. See [Example 5](#materialized-vs-on-query) for details. - -### Lens - -None. The query and SDL are sufficient. DefraDB applies the view as a virtual projection over the `Transaction` collection. - -### Commands - -```shell -# 1) initialize the view -viewkit view init transaction-view - -# 2) add the query -viewkit view add query \ - "Transaction { hash from to value blockNumber gasUsed gasPrice }" \ - --name transaction-view - -# 3) add the SDL -viewkit view add sdl \ - "type TransactionView @materialized(if: false) { hash: String from: String to: String value: String blockNumber: Int gasUsed: String gasPrice: String }" \ - --name transaction-view - -# 4) inspect, test, and deploy -viewkit view inspect transaction-view -viewkit view test transaction-view -viewkit view deploy transaction-view --target local -``` - -### Querying the result - -```graphql -{ - TransactionView( - filter: { to: { _eq: "0x68b3465833fb72A70ecDF485E0e4C7bD8665Fc45" } } - order: { blockNumber: DESC } - limit: 10 - ) { - hash - from - to - value - blockNumber - gasUsed - gasPrice - } -} -``` - -## Materialized vs on-query - -Understand when to use `@materialized(if: true)` vs `@materialized(if: false)`. - -The `@materialized` directive controls when the View's output is computed: - -| `@materialized` | When it computes | Query speed | Storage | Best for | -| --- | --- | --- | --- | --- | -| `if: true` | At write time. Host pre-computes and stores results. | Fast. Data is already materialized. | Higher. Host stores the output collection. | Frequently queried data (e.g. token transfers in a UI). | -| `if: false` | At query time. Host computes on the fly. | Slower. Depends on data volume. | Lower. No pre-stored output. | Large datasets queried occasionally, or during development. | - -### Same view in two modes - -Materialized (pre-computed): - -```graphql -type EventView @materialized(if: true) { - hash: String - from: String - to: String - blockNumber: Int - logAddress: String - event: String - signature: String - arguments: [String] -} -``` - -On-query (virtual): - -```graphql -type EventView @materialized(if: false) { - hash: String - from: String - to: String - blockNumber: Int - logAddress: String - event: String - signature: String - arguments: [String] -} -``` - -### Switching modes - -To toggle materialization on an existing view, update the SDL: - -```shell -# switch to materialized -viewkit view add sdl \ - "type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ - --name event-view - -# or switch to on-query -viewkit view add sdl \ - "type EventView @materialized(if: false) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ - --name event-view -``` - -Then redeploy: - -```shell -viewkit view test event-view -viewkit view deploy event-view --target local -``` - -{% admonition(type="tip") %} -Use `@materialized(if: false)` while developing and iterating on a View. Switch to `@materialized(if: true)` once the View is stable and you need fast queries in production. -{% end %} - -## Editing and updating a view - -Modify an existing View without starting from scratch. This example builds on the `erc20-events` View from [Example 3](#decode-multiple-event-types) and shows the full edit-update lifecycle: add an SDL field, swap a lens, inspect revisions, roll back, test, and redeploy. - -### Starting point - -Assume you already have `erc20-events` deployed with: - -- Query: `Log { address topics data transactionHash blockNumber transaction { hash from to } }` -- SDL: `type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }` -- Lens: `decode-erc20` (decode_log, Transfer + Approval ABI) - -### Swap the lens to decode three event types - -Remove the old lens, then add a new one with an updated ABI that includes `Transfer`, `Approval`, and `Transfer` (ERC-721, which has a `tokenId` instead of `value`): - -```shell -# remove the old lens -viewkit view remove lens \ - --label "decode-erc20" \ - --name erc20-events - -# add a new lens with three event types -viewkit view add lens \ - --label "decode-multi" \ - --url "https://raw.githubusercontent.com/shinzonetwork/wasm-bucket/main/bucket/decode_log/decode_log.wasm" \ - --args '{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]},{\"type\":\"event\",\"name\":\"Approval\",\"inputs\":[{\"type\":\"address\",\"name\":\"owner\",\"indexed\":true},{\"type\":\"address\",\"name\":\"spender\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]},{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"tokenId\",\"indexed\":true}]}]"}' \ - --name erc20-events -``` - -### Inspect with revision history - -Every `add` and `remove` creates a new revision. To see the full history: - -```shell -viewkit view inspect erc20-events --verbose -``` - -This shows the current state (query, SDL, lenses) and all past revisions, each with a version number. - -### Roll back if something went wrong - -If the updated ABI doesn't work as expected, revert to the previous version: - -```shell -# roll back to the most recent previous version -viewkit view rollback erc20-events -``` - -Or roll back to a specific version: - -```shell -viewkit view rollback erc20-events --version 3 -``` - -### Test and redeploy - -```shell -# validate the updated view compiles -viewkit view test erc20-events - -# deploy locally to verify in the playground -viewkit view deploy erc20-events --target local - -# once verified, deploy to devnet -viewkit view deploy erc20-events --target devnet --rpc http://34.29.171.79:8545/ -``` - -### Delete a view (if needed) - -To remove a view bundle from your local machine entirely: - -```shell -viewkit view delete erc20-events -``` - -This deletes the local bundle. It does not remove a view that has already been deployed to devnet. On-chain registrations are permanent. To update a deployed view, deploy a new version with the same name. - -## Querying a deployed view - -GraphQL queries you can run against a deployed View's output collection. These examples assume the `erc20-events` View from Example 3 is deployed and receiving data. - -### Basic query for the latest 10 events - -```graphql -{ - EventView( - order: { blockNumber: DESC } - limit: 10 - ) { - hash - from - to - blockNumber - logAddress - event - signature - arguments - } -} -``` - -### Filter by contract and event type - -```graphql -{ - EventView( - filter: { - _and: [ - { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } - { event: { _eq: "Transfer" } } - ] - } - limit: 10 - ) { - hash - from - to - arguments - blockNumber - } -} -``` - -### Filter by block range - -```graphql -{ - EventView( - filter: { blockNumber: { _gte: 19540000 } } - ) { - hash - from - to - event - blockNumber - } -} -``` - -### Filter by transaction hash - -```graphql -{ - EventView( - filter: { hash: { _eq: "0xabc123..." } } - ) { - hash - from - to - event - signature - arguments - blockNumber - } -} -``` - -For the full list of Viewkit commands and GraphQL filter operators, see the [Viewkit reference](/reference/components/viewkit/). For more on lenses, available modules, and how to chain them, see the [Lenses guide](/reference/components/lens/). For troubleshooting and common errors, see the [FAQ](/run/operations/troubleshooting/). - -## Need help - -{{ need_help(client="Viewkit", repo_name="shinzo-view-creator", repo="https://github.com/shinzonetwork/shinzo-view-creator/issues") }} diff --git a/content/build/how-to/configure-attestation-thresholds/index.md b/content/build/how-to/configure-attestation-thresholds/index.md index 5f722c8..53a9a46 100644 --- a/content/build/how-to/configure-attestation-thresholds/index.md +++ b/content/build/how-to/configure-attestation-thresholds/index.md @@ -3,4 +3,97 @@ title = "Configure attestation thresholds" description = "How to require a minimum number of Generator attestations before query results are returned in your app." +++ -This page is coming soon. +Shinzo data is signed by the Generator clients that produced it, and Host clients keep attestation records that track how many independent Generator clients signed the same data. With the app-sdk you can set a bar: only return documents whose attestation count meets your threshold. The threshold is a query-time filter, not a system-wide setting, so one app can apply different bars to different queries. + +{% admonition(type="warning") %} +The attestation helpers described here live on the `Feature/attestationFilter` branch of the app-sdk and are not merged into `main` yet. They also depend on pushed replication, which is currently blocked by the DefraDB version mismatch described in [Subscribe to Views with the app-sdk](/build/how-to/subscribe-to-views/). This page documents the API as implemented on that branch so you can build against it ahead of the merge. +{% end %} + +## Add attestation records for a View + +Attestation records are segmented per View, so your app only receives records for the data it cares about. Opt in per View with `AddAttestationRecordCollection`: + +```go +import "github.com/shinzonetwork/shinzo-app-sdk/pkg/attestation" + +err := attestation.AddAttestationRecordCollection(context.Background(), myNode, myView.Name) +if err != nil { + if strings.Contains(err.Error(), "collection already exists") { + // Records for this View were added before. Informational and safe to ignore. + } else { + panic(err) + } +} +``` + +This works like `SubscribeTo`: it adds an `AttestationRecord_` collection to your embedded DefraDB instance and registers it for passive replication, so Host clients push the View's attestation records alongside its documents. Call it once per View you want to filter, after subscribing to the View itself. + +## Choose configured or per-query thresholds + +Four helpers cover the two ways to set the bar. All four work like `defra.QuerySingle` and `defra.QueryArray`, except they drop results that fail the attestation check. Your result struct needs a `DocID` field, because the filter matches documents to their attestation records by DocID. + +The configured pair reads the threshold from `shinzo.minimum_attestations` in your config: + +```go +transfers, err := attestation.QueryArrayWithConfiguredAttestationFilter[Transfer](ctx, myNode, query) +transfer, err := attestation.QuerySingleWithConfiguredAttestationFilter[Transfer](ctx, myNode, query) +``` + +The per-call pair takes the threshold as an argument: + +```go +transfers, err := attestation.QueryArrayWithAttestationFilter[Transfer](ctx, myNode, query, 3) +transfer, err := attestation.QuerySingleWithAttestationFilter[Transfer](ctx, myNode, query, 3) +``` + +Set the config default in `config.yaml`: + +```yaml +shinzo: + minimum_attestations: 2 +``` + +Which style to use depends on how uniform your trust requirements are: + +| Situation | Approach | +| --- | --- | +| One threshold covers the whole app | Configured helpers with `minimum_attestations` | +| A wallet display where showing something fast beats certainty | Per-call threshold of 1 | +| A high-value flow like a settlement or payout | Per-call threshold of 3 or more | +| A mix of casual and critical reads in one app | Configured default, per-call overrides where it matters | + +Make sure you added the attestation record collection for any View you query through these helpers. Without it there are no records to filter on, and every result fails the check. + +## Debug an empty result set + +If a filtered query returns nothing but the unfiltered equivalent has rows, the filter is doing its job and your data is under-attested. Inspect the records directly to see why. An attestation record ties one of your View's documents to the evidence behind it: + +- `attested_doc` is the DocID of the View document being attested to. +- `source_doc` links back to the source document the attestation came from. +- `CIDs` are the signed commit CIDs backing the attestation. +- `doc_type` names the attested collection. +- `vote_count` is a CRDT counter that Host clients increment as they observe more Generator clients signing the same data. + +Query the records for the document that went missing: + +```graphql +{ + ____AttestationRecord( + filter: { attested_doc: { _eq: "" } } + ) { + attested_doc + source_doc + CIDs + doc_type + vote_count + } +} +``` + +If `vote_count` (or the number of records) is below your threshold, the filter correctly excluded the document. Lower the threshold, or wait for more Generator clients to attest. Note that attestations only accumulate while Generator clients are actually signing the underlying data, so a quiet View on a testnet may legitimately sit at a low count. + +To check signatures and CIDs by hand, see [Verify data with signatures and CIDs](/build/how-to/verify-data/). For the reasoning behind per-query trust, see [Attestation as a query filter](/build/explanation/attestation-as-a-query-filter/), and [Attestation](/understand/core-concepts/attestation/) for the platform-level picture. + +## Need help + +{{ need_help(client="app-sdk", repo_name="app-sdk", repo="https://github.com/shinzonetwork/app-sdk/issues") }} diff --git a/content/build/how-to/connect-to-a-host/index.md b/content/build/how-to/connect-to-a-host/index.md index 18b9035..226a1f1 100644 --- a/content/build/how-to/connect-to-a-host/index.md +++ b/content/build/how-to/connect-to-a-host/index.md @@ -3,4 +3,87 @@ title = "Connect your app to a Host" description = "How to connect an application to a Shinzo Host: P2P connection strings for embedded Go apps and GraphQL endpoints for direct-query TypeScript apps." +++ -This page is coming soon. +How you connect to a Host depends on your architecture. A local-first Go app embeds DefraDB and peers with Hosts over libp2p, so it needs connection strings. A direct-query TypeScript app talks to a Host's GraphQL endpoint over HTTP, so it needs an endpoint URL. Both start with the discovery step in [Find Views and Hosts](/build/how-to/find-views-and-hosts/), which explains where each value comes from. + +A Host client exposes four interfaces: + +| Port | Interface | +| --- | --- | +| 9181 | GraphQL API at `/api/v0/graphql` | +| 9182 | GraphQL playground, when enabled | +| 9171 | libp2p peering | +| 8080 | Health and self-description endpoints | + +Public Hosts vary in how they publish these, so always read the actual `connection_string` and `endpoint_address` from the registry rather than assuming ports. + +## Check a Host's health first + +Before wiring a Host into your app, check that it is alive and processing data: + +```shell +curl -s -H "Accept: application/json" http://:8080/health | jq '{status, current_block}' +``` + +```json +{ + "status": "healthy", + "current_block": 25903651 +} +``` + +The `Accept: application/json` header asks for the JSON form; a browser gets an HTML status page instead. A `healthy` status with a recent `current_block` means the Host is peered and syncing. The same server answers `GET /registration` with the Host's DID, `connection_string`, and `endpoint_address`, which is handy for confirming a Host's identity before you trust its data. + +## Embedded app (Go) + +A local-first app receives pushed data over libp2p, so the connection happens in the app-sdk config. Take the Host's `connection_string` from the registry: + +```shell +curl -s http://testnet.shinzo.network:1317/shinzonetwork/host/v1/hosts \ + | jq -r '.hosts[].connection_string' +``` + +Add one or more of them to `defradb.p2p.bootstrap_peers` in your `config.yaml`: + +```yaml +defradb: + p2p: + enabled: true + bootstrap_peers: + - "/ip4/34.66.172.230/tcp/9171/p2p/12D3KooWKVCMswzcXYe9kW2z7nSB9YUWjVVLsMbJnBjVPFUQkbQ7" + listen_addr: "/ip4/127.0.0.1/tcp/9171" +``` + +The full multiaddr form `/ip4//tcp/9171/p2p/` always works, but it is not required. Bare IPs (`34.66.172.230`) and `ip:port` pairs (`34.66.172.230:9171`) also work, because the client discovers the peer ID during the connection handshake and fills it in for you. Listing several bootstrap peers makes the first connection more reliable, since registered Hosts come and go on a testnet. + +Once peered, subscribe to a View and data starts arriving. [Subscribe to Views with the app-sdk](/build/how-to/subscribe-to-views/) covers that flow. + +## Direct-query app (TypeScript) + +A direct-query app never peers with anything. It reads the Host's `endpoint_address` from the registry and POSTs signed GraphQL requests to it: + +```shell +curl -s http://testnet.shinzo.network:1317/shinzonetwork/host/v1/hosts \ + | jq -r '.hosts[].endpoint_address' +``` + +The endpoint already includes the API path, so you POST straight to it: + +```shell +curl -s -X POST "http://34.66.172.230/api/v0/graphql" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ Erc20Event(limit: 1) { blockNumber } }", "extensions": { ... } }' +``` + +View queries carry a signature in the `extensions` envelope, and one billed query maps to one pool: a request may touch only one View collection, and its `pool_address` extension names the pool it bills to. When a Host enforces billing, rejections come back as plain errors: `403` if the request signature is missing, stale, or fails verification, and `402` if the signer's query balance is too low. [Query your first View](/build/tutorials/query-your-first-view/) builds the signing flow end to end. + +{% admonition(type="note") %} +Billing enforcement is rolling out on the testnet, so some Hosts still answer unsigned queries. Signed requests are the supported interface either way; treat unsigned access as a convenience that will go away. +{% end %} + +## Prefer your own Host + +Public Hosts are shared infrastructure. If you want guaranteed availability, or you do not want a third party to see your queries at all, you can point everything above at a Host you run yourself. See [Use your own infrastructure](/build/how-to/use-your-own-infrastructure/). + +## Need help + +{{ need_help(client="Host", repo_name="shinzo-host-client", repo="https://github.com/shinzonetwork/shinzo-host-client/issues") }} diff --git a/content/build/how-to/find-views-and-hosts/index.md b/content/build/how-to/find-views-and-hosts/index.md index 627bf35..4d3d91b 100644 --- a/content/build/how-to/find-views-and-hosts/index.md +++ b/content/build/how-to/find-views-and-hosts/index.md @@ -3,4 +3,122 @@ title = "Find Views and Hosts" description = "How to discover registered Views, serving Hosts, and pools on the Shinzo network using the Explorer, Studio, the REST API, or the TypeScript SDK." +++ -This page is coming soon. +Views, Hosts, and Generators are all registered on ShinzoHub, so discovery is a matter of reading the registry. There are four ways to do it: the Explorer in a browser, Shinzo Studio, the chain's REST API, or the TypeScript SDK. Pick whichever fits the job; they all read the same on-chain state. + +Two fields come up constantly, so get familiar with them first: + +- `connection_string` is a libp2p multiaddr, like `/ip4/34.66.172.230/tcp/9171/p2p/12D3KooW...`. It is how peers dial each other. Your local-first app needs it. +- `endpoint_address` is the URL of a Host's GraphQL API, like `http://34.66.172.230/api/v0/graphql`. It is what direct-query apps POST to. + +## Browse the Explorer + +The [Shinzo Explorer](https://explorer.shinzo.network/shinzohub) renders the registry as web pages. Open it and switch between the **Blocks**, **Transactions**, **Generators**, **Hosts**, and **Validators** tabs. + +The **Hosts** tab lists every registered Host client with its connection details, and the **Generators** tab lists the registered Generator clients. Use the Explorer when you want to eyeball the network: who is online, what is registered, and whether anything changed recently. + +## Browse the View catalog in Studio + +[Shinzo Studio](https://studio.shinzo.network/) shows registered Views in its catalog, each with its definition, pool, and serving Hosts. Studio is the better browser experience when your question is "what Views exist and what do they return", because it shows the View's SDL alongside its network status. You can also deploy your own View from the same UI; see [Create and deploy Views in Shinzo Studio](/build/how-to/use-shinzo-studio/). + +## Query the REST API + +ShinzoHub exposes a REST API at `http://testnet.shinzo.network:1317`. This is the option for scripts and terminals, and the responses below are real testnet output. + +List registered Views: + +```shell +curl -s "http://testnet.shinzo.network:1317/shinzonetwork/view/v1/views?include_data=true" \ + | jq -r '.views[] | .name + " " + .address' +``` + +```output +Studio_v1_Erc20TransferUSDC 0xD3084cAddCe8E1bab07C6eDd8afb835566904C6B +Erc20Event 0xEAc245f905e0aAcF3b9Fe27153F2AaF485dc1B48 +FilteredAndDecodedLogs 0xa1226B03c54789e9Bf8876ac956aBbD1bDf5B654 +... +``` + +List registered Host clients: + +```shell +curl -s "http://testnet.shinzo.network:1317/shinzonetwork/host/v1/hosts" \ + | jq -r '.hosts[] | .connection_string + " " + .endpoint_address' +``` + +```output +/ip4/34.66.172.230/tcp/9171/p2p/12D3KooWKVCMswzcXYe9kW2z7nSB9YUWjVVLsMbJnBjVPFUQkbQ7 http://34.66.172.230/api/v0/graphql +/ip4/34.63.186.249/tcp/9171/p2p/12D3KooWSqvLctTtcQLvqSVZU4sTCUWxCX9z4NeFpSHnmVWBiFMZ http://34.63.186.249/api/v0/graphql +... +``` + +List registered Generator clients: + +```shell +curl -s "http://testnet.shinzo.network:1317/shinzonetwork/indexer/v1/indexers" \ + | jq -r '.indexers[] | .operator_address + " " + .source_chain' +``` + +List the pools serving a View, with membership and activity: + +```shell +curl -s "http://testnet.shinzo.network:1317/shinzonetwork/pool/v1/views/0xEAc245f905e0aAcF3b9Fe27153F2AaF485dc1B48/pools" \ + | jq -c '.details[] | {pool: .pool.pool_address, hosts: (.hosts | length), is_active}' +``` + +```output +{"pool":"0xDbc3bE7CBd8Dc8901E3BbbeA1A740BE490dAe23B","hosts":3,"is_active":true} +``` + +A pool becomes active once at least 3 Hosts have joined it, so `is_active` is the quick check for whether a View is being served. The registry moves over time, so expect different names and addresses when you run these. + +## Query with the TypeScript SDK + +The same registry reads are available from `@shinzo/shinzohub` if your app needs them programmatically. The client extends a viem public client with ShinzoHub actions: + +```ts +import { createPublicClient, http } from "viem"; +import { shinzoHubActions } from "@shinzo/shinzohub"; +import { shinzoHubTestnet } from "@shinzo/shinzohub/chains"; + +const client = createPublicClient({ + chain: shinzoHubTestnet, + transport: http(), +}).extend(shinzoHubActions); + +const { views } = await client.listViews({ limit: 25, includeMetadata: true }); +const { hosts } = await client.listHosts({ limit: 100 }); +const pools = await client.listViewPools({ viewAddress: views[0].viewAddress }); +``` + +`listViews` returns registered Views (with the parsed query, SDL, and lens metadata when `includeMetadata` is set), `listHosts` returns Host clients with their `endpointAddress`, and `listViewPools` returns the pools for a View with member Hosts and an `isActive` flag. `getNetworkUnitPrice` reads the network-wide unit price for queries. The [Query your first View](/build/tutorials/query-your-first-view/) tutorial walks through combining these into a full query flow. + +## Ask a Host about itself + +Any running Host client describes itself over its health server, on port 8080 by default: + +```shell +curl -s -H "Accept: application/json" http://:8080/registration | jq .registration +``` + +```json +{ + "did": "did:key:zQ3shQKyThhTw3M83ZcTobUQER5mxNbSrW3Yrds3if4mPPrje", + "connection_string": "/ip4/203.0.113.10/tcp/9171/p2p/12D3KooWB1K1k67DNEcxShBq3o15LQrKHcBkRxzv6AuFR8p8idqJ", + "endpoint_address": "http://203.0.113.10/api/v0/graphql" +} +``` + +This is the same information a Host publishes to the registry when it registers, so it is the quickest way to check what a specific Host claims to be. See [Connect your app to a Host](/build/how-to/connect-to-a-host/) for the health endpoint and the rest of the port layout. + +## What does not exist + +Viewkit has no list or discovery commands. It builds and deploys View bundles, and that is all. Discovery happens on-chain through the paths above, or through the Explorer and Studio UIs listed in [Tools](/reference/tools/). + +## Where to next + +- [Connect your app to a Host](/build/how-to/connect-to-a-host/) to put a `connection_string` or `endpoint_address` to work. +- [Query your first View](/build/tutorials/query-your-first-view/) for the full flow from discovery to a signed query. + +## Need help + +{{ need_help(client="Host", repo_name="shinzo-host-client", repo="https://github.com/shinzonetwork/shinzo-host-client/issues") }} diff --git a/content/build/how-to/query-data/index.md b/content/build/how-to/query-data/index.md index 9461817..c627e20 100644 --- a/content/build/how-to/query-data/index.md +++ b/content/build/how-to/query-data/index.md @@ -1,6 +1,217 @@ +++ title = "Query data" +aliases = ["/hosts/examples", "/build/query-data/"] description = "How to query Shinzo data with GraphQL: filters, ordering, nested documents, DocIDs, CIDs, and signatures." +++ -This page is coming soon. +You query Shinzo data with GraphQL, and there are two places to run it. A local-first app queries its embedded DefraDB instance through the app-sdk helpers, with no network call. A direct-query app POSTs the same query to a Host client's `/api/v0/graphql` endpoint over HTTP. Same language, same collections, same filter operators on both. Only the transport changes. [Connect your app to a Host](/build/how-to/connect-to-a-host/) covers the wiring. + +Here are the patterns you'll reach for most. The examples use primitive collections (blocks, transactions, logs), and they all work against a View's output collection too. + +{% admonition(type="note") %} +Collection names are prefixed with `____`, derived from the `chain.name` and `chain.network` settings of the Generator client that indexed the data (for example `____Block` or `Optimism__Mainnet__Block`). The examples below use the `____` placeholder. Substitute the prefix that matches your chain. See the [chain config](/run/run-a-generator/config-reference#chain) for details. +{% end %} + +## Get the latest N documents + +Order by a field and cap the result with `limit`. This is the pattern behind most "recent activity" displays. + +```graphql +{ + ____Block(limit: 10, order: { number: DESC }) { + _docID + number + timestamp + hash + } +} +``` + +The same shape works for a View. This query fetches the 10 most recent decoded events from a View collection: + +```graphql +{ + EventView(limit: 10, order: { blockNumber: DESC }) { + hash + from + to + blockNumber + logAddress + event + arguments + } +} +``` + +## Fetch a document by DocID or CID + +When you already know a document's `_docID`, pass it as the `docID` argument to fetch exactly that document: + +```graphql +{ + ____Transaction(docID: "bae-25fb059c-f232-5305-8a5d-0162f01e43e6") { + _docID + blockHash + blockNumber + hash + to + from + value + } +} +``` + +Documents are also content-addressed. Passing a commit CID as the `cid` argument resolves the document at that exact version: + +```graphql +{ + ____Transaction(cid: "bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy") { + _docID + blockHash + blockNumber + hash + to + from + value + } +} +``` + +```json +{ + "data": { + "____Transaction": [ + { + "_docID": "bae-25fb059c-f232-5305-8a5d-0162f01e43e6", + "blockHash": "0x9ea35b3bd9e71c57617cc30394b22f607b735f2eea7a0db974cf02ad54de98fb", + "blockNumber": 23902272, + "from": "0x654a6BCe2C6F0aF68eAdCFEaD06bB49C398B3F98", + "hash": "0x61b79fc417ef183e1798681c59481410dd79f919d11806a6e7e77ebd0a744f78", + "to": "0x677f857da5e7C42b823655290cc40ff401e138D3", + "value": "1000000000" + } + ] + } +} +``` + +You usually get a CID from a document's `_version` field or from an attestation record. [Verify data with signatures and CIDs](/build/how-to/verify-data/) covers that flow. + +## Filter by field values + +The `filter` argument narrows results with operators like `_eq`, `_geq`, `_and`, and `_like`. This query returns blocks above a height: + +```graphql +{ + ____Block(filter: { number: { _geq: 19540000 } }) { + _docID + number + hash + } +} +``` + +Combine conditions with `_and`. This query returns `Transfer` events decoded from one contract: + +```graphql +{ + EventView( + filter: { + _and: [ + { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } + { event: { _eq: "Transfer" } } + ] + } + limit: 10 + ) { + hash + from + to + arguments + blockNumber + } +} +``` + +The full operator table lives in the [Viewkit reference](/reference/components/viewkit/#filter-operators). + +## Get a block with nested data + +Relations are nested in the query, so one round trip fetches a block with its transactions and their logs: + +```graphql +{ + ____Block(limit: 1) { + _docID + number + timestamp + hash + gasUsed + gasLimit + baseFeePerGas + parentHash + miner + transactions { + hash + transactionIndex + _docID + logs { + transactionHash + address + topics + data + } + } + } +} +``` + +Nested selections accept their own `filter`, `order`, and `limit` arguments, so you can shape each level independently. + +## Count the transactions in a block + +There is no aggregate count field, but the `transactionIndex` values within a block are zero-based and contiguous. Fetch the highest `transactionIndex` and add one: + +```graphql +{ + ____Block(filter: { number: { _eq: 23901130 } }) { + number + transactions( + limit: 1 + filter: { blockNumber: { _eq: 23901130 } } + order: { transactionIndex: DESC } + ) { + transactionIndex + } + } +} +``` + +The total transaction count is the returned `transactionIndex` plus 1. + +## Check who signed a document + +Every document carries signed commits. Selecting `_version` returns the CID and signature for each commit, which is the starting point for verifying data: + +```graphql +{ + ____Block(limit: 10, order: { number: DESC }) { + number + _docID + _version { + cid + signature { + identity + value + type + } + } + } +} +``` + +The `identity` field is the public key of the Generator client that signed the commit. For commit metadata, attestation records, and CID navigation, see [Verify data with signatures and CIDs](/build/how-to/verify-data/). + +## Need help + +{{ need_help(client="Host", repo_name="shinzo-host-client", repo="https://github.com/shinzonetwork/shinzo-host-client/issues") }} diff --git a/content/build/how-to/subscribe-to-views/index.md b/content/build/how-to/subscribe-to-views/index.md index 9898a5e..94318d9 100644 --- a/content/build/how-to/subscribe-to-views/index.md +++ b/content/build/how-to/subscribe-to-views/index.md @@ -3,4 +3,157 @@ title = "Subscribe to Views with the app-sdk" description = "How to configure and start an embedded DefraDB instance, subscribe to Views, and receive pushed data in a Go application." +++ -This page is coming soon. +A local-first app embeds a DefraDB instance through the [app-sdk](https://github.com/shinzonetwork/shinzo-app-sdk), subscribes to Views, and lets Host clients push pre-processed data to it over P2P. This page covers the mechanics: configuration, startup and shutdown, subscribing, and querying what arrives. For why the model works this way, see [The Shinzo app model](/build/explanation/the-app-model/). For filtering results by Generator attestations, see [Configure attestation thresholds](/build/how-to/configure-attestation-thresholds/). + +Install the SDK with Go modules: + +```shell +go get github.com/shinzonetwork/shinzo-app-sdk +``` + +{% admonition(type="warning") %} +Pushed replication is currently blocked by a version mismatch. The app-sdk pins DefraDB v0.20 while public Host clients run DefraDB v1.0, and documents published by a v1.0 Host cannot be parsed by a v0.20 app, so nothing arrives yet. Everything on this page is correct against the current app-sdk: your app connects and subscribes successfully, and its queries return empty until the SDK ships a compatible DefraDB. If you need data in a Go app today, query a Host directly instead, as described in [Query data](/build/how-to/query-data/). +{% end %} + +## Configure the SDK + +The SDK loads a YAML config file. A minimal one for an app that only holds Shinzo data: + +```yaml +defradb: + url: "http://localhost:9181" + keyring_secret: "dev-secret" + p2p: + enabled: true + bootstrap_peers: + - "/ip4/34.66.172.230/tcp/9171/p2p/12D3KooWKVCMswzcXYe9kW2z7nSB9YUWjVVLsMbJnBjVPFUQkbQ7" + listen_addr: "/ip4/127.0.0.1/tcp/9171" + store: + path: "./.defra" + +shinzo: + minimum_attestations: 1 + +logger: + development: true +``` + +Three keys deserve attention: + +- `defradb.keyring_secret` encrypts the local keyring that holds your node's identity, so the app keeps the same P2P identity across restarts. It can also come from the `DEFRA_KEYRING_SECRET` environment variable. +- `defradb.p2p.enabled` must be `true`, or the instance starts with networking off and nothing can be pushed to it. `defradb.p2p.bootstrap_peers` lists the Host clients to dial; [Connect your app to a Host](/build/how-to/connect-to-a-host/) shows where to get theirs. +- `shinzo.minimum_attestations` sets the default attestation threshold for filtered queries. It only needs a valid value until you start using the attestation query helpers. +- `logger.development` keeps DefraDB's logs visible while you develop. Set it to `false` in production; DefraDB logs a lot. + +Load the file with `config.LoadConfig`: + +```go +shinzoConfig, err := config.LoadConfig("config.yaml") +if err != nil { + panic(err) +} +``` + +In tests, where the working directory is unpredictable, the `file.FindFile` helper locates the config by walking up from the current directory: + +```go +configPath, err := file.FindFile("config.yaml") +if err != nil { + panic(err) +} +shinzoConfig, err := config.LoadConfig(configPath) +``` + +If you pass `nil` instead of a loaded config, the SDK builds a default config for you. That is fine for a quick experiment, but real apps should manage a file: the defaults cannot know your bootstrap peers. + +## Start and stop the embedded instance + +`defra.StartDefraInstance` boots the embedded DefraDB node. Its second argument is a `SchemaApplier`, which decides what non-View schema gets applied at startup: + +```go +myNode, _, err := defra.StartDefraInstance( + shinzoConfig, + &defra.MockSchemaApplierThatSucceeds{}, + nil, + nil, +) +if err != nil { + panic(err) +} +defer myNode.Close(context.Background()) +``` + +The two `nil` arguments are optional node options and a replication filter, which most apps do not need. The second return value is the network handler, which you can ignore for basic subscriptions. + +Pick the `SchemaApplier` that matches how you use DefraDB: + +- `MockSchemaApplierThatSucceeds` applies nothing. Use it when DefraDB only holds Shinzo data, which is the common case. +- `SchemaApplierFromFile` reads a schema from a file, and `SchemaApplierFromProvidedSchema` takes a schema string. Use either when your app also stores its own documents in the same DefraDB instance; your collections go in the schema, and View collections arrive later through subscriptions. + +Whatever you start, close it. `myNode.Close(context.Background())` shuts the node down cleanly, and `defer` is the easiest way to guarantee it runs. + +## Subscribe to a View + +Subscribing is what turns a registered View into pushed data. Define the View and call `SubscribeTo`: + +```go +sdl := `type Studio_v1_Erc20TransferUSDC { + tokenAddress: String + hash: String + blockNumber: Int + from: String + to: String + amount: String +}` + +view := views.View{ + Name: "Studio_v1_Erc20TransferUSDC", + Sdl: &sdl, +} + +err = view.SubscribeTo(context.Background(), myNode) +if err != nil { + if strings.Contains(err.Error(), "collection already exists") { + // You have subscribed before. The error is informational and safe to ignore. + } else { + panic(err) + } +} +``` + +Only the `Name`, `Sdl`, and optionally `Query` fields of the `View` struct matter for subscribing. The `Query` field is not shown here because the SDL alone is enough to receive and store documents. + +`SubscribeTo` does two things. It applies the View's SDL to your embedded instance, so the collection exists locally and can be queried. And it registers that collection as a topic in DefraDB's passive replication, which is the signal that tells connected Host clients to push the View's documents to you. + +The "collection already exists" error is common and expected: it means you have subscribed to this View before, so the collection is already in place. It is informational and safe to ignore. Other errors are real and should not be swallowed. + +## Query the pushed data + +Once subscribed, Host clients push the View's documents into your local collection as they process new blocks. Query the collection with the SDK's generic helpers and a Go struct that matches the fields: + +```go +type Transfer struct { + TokenAddress string `json:"tokenAddress"` + From string `json:"from"` + To string `json:"to"` + Amount string `json:"amount"` + BlockNumber int `json:"blockNumber"` +} + +transfers, err := defra.QueryArray[Transfer]( + context.Background(), + myNode, + `query { Studio_v1_Erc20TransferUSDC(limit: 10) { tokenAddress from to amount blockNumber } }`, +) +if err != nil { + panic(err) +} +``` + +`defra.QueryArray[T]` returns a slice of `T`, and `defra.QuerySingle[T]` returns one document when you expect exactly one. The GraphQL itself is the same language you would run against a Host's endpoint, so [Query data](/build/how-to/query-data/) applies here too. + +Until the DefraDB version gap described at the top is closed, these queries return empty results. Once a compatible app-sdk release lands, the same code starts returning rows a few moments after subscribing. + +## Need help + +{{ need_help(client="app-sdk", repo_name="app-sdk", repo="https://github.com/shinzonetwork/app-sdk/issues") }} diff --git a/content/build/how-to/use-shinzo-studio/index.md b/content/build/how-to/use-shinzo-studio/index.md index 02a69c0..f3ed0bd 100644 --- a/content/build/how-to/use-shinzo-studio/index.md +++ b/content/build/how-to/use-shinzo-studio/index.md @@ -3,4 +3,49 @@ title = "Create and deploy Views in Shinzo Studio" description = "How to create, deploy, and query Views from your browser with Shinzo Studio, without installing the CLI." +++ -This page is coming soon. +[Shinzo Studio](https://studio.shinzo.network/) covers the same create, deploy, and query flow as Viewkit for developers who prefer a browser UI. Nothing installs locally; you need a browser wallet and some testnet SHNZ. + +## Connect your wallet + +1. Get testnet SHNZ from the [faucet](https://faucet.shinzo.network/) if you do not have any. Deploying a View and funding a query balance are on-chain transactions, so the wallet needs a small balance for network fees on top of whatever you spend. +1. Open [studio.shinzo.network](https://studio.shinzo.network/) and click **Connect Wallet**. +1. Choose a wallet from the list and approve the connection. +1. Studio asks you to switch to the Shinzo network. Confirm the network change in your wallet when prompted. The network uses chain ID 91273001; depending on the Studio deployment, your wallet may label it Shinzo testnet or Shinzo devnet. + +Once connected, Studio shows your wallet balance and query balance, and the View catalog becomes available. + +## Create a View + +1. Start a new View from the catalog. Studio offers templates for the common cases, built on the same prebuilt lenses the CLI uses: **Decode Contract Events** (decodes verified event logs for one contract on Ethereum mainnet), **ERC20 Transfers** (normalized transfer rows for one token contract), and **ERC20 Balances** (account balances and transfer counts for one token contract). +1. Enter what the template asks for. For the decode template that is the contract address, for example `0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48` for USDC. Studio fetches the contract's verified ABI from Sourcify and picks out the event definitions, so the ABI has to be verified there. If Sourcify has no verified ABI for the contract, Studio tells you and stops. +1. Review the generated View definition: the source query, the SDL, and the lens definition with its arguments. This is exactly the bundle Viewkit would produce for the same inputs, and it is worth reading before you sign anything. + +## Deploy the View + +1. Click **Deploy View**. Studio validates the definition, builds the View bundle, and asks your wallet to sign the registration transaction. +1. Confirm in your wallet. Studio sends the transaction and waits for confirmation. +1. Wait for registration to complete. Registration on ShinzoHub finishes asynchronously, so Studio polls until the View reaches REGISTERED status. This can take a short while; you can inspect the View definition in the meantime. + +After registration, Studio prompts you to back the View with demand. Creating demand bonds SHNZ to the View's pool and is how Host clients see that the View should be served. The first demand creates the pool, and the pool becomes active once at least 3 Hosts have joined it. Until then Studio shows the pool as waiting for Hosts. + +## Fund your query balance + +1. Open your View and click **Add query funds** (also shown as **Add to query balance**). +1. Enter an amount of SHNZ and approve the deposit transaction in your wallet. +1. Wait for the **Query balance funded** confirmation. + +{% admonition(type="warning") %} +Query billing is still being finalized on the testnet, and the details here may change. Fund small amounts for now, and expect the metering rules to evolve. There is no dedicated billing guide yet. +{% end %} + +## Query the View + +1. Open the View and click **Query this view**. +1. Pick a Host with **Select direct query host**. Studio lists the Hosts currently serving the View's pool, so choose one of those; a Host outside the pool does not have the View's data. +1. Write or adjust the GraphQL query in the editor and run it. Each run asks your wallet for one signature and spends from your prepaid query balance, so keep an eye on the balance Studio shows. If it runs out, Studio refuses to send and tells you to add funds first. + +The results come straight from the Host you picked. To script the same flow outside the browser, see [Query your first View](/build/tutorials/query-your-first-view/). To build the same View from the terminal instead, see [Create your first View](/build/tutorials/create-your-first-view/). + +## Need help + +{{ need_help(client="Shinzo SDK", repo_name="web", repo="https://github.com/shinzonetwork/web/issues") }} diff --git a/content/build/how-to/use-your-own-infrastructure/index.md b/content/build/how-to/use-your-own-infrastructure/index.md index b54b88f..d8db76f 100644 --- a/content/build/how-to/use-your-own-infrastructure/index.md +++ b/content/build/how-to/use-your-own-infrastructure/index.md @@ -3,4 +3,33 @@ title = "Use your own infrastructure" description = "How to run your own Host as a private Direct Client for your app: the strongest privacy and control, at the cost of operating a node." +++ -This page is coming soon. +Everything in the Build apps section works against public infrastructure, and for most apps that is the right trade. But public Hosts are shared: they see your queries, their availability is not yours to control, and your app inherits their trust posture. Running your own Host client as a private Direct Client flips that. You get the strongest query privacy available today and full control over the data path, at the cost of operating a node. + +The privacy point is worth stating plainly. When your app queries a public Host, that Host's operator can observe what you ask. When your app embeds DefraDB and receives pushed data, the Hosts you peer with still see what you subscribe to. Querying a Host you run yourself is the only configuration today where no third party can observe your queries, because there is no third party in the path at all. [Privacy](/understand/core-concepts/privacy/) lays out the full model. + +## Pick a privacy tier + +A private Host client comes in two tiers, and the difference is how much of the public network it talks to. + +A standard private setup keeps ShinzoHub connected so your Host still fetches and runs the public Views, but makes the Host invisible: you skip registration, and you set `defradb.p2p.bootstrap_peers` to your own Generator client only. You get the public Views without exposing your read path to the network. + +A fully air-gapped setup cuts ShinzoHub off entirely by setting `shinzo.hub_base_url` to an empty string. The Host contacts nothing except the Generator client you point it at, and nothing about it is published anywhere. This is the most private setup possible, and it means you manage View definitions yourself. + +## Set it up + +This page is the bridge, not the guide. The run section has the real instructions: + +1. Run a Generator client to produce the signed primitive data. Start with [Run a Generator](/run/run-a-generator/install/) for installation and configuration. +1. Run a Host client against it. [Install a Host](/run/run-a-host/install/) covers the binary and ports, and [Private Hosts](/run/run-a-host/private-hosts/) walks through both privacy tiers with complete configs, including how to load Views onto an air-gapped Host. +1. Point your app at your Host exactly as you would at a public one. The `connection_string` and `endpoint_address` come from your Host's own `/registration` endpoint. + +Expect to operate real infrastructure: a synced source-chain node or a managed endpoint for the Generator, disk for DefraDB, and the usual monitoring. + +## Back to building + +- [Connect your app to a Host](/build/how-to/connect-to-a-host/): everything on that page works unchanged against your own Host. +- [Choosing an app architecture](/build/explanation/choosing-an-architecture/): where running your own Host sits relative to the other two models. + +## Need help + +{{ need_help(client="Host", repo_name="shinzo-host-client", repo="https://github.com/shinzonetwork/shinzo-host-client/issues") }} diff --git a/content/build/how-to/verify-data/index.md b/content/build/how-to/verify-data/index.md index 6023562..578fe32 100644 --- a/content/build/how-to/verify-data/index.md +++ b/content/build/how-to/verify-data/index.md @@ -3,4 +3,143 @@ title = "Verify data with signatures and CIDs" description = "How to verify who signed your data and navigate attestations, commits, and documents via CIDs." +++ -This page is coming soon. +Every document a Shinzo client serves is content-addressed and signed, so you can check where a piece of data came from instead of trusting the server that returned it. This page covers the verification queries: reading signatures off documents, tracing documents back to their attestations, and resolving CIDs to commits and documents. + +The examples query primitive collections through a Host client, and the same queries work in a local-first app's embedded DefraDB instance. Both surfaces share the query language, as described in [Query data](/build/how-to/query-data/). + +{% admonition(type="note") %} +Collection names are prefixed with `____`, derived from the `chain.name` and `chain.network` settings of the Generator client that indexed the data. Substitute the prefix that matches your chain. +{% end %} + +## Check who signed a document + +Every document carries signed commits in its `_version` field. Each entry has the commit's `cid` and a `signature` with the signer's `identity` (a public key), the signature `value`, and the signature `type`. + +```graphql +{ + ____Block(limit: 1, order: { number: DESC }) { + number + _docID + _version { + cid + signature { + identity + value + type + } + } + } +} +``` + +```json +{ + "data": { + "____Block": [ + { + "number": 23902272, + "_docID": "bae-91bd3f16-ccb1-5c35-b098-45672ee6fd48", + "_version": [ + { + "cid": "bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy", + "signature": { + "identity": "0348621aed3cb78ade074e86a3d650dfdfad0c110b274c0633b331d1b0a41ddd99", + "type": "ES256K", + "value": "MEUCIQCjfh3m0RNv4j094aW5YPEeF+GCMFWEGy0hiAcga7HKbQIgc54AV7WSdXZVyGH7jOuLcXJ6w5fDQSUdrlzgZhDkBTw=" + } + } + ] + } + ] + } +} +``` + +The `identity` is the public key of the Generator client that signed the commit. Comparing identities across documents tells you whether two pieces of data came from the same Generator client. + +## Trace a document back to its attestations + +Host clients maintain attestation records that track which Generator clients signed off on a document. Query the `AttestationRecord` collection and filter by the document you care about: + +```graphql +{ + ____AttestationRecord( + filter: { attested_doc: { _eq: "bae-91bd3f16-ccb1-5c35-b098-45672ee6fd48" } } + ) { + attested_doc + source_doc + CIDs + doc_type + vote_count + } +} +``` + +```json +{ + "data": { + "____AttestationRecord": [ + { + "attested_doc": "bae-91bd3f16-ccb1-5c35-b098-45672ee6fd48", + "source_doc": ["bae-25fb059c-f232-5305-8a5d-0162f01e43e6"], + "CIDs": ["bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy"], + "doc_type": "____Block", + "vote_count": 1 + } + ] + } +} +``` + +The fields matter for different reasons. `CIDs` links the record to the signed commits it attests to. `doc_type` names the attested collection. `vote_count` is a CRDT counter that goes up as more Generator clients are observed signing the same data, so it tells you how much independent agreement the document has. To filter query results by that count automatically, see [Configure attestation thresholds](/build/how-to/configure-attestation-thresholds/). + +## Resolve a CID to its commit or document + +A CID from `_version` or from an attestation record resolves in two directions. + +Query `_commits` for the commit-level metadata, including the signature over that exact commit: + +```graphql +{ + _commits(cid: "bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy") { + cid + docID + fieldName + schemaVersionId + signature { + type + value + identity + } + } +} +``` + +Or pass the same CID as the `cid` argument on the collection to resolve the document at that version: + +```graphql +{ + ____Transaction(cid: "bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy") { + _docID + blockNumber + hash + to + from + value + } +} +``` + +Because the CID is derived from the content, the document it resolves to is exactly the version that was signed. A Host that altered the data would produce a different CID. + +## Verify a whole block at once + +Signing every document individually would be slow, so Generator clients also sign per block. After writing a block's documents, the Generator client computes a Merkle root over their CIDs, signs the root, and writes a `BlockSignature` document. Snapshot signatures do the same across block ranges for faster initial sync. Verifying one block-level signature covers every primitive document in that block. The two-level Merkle structure is described in the [architecture reference](/reference/architecture/), and [Attestation](/understand/core-concepts/attestation/) explains how Host clients turn these signatures into attestation records. + +{% admonition(type="note") %} +Signatures and CIDs prove who produced your data and that it was not altered in transit. They do not prove completeness (that no matching documents were withheld from your result) or freshness (that you are seeing the latest state). Closing those gaps is roadmap work; see [Privacy](/understand/core-concepts/privacy/) for how Shinzo frames the remaining trust assumptions. +{% end %} + +## Need help + +{{ need_help(client="Host", repo_name="shinzo-host-client", repo="https://github.com/shinzonetwork/shinzo-host-client/issues") }} diff --git a/content/build/how-to/view-recipes/index.md b/content/build/how-to/view-recipes/index.md index 176b7db..b10d550 100644 --- a/content/build/how-to/view-recipes/index.md +++ b/content/build/how-to/view-recipes/index.md @@ -1,6 +1,629 @@ +++ title = "View recipes" +aliases = ["/views/examples", "/build/create-a-view/examples/"] description = "How to build common Views: decode event logs, filter by contract, decode multiple event types, transaction Views, and editing or rolling back a View." +++ -This page is coming soon. +Seven recipes for the Views people build most often, from a basic event decode to editing and rolling back a deployed View. Each recipe states its goal, shows the View components (query, SDL, lens), gives the Viewkit commands to build it, and ends with the GraphQL query you run against the result. + +If you have never built a View, work through [Create your first View](/build/tutorials/create-your-first-view/) first. When none of the prebuilt lenses does what you need, see [Write and test a custom lens](/build/how-to/write-a-lens/). + +## Primitive data + +Views query the primitive collections that Generator clients produce. All collection names are prefixed with `____`, derived from the Generator's `chain.name` and `chain.network` settings. Viewkit lets you use short names like `Log`, and the Host client auto-prefixes them at runtime. + +| Collection | Common fields | Typical use | +| --- | --- | --- | +| `Log` | `address`, `topics`, `data`, `transactionHash`, `blockNumber` | Event decoding (fungible tokens, NFTs, governance) | +| `Transaction` | `hash`, `from`, `to`, `value`, `blockNumber`, `status`, `gasUsed` | Transaction analytics | +| `Block` | `number`, `hash`, `timestamp`, `miner`, `gasUsed`, `gasLimit` | Block-level metadata | +| `AccessListEntry` | `address`, `storageKeys`, `blockNumber` | Access lists | + +There is no `Event` collection. Raw event data lives in `Log`, where `topics` holds indexed parameters and `data` holds non-indexed ones. A lens decodes those raw fields into structured output. For the full list of primitive collections, including `BlockSignature` and `SnapshotSignature`, see [Views for builders](/build/concepts/views-for-builders/#primitive-data-views-operate-on). + +## Decode event logs + +Goal: decode all `Transfer` events from fungible token contracts into structured records. This is the simplest useful View that includes a lens: it decodes raw log `topics` and `data` into named fields using an ABI. + +### Query + +```graphql +Log { address topics data transactionHash blockNumber transaction { hash from to } } +``` + +The query selects raw log fields plus the nested `transaction` relation. The `decode_log` lens uses `transaction.hash`, `transaction.from`, and `transaction.to` to populate the output's `hash`, `from`, and `to` fields. + +### SDL + +```graphql +type EventView @materialized(if: true) { + hash: String + from: String + to: String + blockNumber: Int + logAddress: String + event: String + signature: String + arguments: [String] +} +``` + +The `decode_log` lens outputs these fields: + +- `hash`, `from`, `to`: from the parent transaction. +- `blockNumber`: block the log was emitted in. +- `logAddress`: the contract that emitted the log. +- `event`: decoded event name (e.g. `"Transfer"`). +- `signature`: event signature (e.g. `"Transfer(address,address,uint256)"`). +- `arguments`: array of decoded parameters as JSON strings. + +`@materialized(if: true)` tells DefraDB to pre-compute and store the output. See [Choose materialized vs on-query](#choose-materialized-vs-on-query) for the tradeoff. + +### Lens + +| Lens | Purpose | Arguments | +| --- | --- | --- | +| `decode_log` | ABI-decode log events into named fields | `{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",...}]"}` | + +The `decode_log` lens takes an `abi` argument: a stringified JSON array of event definitions. For the `Transfer` event of a fungible token: + +```json +[{"type":"event","name":"Transfer","inputs":[{"type":"address","name":"from","indexed":true},{"type":"address","name":"to","indexed":true},{"type":"uint256","name":"value","indexed":false}]}] +``` + +### Commands + +1. Initialize the View: + + ```shell + viewkit view init event-view + ``` + +1. Add the query (raw log shape with the transaction relation): + + ```shell + viewkit view add query \ + "Log { address topics data transactionHash blockNumber transaction { hash from to } }" \ + --name event-view + ``` + +1. Add the SDL (output schema matching the `decode_log` output): + + ```shell + viewkit view add sdl \ + "type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ + --name event-view + ``` + +1. Attach the decode lens with the Transfer ABI: + + ```shell + viewkit view add lens \ + --label "decode-transfer" \ + --url "https://raw.githubusercontent.com/shinzonetwork/wasm-bucket/main/bucket/decode_log/decode_log.wasm" \ + --args '{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]}]"}' \ + --name event-view + ``` + +1. Inspect to confirm everything is attached: + + ```shell + viewkit view inspect event-view + ``` + +1. Test locally (optional but recommended): + + ```shell + viewkit view test event-view + ``` + +1. Deploy locally and explore in the playground: + + ```shell + viewkit view deploy event-view --target local + ``` + +### Query the result + +Once deployed, open the DefraDB Playground (URL printed in the terminal) and run: + +```graphql +{ + EventView(limit: 10, order: { blockNumber: DESC }) { + hash + from + to + blockNumber + logAddress + event + signature + arguments + } +} +``` + +This returns decoded `Transfer` events across all contracts. To narrow down to a specific token, see [Filter by contract address](#filter-by-contract-address). + +## Filter by contract address + +Goal: decode `Transfer` events from one specific contract only, such as a single token. Without a filter lens, `decode_log` processes every log on the chain. You filter the output using GraphQL queries against the `logAddress` field. + +### Query and SDL + +Same as [Decode event logs](#decode-event-logs). The query, SDL, and lens are identical. The filtering happens at query time, not at the lens level. + +### Commands + +Same as the previous recipe. Create a View named `usdc-event` with the same query, SDL, and lens. + +### Filter to one contract + +```graphql +{ + EventView( + filter: { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } + order: { blockNumber: DESC } + limit: 10 + ) { + hash + from + to + blockNumber + event + signature + arguments + } +} +``` + +The `logAddress` field contains the contract address that emitted the log. Filter on it to narrow results to one contract. + +### Filter by sender or receiver + +The `from` and `to` fields come from the parent transaction, not the event's indexed parameters. To filter by the event's `from`/`to` (the actual transfer sender and receiver), use the `arguments` field. With `decode_log_str` (which serializes `arguments` as a JSON string), you can use `_like`: + +```graphql +{ + EventView( + filter: { + _and: [ + { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } + { arguments: { _like: "%0x28C6c06298d514De0879A2640AB71F86b50Ce4E5%" } } + ] + } + ) { + hash + from + to + arguments + blockNumber + } +} +``` + +{% admonition(type="tip", title="decode_log vs decode_log_str") %} +`decode_log` outputs `arguments` as a JSON array (`[String]` in SDL). `decode_log_str` outputs it as a JSON string (`String` in SDL), which enables `_like` filtering in DefraDB queries. Use `decode_log_str` when you need to filter on decoded parameter values. The `_str` variant uses the same URL pattern but with `decode_log_str` in the path. Both lenses are listed in the [lens reference](/reference/components/lens/). +{% end %} + +## Decode multiple event types + +Goal: decode both `Transfer` and `Approval` events from a single contract in one View. Pass both event definitions in the ABI argument to `decode_log`. + +### Query + +```graphql +Log { address topics data transactionHash blockNumber transaction { hash from to } } +``` + +### SDL + +```graphql +type EventView @materialized(if: true) { + hash: String + from: String + to: String + blockNumber: Int + logAddress: String + event: String + signature: String + arguments: [String] +} +``` + +### Lens + +The ABI argument includes both `Transfer` and `Approval` event definitions. The `decode_log` lens matches each log's `topics[0]` against the event signature hash and decodes accordingly. + +```json +[ + {"type":"event","name":"Transfer","inputs":[{"type":"address","name":"from","indexed":true},{"type":"address","name":"to","indexed":true},{"type":"uint256","name":"value","indexed":false}]}, + {"type":"event","name":"Approval","inputs":[{"type":"address","name":"owner","indexed":true},{"type":"address","name":"spender","indexed":true},{"type":"uint256","name":"value","indexed":false}]} +] +``` + +### Commands + +```shell +# 1) initialize the View +viewkit view init erc20-events + +# 2) add the query +viewkit view add query \ + "Log { address topics data transactionHash blockNumber transaction { hash from to } }" \ + --name erc20-events + +# 3) add the SDL +viewkit view add sdl \ + "type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ + --name erc20-events + +# 4) attach the decode lens with both Transfer and Approval in the ABI +viewkit view add lens \ + --label "decode-erc20" \ + --url "https://raw.githubusercontent.com/shinzonetwork/wasm-bucket/main/bucket/decode_log/decode_log.wasm" \ + --args '{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]},{\"type\":\"event\",\"name\":\"Approval\",\"inputs\":[{\"type\":\"address\",\"name\":\"owner\",\"indexed\":true},{\"type\":\"address\",\"name\":\"spender\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]}]"}' \ + --name erc20-events + +# 5) inspect, test, and deploy +viewkit view inspect erc20-events +viewkit view test erc20-events +viewkit view deploy erc20-events --target local +``` + +### Transfers only + +```graphql +{ + EventView( + filter: { + _and: [ + { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } + { event: { _eq: "Transfer" } } + ] + } + limit: 10 + ) { + hash + from + to + event + arguments + blockNumber + } +} +``` + +### Approvals only + +```graphql +{ + EventView( + filter: { + _and: [ + { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } + { event: { _eq: "Approval" } } + ] + } + limit: 10 + ) { + hash + from + to + event + arguments + blockNumber + } +} +``` + +The `event` field lets you distinguish between event types in the same View collection. + +## Query transactions without a lens + +Goal: expose all transactions sent to a specific contract. This View queries `Transaction` documents directly, and needs no lens because nothing is being decoded. + +### Query + +```graphql +Transaction { hash from to value blockNumber gasUsed gasPrice } +``` + +### SDL + +```graphql +type TransactionView @materialized(if: false) { + hash: String + from: String + to: String + value: String + blockNumber: Int + gasUsed: String + gasPrice: String +} +``` + +Here we use `@materialized(if: false)`: the View is computed on query, not pre-stored. This makes sense for transaction data, which is large and queried less frequently than decoded events. See [Choose materialized vs on-query](#choose-materialized-vs-on-query) for details. + +### Lens + +None. The query and SDL are sufficient. DefraDB applies the View as a virtual projection over the `Transaction` collection. + +### Commands + +```shell +# 1) initialize the View +viewkit view init transaction-view + +# 2) add the query +viewkit view add query \ + "Transaction { hash from to value blockNumber gasUsed gasPrice }" \ + --name transaction-view + +# 3) add the SDL +viewkit view add sdl \ + "type TransactionView @materialized(if: false) { hash: String from: String to: String value: String blockNumber: Int gasUsed: String gasPrice: String }" \ + --name transaction-view + +# 4) inspect, test, and deploy +viewkit view inspect transaction-view +viewkit view test transaction-view +viewkit view deploy transaction-view --target local +``` + +### Query the result + +```graphql +{ + TransactionView( + filter: { to: { _eq: "0x68b3465833fb72A70ecDF485E0e4C7bD8665Fc45" } } + order: { blockNumber: DESC } + limit: 10 + ) { + hash + from + to + value + blockNumber + gasUsed + gasPrice + } +} +``` + +## Choose materialized vs on-query + +Goal: decide whether a View's output should be pre-computed and stored, or computed fresh on each query. + +The `@materialized` directive controls when the View's output is computed: + +| `@materialized` | When it computes | Query speed | Storage | Best for | +| --- | --- | --- | --- | --- | +| `if: true` | At write time. The Host pre-computes and stores results. | Fast. Data is already materialized. | Higher. The Host stores the output collection. | Frequently queried data (e.g. token transfers in a UI). | +| `if: false` | At query time. The Host computes on the fly. | Slower. Depends on data volume. | Lower. No pre-stored output. | Large datasets queried occasionally, or during development. | + +### Same View in two modes + +Materialized (pre-computed): + +```graphql +type EventView @materialized(if: true) { + hash: String + from: String + to: String + blockNumber: Int + logAddress: String + event: String + signature: String + arguments: [String] +} +``` + +On-query (virtual): + +```graphql +type EventView @materialized(if: false) { + hash: String + from: String + to: String + blockNumber: Int + logAddress: String + event: String + signature: String + arguments: [String] +} +``` + +### Switch modes + +To toggle materialization on an existing View, update the SDL: + +```shell +# switch to materialized +viewkit view add sdl \ + "type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ + --name event-view + +# or switch to on-query +viewkit view add sdl \ + "type EventView @materialized(if: false) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }" \ + --name event-view +``` + +Then redeploy: + +```shell +viewkit view test event-view +viewkit view deploy event-view --target local +``` + +{% admonition(type="tip") %} +Use `@materialized(if: false)` while developing and iterating on a View. Switch to `@materialized(if: true)` once the View is stable and you need fast queries in production. +{% end %} + +## Edit and roll back a View + +Goal: modify an existing View without starting from scratch. This recipe builds on the `erc20-events` View from [Decode multiple event types](#decode-multiple-event-types) and shows the full edit lifecycle: swap a lens, inspect revisions, roll back, test, and redeploy. + +### Starting point + +Assume you already have `erc20-events` deployed with: + +- Query: `Log { address topics data transactionHash blockNumber transaction { hash from to } }` +- SDL: `type EventView @materialized(if: true) { hash: String from: String to: String blockNumber: Int logAddress: String event: String signature: String arguments: [String] }` +- Lens: `decode-erc20` (`decode_log`, Transfer + Approval ABI) + +### Swap the lens + +Remove the old lens, then add a new one with an updated ABI that decodes three event types (the third `Transfer` variant uses a `tokenId` parameter instead of `value`): + +```shell +# remove the old lens +viewkit view remove lens \ + --label "decode-erc20" \ + --name erc20-events + +# add a new lens with three event types +viewkit view add lens \ + --label "decode-multi" \ + --url "https://raw.githubusercontent.com/shinzonetwork/wasm-bucket/main/bucket/decode_log/decode_log.wasm" \ + --args '{"abi":"[{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]},{\"type\":\"event\",\"name\":\"Approval\",\"inputs\":[{\"type\":\"address\",\"name\":\"owner\",\"indexed\":true},{\"type\":\"address\",\"name\":\"spender\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"value\",\"indexed\":false}]},{\"type\":\"event\",\"name\":\"Transfer\",\"inputs\":[{\"type\":\"address\",\"name\":\"from\",\"indexed\":true},{\"type\":\"address\",\"name\":\"to\",\"indexed\":true},{\"type\":\"uint256\",\"name\":\"tokenId\",\"indexed\":true}]}]"}' \ + --name erc20-events +``` + +### Inspect the revision history + +Every `add` and `remove` creates a new revision. To see the full history: + +```shell +viewkit view inspect erc20-events --verbose +``` + +This shows the current state (query, SDL, lenses) and all past revisions, each with a version number. + +### Roll back if something went wrong + +If the updated ABI doesn't work as expected, revert to the previous version: + +```shell +# roll back to the most recent previous version +viewkit view rollback erc20-events +``` + +Or roll back to a specific version: + +```shell +viewkit view rollback erc20-events --version 3 +``` + +### Test and redeploy + +```shell +# validate the updated View compiles +viewkit view test erc20-events + +# deploy locally to verify in the playground +viewkit view deploy erc20-events --target local + +# once verified, deploy to the public testnet +viewkit view deploy erc20-events --target devnet --rpc http://testnet.shinzo.network:8545/ +``` + +{% admonition(type="note") %} +The CLI's network target is called `devnet`, but pointed at `http://testnet.shinzo.network:8545/` it deploys to the public testnet. Registration is an on-chain transaction, so the wallet you deploy from needs testnet tokens from the [faucet](https://faucet.shinzo.network/). +{% end %} + +### Delete a View + +To remove a View bundle from your local machine entirely: + +```shell +viewkit view delete erc20-events +``` + +This deletes the local bundle. It does not remove a View that has already been deployed to the network. On-chain registrations are permanent. To update a deployed View, deploy a new version with the same name. + +## Query a deployed View + +Goal: run GraphQL queries against a deployed View's output collection. These examples assume the `erc20-events` View from [Decode multiple event types](#decode-multiple-event-types) is deployed and receiving data. + +### Get the latest 10 events + +```graphql +{ + EventView( + order: { blockNumber: DESC } + limit: 10 + ) { + hash + from + to + blockNumber + logAddress + event + signature + arguments + } +} +``` + +### Filter by contract and event type + +```graphql +{ + EventView( + filter: { + _and: [ + { logAddress: { _eq: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48" } } + { event: { _eq: "Transfer" } } + ] + } + limit: 10 + ) { + hash + from + to + arguments + blockNumber + } +} +``` + +### Filter by block range + +```graphql +{ + EventView( + filter: { blockNumber: { _geq: 19540000 } } + ) { + hash + from + to + event + blockNumber + } +} +``` + +### Filter by transaction hash + +```graphql +{ + EventView( + filter: { hash: { _eq: "0xabc123..." } } + ) { + hash + from + to + event + signature + arguments + blockNumber + } +} +``` + +For the full list of Viewkit commands and GraphQL filter operators, see the [Viewkit reference](/reference/components/viewkit/). For the prebuilt lens catalog and how to chain lenses, see the [lens reference](/reference/components/lens/). For more query patterns, see [Query data](/build/how-to/query-data/). For troubleshooting and common errors, see the [FAQ](/run/operations/troubleshooting/). + +## Need help + +{{ need_help(client="Viewkit", repo_name="shinzo-view-creator", repo="https://github.com/shinzonetwork/shinzo-view-creator/issues") }} diff --git a/content/build/how-to/write-a-lens/index.md b/content/build/how-to/write-a-lens/index.md index 77bb403..31e936d 100644 --- a/content/build/how-to/write-a-lens/index.md +++ b/content/build/how-to/write-a-lens/index.md @@ -1,6 +1,178 @@ +++ title = "Write and test a custom lens" -description = "How to author a WebAssembly lens with the AssemblyScript SDK, test it locally, and attach it to a View." +description = "How to author a WebAssembly lens with the Rust SDK, test it locally, and attach it to a View." +++ -This page is coming soon. +Most Views never need a custom lens. The prebuilt lenses in the wasm-bucket cover log decoding and function-call decoding, and the [lens reference](/reference/components/lens/) lists them with their arguments. Write your own only when the transform you need does not exist. + +A lens is a WebAssembly module that sits between a View's query and its output. Every Host client runs the same lens over the same documents through LensVM, so the output has to be identical everywhere. For each input document, the lens returns a transformed document, or drops the document entirely. + +## Scaffold the lens + +The wasm-bucket lenses are written in Rust against the `lens_sdk` crate, and that SDK is the path documented here. Create a new library crate: + +```toml +[package] +name = "drop_old_logs" +version = "0.1.0" +edition = "2018" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +serde_json = "1.0" +lens_sdk = "^0.7.0" +``` + +The lens below drops every log emitted before a fixed block height and passes everything else through unchanged. It shows the three things every lens has: an `alloc` export so the host runtime can hand over memory, a `transform` export that does the work, and the tagged-buffer convention (`JSON_TYPE_ID`, `EOS_TYPE_ID`, `ERROR_TYPE_ID`) that tells the runtime what came back. + +```rust +use std::collections::HashMap; +use std::error; + +use lens_sdk::option::StreamOption::{EndOfStream, None, Some}; +use lens_sdk::StreamOption; +use serde_json::Value; + +#[link(wasm_import_module = "lens")] +extern "C" { + fn next() -> *mut u8; +} + +const MIN_BLOCK: i64 = 19_000_000; + +#[no_mangle] +pub extern "C" fn alloc(size: usize) -> *mut u8 { + lens_sdk::alloc(size) +} + +#[no_mangle] +pub extern "C" fn transform() -> *mut u8 { + match try_transform() { + Ok(Some(json)) => tagged_mem(lens_sdk::JSON_TYPE_ID, &json), + Ok(None) => lens_sdk::nil_ptr(), + Ok(EndOfStream) => tagged_mem(lens_sdk::EOS_TYPE_ID, &[]), + Err(e) => tagged_mem(lens_sdk::ERROR_TYPE_ID, e.to_string().as_bytes()), + } +} + +fn try_transform() -> Result>, Box> { + let ptr = unsafe { next() }; + let doc = match lens_sdk::try_from_mem::>(ptr)? { + Some(v) => v, + None => return Ok(None), + EndOfStream => return Ok(EndOfStream), + }; + + if below_min_block(&doc) { + return Ok(None); // a nil return drops the document + } + + Ok(Some(serde_json::to_vec(&doc)?)) +} + +fn below_min_block(doc: &HashMap) -> bool { + let block_number = doc.get("blockNumber").and_then(|v| v.as_i64()).unwrap_or(0); + block_number < MIN_BLOCK +} + +// The same tagged-buffer layout the wasm-bucket lenses use. +fn tagged_mem(type_id: i8, data: &[u8]) -> *mut u8 { + let total = 1 + 4 + data.len(); + let ptr = lens_sdk::alloc(total); + unsafe { + *ptr = type_id as u8; + let len_bytes = (data.len() as u32).to_le_bytes(); + std::ptr::copy_nonoverlapping(len_bytes.as_ptr(), ptr.add(1), 4); + if !data.is_empty() { + std::ptr::copy_nonoverlapping(data.as_ptr(), ptr.add(5), data.len()); + } + } + ptr +} +``` + +Two things to notice. The `next()` function is imported from the `lens` module: the runtime calls `transform`, and `transform` pulls the next input document by calling `next()`. And dropping a document is just returning a nil pointer, which is how filters are expressed as lenses. + +Lenses that need arguments, like the `abi` parameter of `decode_log`, add a `set_param` export that deserializes a parameters struct once at load time. The [decode_log source](https://github.com/shinzonetwork/wasm-bucket/tree/main/bucket/decode_log) is the best real-world reference for that pattern. + +## Keep the logic testable + +The FFI shell around a lens is tedious to test, so keep it thin. Everything in `try_transform` after the deserialization is ordinary Rust: put the decision in a pure function like `below_min_block` and cover it with `cargo test`. + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn drops_old_logs() { + let mut doc = HashMap::new(); + doc.insert("blockNumber".to_string(), Value::from(18_000_000)); + assert!(below_min_block(&doc)); + } + + #[test] + fn keeps_new_logs() { + let mut doc = HashMap::new(); + doc.insert("blockNumber".to_string(), Value::from(20_000_000)); + assert!(!below_min_block(&doc)); + } +} +``` + +## Build the module + +Compile to WebAssembly with the `wasm32-unknown-unknown` target: + +```shell +rustup target add wasm32-unknown-unknown +cargo build --target wasm32-unknown-unknown --release +``` + +The module lands at `target/wasm32-unknown-unknown/release/drop_old_logs.wasm`. + +{% admonition(type="note") %} +Hosts run lenses through LensVM, and the runtime depends on where the lens executes: production Host clients use wazero, while Viewkit's local testing uses wasmer. Both enforce the same rules, so a lens that is deterministic in one is deterministic in the other. +{% end %} + +## Attach it to a View and test the whole thing + +A local build attaches to a View with `--path` instead of `--url`: + +```shell +viewkit view add lens \ + --label "drop-old-logs" \ + --path ./target/wasm32-unknown-unknown/release/drop_old_logs.wasm \ + --args '{}' \ + --name my-view +``` + +Then run the full-View check, which spins up a local node, applies the lens, and validates the output: + +```shell +viewkit view test my-view +viewkit view deploy my-view --target local +``` + +If the View compiles and the playground shows the filtered rows, the lens is doing its job. From here the usual cycle applies: edit, rebuild, `viewkit view test`, redeploy. + +## Respect the determinism rules + +Every Host client runs your lens over the same documents, and the results are compared. If two Hosts disagree, something is wrong, so a lens must produce exactly the same output on every run. That rules out: + +- Reading the system clock or generating random numbers. +- Making network calls. +- Depending on file system state. +- Using floating-point arithmetic, which can vary across WASM runtimes. + +The hardcoded `MIN_BLOCK` in the example is deliberate: a block height read from the document is deterministic, while "logs older than 24 hours" is not, because it depends on the clock. + +## The AssemblyScript alternative + +Rust is the preferred path for production lenses: it has the best tooling and optimization, with output around 200 to 300 KB. If you would rather write something closer to TypeScript, the AssemblyScript SDK produces noticeably smaller modules, around 73 KB, which means less overhead when Host clients download the View bundle. Both SDKs live in the [sourcenetwork/lens](https://github.com/sourcenetwork/lens) repo, and the [lens reference](/reference/components/lens/) has the details. + +## Need help + +{{ need_help(client="Viewkit", repo_name="shinzo-view-creator", repo="https://github.com/shinzonetwork/shinzo-view-creator/issues") }} diff --git a/content/build/query-data/index.md b/content/build/query-data/index.md deleted file mode 100644 index 4aa6562..0000000 --- a/content/build/query-data/index.md +++ /dev/null @@ -1,347 +0,0 @@ -+++ -title = "Query data" -aliases = ["/hosts/examples"] -description = "GraphQL query examples and patterns for querying indexed data through a Shinzo Host." -+++ - -This page lists common GraphQL query examples for indexed chain data. The examples focus on blocks, transactions, attestations, signatures, and document navigation using DocIDs and CIDs. - -{% admonition(type="note") %} -Collection names are prefixed with `____`, derived from the `chain.name` and `chain.network` settings of the Generator client that indexed the data (for example `____Block` or `Optimism__Mainnet__Block`). The examples below use the `____` placeholder. Substitute the prefix that matches your chain. See the [chain config](/run/run-a-generator/config-reference#chain) for details. -{% end %} - -## Querying a block with nested data - -Fetch a single block with nested sub-documents. - -```graphql -{ - ____Block(limit:1){ - _docID - number - timestamp - hash - nonce - difficulty - size - stateRoot - gasUsed - gasLimit - baseFeePerGas - logsBloom - uncles - sha3Uncles - receiptsRoot - parentHash - extraData - miner - difficulty - totalDifficulty - transactions{ - hash - blockHash - block_id - _docID - # ...additional fields - logs{ - blockHash - transactionHash - address - topics - data - # ...additional fields - } - accessList{ - storageKeys - address - transaction_id - # ...additional fields - } - } - } -} -``` - -## Blocks with signatures (verifiability) - -Verify who signed a block record and inspect the cryptographic metadata. - -```graphql -{ - ____Block(limit: 10, order: {number: DESC}) { - number - _docID - _version { - cid - signature { - identity - value - type - } - } - } -} -``` - -## Fetching a document by DocID - -Retrieve an exact document when you already know its `_docID`. - -```graphql -query { - ____Block(docID: ) { - _docID - number - _count(transactions:{}) - hash - transactions(order: {transactionIndex: DESC}) { - transactionIndex - _docID - } - } -} -``` - -## Attestations and document navigation - -Attestation records link documents to one or more CIDs. These CIDs can then be used to navigate to commit metadata or directly to the underlying document. - -### AttestationRecord - -```graphql -{ - ____AttestationRecord(limit:10){ - attested_doc - source_doc - CIDs - _docID - doc_type - } -} -``` - -#### Response - -```json -[..., { - "CIDs": [ - "bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy" - ], - "_docID": "bae-00000035-bd9b-5938-a55f-3a477dac226a", - "attested_doc": "bae-25fb059c-f232-5305-8a5d-0162f01e43e6", - "doc_type": "____Transaction", - "source_doc": "bae-25fb059c-f232-5305-8a5d-0162f01e43e6" -},...] -``` - -### CID to commit details - -Given a CID from an attestation record, you can query commit-level metadata and signatures. - -```graphql -{ - _commits( - cid:"bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy" - ){ - cid - docID - fieldName - schemaVersionId - signature{ - type - value - identity - } - } -} -``` - -#### Response - -```json -{ - "data": { - "_commits": [ - { - "cid": "bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy", - "docID": "bae-25fb059c-f232-5305-8a5d-0162f01e43e6", - "fieldName": "_C", - "schemaVersionId": "bafyreiagteeodcsrofk3s4fhubdi7jdzjeovhvpx4yayxkcxw2gm4zlcru", - "signature": { - "identity": "0348621aed3cb78ade074e86a3d650dfdfad0c110b274c0633b331d1b0a41ddd99", - "type": "ES256K", - "value": "MEUCIQCjfh3m0RNv4j094aW5YPEeF+GCMFWEGy0hiAcga7HKbQIgc54AV7WSdXZVyGH7jOuLcXJ6w5fDQSUdrlzgZhDkBTw=" - } - } - ] - } -} -``` - -### CID to document - -The same CID can be used to directly resolve the document itself. - -```graphql -{ - ____Transaction(cid:"bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy"){ - _docID - block_id - blockHash - blockNumber - hash - to - from - transactionIndex - value - # ... other fields - } -} -``` - -#### Response - -```json -{ - "data": { - "____Transaction": [ - { - "_docID": "bae-25fb059c-f232-5305-8a5d-0162f01e43e6", - "blockHash": "0x9ea35b3bd9e71c57617cc30394b22f607b735f2eea7a0db974cf02ad54de98fb", - "blockNumber": 23902272, - "block_id": "bae-91bd3f16-ccb1-5c35-b098-45672ee6fd48", - "from": "0x654a6BCe2C6F0aF68eAdCFEaD06bB49C398B3F98", - "hash": "0x61b79fc417ef183e1798681c59481410dd79f919d11806a6e7e77ebd0a744f78", - "to": "0x677f857da5e7C42b823655290cc40ff401e138D3", - "transactionIndex": 130, - "value": "1000000000" - } - ] - } -} -``` - -### From CID to document directly - -```graphql -{ - ____Transaction(cid:"bafyreibtbym4uht5dppohohg4wg66tdg4r253ws2i4wshc2gtwje6e25sy"){ - _docID - block_id - blockHash - blockNumber - hash - to - from - transactionIndex - value - # ... other fields - } -} -``` - -#### Response - -```json -{ - "data": { - "____Transaction": [ - { - "_docID": "bae-25fb059c-f232-5305-8a5d-0162f01e43e6", - "blockHash": "0x9ea35b3bd9e71c57617cc30394b22f607b735f2eea7a0db974cf02ad54de98fb", - "blockNumber": 23902272, - "block_id": "bae-91bd3f16-ccb1-5c35-b098-45672ee6fd48", - "from": "0x654a6BCe2C6F0aF68eAdCFEaD06bB49C398B3F98", - "hash": "0x61b79fc417ef183e1798681c59481410dd79f919d11806a6e7e77ebd0a744f78", - "to": "0x677f857da5e7C42b823655290cc40ff401e138D3", - "transactionIndex": 130, - "value": "1000000000" - } - ] - } -} -``` - -## DocID-based queries - -```graphql -{ - ____Transaction(docID:"bae-25fb059c-f232-5305-8a5d-0162f01e43e6"){ - _docID - block_id - blockHash - blockNumber - hash - to - from - transactionIndex - value - # ... other fields - } -} -``` - -#### Response - -```json -{ - "data": { - "____Transaction": [ - { - "_docID": "bae-25fb059c-f232-5305-8a5d-0162f01e43e6", - "blockHash": "0x9ea35b3bd9e71c57617cc30394b22f607b735f2eea7a0db974cf02ad54de98fb", - "blockNumber": 23902272, - "block_id": "bae-91bd3f16-ccb1-5c35-b098-45672ee6fd48", - "from": "0x654a6BCe2C6F0aF68eAdCFEaD06bB49C398B3F98", - "hash": "0x61b79fc417ef183e1798681c59481410dd79f919d11806a6e7e77ebd0a744f78", - "to": "0x677f857da5e7C42b823655290cc40ff401e138D3", - "transactionIndex": 130, - "value": "1000000000" - } - ] - } -} -``` - -## Filters ordering and limits - -### Number of transactions in a specific block - -```graphql -query { - ____Block( filter: { number: { _eq: 23901130 } } ){ - _docID - number - hash - receiptsRoot - size - gasUsed - transactions( - limit: 1, - filter: { blockNumber: { _eq: 23901130 } } - order: { transactionIndex: DESC } - ) { - transactionIndex # highest index within the block / +1 to get tx count - } - } -} -``` - -The total transaction count is `highest transactionIndex + 1`. - -## Block with transaction count - -```graphql -query { - ____Block(limit:10) { - _docID - number - hash - _count(transactions:{}) - } -} -``` - -## Need help - -{{ need_help(client="Host", repo_name="shinzo-host-client", repo="https://github.com/shinzonetwork/shinzo-host-client/issues") }} diff --git a/content/reference/components/viewkit/index.md b/content/reference/components/viewkit/index.md index 93baa8a..eeea336 100644 --- a/content/reference/components/viewkit/index.md +++ b/content/reference/components/viewkit/index.md @@ -37,14 +37,13 @@ When querying a deployed view's output collection, DefraDB supports these filter | Operator | Meaning | Example | | --- | --- | --- | -| `_eq` | Equal | `{ logAddress: { _eq: "0x..." } }` | -| `_ne` | Not equal | `{ event: { _ne: "Approval" } }` | -| `_gt` / `_gte` | Greater than / greater than or equal | `{ blockNumber: { _gte: 19540000 } }` | -| `_lt` / `_lte` | Less than / less than or equal | `{ blockNumber: { _lte: 19541000 } }` | +| `_eq` / `_neq` | Equal / not equal | `{ logAddress: { _eq: "0x..." } }` | +| `_gt` / `_geq` | Greater than / greater than or equal | `{ blockNumber: { _geq: 19540000 } }` | +| `_lt` / `_leq` | Less than / less than or equal | `{ blockNumber: { _leq: 19541000 } }` | +| `_in` / `_nin` | In / not in a list of values | `{ event: { _in: ["Transfer", "Approval"] } }` | | `_and` | Logical AND | `{ _and: [{ logAddress: { _eq: "0x..." } }, { event: { _eq: "Transfer" } }] }` | | `_or` | Logical OR | `{ _or: [{ from: { _eq: "0x..." } }, { to: { _eq: "0x..." } }] }` | -| `_like` | Substring match (strings) | `{ arguments: { _like: "%0xAddress%" } }` | -| `_any` | Any element in array matches | `{ topics: { _any: { _eq: "0xddf252..." } } }` | +| `_like` / `_ilike` | Substring match, case-sensitive / case-insensitive (strings) | `{ arguments: { _like: "%0xAddress%" } }` | ## What happens during deploy