Green Earth API

An API server for handling bluesky content recommendation requests.

Contributing

Interested in contributing? We'd love to have you!

First, please join our discord and introduce yourself: https://discord.com/invite/8bWEyrkrJC. Unless you've joined the discord and engaged with the community there, all issues/PRs will be auto-closed.

Prerequisites

Python 3.13+
pipenv

Installation

Clone the repository
Install dependencies:
```
pipenv install
```
This installs all required packages and the greenearth-api package itself in editable mode (development install). This allows scripts in scripts/ to import from app.* without path manipulation.
Install development dependencies:
```
pipenv install --dev
```

Running the Server

Start the development server with auto-reload:

pipenv run uvicorn src.app.main:app --reload

The API will be available at http://localhost:8000.

Note: The server refuses to start without GE_ELASTICSEARCH_API_KEY and GE_FEED_CONTEXT_SECRET. Most local development queries Elasticsearch and therefore requires a valid Elasticsearch API key; a dummy key is suitable only for tests or paths that do not access Elasticsearch. The feed-context secret may use a local dummy value. See .env.example for the full configuration. In stage/prod, GE_POSTHOG_API_KEY is also required.

Running Tests

Execute all tests:

pipenv run pytest

Run tests with verbose output:

pipenv run pytest -v

Firestore (Local Development)

The API uses Cloud Firestore for persistent user data. For local development use the Firestore Emulator so you don't need a live GCP project.

Install the emulator

# Requires Node.js
npm install -g firebase-tools

The frontend repository is the sole owner of Firebase configuration, including the Firestore emulator and Emulator UI settings.

Start the emulator

cd ../frontend
npm run emulators:firestore

The Emulator UI is available at:

http://127.0.0.1:4000

Configure the API to use it

Set the following in your .env (see .env.example):

GE_FIRESTORE_EMULATOR_HOST=127.0.0.1:8080
GE_FIRESTORE_PROJECT=greenearth-471522

The API reads this on startup and routes all Firestore traffic to the emulator. No GCP credentials are needed when the emulator is active. The real project identifier selects the same local namespace used by the frontend; GE_FIRESTORE_EMULATOR_HOST ensures requests do not reach production.

Note: The emulator does not persist data across restarts by default. Add --export-on-exit / --import flags if you want persistence between sessions.

API Key Management

The API uses a Firestore-backed multi-key system. Each key is tied to an owner email and tracks monthly call count. Keys are issued manually using the scripts/apikeys.py CLI.

Key format

gea_<8-char key_id><48-char secret>

The plaintext key is shown once at generation and never stored. Authenticate requests by passing the key in the X-API-Key header.

Keys also carry an admin flag, required to access /admin/* endpoints. Existing keys default to non-admin. Only issue admin keys to the greenearth team.

CLI commands

Run all commands from the api/ directory:

# Issue a new key
pipenv run python scripts/apikeys.py generate alice@example.com

# Issue an admin key (greenearth team only)
pipenv run python scripts/apikeys.py generate alice@greenearth.social --admin

# List all keys
pipenv run python scripts/apikeys.py list

# Show usage stats for a key
pipenv run python scripts/apikeys.py usage <key_id>

# Deactivate a key
pipenv run python scripts/apikeys.py revoke <key_id>

Local development

Point the CLI at the Firestore emulator so you don't touch the real database:

GE_FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 pipenv run python scripts/apikeys.py generate test@example.com

Deployment order

Generate keys before deploying. The service reads keys from Firestore at request time. If no keys exist when the service starts, every request returns 401.

# 1. Generate a key in the target environment's Firestore
GE_FIRESTORE_PROJECT=greenearth-471522 GE_FIRESTORE_DATABASE=greenearth-stage \
  pipenv run python scripts/apikeys.py generate alice@example.com

# 2. Deploy
./scripts/deploy.sh --environment stage

# 3. Verify
curl https://<service-url>/ -H "X-API-Key: gea_..."

Repeat with GE_FIRESTORE_DATABASE=greenearth-prod and --environment prod for production.

API Documentation

Interactive API documentation is automatically generated by FastAPI:

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

Deployment

The API is deployed to Google Cloud Run using buildpacks.

Prerequisites for Deployment

gcloud CLI installed and authenticated
kubectl installed (for accessing Elasticsearch internal load balancer)
Appropriate GCP project permissions

First-Time Setup

Run the setup script once per environment to configure GCP resources:

# For staging environment (default)
./scripts/gcp_setup.sh

# For production environment
ENVIRONMENT=prod ./scripts/gcp_setup.sh

# With explicit Elasticsearch configuration
GE_ELASTICSEARCH_URL="https://custom-es:9200" \
GE_ELASTICSEARCH_API_KEY="your-api-key" \
./scripts/gcp_setup.sh

This script will:

Enable required GCP APIs (Cloud Run, Secret Manager, etc.)
Create a service account with appropriate IAM roles
Configure the Elasticsearch connection using GE_ELASTICSEARCH_URL as non-secret config and GE_ELASTICSEARCH_API_KEY as a Secret Manager secret
Create Secret Manager secrets for GE_FEED_CONTEXT_SECRET (auto-generated), GE_POSTHOG_API_KEY, GE_PERSPECTIVE_API_KEY, and GE_BSKY_APP_PASSWORD
Verify VPC connector for internal network access
Ensure the environment's Firestore database exists

Firestore rules, indexes, and TTL policies are owned by the frontend repository and applied by its stage and production deployment workflows.

After adopting frontend-owned Firestore deployment, rerun ./scripts/gcp_setup.sh once before the next frontend deployment. This grants the frontend deployment service account the index-administrator permission required for composite indexes and TTL policies. The role is project-wide, so either environment's setup is sufficient.

Before API deployment in stage/prod, run the inference setup script so domain mapping and DNS are in place for stable inference hostnames:

# stage
cd ../engagement-prediction/inference_service
GE_ENVIRONMENT=stage ./gcp_setup.sh

# prod
GE_ENVIRONMENT=prod ./gcp_setup.sh

Note: The API uses a separate readonly Elasticsearch API key (elasticsearch-api-key-readonly) that only has read access. This key is created by running scripts/k8s_recreate_api_key.sh in the ingex/ingest directory, which creates both the ingest (read/write) and API (readonly) keys.

Deploying the Service

Deploy to Cloud Run:

# Deploy to staging (default)
./scripts/deploy.sh

# Deploy to production
./scripts/deploy.sh --environment prod

# Deploy with custom configuration
./scripts/deploy.sh \
  --environment prod \
  --min-instances 2 \
  --max-instances 50

The deployment script will:

Refuse to run with a dirty working tree (see below)
Generate requirements.txt from Pipfile
Auto-detect the Elasticsearch internal load balancer IP
Build the container using Google Cloud buildpacks
Deploy to Cloud Run with proper environment variables and secrets
Stamp the deployed git sha onto the revision and the debug feed records
Preserve existing public feed descriptions while syncing other generator metadata

API deployments do not change Firebase configuration. Deploy Firebase rules, indexes, TTL policies, Functions, and Hosting from the frontend repository.

When an API release depends on new Firestore configuration, deploy that configuration to the target environment before deploying the API revision:

cd ../frontend
./scripts/deploy-firestore.sh stage  # or: ./scripts/deploy-firestore.sh prod

cd ../api
./scripts/deploy.sh                  # or: ./scripts/deploy.sh --environment prod

This ordering is required for the shared user-history cache. Its documents contain base64-encoded embedding arrays and require the user_history_cache.* indexing exemption from the frontend repository's firestore.indexes.json. If the API is deployed first, cache writes fail open: feed generation continues from Elasticsearch, but the cache cannot populate and each request repeats the uncached work.

Deployments must be from a clean tree (git sha traceability)

So we always know exactly what code is live, deploy.sh refuses to deploy with uncommitted changes. Deploying an unpushed branch is fine — only a dirty working tree is rejected. Commit or stash first, then deploy.

Each deploy stamps its short git sha in three places:

Cloud Run env var GE_GIT_SHA — the running app reports it at GET /health ({"status":"ok","git_sha":"e9f07f5"}). Use this to confirm which revision is live.
Cloud Run label git-sha=<sha> — tags the service/revision so past deployments are identifiable when picking a rollback target (gcloud run revisions list --format='value(metadata.name,metadata.labels.git-sha)').
Debug feed display names + descriptions — every internal ("debug") feed record is published as e.g. GE e2 S e9f07f5, with Built by Caterpie · e9f07f5 in the description. The public prod GreenEarth feeds are left unstamped.

Reporting a bug against a feed? Open the debug feed in Bluesky and copy the trailing sha from its name (e.g. e9f07f5) into the report — it pins the bug to the exact deployed code. You can also read the live sha off GET /health.

Inference endpoint resolution order during deploy:

Explicit GE_INFERENCE_BASE_URL / --inference-base-url (best for local overrides)
Mapped domain from GE_INFERENCE_DOMAIN (or env default)
If mapping is disabled and no base URL is provided, two_tower calls will fail

Default mapped inference domains:

stage: https://inference-stage.greenearth.social
prod: https://inference.greenearth.social

Configuration Options

You can override deployment defaults using environment variables or command-line flags:

# Using environment variables
PROJECT_ID=your-project \
  REGION=us-west1 \
  ENVIRONMENT=prod \
  GE_ELASTICSEARCH_URL=https://custom-es:9200 \
  ./scripts/deploy.sh

# Using command-line flags
./scripts/deploy.sh \
  --project-id your-project \
  --region us-west1 \
  --environment prod \
  --min-instances 2 \
  --max-instances 50

Available configuration inputs across gcp_setup.sh and deploy.sh:

PROJECT_ID - GCP project ID (default: greenearth-471522)
REGION - GCP region (default: us-east1)
ENVIRONMENT - Environment name (default: stage)
GE_ELASTICSEARCH_URL - Elasticsearch endpoint (auto-detected by deploy.sh if not set)
GE_ELASTICSEARCH_API_KEY - Elasticsearch readonly API key (accepted by gcp_setup.sh, then stored in Secret Manager for deploys)
GE_FEED_CONTEXT_SECRET - HMAC secret for feed context tokens (auto-generated by gcp_setup.sh, stored in Secret Manager)
GE_POSTHOG_API_KEY - PostHog project API key (required in stage/prod; pass to gcp_setup.sh via --posthog-api-key)
GE_PERSPECTIVE_API_KEY - Perspective API key for toxicity scoring (pass to gcp_setup.sh via --perspective-api-key)
GE_BSKY_APP_PASSWORD - Bluesky app password for feed publishing (pass to gcp_setup.sh via --bsky-app-password)
GE_INFERENCE_BASE_URL - Explicit inference endpoint override (highest priority)
GE_INFERENCE_DOMAIN - Domain-mapped inference host used when base URL override is not set
GE_ENABLE_INFERENCE_DOMAIN_MAPPING - Toggle mapped-domain resolution in deploy.sh (default: true)
API_INSTANCES_MIN - Minimum instances (default: 1)
API_INSTANCES_MAX - Maximum instances (default: 20)

For occasional local inference development while keeping stage/prod defaults:

GE_INFERENCE_BASE_URL="http://127.0.0.1:8001" ./scripts/deploy.sh --environment stage

Accessing the Deployed Service

After deployment, the script will output the service URL:

Service URL: https://greenearth-api-<hash>-<region>.a.run.app

Test the deployed service:

# Health check
curl https://greenearth-api-<hash>-<region>.a.run.app/health

# API documentation
open https://greenearth-api-<hash>-<region>.a.run.app/docs

Feed Generator (AT Protocol)

The API serves as an AT Protocol feed generator, implementing the app.bsky.feed.describeFeedGenerator and app.bsky.feed.getFeedSkeleton XRPC endpoints.

Available Feeds

See src/app/feeds.py

Testing Feeds in Development

Bluesky's AppView needs to reach your feed generator over the public internet. Use Tailscale Funnel or NGrok to expose the local server at a stable public URL.

1. Pull secrets and configure `.env`

PROJECT_ID="greenearth-471522"

# NOTE: Get these secrets from GCP.
cp .env.example .env

cat >> .env <<EOF
export GE_ELASTICSEARCH_API_KEY="$GE_ELASTICSEARCH_API_KEY"
export GE_INFERENCE_API_KEY="$GE_INFERENCE_API_KEY"
export GE_FEED_CONTEXT_SECRET="$GE_FEED_CONTEXT_SECRET"
export GE_PERSPECTIVE_API_KEY="$GE_PERSPECTIVE_API_KEY"
EOF
source .env

cat >> .env <<EOF
export GE_ELASTICSEARCH_URL="https://localhost:9200"
export GE_ELASTICSEARCH_VERIFY_SSL="false"
export GE_INFERENCE_BASE_URL="https://inference.greenearth.social"
export GE_FIRESTORE_EMULATOR_HOST="127.0.0.1:8080"
export GE_FIRESTORE_PROJECT="greenearth-471522"
export GE_BSKY_APP_PASSWORD="<your-dev-account-app-password>"
EOF
source .env

2. Start supporting services (separate terminals)

Firestore emulator (requires Java — install with brew install --cask temurin):

cd ../frontend
npm run emulators:firestore

Elasticsearch port-forward (prod ES runs on a private VPC):

gcloud container clusters get-credentials greenearth-prod-cluster \
  --region=us-east1 --project=greenearth-471522
kubectl port-forward svc/greenearth-es-internal-lb 9200:9200 -n greenearth-prod

3. Expose the API via Tailscale Funnel (or ngrok)

# If you see a "shields-up" error, run this first:
sudo tailscale set --shields-up=false

tailscale funnel 8000
# prints your stable URL, e.g. https://your-machine.tail1234.ts.net

Alternatively, use ngrok — paid accounts get a stable URL, free accounts get a random URL that changes on each restart (requiring a re-run of publish_feed.py):

ngrok http --url your-subdomain.ngrok.dev 8000  # paid, stable
ngrok http 8000                                  # free, random URL

4. Start the API

export GE_FEED_GENERATOR_DID="did:web:your-machine.tail1234.ts.net"
pipenv run uvicorn src.app.main:app --reload --port 8000

Verify it's reachable:

curl https://your-machine.tail1234.ts.net/.well-known/did.json

5. Publish feeds to your dev Bluesky account

Use a dedicated dev Bluesky account (e.g. caterpie-internal.bsky.social). Get an App Password for it and set GE_BSKY_APP_PASSWORD in .env. The --handle must include the full domain.

# Publish a single feed
pipenv run python scripts/publish_feed.py \
  --handle caterpie-internal.bsky.social \
  --feed-name unranked-your-feed \
  --environment dev \
  --app-password $GE_BSKY_APP_PASSWORD

# Or publish all feeds at once
pipenv run python scripts/publish_feed.py \
  --handle caterpie-internal.bsky.social \
  --all \
  --environment dev \
  --app-password $GE_BSKY_APP_PASSWORD

Because Tailscale Funnel gives you a stable hostname, you only need to publish once — the URL doesn't change between sessions.

Other useful publish_feed.py flags:

--delete / --delete-all — remove feed records
--list — list all published feeds under the handle
--generator-did — override GE_FEED_GENERATOR_DID
--pds — use a different PDS (default: https://bsky.social)

One-time public feed-description migration

Public feed descriptions are account-managed copy. Routine deployments preserve their current description and descriptionFacets; they do not append or recompose the attribution line. Use the dedicated migration script when that copy intentionally changes.

The current migration replaces only either legacy attribution:

Built by GreenEarth (www.greenearth.social).
Built by GreenEarth (https://www.greenearth.social).

with:

Built by Green Earth (https://www.greenearth.social).

Everything else in each existing description is retained. The script is idempotent, will not append the new text when no legacy attribution is present, and exits non-zero if a targeted record is missing or needs manual attention. It reads the appropriate Bluesky app password from GCP Secret Manager unless GE_BSKY_APP_PASSWORD or --app-password is supplied.

Preview and then apply it once in each environment:

pipenv run python scripts/update_feed_descriptions.py --environment stage --dry-run
pipenv run python scripts/update_feed_descriptions.py --environment stage

pipenv run python scripts/update_feed_descriptions.py --environment prod --dry-run
pipenv run python scripts/update_feed_descriptions.py --environment prod

Stage targets the public feed configurations published under their Caterpie rkeys; production targets your-feed, best-of-friends, and random on the GreenEarth account. Records with description facets are deliberately left for manual review because changing text would invalidate their byte offsets.

Public feed pins are managed from the pinned_post_content entries in src/app/feeds.py. Their SETTINGS links use markdown syntax, which scripts/manage_pinned_posts.py converts into Bluesky rich-text facets.

The deployment lifecycle is deliberately change-aware:

The script fingerprints the three configured messages and its managed-post schema version. The fingerprint and resolved URIs are stored on the Cloud Run revision as GE_PINNED_POST_CONFIG_SHA and GE_PINNED_POST_<FEED>_URI.
If the fingerprint matches the currently deployed revision, deploy.sh reuses its URIs without logging into Bluesky.
A changed message/link, a missing deployed state, or an intentional schema version bump runs the authenticated sync. It scans the publisher's post records for an exact text-and-link match; an existing match is reused, while changed content is published as a normal TID-keyed Bluesky post with a new URI.
./scripts/deploy.sh --sync-pinned-posts forces an authenticated verification when recovering from a deleted record. It still does not create a duplicate when an exact matching post already exists.
Previous managed posts are retained because an older Cloud Run revision or rollback may still reference them. A required pin-sync failure stops deployment before Cloud Run is changed.

Production publishes pins under greenearth-social.bsky.social; stage/dev uses caterpie-internal.bsky.social. Feed generator metadata is synchronized later in the same deployment, but existing public descriptions are preserved. Managed pinned-post publication remains part of the deployment lifecycle described above.

6. View the feed in Bluesky

Open bsky.app, log in as the dev account, and navigate to the Feeds tab to find and open the published feed.

Feed Transparency API

A web-based API (/api/feeds) surfaces pipeline observability for all users — every feed load writes lightweight pipeline metadata, and the detail endpoint returns it merged with hydrated Bluesky post data (author, media, engagement) from the public API. This is available regardless of whether debug_feeds is enabled for the account.

# List up to 100 recent feed loads for your account (within the last 24 hours)
curl http://localhost:8000/api/feeds \
  -H "Authorization: Bearer <firebase-custom-token>"

# Full detail for one feed load (snapshot + hydrated posts)
curl http://localhost:8000/api/feeds/<request-id> \
  -H "Authorization: Bearer <firebase-custom-token>"

Responses use camelCase — the frontend can consume them directly without mapping.

Authentication uses Firebase custom tokens (Authorization: Bearer <token>) rather than API keys. Token uid must be a did:plc:… — the prefix is stripped to form the Firestore document key.

Debugging Feeds (CLI)

Turn on full pipeline capture for your account (stores full candidate/ranker data):

pipenv run scripts/feed_debug.py [username].bsky.social --environment stage --enable

To see your feed loads:

pipenv run scripts/feed_debug.py [username].bsky.social --environment stage --list

To look at one:

pipenv run scripts/feed_debug.py [username].bsky.social --environment stage --show [id]

Disable when done (full capture has storage/perf cost):

pipenv run scripts/feed_debug.py [username].bsky.social --environment stage --disable

User-History Feature Cache

The two-tower generator and heavy ranker share a per-user Firestore document (user_history_cache) containing the user's recent likes and hydrated post features. Cache entries use this fixed, bounded stale-while-refresh lifecycle:

Through 10 minutes, an entry is fresh and is returned directly.
After 10 and before 30 minutes, the stale entry is returned immediately while a managed background task refreshes it from Elasticsearch.
At 30 minutes, the entry is a hard miss and the request synchronously rebuilds it rather than serving older recommendation inputs.

Stale refreshes use a 30-second transactional Firestore lease, so API instances do not duplicate Elasticsearch work. A failed refresh releases the lease and sets a 60-second retry cooldown; the stale value remains usable until its fixed hard expiry. Cold misses are still fetched synchronously because the caller needs a value, but persistence happens in the background and is drained during shutdown. Request-path cache reads fail open after 500 ms; background writes and lease releases have a separate 5-second timeout.

Lookups record user_history.cache.age_seconds and user_history.cache.lookup_count; background work records refresh_count and write_count, each labelled by outcome. Firestore native TTL uses expires_at, which is set to the fixed 30-minute maximum serving age. These policy values are code-level constants and intentionally cannot vary by deployment environment; changing them requires a code change and a new API deployment. See src/app/lib/user_history_cache.py.

Popularity Candidate Cache

Popularity candidates are identical for every user, so the API keeps one shared pool of them in Firestore (popularity_cache) instead of running its ~1.5s Elasticsearch query per request. Requests filter their own exclusions out of the pool in memory; only a background refresh touches Elasticsearch, and only one instance at a time (a transactional lease on the document). A user request never waits on a refresh — a stale pool is served while the new one is built. See src/app/lib/candidates/popularity_cache.py.

There is one document per (freshness window, video_only) combination, created on demand — real traffic typically populates two or three of them.

Variable	Default	What it controls
`GE_POPULARITY_CACHE_POOL_SIZE`	500	Candidates per pool. Must stay well above one feed's popularity allocation so heavily-excluded users still fill their slate.
`GE_POPULARITY_CACHE_TTL_SEC`	300	How long a pool is served before a refresh is triggered.
`GE_POPULARITY_CACHE_LOCAL_TTL_SEC`	30	How long an instance reuses its in-memory copy before re-reading Firestore.
`GE_POPULARITY_CACHE_LEASE_SEC`	60	Refresh lease. Must exceed one pool query plus its write.

Observability. Every lookup records candidates.popularity_cache.age_seconds (plus lookup_count / refresh_count, labelled by outcome). A pool served at 20 minutes or older also logs at ERROR — refreshes are failing, and that is the condition worth alerting on.

Firestore. The payload field is exempted from indexing in the frontend repo's firestore.indexes.json; without that exemption a deployed write of a blob this size is rejected. Deploy the exemption before the API release that starts writing it.

Analytics (PostHog)

This service and the frontend write to the same PostHog project, so every event this service emits is annotated to identify its producer. Annotations are applied centrally in src/app/lib/posthog_client.py — call sites never set them.

Property	Value here	Purpose
`surface`	`greenearth_api`	Which producer emitted the event. The frontend stamps `greenearth_web`.
`schema_version`	`1`	Version of this surface's event schema; scoped to `surface`, versioned independently of the frontend.

Conventions:

Filter on surface in any insight that should cover one producer rather than both. Event names are not namespaced, so a name collision between the two surfaces is possible and only surface separates them.
schema_version is only meaningful alongside surface — the two surfaces' version numbers are unrelated. Bump it when an existing event's properties change shape in a way that would break a saved insight.
Annotations are applied after caller-supplied properties, so an event property can never overwrite the partition key.
scripts/backfill_posthog.py stamps the same annotations, so historical API-origin events are not a gap in the partition.

User identity

distinct_id is the user's did:plc:… on both surfaces (the frontend's Firebase uid is that same DID), so a user is one PostHog person across both. Feature flags are evaluated on the DID for the same reason. The DID is deliberately the key: Bluesky handles are mutable, so keying on the handle would fork a person on every rename and detach their history.

The handle is the identifier a human reads, and rides along on every event:

Property	Purpose
`$set: {username: <handle>}`	Populates the PostHog person display name. `$set`, not `$set_once`, so a rename propagates.
`user_handle`	Event-level copy, so an insight can break down by handle without joining to the person.

Both are best-effort — when the handle can't be resolved they are omitted rather than written as null, so a transient failure never erases a handle PostHog already has. feedLoaded takes the handle from the live PLC resolution; interaction events read it from the Firestore user doc that the same request already wrote, avoiding an extra directory round-trip on a background path.

Elasticsearch Query Profiling

The API logs any ES query that exceeds GE_SLOW_ES_THRESHOLD_MS (default 500 ms) to Cloud Run as a slow_es_query line. Two scripts turn those logs into actionable profiles.

Pull slow queries from Cloud Logging

# Stage — past 3 weeks, up to 200 entries
./scripts/pull_slow_es_queries.sh --environment stage --hours 504 > /tmp/slow_queries.ndjson

# Production
./scripts/pull_slow_es_queries.sh --environment prod --hours 504 --limit 200 > /tmp/slow_queries.ndjson

Requires gcloud authenticated: gcloud auth application-default login

Replay and profile

Replay each logged query against ES with profile: true to get per-shard, per-phase timing. Run from the api/ directory:

# Dry-run: inspect query shapes without hitting ES
pipenv run python scripts/profile_es_queries.py --dry-run < /tmp/slow_queries.ndjson

# Full replay — requires ES port-forward (see Testing Feeds in Development above)
export GE_ELASTICSEARCH_URL="https://localhost:9200"
export GE_ELASTICSEARCH_API_KEY="<your-key>"
export GE_ELASTICSEARCH_VERIFY_SSL="false"

pipenv run python scripts/profile_es_queries.py --top 10 < /tmp/slow_queries.ndjson

--top N limits output to the N slowest queries sorted by logged elapsed time.

What to look for

Profile symptom	Likely cause
`vector_ops_count` > 50 k on a kNN query	`knn.filter` clause triggering brute-force HNSW scoring
`max_fetch_ms` >> `max_query_ms`	Embedding arrays included in `_source`
One shard much slower than others	Shard hot-spot / imbalanced index

Project Structure

greenearth/api/
├── src/
│   └── app/
│       ├── __init__.py
│       ├── conftest.py             # Shared test fixtures (auth bypasses)
│       ├── documents.py            # Firestore Pydantic document models
│       ├── feeds.py                # Feed config definitions
│       ├── main.py                 # FastAPI entry point
│       ├── models.py               # Pipeline request/response models
│       ├── models_feed_debug.py    # Feed-debug API response views
│       ├── security.py             # X-API-Key auth
│       ├── lib/
│       │   ├── candidates/         # Candidate generators
│       │   ├── rankers/            # Ranking models
│       │   ├── diversify.py        # MMR reranking
│       │   ├── feed_debug.py       # Per-request pipeline recorder
│       │   ├── firebase_auth.py    # Firebase token verification
│       │   ├── firestore.py        # Typed Firestore helpers
│       │   ├── post_hydration.py   # Bluesky post metadata + cache
│       │   └── ...                 # ES client, inference, metrics, etc.
│       └── routers/
│           ├── candidates.py       # POST /candidates/generate
│           ├── rank.py             # POST /rank/predict
│           ├── diversify.py        # POST /diversify
│           ├── skylight.py         # /skylight/search, /skylight/similar
│           ├── xrpc.py             # AT Protocol feed generator XRPC
│           ├── feed_debug.py       # GET /api/feeds, GET /api/feeds/{id}
│           └── health.py           # GET /health
├── scripts/
│   ├── deploy.sh                  # Cloud Run deployment
│   ├── gcp_setup.sh               # GCP environment setup
│   ├── apikeys.py                 # API key management
│   ├── feed_debug.py              # CLI debug tool
│   ├── manage_pinned_posts.py     # Change-aware public feed pin publication
│   ├── update_feed_descriptions.py # One-time public description migration
│   └── publish_feed.py            # Publish/update feed generator records
├── .gcloudignore                  # Files to exclude from deployment
├── .python-version
├── Pipfile                        # pipenv dependencies
├── Procfile                       # Process definition for buildpacks (prod)
└── README.md

Name		Name	Last commit message	Last commit date
.github		.github
assets/icons		assets/icons
monitoring		monitoring
scripts		scripts
src/app		src/app
.env.example		.env.example
.gcloudignore		.gcloudignore
.gitignore		.gitignore
.python-version		.python-version
Pipfile		Pipfile
Pipfile.lock		Pipfile.lock
Procfile		Procfile
README.md		README.md
pyproject.toml		pyproject.toml

Uh oh!

Latest commit

History

Folders and files

Repository files navigation