seo: resolve the Search Console page-indexing buckets - #108
Merged
Merged
Conversation
Search Console's 2026-09-21 page-indexing export for suedeai.org reports 13 indexed against 22 not indexed, with 10 URLs under "Discovered - currently not indexed" and 5 under "Crawled - currently not indexed". Twelve indexable pages sit outside both the header nav and the footer, so the only links pointing at them came from body copy: /about/ had two inbound pages, /voice/terms/ and /voice/privacy/ two each, /ai-voice-protection/ and /ai-likeness-protection/ three each. Weak internal linking is the usual reason Google discovers a URL and never spends crawl budget on it. Add a footer explore nav carrying all twelve, so every page on the site links to every one of them. /llms.txt, /llms-full.txt, /docs/accomplishments.md and the accomplishments JSON answer 200 but are machine-readable records, not pages. Bare, they get crawled and then filed under "Crawled - currently not indexed", a critical bucket. Send X-Robots-Tag: noindex, follow so they are classified as what they are; follow keeps the links inside them crawlable, and noindex does not stop the AI crawlers robots.txt invites from reading them. Sitemap lastmod values had drifted behind the pages: /about/ still declared 2026-08-27 although the Suede Labs to Suede AI rename rewrote it on 2026-09-15, and every page changed again here. A lastmod that says nothing changed tells Google not to recrawl, which is the opposite of what these pages need. Guard all three in verify_site.py: the footer nav is checked on every page the repository serves rather than the PAGES subset, the noindex headers against NOINDEX_ASSET_PATHS, and each sitemap <loc> must resolve to a served, non-noindexed page that is not a redirect source, with a well-formed lastmod. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBkyDpU9tcCeLydQ85iskj
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd12503f9b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
index.html is the one page that loads styles.css rather than assets/css/site.css, so the .site-footer__explore rules added with the nav never reached it. There .site-footer is a five-column grid in which .site-footer__topics and .site-footer__ecosystem each claim grid-column 1 / -1; an unstyled nav auto-places into a single cell and renders the twelve links cramped into one column. The home footer already carries .site-footer__topics with eight of the twelve, so a second nav would have duplicated them as well as broken the layout. Drop the explore nav here and add the four links topics was missing: /investors/ (the header links the #work-with-us anchor, not the page), /book-a-call/, /voice/terms/ and /voice/privacy/. All twelve keep a sitewide inbound link from all 29 other pages plus this one, and the home page needs no new CSS. Check the links rather than the markup around them, so the guard covers both footer shapes and does not pin either page to a class name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBkyDpU9tcCeLydQ85iskj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A structural pass over the 2026-09-21 Search Console page-indexing export for
suedeai.org(All known pages view: 13 indexed / 22 not indexed).The export contains counts only, no URLs. Everything below is reconstructed from the repository — the served route surface, the redirect table, the internal link graph, and git history. Where a row is inferred rather than confirmed, it says so. A second agent is pulling the per-reason URL lists from the Search Console UI; those lists should be checked against this table before anyone marks the buckets resolved.
The environment's network policy blocks outbound HTTPS to
suedeai.org, so nothing here was verified against the live site. Verification was structural, against the repo.The URL surface
Static site on Vercel,
cleanUrls: true,trailingSlash: true, canonical hosthttps://suedeai.org. 30 HTML routes served, 25 insitemap.xml, 5 correctly excluded because they carrynoindex, follow(the four/thanks/pages and/welcome-back/). 24 redirect rules invercel.json.robots.txtdisallows only/api/.Structural checks that came back clean and needed no change: no broken internal links, no sitemap entry that redirects or 404s, no redirect chains, all 72 referenced assets resolve, every page has a unique
<title>, meta description and self-referencing canonical, and no page carriesnoindexthat shouldn't.The 22, mapped
Page with redirect (6) — expected, not a defect
/the-hidden-tax…/,/how-record-labels-trap-artists…/,/the-system-was-never-built-for-you/, etc.)/voice/→/voice/support//blog/,/feed/, or a/wp-sitemap.xmlvariantThis bucket tripling 2 → 6 is not a regression — it is the 404 fix landing. #63's own commit message says Search Console had failed page-indexing validation because four old blog posts plus legacy feed and sitemap URLs still returned 404. Those URLs moved out of "Not found (404)" (3 → 1) and into "Page with redirect", which is exactly the intended trade. GSC lists this bucket as informational.
I checked all four defect classes the brief asks about and found none: no sitemap
<loc>is a redirect source, no internal link or canonical points at a redirecting URL, there are no A→B→C chains, and there is no host-level (www / .org) redirect rule in the repo. All 6 are expected-and-fine; 0 are defects.Caveat on dating: git history for this repo starts 2026-08-20, after the May baseline, so the pre-August redirect state could not be read from git. The 2 → 6 attribution rests on #63's and #74's commit messages, which are explicit.
Not found (404) (1) — not fixed, needs the URL
Nothing in the repository points at a dead path. Internal links, sitemap entries, canonicals, JSON-LD and image references were all cross-checked against the served routes — zero misses. So the 404 is a URL Google learned about externally, and I could not identify it structurally.
Two candidate families, neither actionable without the URL:
llms.txtlists TechBullion syndications with no matching redirect (…launches-ios-apps,…avalanche-build-games…,…ip-visibility-audit…,…seo-and-ai-cohort), which suggests the old site carried corresponding posts. I deliberately did not add redirects for guessed slugs — a guarded redirect table (LEGACY_REDIRECTSinverify_site.py) should not accumulate rules for URLs that may never have existed..vercelignorelanded, every tracked.md,.py,tests/,scripts/and.github/file answered 200 (audited 2026-09-11, per the file's own comment). They now 404 — correctly. If the 404 is one of these, the right action is none.Needs a human: paste the URL and this is a two-minute fix — a 301 if it is a real retired page, nothing if it is a retired internal file.
Discovered – currently not indexed (10) + Crawled – currently not indexed (5) — fixed
The header nav links 7 internal pages; the footer linked 5. That leaves 12 indexable pages reachable only from body copy — and 12 lines up closely with the 15 in these two buckets. Orphan pages are the standard cause of "Discovered".
/about//voice/privacy//voice/terms//ai-voice-protection//ai-likeness-protection//human-authenticity-layer//investors//book-a-call//sharp-excerpt//full-preview//agentic-commerce//creator-ownership//llms.txt,/llms-full.txt,/docs/accomplishments.mdX-Robots-Tag: noindex, followConfidence: medium-high on the mechanism (the orphan set is measured, the bucket assignment per URL is inferred), high on the fix being correct regardless of which page sits in which bucket.
What changed
assets/css/site.cssget a new.site-footer__explorenav, styled by extending the existing.site-footer__ecosystemrules so it inherits the footer's hairline and rhythm (CSS cache-buster bumped tov=20260921).index.htmlis the one page onstyles.css, and its footer already carried a.site-footer__topicsnav with 8 of the 12 — it gains the other 4 there instead, so no links are duplicated and the home page needs no new CSS.X-Robots-Tag: noindex, followon/llms.txt,/llms-full.txt,/docs/accomplishments.md,/content/*.jsonand/LICENSE. These answer 200 but are records, not pages.followkeeps their links crawlable, andnoindexdoes not stop the AI crawlersrobots.txtinvites from reading them.lastmodrefreshed to 2026-09-21. The values had drifted:/about/still declared2026-08-27although the Suede Labs → Suede AI rename rewrote it on 2026-09-15, and every page changes again in this PR. Alastmodthat says nothing changed tells Google not to recrawl.verify_site.py— the footer's links checked on every page the repo serves (by link, not by markup, so it covers both footer shapes without pinning either to a class name), noindex headers againstNOINDEX_ASSET_PATHS, and every sitemap<loc>required to resolve to a served, non-noindexed page that is not a redirect source, with a well-formedlastmod. Each guard was negative-tested.What I deliberately left alone
/about/(226 words),/full-preview/(253) and/book-a-call/(310) are genuinely thin, but expanding them is content work, not an indexing fix, and the brief scopes out wholesale rewriting. Worth doing separately if linking alone doesn't move them./full-preview/~/sharp-excerpt/and/ai-voice-protection/~/ai-likeness-protection/share ~0.35 and ~0.33 Jaccard vocabulary overlap. That is normal for a topic cluster, not duplication. Flagging, not acting.calendar.app.googlelink added to every footer inff80830— an external sitewide link, and it sits outside.site-footer__innerso it renders unstyled. Out of scope here.#work-with-usheader link where every other page would link/investors/. The footer now covers the page, so this is cosmetic rather than an indexing problem.site.webmanifesthas nostart_url. Minor, unrelated to these four buckets.Validation
python3 tests/verify_site.py→ PASS (18 pages + core assets).node --test tests/*.test.js→ 35/35. All 30 pages parse. Internal link graph, asset references and sitemap re-checked after the change: 0 broken links, 0 missing assets, 0 sitemap defects.Not validated: anything requiring the live site. The network policy blocks outbound HTTPS to
suedeai.org, so the redirect behaviour, the new headers and the 301 chain lengths are verified againstvercel.jsononly. Thesmoke-deployedCI job will exercise the real deployment after merge.Still needs a human or the GSC URL lists
lastmodrefresh so the declared dates still match git, per the convention seo: give each sitemap URL its real lastmod instead of one shared date #78 and Sitemap: /book last changed 2026-09-04, not 2026-09-05 #92 set.🤖 Generated with Claude Code
https://claude.ai/code/session_01EBkyDpU9tcCeLydQ85iskj