Conversation
- Add verified metadata_as_code_dark_data.ipynb cookbook with 3-layer test verification - Add initial draft cookbooks (curated_data_products, lineage_graph_observability) and sample manuals
- Add verified curated_data_products.ipynb with 3-layer test verification - Add data contract SLA aspect and AI golden queries grounding
- Add verified lineage_graph_observability.ipynb with 3-layer test verification - Add programmatic Lineage API querying, topology visualization, and root-cause drift isolation
… and branding - Remove unused tabulate dependency and BadRequest import - Eliminate directional words and 1st-person plural pronouns - Replace em dashes with standard commas per style guide - Update Universal Catalog references to Knowledge Catalog
…a_products with style guide - Add architectural rationale for zero-copy data mesh packaging, machine-readable SLAs, and protobuf integer indexing - Explain golden queries grounding for AI agents and dual-consumer patterns - Eliminate directional words and 1st-person plural pronouns per Google Developer Style Guide - Replace em dashes with standard punctuation - Remove unused pyyaml dependency from setup
…ry data, HITL governance, and schema drift RCA - Ingest authentic e-commerce dataset from bigquery-public-data.thelook_ecommerce into 3-tier pipeline - Implement human-in-the-loop (HITL) metadata review and approval promotion - Inject upstream schema drift into users_raw and detect impact across lineage graph - Implement recursive root-cause analysis (trace_root_cause) to isolate originating fault - Add lineage explainability auditing with BigQuery dry-run syntax verification - Align with Google Developer Style Guide and pass 3-layer test suite
…uganizer IDs - Set utm_campaign=b543855766 for metadata_as_code_dark_data.ipynb - Set utm_campaign=b543859711 for curated_data_products.ipynb - Set utm_campaign=b543859650 for lineage_graph_observability.ipynb
…ing layers on live GCP - Set default LOCATION to US for seamless bigquery-public-data multi-region CTAS - Robustify provision_dataset to ensure dataset location matches target region - Use pure pandas DataFrame row-dict conversion for preview queries - Pass Layer 1 Unit Tests (16/16 tests), Layer 2 Cleanup, and Layer 3 E2E Execution on live environment
…nt automatically - Add explicit Callout Note in Cell 3 explaining bigquery-public-data US multi-region requirement - Automatically normalize LOCATION to US in Cell 4 for public data pipeline execution - Pass all 3 testing layers top-to-bottom on live GCP
…bformat v4 schema compliance
… with Gemini 3.6 Flash - Upgrades reasoning model to gemini-3.6-flash and removes deprecated sampling arguments - Aligns markdown headings with Google Developer Style Guide sentence-case conventions - Enhances fail-fast parameter validation with descriptive remediation instructions - Implements BigQuery, OpenLineage, Dataplex aspect types, and cryptographic audit certificates
…prise_governance_observability
…& Gemini - Refactor architecture to official Google Cloud Knowledge Catalog Data Products and Aspect Types - Implement deterministic in-table timestamp freshness SLA scorecard audit - Add live Knowledge Catalog metadata discovery and Gemini grounded query generation - Integrate BigQuery dry-run cost estimation and syntax safety guardrails - Streamline to 27 procedural cells with unnumbered sentence-case headings and clean English comments - Align top Open in Colab badge with GoogleCloudPlatform/knowledge-catalog main branch
…ebook and preserve WIP version - cookbooks/enterprise_governance_observability.ipynb: Update to final 3-Gate certified Level 200 notebook (BigQuery staging, negative drift falsification, HITL state machine, ACL search, Gemini 3.7 Flash, and Lineage process discovery). - cookbooks/enterprise_governance_observability_wip.ipynb: Preserve intermediate WIP notebook version. - cookbooks/enterprise_governance_observability_preview.md: Add companion markdown preview.
…ds and rebrand - Remediate AST taint and eliminate silent fallback assignments in except blocks - Enforce Google Cloud 2026 rebrand invariants and remove banned acronyms - Standardize headings to sentence-case and add DevRel UTM attribution tags - Harden teardown resilience and add bounded retry for asynchronous indexing
…tructure and standard parameter contracts
…ucture, and cleanup safety - Harmonize parameter contracts across all 3 cookbooks: standardize on PROJECT_ID = "your-project-id", REGION = "us-central1", GEMINI_LOCATION = "global", and MODEL_NAME = "gemini-3.7-flash" - Enforce atomic cell granularity: separate dependency installation, parameter configuration, and SDK client initialization - Enforce safe standalone teardown with ENABLE_CLEANUP = False boolean guard to prevent accidental resource deletion during Run All - Implement negative validation gates across cookbooks to verify contract error interception - Eliminate legacy acronyms and directional language in compliance with Google Cloud documentation standards
…bservability Colab badge - Point Colab badge URL to official GoogleCloudPlatform/knowledge-catalog repository - Add standard utm_source=devrel, utm_medium=colab_badge, and campaign parameters
… in dark data cookbook
…ode ENABLE_CLEANUP flags - Remove ENABLE_CLEANUP boolean safety flags and dead-code wrappers across all teardown cells - Trust interactive learners to execute cells on demand, enabling headless CI/CD to verify full deletion - Fortify teardown resilience using solely in locals() guards and google.api_core.exceptions.NotFound handling - Verified 3-stage idempotent cleanup (partial state, active execution, repeat run) passes with Exit Code 0
…h, and ungrounded AI logic - Fix fatal NameError: LOCATION in curated_data_products by aligning LOCATION and REGION contracts - Connect live Knowledge Catalog Aspect retrieval to consumer SLA auditing instead of relying on local memory - Ground Gemini query generation with real-time measured telemetry to eliminate ungrounded hallucinations - Deconstruct monolithic Cell 20 in enterprise_governance_observability into focused Step 3A & 3B atomic cells - Add resilient 120s polling for asynchronous BigQuery system entry harvesting to eliminate 404 race condition - Connect catalog search results dynamically to downstream prompt context, eliminating ghost searches - Transparently report Data Lineage propagation latency without synthetic mock edge injections - Enhance negative test in metadata_as_code_dark_data with visible diagnostic error traces - Verified: all 3 cookbooks pass 100% across AST linters, taint analyzers, and cleanup suites with 0 errors
…dize strictly on REGION - Remove redundant LOCATION = "us-central1" and REGION = LOCATION band-aid - Standardize all BigQuery, Dataplex, and Aspect references across Cells 5, 9, 11, 17 on REGION - Resolves RULE_STANDARD_PARAMETER_NAMES warning (0 errors, 0 warnings)
…ks, eliminate mocks, harden teardown
…search query syntax
…folding, and adopt colab-native callouts - Fix Dataplex search query syntax in enterprise governance cookbook to prevent 60s timeout - Refocus learning objectives on core Knowledge Catalog and governance architecture - Add architectural context contrasting enterprise DataScan/Audit Logs/Eventarc against tutorial procedural gates - Expand AspectType and scoped system entry update contracts with architectural diagram - Convert markdown alert syntax across all 3 cookbooks to Colab-native emoji callouts
…ounding cookbook - Implement 3-tier Knowledge Catalog retrieval stack (search_entries, lookup_entry, lookup_context) - Demonstrate semantic vs. predicate-only search and ANN Top-K vs. unbounded pagination - Implement column-level PII audit with EntryView.CUSTOM and client-side unmasked filtering - Implement context hydration under context_budget and grounded AI agent with gemini-3.7-flash - Add fail-fast parameter checks, Colab session auth, and standalone idempotent teardown - Update .gitignore with internal test receipt and preview patterns
…louts, and harden observability across 4 cookbooks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This pull request adds cookbooks for Google Cloud Knowledge Catalog. These recipes demonstrate end-to-end patterns for enterprise data governance, curated data products, and multimodal dark data extraction.
Included Cookbooks
metadata_as_code_dark_data.ipynb• Strict Pydantic schema validation
• Programmatic Knowledge Catalog custom aspect registration and tagging
• Round-trip aspect verification
curated_data_products.ipynb• Machine-readable contract SLAs
• Golden queries aspect for AI grounding
• Grounded BigQuery SQL generation.
enterprise_governance_observability.ipynb• OpenLineage lifecycle events & schema facets
• Automated data quality diagnostic gate & poisoning prevention
• Custom aspect governance.