Go: migrate control flow graph to shared CFG library #2

[owen-mc]

This PR migrates the Go control-flow graph (CFG) from its bespoke, Go-specific implementation to the shared CFG library. Broadly speaking, the commits are in these groups:

• The first commit changes the toString and getLocation predicates for many CFG classes and accepts all test changes.
• Then there are a series of commits doing preliminary work such as adjusting tests or queries.
• Core migration — Switch Go over to the shared CFG library and reimplement the language-specific pieces (the IR / node kinds) on top of it. The test changes are included in this commit, but there aren't too many because of the preceding commits.
• A commit adding a print-CFG query and a commit adding CFG consistency queries, including their outputs.
• IR clean-up — A sequence of small, focused simplifications that remove redundancy in the generated CFG nodes now that we're on the shared framework, e.g. unifying incdec-rhs/compound-rhs, folding zero-init and write nodes together, merging result-init into result-zero-init, dropping implicit slice-bound nodes, and no longer emitting CFG nodes for subexpressions of constant expressions.

I've tried to always make it so that a commit contains any test changes which it causes, so their effect can easily be seen while reviewing.

Note that additionalNodes is quite a lot bigger than in other languages, like java and C#. The reason for the disparity is architectural: Go's dataflow nodes are CFG-instruction-based (MkInstructionNode), not AST-based like Java/C#'s TExprNode. Switching to AST-keyed expr nodes (or mapping values via injects) would make Go's CFG implementation more like java and C#, but that's a larger IR redesign we'd do separately.

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[Copilot]

Pull request overview

Migrates Go control-flow, SSA, and data-flow modeling to the shared CFG framework, including schema support and updated test baselines.

Changes:

• Integrates shared CFG/basic-block infrastructure and no-return modeling.
• Adds synthesized range-element nodes with upgrade/downgrade support.
• Updates framework models, inline annotations, and generated expectations.

Show a summary per file

File	Description
go/ql/test/query-tests/Security/CWE-918/RequestForgery.expected	Updates data-flow node labels.
go/ql/test/query-tests/Security/CWE-918/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/query-tests/Security/CWE-770/UncontrolledAllocationSize.expected	Updates extraction-node labels.
go/ql/test/query-tests/Security/CWE-770/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/query-tests/Security/CWE-643/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-640/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-601/OpenUrlRedirect/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/query-tests/Security/CWE-601/BadRedirectCheck/main.go	Relocates source annotations.
go/ql/test/query-tests/Security/CWE-601/BadRedirectCheck/cves.go	Relocates source annotation.
go/ql/test/query-tests/Security/CWE-601/BadRedirectCheck/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-601/BadRedirectCheck/BadRedirectCheck.go	Relocates source annotation.
go/ql/test/query-tests/Security/CWE-347/MissingJwtSignatureCheck.expected	Updates SSA locations and labels.
go/ql/test/query-tests/Security/CWE-347/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-327/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-326/InsufficientKeySize.expected	Updates result ordering and SSA locations.
go/ql/test/query-tests/Security/CWE-322/InsecureHostKeyCallback.expected	Updates SSA and extraction nodes.
go/ql/test/query-tests/Security/CWE-312/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/query-tests/Security/CWE-295/DisabledCertificateCheck/DisabledCertificateCheck.expected	Updates assignment and literal labels.
go/ql/test/query-tests/Security/CWE-190/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-117/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-089/StringBreak.expected	Updates extraction-node labels.
go/ql/test/query-tests/Security/CWE-089/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/query-tests/Security/CWE-089/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/query-tests/Security/CWE-079/StoredXss.expected	Updates extraction and SSA nodes.
go/ql/test/query-tests/Security/CWE-079/stored.go	Relocates source annotation.
go/ql/test/query-tests/Security/CWE-079/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/query-tests/Security/CWE-078/StoredCommand.expected	Updates extraction-node labels.
go/ql/test/query-tests/Security/CWE-078/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-078/CommandInjection.expected	Updates SSA source ranges.
go/ql/test/query-tests/Security/CWE-022/UnsafeUnzipSymlink.expected	Updates SSA locations.
go/ql/test/query-tests/Security/CWE-022/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-020/MissingRegexpAnchor/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/Security/CWE-020/IncompleteHostnameRegexp/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/query-tests/RedundantCode/UnreachableStatement/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/query-tests/RedundantCode/RedundantRecover/tst.go	Removes an obsolete alert annotation.
go/ql/test/query-tests/RedundantCode/RedundantRecover/RedundantRecover.expected	Updates recover-call results.
go/ql/test/query-tests/RedundantCode/DeadStoreOfLocal/DeadStoreOfLocal.expected	Updates assignment instruction labels.
go/ql/test/query-tests/RedundantCode/DeadStoreOfLocal/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/query-tests/RedundantCode/DeadStoreOfField/DeadStoreOfField.expected	Updates assignment label.
go/ql/test/query-tests/InconsistentCode/UnhandledCloseWritableHandle/CONSISTENCY/CfgConsistency.expected	Adds defer CFG baseline.
go/ql/test/query-tests/InconsistentCode/MissingErrorCheck/MissingErrorCheck.expected	Updates SSA locations.
go/ql/test/library-tests/semmle/go/Types/notype.ql	Excludes synthesized valid types.
go/ql/test/library-tests/semmle/go/security/SafeUrlFlow/SafeUrlFlow.expected	Updates dereference labels.
go/ql/test/library-tests/semmle/go/Scopes/EntityWrite.expected	Updates parameter-init nodes.
go/ql/test/library-tests/semmle/go/PrintAst/PrintAstExcludeComments.expected	Adds range-element AST nodes.
go/ql/test/library-tests/semmle/go/PrintAst/PrintAst.expected	Adds range-element AST nodes.
go/ql/test/library-tests/semmle/go/PrintAst/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/library-tests/semmle/go/IR/test.expected	Updates extraction instruction labels.
go/ql/test/library-tests/semmle/go/frameworks/Yaml/yaml.go	Updates inline model expectations.
go/ql/test/library-tests/semmle/go/frameworks/XNetHtml/SqlInjection.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/frameworks/XNetHtml/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/library-tests/semmle/go/frameworks/WebSocket/RemoteFlowSources.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/frameworks/WebSocket/Read.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/frameworks/WebSocket/CONSISTENCY/DataFlowConsistency.expected	Updates dereference label.
go/ql/test/library-tests/semmle/go/frameworks/Twirp/server/main.go	Relocates handler/source annotations.
go/ql/test/library-tests/semmle/go/frameworks/TaintSteps/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/library-tests/semmle/go/frameworks/SystemCommandExecutors/CONSISTENCY/DataFlowConsistency.expected	Updates dereference label.
go/ql/test/library-tests/semmle/go/frameworks/Revel/test.expected	Records inline expectation mismatch.
go/ql/test/library-tests/semmle/go/frameworks/Revel/Revel.go	Relocates response-body annotation.
go/ql/test/library-tests/semmle/go/frameworks/Revel/OpenRedirect.expected	Updates dereference labels.
go/ql/test/library-tests/semmle/go/frameworks/Protobuf/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/library-tests/semmle/go/frameworks/Protobuf/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/library-tests/semmle/go/frameworks/gqlgen/graph/schema.resolvers.go	Relocates resolver annotation.
go/ql/test/library-tests/semmle/go/frameworks/GoMicro/main.go	Relocates request annotation.
go/ql/test/library-tests/semmle/go/frameworks/GoMicro/LogInjection.expected	Updates parameter SSA location.
go/ql/test/library-tests/semmle/go/frameworks/GoMicro/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/library-tests/semmle/go/frameworks/GoKit/main.go	Relocates endpoint annotations.
go/ql/test/library-tests/semmle/go/frameworks/Gin/Gin.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/frameworks/Gin/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/library-tests/semmle/go/frameworks/Fasthttp/fasthttp.go	Updates inline source expectations.
go/ql/test/library-tests/semmle/go/frameworks/Fasthttp/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/library-tests/semmle/go/frameworks/Echo/CONSISTENCY/DataFlowConsistency.expected	Updates dereference label.
go/ql/test/library-tests/semmle/go/frameworks/Chi/CONSISTENCY/DataFlowConsistency.expected	Updates dereference label.
go/ql/test/library-tests/semmle/go/frameworks/Beego/test.go	Relocates source annotation.
go/ql/test/library-tests/semmle/go/frameworks/Beego/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/library-tests/semmle/go/frameworks/Afero/CONSISTENCY/DataFlowConsistency.expected	Updates dereference label.
go/ql/test/library-tests/semmle/go/dataflow/VarArgs/CONSISTENCY/DataFlowConsistency.expected	Removes resolved consistency failure.
go/ql/test/library-tests/semmle/go/dataflow/ThreatModels/CONSISTENCY/DataFlowConsistency.expected	Updates dereference label.
go/ql/test/library-tests/semmle/go/dataflow/SSA/VarUses.expected	Updates result-read nodes.
go/ql/test/library-tests/semmle/go/dataflow/SliceExpressions/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/library-tests/semmle/go/dataflow/ReadsAndWrites/writesField.expected	Updates field-write instructions.
go/ql/test/library-tests/semmle/go/dataflow/ReadsAndWrites/writesElement.expected	Updates element-write instructions.
go/ql/test/library-tests/semmle/go/dataflow/ReadsAndWrites/readsMethod.expected	Updates implicit dereference label.
go/ql/test/library-tests/semmle/go/dataflow/ReadsAndWrites/readsField.expected	Updates implicit dereference label.
go/ql/test/library-tests/semmle/go/dataflow/ReadsAndWrites/readsElement.expected	Updates implicit dereference label.
go/ql/test/library-tests/semmle/go/dataflow/PostUpdateNodes/test.expected	Updates post-update dereference labels.
go/ql/test/library-tests/semmle/go/dataflow/Nodes/resultParameters.go	Relocates result-node annotations.
go/ql/test/library-tests/semmle/go/dataflow/Nodes/ResultNode.expected	Updates result-read nodes.
go/ql/test/library-tests/semmle/go/dataflow/Nodes/CallNode_getResult_int.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/dataflow/Nodes/BinaryOperationNodes.expected	Updates compound-assignment label.
go/ql/test/library-tests/semmle/go/dataflow/HiddenNodes/test.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/dataflow/GlobalValueNumbering/GlobalValueNumber.expected	Updates CFG instruction locations.
go/ql/test/library-tests/semmle/go/dataflow/FunctionInputsAndOutputs/FunctionOutput_isResult_int.expected	Updates result extraction labels.
go/ql/test/library-tests/semmle/go/dataflow/FunctionInputsAndOutputs/FunctionOutput_getExitNode.expected	Updates output exit nodes.
go/ql/test/library-tests/semmle/go/dataflow/FunctionInputsAndOutputs/FunctionOutput_getEntryNode.expected	Updates zero-init nodes.
go/ql/test/library-tests/semmle/go/dataflow/FunctionInputsAndOutputs/FunctionInput_getExitNode.expected	Updates parameter-init nodes.
go/ql/test/library-tests/semmle/go/dataflow/FunctionInputsAndOutputs/FunctionInput_getEntryNode.expected	Updates SSA source ranges.
go/ql/test/library-tests/semmle/go/dataflow/ExternalValueFlow/steps.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/dataflow/ExternalValueFlow/srcs.expected	Updates source-node locations.
go/ql/test/library-tests/semmle/go/dataflow/ExternalTaintFlow/steps.expected	Updates extraction-node labels.
go/ql/test/library-tests/semmle/go/dataflow/ExternalTaintFlow/srcs.expected	Updates source-node locations.
go/ql/test/library-tests/semmle/go/dataflow/DefaultTaintSanitizer/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/library-tests/semmle/go/controlflow/ControlFlowGraph/NoretFunctions.expected	Updates normal-return classification.
go/ql/test/library-tests/semmle/go/controlflow/ControlFlowGraph/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/library-tests/semmle/go/concepts/Regexp/RegexpPattern.expected	Updates extraction and SSA nodes.
go/ql/test/library-tests/semmle/go/concepts/HTTP/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/experimental/Unsafe/WrongUsageOfUnsafe.expected	Updates SSA location.
go/ql/test/experimental/Unsafe/CONSISTENCY/CfgConsistency.expected	Adds CFG consistency baseline.
go/ql/test/experimental/InconsistentCode/CONSISTENCY/CfgConsistency.expected	Adds defer-loop CFG baseline.
go/ql/test/experimental/frameworks/CleverGo/CONSISTENCY/DataFlowConsistency.expected	Updates dereference label.
go/ql/test/experimental/CWE-942/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/experimental/CWE-918/SSRF.expected	Updates dereference and extraction labels.
go/ql/test/experimental/CWE-918/CONSISTENCY/DataFlowConsistency.expected	Updates consistency output.
go/ql/test/experimental/CWE-840/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/experimental/CWE-807/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/experimental/CWE-74/DsnInjectionLocal.expected	Updates dereference labels.
go/ql/test/experimental/CWE-369/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/experimental/CWE-321-V2/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/experimental/CWE-287/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/experimental/CWE-285/PamAuthBypass.expected	Updates extraction-node label.
go/ql/test/experimental/CWE-203/CONSISTENCY/DataFlowConsistency.expected	Updates dereference labels.
go/ql/test/example-tests/snippets/varwrite.expected	Updates assignment label.
go/ql/test/example-tests/snippets/typeinfo.expected	Updates parameter-init nodes.
go/ql/test/example-tests/snippets/fieldwrite.expected	Updates assignment label.
go/ql/src/RedundantCode/UnreachableStatement.ql	Reworks unreachable-statement detection.
go/ql/src/RedundantCode/DeadStoreOfLocal.ql	Uses shared parameter initialization.
go/ql/src/experimental/IntegerOverflow/RangeAnalysis.qll	Adapts increment/decrement analysis.
go/ql/lib/upgrades/b1341734d6870b105e5c9d168ce7dec25d7f72d0/upgrade.properties	Declares range-element schema upgrade.
go/ql/lib/semmle/go/StringOps.qll	Handles omitted slice lower bounds.
go/ql/lib/semmle/go/Stmt.qll	Adds the range-element AST API.
go/ql/lib/semmle/go/Scopes.qll	Adds no-normal-return modeling hook.
go/ql/lib/semmle/go/PrintAst.qll	Makes the local overlay optional.
go/ql/lib/semmle/go/frameworks/Zap.qll	Migrates no-return model.
go/ql/lib/semmle/go/frameworks/stdlib/Os.qll	Migrates os.Exit model.
go/ql/lib/semmle/go/frameworks/stdlib/Log.qll	Migrates fatal-log model.
go/ql/lib/semmle/go/frameworks/Revel.qll	Adapts implicit field-read traversal.
go/ql/lib/semmle/go/frameworks/Logrus.qll	Migrates fatal/panic models.
go/ql/lib/semmle/go/frameworks/Glog.qll	Migrates fatal/exit models.
go/ql/lib/semmle/go/Expr.qll	Synthesizes key-value expression types.
go/ql/lib/semmle/go/dataflow/SsaImpl.qll	Connects SSA to the shared CFG.
go/ql/lib/semmle/go/dataflow/internal/TaintTrackingUtil.qll	Refines switch-edge filtering.
go/ql/lib/semmle/go/dataflow/internal/DataFlowNodes.qll	Adapts reachable and compound nodes.
go/ql/lib/semmle/go/dataflow/GlobalValueNumbering.qll	Anchors side-effect lookup to CFG entry.
go/ql/lib/semmle/go/controlflow/BasicBlocks.qll	Replaces bespoke basic blocks.
go/ql/lib/semmle/go/Concepts.qll	Migrates heuristic fatal logging model.
go/ql/lib/printCfg.ql	Adds the Go CFG viewer query.
go/ql/lib/go.dbscheme	Adds @rangeelementexpr.
go/ql/consistency-queries/CfgConsistency.ql	Enables shared CFG consistency checks.
go/extractor/extractor.go	Extracts synthesized range-element nodes.
go/extractor/dbscheme/tables.go	Registers the new expression kind.
go/downgrades/23f2f56b5d3a846b4f73e3fa62510e36f934fb46/upgrade.properties	Configures downgrade transforms.
go/downgrades/23f2f56b5d3a846b4f73e3fa62510e36f934fb46/has_location.ql	Removes synthesized-node locations.
go/downgrades/23f2f56b5d3a846b4f73e3fa62510e36f934fb46/exprs.ql	Reparents range variables on downgrade.

Review details

• Files reviewed: 191/192 changed files
• Comments generated: 3
• Review effort level: Medium

[aschackmull]

Right. The shared lib actually supports both AST setups, but it does expect just a single body AstNode when the case bodies are nested under the case clauses. But since Go appears to have a sequence of statements as the body of a case clause, then I guess this is the easiest.

[owen-mc]

I suppose we could extract a block statement to make it fit in with the shared CFG library better. But the workaround in ql is not too bad.

[owen-mc]

Thanks for the review - lots of good suggestions. I think I've addressed them all, each in its own commit. (I only force-pushed because I rebased on main - I didn't edit any commits.)

[aschackmull]

Could you comment on the consistency failures?