[ENH] Support AND queries for categorical variables other than sex

[alyssadai]

• Closes Update query logic and API endpoints to support categorical AND queries #589

Changes proposed in this pull request:

• Switch diagnosis, assessment, imaging modality from string to list fields in query body
• Replace pipeline_name and pipeline_version with single pipeline field that accepts a list of {"name": ..., "version": ...} objects
• Return error response when pipeline version is specified without specifying a pipeline name
• Update SPARQL query logic to generate >1 triple for the same variable when multiple values are passed
  • add custom SPARQL triple generator for >1 imaging modality or pipeline name/version (since each imaging modality / pipeline is its own instance)
• Update catalog mode query logic to only match dataset if its dataset metadata includes all specified terms for list fields

Note

The default SPARQL query is not expected to change since this PR only updates the logic when a filter is applied

Checklist

This section is for the PR reviewer

• PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF], [MODEL], [DOC]) (see our Contributing Guidelines for more info)
• PR has a label for the release changelog or skip-release (to be applied by maintainers only)
• PR links to GitHub issue with mention Closes #XXXX
• Tests pass
• Checks pass
• If the PR changes the SPARQL query template, the default Neurobagel query file has also been regenerated

For new features:

• Tests have been added

For bug fixes:

• There is at least one test that would fail under the original bug conditions.

Summary by Sourcery

Enable validated multi-value categorical filtering with AND semantics across subject and dataset queries.

New Features:

• Support AND queries across multiple diagnosis, assessment, imaging modality, and pipeline filters.
• Replace separate pipeline name and version query parameters with validated pipeline objects.
• Require pipeline names when pipeline versions are specified.

Bug Fixes:

• Ensure catalog-mode dataset matching requires all requested categorical terms.

Enhancements:

• Update SPARQL query generation to represent multiple categorical values and pipeline or acquisition instances correctly.

Tests:

• Add coverage for multi-value categorical and pipeline queries, catalog AND matching, validation errors, and empty filters.

[sourcery-ai]

Reviewer's Guide

Adds AND filtering for categorical diagnosis, assessment, imaging modality, and pipeline criteria by introducing list-based request fields, structured pipeline validation, instance-aware SPARQL generation, and all-term catalog matching, with comprehensive validation and query-generation coverage.

Sequence diagram for multi-value categorical query generation

sequenceDiagram
    participant Client
    participant API as post_subjects
    participant Model as DatasetsQueryModel
    participant Utility as create_query
    participant SPARQL

    Client->>API: POST query with list filters
    API->>Model: Validate diagnosis, assessment, image_modal, pipeline
    Model-->>API: Validated query
    API->>Utility: create_query(..., pipeline)
    Utility->>SPARQL: Generate one triple per categorical value
    Utility->>SPARQL: Generate distinct pipeline instances
    SPARQL-->>API: AND-constrained query
    API-->>Client: Matching subjects

Loading

Flow diagram for catalog AND matching

flowchart LR
    Query[Query categorical list filters] --> Match[catalog_dataset_matches_categorical_filter]
    Dataset[Dataset metadata terms] --> Match
    Match --> All{all requested terms present?}
    All -->|Yes| Include[Include dataset]
    All -->|No| Exclude[Exclude dataset]

Loading

File-Level Changes

Change	Details	Files
Expand query request models to support AND filters and structured pipeline criteria.	Convert diagnosis, assessment, and imaging modality inputs to validated lists. Replace separate pipeline name/version fields with a list of name/version objects. Reject pipeline versions that lack a corresponding name and forbid unknown pipeline fields. Update API forwarding and dataset filter selection for the new request shape.	app/api/models.py app/api/crud.py
Generate SPARQL patterns that require every requested categorical value and preserve acquisition/pipeline instance relationships.	Emit one predicate triple per diagnosis, assessment, or modality value. Generate numbered acquisition and pipeline variables for multiple nested instances. Add pipeline name/version triples using the same pipeline variable and ignore empty nested filters. Update serialization and filter-presence detection for list-valued models.	app/api/sparql_models.py app/api/utility.py
Apply AND semantics to catalog-mode categorical metadata matching.	Match datasets only when all requested terms occur in the corresponding metadata list. Retain unfiltered behavior for null and empty filters.	app/api/utility.py
Add and update tests covering the new query contract and matching behavior.	Update existing API, catalog, and SPARQL fixtures to use list fields and structured pipelines. Test multi-value phenotypic and imaging SPARQL generation, including distinct nested variables. Test pipeline validation, empty-filter handling, and catalog AND matching.	tests/test_datasets.py tests/test_query_field_validation.py tests/test_sparql.py tests/test_utility.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#589	Update the query API to accept list-valued diagnosis, assessment, and imaging modality filters, replace pipeline_name and pipeline_version with a structured pipeline list, and validate that pipeline versions require names.	✅
#589	Update SPARQL query generation and models so multiple categorical values produce AND-matching triples, including distinct acquisition and pipeline instances for multiple imaging modalities and pipelines.	✅
#589	Update catalog-mode matching so datasets are returned only when their metadata contains all requested categorical terms.	✅

Possibly linked issues

• Update query logic and API endpoints to support categorical AND queries #589: The PR directly implements the issue’s requested API, validation, SPARQL, and catalog-mode categorical AND query changes.
• Update query logic and API endpoints to support categorical AND queries #589: PR adds the issue’s categorical phenotypic SPARQL filtering and AND semantics, while extending support to imaging and pipelines.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codecov]

Codecov Report

❌ Patch coverage is 99.18699% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.09%. Comparing base (9971f37) to head (92c558d).

Files with missing lines	Patch %	Lines
app/api/utility.py	96.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #594      +/-   ##
==========================================
+ Coverage   95.20%   96.09%   +0.89%     
==========================================
  Files          29       29              
  Lines        1355     1435      +80     
  Branches       90       97       +7     
==========================================
+ Hits         1290     1379      +89     
+ Misses         36       31       -5     
+ Partials       29       25       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}