Add pagination support for generic table API by MonkeyCanCode · Pull Request #5533 · apache/polaris

MonkeyCanCode · 2026-09-17T05:20:52Z

Per ML, this PR honors the pagination support for generic table API. This is partial fix for #5311

Here is sample output:

# Without pagination

## command
(.venv) ➜  polaris git:(generic_table_pagination) polaris --profile dev generic-tables list --catalog polaris_demo --namespace DELTA_NS.PUBLIC
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE2"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE3"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE4"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE5"}

## log
2026-09-17 00:06:54,944 INFO  [org.apa.pol.ser.cat.gen.GenericTableCatalogHandler] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000043,POLARIS] [,,,] (executor-thread-1) Initializing non-federated catalog
> IDL2026-09-17 00:06:54,962 INFO  [io.qua.htt.access-log] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000043,POLARIS] [,,,] (executor-thread-1) 127.0.0.1 - root [17/Sep/2026:00:06:54 -0500] "GET /api/catalog/polaris/v1/polaris_demo/namespaces/DELTA_NS%1FPUBLIC/generic-tables HTTP/1.1" 200 304

# With pagination

## command
(.venv) ➜  polaris git:(generic_table_pagination) polaris --profile dev generic-tables list --catalog polaris_demo --namespace DELTA_NS.PUBLIC --page-size 2
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE2"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE3"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE4"}
{"namespace": ["DELTA_NS", "PUBLIC"], "name": "PEOPLE5"}

## log
> IDL2026-09-17 00:07:27,884 INFO  [org.apa.pol.ser.cat.gen.GenericTableCatalogHandler] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000045,POLARIS] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-09-17 00:07:27,911 INFO  [io.qua.htt.access-log] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000045,POLARIS] [,,,] (executor-thread-1) 127.0.0.1 - root [17/Sep/2026:00:07:27 -0500] "GET /api/catalog/polaris/v1/polaris_demo/namespaces/DELTA_NS%1FPUBLIC/generic-tables?pageToken=&pageSize=2 HTTP/1.1" 200 171
2026-09-17 00:07:27,937 INFO  [org.apa.pol.ser.cat.gen.GenericTableCatalogHandler] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000046,POLARIS] [,,,] (executor-thread-1) Initializing non-federated catalog
> IDL2026-09-17 00:07:27,957 INFO  [io.qua.htt.access-log] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000046,POLARIS] [,,,] (executor-thread-1) 127.0.0.1 - root [17/Sep/2026:00:07:27 -0500] "GET /api/catalog/polaris/v1/polaris_demo/namespaces/DELTA_NS%1FPUBLIC/generic-tables?pageToken=OikKAfqAcMSAdvqAdEBlgGnQ-_s%3D&pageSize=2 HTTP/1.1" 200 172er:quarkusDev
2026-09-17 00:07:27,980 INFO  [org.apa.pol.ser.cat.gen.GenericTableCatalogHandler] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000047,POLARIS] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-09-17 00:07:27,980 INFO  [io.qua.htt.access-log] [4cbd359a-439e-45d1-9db7-2b55790a8e80_0000000000000000047,POLARIS] [,,,] (executor-thread-1) 127.0.0.1 - root [17/Sep/2026:00:07:27 -0500] "GET /api/catalog/polaris/v1/polaris_demo/namespaces/DELTA_NS%1FPUBLIC/generic-tables?pageToken=OikKAfqAcMSAdvqAdEBlgGnU-_s%3D&pageSize=2 HTTP/1.1" 200 93

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

…ties

Copilot

🟡 Changes recommended

Address unsized CLI pagination, compatibility and disabled-pagination coverage, page-uniqueness assertions, and user-facing documentation.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds end-to-end pagination for generic-table listings across the REST service, catalog layer, integration tests, and Python CLI.

Changes:

Threads page tokens through catalog and REST layers.
Adds CLI pagination support and tests.
Adds unit/integration coverage and a changelog entry.

File summaries

File	Summary
`runtime/service/src/test/java/org/apache/polaris/service/catalog/generic/PolarisGenericTableCatalogHandlerAuthzTest.java`	Updates authorization coverage for generic-table listing.
`runtime/service/src/test/java/org/apache/polaris/service/catalog/generic/AbstractPolarisGenericTableCatalogTest.java`	Adds unit coverage for paginated catalog behavior.
`runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/PolarisGenericTableCatalog.java`	Implements metastore-backed pagination.
`runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/GenericTableCatalogHandler.java`	Handles page parameters and continuation tokens.
`runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/GenericTableCatalogAdapter.java`	Forwards pagination parameters.
`polaris-core/src/main/java/org/apache/polaris/core/catalog/GenericTableCatalog.java`	Adds the paginated catalog API.
`integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java`	Adds REST pagination integration coverage.
`integration-tests/src/main/java/org/apache/polaris/service/it/env/GenericTableApi.java`	Supports paginated REST test requests.
`client/python/tests/test_generic_tables_command.py`	Tests CLI pagination behavior.
`client/python/apache_polaris/cli/command/generic_tables.py`	Adds generic-table pagination handling.
`client/python/apache_polaris/cli/command/__init__.py`	Wires pagination options into the CLI.
`CHANGELOG.md`	Documents the pagination feature.

Review details

Suppressed comments (4)

client/python/apache_polaris/cli/command/generic_tables.py:76

paginate returns after one request when page_size is None. However, with LIST_PAGINATION_MAX_PAGE_SIZE configured, the new server handler can return a continuation token even when the client omits --page-size; this loop then discards all subsequent generic tables. Please make the shared pagination path follow next_page_token for unsized requests as well, or otherwise pass an explicit size.

            for resp in paginate(
                generic_api.list_generic_tables,
                page_size=self.page_size,
                prefix=catalog_name,
                namespace=ns_str,

integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:2966

The new integration test covers only pagination-enabled catalogs. Please add a generic-table case with LIST_PAGINATION_ENABLED=false (including an invalid token), matching testNonPaginatedListTablesViewNamespaces, because this handler now introduces a separate PageToken.build path whose disabled-feature behavior is not covered for this endpoint.

  @Test
  public void testPaginatedListGenericTables() {

integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:2988

This assertion only sums page sizes, so a broken cursor that repeats an identifier while omitting another still passes as long as it returns 30 entries. Since pagination correctness depends on stable, non-overlapping pages, collect the identifiers and assert that they equal the 30 tables created above for each page size.

          assertThat(response.getIdentifiers().size()).isLessThanOrEqualTo(pageSize);
          total += response.getIdentifiers().size();
          pageToken = response.getNextPageToken();
        } while (pageToken != null);
        assertThat(total).as("Total paginated results for pageSize = " + pageSize).isEqualTo(30);

runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/GenericTableCatalogHandler.java:112

This introduces a user-visible pagination contract for generic-table listing, but the tracked generic-table guide still only documents an unqualified “list all” request and a response with a null token (site/content/in-dev/unreleased/generic-table.md:128-146). Please update the REST guide and the generic-tables CLI section to explain page-size/page-token, continuation handling, and the LIST_PAGINATION_ENABLED behavior so clients can use the new API correctly.

    PageToken pageRequest =
        PageToken.build(pageToken, pageSize, maxPageSize(), this::shouldDecodeToken);
    Page<TableIdentifier> page = genericTableCatalog.listGenericTables(parent, pageRequest);
    return ListGenericTablesResponse.builder()
        .setIdentifiers(new LinkedHashSet<>(page.items()))
        .setNextPageToken(page.encodedResponseToken())

Files reviewed: 12/12 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+  default Page<TableIdentifier> listGenericTables(Namespace namespace, PageToken pageToken) {
+    return Page.fromItems(listGenericTables(namespace));


MonkeyCanCode · 2026-09-17T06:51:44Z

  backend, since `S3FileIO`, `GCSFileIO`, `ADLSFileIO` and `HadoopFileIO` all implement
  `DelegateFileIO`.
 - Async task retries no longer fail with a `NullPointerException` when the task entity has already been dropped by a previous attempt. Such a retry is now recognized as an already-completed task and exits cleanly, instead of exhausting all retry attempts and logging a `NullPointerException` on each one.
+- Honored pagination for generic table API.


This is not correct. --page-size is global option and it is already documented in command-line-interface.md

dimas-b

LGTM 👍 Just one comment about SPI evolution. Thanks, @MonkeyCanCode !

flyrain

+1 Thanks @MonkeyCanCode !

ayushtkn

Thanx @MonkeyCanCode for chasing this, some minor stuff in the CHANGELOG, maybe due to conflict resolution, else the code changes LGTM

ayushtkn · 2026-09-18T08:05:53Z

  scheme stripping (e.g. `s3://bucket/path` → `//bucket/path`). `//` and `///` are retained so
  scheme-root ancestors remain visible to the overlap check.

+>>>>>>> main


this maybe came in from conflict resolution

Yes. Thanks for review and this extra line is now removed.

MonkeyCanCode added 3 commits September 17, 2026 00:14

Added pagination support for generic tables API

93ed176

Added pagination support for generic tables API

c20cf4e

Undo change in runtime/defaults/src/main/resources/application.proper…

6ff0c40

…ties

Copilot AI lite review requested due to automatic review settings September 17, 2026 05:20

github-project-automation Bot added this to Basic Kanban Board Sep 17, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Sep 17, 2026

Copilot started reviewing on behalf of MonkeyCanCode September 17, 2026 05:21 View session

MonkeyCanCode requested review from dimas-b, flyrain and singhpk234 September 17, 2026 05:21

Copilot AI reviewed Sep 17, 2026

View reviewed changes

MonkeyCanCode requested a review from jbonofre September 17, 2026 06:41

dimas-b reviewed Sep 17, 2026

View reviewed changes

Comment thread polaris-core/src/main/java/org/apache/polaris/core/catalog/GenericTableCatalog.java Outdated

flyrain previously approved these changes Sep 18, 2026

View reviewed changes

github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Sep 18, 2026

MonkeyCanCode added 2 commits September 17, 2026 23:30

Fix conflict

be59868

Remove un-used listGenericTables in GenericTableCatalog.java

9a8e01d

MonkeyCanCode dismissed flyrain’s stale review via 9a8e01d September 18, 2026 04:53

ayushtkn reviewed Sep 18, 2026

View reviewed changes

Remove extra line after fixed merge conflict

7d05d32

ayushtkn approved these changes Sep 18, 2026

View reviewed changes

Navigation Menu

Add pagination support for generic table API - #5533

Add pagination support for generic table API#5533
MonkeyCanCode wants to merge 6 commits into
apache:mainfrom
MonkeyCanCode:generic_table_pagination

MonkeyCanCode commented Sep 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

MonkeyCanCode Sep 17, 2026

Uh oh!

dimas-b left a comment

Uh oh!

Uh oh!

flyrain left a comment

Uh oh!

ayushtkn left a comment

Uh oh!

ayushtkn Sep 18, 2026

Uh oh!

MonkeyCanCode Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		default Page<TableIdentifier> listGenericTables(Namespace namespace, PageToken pageToken) {
		return Page.fromItems(listGenericTables(namespace));

Conversation

MonkeyCanCode commented Sep 17, 2026

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (4)

Uh oh!

MonkeyCanCode Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

MonkeyCanCode Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants