Skip to content

fix(#668): harden storage payload codecs - #672

Open
cofin wants to merge 2 commits into
mainfrom
fix/storage-payload-codecs
Open

fix(#668): harden storage payload codecs#672
cofin wants to merge 2 commits into
mainfrom
fix/storage-payload-codecs

Conversation

@cofin

@cofin cofin commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

  • decode JSONL payloads through PyArrow's native JSON reader
  • reject unsupported row/Arrow storage write-format combinations before side effects
  • document native inference and the full-payload memory boundary

Details

Exact empty JSONL payloads preserve the existing empty-table result; ordinary JSON arrays retain their existing decoder. Invalid sync and async writes now fail before serialization, encoding, backend resolution, I/O, or diagnostic telemetry.

Closes #668.
Closes #669.

Verification

  • 17 focused payload tests and full storage unit suite
  • Ruff and focused mypy
  • mypyc performance build
  • 100k-row JSONL benchmark improved from 0.189508s to 0.027407s mean

@codecov-commenter

codecov-commenter commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.29630% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 77.28%. Comparing base (a3a3012) to head (be17fdb).

Files with missing lines Patch % Lines
sqlspec/storage/pipeline.py 93.75% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #672      +/-   ##
==========================================
+ Coverage   77.26%   77.28%   +0.02%     
==========================================
  Files         475      475              
  Lines       67901    67920      +19     
  Branches     9335     9338       +3     
==========================================
+ Hits        52462    52494      +32     
+ Misses      12035    12025      -10     
+ Partials     3404     3401       -3     
Flag Coverage Δ
integration 60.79% <33.33%> (-0.02%) ⬇️
py3.10 75.60% <96.29%> (+0.02%) ⬆️
py3.11 64.52% <96.29%> (-11.07%) ⬇️
py3.12 75.61% <96.29%> (+0.03%) ⬆️
py3.13 75.62% <96.29%> (+1.38%) ⬆️
py3.14 76.49% <96.29%> (+0.03%) ⬆️
unit 64.81% <96.29%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/storage/_arrow_payload.py 93.33% <100.00%> (+34.24%) ⬆️
sqlspec/storage/_utils.py 97.43% <100.00%> (+0.29%) ⬆️
sqlspec/storage/pipeline.py 86.19% <93.75%> (+1.03%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cofin
cofin marked this pull request as ready for review August 1, 2026 22:28
@cofin
cofin force-pushed the fix/storage-payload-codecs branch from 1eae917 to be17fdb Compare August 1, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reject unsupported storage write formats instead of emitting mislabeled payloads Parse JSONL storage payloads with PyArrow's native JSON reader

2 participants