Skip to content

[FEA] Add DECIMAL decoding for Parquet VARIANT values #23817

Description

@nartal1

Is your feature request related to a problem? Please describe.

cast_variant supports STRING, signed integers, BOOL8, FLOAT32, and FLOAT64 but it cannot decode encoded DECIMAL4/8/16 payloads. A common analytical query therefore requires CPU execution even when path extraction is already supported:

SELECT try_variant_get(v, '$.amount', 'decimal(18,2)')
FROM events;

Describe the solution you'd like

Extend cast_variant and extract_variant_field to accept cuDF DECIMAL32/64/128 targets and their target scale.

Variant decimals carry scale per value, while a cuDF decimal column has one column-wide scale. The contract must therefore define:

  • source scale/sign decoding and normalization to the requested decimal storage width and scale.
  • discarded fractional digits, rounding, and overflow when rows have different source scales.
  • a lossless source representation for downstream DECIMAL-to-STRING/integer/BOOL/FLOAT conversions.
  • null, Variant-null, type-mismatch, malformed/truncated payload and overflow behavior.
  • status values from variant_operation_status when the optional status path is requested, including TYPE_MISMATCH, MALFORMED_VARIANT, and OVERFLOW.

cuDF data_type carries width and scale, not Spark's declared precision; cudf-spark remains responsible for DECIMAL(p,s) precision and Spark TRY rules.

Describe alternatives you've considered

Decoding Variant decimal bytes in JNI or Scala duplicates the format. Converting through DOUBLE loses precision.

Additional context

  • Parent story: cuDF #22312
  • Logical type inspection #23182 identifies DECIMAL rows but does not decode them.
  • Operation status was added by PR #23560.

Definition of done:

  • C++ tests cover DECIMAL4/8/16, DECIMAL32/64/128 targets, mixed per-row source scales, zero/sign, width boundaries, rescaling/rounding, overflow, mixed rows, nulls, slices and malformed payloads.
  • The lossless source-decoding path round-trips unscaled values and scale for every supported payload width.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions