Is your feature request related to a problem? Please describe.
cast_variant supports STRING, signed integers, BOOL8, FLOAT32, and FLOAT64 but it cannot decode encoded DECIMAL4/8/16 payloads. A common analytical query therefore requires CPU execution even when path extraction is already supported:
SELECT try_variant_get(v, '$.amount', 'decimal(18,2)')
FROM events;
Describe the solution you'd like
Extend cast_variant and extract_variant_field to accept cuDF DECIMAL32/64/128 targets and their target scale.
Variant decimals carry scale per value, while a cuDF decimal column has one column-wide scale. The contract must therefore define:
- source scale/sign decoding and normalization to the requested decimal storage width and scale.
- discarded fractional digits, rounding, and overflow when rows have different source scales.
- a lossless source representation for downstream DECIMAL-to-STRING/integer/BOOL/FLOAT conversions.
- null, Variant-null, type-mismatch, malformed/truncated payload and overflow behavior.
- status values from
variant_operation_status when the optional status path is requested, including TYPE_MISMATCH, MALFORMED_VARIANT, and OVERFLOW.
cuDF data_type carries width and scale, not Spark's declared precision; cudf-spark remains responsible for DECIMAL(p,s) precision and Spark TRY rules.
Describe alternatives you've considered
Decoding Variant decimal bytes in JNI or Scala duplicates the format. Converting through DOUBLE loses precision.
Additional context
- Parent story: cuDF #22312
- Logical type inspection #23182 identifies DECIMAL rows but does not decode them.
- Operation status was added by PR #23560.
Definition of done:
- C++ tests cover DECIMAL4/8/16, DECIMAL32/64/128 targets, mixed per-row source scales, zero/sign, width boundaries, rescaling/rounding, overflow, mixed rows, nulls, slices and malformed payloads.
- The lossless source-decoding path round-trips unscaled values and scale for every supported payload width.
Is your feature request related to a problem? Please describe.
cast_variantsupports STRING, signed integers, BOOL8, FLOAT32, and FLOAT64 but it cannot decode encoded DECIMAL4/8/16 payloads. A common analytical query therefore requires CPU execution even when path extraction is already supported:Describe the solution you'd like
Extend
cast_variantandextract_variant_fieldto accept cuDF DECIMAL32/64/128 targets and their target scale.Variant decimals carry scale per value, while a cuDF decimal column has one column-wide scale. The contract must therefore define:
variant_operation_statuswhen the optional status path is requested, includingTYPE_MISMATCH,MALFORMED_VARIANT, andOVERFLOW.cuDF
data_typecarries width and scale, not Spark's declared precision; cudf-spark remains responsible forDECIMAL(p,s)precision and Spark TRY rules.Describe alternatives you've considered
Decoding Variant decimal bytes in JNI or Scala duplicates the format. Converting through DOUBLE loses precision.
Additional context
Definition of done: