Skip to content

Fix grouped DataFrame string output and Arrow exception framing - #1253

Draft
SparkSnail wants to merge 2 commits into
mainfrom
users/shinyang/fix-string-data-frame-column
Draft

Fix grouped DataFrame string output and Arrow exception framing#1253
SparkSnail wants to merge 2 commits into
mainfrom
users/shinyang/fix-string-data-frame-column

Conversation

@SparkSnail

Copy link
Copy Markdown
Contributor

Fixes #1231

Grouped DataFrame UDFs returning StringDataFrameColumn can fail during Arrow conversion. If the failure occurs after the Arrow stream starts, the JVM may misinterpret the exception payload as Arrow data and report a misleading ByteBuffer.allocate error.

This change:

  • Adds bounded Arrow conversion for ordinary string columns, preserving nulls, empty strings, Unicode, empty results, and supported mixed primitive types.
  • Keeps grouped output schemas consistent across empty and non-empty groups.
  • Updates all four Arrow UDF output paths to prepare the first batch before starting the stream, close intact streams before reporting errors, and stop writing after a partial write failure.
  • Preserves the original .NET exception even when cleanup or diagnostic logging fails.

Validation:

  • Worker builds passed for net472, net48, and net8.0.
  • Worker unit tests: 224 passed; one Linux-only test skipped on Windows.
  • Spark 3.3.4 + Java 11 + .NET 8 E2E: 9 passed, including the issue reproduction, empty/mixed/null cases, original exception propagation, and existing four-path Arrow regressions.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@SparkSnail

Copy link
Copy Markdown
Contributor Author

/AzurePipelines run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@SparkSnail

Copy link
Copy Markdown
Contributor Author

/AzurePipelines run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]:When returning a StringDataFrameColumn from the apply method, a java.lang.IllegalArgumentException is thrown

1 participant