Add Spark 4.0 DataFrame row retrieval support

[SparkSnail]

Summary

Adds Spark 4.0.x support for DataFrame.Tail(int) and parameterless ToLocalIterator(), and fixes cleanup when local iteration ends early.

Changes

• Enable the Spark 4.0.x row retrieval paths while preserving Spark 2/3 version routing.
• Replace finalizer-based iterator cleanup with deterministic enumerator disposal: drain the active partition and send the stop request before closing the socket.
• Add unit and E2E coverage for both prefetch settings, empty and multiple partitions, early disposal, cleanup failures, and JVM error propagation.
• Include the new tests in the Spark 4.0.0–4.0.4 Windows/Linux CI lanes. Applicable Spark 3 tests remain included.

The cleanup fix applies to both Spark 3 and Spark 4 without changing the wire protocol.

Local validation

• Core unit tests: 193/193 passed.
• Row retrieval E2E: 8/8 passed on each Windows/Linux combination with Spark 4.0.4 and Spark 3.5.3.
• Combined Spark 4 compatibility and row retrieval tests: 17/17 passed on Windows.
• Builds, formatting, and YAML checks passed.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).