Skip to content

[avro] Optimize array in avro#8567

Open
yuzelin wants to merge 2 commits into
apache:masterfrom
yuzelin:avro-bytes-array
Open

[avro] Optimize array in avro#8567
yuzelin wants to merge 2 commits into
apache:masterfrom
yuzelin:avro-bytes-array

Conversation

@yuzelin

@yuzelin yuzelin commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Purpose

This is very similar to the #6256. This PR refactor the Array writing in avro.

I have test Array<INT>, Array<BIGINT> and Array<String>. The results are:

  1. Array<INT>, no obvious improvement:
before:
avro-array-int-compact-benchmark:                                                                    Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_avro-array-int-compact-benchmark_write-and-compact                                         14709 / 14709            271.9           3677.2       1.0X

after:
avro-array-int-compact-benchmark:                                                                    Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_avro-array-int-compact-benchmark_write-and-compact                                         14336 / 14336            279.0           3583.9       1.0X
  1. Array<BIGINT>, about 18.7% improvement:
before:
avro-array-bigint-compact-benchmark:                                                                 Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_avro-array-bigint-compact-benchmark_write-and-compact                                      22452 / 22452            178.2           5612.9       1.0X

after:
avro-array-bigint-compact-benchmark:                                                                 Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_avro-array-bigint-compact-benchmark_write-and-compact                                      18257 / 18257            219.1           4564.2       1.0X
  1. Array<STRING>, about 28.2% improvement:
before:
avro-array-string-compact-benchmark:                                                                 Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_avro-array-string-compact-benchmark_write-and-compact                                      15023 / 15023            266.3           3755.8       1.0X

after:
avro-array-string-compact-benchmark:                                                                 Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_avro-array-string-compact-benchmark_write-and-compact                                      10792 / 10792            370.7           2697.9       1.0X

Tests

AvroBytesArrayTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant