Skip to content

Commit 65c5775

Browse files
pan3793dongjoon-hyun
authored andcommitted
[SPARK-53420][BUILD] Upgrade Parquet to 1.16.0
### What changes were proposed in this pull request? Parquet Java 1.16.0 Release Notes: https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.16.0 ### Why are the changes needed? Keep Parquet update to date, benefit from upstream bugfixes and improvements. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GHA. Run TPC-DS 300G (query time in seconds), no surprise compared to Parquet 1.15.2 | query_name | parquet 1.15.2 | parquet 1.16.0 RC2 | | :--------- | :----------- | :----------- | | q01.sql | 31 | 29 | | q02.sql | 26 | 20 | | q03.sql | 9 | 9 | | q04.sql | 67 | 67 | | q05.sql | 14 | 14 | | q06.sql | 8 | 8 | | q07.sql | 5 | 5 | | q08.sql | 6 | 6 | | q09.sql | 13 | 12 | | q10.sql | 6 | 6 | | q11.sql | 31 | 31 | | q12.sql | 4 | 3 | | q13.sql | 5 | 5 | | q14a.sql | 52 | 53 | | q14b.sql | 41 | 41 | | q15.sql | 4 | 4 | | q16.sql | 16 | 16 | | q17.sql | 7 | 7 | | q18.sql | 6 | 6 | | q19.sql | 6 | 6 | | q20.sql | 3 | 3 | | q21.sql | 5 | 4 | | q22.sql | 10 | 10 | | q23a.sql | 76 | 74 | | q23b.sql | 80 | 80 | | q24a.sql | 39 | 39 | | q24b.sql | 32 | 31 | | q25.sql | 7 | 7 | | q26.sql | 4 | 3 | | q27.sql | 3 | 3 | | q28.sql | 17 | 16 | | q29.sql | 9 | 9 | | q30.sql | 9 | 9 | | q31.sql | 13 | 13 | | q32.sql | 2 | 2 | | q33.sql | 5 | 6 | | q34.sql | 5 | 5 | | q35.sql | 13 | 13 | | q36.sql | 4 | 4 | | q37.sql | 7 | 6 | | q38.sql | 13 | 12 | | q39a.sql | 7 | 7 | | q39b.sql | 7 | 6 | | q40.sql | 5 | 6 | | q41.sql | 0 | 0 | | q42.sql | 1 | 2 | | q43.sql | 4 | 4 | | q44.sql | 7 | 7 | | q45.sql | 3 | 3 | | q46.sql | 7 | 7 | | q47.sql | 11 | 12 | | q48.sql | 5 | 5 | | q49.sql | 7 | 7 | | q50.sql | 16 | 16 | | q51.sql | 15 | 14 | | q52.sql | 2 | 2 | | q53.sql | 2 | 2 | | q54.sql | 4 | 5 | | q55.sql | 1 | 2 | | q56.sql | 4 | 4 | | q57.sql | 12 | 9 | | q58.sql | 4 | 6 | | q59.sql | 14 | 10 | | q60.sql | 7 | 7 | | q61.sql | 4 | 3 | | q62.sql | 4 | 5 | | q63.sql | 2 | 2 | | q64.sql | 21 | 21 | | q65.sql | 12 | 12 | | q66.sql | 6 | 7 | | q67.sql | 46 | 45 | | q68.sql | 7 | 6 | | q69.sql | 10 | 10 | | q70.sql | 5 | 5 | | q71.sql | 6 | 6 | | q72.sql | 17 | 16 | | q73.sql | 4 | 4 | | q74.sql | 24 | 24 | | q75.sql | 24 | 21 | | q76.sql | 12 | 10 | | q77.sql | 6 | 5 | | q78.sql | 31 | 31 | | q79.sql | 5 | 5 | | q80.sql | 9 | 8 | | q81.sql | 8 | 7 | | q82.sql | 11 | 11 | | q83.sql | 4 | 4 | | q84.sql | 5 | 4 | | q85.sql | 8 | 7 | | q86.sql | 3 | 3 | | q87.sql | 14 | 13 | | q88.sql | 13 | 13 | | q89.sql | 4 | 3 | | q90.sql | 4 | 3 | | q91.sql | 3 | 4 | | q92.sql | 2 | 2 | | q93.sql | 21 | 20 | | q94.sql | 9 | 9 | | q95.sql | 62 | 61 | | q96.sql | 5 | 3 | | q97.sql | 12 | 11 | | q98.sql | 4 | 3 | | q99.sql | 6 | 6 | ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52165 from pan3793/parquet-1.16.0. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 7635204 commit 65c5775

File tree

2 files changed

+7
-7
lines changed

2 files changed

+7
-7
lines changed

dev/deps/spark-deps-hadoop-3-hive-2.3

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -240,12 +240,12 @@ orc-shims/2.2.0//orc-shims-2.2.0.jar
240240
oro/2.0.8//oro-2.0.8.jar
241241
osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
242242
paranamer/2.8.3//paranamer-2.8.3.jar
243-
parquet-column/1.15.2//parquet-column-1.15.2.jar
244-
parquet-common/1.15.2//parquet-common-1.15.2.jar
245-
parquet-encoding/1.15.2//parquet-encoding-1.15.2.jar
246-
parquet-format-structures/1.15.2//parquet-format-structures-1.15.2.jar
247-
parquet-hadoop/1.15.2//parquet-hadoop-1.15.2.jar
248-
parquet-jackson/1.15.2//parquet-jackson-1.15.2.jar
243+
parquet-column/1.16.0//parquet-column-1.16.0.jar
244+
parquet-common/1.16.0//parquet-common-1.16.0.jar
245+
parquet-encoding/1.16.0//parquet-encoding-1.16.0.jar
246+
parquet-format-structures/1.16.0//parquet-format-structures-1.16.0.jar
247+
parquet-hadoop/1.16.0//parquet-hadoop-1.16.0.jar
248+
parquet-jackson/1.16.0//parquet-jackson-1.16.0.jar
249249
pickle/1.5//pickle-1.5.jar
250250
py4j/0.10.9.9//py4j-0.10.9.9.jar
251251
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar

pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,7 @@
140140
<kafka.version>3.9.1</kafka.version>
141141
<!-- After 10.17.1.0, the minimum required version is JDK19 -->
142142
<derby.version>10.16.1.1</derby.version>
143-
<parquet.version>1.15.2</parquet.version>
143+
<parquet.version>1.16.0</parquet.version>
144144
<orc.version>2.2.0</orc.version>
145145
<orc.classifier>shaded-protobuf</orc.classifier>
146146
<jetty.version>11.0.25</jetty.version>

0 commit comments

Comments
 (0)