JDK17 Benchmark

Nowadays you can get your JDK from many different sources. In most cases they only differ on license and support, but there are also alternatives with real technical differences. Two of those alternatives are GraalVM and Semeru/OpenJ9. Here we’ll benchmark these against the traditional JDK available from Oracle and others.

The JDK:s used are:

  • HotSpot: JDK 17.0.1, Java HotSpot(TM) 64-Bit Server VM, 17.0.1+12-LTS-39 (The traditional JDK from Oracle.)
  • GraalVM EE: JDK 17.0.1, Java HotSpot(TM) 64-Bit Server VM, 17.0.1+12-LTS-jvmci-21.3-b05 (GraalVM is based on HotSpot/OpenJDK but has a new/different JIT compiler. Further it comes in 2 flavours – Community Edition and Enterprise Edition. This is the Enterprise Edition which is NOT FREE.)
  • OpenJ9: JDK 17.0.1, Eclipse OpenJ9 VM, openj9-0.29.1-m1 (Makes use of the OpenJDK class library, but other than that this is an alternative Java Virtual Machine from IBM. Note that this is a pre-release version downloaded from [GitHub](http://GraalVM is a Java VM and JDK based on HotSpot/OpenJDK, implemented in Java. It supports additional programming languages and execution modes, like ahead-of-time compilation of Java applications for fast startup and low memory footprint. The first production-ready version, GraalVM 19.0, was released in May 2019.[5] The most recent version is GraalVM 21.0.0, made available in January 2021. Major differentiators of GraalVM compared to the base JDK are: GraalVM Compiler, a JIT compiler for Java GraalVM Native Image, allowing the ahead-of-time compilation of Java applications Truffle Language Implementation framework and the GraalVM SDK, to implement additional programming language runtimes LLVM Runtime and JavaScript Runtime). The latest official release did not support Java 17.)

The JDK:s are evaluated by executing 3 different linear algebra operations implemented in 3 different code libraries. This is done for a number of different matrix sizes ranging from 100×100 to 1000×1000.

Operations

  • Matrix Multiplication
  • Solve Equation System
  • Singular Value Decomposition

Libraries

  • Apache Commons Math (ACM) v3.6.1
  • Efficient Java Matrix Library (EJML) v0.41
  • ojAlgo v49.2.1

Results

The charts below show the number of operations per minute on the y-axis and matrix size (square) on the x-axis.

Matrix Multiplication

Solve Equation System

Singular Value Decomposition

Traditional HotSpot

This is the reference implementation and would be the default choice in most cases. The main purpose of this benchmark is to see how the alternatives perform compared to this.

GraalVM EE

GraalVM EE tracks the Traditional HotSpot closely in most cases, but for matrix multiplication with EJML or ojAlgo it is clearly a faster alternative. Remember that this is the Enterprise Edition (EE) which is NOT FREE. Previous benchmarks have included the Community Edition (CE) as well. Those benchmarks indicated that the CE was not as fast as the EE or as the Traditional HotSpot.

If you are willing to pay for the JDK you should also investigate other commercial alternatives (like Azul Platform Prime / Zing)

If you’re limited to free alternatives then (based on previous benchmarks) GraalVM is not faster.

Previous benchmarks that include GraalVM:

Semeru/OpenJ9

OpenJ9 performs well on matrix multiplication with ojAlgo (but not as good as GraalVM EE). In all other cases it is either on par with or worse than the others. Bear in mind that this is a pre-release version. When a version with support for Java 17 is officially released the published results will be verified (and this post corrected/updated if necessary).

Previous benchmarks that include OpenJ9:

Other Alternatives?

Another alternative, with actual technical differences, is Azul Platform Prime (Zing). In Azul’s Evaluation Agreement they clearly state that you are not allowed to publish benchmark results (without their consent). That’s why it’s not included here. (They haven’t released a Java 17 version yet either.) In the past Zing has been included in various benchmarks and results published (with consent). That includes a full Java-Matrix-Benchmark execution.

Previous benchmarks that include Zing:

Benchmark Details

The tests were executed on a Google Cloud Platform Compute Engine: c2-standard-4 (4 vCPUs Intel Cascade Lake Xeon, 16 GB memory) running Debian GNU/Linux.

The benchmark code is here: https://github.com/optimatika/ojAlgo-linear-algebra-benchmark

The raw results for this benchmark (JMH console output and csv files) can be found here: https://github.com/optimatika/ojAlgo-linear-algebra-benchmark/tree/develop/results/2021/11/jdk17-benchmark