JEP 230: Microbenchmark SuiteJEP 230: Microbenchmark SuiteAuthorsClaes Redestad, Staffan Friberg, Aleksey ShipilevOwnerClaes RedestadTypeFeatureScopeJDKStatusClosed / DeliveredRelease12ComponentperformanceDiscussionjdk dash dev at openjdk dot java dot netEffortSDurationMReviewed byJoe Darcy, Mikael VidstedtEndorsed byMikael VidstedtCreated2014/07/17 00:07Updated2019/02/27 12:31Issue8050952Summary
Add a basic suite of microbenchmarks to the JDK source code, and make it
easy for developers to run existing microbenchmarks and create new ones.
Goals

Based on the [Java Microbenchmark Harness (JMH)][1]
Stable and tuned benchmarks, targeted for continuous performance
testing

A stable and non-moving suite after the Feature Complete milestone
of a feature release, and for non-feature releases
Support comparison to previous JDK releases for applicable tests


Simplicity

Easy to add new benchmarks
Easy to update tests as APIs and options change, are deprecated, or
are removed during development
Easy to build
Easy to find and run a benchmark


Support JMH updates
Include an initial set of around a hundred benchmarks in the suite

Non-Goals


It is not a goal to provide benchmarks for new JDK features.
Adding benchmarks for new features will be done as part of those
projects.


It is not a goal to create a complete set of benchmarks covering
everything in the JDK.  The suite will continue to be expanded over
time with newly-written benchmarks, or through efforts specifically
targeted at extending its coverage.


It is not a goal to provide a solution for handling binary
dependencies in microbenchmarks.  Support for this might be added
later.


Description
The microbenchmark suite will be co-located with the JDK source code in a
single directory and, when built, will produce a single JAR
file. Co-location will simplify adding and locating benchmarks
during development. When running benchmarks, JMH provides powerful
filtering capabilities that allow the user to run only the benchmarks
that are currently of interest. The exact location remains to be
determined.
Benchmarking generally require comparisons to an earlier build, or even
release, so the microbenchmarks must support JDK(N), for benchmarks
targeting features in the new JDK, and JDK(N-1), for benchmarks targeting
features existing in an earlier release. This means, for JDK 12, that the
structure and build scripts must support compiling benchmarks for both
JDK 12 and JDK 11. The benchmarks will further be divided by using Java
package names that describe the area of the JDK they are testing.
The following directory structure is proposed:
jdk/jdk
   .../make/test (Shared folder for Makefiles)
   .../test (Shared folder for functional tests)
      .../micro/org/openjdk/bench
         .../java (subdirectories similar to JDK packages and modules)
         .../vm (subdirectories similar to HotSpot components)
The building of the microbenchmark suite will be integrated with the
normal JDK build system. It will be a separate target that is not
executed during normal JDK builds in order to keep the build time low for
developers and others not interested in building the microbenchmark
suite. To build the microbenchmark suite the user will have to
specifically run make build-microbenchmark or similar. Additionally
running benchmarks using make test TEST="micro:regexp" will be
supported. Instructions on how to set up your local environment will be
documented in docs/testing.md|html.
The benchmarks will all depend on JMH in much the same way
that some unit tests depend on TestNG or jtreg, so while the dependency
on JMH is new there are other parts of the build that have similar
dependencies. One difference compared to jtreg is that JMH is both used
during the build and is packaged as part of the resulting JAR file.
The set of benchmarks in the microbenchmark suite will be imported
from of the JMH JDK Microbenchmarks project.[2] These constitute a set
of tuned and tested microbenchmarks that is already in use internally. An open
question is whether to migrate this standalone project in its entirety to the
co-located suite, or to keep it as a stabilization forest for more long-lived
regression tests.
Any user, however, is still expected to make sure that other parameters such
as the execution machine and the JDK are stable and comparable when
doing analysis. Benchmarks are expected, in the common case, to be able
to finish a complete run in less than a minute.  This is not a wrapper framework
for large or long-running benchmarks; the goal is to provide a suite of fast and
targeted benchmarks. In some exceptional cases benchmarks may require
longer warmup or runtime in order to achieve stable results, but that should
be avoided as much as possible. It is not a goal of the suite to act as a general
wrapper for larger workloads; the intent, rather, is to extract a critical component
or method from a larger benchmark and stress only that part as a
microbenchmark.
As part of this project a new page on
wiki.openjdk.java.net will be created to
help explain how to develop new benchmarks and describe what the
requirements are for adding a benchmark. The requirements will mandate
adherence to coding standards, reproducible performance, and clear
documentation of the benchmark and what it is measuring.
Alternatives
Keep maintaining the microbenchmark suite as a separate project[2].
Co-location simplifies adding benchmarks for new features, especially
in a world where a large fraction of all new feature development is done
in project repositories (Valhalla, Amber etc). A case that has proven
to be especially convoluted in the separate project model is to test
changes to javac itself, which then requires an explicit rebuild of the
benchmark suite using each respective JDK. Co-locating would solve
this particular use case more elegantly, while not prohibiting use pre-built
benchmark bundles for performance tracking of stable tests over longer
time periods.
Testing
The microbenchmarks will be validated by the performance team as part of
regular performance testing to ensure that only stable, tuned, and
accurate microbenchmarks are added. Evaluation and profiling will also be
done for benchmarks on a case-by-case basis to ensure that it tests
the intended functionality. All tests must be run multiple times on all applicable
platforms to ensure that they are stable.
Risks


A JDK build will depend on a version of JMH. The JDK build system
will need to be able to download or in some other way access to a
binary version of JMH, and it must be possible with reasonable effort
to update to a newer version of JMH when required. This should be an
easy-to-solve issue since we have similar dependencies for TestNG and
other libraries. The JMH dependency will only be visible for someone
trying to build the microbenchmark suite; other build targets do not
depend on JMH.


A significant increase in build time. The benchmark suite is not
required for regular developer JDK builds; it will be a separate
build target.


Potential test instability in new benchmarks. Thorough review and
testing of each added benchmark is required.


Increase in source-code repository size. Microbenchmarks are small,
as the name suggests, but there will over time hopefully be a large
set of them.  The main size will be due to resources that will need
to be handled outside of the repository.


Binary storage needs to be available to allow creation of certain
types of benchmarks.  Some benchmarks rely on external libraries or
large static resources such as XML documents and source-code files
used as part of the benchmark.  We currently have no solution for
this, but this will only limit benchmarks with this type of
dependence.  Small, non-binary resources might be allowed as part
of the microbenchmark source code.


Dependencies
The microbenchmark suite will depend on the Java Microbenchmark Harness
version 1.12 or later.
[1] http://openjdk.java.net/projects/code-tools/jmh
[2] http://openjdk.java.net/projects/code-tools/jmh-jdk-microbenchmarks/
InstallingContributingSponsoringDevelopers' GuideVulnerabilitiesJDK GA/EA BuildsMailing listsWiki
                      · IRCBylaws · CensusLegalWorkshopJEP ProcessSource codeMercurialGitHubToolsGitjtreg harnessGroups(overview)AdoptionBuildClient LibrariesCompatibility & Specification ReviewCompilerConformanceCore LibrariesGoverning BoardHotSpotIDE Tooling & SupportInternationalizationJMXMembersNetworkingPortersQualitySecurityServiceabilityVulnerabilityWebProjects(overview,
      archive)AmberBabylonCRaCCaciocavalloClosuresCode ToolsCoinCommon VM InterfaceCompiler GrammarDetroitDevelopers' GuideDevice I/ODukeGalahadGraalIcedTeaJDK 7JDK 8JDK 8 UpdatesJDK 9JDK
      (…,
       21,
       22,
       23)JDK UpdatesJavaDoc.NextJigsawKonaKullaLambdaLanaiLeydenLilliputLocale EnhancementLoomMemory Model UpdateMetropolisMission ControlMulti-Language VMNashornNew I/OOpenJFXPanamaPenrosePort: AArch32Port: AArch64Port: BSDPort: HaikuPort: Mac OS XPort: MIPSPort: MobilePort: PowerPC/AIXPort: RISC-VPort: s390xPortolaSCTPShenandoahSkaraSumatraTiered AttributionTsanType AnnotationsValhallaVeronaVisualVMWakefieldZeroZGC

        © 2024 Oracle Corporation and/or its affiliates
        Terms of Use
        ·
        
            License: GPLv2
        · Privacy
        · Trademarks
