From d7544b21c0cae57807939c2a9940ef7243d1ba44 Mon Sep 17 00:00:00 2001 From: Erwan Viollet Date: Tue, 3 Feb 2026 11:52:33 +0100 Subject: [PATCH 1/9] Profiling - Refactor java documentation - Simplify documentation to only include onboarding steps - Move internal details to the troubleshooting section - Move GraalVM specifics to a dedicated page --- content/en/profiler/enabling/graalvm.md | 89 ++++++ content/en/profiler/enabling/java.md | 289 +++--------------- .../profiler/profiler_troubleshooting/java.md | 142 +++++++++ .../profiling/profiling-languages.html | 8 + 4 files changed, 276 insertions(+), 252 deletions(-) create mode 100644 content/en/profiler/enabling/graalvm.md diff --git a/content/en/profiler/enabling/graalvm.md b/content/en/profiler/enabling/graalvm.md new file mode 100644 index 00000000000..084a33fb7dd --- /dev/null +++ b/content/en/profiler/enabling/graalvm.md @@ -0,0 +1,89 @@ +--- +title: Enabling the Profiler for GraalVM Native Image +code_lang: graalvm +type: multi-code-lang +code_lang_weight: 15 +further_reading: + - link: 'getting_started/profiler' + tag: 'Documentation' + text: 'Getting Started with Profiler' + - link: 'profiler/profile_visualizations' + tag: 'Documentation' + text: 'Learn more about available profile visualizations' + - link: 'profiler/profiler_troubleshooting/java' + tag: 'Documentation' + text: 'Fix problems you encounter while using the profiler' +--- + +
+Datadog Profiler support for GraalVM native-image is in beta. Datadog recommends evaluating the profiler in a non-sensitive environment before deploying in production. +
+ +This page describes how to enable profiling for applications compiled as GraalVM native images. For standard JVM applications, see [Enabling the Java Profiler][1]. + +## Requirements + +For a summary of the minimum and recommended runtime and tracer versions across all languages, read [Supported Language and Tracer Versions][2]. + +Minimum version +: GraalVM 17+ + +Supported platforms +: Linux, Windows, macOS + +Available profile types +: CPU, Allocations + +## Installation + +### 1. Build your native image with the Datadog tracer + +Follow the [Tracer Setup Instructions][3] to build your GraalVM native image with the Datadog Java Profiler instrumentation. + +### 2. Run with profiling enabled + +After the service binary is built, enable the profiler using environment variables: + +```shell +export DD_SERVICE= +export DD_ENV= +export DD_VERSION= +export DD_PROFILING_ENABLED=true +export DD_PROFILING_DIRECTALLOCATION_ENABLED=true +./my_service +``` + +### 3. Verify profiles are collected + +After a couple of minutes, your profiles appear on the [Datadog APM > Profiler page][4]. + +## Configuration + +| Environment variable | Description | +|-----------------------------------|--------------------------------------------------------------------------| +| `DD_PROFILING_ENABLED` | Set to `true` to enable the profiler. | +| `DD_PROFILING_DIRECTALLOCATION_ENABLED` | Set to `true` to enable allocation profiling. | +| `DD_SERVICE` | The [service][5] name. | +| `DD_ENV` | The [environment][5] name. | +| `DD_VERSION` | The [version][5] of your service. | + +## Limitations + +- Only JFR-based profiling is supported for GraalVM native-image applications. +- Wallclock and live heap profiling are not available. + +## Not sure what to do next? + +The [Getting Started with Profiler][6] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. + +## Further Reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /profiler/enabling/java/ +[2]: /profiler/enabling/supported_versions/ +[3]: /tracing/trace_collection/compatibility/java/?tab=graalvm#setup +[4]: https://app.datadoghq.com/profiling +[5]: /getting_started/tagging/unified_service_tagging +[6]: /getting_started/profiler/ + diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index 6490ffdbf32..cd4acb47af8 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -21,69 +21,40 @@ The profiler is shipped within Datadog tracing libraries. If you are already usi ## Requirements -For a summary of the minimum and recommended runtime and tracer versions across all languages, read [Supported Language and Tracer Versions][13]. - -As of dd-trace-java 1.0.0, you have two options for the engine that generates profile data for Java applications: [Java Flight Recorder (JFR)][2] or the Datadog Profiler. As of dd-trace-java 1.7.0, Datadog Profiler is the default. Each profiler engine has different side effects, requirements, available configurations, and limitations, and this page describes each. You can enable either one or both engines. Enabling both captures both profile types at the same time. +For a summary of the minimum and recommended runtime and tracer versions across all languages, read [Supported Language and Tracer Versions][2]. {{< tabs >}} -{{% tab "Datadog Profiler" %}} - -Supported operating systems: -- Linux - -Minimum JDK versions: -- OpenJDK 8u352+, 11.0.17+, 17.0.5+, 21+ (including builds on top of it: Amazon Corretto, Azul Zulu, and others) -- Oracle JDK 8u351+, 11.0.17+, 17.0.5+, 21+ -- OpenJ9 JDK 8u372+, 11.0.18+, 17.0.6+ (used on Eclipse OpenJ9, IBM JDK, IBM Semeru Runtime). - -**Note:** The profiler is disabled by default for OpenJ9 due to the possibility of crashing JVM caused by a subtle bug in JVTMI implementation. If you are **not** experiencing any crashes, you can enable the profiler by adding `-Ddd.profiling.ddprof.enabled=true` or `DD_PROFILING_DDPROF_ENABLED=true`. -- Azul Platform Prime 23.05.0.0+ (formerly Azul Zing) +{{% tab "Linux" %}} +**Minimum JDK versions**: OpenJDK 8u352+, 11.0.17+, 17.0.5+, or 21+ -**Note:** The Datadog Profiler is disabled on the GraalVM compiler (JVMCI) and needs to be enabled explicitly with `-Ddd.profiling.ddprof.enabled=true` or `DD_PROFILING_DDPROF_ENABLED=true`. +**Available profile types**: +- CPU +- Wallclock (latency) +- Allocations +- Live heap (memory leak detection) -The Datadog Profiler uses the JVMTI `AsyncGetCallTrace` function, in which there is a [known issue][1] prior to JDK release 17.0.5. This fix was backported to 11.0.17 and 8u352. The Datadog Profiler is not enabled unless the JVM the profiler is deployed into has this fix. Upgrade to at least 8u352, 11.0.17, 17.0.5, or the latest non-LTS JVM version to use the Datadog Profiler. - -[1]: https://bugs.openjdk.org/browse/JDK-8283849 {{% /tab %}} +{{% tab "Windows / macOS" %}} -{{% tab "JFR" %}} - -Supported operating systems: -- Linux -- Windows - -Minimum JDK versions: -- OpenJDK [1.8.0.262/8u262+][3], 11+ (including builds on top of it: Amazon Corretto, and others) -- Oracle JDK 11+ (Enabling the JFR may require a commercial license from Oracle. Reach out to your Oracle representative to confirm whether this is part of your license) -- Azul Zulu 8 (version 1.8.0.212/8u212+), 11+ -- GraalVM 17+ - both, JIT and AOT (native-image) versions - -Because non-LTS JDK versions may not contain stability and performance fixes related to the Datadog Profiler library, use versions 8, 11, and 17 of the Long Term Support JDK. +**Minimum JDK versions**: OpenJDK 8u262+, 11+ -Additional requirements for profiling [Trace to Profiling integration][12]: - - OpenJDK 17.0.5+ and `dd-trace-java` version 1.17.0+ - - OpenJDK 11.0.17+ and `dd-trace-java` version 1.17.0+ - - OpenJDK 8 8u352+ and `dd-trace-java` version 1.17.0+ - - OpenJ9 17.0.6+ and `dd-trace-java` version 1.17.0+ - - OpenJ9 11.0.18+ and `dd-trace-java` version 1.17.0+ - - OpenJ9 8.0.362+ and `dd-trace-java` version 1.17.0+ - -[3]: /profiler/profiler_troubleshooting/java/#java-8-support -[12]: /profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces +**Available profile types**: +- CPU +- Allocations {{% /tab %}} {{< /tabs >}} All JVM-based languages, such as Java, Scala, Groovy, Kotlin, and Clojure are supported. -Continuous Profiler is not supported on some serverless platforms, such as AWS Lambda. +Continuous Profiler is not supported on serverless platforms such as AWS Lambda. ## Installation To begin profiling applications: -1. Ensure Datadog Agent v6+ is installed and running. Datadog recommends using [Datadog Agent v7+][4]. If you don't have APM enabled to set up your application to send data to Datadog, in your Agent, set the `DD_APM_ENABLED` environment variable to `true` and listening to the port `8126/TCP`. +1. Ensure Datadog Agent v6+ is installed and running. Datadog recommends using [Datadog Agent v7+][3]. If you don't have APM enabled to set up your application to send data to Datadog, in your Agent, set the `DD_APM_ENABLED` environment variable to `true` and listening to the port `8126/TCP`. 2. Download `dd-java-agent.jar`, which contains the Java Agent class files: @@ -108,11 +79,11 @@ To begin profiling applications: **Note**: Profiler is available in the `dd-java-agent.jar` library in versions 0.55+. 3. Enable the profiler by setting `-Ddd.profiling.enabled` flag or `DD_PROFILING_ENABLED` environment variable to `true`. Specify `dd.service`, `dd.env`, and `dd.version` so you can filter and group your profiles across these dimensions: + {{< tabs >}} {{% tab "Command arguments" %}} -Invoke your service: -```diff +```shell java \ -javaagent:dd-java-agent.jar \ -Ddd.service= \ @@ -125,7 +96,7 @@ java \ {{% /tab %}} {{% tab "Environment variables" %}} -```diff +```shell export DD_SERVICE= export DD_ENV= export DD_VERSION= @@ -138,194 +109,11 @@ java \ {{% /tab %}} {{< /tabs >}} -{{% collapse-content title="(Optional) Build and run Graal native-image" level="h4" %}} - -Follow the [Tracer Setup Instructions][14] to build your Graal native image with the Datadog Java Profiler. - -When the service binary is built, you can use environment variables to enable and configure the Datadog Java Profiler: - - ```shell - DD_PROFILING_ENABLED=true DD_PROFILING_DIRECTALLOCATION_ENABLED=true ./my_service - ``` - -**Note**: Only JFR-based profiling is supported for the GraalVM native-image applications. None of the DDPROF related configuration options are effective. -{{% /collapse-content %}} - - **Note**: The `-javaagent` argument needs to be before `-jar`. This adds it as a JVM option rather than an application argument. For example, `java -javaagent:dd-java-agent.jar ... -jar my-service.jar -more-flags`. For more information, see the [Oracle documentation][6]. - -4. Optional: Set up [Source Code Integration][7] to connect your profiling data with your Git repositories. - -5. A couple of minutes after you start your application, your profiles appear on the [Datadog APM > Profiling page][8]. If they do not, refer to the [Troubleshooting][16] guide. - -### Enabling CPU profiler engine options - -Since dd-trace-java version 1.5.0, you have two options for the CPU profiler used, Datadog or Java Flight Recorder (JFR). Since version 1.7.0, Datadog is the default, but you can also optionally enable JFR for CPU profiling. You can enable either one or both engines. Enabling both captures both profile types at the same time. - -The Datadog profiler records the active span on every sample, which improves the fidelity of the Trace to Profiling integration and Endpoint profiling features. Enabling this engine supports much better integration with APM tracing. - -The Datadog profiler consists of several profiling engines, including CPU, wallclock, allocation, and memory leak profilers. - - -{{< tabs >}} -{{% tab "Datadog Profiler" %}} -_Please refer to the Minimum JDK version requirements for enabling DataDog profiler._ - -The Datadog profiler is enabled by default in dd-trace-java versions 1.7.0+. Datadog CPU profiling is scheduled through perf events and is more accurate than JFR CPU profiling. To enable CPU profiling: - -``` -export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+ -export DD_PROFILING_DDPROF_CPU_ENABLED=true -``` - -or: - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.cpu.enabled=true -``` - -For JDK Mission Control (JMC) users, the Datadog CPU sample event is `datadog.ExecutionSample`. - -#### Linux settings - -The CPU engine works on most systems, but if the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler can't use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using `itimer`. Set `/proc/sys/kernel/perf_event_paranoid` to `2` or lower with the following command: - -``` -sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' -``` - -{{% /tab %}} - -{{% tab "JFR" %}} - -For version 1.7.0+, to switch from the default Datadog to JFR CPU profiling: - -``` -export DD_PROFILING_DDPROF_CPU_ENABLED=false -``` -or: -``` --Ddd.profiling.ddprof.cpu.enabled=false -``` -For JDK Mission Control (JMC) users, the JFR CPU sample event is `jdk.ExecutionSample`. - -{{% /tab %}} -{{< /tabs >}} - - -### Datadog profiler wallclock engine - -The wallclock profiling engine is useful for profiling latency and integrates tightly with APM tracing. The engine samples all threads, on- or off-CPU, with active tracing activity and can be used to diagnose trace or span latency. The engine has been enabled by default since 1.7.0. - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.wall.enabled=true -``` - -For version 1.7.0+, to disable the wallclock profiler: - -``` -export DD_PROFILING_DDPROF_WALL_ENABLED=false -``` -or: -``` --Ddd.profiling.ddprof.wall.enabled=false -``` - -For JMC users, the `datadog.MethodSample` event is emitted for wallclock samples. - -The wallclock engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. - -### Profiler allocation engine - -{{< tabs >}} -{{% tab "JFR" %}} -The JFR based allocation profiling engine is enabled by default since JDK 16. -The reason it's not enabled by default for JDK 8 and 11, is that an allocation intensive -application can lead to high overhead and large recording sizes. -To enable it for JDK 8 and 11, add the following: - -``` -export DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB -``` - -or: - -``` --Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB -``` -{{% /tab %}} - -{{% tab "Datadog Profiler" %}} - -The Datadog allocation profiling engine contextualizes allocation profiles, which supports allocation profiles filtered by endpoint. -In dd-java-agent earlier than v1.28.0 it is **disabled** by default. The allocation profiler relies on JVMTI APIs which could crash before OpenJDK 21.0.3 and is disabled on older JDK versions. Enable it -with: - -``` -export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+ -export DD_PROFILING_DDPROF_ALLOC_ENABLED=true # this is the default in v1.28.0+ on OpenJDK 21.0.3+ -``` - -or: - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.alloc.enabled=true # this is the default in v1.17.0+ -``` - -For JMC users, the Datadog allocation events are `datadog.ObjectAllocationInNewTLAB` and `datadog.ObjectAllocationOutsideTLAB`. - -The allocation profiler engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. -{{% /tab %}} - -{{< /tabs >}} - -### Live-heap profiler engine - -_Since: v1.39.0. Requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+._ - -The live-heap profiler engine is useful for investigating the overall memory usage of your service and identifying potential memory leaks. -The engine samples allocations and keeps track of whether those samples survived the most recent garbage collection cycle. The number of surviving samples is used to estimate the number of live objects in the heap. -The number of tracked samples is limited to avoid unbounded growth of the profiler's memory usage. - -The engine is disabled by default, but you can enable it with: - -``` -export DD_PROFILING_DDPROF_LIVEHEAP_ENABLED=true -``` - -or: - -``` --Ddd.profiling.ddprof.liveheap.enabled=true -``` - -For JMC users, the Datadog live-heap event is `datadog.HeapLiveObject`. - -The allocation engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. - -### Collecting native stack traces - -If the Datadog profiler CPU or wallclock engines are enabled, you can collect native stack traces. Native stack traces include things like JVM internals, native libraries used by your application or the JVM, and syscalls. - -
Native stack traces are not collected by default because usually they do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before you try using it in production.
- -To enable native stack trace collection, understanding that it can destabilize your application, set: - -``` -export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+ -export DD_PROFILING_DDPROF_CSTACK=dwarf -``` - -or: - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.cstack=dwarf -``` + **Note**: The `-javaagent` argument needs to be before `-jar`. This adds it as a JVM option rather than an application argument. For more information, see the [Oracle documentation][4]. +4. Optional: Set up [Source Code Integration][5] to connect your profiling data with your Git repositories. +5. After a couple of minutes, your profiles appear on the [Datadog APM > Profiling page][6]. ## Configuration @@ -335,32 +123,29 @@ You can configure the profiler using the following environment variables: | ------------------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------ | | `DD_PROFILING_ENABLED` | Boolean | Alternate for `-Ddd.profiling.enabled` argument. Set to `true` to enable profiler. | | `DD_PROFILING_ALLOCATION_ENABLED` | Boolean | Alternate for `-Ddd.profiling.allocation.enabled` argument. Set to `true` to enable the allocation profiler. It requires the profiler to be enabled already. | -| `DD_ENV` | String | The [environment][10] name, for example: `production`. | -| `DD_SERVICE` | String | The [service][10] name, for example, `web-backend`. | -| `DD_VERSION` | String | The [version][10] of your service. | +| `DD_ENV` | String | The [environment][7] name, for example: `production`. | +| `DD_SERVICE` | String | The [service][7] name, for example, `web-backend`. | +| `DD_VERSION` | String | The [version][7] of your service. | | `DD_TAGS` | String | Tags to apply to an uploaded profile. Must be a list of `:` separated by commas such as: `layer:api, team:intake`. | +## GraalVM native-image + +For GraalVM native-image applications, see [Enabling the Profiler for GraalVM Native Image][8]. + ## Not sure what to do next? -The [Getting Started with Profiler][11] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. +The [Getting Started with Profiler][9] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. ## Further Reading {{< partial name="whats-next/whats-next.html" >}} [1]: /tracing/trace_collection/ -[2]: https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/about.htm -[3]: /profiler/profiler_troubleshooting/#java-8-support -[4]: https://app.datadoghq.com/account/settings/agent/latest?platform=overview -[5]: https://app.datadoghq.com/account/settings/agent/6?platform=overview -[6]: https://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html -[7]: /integrations/guide/source-code-integration/?tab=java -[8]: https://app.datadoghq.com/profiling -[9]: /profiler/profiler_troubleshooting/#enabling-the-allocation-profiler -[10]: /getting_started/tagging/unified_service_tagging -[11]: /getting_started/profiler/ -[12]: /profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces -[13]: /profiler/enabling/supported_versions/ -[14]: /tracing/trace_collection/compatibility/java/?tab=graalvm#setup -[15]: https://docs.datadoghq.com/profiler/enabling/java/?tab=datadogprofiler# -[16]: /profiler/profiler_troubleshooting/java/ +[2]: /profiler/enabling/supported_versions/ +[3]: https://app.datadoghq.com/account/settings/agent/latest?platform=overview +[4]: https://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html +[5]: /integrations/guide/source-code-integration/?tab=java +[6]: https://app.datadoghq.com/profiling +[7]: /getting_started/tagging/unified_service_tagging +[8]: /profiler/enabling/graalvm/ +[9]: /getting_started/profiler/ diff --git a/content/en/profiler/profiler_troubleshooting/java.md b/content/en/profiler/profiler_troubleshooting/java.md index a136d29f13b..0b6e323ef6e 100644 --- a/content/en/profiler/profiler_troubleshooting/java.md +++ b/content/en/profiler/profiler_troubleshooting/java.md @@ -9,6 +9,68 @@ further_reading: text: 'APM Troubleshooting' --- +## Profiler engines + +The Java profiler uses two different profiling engines depending on your operating system: + +| Engine | Platforms | Description | +|--------|-----------|-------------| +| **Datadog Profiler** | Linux | A native profiler based on async-profiler. Uses the `AsyncGetCallTrace` JVM API and perf events for accurate CPU sampling. Provides the full set of profile types including wallclock profiling for latency analysis. Default since dd-trace-java v1.7.0. | +| **Java Flight Recorder (JFR)** | Windows, macOS, Linux (fallback) | Uses the JVM's built-in flight recorder. Available on all platforms but with fewer profile types. On Linux, used as a fallback when the Datadog Profiler cannot be enabled. | + +The profiler automatically selects the appropriate engine based on your environment. You do not need to configure this manually. + +## Supported versions and profile types + +### Linux (Datadog Profiler) + +On Linux, the Datadog profiler engine provides the full set of profile types. + +| JDK Version | CPU | Wallclock | Allocation | Live Heap | +|-------------|:---:|:---------:|:----------:|:---------:| +| OpenJDK 8u352+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ (21.0.3+) | +| Oracle JDK 8u351+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ (21.0.3+) | +| OpenJ9 8u372+, 11.0.18+, 17.0.6+ | ✓ | ✓ | ✓ | ✓ | +| Azul Platform Prime 23.05.0.0+ | ✓ | ✓ | ✓ | ✓ | + +
+Version requirements: The Datadog Profiler requires specific JDK versions due to a fix for a known issue in the AsyncGetCallTrace JVM API. This fix was backported to 8u352, 11.0.17, and 17.0.5. +
+ +**Live heap profiling** requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+ due to JVMTI API stability requirements. + +### Windows and macOS (JFR) + +On Windows and macOS, the profiler uses Java Flight Recorder (JFR). + +| JDK Version | CPU | Allocation | +|-------------|:---:|:----------:| +| OpenJDK 8u262+, 11+ | ✓ | ✓ (JDK 16+ by default) | +| Oracle JDK 11+ | ✓ | ✓ | +| Azul Zulu 8u212+, 11+ | ✓ | ✓ | +| GraalVM 17+ (JIT mode) | ✓ | ✓ | + +
+Oracle JDK 11+ may require a commercial license from Oracle to enable JFR. Contact your Oracle representative to confirm whether JFR is included in your license. +
+ +### GraalVM native-image + +For applications compiled as GraalVM native images, see [Enabling the Profiler for GraalVM Native Image][3]. + +| Platform | CPU | Allocation | +|----------|:---:|:----------:| +| Linux, Windows, macOS | ✓ | ✓ | + +### Trace to profiling integration + +The [Trace to Profiling integration][4] identifies code hotspots in slow traces. The following minimum versions are required: + +| JDK | Minimum version | dd-trace-java version | +|-----|-----------------|----------------------| +| OpenJDK | 8u352, 11.0.17, 17.0.5 | 1.17.0+ | +| OpenJ9 | 8u362, 11.0.18, 17.0.6 | 1.17.0+ | + ## Missing profiles in the profile search page If you've configured the profiler and don't see profiles in the profile search page, turn on [debug mode][1] and [open a support ticket][2] with debug files and the following information: @@ -205,9 +267,89 @@ Below are basic troubleshooting steps for resolving those issues: ``` DD_PROFILING_TEMPDIR: ``` +## Linux-specific: Degraded CPU profile quality + +On Linux, the profiler uses perf events for accurate CPU sampling. If the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler falls back to a less accurate sampling method. + +To improve CPU profile quality, set the value to `2` or lower: + +```shell +sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' +``` + +## OpenJ9 JVM crashes + +The profiler is disabled by default on OpenJ9 due to potential JVM crashes caused by a JVMTI implementation issue. If you are not experiencing crashes and want to enable the profiler: + +```shell +export DD_PROFILING_DDPROF_ENABLED=true +``` + +or: + +``` +-Ddd.profiling.ddprof.enabled=true +``` + +## GraalVM compiler (JVMCI) + +The Datadog profiler is disabled by default on the GraalVM compiler. To enable it: + +```shell +export DD_PROFILING_DDPROF_ENABLED=true +``` + +or: + +``` +-Ddd.profiling.ddprof.enabled=true +``` + +## Advanced profiler engine configuration + +The following settings allow fine-grained control over the profiler engines. These are typically not needed for standard use cases. + +| Environment variable | System property | Description | +|---------------------|-----------------|-------------| +| `DD_PROFILING_DDPROF_ENABLED` | `-Ddd.profiling.ddprof.enabled` | Enable the Datadog profiler engine (Linux only, default: true since v1.7.0) | +| `DD_PROFILING_DDPROF_CPU_ENABLED` | `-Ddd.profiling.ddprof.cpu.enabled` | Enable CPU profiling with the Datadog engine | +| `DD_PROFILING_DDPROF_WALL_ENABLED` | `-Ddd.profiling.ddprof.wall.enabled` | Enable wallclock profiling (default: true since v1.7.0) | +| `DD_PROFILING_DDPROF_ALLOC_ENABLED` | `-Ddd.profiling.ddprof.alloc.enabled` | Enable allocation profiling with the Datadog engine | +| `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED` | `-Ddd.profiling.ddprof.liveheap.enabled` | Enable live heap profiling (requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+) | +| `DD_PROFILING_ENABLED_EVENTS` | `-Ddd.profiling.enabled.events` | Enable specific JFR events (for example: `jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB`) | + +## Collecting native stack traces + +You can collect native stack traces (JVM internals, native libraries, syscalls) by enabling the following setting: + +
Native stack traces are not collected by default because they usually do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before using it in production.
+ +```shell +export DD_PROFILING_DDPROF_CSTACK=dwarf +``` + +or: + +``` +-Ddd.profiling.ddprof.cstack=dwarf +``` + +## JDK Mission Control (JMC) event reference + +If you are analyzing profiles with JDK Mission Control, the following events are emitted by the profiler: + +| Profile type | JFR event | Datadog event | +|--------------|-----------|---------------| +| CPU | `jdk.ExecutionSample` | `datadog.ExecutionSample` | +| Wallclock | - | `datadog.MethodSample` | +| Allocation | `jdk.ObjectAllocationInNewTLAB`, `jdk.ObjectAllocationOutsideTLAB` | `datadog.ObjectAllocationInNewTLAB`, `datadog.ObjectAllocationOutsideTLAB` | +| Live heap | - | `datadog.HeapLiveObject` | + ## Further Reading {{< partial name="whats-next/whats-next.html" >}} [1]: /tracing/troubleshooting/#debugging-and-logging [2]: /help/ +[3]: /profiler/enabling/graalvm/ +[4]: /profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces diff --git a/layouts/partials/profiling/profiling-languages.html b/layouts/partials/profiling/profiling-languages.html index 729df4b448b..4c25e6542c4 100644 --- a/layouts/partials/profiling/profiling-languages.html +++ b/layouts/partials/profiling/profiling-languages.html @@ -16,6 +16,14 @@ +
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/java.png" "class" "img-fluid" "alt" "GraalVM" "width" "400") }} + GraalVM +
+ +
From 154814c8b4d33d48226d4a0b378506de0c10b222 Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Wed, 4 Feb 2026 11:27:39 +0100 Subject: [PATCH 2/9] Minor updates on java.md --- content/en/profiler/enabling/java.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index cd4acb47af8..b43cda00973 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -26,8 +26,6 @@ For a summary of the minimum and recommended runtime and tracer versions across {{< tabs >}} {{% tab "Linux" %}} -**Minimum JDK versions**: OpenJDK 8u352+, 11.0.17+, 17.0.5+, or 21+ - **Available profile types**: - CPU - Wallclock (latency) @@ -35,9 +33,7 @@ For a summary of the minimum and recommended runtime and tracer versions across - Live heap (memory leak detection) {{% /tab %}} -{{% tab "Windows / macOS" %}} - -**Minimum JDK versions**: OpenJDK 8u262+, 11+ +{{% tab "Windows" %}} **Available profile types**: - CPU From 653aea32c3c27206ec1482caa3198be7ebf52686 Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Wed, 4 Feb 2026 18:14:14 +0100 Subject: [PATCH 3/9] Update requirement section - Move back requirements from Java Troubleshooting page to enablement page - Update the JDK details --- content/en/profiler/enabling/java.md | 20 +++++++++++++++++++ .../profiler/profiler_troubleshooting/java.md | 18 ----------------- 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index b43cda00973..db72b8cc4e4 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -31,6 +31,16 @@ For a summary of the minimum and recommended runtime and tracer versions across - Wallclock (latency) - Allocations - Live heap (memory leak detection) +- Exception profiling + +**Note**: On Linux, the Datadog profiler engine (ddprof) provides the full set of profile types. + +| JDK Version | CPU | Wallclock | Allocation | Live Heap | +|-------------|:---:|:---------:|:----------:|:---------:| +| OpenJDK 8u352+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ | +| Oracle JDK 8u351+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ | +| OpenJ9 JDK 8u372+, 11.0.18+, 17.0.6+ | ✓ | ✓ | ✓ | ✓ | +| Azul Platform Prime 23.05.0.0+ | ✓ | ✓ | ✓ | ✓ | {{% /tab %}} {{% tab "Windows" %}} @@ -39,6 +49,16 @@ For a summary of the minimum and recommended runtime and tracer versions across - CPU - Allocations +**Minimum JDK Versions**: + +| JDK Distribution | Minimum Version | Notes | +|------------------|-----------------|-------| +| OpenJDK | 8u282+, 11.0.17+, 17.0.5+, 21.0.3+ | Includes builds from Amazon Corretto, Azul Zulu, Eclipse Temurin, BellSoft Liberica, and other OpenJDK-based distributions | +| Oracle JDK | 11.0.17+, 17.0.5+, 21.0.3+ | JFR may require commercial license. Oracle JDK 8u40+ supported with limited features (CPU profiling only) | +| Azul Zulu | 8u212+, 11.0.17+, 17.0.5+, 21.0.3+ | | +| OpenJ9 JDK | 8u372+, 11.0.18+, 17.0.6+ | Used in Eclipse OpenJ9, IBM JDK, IBM Semeru Runtime | +| GraalVM | 17.0.11+, 21.0.3+ | JIT mode only; native-image (AOT) not supported | + {{% /tab %}} {{< /tabs >}} diff --git a/content/en/profiler/profiler_troubleshooting/java.md b/content/en/profiler/profiler_troubleshooting/java.md index 0b6e323ef6e..2e851e74adc 100644 --- a/content/en/profiler/profiler_troubleshooting/java.md +++ b/content/en/profiler/profiler_troubleshooting/java.md @@ -24,15 +24,6 @@ The profiler automatically selects the appropriate engine based on your environm ### Linux (Datadog Profiler) -On Linux, the Datadog profiler engine provides the full set of profile types. - -| JDK Version | CPU | Wallclock | Allocation | Live Heap | -|-------------|:---:|:---------:|:----------:|:---------:| -| OpenJDK 8u352+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ (21.0.3+) | -| Oracle JDK 8u351+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ (21.0.3+) | -| OpenJ9 8u372+, 11.0.18+, 17.0.6+ | ✓ | ✓ | ✓ | ✓ | -| Azul Platform Prime 23.05.0.0+ | ✓ | ✓ | ✓ | ✓ | -
Version requirements: The Datadog Profiler requires specific JDK versions due to a fix for a known issue in the AsyncGetCallTrace JVM API. This fix was backported to 8u352, 11.0.17, and 17.0.5.
@@ -41,15 +32,6 @@ On Linux, the Datadog profiler engine provides the full set of profile types. ### Windows and macOS (JFR) -On Windows and macOS, the profiler uses Java Flight Recorder (JFR). - -| JDK Version | CPU | Allocation | -|-------------|:---:|:----------:| -| OpenJDK 8u262+, 11+ | ✓ | ✓ (JDK 16+ by default) | -| Oracle JDK 11+ | ✓ | ✓ | -| Azul Zulu 8u212+, 11+ | ✓ | ✓ | -| GraalVM 17+ (JIT mode) | ✓ | ✓ | -
Oracle JDK 11+ may require a commercial license from Oracle to enable JFR. Contact your Oracle representative to confirm whether JFR is included in your license.
From c38c42872b071fb04010afa94b1d85371419a5c5 Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Tue, 10 Feb 2026 09:29:52 +0100 Subject: [PATCH 4/9] Update troubleshooting section - Add profiling engine configuration to troubleshooting documentation - Format all configuration in tables for improved readability --- content/en/profiler/enabling/java.md | 3 + .../profiler/profiler_troubleshooting/java.md | 104 +++++++++++++++--- 2 files changed, 92 insertions(+), 15 deletions(-) diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index db72b8cc4e4..87d6c0ae5b5 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -152,6 +152,8 @@ For GraalVM native-image applications, see [Enabling the Profiler for GraalVM Na The [Getting Started with Profiler][9] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. +For additional configuration options and troubleshooting guidance, see [Java Profiler Troubleshooting][10]. + ## Further Reading {{< partial name="whats-next/whats-next.html" >}} @@ -165,3 +167,4 @@ The [Getting Started with Profiler][9] guide takes a sample service with a perfo [7]: /getting_started/tagging/unified_service_tagging [8]: /profiler/enabling/graalvm/ [9]: /getting_started/profiler/ +[10]: /profiler/profiler_troubleshooting/java/ diff --git a/content/en/profiler/profiler_troubleshooting/java.md b/content/en/profiler/profiler_troubleshooting/java.md index 2e851e74adc..ba3b51ad41c 100644 --- a/content/en/profiler/profiler_troubleshooting/java.md +++ b/content/en/profiler/profiler_troubleshooting/java.md @@ -13,28 +13,102 @@ further_reading: The Java profiler uses two different profiling engines depending on your operating system: -| Engine | Platforms | Description | -|--------|-----------|-------------| -| **Datadog Profiler** | Linux | A native profiler based on async-profiler. Uses the `AsyncGetCallTrace` JVM API and perf events for accurate CPU sampling. Provides the full set of profile types including wallclock profiling for latency analysis. Default since dd-trace-java v1.7.0. | -| **Java Flight Recorder (JFR)** | Windows, macOS, Linux (fallback) | Uses the JVM's built-in flight recorder. Available on all platforms but with fewer profile types. On Linux, used as a fallback when the Datadog Profiler cannot be enabled. | +| Engine | Platforms | +|--------|-----------| +| **Datadog Profiler** | Linux | +| **Java Flight Recorder (JFR)** | Windows, macOS, Linux (fallback) | -The profiler automatically selects the appropriate engine based on your environment. You do not need to configure this manually. +- **Datadog Profiler**: The Datadog Java profiler was implemented on top of async-profiler and used JVM sampling hooks such as `AsyncGetCallTrace` (and, on Linux, the kernel perf subsystem) to collect stacks and samples. To improve robustness the tracer now performs its own stack unwinding / stack walking in newer agent releases: starting with dd-trace-java 1.55.0 the Datadog profiler uses its own stable stack-walker implementation instead of relying solely on `AsyncGetCallTrace`. That reduces ASGCT-related crashes and gives the profiler more control over unwinding across JVMs and GC modes. +- **Java Flight Recorder (JFR)**: Uses the JVM's built-in flight recorder. Available on all platforms but with fewer profile types. On Linux, used as a fallback when the Datadog Profiler cannot be enabled. -## Supported versions and profile types +The profiler automatically selects the appropriate engine based on your environment. You do not need to configure this manually. Below are the different profile types and the related configurations. + +## CPU profiling + +Datadog CPU profiling is scheduled through perf events and is more accurate than JFR CPU profiling. + +| Engine | Environment Variable | System Property | JMC Event | Notes | +|--------|---------------------|-----------------|-----------|-------| +| **Datadog** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_CPU_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.cpu.enabled=true` | `datadog.ExecutionSample` | Default on Linux | +| **JFR** | `DD_PROFILING_DDPROF_CPU_ENABLED=false` | `-Ddd.profiling.ddprof.cpu.enabled=false` | `jdk.ExecutionSample` | To switch from Datadog to JFR (v1.7.0+) | + +### Linux settings + +The CPU engine works on most systems, but if the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler can't use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using itimer. Set `/proc/sys/kernel/perf_event_paranoid` to `2` or lower with the following command: + +```shell +sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' +``` + +## Wallclock + +The Datadog profiler wallclock engine: +- Samples all threads, on- or off-CPU, with active tracing activity +- Can be used to diagnose trace or span latency +- Is useful for profiling latency and integrates tightly with APM tracing +- Enabled by default +- Is specific to the Datadog profiler and is not available on Windows + +| Configuration | Environment Variable | System Property | JMC Event | +|---------------|---------------------|-----------------|-----------| +| **Enable** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_WALL_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.wall.enabled=true` | `datadog.MethodSample` | + +**Note**: The wallclock engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. -### Linux (Datadog Profiler) +## Allocation profiling -
-Version requirements: The Datadog Profiler requires specific JDK versions due to a fix for a known issue in the AsyncGetCallTrace JVM API. This fix was backported to 8u352, 11.0.17, and 17.0.5. -
+The Datadog allocation profiling engine: +- Contextualizes allocation profiles with endpoint information +- Supports allocation profiles filtered by endpoint +- Relies on JVMTI APIs; automatically disabled on OpenJDK < 21.0.3 to prevent stability issues +- For JMC users, the Datadog allocation events are `datadog.ObjectAllocationInNewTLAB` and `datadog.ObjectAllocationOutsideTLAB` + +The JFR-based allocation profiling engine: +- Enabled by default since JDK 16 +- Not enabled by default for JDK 8 and 11, as allocation-intensive applications can lead to high overhead and large recording sizes +- For JMC users, the JFR allocation events are `jdk.ObjectAllocationInNewTLAB` and `jdk.ObjectAllocationOutsideTLAB` + +| Engine | Environment Variable | System Property | +|--------|---------------------|-----------------| +| **Datadog** (default) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_ALLOC_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.alloc.enabled=true` | +| **JFR** | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | + +**Note**: The allocation profiler engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. + +## Live heap profiling + +The live-heap profiler engine: +- Useful for investigating the overall memory usage of service and identifying potential memory leaks +- Samples allocations and keeps track of whether those samples survived the most recent garbage collection cycle +- Uses the number of surviving samples to estimate the number of live objects in the heap +- Limits the number of tracked samples to avoid unbounded growth of the profiler's memory usage +- Disabled by default +- Not available on Windows +- For JMC users, the Datadog live-heap event is `datadog.HeapLiveObject` + +| Configuration | Environment Variable | System Property | +|---------------|---------------------|-----------------| +| **Enable** (Requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+) | `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED=true` | `-Ddd.profiling.ddprof.liveheap.enabled=true` | + +**Note**: The live-heap engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. + +## Collecting native stack traces + +If the Datadog profiler CPU or wallclock engines are enabled, you can collect native stack traces. Native stack traces include: +- JVM internals +- Native libraries used by your application or the JVM +- Syscalls + +| Configuration | Environment Variable | System Property | +|---------------|---------------------|-----------------| +| **Enable** | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_CSTACK=dwarf` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.cstack=dwarf` | + +
Native stack traces are not collected by default because usually they do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before you try using it in production.
+ +## Supported versions and profile types -**Live heap profiling** requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+ due to JVMTI API stability requirements. -### Windows and macOS (JFR) -
-Oracle JDK 11+ may require a commercial license from Oracle to enable JFR. Contact your Oracle representative to confirm whether JFR is included in your license. -
### GraalVM native-image From 368c69ae0bc7cb1e9f02e3f33f2dd2ace94d1e20 Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Wed, 11 Feb 2026 12:19:27 +0100 Subject: [PATCH 5/9] Update Troubleshooting and enablement sections --- content/en/profiler/enabling/java.md | 18 ++++-- .../profiler/profiler_troubleshooting/java.md | 58 ++++++++++--------- 2 files changed, 44 insertions(+), 32 deletions(-) diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index 87d6c0ae5b5..fda9254c822 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -62,9 +62,10 @@ For a summary of the minimum and recommended runtime and tracer versions across {{% /tab %}} {{< /tabs >}} -All JVM-based languages, such as Java, Scala, Groovy, Kotlin, and Clojure are supported. +- All JVM-based languages, such as Java, Scala, Groovy, Kotlin, and Clojure are supported. + +- Java Continuous Profiler is not supported on serverless platforms such as AWS Lambda. -Continuous Profiler is not supported on serverless platforms such as AWS Lambda. ## Installation @@ -129,7 +130,14 @@ java \ 4. Optional: Set up [Source Code Integration][5] to connect your profiling data with your Git repositories. -5. After a couple of minutes, your profiles appear on the [Datadog APM > Profiling page][6]. +5. For more information on available profile types , see [Profile Types][11]. + +5. After a couple of minutes, your profiles appear on the [Datadog APM > Profiling page][6]. + +7. For GraalVM native-image applications, see [Enabling the Profiler for GraalVM Native Image][8]. + + + ## Configuration @@ -144,9 +152,7 @@ You can configure the profiler using the following environment variables: | `DD_VERSION` | String | The [version][7] of your service. | | `DD_TAGS` | String | Tags to apply to an uploaded profile. Must be a list of `:` separated by commas such as: `layer:api, team:intake`. | -## GraalVM native-image -For GraalVM native-image applications, see [Enabling the Profiler for GraalVM Native Image][8]. ## Not sure what to do next? @@ -168,3 +174,5 @@ For additional configuration options and troubleshooting guidance, see [Java Pro [8]: /profiler/enabling/graalvm/ [9]: /getting_started/profiler/ [10]: /profiler/profiler_troubleshooting/java/ +[11]: /profiler/profile_types/?tab=java + diff --git a/content/en/profiler/profiler_troubleshooting/java.md b/content/en/profiler/profiler_troubleshooting/java.md index ba3b51ad41c..62fed2405e8 100644 --- a/content/en/profiler/profiler_troubleshooting/java.md +++ b/content/en/profiler/profiler_troubleshooting/java.md @@ -53,7 +53,6 @@ The Datadog profiler wallclock engine: |---------------|---------------------|-----------------|-----------| | **Enable** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_WALL_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.wall.enabled=true` | `datadog.MethodSample` | -**Note**: The wallclock engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. ## Allocation profiling @@ -73,7 +72,14 @@ The JFR-based allocation profiling engine: | **Datadog** (default) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_ALLOC_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.alloc.enabled=true` | | **JFR** | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | -**Note**: The allocation profiler engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. + +**Note**: On Java 15 and lower, the allocation profiler is turned off by default because it can overwhelm the profiler in allocation-heavy applications. Alternatively for JFR, you can enable the following events in your `jfp` override template file +[Learn how to use override templates.](#creating-and-using-a-jfr-template-override-file) + +``` +jdk.ObjectAllocationInNewTLAB#enabled=true +jdk.ObjectAllocationOutsideTLAB#enabled=true +``` ## Live heap profiling @@ -92,6 +98,26 @@ The live-heap profiler engine: **Note**: The live-heap engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. + +## Heap Profiling +
This feature requires at least Java 11.0.12, 15.0.4, 16.0.2, 17.0.3 or 18 and newer
+ +To enable the heap profiler, start your application with the `-Ddd.profiling.heap.enabled=true` JVM setting or the `DD_PROFILING_HEAP_ENABLED=true` environment variable. + +Alternatively, you can enable the following events in your `jfp` [override template file](#creating-and-using-a-jfr-template-override-file): + +``` +jdk.OldObjectSample#enabled=true +``` + + +## Heap histogram metrics + +
This feature requires at least Java 17.0.9 or newer and does not work with ZGC
+ +To enable the heap histogram metrics, start your application with the `-Ddd.profiling.heap.histogram.enabled=true` JVM setting or the `DD_PROFILING_HEAP_HISTOGRAM_ENABLED=true` environment variable. + + ## Collecting native stack traces If the Datadog profiler CPU or wallclock engines are enabled, you can collect native stack traces. Native stack traces include: @@ -104,12 +130,6 @@ If the Datadog profiler CPU or wallclock engines are enabled, you can collect na | **Enable** | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_CSTACK=dwarf` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.cstack=dwarf` |
Native stack traces are not collected by default because usually they do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before you try using it in production.
- -## Supported versions and profile types - - - - ### GraalVM native-image For applications compiled as GraalVM native images, see [Enabling the Profiler for GraalVM Native Image][3]. @@ -242,7 +262,7 @@ The Datadog exception profiler has a small footprint and overhead under normal c To disable exception profiling, start the tracer with the `-Ddd.integration.throwables.enabled=false` JVM setting. -Remember to turn this setting back on after you've returned to a more typical rate of exceptions. +Note: Turn this setting back on after you've returned to a more typical rate of exceptions. ## Java 8 support @@ -363,7 +383,7 @@ or: ## Advanced profiler engine configuration -The following settings allow fine-grained control over the profiler engines. These are typically not needed for standard use cases. +The following settings allow fine-grained control over the profiler engines. These are typically not needed for standard use cases. For detailed information about each profiler type, see the corresponding sections above: [CPU profiling](#cpu-profiling), [Wallclock](#wallclock), [Allocation profiling](#allocation-profiling), and [Live heap profiling](#live-heap-profiling). | Environment variable | System property | Description | |---------------------|-----------------|-------------| @@ -374,25 +394,9 @@ The following settings allow fine-grained control over the profiler engines. The | `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED` | `-Ddd.profiling.ddprof.liveheap.enabled` | Enable live heap profiling (requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+) | | `DD_PROFILING_ENABLED_EVENTS` | `-Ddd.profiling.enabled.events` | Enable specific JFR events (for example: `jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB`) | -## Collecting native stack traces - -You can collect native stack traces (JVM internals, native libraries, syscalls) by enabling the following setting: - -
Native stack traces are not collected by default because they usually do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before using it in production.
- -```shell -export DD_PROFILING_DDPROF_CSTACK=dwarf -``` - -or: - -``` --Ddd.profiling.ddprof.cstack=dwarf -``` - ## JDK Mission Control (JMC) event reference -If you are analyzing profiles with JDK Mission Control, the following events are emitted by the profiler: +If you are analyzing profiles with JDK Mission Control, the following table provides a quick reference for events emitted by the profiler. For detailed information about each profile type, see the corresponding sections above: [CPU profiling](#cpu-profiling), [Wallclock](#wallclock), [Allocation profiling](#allocation-profiling), and [Live heap profiling](#live-heap-profiling). | Profile type | JFR event | Datadog event | |--------------|-----------|---------------| From 60e177c408a0f372038e71379668d27ee1e07266 Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Wed, 18 Feb 2026 12:43:42 +0100 Subject: [PATCH 6/9] Update troubleshooting section - Remove OpenJ9 from JFR support - Remove redundant and Obsolete data - Update Allocation profiler section --- content/en/profiler/enabling/java.md | 6 +-- .../profiler/profiler_troubleshooting/java.md | 52 ++++++------------- 2 files changed, 16 insertions(+), 42 deletions(-) diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index fda9254c822..b3e52e6a7f4 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -56,17 +56,13 @@ For a summary of the minimum and recommended runtime and tracer versions across | OpenJDK | 8u282+, 11.0.17+, 17.0.5+, 21.0.3+ | Includes builds from Amazon Corretto, Azul Zulu, Eclipse Temurin, BellSoft Liberica, and other OpenJDK-based distributions | | Oracle JDK | 11.0.17+, 17.0.5+, 21.0.3+ | JFR may require commercial license. Oracle JDK 8u40+ supported with limited features (CPU profiling only) | | Azul Zulu | 8u212+, 11.0.17+, 17.0.5+, 21.0.3+ | | -| OpenJ9 JDK | 8u372+, 11.0.18+, 17.0.6+ | Used in Eclipse OpenJ9, IBM JDK, IBM Semeru Runtime | -| GraalVM | 17.0.11+, 21.0.3+ | JIT mode only; native-image (AOT) not supported | +| GraalVM | 17.0.11+, 21.0.3+ | JIT mode and native-image(AOT) | {{% /tab %}} {{< /tabs >}} - All JVM-based languages, such as Java, Scala, Groovy, Kotlin, and Clojure are supported. -- Java Continuous Profiler is not supported on serverless platforms such as AWS Lambda. - - ## Installation To begin profiling applications: diff --git a/content/en/profiler/profiler_troubleshooting/java.md b/content/en/profiler/profiler_troubleshooting/java.md index 62fed2405e8..e7948762cc6 100644 --- a/content/en/profiler/profiler_troubleshooting/java.md +++ b/content/en/profiler/profiler_troubleshooting/java.md @@ -32,13 +32,7 @@ Datadog CPU profiling is scheduled through perf events and is more accurate than | **Datadog** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_CPU_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.cpu.enabled=true` | `datadog.ExecutionSample` | Default on Linux | | **JFR** | `DD_PROFILING_DDPROF_CPU_ENABLED=false` | `-Ddd.profiling.ddprof.cpu.enabled=false` | `jdk.ExecutionSample` | To switch from Datadog to JFR (v1.7.0+) | -### Linux settings -The CPU engine works on most systems, but if the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler can't use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using itimer. Set `/proc/sys/kernel/perf_event_paranoid` to `2` or lower with the following command: - -```shell -sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' -``` ## Wallclock @@ -70,8 +64,8 @@ The JFR-based allocation profiling engine: | Engine | Environment Variable | System Property | |--------|---------------------|-----------------| | **Datadog** (default) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_ALLOC_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.alloc.enabled=true` | -| **JFR** | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | - +| **JFR** (Java 8) | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | +| **JFR** (Java 16) | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationSample` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationSample` | **Note**: On Java 15 and lower, the allocation profiler is turned off by default because it can overwhelm the profiler in allocation-heavy applications. Alternatively for JFR, you can enable the following events in your `jfp` override template file [Learn how to use override templates.](#creating-and-using-a-jfr-template-override-file) @@ -157,12 +151,12 @@ If you've configured the profiler and don't see profiles in the profile search p ## Reduce overhead from default setup -If the default setup overhead is not acceptable, you can use the profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default: +If the default setup overhead is not acceptable, you can use profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default: - Increases sampling threshold to 500ms for `ThreadSleep`, `ThreadPark`, and `JavaMonitorWait` events compared to 100ms default - Disables `ObjectAllocationInNewTLAB`, `ObjectAllocationOutsideTLAB`, `ExceptionSample`, `ExceptionCount` events -To use the minimal configuration ensure you have `dd-java-agent` version `0.70.0` then change your service invocation to the following: +To use the minimal configuration change your service ensure you have a recent version of java tracer, then change the service invocation to the following: ``` java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=minimal -jar .jar @@ -175,7 +169,7 @@ If you want more granularity in your profiling data, you can specify the `compre - Reduces sampling threshold to 10ms for `ThreadSleep`, `ThreadPark`, and `JavaMonitorWait` events compared to 100ms default - Enables `ObjectAllocationInNewTLAB`, `ObjectAllocationOutsideTLAB`, `ExceptionSample`, `ExceptionCount` events -To use the comprehensive configuration ensure you have `dd-trace-java` version `0.70.0` then change your service invocation to the following: +To use the comprehensive configuration ensure you have a recent version of java tracer, change your service invocation to the following: ``` java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=comprehensive -jar .jar @@ -353,35 +347,11 @@ To improve CPU profile quality, set the value to `2` or lower: sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' ``` -## OpenJ9 JVM crashes -The profiler is disabled by default on OpenJ9 due to potential JVM crashes caused by a JVMTI implementation issue. If you are not experiencing crashes and want to enable the profiler: -```shell -export DD_PROFILING_DDPROF_ENABLED=true -``` -or: -``` --Ddd.profiling.ddprof.enabled=true -``` - -## GraalVM compiler (JVMCI) - -The Datadog profiler is disabled by default on the GraalVM compiler. To enable it: - -```shell -export DD_PROFILING_DDPROF_ENABLED=true -``` - -or: - -``` --Ddd.profiling.ddprof.enabled=true -``` - -## Advanced profiler engine configuration +## Summary - Advanced profiler engine configuration The following settings allow fine-grained control over the profiler engines. These are typically not needed for standard use cases. For detailed information about each profiler type, see the corresponding sections above: [CPU profiling](#cpu-profiling), [Wallclock](#wallclock), [Allocation profiling](#allocation-profiling), and [Live heap profiling](#live-heap-profiling). @@ -396,7 +366,7 @@ The following settings allow fine-grained control over the profiler engines. The ## JDK Mission Control (JMC) event reference -If you are analyzing profiles with JDK Mission Control, the following table provides a quick reference for events emitted by the profiler. For detailed information about each profile type, see the corresponding sections above: [CPU profiling](#cpu-profiling), [Wallclock](#wallclock), [Allocation profiling](#allocation-profiling), and [Live heap profiling](#live-heap-profiling). +If you are analyzing profiles with JDK Mission Control, the following table provides a quick reference for events emitted by the profiler. | Profile type | JFR event | Datadog event | |--------------|-----------|---------------| @@ -405,6 +375,14 @@ If you are analyzing profiles with JDK Mission Control, the following table prov | Allocation | `jdk.ObjectAllocationInNewTLAB`, `jdk.ObjectAllocationOutsideTLAB` | `datadog.ObjectAllocationInNewTLAB`, `datadog.ObjectAllocationOutsideTLAB` | | Live heap | - | `datadog.HeapLiveObject` | +### Advanced Linux settings for CPU profiles + +The CPU profiler engine works on most systems, but if the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler can't use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using itimer. Set `/proc/sys/kernel/perf_event_paranoid` to `2` or lower with the following command: + +```shell +sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' +``` + ## Further Reading {{< partial name="whats-next/whats-next.html" >}} From d604be458fda10833cdc0b43ba5dfcefc73775b4 Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Fri, 20 Feb 2026 12:15:24 +0100 Subject: [PATCH 7/9] Update profiler documentation - Update troubleshooting section - Update params.yaml --- config/_default/params.yaml | 1 + .../profiler/profiler_troubleshooting/java.md | 41 +++++++------------ 2 files changed, 16 insertions(+), 26 deletions(-) diff --git a/config/_default/params.yaml b/config/_default/params.yaml index 716290c3556..80c0b355a88 100644 --- a/config/_default/params.yaml +++ b/config/_default/params.yaml @@ -83,6 +83,7 @@ code_language_ids: kotlin-multiplatform: Kotlin Multiplatform swiftui: Swift UI java: Java + graalvm: GraalVM cpp: 'C++' php: PHP nodejs: "Node.js" diff --git a/content/en/profiler/profiler_troubleshooting/java.md b/content/en/profiler/profiler_troubleshooting/java.md index 5bc3868dde4..407b44dc88f 100644 --- a/content/en/profiler/profiler_troubleshooting/java.md +++ b/content/en/profiler/profiler_troubleshooting/java.md @@ -71,10 +71,15 @@ The JFR-based allocation profiling engine: [Learn how to use override templates.](#creating-and-using-a-jfr-template-override-file) ``` +#Java8 jdk.ObjectAllocationInNewTLAB#enabled=true jdk.ObjectAllocationOutsideTLAB#enabled=true + +#Java16 +jdk.ObjectAllocationSample#enabled=true ``` + ## Live heap profiling The live-heap profiler engine: @@ -96,7 +101,11 @@ The live-heap profiler engine: ## Heap Profiling
This feature requires at least Java 11.0.12, 15.0.4, 16.0.2, 17.0.3 or 18 and newer
-To enable the heap profiler, start your application with the `-Ddd.profiling.heap.enabled=true` JVM setting or the `DD_PROFILING_HEAP_ENABLED=true` environment variable. +To enable the heap profiler, start your application with one of the following: + +| Environment Variable | System Property | +|---------------------|-----------------| +| `DD_PROFILING_HEAP_ENABLED=true` | `-Ddd.profiling.heap.enabled=true` | Alternatively, you can enable the following events in your `jfp` [override template file](#creating-and-using-a-jfr-template-override-file): @@ -109,22 +118,15 @@ jdk.OldObjectSample#enabled=true
This feature requires at least Java 17.0.9 or newer and does not work with ZGC
-To enable the heap histogram metrics, start your application with the `-Ddd.profiling.heap.histogram.enabled=true` JVM setting or the `DD_PROFILING_HEAP_HISTOGRAM_ENABLED=true` environment variable. +To enable the heap histogram metrics, start your application with one of the following: +| Environment Variable | System Property | +|---------------------|-----------------| +| `DD_PROFILING_HEAP_HISTOGRAM_ENABLED=true` | `-Ddd.profiling.heap.histogram.enabled=true` | -## Collecting native stack traces -If the Datadog profiler CPU or wallclock engines are enabled, you can collect native stack traces. Native stack traces include: -- JVM internals -- Native libraries used by your application or the JVM -- Syscalls -| Configuration | Environment Variable | System Property | -|---------------|---------------------|-----------------| -| **Enable** | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_CSTACK=dwarf` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.cstack=dwarf` | - -
Native stack traces are not collected by default because usually they do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before you try using it in production.
-### GraalVM native-image +## GraalVM native-image For applications compiled as GraalVM native images, see [Enabling the Profiler for GraalVM Native Image][3]. @@ -339,19 +341,6 @@ Below are basic troubleshooting steps for resolving those issues: ``` DD_PROFILING_TEMPDIR: ``` -## Linux-specific: Degraded CPU profile quality - -On Linux, the profiler uses perf events for accurate CPU sampling. If the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler falls back to a less accurate sampling method. - -To improve CPU profile quality, set the value to `2` or lower: - -```shell -sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' -``` - - - - ## Summary - Advanced profiler engine configuration From 430856baefa7d3108f06f27cf416600e158d6b52 Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Fri, 20 Feb 2026 16:25:02 +0100 Subject: [PATCH 8/9] Update profile type support --- content/en/profiler/enabling/java.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index b3e52e6a7f4..ac84cb9bd42 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -32,6 +32,7 @@ For a summary of the minimum and recommended runtime and tracer versions across - Allocations - Live heap (memory leak detection) - Exception profiling +- I/O profiling **Note**: On Linux, the Datadog profiler engine (ddprof) provides the full set of profile types. @@ -48,7 +49,9 @@ For a summary of the minimum and recommended runtime and tracer versions across **Available profile types**: - CPU - Allocations - +- I/O profiling +- Exception +- Live heap (on Linux) **Minimum JDK Versions**: | JDK Distribution | Minimum Version | Notes | From cd1680c0ac2764e3e342731f5f88434f15dca04d Mon Sep 17 00:00:00 2001 From: anandreshmi46 <62700652+anandreshmi46@users.noreply.github.com> Date: Fri, 20 Feb 2026 16:32:41 +0100 Subject: [PATCH 9/9] Update java profiler enablement page - Remove ddprof mention --- content/en/profiler/enabling/java.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index ac84cb9bd42..f4d03393efc 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -34,7 +34,7 @@ For a summary of the minimum and recommended runtime and tracer versions across - Exception profiling - I/O profiling -**Note**: On Linux, the Datadog profiler engine (ddprof) provides the full set of profile types. +**Note**: On Linux, the Datadog profiler engine provides the full set of profile types. | JDK Version | CPU | Wallclock | Allocation | Live Heap | |-------------|:---:|:---------:|:----------:|:---------:|