diff --git a/config/_default/params.yaml b/config/_default/params.yaml index 716290c3556..80c0b355a88 100644 --- a/config/_default/params.yaml +++ b/config/_default/params.yaml @@ -83,6 +83,7 @@ code_language_ids: kotlin-multiplatform: Kotlin Multiplatform swiftui: Swift UI java: Java + graalvm: GraalVM cpp: 'C++' php: PHP nodejs: "Node.js" diff --git a/content/en/profiler/enabling/graalvm.md b/content/en/profiler/enabling/graalvm.md new file mode 100644 index 00000000000..084a33fb7dd --- /dev/null +++ b/content/en/profiler/enabling/graalvm.md @@ -0,0 +1,89 @@ +--- +title: Enabling the Profiler for GraalVM Native Image +code_lang: graalvm +type: multi-code-lang +code_lang_weight: 15 +further_reading: + - link: 'getting_started/profiler' + tag: 'Documentation' + text: 'Getting Started with Profiler' + - link: 'profiler/profile_visualizations' + tag: 'Documentation' + text: 'Learn more about available profile visualizations' + - link: 'profiler/profiler_troubleshooting/java' + tag: 'Documentation' + text: 'Fix problems you encounter while using the profiler' +--- + +
+Datadog Profiler support for GraalVM native-image is in beta. Datadog recommends evaluating the profiler in a non-sensitive environment before deploying in production. +
+ +This page describes how to enable profiling for applications compiled as GraalVM native images. For standard JVM applications, see [Enabling the Java Profiler][1]. + +## Requirements + +For a summary of the minimum and recommended runtime and tracer versions across all languages, read [Supported Language and Tracer Versions][2]. + +Minimum version +: GraalVM 17+ + +Supported platforms +: Linux, Windows, macOS + +Available profile types +: CPU, Allocations + +## Installation + +### 1. Build your native image with the Datadog tracer + +Follow the [Tracer Setup Instructions][3] to build your GraalVM native image with the Datadog Java Profiler instrumentation. + +### 2. Run with profiling enabled + +After the service binary is built, enable the profiler using environment variables: + +```shell +export DD_SERVICE= +export DD_ENV= +export DD_VERSION= +export DD_PROFILING_ENABLED=true +export DD_PROFILING_DIRECTALLOCATION_ENABLED=true +./my_service +``` + +### 3. Verify profiles are collected + +After a couple of minutes, your profiles appear on the [Datadog APM > Profiler page][4]. + +## Configuration + +| Environment variable | Description | +|-----------------------------------|--------------------------------------------------------------------------| +| `DD_PROFILING_ENABLED` | Set to `true` to enable the profiler. | +| `DD_PROFILING_DIRECTALLOCATION_ENABLED` | Set to `true` to enable allocation profiling. | +| `DD_SERVICE` | The [service][5] name. | +| `DD_ENV` | The [environment][5] name. | +| `DD_VERSION` | The [version][5] of your service. | + +## Limitations + +- Only JFR-based profiling is supported for GraalVM native-image applications. +- Wallclock and live heap profiling are not available. + +## Not sure what to do next? + +The [Getting Started with Profiler][6] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. + +## Further Reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /profiler/enabling/java/ +[2]: /profiler/enabling/supported_versions/ +[3]: /tracing/trace_collection/compatibility/java/?tab=graalvm#setup +[4]: https://app.datadoghq.com/profiling +[5]: /getting_started/tagging/unified_service_tagging +[6]: /getting_started/profiler/ + diff --git a/content/en/profiler/enabling/java.md b/content/en/profiler/enabling/java.md index ac4f22f5aa8..f4d03393efc 100644 --- a/content/en/profiler/enabling/java.md +++ b/content/en/profiler/enabling/java.md @@ -21,69 +21,56 @@ The profiler is shipped within Datadog tracing libraries. If you are already usi ## Requirements -For a summary of the minimum and recommended runtime and tracer versions across all languages, read [Supported Language and Tracer Versions][13]. - -As of dd-trace-java 1.0.0, you have two options for the engine that generates profile data for Java applications: [Java Flight Recorder (JFR)][2] or the Datadog Profiler. As of dd-trace-java 1.7.0, Datadog Profiler is the default. Each profiler engine has different side effects, requirements, available configurations, and limitations, and this page describes each. You can enable either one or both engines. Enabling both captures both profile types at the same time. +For a summary of the minimum and recommended runtime and tracer versions across all languages, read [Supported Language and Tracer Versions][2]. {{< tabs >}} -{{% tab "Datadog Profiler" %}} - -Supported operating systems: -- Linux - -Minimum JDK versions: -- OpenJDK 8u352+, 11.0.17+, 17.0.5+, 21+ (including builds on top of it: Amazon Corretto, Azul Zulu, and others) -- Oracle JDK 8u351+, 11.0.17+, 17.0.5+, 21+ -- OpenJ9 JDK 8u372+, 11.0.18+, 17.0.6+ (used on Eclipse OpenJ9, IBM JDK, IBM Semeru Runtime). - -**Note:** The profiler is disabled by default for OpenJ9 due to the possibility of crashing JVM caused by a subtle bug in JVTMI implementation. If you are **not** experiencing any crashes, you can enable the profiler by adding `-Ddd.profiling.ddprof.enabled=true` or `DD_PROFILING_DDPROF_ENABLED=true`. -- Azul Platform Prime 23.05.0.0+ (formerly Azul Zing) +{{% tab "Linux" %}} +**Available profile types**: +- CPU +- Wallclock (latency) +- Allocations +- Live heap (memory leak detection) +- Exception profiling +- I/O profiling -**Note:** The Datadog Profiler is disabled on the GraalVM compiler (JVMCI) and needs to be enabled explicitly with `-Ddd.profiling.ddprof.enabled=true` or `DD_PROFILING_DDPROF_ENABLED=true`. +**Note**: On Linux, the Datadog profiler engine provides the full set of profile types. -The Datadog Profiler uses the JVMTI `AsyncGetCallTrace` function, in which there is a [known issue][1] prior to JDK release 17.0.5. This fix was backported to 11.0.17 and 8u352. The Datadog Profiler is not enabled unless the JVM the profiler is deployed into has this fix. Upgrade to at least 8u352, 11.0.17, 17.0.5, or the latest non-LTS JVM version to use the Datadog Profiler. +| JDK Version | CPU | Wallclock | Allocation | Live Heap | +|-------------|:---:|:---------:|:----------:|:---------:| +| OpenJDK 8u352+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ | +| Oracle JDK 8u351+, 11.0.17+, 17.0.5+, 21+ | ✓ | ✓ | ✓ | ✓ | +| OpenJ9 JDK 8u372+, 11.0.18+, 17.0.6+ | ✓ | ✓ | ✓ | ✓ | +| Azul Platform Prime 23.05.0.0+ | ✓ | ✓ | ✓ | ✓ | -[1]: https://bugs.openjdk.org/browse/JDK-8283849 {{% /tab %}} - -{{% tab "JFR" %}} - -Supported operating systems: -- Linux -- Windows - -Minimum JDK versions: -- OpenJDK [1.8.0.262/8u262+][3], 11+ (including builds on top of it: Amazon Corretto, and others) -- Oracle JDK 11+ (Enabling the JFR may require a commercial license from Oracle. Reach out to your Oracle representative to confirm whether this is part of your license) -- Azul Zulu 8 (version 1.8.0.212/8u212+), 11+ -- GraalVM 17+ - both, JIT and AOT (native-image) versions - -Because non-LTS JDK versions may not contain stability and performance fixes related to the Datadog Profiler library, use versions 8, 11, and 17 of the Long Term Support JDK. - -Additional requirements for profiling [Trace to Profiling integration][12]: - - OpenJDK 17.0.5+ and `dd-trace-java` version 1.17.0+ - - OpenJDK 11.0.17+ and `dd-trace-java` version 1.17.0+ - - OpenJDK 8 8u352+ and `dd-trace-java` version 1.17.0+ - - OpenJ9 17.0.6+ and `dd-trace-java` version 1.17.0+ - - OpenJ9 11.0.18+ and `dd-trace-java` version 1.17.0+ - - OpenJ9 8.0.362+ and `dd-trace-java` version 1.17.0+ - -[3]: /profiler/profiler_troubleshooting/java/#java-8-support -[12]: /profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces +{{% tab "Windows" %}} + +**Available profile types**: +- CPU +- Allocations +- I/O profiling +- Exception +- Live heap (on Linux) +**Minimum JDK Versions**: + +| JDK Distribution | Minimum Version | Notes | +|------------------|-----------------|-------| +| OpenJDK | 8u282+, 11.0.17+, 17.0.5+, 21.0.3+ | Includes builds from Amazon Corretto, Azul Zulu, Eclipse Temurin, BellSoft Liberica, and other OpenJDK-based distributions | +| Oracle JDK | 11.0.17+, 17.0.5+, 21.0.3+ | JFR may require commercial license. Oracle JDK 8u40+ supported with limited features (CPU profiling only) | +| Azul Zulu | 8u212+, 11.0.17+, 17.0.5+, 21.0.3+ | | +| GraalVM | 17.0.11+, 21.0.3+ | JIT mode and native-image(AOT) | {{% /tab %}} {{< /tabs >}} -All JVM-based languages, such as Java, Scala, Groovy, Kotlin, and Clojure are supported. - -Continuous Profiler is not supported on some serverless platforms, such as AWS Lambda. +- All JVM-based languages, such as Java, Scala, Groovy, Kotlin, and Clojure are supported. ## Installation To begin profiling applications: -1. Ensure Datadog Agent v6+ is installed and running. Datadog recommends using [Datadog Agent v7+][4]. If you don't have APM enabled to set up your application to send data to Datadog, in your Agent, set the `DD_APM_ENABLED` environment variable to `true` and listening to the port `8126/TCP`. +1. Ensure Datadog Agent v6+ is installed and running. Datadog recommends using [Datadog Agent v7+][3]. If you don't have APM enabled to set up your application to send data to Datadog, in your Agent, set the `DD_APM_ENABLED` environment variable to `true` and listening to the port `8126/TCP`. 2. Download `dd-java-agent.jar`, which contains the Java Agent class files: @@ -108,11 +95,11 @@ To begin profiling applications: **Note**: Profiler is available in the `dd-java-agent.jar` library in versions 0.55+. 3. Enable the profiler by setting `-Ddd.profiling.enabled` flag or `DD_PROFILING_ENABLED` environment variable to `true`. Specify `dd.service`, `dd.env`, and `dd.version` so you can filter and group your profiles across these dimensions: + {{< tabs >}} {{% tab "Command arguments" %}} -Invoke your service: -```diff +```shell java \ -javaagent:dd-java-agent.jar \ -Ddd.service= \ @@ -125,7 +112,7 @@ java \ {{% /tab %}} {{% tab "Environment variables" %}} -```diff +```shell export DD_SERVICE= export DD_ENV= export DD_VERSION= @@ -138,194 +125,16 @@ java \ {{% /tab %}} {{< /tabs >}} -{{% collapse-content title="(Optional) Build and run Graal native-image" level="h4" %}} - -Follow the [Tracer Setup Instructions][14] to build your Graal native image with the Datadog Java Profiler. - -When the service binary is built, you can use environment variables to enable and configure the Datadog Java Profiler: - - ```shell - DD_PROFILING_ENABLED=true DD_PROFILING_DIRECTALLOCATION_ENABLED=true ./my_service - ``` - -**Note**: Only JFR-based profiling is supported for the GraalVM native-image applications. None of the DDPROF related configuration options are effective. -{{% /collapse-content %}} - - **Note**: The `-javaagent` argument needs to be before `-jar`. This adds it as a JVM option rather than an application argument. For example, `java -javaagent:dd-java-agent.jar ... -jar my-service.jar -more-flags`. For more information, see the [Oracle documentation][6]. + **Note**: The `-javaagent` argument needs to be before `-jar`. This adds it as a JVM option rather than an application argument. For more information, see the [Oracle documentation][4]. -4. Optional: Set up [Source Code Integration][7] to connect your profiling data with your Git repositories. +4. Optional: Set up [Source Code Integration][5] to connect your profiling data with your Git repositories. -5. A couple of minutes after you start your application, your profiles appear on the [Datadog APM > Profiling page][8]. If they do not, refer to the [Troubleshooting][16] guide. +5. For more information on available profile types , see [Profile Types][11]. -### Enabling CPU profiler engine options +5. After a couple of minutes, your profiles appear on the [Datadog APM > Profiling page][6]. -Since dd-trace-java version 1.5.0, you have two options for the CPU profiler used, Datadog or Java Flight Recorder (JFR). Since version 1.7.0, Datadog is the default, but you can also optionally enable JFR for CPU profiling. You can enable either one or both engines. Enabling both captures both profile types at the same time. +7. For GraalVM native-image applications, see [Enabling the Profiler for GraalVM Native Image][8]. -The Datadog profiler records the active span on every sample, which improves the fidelity of the Trace to Profiling integration and Endpoint profiling features. Enabling this engine supports much better integration with APM tracing. - -The Datadog profiler consists of several profiling engines, including CPU, wallclock, allocation, and memory leak profilers. - - -{{< tabs >}} -{{% tab "Datadog Profiler" %}} -_Please refer to the Minimum JDK version requirements for enabling DataDog profiler._ - -The Datadog profiler is enabled by default in dd-trace-java versions 1.7.0+. Datadog CPU profiling is scheduled through perf events and is more accurate than JFR CPU profiling. To enable CPU profiling: - -``` -export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+ -export DD_PROFILING_DDPROF_CPU_ENABLED=true -``` - -or: - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.cpu.enabled=true -``` - -For JDK Mission Control (JMC) users, the Datadog CPU sample event is `datadog.ExecutionSample`. - -#### Linux settings - -The CPU engine works on most systems, but if the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler can't use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using `itimer`. Set `/proc/sys/kernel/perf_event_paranoid` to `2` or lower with the following command: - -``` -sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' -``` - -{{% /tab %}} - -{{% tab "JFR" %}} - -For version 1.7.0+, to switch from the default Datadog to JFR CPU profiling: - -``` -export DD_PROFILING_DDPROF_CPU_ENABLED=false -``` -or: -``` --Ddd.profiling.ddprof.cpu.enabled=false -``` -For JDK Mission Control (JMC) users, the JFR CPU sample event is `jdk.ExecutionSample`. - -{{% /tab %}} -{{< /tabs >}} - - -### Datadog profiler wallclock engine - -The wallclock profiling engine is useful for profiling latency and integrates tightly with APM tracing. The engine samples all threads, on- or off-CPU, with active tracing activity and can be used to diagnose trace or span latency. The engine has been enabled by default since 1.7.0. - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.wall.enabled=true -``` - -For version 1.7.0+, to disable the wallclock profiler: - -``` -export DD_PROFILING_DDPROF_WALL_ENABLED=false -``` -or: -``` --Ddd.profiling.ddprof.wall.enabled=false -``` - -For JMC users, the `datadog.MethodSample` event is emitted for wallclock samples. - -The wallclock engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. - -### Profiler allocation engine - -{{< tabs >}} -{{% tab "JFR" %}} -The JFR based allocation profiling engine is enabled by default since JDK 16. -The reason it's not enabled by default for JDK 8 and 11, is that an allocation intensive -application can lead to high overhead and large recording sizes. -To enable it for JDK 8 and 11, add the following: - -``` -export DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB -``` - -or: - -``` --Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB -``` -{{% /tab %}} - -{{% tab "Datadog Profiler" %}} - -The Datadog allocation profiling engine contextualizes allocation profiles, which supports allocation profiles filtered by endpoint. -In dd-java-agent earlier than v1.28.0 it is **disabled** by default. The allocation profiler relies on JVMTI APIs which could crash before OpenJDK 21.0.3 and is disabled on older JDK versions. Enable it -with: - -``` -export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+ -export DD_PROFILING_DDPROF_ALLOC_ENABLED=true # this is the default in v1.28.0+ on OpenJDK 21.0.3+ -``` - -or: - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.alloc.enabled=true # this is the default in v1.17.0+ -``` - -For JMC users, the Datadog allocation events are `datadog.ObjectAllocationInNewTLAB` and `datadog.ObjectAllocationOutsideTLAB`. - -The allocation profiler engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. -{{% /tab %}} - -{{< /tabs >}} - -### Live-heap profiler engine - -_Since: v1.39.0. Requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+._ - -The live-heap profiler engine is useful for investigating the overall memory usage of your service and identifying potential memory leaks. -The engine samples allocations and keeps track of whether those samples survived the most recent garbage collection cycle. The number of surviving samples is used to estimate the number of live objects in the heap. -The number of tracked samples is limited to avoid unbounded growth of the profiler's memory usage. - -The engine is disabled by default, but you can enable it with: - -``` -export DD_PROFILING_DDPROF_LIVEHEAP_ENABLED=true -``` - -or: - -``` --Ddd.profiling.ddprof.liveheap.enabled=true -``` - -For JMC users, the Datadog live-heap event is `datadog.HeapLiveObject`. - -The allocation engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. - -**Note**: Live-heap profiler is based on the Datadog Profiler library and not JFR. Live-heap profile type is not available on Windows. - -### Collecting native stack traces - -If the Datadog profiler CPU or wallclock engines are enabled, you can collect native stack traces. Native stack traces include things like JVM internals, native libraries used by your application or the JVM, and syscalls. - -
Native stack traces are not collected by default because usually they do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before you try using it in production.
- -To enable native stack trace collection, understanding that it can destabilize your application, set: - -``` -export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+ -export DD_PROFILING_DDPROF_CSTACK=dwarf -``` - -or: - -``` --Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+ --Ddd.profiling.ddprof.cstack=dwarf -``` @@ -337,32 +146,32 @@ You can configure the profiler using the following environment variables: | ------------------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------ | | `DD_PROFILING_ENABLED` | Boolean | Alternate for `-Ddd.profiling.enabled` argument. Set to `true` to enable profiler. | | `DD_PROFILING_ALLOCATION_ENABLED` | Boolean | Alternate for `-Ddd.profiling.allocation.enabled` argument. Set to `true` to enable the allocation profiler. It requires the profiler to be enabled already. | -| `DD_ENV` | String | The [environment][10] name, for example: `production`. | -| `DD_SERVICE` | String | The [service][10] name, for example, `web-backend`. | -| `DD_VERSION` | String | The [version][10] of your service. | +| `DD_ENV` | String | The [environment][7] name, for example: `production`. | +| `DD_SERVICE` | String | The [service][7] name, for example, `web-backend`. | +| `DD_VERSION` | String | The [version][7] of your service. | | `DD_TAGS` | String | Tags to apply to an uploaded profile. Must be a list of `:` separated by commas such as: `layer:api, team:intake`. | + + ## Not sure what to do next? -The [Getting Started with Profiler][11] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. +The [Getting Started with Profiler][9] guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem. + +For additional configuration options and troubleshooting guidance, see [Java Profiler Troubleshooting][10]. ## Further Reading {{< partial name="whats-next/whats-next.html" >}} [1]: /tracing/trace_collection/ -[2]: https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/about.htm -[3]: /profiler/profiler_troubleshooting/#java-8-support -[4]: https://app.datadoghq.com/account/settings/agent/latest?platform=overview -[5]: https://app.datadoghq.com/account/settings/agent/6?platform=overview -[6]: https://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html -[7]: /integrations/guide/source-code-integration/?tab=java -[8]: https://app.datadoghq.com/profiling -[9]: /profiler/profiler_troubleshooting/#enabling-the-allocation-profiler -[10]: /getting_started/tagging/unified_service_tagging -[11]: /getting_started/profiler/ -[12]: /profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces -[13]: /profiler/enabling/supported_versions/ -[14]: /tracing/trace_collection/compatibility/java/?tab=graalvm#setup -[15]: https://docs.datadoghq.com/profiler/enabling/java/?tab=datadogprofiler# -[16]: /profiler/profiler_troubleshooting/java/ +[2]: /profiler/enabling/supported_versions/ +[3]: https://app.datadoghq.com/account/settings/agent/latest?platform=overview +[4]: https://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html +[5]: /integrations/guide/source-code-integration/?tab=java +[6]: https://app.datadoghq.com/profiling +[7]: /getting_started/tagging/unified_service_tagging +[8]: /profiler/enabling/graalvm/ +[9]: /getting_started/profiler/ +[10]: /profiler/profiler_troubleshooting/java/ +[11]: /profiler/profile_types/?tab=java + diff --git a/content/en/profiler/profiler_troubleshooting/java.md b/content/en/profiler/profiler_troubleshooting/java.md index 12cf691a59b..407b44dc88f 100644 --- a/content/en/profiler/profiler_troubleshooting/java.md +++ b/content/en/profiler/profiler_troubleshooting/java.md @@ -9,6 +9,140 @@ further_reading: text: 'APM Troubleshooting' --- +## Profiler engines + +The Java profiler uses two different profiling engines depending on your operating system: + +| Engine | Platforms | +|--------|-----------| +| **Datadog Profiler** | Linux | +| **Java Flight Recorder (JFR)** | Windows, macOS, Linux (fallback) | + +- **Datadog Profiler**: The Datadog Java profiler was implemented on top of async-profiler and used JVM sampling hooks such as `AsyncGetCallTrace` (and, on Linux, the kernel perf subsystem) to collect stacks and samples. To improve robustness the tracer now performs its own stack unwinding / stack walking in newer agent releases: starting with dd-trace-java 1.55.0 the Datadog profiler uses its own stable stack-walker implementation instead of relying solely on `AsyncGetCallTrace`. That reduces ASGCT-related crashes and gives the profiler more control over unwinding across JVMs and GC modes. +- **Java Flight Recorder (JFR)**: Uses the JVM's built-in flight recorder. Available on all platforms but with fewer profile types. On Linux, used as a fallback when the Datadog Profiler cannot be enabled. + +The profiler automatically selects the appropriate engine based on your environment. You do not need to configure this manually. Below are the different profile types and the related configurations. + +## CPU profiling + +Datadog CPU profiling is scheduled through perf events and is more accurate than JFR CPU profiling. + +| Engine | Environment Variable | System Property | JMC Event | Notes | +|--------|---------------------|-----------------|-----------|-------| +| **Datadog** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_CPU_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.cpu.enabled=true` | `datadog.ExecutionSample` | Default on Linux | +| **JFR** | `DD_PROFILING_DDPROF_CPU_ENABLED=false` | `-Ddd.profiling.ddprof.cpu.enabled=false` | `jdk.ExecutionSample` | To switch from Datadog to JFR (v1.7.0+) | + + + +## Wallclock + +The Datadog profiler wallclock engine: +- Samples all threads, on- or off-CPU, with active tracing activity +- Can be used to diagnose trace or span latency +- Is useful for profiling latency and integrates tightly with APM tracing +- Enabled by default +- Is specific to the Datadog profiler and is not available on Windows + +| Configuration | Environment Variable | System Property | JMC Event | +|---------------|---------------------|-----------------|-----------| +| **Enable** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_WALL_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.wall.enabled=true` | `datadog.MethodSample` | + + +## Allocation profiling + +The Datadog allocation profiling engine: +- Contextualizes allocation profiles with endpoint information +- Supports allocation profiles filtered by endpoint +- Relies on JVMTI APIs; automatically disabled on OpenJDK < 21.0.3 to prevent stability issues +- For JMC users, the Datadog allocation events are `datadog.ObjectAllocationInNewTLAB` and `datadog.ObjectAllocationOutsideTLAB` + +The JFR-based allocation profiling engine: +- Enabled by default since JDK 16 +- Not enabled by default for JDK 8 and 11, as allocation-intensive applications can lead to high overhead and large recording sizes +- For JMC users, the JFR allocation events are `jdk.ObjectAllocationInNewTLAB` and `jdk.ObjectAllocationOutsideTLAB` + +| Engine | Environment Variable | System Property | +|--------|---------------------|-----------------| +| **Datadog** (default) | `DD_PROFILING_DDPROF_ENABLED=true`
`DD_PROFILING_DDPROF_ALLOC_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true`
`-Ddd.profiling.ddprof.alloc.enabled=true` | +| **JFR** (Java 8) | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | +| **JFR** (Java 16) | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationSample` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationSample` | + +**Note**: On Java 15 and lower, the allocation profiler is turned off by default because it can overwhelm the profiler in allocation-heavy applications. Alternatively for JFR, you can enable the following events in your `jfp` override template file +[Learn how to use override templates.](#creating-and-using-a-jfr-template-override-file) + +``` +#Java8 +jdk.ObjectAllocationInNewTLAB#enabled=true +jdk.ObjectAllocationOutsideTLAB#enabled=true + +#Java16 +jdk.ObjectAllocationSample#enabled=true +``` + + +## Live heap profiling + +The live-heap profiler engine: +- Useful for investigating the overall memory usage of service and identifying potential memory leaks +- Samples allocations and keeps track of whether those samples survived the most recent garbage collection cycle +- Uses the number of surviving samples to estimate the number of live objects in the heap +- Limits the number of tracked samples to avoid unbounded growth of the profiler's memory usage +- Disabled by default +- Not available on Windows +- For JMC users, the Datadog live-heap event is `datadog.HeapLiveObject` + +| Configuration | Environment Variable | System Property | +|---------------|---------------------|-----------------| +| **Enable** (Requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+) | `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED=true` | `-Ddd.profiling.ddprof.liveheap.enabled=true` | + +**Note**: The live-heap engine does not depend on the `/proc/sys/kernel/perf_event_paranoid` setting. + + +## Heap Profiling +
This feature requires at least Java 11.0.12, 15.0.4, 16.0.2, 17.0.3 or 18 and newer
+ +To enable the heap profiler, start your application with one of the following: + +| Environment Variable | System Property | +|---------------------|-----------------| +| `DD_PROFILING_HEAP_ENABLED=true` | `-Ddd.profiling.heap.enabled=true` | + +Alternatively, you can enable the following events in your `jfp` [override template file](#creating-and-using-a-jfr-template-override-file): + +``` +jdk.OldObjectSample#enabled=true +``` + + +## Heap histogram metrics + +
This feature requires at least Java 17.0.9 or newer and does not work with ZGC
+ +To enable the heap histogram metrics, start your application with one of the following: + +| Environment Variable | System Property | +|---------------------|-----------------| +| `DD_PROFILING_HEAP_HISTOGRAM_ENABLED=true` | `-Ddd.profiling.heap.histogram.enabled=true` | + + + +## GraalVM native-image + +For applications compiled as GraalVM native images, see [Enabling the Profiler for GraalVM Native Image][3]. + +| Platform | CPU | Allocation | +|----------|:---:|:----------:| +| Linux, Windows, macOS | ✓ | ✓ | + +### Trace to profiling integration + +The [Trace to Profiling integration][4] identifies code hotspots in slow traces. The following minimum versions are required: + +| JDK | Minimum version | dd-trace-java version | +|-----|-----------------|----------------------| +| OpenJDK | 8u352, 11.0.17, 17.0.5 | 1.17.0+ | +| OpenJ9 | 8u362, 11.0.18, 17.0.6 | 1.17.0+ | + ## Missing profiles in the profile search page If you've configured the profiler and don't see profiles in the profile search page, turn on [debug mode][1] and [open a support ticket][2] with debug files and the following information: @@ -19,12 +153,12 @@ If you've configured the profiler and don't see profiles in the profile search p ## Reduce overhead from default setup -If the default setup overhead is not acceptable, you can use the profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default: +If the default setup overhead is not acceptable, you can use profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default: - Increases sampling threshold to 500ms for `ThreadSleep`, `ThreadPark`, and `JavaMonitorWait` events compared to 100ms default - Disables `ObjectAllocationInNewTLAB`, `ObjectAllocationOutsideTLAB`, `ExceptionSample`, `ExceptionCount` events -To use the minimal configuration ensure you have `dd-java-agent` version `0.70.0` then change your service invocation to the following: +To use the minimal configuration change your service ensure you have a recent version of java tracer, then change the service invocation to the following: ``` java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=minimal -jar .jar @@ -37,7 +171,7 @@ If you want more granularity in your profiling data, you can specify the `compre - Reduces sampling threshold to 10ms for `ThreadSleep`, `ThreadPark`, and `JavaMonitorWait` events compared to 100ms default - Enables `ObjectAllocationInNewTLAB`, `ObjectAllocationOutsideTLAB`, `ExceptionSample`, `ExceptionCount` events -To use the comprehensive configuration ensure you have `dd-trace-java` version `0.70.0` then change your service invocation to the following: +To use the comprehensive configuration ensure you have a recent version of java tracer, change your service invocation to the following: ``` java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=comprehensive -jar .jar @@ -126,7 +260,7 @@ The Datadog exception profiler has a small footprint and overhead under normal c To disable exception profiling, start the tracer with the `-Ddd.integration.throwables.enabled=false` JVM setting. -Remember to turn this setting back on after you've returned to a more typical rate of exceptions. +Note: Turn this setting back on after you've returned to a more typical rate of exceptions. ## Java 8 support @@ -207,9 +341,44 @@ Below are basic troubleshooting steps for resolving those issues: ``` DD_PROFILING_TEMPDIR: ``` + +## Summary - Advanced profiler engine configuration + +The following settings allow fine-grained control over the profiler engines. These are typically not needed for standard use cases. For detailed information about each profiler type, see the corresponding sections above: [CPU profiling](#cpu-profiling), [Wallclock](#wallclock), [Allocation profiling](#allocation-profiling), and [Live heap profiling](#live-heap-profiling). + +| Environment variable | System property | Description | +|---------------------|-----------------|-------------| +| `DD_PROFILING_DDPROF_ENABLED` | `-Ddd.profiling.ddprof.enabled` | Enable the Datadog profiler engine (Linux only, default: true since v1.7.0) | +| `DD_PROFILING_DDPROF_CPU_ENABLED` | `-Ddd.profiling.ddprof.cpu.enabled` | Enable CPU profiling with the Datadog engine | +| `DD_PROFILING_DDPROF_WALL_ENABLED` | `-Ddd.profiling.ddprof.wall.enabled` | Enable wallclock profiling (default: true since v1.7.0) | +| `DD_PROFILING_DDPROF_ALLOC_ENABLED` | `-Ddd.profiling.ddprof.alloc.enabled` | Enable allocation profiling with the Datadog engine | +| `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED` | `-Ddd.profiling.ddprof.liveheap.enabled` | Enable live heap profiling (requires JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+) | +| `DD_PROFILING_ENABLED_EVENTS` | `-Ddd.profiling.enabled.events` | Enable specific JFR events (for example: `jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB`) | + +## JDK Mission Control (JMC) event reference + +If you are analyzing profiles with JDK Mission Control, the following table provides a quick reference for events emitted by the profiler. + +| Profile type | JFR event | Datadog event | +|--------------|-----------|---------------| +| CPU | `jdk.ExecutionSample` | `datadog.ExecutionSample` | +| Wallclock | - | `datadog.MethodSample` | +| Allocation | `jdk.ObjectAllocationInNewTLAB`, `jdk.ObjectAllocationOutsideTLAB` | `datadog.ObjectAllocationInNewTLAB`, `datadog.ObjectAllocationOutsideTLAB` | +| Live heap | - | `datadog.HeapLiveObject` | + +### Advanced Linux settings for CPU profiles + +The CPU profiler engine works on most systems, but if the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler can't use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using itimer. Set `/proc/sys/kernel/perf_event_paranoid` to `2` or lower with the following command: + +```shell +sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' +``` + ## Further Reading {{< partial name="whats-next/whats-next.html" >}} [1]: /tracing/troubleshooting/#debugging-and-logging [2]: /help/ +[3]: /profiler/enabling/graalvm/ +[4]: /profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces diff --git a/layouts/partials/profiling/profiling-languages.html b/layouts/partials/profiling/profiling-languages.html index 729df4b448b..4c25e6542c4 100644 --- a/layouts/partials/profiling/profiling-languages.html +++ b/layouts/partials/profiling/profiling-languages.html @@ -16,6 +16,14 @@ +
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/java.png" "class" "img-fluid" "alt" "GraalVM" "width" "400") }} + GraalVM +
+ +