appsec: add smoke tests for apm standalone by florentinl · Pull Request #6181 · DataDog/system-tests

florentinl · 2026-02-02T14:02:30Z

Motivation

Test that AppSec is correctly enabled and functional in APM Standalone mode (with infra disabled at the agent level).

To do so, I setup a testing class that performs AAP smoke tests using only data intercepted at the agent level. This testing class can then be used through inheritance to create different test suites linked to different features and manifest entries.

Changes

Add appsec helpers in interfaces.agent to perform assertions on traces intercepted after the agent
Add smoke tests to check:
- Threat detection
- RASP
- Remote configuration of rules
- Telemetry emission
- API Security
  => This ensures that all communication protocols used by AAP (traces, trace stats, telemetry and RC) keep getting proxied by the agent.
Add two scenarios and a feature for APM Standalone and ASM Standalone + APM Standalone

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-02-02T14:03:01Z

CODEOWNERS have been resolved as:

tests/appsec/smoke_tests/__init__.py                                    @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/smoke_tests/test_apm_standalone.py                         @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/smoke_tests/utils.py                                       @DataDog/asm-libraries @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
docs/edit/agent-interface-validation-methods.md                         @DataDog/system-tests-core
manifests/agent.yml                                                     @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/interfaces/_agent.py                                              @DataDog/system-tests-core
utils/proxy/_deserializer.py                                            @DataDog/system-tests-core
utils/proxy/traces/trace_v1.py                                          @DataDog/system-tests-core

datadog-official · 2026-02-02T14:24:26Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Other Violations

❄️ 3 New flaky tests detected

    tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_ApiSecurity.test_api_security_smoke[spring-boot-wildfly] from system_tests_suite (Fix with Cursor)

    tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_Rasp.test_lfi_smoke[spring-boot-wildfly] from system_tests_suite (Fix with Cursor)

    tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_Threats.test_attack_detection_smoke[spring-boot-wildfly] from system_tests_suite (Fix with Cursor)

🧪 1 Test failed

tests.remote_config.test_remote_configuration.Test_RemoteConfigurationUpdateSequenceLiveDebugging.test_tracer_update_sequence[apache-mod-7.4-zts]

from system_tests_suite

(Fix with Cursor)

utils.interfaces._core.ValidationError: ("{'path': 'datadog/2/LIVE_DEBUGGING/metricProbe_33a64d99-fbed-5eab-bb10-80735405c09b/config', 'length': 360, 'hashes': [{'algorithm': 'sha256', 'hash': '6daaa0eb13996d340d99983bb014ef17453bad39edf19041f24a87a159ff94fe'}]} should be in cached_target_files property: [{'path': 'datadog/2/LIVE_DEBUGGING/logProbe_22953c88-eadc-4f9a-aa0f-7f6243f4bf8a/config', 'length': 239, 'hashes': [{'algorithm': 'sha256', 'hash': '8176095e451a5f4d49db40e5eadf7d79b0ca6956cf28c83f87d18f4d66ea2583'}]}, {'path': 'datadog/2/LIVE_DEBUGGING/metricProbe_33a64d99-fbed-5eab-bb10-80735405c09b/config', 'length': 365, 'hashes': [{'algorithm': 'sha256', 'hash': '4f12b33894fd7178f2464d3fc2c63223c3ee2a29a5cf0936de60ceee88fd0656'}]}, {'path': 'datadog/2/LIVE_DEBUGGING/spanProbe_kepf0cf2-9top-45cf-9f39-59installed/config', 'length': 188, 'hashes': [{'algorithm': 'sha256', 'hash': 'd22df7cf36e9f2b0134c4f6535a7340b9a4435876b79280f91d80942c9562b5b'}]}]", 'SUCCESS - Add back the initial config along with the second (add multiple). RFC about integrating with remote-config: https://docs.google.com/document/d/1u_G7TOr8wJX0dOM_zUDKuRJgxoJU_hVTd5SeaMucQUs')

self = <tests.remote_config.test_remote_configuration.Test_RemoteConfigurationUpdateSequenceLiveDebugging object at 0x7f38685fb4a0>

    def test_tracer_update_sequence(self):
        """Test update sequence, based on a scenario mocked in the proxy"""
    
        # Index the request number by runtime ID so that we can support applications
        # that spawns multiple worker processes, each running its own RCM client.
        request_number: dict = defaultdict(int)
...

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 99c6401 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

docs/edit/agent-interface-validation-methods.md

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 834583e22b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tests/appsec/smoke_tests/utils.py

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aba9c84c94

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

utils/_context/_scenarios/__init__.py

cbeauchesne

Just a small naming request, then all good.

tests/appsec/smoke_tests/utils.py

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0326ddddad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

manifests/dotnet.yml

…s flaky Separate test classes for each scenario so they can have independent manifest declarations: - Test_AppSecAPMStandalone_*: APPSEC_APM_STANDALONE (tracing enabled) - Test_AppSecStandaloneAPMStandalone_*: APPSEC_STANDALONE_APM_STANDALONE (tracing disabled — traces without appsec data silently dropped) Both sets use identical clean base classes with no workarounds. The Java standalone variants for Threats, Rasp, ApiSecurity, and UserEvents are marked flaky (APPSEC-60872): the Java tracer drops traces from the first request to each code path when DD_APM_TRACING_ENABLED=false. Updated manifests: java, golang, nodejs, php, ruby, cpp_nginx. File-level entries (dotnet, python, agent) already cover both class sets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a6383aa34

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

manifests/ruby.yml

…stem-tests

cbeauchesne · 2026-04-10T16:24:34Z

tests/appsec/smoke_tests/utils.py

+
+"""AppSec smoke tests for the appsec_apm_standalone scenario."""
+
+from utils import interfaces, remote_config as rc, weblog


Just to understand, why do you need base classes here?

So the comment should be removed, thanks for spotting it. Base classes are more generic than just apm standalone.

The idea is to have base classes for generic smoke tests and instantiate them for each configuration we want to test (the first two being "apm standalone" and "apm standalone + appsec standalone" in this PR).

My point is to be able to instantiate the same test in different scenarios and be able to have different manifest entries for each test in each scenario. To allow passing the same test with a configuration but not necessarily with another.

Is there a more idiomatic way to handle this ?

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from 8b69cf1 to e67144b Compare February 2, 2026 14:17

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch 2 times, most recently from 20a2e93 to 8a60060 Compare March 5, 2026 10:55

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch 4 times, most recently from fd4926c to 8679f7a Compare March 26, 2026 14:15

florentinl marked this pull request as ready for review March 27, 2026 14:59

florentinl requested review from a team as code owners March 27, 2026 14:59

florentinl requested review from jandro996, manuel-alvarez-alvarez, r1viollet and taegyunkim and removed request for a team March 27, 2026 14:59

florentinl requested a review from cbeauchesne April 7, 2026 07:38

avara1986 approved these changes Apr 7, 2026

View reviewed changes

christophe-papazian reviewed Apr 7, 2026

View reviewed changes

docs/edit/agent-interface-validation-methods.md Outdated Show resolved Hide resolved

christophe-papazian approved these changes Apr 7, 2026

View reviewed changes

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from e58bf30 to 834583e Compare April 7, 2026 11:03

chatgpt-codex-connector bot reviewed Apr 7, 2026

View reviewed changes

tests/appsec/smoke_tests/utils.py Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Apr 8, 2026

View reviewed changes

utils/_context/_scenarios/__init__.py Show resolved Hide resolved

cbeauchesne requested changes Apr 8, 2026

View reviewed changes

tests/appsec/smoke_tests/utils.py Outdated Show resolved Hide resolved

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from aba9c84 to 0326ddd Compare April 8, 2026 11:18

chatgpt-codex-connector bot reviewed Apr 8, 2026

View reviewed changes

manifests/dotnet.yml Outdated Show resolved Hide resolved

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from 0326ddd to 09567d7 Compare April 8, 2026 11:26

florentinl added 11 commits April 9, 2026 17:45

appsec: add smoke tests for apm standalone

de65962

comment to fill things

5c24214

enable tests for the different tracers

553c2e9

fix format

37dbb5d

v1 trace payload support + small refactor

2d0ae2b

linter

8bc514a

move metastruct deserialization to the proxy

0f95705

add extra coverage

0f7857d

update manifests

77f4736

update docs

006ab1f

move files + renaming

5653051

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from 09567d7 to 6888a95 Compare April 9, 2026 15:45

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from 6888a95 to 2a6383a Compare April 9, 2026 15:47

chatgpt-codex-connector bot reviewed Apr 9, 2026

View reviewed changes

manifests/ruby.yml Show resolved Hide resolved

fix ruby manifest

bda2893

florentinl requested a review from cbeauchesne April 10, 2026 09:41

Merge branch 'main' into florentin.labelle/APPSEC-60883/infra-less-sy…

99c6401

…stem-tests

cbeauchesne reviewed Apr 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

appsec: add smoke tests for apm standalone#6181

appsec: add smoke tests for apm standalone#6181
florentinl wants to merge 14 commits intomainfrom
florentin.labelle/APPSEC-60883/infra-less-system-tests

florentinl commented Feb 2, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 2, 2026 •

edited

Loading

Uh oh!

datadog-official bot commented Feb 2, 2026 •

edited by datadog-prod-us1-4 bot

Loading

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

cbeauchesne left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

cbeauchesne Apr 10, 2026

Uh oh!

florentinl Apr 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		"""AppSec smoke tests for the appsec_apm_standalone scenario."""

		from utils import interfaces, remote_config as rc, weblog

Conversation

florentinl commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official bot commented Feb 2, 2026 • edited by datadog-prod-us1-4 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Other Violations

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cbeauchesne Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

florentinl Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

florentinl commented Feb 2, 2026 •

edited

Loading

github-actions bot commented Feb 2, 2026 •

edited

Loading

datadog-official bot commented Feb 2, 2026 •

edited by datadog-prod-us1-4 bot

Loading

florentinl Apr 10, 2026 •

edited

Loading