[Improvement-17986][task-plugin] Support parameter replacement in Flink and FlinkStream task by macdoor · Pull Request #17987 · apache/dolphinscheduler

macdoor · 2026-02-21T12:30:12Z

Purpose of the pull request

Add parameter replacement support for Flink and Flink Stream task plugins. initScript and rawScript can use DolphinScheduler built-in and custom parameters (e.g. ${system.biz.date}, ${localParams.xxx}), consistent with other task types.

close #17986

Brief change log

In FlinkTask and FlinkStreamTask, apply ParameterUtils.convertParameterPlaceholders() to initScript and rawScript in init() using taskExecutionContext.getPrepareParamsMap()
Add unit tests (FlinkTaskTest, FlinkStreamTaskTest) to verify parameter replacement for ${var} placeholders and $[yyyyMMdd] time expressions
No new dependencies; reuses existing ParameterUtils and task API

Verify this pull request

This change added tests and can be verified as follows:

Added FlinkTaskTest and FlinkStreamTaskTest with test cases:
- testInitReplacesPlaceholdersInInitScriptAndRawScript(): Verifies ${var} replacement with custom parameters
- testInitReplacesTimePlaceholderWhenParamMapContainsScheduleTime(): Verifies $[yyyyMMdd] time placeholder replacement
Run tests: mvn test -pl dolphinscheduler-task-plugin/dolphinscheduler-task-flink,dolphinscheduler-task-plugin/dolphinscheduler-task-flink-stream -Dtest=FlinkTaskTest,FlinkStreamTaskTest
Manually verified: Create a Flink (or Flink Stream) task with placeholders in init script or main script (e.g. ${system.biz.date}), run the workflow and confirm the placeholders are replaced correctly at runtime

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

closes #17986

boring-cyborg · 2026-02-21T12:30:14Z

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

SbloodyS

LGTM. Thanks.

ruanwenjun

It's better to move

FileUtils.generateScriptFile(taskExecutionContext, flinkParameters);

at getScript method, then we can avoid use mockito at UT.

We do not recommend using Mockito in UT, using Mockito makes UT difficult to maintain.

dolphinscheduler-task-plugin/dolphinscheduler-task-flink/pom.xml

dolphinscheduler-task-plugin/dolphinscheduler-task-flink-stream/pom.xml

...link-stream/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkStreamTask.java

...eduler-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkTask.java

…cript(), remove Mockito from FlinkTaskTest - Move FileUtils.generateScriptFile() from init() to getScript() - Apply ParameterUtils.convertParameterPlaceholders() in getScript() before script generation - Refactor FlinkTaskTest without Mockito, use @tempdir and assert on generated file content Ref: PR apache#17987 Co-authored-by: Cursor <cursoragent@cursor.com>

…s without Mockito - Move FileUtils.generateScriptFile() from init() to getScript() - Apply ParameterUtils.convertParameterPlaceholders() in getScript() before script generation - Refactor FlinkTaskTest and FlinkStreamTaskTest without Mockito - Use @tempdir and assert on generated file content Ref: PR apache#17987 Co-authored-by: Cursor <cursoragent@cursor.com>

…nk-stream junit-jupiter, mockito-junit-jupiter, mockito-inline are already in root pom Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

…e site - Remove param replacement from init(), apply in resolveFlinkJdbcUrl/resolveInitScriptContent/resolveMainScriptContent - Consistent with PR apache#17987 review: move replacement to where values are used Co-authored-by: Cursor <cursoragent@cursor.com>

...-stream/src/test/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkStreamTaskTest.java

…subclasses Same package allows direct access to protected getScript() Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

…s without Mockito - Move FileUtils.generateScriptFile() from init() to getScript() - Apply ParameterUtils.convertParameterPlaceholders() in getScript() before script generation - Refactor FlinkTaskTest and FlinkStreamTaskTest without Mockito - Use @tempdir and assert on generated file content Ref: PR apache#17987 Co-authored-by: Cursor <cursoragent@cursor.com>

…nk-stream junit-jupiter, mockito-junit-jupiter, mockito-inline are already in root pom Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

…subclasses Same package allows direct access to protected getScript() Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

...er-task-flink/src/test/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkTaskTest.java

...-stream/src/test/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkStreamTaskTest.java

…ue(contains) in Flink tests - Replace assertTrue(contains) with assertEquals in FlinkTaskTest and FlinkStreamTaskTest - Remove class-level Javadoc per review Ref: PR apache#17987 Co-authored-by: Cursor <cursoragent@cursor.com>

…nk and FlinkStream task

…d unit tests for parameter replacement - Always apply parameter replacement even when paramsMap is empty (to support time placeholders like $[yyyyMMdd]) - Replace Chinese comments with English - Add FlinkTaskTest and FlinkStreamTaskTest with test cases for parameter replacement Co-authored-by: Cursor <cursoragent@cursor.com>

…nk and FlinkStream task - Replace parameter placeholders in initScript and rawScript for FlinkTask and FlinkStreamTask - Add unit tests for parameter replacement and time placeholders with empty params - Add test dependencies to task-flink and task-flink-stream modules

…s without Mockito - Move FileUtils.generateScriptFile() from init() to getScript() - Apply ParameterUtils.convertParameterPlaceholders() in getScript() before script generation - Refactor FlinkTaskTest and FlinkStreamTaskTest without Mockito - Use @tempdir and assert on generated file content Ref: PR apache#17987 Co-authored-by: Cursor <cursoragent@cursor.com>

…nk-stream junit-jupiter, mockito-junit-jupiter, mockito-inline are already in root pom Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

…subclasses Same package allows direct access to protected getScript() Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

…ue(contains) in Flink tests - Replace assertTrue(contains) with assertEquals in FlinkTaskTest and FlinkStreamTaskTest - Remove class-level Javadoc per review Ref: PR apache#17987 Co-authored-by: Cursor <cursoragent@cursor.com>

ruanwenjun

LGTM

SbloodyS

Please fix failed UT. @macdoor

…va 8 - Replace Path.of() with Paths.get() (Path.of is Java 11+) - Replace Files.readString() with Files.readAllBytes() + new String() - Ensures tests pass on both Java 8 and Java 11 Made-with: Cursor

Made-with: Cursor

macdoor · 2026-02-26T08:05:03Z

Hi @SbloodyS
Could you please approve the workflows so the CI can run? THX

Regarding the previous failed UT (Flink task-flink compilation):
The issue was caused by using Java 11-only APIs (Path.of(), Files.readString()). I’ve changed them to Java 8–compatible APIs (Paths.get(), Files.readAllBytes()). Unit tests now pass locally on both Java 8 and Java 11.

Note: Any E2E failure (e.g. PythonTaskE2ETest.testRunPythonTasks_UsingResourceFile with StaleElementReferenceException) is unrelated to this PR, which only modifies the Flink/FlinkStream task plugins and their unit tests.

SbloodyS · 2026-02-26T08:31:55Z

I've restarted CI. @macdoor

…o reduce SonarQube duplication - Add buildScriptWithParameterReplacement() in FlinkTask - FlinkStreamTask delegates to parent method instead of duplicating logic - Remove unused Property import in FlinkStreamTask Made-with: Cursor

macdoor · 2026-02-26T10:02:48Z

Hi @SbloodyS
another CI. Thx
I extract duplicated getScript logic to reduce SonarQube duplication.

sonarqubecloud · 2026-02-26T14:49:32Z

Quality Gate passed

Issues
10 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

macdoor requested review from Gallardot, SbloodyS and caishunfeng as code owners February 21, 2026 12:30

github-actions bot added backend test labels Feb 21, 2026

github-actions bot assigned macdoor Feb 21, 2026

SbloodyS added first time contributor First-time contributor improvement make more easy to user or prompt friendly labels Feb 22, 2026

SbloodyS added this to the 3.4.1 milestone Feb 22, 2026

SbloodyS previously approved these changes Feb 24, 2026

View reviewed changes

SbloodyS requested a review from ruanwenjun February 24, 2026 02:50

ruanwenjun reviewed Feb 24, 2026

View reviewed changes

macdoor dismissed SbloodyS’s stale review via e245f35 February 24, 2026 11:51

macdoor force-pushed the improvement-17986 branch from 9a16038 to 4e40f8d Compare February 24, 2026 12:07

ruanwenjun reviewed Feb 25, 2026

View reviewed changes

...-stream/src/test/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkStreamTaskTest.java Outdated Show resolved Hide resolved

ruanwenjun force-pushed the improvement-17986 branch from 6554359 to bfd71ea Compare February 25, 2026 03:59

ruanwenjun reviewed Feb 25, 2026

View reviewed changes

xiaoxiong.duan and others added 3 commits February 25, 2026 13:44

[Improvement-17986][task-plugin] Support parameter replacement in Fli…

cf7a5f4

…nk and FlinkStream task

段晓雄 and others added 4 commits February 25, 2026 13:44

chore(flink): remove duplicate test deps from task-flink and task-fli…

a3d03d3

…nk-stream junit-jupiter, mockito-junit-jupiter, mockito-inline are already in root pom Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

refactor(flink-test): call task.getScript() directly, remove ForTest …

0ad461f

…subclasses Same package allows direct access to protected getScript() Ref: PR apache#17987 review Co-authored-by: Cursor <cursoragent@cursor.com>

ruanwenjun force-pushed the improvement-17986 branch from 2ebdd88 to 2660988 Compare February 25, 2026 05:44

Merge branch 'dev' into improvement-17986

de2e1a4

ruanwenjun previously approved these changes Feb 25, 2026

View reviewed changes

Merge branch 'dev' into improvement-17986

72592b2

SbloodyS reviewed Feb 26, 2026

View reviewed changes

[Improvement-17986][task-plugin] Fix Flink test compatibility with Ja…

d439b5c

…va 8 - Replace Path.of() with Paths.get() (Path.of is Java 11+) - Replace Files.readString() with Files.readAllBytes() + new String() - Ensures tests pass on both Java 8 and Java 11 Made-with: Cursor

macdoor dismissed ruanwenjun’s stale review via d439b5c February 26, 2026 05:33

ci: trigger Unit Test workflow

19ba74d

Made-with: Cursor

Conversation

macdoor commented Feb 21, 2026 • edited by SbloodyS Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

boring-cyborg bot commented Feb 21, 2026

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

ruanwenjun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ruanwenjun left a comment

Choose a reason for hiding this comment

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

macdoor commented Feb 26, 2026

Uh oh!

SbloodyS commented Feb 26, 2026

Uh oh!

macdoor commented Feb 26, 2026

Uh oh!

sonarqubecloud bot commented Feb 26, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

macdoor commented Feb 21, 2026 •

edited by SbloodyS

Loading

ruanwenjun left a comment •

edited

Loading