Bugfix for issue #6405: Fast registration runs into infinite loop by rambrus · Pull Request #3230 · flyteorg/flytekit

rambrus · 2025-04-15T09:14:44Z

Tracking issue

Why are the changes needed?

In Spark fast registration workflows, entrypoint.py runs into an infinite loop while preparing to run the job

flytekit is taking everything in the current working directory and zip it, the archive file (flyte_wf.zip) is placed in the same directory that is being archived.

shutil.make_archive creates a zip file and archives the whole directory, including the zip file itself. In the case of Databricks clusters, it triggers a recursion and runs into an endless zipping loop.

What changes were proposed in this pull request?

The easiest way to fix this issue to create the archive outside of the directory to be archived.

How was this patch tested?

Built a docker image with the fixed flytekit version and tested it E2E.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

Summary by Bito

This PR fixes a critical bug in Spark fast registration workflow that caused an infinite loop due to recursive archiving. The solution moves archive creation to a temporary directory, preventing the archive file from including itself. The changes also streamline function parameters and improve mock assertions for better test reliability.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 1

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

welcome · 2025-04-15T09:14:48Z

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

flyte-bot · 2025-04-15T09:14:56Z

Code Review Agent Run #5b67c9

Actionable Suggestions - 1

plugins/flytekit-spark/flytekitplugins/spark/task.py - 1
- Temporary directory immediately deleted after creation · Line 241-241

Review Details

Files reviewed - 1 · Commit Range: 6d7148e..6d7148e
- plugins/flytekit-spark/flytekitplugins/spark/task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-04-15T09:18:02Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Bug Fix - Resolve Recursive Zipping Bug	- `task.py` - Replaced archiving in working directory with a temporary directory to prevent recursive zipping and infinite loop.
Testing - Enhance Test Coverage for Spark Task	- `test_spark_task.py` - Updated tests to mock temporary directory creation and archive functions, ensuring proper call validations for the new archiving process.

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

rambrus · 2025-04-15T09:30:15Z

/review

flyte-bot · 2025-04-15T09:30:27Z

Code Review Agent Run #2fd235

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 6d7148e..1172159
- plugins/flytekit-spark/flytekitplugins/spark/task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

pingsutw

Thank you!

flyte-bot · 2025-04-15T10:32:29Z

Code Review Agent Run #15f705

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 6d7148e..1172159
- plugins/flytekit-spark/flytekitplugins/spark/task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

codecov · 2025-04-15T11:37:49Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 52.32%. Comparing base (8abecfe) to head (854d54a).

❗ There is a different number of reports uploaded between BASE (8abecfe) and HEAD (854d54a). Click for more details.

HEAD has 26 uploads less than BASE

Flag BASE (8abecfe) HEAD (854d54a)

28 2

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3230       +/-   ##
===========================================
- Coverage   89.81%   52.32%   -37.49%     
===========================================
  Files         136      214       +78     
  Lines        6599    22371    +15772     
  Branches        0     2925     +2925     
===========================================
+ Hits         5927    11706     +5779     
- Misses        672     9979     +9307     
- Partials        0      686      +686

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

rambrus · 2025-04-15T14:51:05Z

/review

flyte-bot · 2025-04-15T14:51:16Z

Code Review Agent Run #044a16

Actionable Suggestions - 0

Review Details

Files reviewed - 2 · Commit Range: 6d7148e..854d54a
- plugins/flytekit-spark/flytekitplugins/spark/task.py
- plugins/flytekit-spark/tests/test_spark_task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-04-15T15:52:19Z

Code Review Agent Run #1b7b4f

Actionable Suggestions - 0

Review Details

Files reviewed - 2 · Commit Range: 6d7148e..854d54a
- plugins/flytekit-spark/flytekitplugins/spark/task.py
- plugins/flytekit-spark/tests/test_spark_task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

flyte-bot · 2025-04-16T06:52:17Z

Code Review Agent Run #8a0a73

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 854d54a..cb153b4
- plugins/flytekit-spark/tests/test_spark_task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

welcome · 2025-04-17T04:34:29Z

Congrats on merging your first pull request! 🎉

…yteorg#3230) * Bugfix for issue #6405: Fast registration runs into infinite loop Signed-off-by: rambrus <v-rambrus@expediagroup.com> * Replace tempfile.TemporaryDirectory().name with tempfile.mkdtemp() Signed-off-by: rambrus <v-rambrus@expediagroup.com> * Update and add unit tests Signed-off-by: rambrus <v-rambrus@expediagroup.com> * Merge unit tests, fix lint issues Signed-off-by: rambrus <v-rambrus@expediagroup.com> --------- Signed-off-by: rambrus <v-rambrus@expediagroup.com> Signed-off-by: Atharva <atharvakulkarni172003@gmail.com>

Bugfix for issue #6405: Fast registration runs into infinite loop

6d7148e

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

rambrus requested review from Future-Outlier, cosmicBboy, davidmirror-ops, eapolinario, kumare3, pingsutw, samhita-alla, thomasjpfan and wild-endeavor as code owners April 15, 2025 09:14

rambrus mentioned this pull request Apr 15, 2025

[BUG] Fast registration Databricks jobs run into infinite loop with flytekit 1.14.7 flyteorg/flyte#6405

Closed

flyte-bot reviewed Apr 15, 2025

View reviewed changes

Comment thread plugins/flytekit-spark/flytekitplugins/spark/task.py Outdated

Replace tempfile.TemporaryDirectory().name with tempfile.mkdtemp()

1172159

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

pingsutw previously approved these changes Apr 15, 2025

View reviewed changes

Update and add unit tests

854d54a

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

rambrus dismissed pingsutw’s stale review via 854d54a April 15, 2025 14:47

Future-Outlier previously approved these changes Apr 15, 2025

View reviewed changes

Merge unit tests, fix lint issues

cb153b4

Signed-off-by: rambrus <v-rambrus@expediagroup.com>

rambrus dismissed Future-Outlier’s stale review via cb153b4 April 16, 2025 05:46

Future-Outlier approved these changes Apr 17, 2025

View reviewed changes

Future-Outlier merged commit 26db7a3 into flyteorg:master Apr 17, 2025
116 checks passed

Conversation

rambrus commented Apr 15, 2025 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

Summary by Bito

Uh oh!

welcome Bot commented Apr 15, 2025

Uh oh!

flyte-bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #5b67c9

Uh oh!

flyte-bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelist by Bito

Uh oh!

Uh oh!

rambrus commented Apr 15, 2025

Uh oh!

flyte-bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #2fd235

Uh oh!

pingsutw left a comment

Choose a reason for hiding this comment

Uh oh!

flyte-bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #15f705

Uh oh!

codecov Bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rambrus commented Apr 15, 2025

Uh oh!

flyte-bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #044a16

Uh oh!

flyte-bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #1b7b4f

Uh oh!

flyte-bot commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #8a0a73

Uh oh!

Uh oh!

welcome Bot commented Apr 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rambrus commented Apr 15, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Apr 15, 2025 •

edited

Loading

flyte-bot commented Apr 15, 2025 •

edited

Loading

flyte-bot commented Apr 15, 2025 •

edited

Loading

flyte-bot commented Apr 15, 2025 •

edited

Loading

codecov Bot commented Apr 15, 2025 •

edited

Loading

flyte-bot commented Apr 15, 2025 •

edited

Loading

flyte-bot commented Apr 15, 2025 •

edited

Loading

flyte-bot commented Apr 16, 2025 •

edited

Loading