Set `_F_IMG_ID` env var all the way at the bottom during `ImageSpec` builds by redartera · Pull Request #3227 · flyteorg/flytekit

redartera · 2025-04-12T14:03:33Z

Why are the changes needed?

Docker build caching is highly sensitive to the order of operations in a Dockerfile. Currently, the _F_IMG_ID environment variable (which contains a hash of the ImageSpec dataclass) is set near the top of the generated Dockerfile. When this value changes due to modifications in the ImageSpec, it invalidates the cache for that layer and all subsequent layers, causing unnecessary rebuilds of the Python environment even when those layers haven't changed functionally (e.g. python dependencies have remained the same but something changed in the commands or copy block following it)

What changes were proposed in this pull request?

This PR moves the _F_IMG_ID environment variable declaration to the bottom of the generated Dockerfile, after all major build steps (like uv/poetry/pip env installations). This placement ensures that changes to the ImageSpec as a whole only invalidate the layers that are directly affected, preventing costly rebuilds of the Python environment when only commands or other non-environment components change.

How was this patch tested?

The change was tested locally by modifying flytekit and observing Docker build behavior. When the _F_IMG_ID is placed at the bottom of the Dockerfile, changes to components like commands only invalidate the cache for those specific layers and what follows them, preserving the cache for expensive operations like Poetry environment setup.

This improvement reduces build times and registry storage space by maintaining slightly better cache consistency, especially for iterative development workflows where small changes shouldn't trigger full environment rebuilds.

Summary by Bito

This PR optimizes Dockerfile generation by repositioning the _F_IMG_ID environment variable to the end of the build process. This strategic change preserves cache layers for steps unaffected by ImageSpec changes, preventing unnecessary rebuilds when only non-critical configuration changes occur. This significantly improves build times, reduces storage requirements, and enhances overall Docker build performance during development.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 1

flyte-bot · 2025-04-12T14:03:44Z

Code Review Agent Run #e7657b

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: fb9edb3..fb9edb3
- flytekit/image_spec/default_builder.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-04-12T14:05:37Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Optimized Docker Build for Improved Caching	- `default_builder.py` - Repositioned _F_IMG_ID environment variable declaration to the bottom of the Dockerfile and adjusted environment variable setup, thereby refining the build caching mechanism to avoid unnecessary rebuilds.

codecov · 2025-04-12T14:22:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.20%. Comparing base (1bc8302) to head (4853f0c).

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3227       +/-   ##
===========================================
+ Coverage   81.52%   93.20%   +11.68%     
===========================================
  Files         343       37      -306     
  Lines       28627     2370    -26257     
  Branches     2935        0     -2935     
===========================================
- Hits        23338     2209    -21129     
+ Misses       4445      161     -4284     
+ Partials      844        0      -844

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

flyte-bot · 2025-04-12T15:12:15Z

Code Review Agent Run #0b48aa

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: fb9edb3..915badb
- flytekit/image_spec/default_builder.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

Signed-off-by: redartera <reda@artera.ai>

flyte-bot · 2025-04-15T14:22:17Z

Code Review Agent Run #133435

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 915badb..195e795
- flytekit/image_spec/default_builder.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-04-29T12:27:59Z

Code Review Agent Run Status

Limitations and other issues: ❌ Failure - Bito Code Review Skipped - No Changes

Bito didn't review because we did not see any changes in the PR to review.
To trigger review again, type /review in the comment and save.
You can change the settings here, or contact the agent instance creator at eduardo@union.ai.

flyte-bot · 2025-05-02T13:43:10Z

Code Review Agent Run Status

Limitations and other issues: ❌ Failure - Bito Code Review Skipped - No Changes

Bito didn't review because we did not see any changes in the PR to review.
To trigger review again, type /review in the comment and save.
You can change the settings here, or contact the agent instance creator at eduardo@union.ai.

pingsutw

Should we just move $ENV to the end of the Dockerfile? cc @thomasjpfan

thomasjpfan · 2025-05-03T17:50:03Z

Should we just move $ENV to the end of the Dockerfile?

It's hard to say, if one sets any of the UV environment variables, then you'll want the environment variables before the UV install command.

redartera · 2025-05-04T11:24:46Z

Should we just move $ENV to the end of the Dockerfile?

It's hard to say, if one sets any of the UV environment variables, then you'll want the environment variables before the UV install command.

+1 to @thomasjpfan - I preferred to only move _F_IMG_ID because it doesn't seem used during the build but rather at runtime

Signed-off-by: redartera <reda@artera.ai>

flyte-bot · 2025-05-04T12:33:14Z

Code Review Agent Run #b6be55

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 195e795..02c6513
- flytekit/image_spec/default_builder.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-05-06T15:23:12Z

Code Review Agent Run Status

Limitations and other issues: ❌ Failure - Bito Code Review Skipped - No Changes

Bito didn't review because we did not see any changes in the PR to review.
To trigger review again, type /review in the comment and save.
You can change the settings here, or contact the agent instance creator at eduardo@union.ai.

…builds (flyteorg#3227) Signed-off-by: redartera <reda@artera.ai> Signed-off-by: Atharva <atharvakulkarni172003@gmail.com>

redartera requested review from Future-Outlier, cosmicBboy, davidmirror-ops, eapolinario, kumare3, pingsutw, samhita-alla, thomasjpfan and wild-endeavor as code owners April 12, 2025 14:03

redartera force-pushed the default-builder-f-img-id-move-down branch from fb9edb3 to 915badb Compare April 12, 2025 14:08

move _F_IMG_ID to the bottom

195e795

Signed-off-by: redartera <reda@artera.ai>

redartera force-pushed the default-builder-f-img-id-move-down branch from 915badb to 195e795 Compare April 15, 2025 13:17

Merge branch 'master' into default-builder-f-img-id-move-down

4853f0c

Merge branch 'flyteorg:master' into default-builder-f-img-id-move-down

69dbf83

pingsutw reviewed May 3, 2025

View reviewed changes

apply lint

02c6513

Signed-off-by: redartera <reda@artera.ai>

thomasjpfan approved these changes May 4, 2025

View reviewed changes

Merge branch 'flyteorg:master' into default-builder-f-img-id-move-down

2e948c7

pingsutw merged commit 0ef2287 into flyteorg:master May 6, 2025
115 checks passed

Conversation

redartera commented Apr 12, 2025 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Summary by Bito

Uh oh!

flyte-bot commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #e7657b

Uh oh!

flyte-bot commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelist by Bito

Uh oh!

codecov Bot commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

flyte-bot commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #0b48aa

Uh oh!

flyte-bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #133435

Uh oh!

flyte-bot commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run Status

Uh oh!

flyte-bot commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run Status

Uh oh!

pingsutw left a comment

Choose a reason for hiding this comment

Uh oh!

thomasjpfan commented May 3, 2025

Uh oh!

redartera commented May 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flyte-bot commented May 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #b6be55

Uh oh!

flyte-bot commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

redartera commented Apr 12, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Apr 12, 2025 •

edited

Loading

flyte-bot commented Apr 12, 2025 •

edited

Loading

codecov Bot commented Apr 12, 2025 •

edited

Loading

flyte-bot commented Apr 12, 2025 •

edited

Loading

flyte-bot commented Apr 15, 2025 •

edited

Loading

flyte-bot commented Apr 29, 2025 •

edited

Loading

flyte-bot commented May 2, 2025 •

edited

Loading

redartera commented May 4, 2025 •

edited

Loading

flyte-bot commented May 4, 2025 •

edited

Loading

flyte-bot commented May 6, 2025 •

edited

Loading