Image spec builder options by mhotan · Pull Request #3233 · flyteorg/flytekit

mhotan · 2025-04-18T20:36:31Z

Why are the changes needed?

This PR provides the ability to specify image builder specific options to Image Spec.

from flytekit import ImageSpec, task

image = ImageSpec(
    base_image="localhost:30000/flytekit:0.1.5", 
    builder_options={"option": "custom_build_option"}
)

@task(container_image=image)
def main() -> int:
    return 10

In a more realistic example, Union image builder is looking the ability to configure and set registry credentials through this attribute.

from flytekit import ImageSpec, task

# Assume we uploaded credentials
# union create secret --type imagepullsecrets --value-file /path/to/credentials.json  my-custom-credentials

image = ImageSpec(
    base_image="myprivateregistry.com/flytekit:0.1.5", 
    builder_options={"imagepull_secret_name": "my-custom-credentials"}
)

@task(container_image=image, secret_requests=[Secret(key="my-custom-credentials""))
def main() -> int:
    return 10

What changes were proposed in this pull request?

This PR adds:

The ability to specify 'builder_options` dictionary to image spec
A imagespec.with_builder_options to additively accumulate builder_options

How was this patch tested?

Additional unit tests

Summary by Bito

This PR enhances ImageSpec functionality by enabling dictionary-based builder-specific options with type validation and clearer error messages. It improves error handling in image_spec.py and updates docstring formatting to meet pydoclint standards, reducing discrepancies between documented and actual attributes. New test cases verify proper handling of invalid builder_options inputs, strengthening the robustness, reliability and clarity of the image specification API.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 1

welcome · 2025-04-18T20:36:34Z

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

flyte-bot · 2025-04-18T20:36:42Z

Code Review Agent Run #83541f

Actionable Suggestions - 1

flytekit/image_spec/image_spec.py - 1
- Missing validation for new builder_options parameter · Line 106-106

Additional Suggestions - 7

flytekit/image_spec/image_spec.py - 7
- Use modern type annotation syntax · Line 14-14
- Improve exception handling practices · Line 337-337
- Improve docstring formatting and style · Line 30-77
- Add return type and docstring · Line 108-108
- Fix docstring and ternary operator usage · Line 323-326
- Add type annotation for kwargs · Line 398-398
- Remove blank line after docstring · Line 399-399

Review Details

Files reviewed - 3 · Commit Range: 2b8f16d..2b8f16d
- flytekit/image_spec/image_spec.py
- pydoclint-errors-baseline.txt
- tests/flytekit/unit/core/image_spec/test_image_spec.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-04-18T20:41:49Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Enhanced Image Specification	- `image_spec.py` - Added support for builder_options with type validation, improved error handling, and comprehensive docstring updates to better describe class attributes. - `test_image_spec.py` - Introduced new tests covering both positive and negative scenarios for the builder_options functionality, ensuring robust behavior.
Documentation - Updated Documentation for Compliance	- `pydoclint-errors-baseline.txt` - Revised and aligned class and method documentation to meet pydoclint standards, addressing discrepancies in documented attributes.

codecov · 2025-04-18T20:47:05Z

Codecov Report

Attention: Patch coverage is 57.14286% with 9 lines in your changes missing coverage. Please review.

Project coverage is 79.19%. Comparing base (9b100f0) to head (3efb7de).

Files with missing lines	Patch %	Lines
flytekit/image_spec/image_spec.py	57.14%	4 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3233       +/-   ##
===========================================
+ Coverage   52.32%   79.19%   +26.87%     
===========================================
  Files         215      215               
  Lines       22391    22408       +17     
  Branches     2927     2934        +7     
===========================================
+ Hits        11716    17747     +6031     
+ Misses       9988     3817     -6171     
- Partials      687      844      +157

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mhotan · 2025-04-18T20:56:09Z

-        pip_index: Specify the custom pip index url
-        pip_extra_index_url: Specify one or more pip index urls as a list
-        pip_secret_mounts: Specify a list of tuples to mount secret for pip install. Each tuple should contain the path to
+    Attributes:


Updated all this Doc to fix linter

cc @ppiegaze - your upcoming flytekit PR renders these changes moot right? Along with the monodocs build? I've been out of the loop on the doc building stuff for a while, do you know what the plan for that is? or maybe @cosmicBboy?

For context, I was doing my best here to get tests passing under the assumption that I was upholding the current standards. I am not aware of ongoing changes and potential CICD inconsistencies.

thomasjpfan

Minor comments, otherwise LGTM

flyte-bot · 2025-04-18T21:42:15Z

Code Review Agent Run #413df0

Actionable Suggestions - 0

Review Details

Files reviewed - 3 · Commit Range: 2b8f16d..7dc2335
- flytekit/image_spec/image_spec.py
- pydoclint-errors-baseline.txt
- tests/flytekit/unit/core/image_spec/test_image_spec.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

kumare3 · 2025-04-18T22:04:39Z

LGTM

flyte-bot · 2025-04-18T22:52:14Z

Code Review Agent Run #30b4ef

Actionable Suggestions - 0

Additional Suggestions - 1

flytekit/image_spec/image_spec.py - 1
- String literal used in exception raising · Line 159-159

Review Details

Files reviewed - 3 · Commit Range: 7dc2335..3efb7de
- flytekit/image_spec/image_spec.py
- pydoclint-errors-baseline.txt
- tests/flytekit/unit/core/image_spec/test_image_spec.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

wild-endeavor

just a couple questions. This will not be used at all by the default builder correct?

wild-endeavor · 2025-04-19T03:00:34Z

-        pip_index: Specify the custom pip index url
-        pip_extra_index_url: Specify one or more pip index urls as a list
-        pip_secret_mounts: Specify a list of tuples to mount secret for pip install. Each tuple should contain the path to
+    Attributes:


cc @ppiegaze - your upcoming flytekit PR renders these changes moot right? Along with the monodocs build? I've been out of the loop on the doc building stuff for a while, do you know what the plan for that is? or maybe @cosmicBboy?

wild-endeavor · 2025-04-19T03:01:17Z

            return None

-    def _update_attribute(self, attr_name: str, values: Union[str, List[str]]) -> "ImageSpec":
+    def _update_attribute(self, attr_name: str, values: Union[str, List[str], Dict[str, Any]]) -> "ImageSpec":


should the type hint here (and in the class definition) really be dict[str, Any] rather than just dict[str, str]? What are the other types users might add?

I can see a future, however don't have a specific example, where int and potentially List[str] be used but it seems builder dependent. Happy to constrain to Dict[str, str] and we could later decide to move toward Dict[str, Any] if we find compelling future cases.

mhotan · 2025-04-21T22:34:57Z

just a couple questions. This will not be used at all by the default builder correct?

Correct, not be used by the default builder. That being said, it is not currently restricted for future use at this time

Provide the ability to specify image `builder` specific options per Image Spec. Signed-off-by: Mike Hotan <mike@union.ai>

Signed-off-by: Mike Hotan <mike@union.ai>

flyte-bot · 2025-04-21T23:42:16Z

Code Review Agent Run #c907a9

Actionable Suggestions - 0

Additional Suggestions - 1

flytekit/image_spec/image_spec.py - 1
- String literal used directly in exception · Line 159-159

Review Details

Files reviewed - 3 · Commit Range: 3efb7de..d6b6494
- flytekit/image_spec/image_spec.py
- pydoclint-errors-baseline.txt
- tests/flytekit/unit/core/image_spec/test_image_spec.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at haytham@union.ai.

Documentation & Help

AI Code Review powered by

welcome · 2025-04-22T17:15:52Z

Congrats on merging your first pull request! 🎉

* Image spec builder options Provide the ability to specify image `builder` specific options per Image Spec. Signed-off-by: Mike Hotan <mike@union.ai> * Add builder_options validation Signed-off-by: Mike Hotan <mike@union.ai> * updates Signed-off-by: Mike Hotan <mike@union.ai> --------- Signed-off-by: Mike Hotan <mike@union.ai>

* Adds ImageSpec.with_runtime_packages (#3231) * Adds ImageSpec.with_dev_dependencies Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Fix Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Add tests for noop builder Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Use runtime_packages Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Add docs abount how to use runtime packages Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Less diffs Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Fix formatting Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Fix docstring Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Dix docstring Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Let pip default to user by itself to be more compatible Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> --------- Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Image spec builder options (#3233) * Image spec builder options Provide the ability to specify image `builder` specific options per Image Spec. Signed-off-by: Mike Hotan <mike@union.ai> * Add builder_options validation Signed-off-by: Mike Hotan <mike@union.ai> * updates Signed-off-by: Mike Hotan <mike@union.ai> --------- Signed-off-by: Mike Hotan <mike@union.ai> --------- Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> Signed-off-by: Mike Hotan <mike@union.ai> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>

* Image spec builder options Provide the ability to specify image `builder` specific options per Image Spec. Signed-off-by: Mike Hotan <mike@union.ai> * Add builder_options validation Signed-off-by: Mike Hotan <mike@union.ai> * updates Signed-off-by: Mike Hotan <mike@union.ai> --------- Signed-off-by: Mike Hotan <mike@union.ai> Signed-off-by: Atharva <atharvakulkarni172003@gmail.com>

mhotan requested review from Future-Outlier, cosmicBboy, davidmirror-ops, eapolinario, kumare3, pingsutw, samhita-alla, thomasjpfan and wild-endeavor as code owners April 18, 2025 20:36

mhotan force-pushed the mhotan/image_spec_builder_options branch from 2b8f16d to e84b0dc Compare April 18, 2025 20:37

mhotan force-pushed the mhotan/image_spec_builder_options branch from 24ce8ae to 7dc2335 Compare April 18, 2025 20:55

mhotan commented Apr 18, 2025

View reviewed changes

thomasjpfan previously approved these changes Apr 18, 2025

View reviewed changes

Comment thread flytekit/image_spec/image_spec.py Outdated

Comment thread tests/flytekit/unit/core/image_spec/test_image_spec.py

mhotan dismissed thomasjpfan’s stale review via 17157d2 April 18, 2025 21:44

mhotan force-pushed the mhotan/image_spec_builder_options branch from 17157d2 to 3efb7de Compare April 18, 2025 21:45

kumare3 approved these changes Apr 18, 2025

View reviewed changes

flyteorg deleted a comment from flyte-bot Apr 19, 2025

wild-endeavor approved these changes Apr 19, 2025

View reviewed changes

mhotan added 3 commits April 22, 2025 08:39

Image spec builder options

b567cfe

Provide the ability to specify image `builder` specific options per Image Spec. Signed-off-by: Mike Hotan <mike@union.ai>

Add builder_options validation

bc16a2c

Signed-off-by: Mike Hotan <mike@union.ai>

updates

d6b6494

Signed-off-by: Mike Hotan <mike@union.ai>

mhotan force-pushed the mhotan/image_spec_builder_options branch from 3efb7de to d6b6494 Compare April 21, 2025 22:39

mhotan merged commit bba955f into master Apr 22, 2025
115 of 116 checks passed

wild-endeavor mentioned this pull request Apr 22, 2025

[v1.15 Backport] image spec builder options 3233 & 3231 #3239

Closed

mhotan mentioned this pull request Apr 22, 2025

mhotan/v1.15 bp 3231 3239 #3240

Merged

Conversation

mhotan commented Apr 18, 2025 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Summary by Bito

Uh oh!

welcome Bot commented Apr 18, 2025

Uh oh!

flyte-bot commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #83541f

Uh oh!

flyte-bot commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelist by Bito

Uh oh!

codecov Bot commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mhotan Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

wild-endeavor Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

mhotan Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

thomasjpfan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

flyte-bot commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #413df0

Uh oh!

kumare3 commented Apr 18, 2025

Uh oh!

flyte-bot commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #30b4ef

Uh oh!

wild-endeavor left a comment

Choose a reason for hiding this comment

Uh oh!

wild-endeavor Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

wild-endeavor Apr 19, 2025

Choose a reason for hiding this comment

Uh oh!

mhotan Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

mhotan commented Apr 21, 2025

Uh oh!

flyte-bot commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #c907a9

Uh oh!

Uh oh!

welcome Bot commented Apr 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mhotan commented Apr 18, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Apr 18, 2025 •

edited

Loading

flyte-bot commented Apr 18, 2025 •

edited

Loading

codecov Bot commented Apr 18, 2025 •

edited

Loading

flyte-bot commented Apr 18, 2025 •

edited

Loading

flyte-bot commented Apr 18, 2025 •

edited

Loading

flyte-bot commented Apr 21, 2025 •

edited

Loading