Skip to content

[feature](mtmv)Support iceberg partition refresh.#44726

Merged
Jibing-Li merged 1 commit into
apache:masterfrom
Jibing-Li:iceberg
Dec 16, 2024
Merged

[feature](mtmv)Support iceberg partition refresh.#44726
Jibing-Li merged 1 commit into
apache:masterfrom
Jibing-Li:iceberg

Conversation

@Jibing-Li
Copy link
Copy Markdown
Contributor

@Jibing-Li Jibing-Li commented Nov 28, 2024 •
edited
Loading

What problem does this PR solve?

Previously, when using Iceberg to create MTMV, it was not possible to perceive changes in partition lists and data, so only refresh materialized view mv1 complete could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link
Copy Markdown

doris-robot commented Nov 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li Jibing-Li force-pushed the iceberg branch 3 times, most recently from 870c5ab to ea79a6a Compare November 28, 2024 09:16
@Jibing-Li Jibing-Li marked this pull request as ready for review November 28, 2024 09:17
@Jibing-Li
Copy link
Copy Markdown
Contributor Author

Jibing-Li commented Nov 28, 2024

run buildall

morrySnow
morrySnow reviewed Nov 29, 2024
View reviewed changes
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
zddr
zddr reviewed Nov 29, 2024
View reviewed changes
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
Comment thread regression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy Outdated
Comment thread regression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy
@Jibing-Li Jibing-Li force-pushed the iceberg branch 10 times, most recently from 235a77f to 7ef6b3a Compare December 6, 2024 06:08
@Jibing-Li
Copy link
Copy Markdown
Contributor Author

Jibing-Li commented Dec 6, 2024

run buildall

zddr
zddr reviewed Dec 10, 2024
View reviewed changes
Comment thread fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergPartitionInfo.java
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
@Jibing-Li Jibing-Li force-pushed the iceberg branch 4 times, most recently from 708edf0 to 1df4979 Compare December 10, 2024 08:28
@Jibing-Li
Copy link
Copy Markdown
Contributor Author

Jibing-Li commented Dec 10, 2024

run buildall

zddr
zddr approved these changes Dec 10, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@zddr zddr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 10, 2024

PR approved by anyone and no changes requested.

@Jibing-Li
Copy link
Copy Markdown
Contributor Author

Jibing-Li commented Dec 10, 2024

run p0

morrySnow
morrySnow reviewed Dec 13, 2024
View reviewed changes
Comment thread fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java Outdated
Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
@Jibing-Li
Copy link
Copy Markdown
Contributor Author

Jibing-Li commented Dec 16, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 16, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Dec 16, 2024
@Jibing-Li Jibing-Li merged commit f49e609 into apache:master Dec 16, 2024
github-actions Bot pushed a commit that referenced this pull request Dec 16, 2024
@Jibing-Li
[feature](mtmv)Support iceberg partition refresh. (#44726)
564b3b6 
### What problem does this PR solve?

Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None
@github-actions github-actions Bot mentioned this pull request Dec 16, 2024
Closed
github-actions Bot pushed a commit that referenced this pull request Dec 16, 2024
@Jibing-Li
[feature](mtmv)Support iceberg partition refresh. (#44726)
7ec73c1 
### What problem does this PR solve?

Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None
@github-actions github-actions Bot mentioned this pull request Dec 16, 2024
Closed
@Jibing-Li Jibing-Li deleted the iceberg branch December 16, 2024 08:31
zddr pushed a commit to zddr/incubator-doris that referenced this pull request Jun 19, 2025
@Jibing-Li @zddr
[feature](mtmv)Support iceberg partition refresh. (apache#44726)
22d29ef 
### What problem does this PR solve?

Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None
zddr pushed a commit to zddr/incubator-doris that referenced this pull request Jun 19, 2025
@Jibing-Li @zddr
[feature](mtmv)Support iceberg partition refresh. (apache#44726)
e82fc3b 
Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

None
@zddr zddr mentioned this pull request Jun 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 23, 2025
@zddr @Jibing-Li @BePPPower
branch-3.1: [enhance](mtmv)MTMV support iceberg/hudi/paimon (#52040)
4234a27 
Cherry-pick from
#43959
#44419
#44415
#44567
#44673
#44998
#45273
#44911
#44726
#45652
#45659
#46257
#46641
#47026
#47166
#48172
#49956
#50979

---------

Co-authored-by: James <lijibing@selectdb.com>
Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Jibing-Li @HappenLee
[feature](mtmv)Support iceberg partition refresh. (apache#44726)
af53a73 
### What problem does this PR solve?

Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zddr zddr zddr approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Jibing-Li @doris-robot @zddr @morrySnow