-
Notifications
You must be signed in to change notification settings - Fork 948
[KIP-932] : Implement Share consumer interface with poll API #2217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Kaushik Raina (k-raina)
wants to merge
22
commits into
dev_kip-932_queues-for-kafka
Choose a base branch
from
dev_kip-932_share_consumer_poll
base: dev_kip-932_queues-for-kafka
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
cc164a1
DGS-23025 Add AssociatedNameStrategy (#2194)
rayokota 51ec85d
KIP-932 : Implement Share consumer interface with poll API
k-raina 0ed043f
Fix typo (#1691)
lpsinger fc5ef4f
BUILD-KIP932: Commit to build branch with kip-932 librdkafka branch
k-raina 40e8a6f
Implement async context manager protocol for AIO clients (#2180)
lpsinger d313cac
Fix return type signature for context managers (#2181)
lpsinger f3af6aa
Remove callback wrapping for oauth_cb in AIO clients (#2219)
ojasvajain e23df2e
Address copilor comments
k-raina 88b4a3a
Address first pass feedback
k-raina 852d090
Style fix
k-raina 410a59c
Add enableAt to RuleSet (#2218)
rayokota 73c7953
Avoid redundant lookup_schema call in _get_reader_schema when use_sch…
Jthai006 ea948da
v2.14.0rc1 (#2224)
emasab d80ff36
Add current release candidate to setup_all_versions.py (#2226)
emasab a2f066d
v2.14.0 (#2227)
emasab 6c3bd6d
DGS-23862 Fix URL joining in Python client (#2228)
rayokota d59b196
Style fix
k-raina 48208f4
Address feedback
k-raina f5f03dc
Fix uv venv picking up Python 3.14 on new Semaphore AMI (#2233)
k-raina 1e41691
Address feedback
k-raina 59db830
Merge remote-tracking branch 'origin/master' into dev_kip-932_share_c…
k-raina 9580e6d
Style fix
k-raina File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,7 +8,10 @@ execution_time_limit: | |
| global_job_config: | ||
| env_vars: | ||
| - name: LIBRDKAFKA_VERSION | ||
| value: v2.13.2 | ||
| value: v2.14.0 | ||
| # TODO KIP-932: Remove LIBRDKAFKA_BRANCH once LIBRDKAFKA_VERSION includes share consumer support | ||
| - name: LIBRDKAFKA_BRANCH | ||
| value: dev_kip-932_queues-for-kafka | ||
| prologue: | ||
| commands: | ||
| - checkout | ||
|
|
@@ -26,13 +29,13 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: osx | ||
| - name: ARCH | ||
| value: x64 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=x64 | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # TODO KIP-932: Add LIBRDKAFKA_BRANCH fallback for share consumer support | ||
| - PIP_INSTALL_OPTIONS="--user" tools/wheels/build-wheels.sh "${LIBRDKAFKA_VERSION#v}" wheelhouse 2.16.2 | ||
| - tar -czf wheelhouse-macOS-${ARCH}.tgz wheelhouse | ||
| - artifact push workflow wheelhouse-macOS-${ARCH}.tgz --destination artifacts/wheels-${OS_NAME}-${ARCH}.tgz/ | ||
|
|
@@ -48,17 +51,17 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: osx | ||
| - name: ARCH | ||
| value: x64 | ||
| - name: CIBW_SKIP | ||
| value: cp36-* cp37-* cp38-* cp39-* cp310-* cp311-* cp312-* | ||
| - name: CIBW_ENVIRONMENT_MACOS | ||
| value: MACOSX_DEPLOYMENT_TARGET=13 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=x64 | ||
| - sem-version python 3.13 | ||
| - pip install uv | ||
| # TODO KIP-932: Add LIBRDKAFKA_BRANCH fallback for share consumer support | ||
| - PIP_INSTALL_OPTIONS="--user" tools/wheels/build-wheels.sh "${LIBRDKAFKA_VERSION#v}" wheelhouse | ||
| - tar -czf wheelhouse-macOS-${ARCH}-py313-plus.tgz wheelhouse | ||
| - artifact push workflow wheelhouse-macOS-${ARCH}-py313-plus.tgz --destination artifacts/wheels-${OS_NAME}-${ARCH}-py313-plus.tgz/ | ||
|
|
@@ -76,13 +79,13 @@ blocks: | |
| value: osx | ||
| - name: CIBW_ARCHS | ||
| value: arm64 | ||
| - name: ARCH | ||
| value: arm64 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=arm64 | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # TODO KIP-932: Add LIBRDKAFKA_BRANCH fallback for share consumer support | ||
| - PIP_INSTALL_OPTIONS="--user" tools/wheels/build-wheels.sh "${LIBRDKAFKA_VERSION#v}" wheelhouse 2.16.2 | ||
| - tar -czf wheelhouse-macOS-${ARCH}.tgz wheelhouse | ||
| - artifact push workflow wheelhouse-macOS-${ARCH}.tgz --destination artifacts/wheels-${OS_NAME}-${ARCH}.tgz/ | ||
|
|
@@ -100,17 +103,17 @@ blocks: | |
| value: osx | ||
| - name: CIBW_ARCHS | ||
| value: arm64 | ||
| - name: ARCH | ||
| value: arm64 | ||
| - name: CIBW_SKIP | ||
| value: cp38-* cp39-* cp310-* cp311-* cp312-* | ||
| - name: CIBW_ENVIRONMENT_MACOS | ||
| value: MACOSX_DEPLOYMENT_TARGET=13 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=arm64 | ||
| - sem-version python 3.13 | ||
| - pip install uv | ||
| # TODO KIP-932: Add LIBRDKAFKA_BRANCH fallback for share consumer support | ||
| - PIP_INSTALL_OPTIONS="--user" tools/wheels/build-wheels.sh "${LIBRDKAFKA_VERSION#v}" wheelhouse | ||
| - tar -czf wheelhouse-macOS-${ARCH}-py313-plus.tgz wheelhouse | ||
| - artifact push workflow wheelhouse-macOS-${ARCH}-py313-plus.tgz --destination artifacts/wheels-${OS_NAME}-${ARCH}-py313-plus.tgz/ | ||
|
|
@@ -126,13 +129,13 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: linux | ||
| - name: ARCH | ||
| value: arm64 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=arm64 | ||
| - sem-version python 3.13 | ||
| - pip install uv | ||
| # TODO KIP-932: Add LIBRDKAFKA_BRANCH fallback for share consumer support | ||
| - ./tools/wheels/build-wheels.sh "${LIBRDKAFKA_VERSION#v}" wheelhouse | ||
| - tar -czf wheelhouse-linux-${ARCH}.tgz wheelhouse | ||
| - artifact push workflow wheelhouse-linux-${ARCH}.tgz --destination artifacts/wheels-${OS_NAME}-${ARCH}.tgz/ | ||
|
|
@@ -148,13 +151,13 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: linux | ||
| - name: ARCH | ||
| value: x64 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=x64 | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # TODO KIP-932: Add LIBRDKAFKA_BRANCH fallback for share consumer support | ||
| - ./tools/wheels/build-wheels.sh "${LIBRDKAFKA_VERSION#v}" wheelhouse | ||
| - tar -czf wheelhouse-linux-${ARCH}.tgz wheelhouse | ||
| - artifact push workflow wheelhouse-linux-${ARCH}.tgz --destination artifacts/wheels-${OS_NAME}-${ARCH}.tgz/ | ||
|
|
@@ -187,6 +190,7 @@ blocks: | |
| commands: | ||
| - sem-version python 3.11.9 | ||
| - bash tools/mingw-w64/semaphore_commands.sh | ||
| # TODO KIP-932: Add LIBRDKAFKA_BRANCH fallback for share consumer support | ||
| - bash tools/wheels/install-librdkafka.sh $env:LIBRDKAFKA_VERSION.TrimStart("v") dest | ||
| - tools/wheels/build-wheels.bat x64 win_amd64 dest wheelhouse | ||
| - tar -czf wheelhouse-windows-${Env:ARCH}.tgz wheelhouse | ||
|
|
@@ -200,10 +204,9 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: linux | ||
| - name: ARCH | ||
| value: x64 | ||
| prologue: | ||
| commands: | ||
| - export ARCH=x64 | ||
| - '[[ -z $DOCKERHUB_APIKEY ]] || docker login --username $DOCKERHUB_USER --password $DOCKERHUB_APIKEY' | ||
| - sudo apt-get update -qq && sudo apt-get install -y -qq clang-format | ||
| jobs: | ||
|
|
@@ -212,7 +215,7 @@ blocks: | |
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # use a virtualenv | ||
| - uv venv _venv && source _venv/bin/activate | ||
| - uv venv _venv --python "$(command -v python)" && source _venv/bin/activate | ||
| - chmod u+r+x tools/source-package-verification.sh | ||
| - tools/source-package-verification.sh | ||
| - name: Build and Tests with 'consumer' group protocol | ||
|
|
@@ -221,7 +224,7 @@ blocks: | |
| - sem-version java 17 | ||
| - pip install uv | ||
| # use a virtualenv | ||
| - uv venv _venv && source _venv/bin/activate | ||
| - uv venv _venv --python "$(command -v python)" && source _venv/bin/activate | ||
| - chmod u+r+x tools/source-package-verification.sh | ||
| - export TEST_CONSUMER_GROUP_PROTOCOL=consumer | ||
| - tools/source-package-verification.sh | ||
|
|
@@ -230,7 +233,7 @@ blocks: | |
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # use a virtualenv | ||
| - uv venv _venv && source _venv/bin/activate | ||
| - uv venv _venv --python "$(command -v python)" && source _venv/bin/activate | ||
| - chmod u+r+x tools/source-package-verification.sh | ||
| - export RUN_COVERAGE=true | ||
| - tools/source-package-verification.sh | ||
|
|
@@ -247,15 +250,14 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: linux | ||
| - name: ARCH | ||
| value: arm64 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=arm64 | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # use a virtualenv | ||
| - uv venv _venv && source _venv/bin/activate | ||
| - uv venv _venv --python "$(command -v python)" && source _venv/bin/activate | ||
| - chmod u+r+x tools/source-package-verification.sh | ||
| - tools/source-package-verification.sh | ||
| - name: "Source package verification with Python 3 (OSX x64) +docs" | ||
|
|
@@ -267,15 +269,14 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: osx | ||
| - name: ARCH | ||
| value: x64 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=x64 | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # use a virtualenv | ||
| - uv venv _venv && source _venv/bin/activate | ||
| - uv venv _venv --python "$(command -v python)" && source _venv/bin/activate | ||
| - chmod u+r+x tools/source-package-verification.sh | ||
| - tools/source-package-verification.sh | ||
| - name: "Source package verification with Python 3 (OSX arm64) +docs" | ||
|
|
@@ -287,15 +288,14 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: osx | ||
| - name: ARCH | ||
| value: arm64 | ||
| jobs: | ||
| - name: Build | ||
| commands: | ||
| - export ARCH=arm64 | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
| # use a virtualenv | ||
| - uv venv _venv && source _venv/bin/activate | ||
| - uv venv _venv --python "$(command -v python)" && source _venv/bin/activate | ||
| - chmod u+r+x tools/source-package-verification.sh | ||
| - tools/source-package-verification.sh | ||
| - name: "Ducktape Performance Tests (Linux x64)" | ||
|
|
@@ -307,8 +307,6 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: linux | ||
| - name: ARCH | ||
| value: x64 | ||
| - name: BENCHMARK_BOUNDS_CONFIG | ||
| value: tests/ducktape/producer_benchmark_bounds.json | ||
| - name: BENCHMARK_ENVIRONMENT | ||
|
|
@@ -320,19 +318,26 @@ blocks: | |
| - name: Build and Tests | ||
| commands: | ||
| # Setup Python environment | ||
| - export ARCH=x64 | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
| - uv venv _venv && source _venv/bin/activate | ||
| - uv venv _venv --python "$(command -v python)" && source _venv/bin/activate | ||
|
|
||
| # Install ducktape framework and additional dependencies | ||
| - uv pip install ducktape psutil | ||
|
|
||
| # Install existing test requirements | ||
| - uv pip install -r requirements/requirements-tests-install.txt | ||
|
|
||
| # TODO KIP-932: Remove LIBRDKAFKA_BRANCH fallback once LIBRDKAFKA_VERSION includes share consumer support | ||
| # Build and install confluent-kafka from source | ||
| - lib_dir=dest/runtimes/$OS_NAME-$ARCH/native | ||
| - tools/wheels/install-librdkafka.sh "${LIBRDKAFKA_VERSION#v}" dest | ||
| - | | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We need these changes in other jobs as well. Either add them or add a TODO for those. |
||
| if [[ -n $LIBRDKAFKA_BRANCH ]]; then | ||
| tools/wheels/build-librdkafka-branch.sh "$LIBRDKAFKA_BRANCH" dest | ||
| else | ||
| tools/wheels/install-librdkafka.sh "${LIBRDKAFKA_VERSION#v}" dest | ||
| fi | ||
| - export CFLAGS="$CFLAGS -I${PWD}/dest/build/native/include" | ||
| - export LDFLAGS="$LDFLAGS -L${PWD}/${lib_dir}" | ||
| - export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$PWD/$lib_dir" | ||
|
|
@@ -411,11 +416,10 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: linux | ||
| - name: ARCH | ||
| value: x64 | ||
| jobs: | ||
| - name: Verify | ||
| commands: | ||
| - export ARCH=x64 | ||
| - sem-version python 3.9 | ||
| - artifact pull workflow artifacts | ||
| - cd artifacts && ls *.tgz |xargs -n1 tar -xvf && cd .. | ||
|
|
@@ -432,11 +436,10 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: linux | ||
| - name: ARCH | ||
| value: arm64 | ||
| jobs: | ||
| - name: Verify | ||
| commands: | ||
| - export ARCH=arm64 | ||
| - sem-version python 3.9 | ||
| - artifact pull workflow artifacts | ||
| - cd artifacts && ls *.tgz |xargs -n1 tar -xvf && cd .. | ||
|
|
@@ -454,11 +457,10 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: osx | ||
| - name: ARCH | ||
| value: x64 | ||
| jobs: | ||
| - name: Verify | ||
| commands: | ||
| - export ARCH=x64 | ||
| - checkout | ||
| - sem-version python 3.11 | ||
| - pip install uv | ||
|
|
@@ -487,13 +489,12 @@ blocks: | |
| env_vars: | ||
| - name: OS_NAME | ||
| value: osx | ||
| - name: ARCH | ||
| value: arm64 | ||
| jobs: | ||
| - name: Verify | ||
| commands: | ||
| - checkout | ||
| - sem-version python 3.11 | ||
| - export ARCH=arm64 | ||
| - pip install uv | ||
| - uv pip install --system pytest | ||
| - uv pip install --system -r requirements/requirements-tests-install.txt | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| typing-extensions; python_version < "3.11" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a TODO to remove this
env_varswhen theLIBRDKAFKA_VERSIONhas QfK changes.