Sync fork#1
Open
kislaykishore wants to merge 1337 commits intokislaykishore:masterfrom
Open
Conversation
### Description The Issue Tests passed on the first run but failed on subsequent runs (-count > 1) with 0 spans detected. This was caused by stale global state: the tracer was captured once at startup and became "orphaned" when the test suite moved to the next iteration, sending spans to a dead provider. The Fix: Changed GCSFuseTracer() to fetch the tracer from the current global provider instead of a one-time package variable and move it to OtelTracer traceHandle struct. ### Link to the issue in case of a bug fix. b/475710717 b/476297291 ### Testing details 1. Manual - Yes Ran the below command where tests were run 5 times, multiple times to check they are passing. /usr/local/go/bin/go test -v -fullpath -timeout 30s -count 5 -run "^TestTracingTestSuite$" github.com/googlecloudplatform /gcsfuse/v3/internal/fs -testify.m "^(TestTraceLookupInode)$" Also verified that the tracing is working fine by exporting the traces to google cloud trace exporter. 2. Unit tests - existing tests need to pass. 3. Integration tests - tests need to pass. ### Any backward incompatible change? If so, please explain. NA
…4282) * refactoring refactoring * review comment
…n inode (#4259) - Adding changes to ensure minObject is updated in MrdInstance as well when it's updated in inode. Also, recreating the MRDPool if the object generation changes. - Ensuring that we decrement the refCount from simple reader only when it was incremented from that reader as well.
Upgrading go sdk to 1.59.1
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.1 to 0.6.2. - [Release notes](https://github.com/pyasn1/pyasn1/releases) - [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst) - [Commits](pyasn1/pyasn1@v0.6.1...v0.6.2) --- updated-dependencies: - dependency-name: pyasn1 dependency-version: 0.6.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.1 to 0.6.2. - [Release notes](https://github.com/pyasn1/pyasn1/releases) - [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst) - [Commits](pyasn1/pyasn1@v0.6.1...v0.6.2) --- updated-dependencies: - dependency-name: pyasn1 dependency-version: 0.6.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.1 to 0.6.2. - [Release notes](https://github.com/pyasn1/pyasn1/releases) - [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst) - [Commits](pyasn1/pyasn1@v0.6.1...v0.6.2) --- updated-dependencies: - dependency-name: pyasn1 dependency-version: 0.6.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump the go-dependencies group across 1 directory with 8 updates (#4268) Bumps the go-dependencies group with 6 updates in the / directory: | Package | From | To | | --- | --- | --- | | [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) | `1.58.0` | `1.59.0` | | [github.com/fsouza/fake-gcs-server](https://github.com/fsouza/fake-gcs-server) | `1.52.2` | `1.52.3` | | [github.com/go-viper/mapstructure/v2](https://github.com/go-viper/mapstructure) | `2.4.0` | `2.5.0` | | [github.com/prometheus/common](https://github.com/prometheus/common) | `0.67.4` | `0.67.5` | | [golang.org/x/net](https://github.com/golang/net) | `0.48.0` | `0.49.0` | | [google.golang.org/api](https://github.com/googleapis/google-api-go-client) | `0.257.0` | `0.259.0` | Updates `cloud.google.com/go/storage` from 1.58.0 to 1.59.0 - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.58.0...spanner/v1.59.0) Updates `github.com/fsouza/fake-gcs-server` from 1.52.2 to 1.52.3 - [Release notes](https://github.com/fsouza/fake-gcs-server/releases) - [Commits](fsouza/fake-gcs-server@v1.52.2...v1.52.3) Updates `github.com/go-viper/mapstructure/v2` from 2.4.0 to 2.5.0 - [Release notes](https://github.com/go-viper/mapstructure/releases) - [Changelog](https://github.com/go-viper/mapstructure/blob/main/CHANGELOG.md) - [Commits](go-viper/mapstructure@v2.4.0...v2.5.0) Updates `github.com/prometheus/common` from 0.67.4 to 0.67.5 - [Release notes](https://github.com/prometheus/common/releases) - [Changelog](https://github.com/prometheus/common/blob/main/CHANGELOG.md) - [Commits](prometheus/common@v0.67.4...v0.67.5) Updates `golang.org/x/net` from 0.48.0 to 0.49.0 - [Commits](golang/net@v0.48.0...v0.49.0) Updates `golang.org/x/sys` from 0.39.0 to 0.40.0 - [Commits](golang/sys@v0.39.0...v0.40.0) Updates `golang.org/x/text` from 0.32.0 to 0.33.0 - [Release notes](https://github.com/golang/text/releases) - [Commits](golang/text@v0.32.0...v0.33.0) Updates `google.golang.org/api` from 0.257.0 to 0.259.0 - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.257.0...v0.259.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.59.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/fsouza/fake-gcs-server dependency-version: 1.52.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/go-viper/mapstructure/v2 dependency-version: 2.5.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/prometheus/common dependency-version: 0.67.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: golang.org/x/net dependency-version: 0.49.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/sys dependency-version: 0.40.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: golang.org/x/text dependency-version: 0.33.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: google.golang.org/api dependency-version: 0.259.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump urllib3 in /perfmetrics/scripts/micro_benchmarks (#4254) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.3. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.5.0...2.6.3) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.6.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump urllib3 (#4242) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.5.0...2.6.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.6.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * revert fake-gcs-server dependecy upgrade due to linux text failure * review comments --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…4283) * feat(simple_reader): changing the default value for max-background * minor change in unit test * Updating the value based on the latest performance
…n GKE and nonGKE environments. (#4277) * create separate package for managing kernel optimisations and contract for GKE * update logs * remove indentation as the file is not read by humans and logs are already json formatted in GKE * review fixes for gemini comments * add more tests and review fixes * fix review comments * kernelparam -> kernelparams * local changes * update comment and usage * fix usage * do nothing when params are empty and fix log structure * fix param name * fix review comments
…ownloader (#4281) * Invalidate metadata Cache for NewMultiRangeDownloader * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix formatting --------- Co-authored-by: Ashmeen Kaur <57195160+ashmeenkaur@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* delete implicit dir with type cache deprecation * rebase * delete implicit dir * some cleanup * small fix * review comments by gemini * remove erasing from type cache twice * Small fix
* add stall timeout for direct path connectivity * Gemini suggestions Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Gemini suggestions Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * lint and linux test fix --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…ze file (#4288) * feat(simple_reader): optimize mrd pool size based on object-size * Updating the heuristic value * pool-size 3 gives regression for 500mb to 900mb file, hence keeping 4 only * fixing unit test * Fixing unit test * minor comment update * Adding minor review comments
…ing (#4292) * add configs for metadata prefetching * rename flags/configs
* Handle clobbered error in MRD Pool * Handle Clobbered error in MRDWrapper * Address comments
…se Test migration] (#4274) * remove spaces in test_config and replace with commmas * gemini reviews resolve * add parsing logic
* fix network unreachable error by adding retries * add unit tests * increase retry * gemini review comments * keeping changes for storage client only * renaming function * review comment * review comment * case in sesitive check
… to directory metadata prefetcher (#4301) * avoid prefetch storms (edge case) * review comments
…egration tests. (#4266) ### Description Disable **grpc** metrics to avoid **otel** plugin error log in integration tests. **ERROR: [otel-plugin] ctx passed into client side stats handler metrics event handling has no client attempt data present** is logged when running any integration tests with grpc protocol and is a known issue in otel grpc plugin and we already disabled grpc metrics in production, doing the same for storage client creation for tests to ensure the noise from this logs removed in tests as well. ### Link to the issue in case of a bug fix. b/475117270 ### Testing details 1. Manual Go to the directory: cd ~/gcsfuse/tools/integration_tests/buffered_read . Run the bufferred read package integration tests. go test . -test.timeout 5400s -test.parallel 1 --integrationTest -test.v --testbucket=thrivikramks_test_runs_zb --zonal=true -test.count 1 without changes in this PR, you would see logs like ERROR: [otel-plugin] ctx passed into client side stats handler metrics event handling has no client attempt data present with changes in this PR, this error is no longer observed. 2. Unit tests - Existing tests need to pass. 3. Integration tests - Existing tests need to pass. ### Any backward incompatible change? If so, please explain. NA
…on] (#4131) * monitoring test migration * Correction of HNS flags for TestPromGrpcMetricsSuite * lint correction * Add cache dir flag * BidiReadObject grpc metric error resolve for HNS * \Skip grpc metrics monitoring tests if GCE VM
* update documentation * Update docs/troubleshooting.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Gemini review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/semantics.md Co-authored-by: Nitin Garg <113666283+gargnitingoogle@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Nitin Garg <113666283+gargnitingoogle@users.noreply.github.com>
gcs/reader_count tracks the number of open and closed readers. ReadHandle label doesn't make sense there.
…xist while updating minObject (#4299) - Ensuring MRDPool is not recreated if it doesn't exist while updating minObject. - Making updateMRD no op for regional buckets. - Making file UTs run for both zonal & non zonal bucket types. - Updating fake bucket to handle append/overwrite.
Updating the retry logic in simple reader and adding some UTs
… 1.26.1 (#4624) Upgrading golang version to 1.26.2 due to multiple CVEs in 1.26.1
#4503) * fixing lint and better formatting * swapping the file content - incorrectly swapped during rebase * review comments * fixing formatting
…s becomes reliable (#4628) * feat: add skipDirectPathEnforcement parameter to createGRPCClientHandle to allow conditional DirectPath enforcement * chore: increase directPathDetectionTimeout from 10 seconds to 5 minutes * simplifying a bit * making direct-path verification non-fatal * removing the timeout from the client creation * minor change * removing empty line
* pr review * refactor: update runtime table visualization to support timeline segments and detailed status icons * simplify retry logic * nested lock issue resolve * increase lock timeout * remove lock based status tracking * fix the attempt numbers --------- Co-authored-by: Mohit Yadav <mohitkyadav@google.com>
Fixes a memory leak in the buffered reader code during multi-block reads. Previously, if an initial block downloaded successfully but a later block failed (triggering a fallback), the reference counts on the successfully downloaded blocks were never decremented. This PR addresses the leak by catching gcsx.FallbackToAnotherReader errors and calling a new releaseInflightBlocks helper to immediately invoke callbacks and release references for any in-flight blocks.
…egy (#4635) Updating the direct path enforcement strategy: - For zonal buckets, log direct path verification status but do not block client creation on it - For non-zonal buckets, block grpc client creation on direct path verification status. In case of failure in detecting direct path status, fallback to http client would happen based on the set grpc-strategy Also, updated the timeout to 15 seconds for direct path verification.
* migration buffered read * mountdir and env changes * lint correction * gemini code reviews * use standard setup and teardown functions * cleanup and format test
* updating otel library from 1.42.0 to 1.43.0 * updating otel library from 1.42.0 to 1.43.0 * updating remaining otel dependencies * updating remaining otel dependencies
### Description Adding the metric gen code for read/block_sizes metric. This is a histogram metric and the details for the metric are available in metrics.yaml . ### Link to the issue in case of a bug fix. b/504892196 ### Testing details 1. Manual - NA 2. Unit tests - Auto generated unittests have been added and need to pass. 3. Integration tests - NA ### Any backward incompatible change? If so, please explain. N/A
### Description Adding metric for tracking streaming write fallback with reasons metric-gen code. ### Link to the issue in case of a bug fix. b/504892393 ### Testing details 1. Manual - NA (given it is auto generated code, existing tests passing and new tests added getting passed is good enough) 2. Unit tests - otel_metrics_test.go updated. 3. Integration tests - NA ### Any backward incompatible change? If so, please explain. N/A
* fix: prevent symlink attack in cmd/ by using O_NOFOLLOW Updated `cmd/mount.go` to open the wire log file with `syscall.O_NOFOLLOW` instead of `os.Create`, which prevents following symlinks. Updated `cmd/legacy_main.go` to open the stderr log file with `syscall.O_NOFOLLOW` when running in daemon mode, which also prevents following symlinks. This fixes the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files (like /etc/shadow) when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: prevent symlink attack in log file creation across the project Updated `internal/logger/logger.go`, `cmd/mount.go`, and `cmd/legacy_main.go` to open log files with the `unix.O_NOFOLLOW` flag. This prevents the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Also updated to use `unix.O_NOFOLLOW` consistently instead of `syscall.O_NOFOLLOW`. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: prevent symlink attack in log file creation across the project Updated `internal/logger/logger.go`, `cmd/mount.go`, and `cmd/legacy_main.go` to open log files with the `unix.O_NOFOLLOW` flag. This prevents the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Also updated to use `unix.O_NOFOLLOW` consistently instead of `syscall.O_NOFOLLOW`. Added robust unit tests to ensure that these log files cannot be overwritten via symlinks. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: prevent symlink attack in log file creation across the project Updated `internal/logger/logger.go`, `cmd/mount.go`, and `cmd/legacy_main.go` to open log files with the `unix.O_NOFOLLOW` flag. This prevents the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Also updated to use `unix.O_NOFOLLOW` consistently instead of `syscall.O_NOFOLLOW`. Added robust unit tests to ensure that these log files cannot be overwritten via symlinks. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: prevent symlink attack in log file creation across the project Updated `internal/logger/logger.go`, `cmd/mount.go`, and `cmd/legacy_main.go` to open log files with the `unix.O_NOFOLLOW` flag. This prevents the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Also updated to use `unix.O_NOFOLLOW` consistently instead of `syscall.O_NOFOLLOW`. Added robust unit tests to ensure that these log files cannot be overwritten via symlinks. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * testcase review comments * fix: prevent symlink attack in log file creation across the project Updated `internal/logger/logger.go`, `cmd/mount.go`, and `cmd/legacy_main.go` to open log files with the `unix.O_NOFOLLOW` flag. This prevents the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Also updated to use `unix.O_NOFOLLOW` consistently instead of `syscall.O_NOFOLLOW`. Added robust unit tests to ensure that these log files cannot be overwritten via symlinks. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: prevent symlink attack in log file creation across the project Updated `internal/logger/logger.go`, `cmd/mount.go`, and `cmd/legacy_main.go` to open log files with the `unix.O_NOFOLLOW` flag. This prevents the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Also updated to use `unix.O_NOFOLLOW` consistently instead of `syscall.O_NOFOLLOW`. Added robust unit tests to ensure that these log files cannot be overwritten via symlinks. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: prevent symlink attack in log file creation across the project Updated `internal/logger/logger.go`, `cmd/mount.go`, and `cmd/legacy_main.go` to open log files with the `unix.O_NOFOLLOW` flag. This prevents the Container-to-Host escape vulnerability reported where an unprivileged user could overwrite arbitrary host files when gcsfuse runs as root by creating symlinks in places where gcsfuse writes logs. Also updated to use `unix.O_NOFOLLOW` consistently instead of `syscall.O_NOFOLLOW`. Added robust unit tests to ensure that these log files cannot be overwritten via symlinks. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
…arsing (#4620) * enforce log-file strictness and centralize log filename parsing * remove .log addition * revert an if block changes
When takeover offset doesn't match the requested offset, logs are more useful if we know what each value was.
…not empty (#4641) * fix: release lock in renameHierarchicalDir Fixes issue #4636 where `newDirInode` lock was never released in `renameHierarchicalDir` when `checkDirNotEmpty` returned an error because the directory was not empty. The fix adds `newDirInode` to `pendingInodes` immediately after successfully getting it, ensuring it's released by the deferred function regardless of subsequent errors. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: release lock in renameHierarchicalDir on error Fixes issue #4636 where `newDirInode` lock was never released in `renameHierarchicalDir` when `checkDirNotEmpty` returned an error because the directory was not empty. Added an integration test to cover renaming a directory to an existing non-empty directory. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * fix: release lock in renameHierarchicalDir on error Fixes issue #4636 where `newDirInode` lock was never released in `renameHierarchicalDir` when `checkDirNotEmpty` returned an error because the directory was not empty. Added an integration test to cover renaming a directory to an existing non-empty directory. Co-authored-by: vadlakondaswetha <101323867+vadlakondaswetha@users.noreply.github.com> * code review comments --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
### Description
Adding support for read block sizes metrics.
This metric tracks the read block size requests coming from kernel to GCSFuse into a histogram metric with the following bucket sizes.
{8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576, 2097152, 4194304, 8388608, 16777216, 33554432, 67108864, 134217728}
### Link to the issue in case of a bug fix.
b/499889366
### Testing details
1. Manual .
Set the parameter of **cloud-metrics-export-interval-secs** to 60 secs and execute the below fio command with a **bssplit** to be able to run random read with different block sizes from kernel to show up the metrics as a heatmap or histogram.
Below are heat map & bar chart representations of the metric: custom.googleapis.com/gcsfuse/read/block_sizes .
fio --name=gcsfuse-precise-test --rw=randread --size=512M --bssplit=8k/20:64k/30:1m/40:128m/10 --numjobs=1
<img width="1115" height="813" alt="Screenshot 2026-04-20 at 5 58 03 AM" src="https://github.com/user-attachments/assets/9c93fe36-7038-432e-9735-3ab679054a80" />
<img width="1118" height="816" alt="Screenshot 2026-04-20 at 5 59 06 AM" src="https://github.com/user-attachments/assets/e7c40631-eaa9-46b2-829c-9ae134810ae5" />
Also scraped the metrics from the prometheus export to check the values are as expected for a histogram metric.
with prometheus port 8080 and hitting the metrics endpoint to search for the read/block_sizes metric, here is a sample result.
curl -s http://localhost:8080/metrics | grep "read_block_sizes"
read_block_sizes_bucket{le="8192"} 218
read_block_sizes_bucket{le="16384"} 232
read_block_sizes_bucket{le="32768"} 269
read_block_sizes_bucket{le="65536"} 558
read_block_sizes_bucket{le="131072"} 4225
read_block_sizes_bucket{le="262144"} 4225
read_block_sizes_bucket{le="524288"} 4225
read_block_sizes_bucket{le="1.048576e+06"} 4225
read_block_sizes_bucket{le="2.097152e+06"} 4225
read_block_sizes_bucket{le="4.194304e+06"} 4225
read_block_sizes_bucket{le="8.388608e+06"} 4225
read_block_sizes_bucket{le="1.6777216e+07"} 4225
read_block_sizes_bucket{le="3.3554432e+07"} 4225
read_block_sizes_bucket{le="6.7108864e+07"} 4225
read_block_sizes_bucket{le="1.34217728e+08"} 4225
read_block_sizes_bucket{le="+Inf"} 4225
read_block_sizes_sum 4.98950144e+08
read_block_sizes_count 4225
2. Unit tests - added
3. Integration tests - added
### Any backward incompatible change? If so, please explain.
N/A
### Description Merge pending dependabot PRs. Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.3.1 to 9.0.3. PR: [4615](#4615) ### Link to the issue in case of a bug fix. b/509411478 ### Testing details 1. Manual - NA 2. Unit tests - NA 3. Integration tests - NA 4. Performance test - NA (Updates to performance testing scripts themselves) ### Any backward incompatible change? If so, please explain. NA
### Description
Adding streaming write fallback metrics that tracks the reason for the fallback and also which file mode & flag it fell back to staging writes. Useful for understanding why streaming writes fallback hasn't been picked .
### Link to the issue in case of a bug fix.
b/499889281
### Testing details
1. Manual
Ran GCSFuse with streaming writes enabled.
Executed the command:
Existing File fallback test:
echo "test_data" > existing_file_test.txt on a new file.
Grepped the prometheus endpoint for the metrics using below:
curl -s http://localhost:8080/metrics | grep "streaming_write_fallback"
No output for first time given that the file is new and streaming writes are allowed,
got output like below:
fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 1
Executed the command multiple times in loops and given each time streaming writes cannot be used for the same reason of existing file, saw the below.
fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 2
fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 7
fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 14
echo "test_data" > not_enabled_test.txt
got an output like below:
fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="not_enabled"} 2
Ran the following to test the concurrent limit breached.
for i in {1..10}; do dd if=/dev/zero of=limit_breached_test_$i.txt bs=1M count=10 & done; wait
Got the output for the metric fallback with appropriate reason.
fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="concurrent_limit_breached"} 7
similarly tested for not enabled and out_of_order_writes
2. Unit tests - added
3. Integration tests - NA
### Any backward incompatible change? If so, please explain.
N/A
…est migration] (#4543) * update test_config.yaml * setup file migration * concurrent listing migration * concurrent read migration * update concurrent operations flags * lint correction * concurrent operations mounted tests passing * correct create dir, and remove func for mountGCSFuse * lint correction * zonal flag addition * gemini code review comment address * remove log import * lint correction and rebase * lint correction for parallel tests
…uncated offsets (#4656)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync fork