Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -925,6 +925,8 @@ SyncThreshold = 5 # Default
LeaseDuration = '0s' # Default
NodeIsSyncingEnabled = false # Default
FinalizedBlockPollInterval = '5s' # Default
HistoricalBalanceCheckAddress = '0x0000000000000000000000000000000000000000' # Default
FinalizedStateCheckFailureThreshold = 0 # Default
EnforceRepeatableRead = true # Default
DeathDeclarationDelay = '1m' # Default
NewHeadsPollInterval = '0s' # Default
Expand Down Expand Up @@ -1003,6 +1005,25 @@ reported based on latest block and finality depth.

Set to 0 to disable.

### HistoricalBalanceCheckAddress
```toml
HistoricalBalanceCheckAddress = '0x0000000000000000000000000000000000000000' # Default
```
HistoricalBalanceCheckAddress is the probe account used by the historical balance health check.
The check executes `eth_getBalance` for this address at the latest finalized block.
Finalized block selection follows chain finality settings:
- `FinalityTagEnabled = true`: use `finalized` tag
- `FinalityTagEnabled = false`: use `latest - FinalityDepth`
The check is only active when `FinalizedStateCheckFailureThreshold > 0`.

### FinalizedStateCheckFailureThreshold
```toml
FinalizedStateCheckFailureThreshold = 0 # Default
```
FinalizedStateCheckFailureThreshold is the number of consecutive failures of the finalized state availability check
before the node is marked as FinalizedStateNotAvailable.
Set to 0 to disable the check.

### EnforceRepeatableRead
```toml
EnforceRepeatableRead = true # Default
Expand Down Expand Up @@ -1068,6 +1089,7 @@ Fatal = '(: |^)fatal' # Example
ServiceUnavailable = '(: |^)service unavailable' # Example
TooManyResults = '(: |^)too many results' # Example
MissingBlocks = '(: |^)invalid block range' # Example
FinalizedStateUnavailable = '(missing trie node|state not available|historical state unavailable)' # Example
```
Errors enable the node to provide custom regex patterns to match against error messages from RPCs.

Expand Down Expand Up @@ -1167,6 +1189,12 @@ MissingBlocks = '(: |^)invalid block range' # Example
```
MissingBlocks is a regex pattern to match an eth_getLogs error indicating the rpc server is permanently missing some blocks in the requested block range

### FinalizedStateUnavailable
```toml
FinalizedStateUnavailable = '(missing trie node|state not available|historical state unavailable)' # Example
```
FinalizedStateUnavailable is a regex pattern to match errors indicating the RPC cannot serve historical state at the finalized block (e.g., pruned/non-archive node)

## OCR
```toml
[OCR]
Expand Down
14 changes: 7 additions & 7 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ require (
github.com/smartcontractkit/chainlink-evm/gethwrappers v0.0.0-20251022073203-7d8ae8cf67c1
github.com/smartcontractkit/chainlink-framework/capabilities v0.0.0-20250818175541-3389ac08a563
github.com/smartcontractkit/chainlink-framework/chains v0.0.0-20260317132927-e8bc2c7b01f1
github.com/smartcontractkit/chainlink-framework/metrics v0.0.0-20251020150604-8ab84f7bad1a
github.com/smartcontractkit/chainlink-framework/multinode v0.0.0-20251021173435-e86785845942
github.com/smartcontractkit/chainlink-framework/metrics v0.0.0-20260310180305-3ee91a6d9ae9
github.com/smartcontractkit/chainlink-framework/multinode v0.0.0-20260317161400-793ab9bc5b53
github.com/smartcontractkit/chainlink-protos/cre/go v0.0.0-20260226130359-963f935e0396
github.com/smartcontractkit/chainlink-protos/svr v1.1.1-0.20260203131522-bb8bc5c423b3
github.com/smartcontractkit/chainlink-tron/relayer v0.0.11-0.20250815105909-75499abc4335
Expand All @@ -46,8 +46,8 @@ require (
github.com/tidwall/gjson v1.18.0
github.com/ugorji/go/codec v1.2.12
github.com/umbracle/ethgo v0.1.3
go.opentelemetry.io/otel v1.39.0
go.opentelemetry.io/otel/metric v1.39.0
go.opentelemetry.io/otel v1.41.0
go.opentelemetry.io/otel/metric v1.41.0
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.27.1
golang.org/x/crypto v0.48.0
Expand Down Expand Up @@ -199,10 +199,10 @@ require (
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.36.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.36.0 // indirect
go.opentelemetry.io/otel/log v0.15.0 // indirect
go.opentelemetry.io/otel/sdk v1.39.0 // indirect
go.opentelemetry.io/otel/sdk v1.41.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.15.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.39.0 // indirect
go.opentelemetry.io/otel/trace v1.39.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.41.0 // indirect
go.opentelemetry.io/otel/trace v1.41.0 // indirect
go.opentelemetry.io/proto/otlp v1.9.0 // indirect
golang.org/x/mod v0.32.0 // indirect
golang.org/x/net v0.49.0 // indirect
Expand Down
28 changes: 14 additions & 14 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -662,10 +662,10 @@ github.com/smartcontractkit/chainlink-framework/capabilities v0.0.0-202508181755
github.com/smartcontractkit/chainlink-framework/capabilities v0.0.0-20250818175541-3389ac08a563/go.mod h1:jP5mrOLFEYZZkl7EiCHRRIMSSHCQsYypm1OZSus//iI=
github.com/smartcontractkit/chainlink-framework/chains v0.0.0-20260317132927-e8bc2c7b01f1 h1:aUdjMnHpriMkEwsgeqQ/ZuNBjrWw6c46HG57TuPPEbE=
github.com/smartcontractkit/chainlink-framework/chains v0.0.0-20260317132927-e8bc2c7b01f1/go.mod h1:kGprqyjsz6qFNVszOQoHc24wfvCjyipNZFste/3zcbs=
github.com/smartcontractkit/chainlink-framework/metrics v0.0.0-20251020150604-8ab84f7bad1a h1:pr0VFI7AWlDVJBEkcvzXWd97V8w8QMNjRdfPVa/IQLk=
github.com/smartcontractkit/chainlink-framework/metrics v0.0.0-20251020150604-8ab84f7bad1a/go.mod h1:jo+cUqNcHwN8IF7SInQNXDZ8qzBsyMpnLdYbDswviFc=
github.com/smartcontractkit/chainlink-framework/multinode v0.0.0-20251021173435-e86785845942 h1:T/eCDsUI8EJT4n5zSP4w1mz4RHH+ap8qieA17QYfBhk=
github.com/smartcontractkit/chainlink-framework/multinode v0.0.0-20251021173435-e86785845942/go.mod h1:2JTBNp3FlRdO/nHc4dsc9bfxxMClMO1Qt8sLJgtreBY=
github.com/smartcontractkit/chainlink-framework/metrics v0.0.0-20260310180305-3ee91a6d9ae9 h1:GK+2aFpW/Z5ZnMGCa9NU6o7LKHQ/9xJVZx2yMAMudnc=
github.com/smartcontractkit/chainlink-framework/metrics v0.0.0-20260310180305-3ee91a6d9ae9/go.mod h1:HG/aei0MgBOpsyRLexdKGtOUO8yjSJO3iUu0Uu8KBm4=
github.com/smartcontractkit/chainlink-framework/multinode v0.0.0-20260317161400-793ab9bc5b53 h1:xOcX4Mp6udpFErDcRiMghcezT85DXHouJlEp60MskH8=
github.com/smartcontractkit/chainlink-framework/multinode v0.0.0-20260317161400-793ab9bc5b53/go.mod h1:n865LsUxibw9oJM0pH74EBiejJ/x/AgIGHaD99D3SDY=
github.com/smartcontractkit/chainlink-protos/cre/go v0.0.0-20260226130359-963f935e0396 h1:03tbcwjyIEjvHba1IWOj1sfThwebm2XNzyFHSuZtlWc=
github.com/smartcontractkit/chainlink-protos/cre/go v0.0.0-20260226130359-963f935e0396/go.mod h1:Jqt53s27Tr0jDl8mdBXg1xhu6F8Fci8JOuq43tgHOM8=
github.com/smartcontractkit/chainlink-protos/linking-service/go v0.0.0-20251002192024-d2ad9222409b h1:QuI6SmQFK/zyUlVWEf0GMkiUYBPY4lssn26nKSd/bOM=
Expand Down Expand Up @@ -790,8 +790,8 @@ go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0 h1:YH4g8lQroajqUwWbq/tr2QX1JFmEXaDLgG+ew9bLMWo=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0/go.mod h1:fvPi2qXDqFs8M4B4fmJhE92TyQs9Ydjlg3RvfUp+NbQ=
go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
go.opentelemetry.io/otel v1.41.0 h1:YlEwVsGAlCvczDILpUXpIpPSL/VPugt7zHThEMLce1c=
go.opentelemetry.io/otel v1.41.0/go.mod h1:Yt4UwgEKeT05QbLwbyHXEwhnjxNO6D8L5PQP51/46dE=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.12.2 h1:06ZeJRe5BnYXceSM9Vya83XXVaNGe3H1QqsvqRANQq8=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.12.2/go.mod h1:DvPtKE63knkDVP88qpatBj81JxN+w1bqfVbsbCbj1WY=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.12.2 h1:tPLwQlXbJ8NSOfZc4OkgU5h2A38M4c9kfHSVc4PFQGs=
Expand All @@ -814,18 +814,18 @@ go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.36.0 h1:G8Xec/SgZQricwW
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.36.0/go.mod h1:PD57idA/AiFD5aqoxGxCvT/ILJPeHy3MjqU/NS7KogY=
go.opentelemetry.io/otel/log v0.15.0 h1:0VqVnc3MgyYd7QqNVIldC3dsLFKgazR6P3P3+ypkyDY=
go.opentelemetry.io/otel/log v0.15.0/go.mod h1:9c/G1zbyZfgu1HmQD7Qj84QMmwTp2QCQsZH1aeoWDE4=
go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
go.opentelemetry.io/otel/metric v1.41.0 h1:rFnDcs4gRzBcsO9tS8LCpgR0dxg4aaxWlJxCno7JlTQ=
go.opentelemetry.io/otel/metric v1.41.0/go.mod h1:xPvCwd9pU0VN8tPZYzDZV/BMj9CM9vs00GuBjeKhJps=
go.opentelemetry.io/otel/sdk v1.41.0 h1:YPIEXKmiAwkGl3Gu1huk1aYWwtpRLeskpV+wPisxBp8=
go.opentelemetry.io/otel/sdk v1.41.0/go.mod h1:ahFdU0G5y8IxglBf0QBJXgSe7agzjE4GiTJ6HT9ud90=
go.opentelemetry.io/otel/sdk/log v0.15.0 h1:WgMEHOUt5gjJE93yqfqJOkRflApNif84kxoHWS9VVHE=
go.opentelemetry.io/otel/sdk/log v0.15.0/go.mod h1:qDC/FlKQCXfH5hokGsNg9aUBGMJQsrUyeOiW5u+dKBQ=
go.opentelemetry.io/otel/sdk/log/logtest v0.13.0 h1:9yio6AFZ3QD9j9oqshV1Ibm9gPLlHNxurno5BreMtIA=
go.opentelemetry.io/otel/sdk/log/logtest v0.13.0/go.mod h1:QOGiAJHl+fob8Nu85ifXfuQYmJTFAvcrxL6w5/tu168=
go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
go.opentelemetry.io/otel/sdk/metric v1.41.0 h1:siZQIYBAUd1rlIWQT2uCxWJxcCO7q3TriaMlf08rXw8=
go.opentelemetry.io/otel/sdk/metric v1.41.0/go.mod h1:HNBuSvT7ROaGtGI50ArdRLUnvRTRGniSUZbxiWxSO8Y=
go.opentelemetry.io/otel/trace v1.41.0 h1:Vbk2co6bhj8L59ZJ6/xFTskY+tGAbOnCtQGVVa9TIN0=
go.opentelemetry.io/otel/trace v1.41.0/go.mod h1:U1NU4ULCoxeDKc09yCWdWe+3QoyweJcISEVa1RBzOis=
go.opentelemetry.io/proto/otlp v1.9.0 h1:l706jCMITVouPOqEnii2fIAuO3IVGBRPV5ICjceRb/A=
go.opentelemetry.io/proto/otlp v1.9.0/go.mod h1:xE+Cx5E/eEHw+ISFkwPLwCZefwVjY+pqKg1qcK03+/4=
go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
Expand Down
42 changes: 27 additions & 15 deletions pkg/client/helpers_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ type TestClientErrors struct {
serviceUnavailable string
tooManyResults string
missingBlocks string
finalizedStateUnavailable string
}

func NewTestClientErrors() TestClientErrors {
Expand Down Expand Up @@ -83,23 +84,26 @@ func (c *TestClientErrors) L2FeeTooHigh() string { return c.l2FeeTooH
func (c *TestClientErrors) L2Full() string { return c.l2Full }
func (c *TestClientErrors) TransactionAlreadyMined() string { return c.transactionAlreadyMined }
func (c *TestClientErrors) Fatal() string { return c.fatal }
func (c *TestClientErrors) ServiceUnavailable() string { return c.serviceUnavailable }
func (c *TestClientErrors) TooManyResults() string { return c.tooManyResults }
func (c *TestClientErrors) MissingBlocks() string { return c.missingBlocks }
func (c *TestClientErrors) ServiceUnavailable() string { return c.serviceUnavailable }
func (c *TestClientErrors) TooManyResults() string { return c.tooManyResults }
func (c *TestClientErrors) MissingBlocks() string { return c.missingBlocks }
func (c *TestClientErrors) FinalizedStateUnavailable() string { return c.finalizedStateUnavailable }

type TestNodePoolConfig struct {
NodePollFailureThreshold uint32
NodePollInterval time.Duration
NodeSelectionMode string
NodeSyncThreshold uint32
NodeLeaseDuration time.Duration
NodeIsSyncingEnabledVal bool
NodeFinalizedBlockPollInterval time.Duration
NodeErrors config.ClientErrors
EnforceRepeatableReadVal bool
NodeDeathDeclarationDelay time.Duration
NodeNewHeadsPollInterval time.Duration
ExternalRequestMaxResponseSizeVal uint32
NodePollFailureThreshold uint32
NodePollInterval time.Duration
NodeSelectionMode string
NodeSyncThreshold uint32
NodeLeaseDuration time.Duration
NodeIsSyncingEnabledVal bool
NodeFinalizedBlockPollInterval time.Duration
HistoricalBalanceCheckAddressVal string
NodeErrors config.ClientErrors
EnforceRepeatableReadVal bool
NodeDeathDeclarationDelay time.Duration
NodeNewHeadsPollInterval time.Duration
ExternalRequestMaxResponseSizeVal uint32
FinalizedStateCheckFailureThresholdVal uint32
}

func (tc TestNodePoolConfig) PollFailureThreshold() uint32 { return tc.NodePollFailureThreshold }
Expand All @@ -118,6 +122,10 @@ func (tc TestNodePoolConfig) FinalizedBlockPollInterval() time.Duration {
return tc.NodeFinalizedBlockPollInterval
}

func (tc TestNodePoolConfig) HistoricalBalanceCheckAddress() string {
return tc.HistoricalBalanceCheckAddressVal
}

func (tc TestNodePoolConfig) NewHeadsPollInterval() time.Duration {
return tc.NodeNewHeadsPollInterval
}
Expand All @@ -142,6 +150,10 @@ func (tc TestNodePoolConfig) ExternalRequestMaxResponseSize() uint32 {
return tc.ExternalRequestMaxResponseSizeVal
}

func (tc TestNodePoolConfig) FinalizedStateCheckFailureThreshold() uint32 {
return tc.FinalizedStateCheckFailureThresholdVal
}

func NewChainClientWithTestNode(
t *testing.T,
nodeCfg multinode.NodeConfig,
Expand Down
47 changes: 47 additions & 0 deletions pkg/client/rpc_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"math/big"
"net/http"
"net/url"
"regexp"
"strconv"
"sync/atomic"
"time"
Expand Down Expand Up @@ -105,6 +106,7 @@ type RPCClient struct {
finalityTagEnabled bool
finalityDepth uint32
safeDepth uint32
historicalBalanceCheckAddress common.Address
externalRequestMaxResponseSize uint32

beholderMetrics *rpcClientMetrics
Expand Down Expand Up @@ -144,6 +146,7 @@ func NewRPCClient(
finalityTagEnabled: supportsFinalityTags,
finalityDepth: finalityDepth,
safeDepth: safeDepth,
historicalBalanceCheckAddress: common.HexToAddress(cfg.HistoricalBalanceCheckAddress()),
externalRequestMaxResponseSize: externalRequestMaxResponseSize,
}
r.cfg = cfg
Expand Down Expand Up @@ -193,6 +196,50 @@ func (r *RPCClient) ClientVersion(ctx context.Context) (version string, err erro
return version, nil
}

// CheckFinalizedStateAvailability verifies if the RPC can serve historical state at the finalized block.
// This is used to detect non-archive nodes that cannot serve state queries for older blocks.
// Returns multinode.ErrFinalizedStateUnavailable if the error matches the FinalizedStateUnavailable pattern.
// The decision to call this method is made by multinode based on its configuration.
func (r *RPCClient) CheckFinalizedStateAvailability(ctx context.Context) error {
var blockNumber *big.Int
if r.finalityTagEnabled {
blockNumber = big.NewInt(rpc.FinalizedBlockNumber.Int64())
} else {
latestBlock, err := r.BlockNumber(ctx)
if err != nil {
return fmt.Errorf("fetching latest block number failed: %w", err)
}
latest := int64(latestBlock)
finalizedHeight := max(int64(0), latest-int64(r.finalityDepth))
blockNumber = big.NewInt(finalizedHeight)
}
_, err := r.BalanceAt(ctx, r.historicalBalanceCheckAddress, blockNumber)
if err != nil {
if r.isFinalizedStateUnavailableError(err) {
return fmt.Errorf("%w: %w", multinode.ErrFinalizedStateUnavailable, err)
}
return fmt.Errorf("fetching balance for address %s at block %s failed: %w", r.historicalBalanceCheckAddress.String(), blockNumber.String(), err)
}
return nil
}

// isFinalizedStateUnavailableError checks if the error matches the FinalizedStateUnavailable regex pattern.
func (r *RPCClient) isFinalizedStateUnavailableError(err error) bool {
if err == nil {
return false
}
pattern := r.clientErrors.FinalizedStateUnavailable()
if pattern == "" {
return false
}
re, compileErr := regexp.Compile(pattern)
if compileErr != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log critical error. Without a correct regexp, check if disabled. We need to be as loud as possible to indicate the issue, as system does not behave as configurer expects it to behave

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added a Criticalw log when the FinalizedStateUnavailable regex fails to compile:

r.rpcLog.Criticalw("FinalizedStateUnavailable regex pattern is invalid; finalized state availability check is effectively disabled", "pattern", pattern, "err", compileErr)

  • Used Critical specifically because without a valid regex, the check silently stops classifying errors as ErrFinalizedStateUnavailable, meaning non-archive nodes would never be marked FinalizedStateNotAvailable the system misbehaves without any indication.

r.rpcLog.Criticalw("FinalizedStateUnavailable regex pattern is invalid; finalized state availability check is effectively disabled", "pattern", pattern, "err", compileErr)
return false
}
return re.MatchString(err.Error())
}

func (r *RPCClient) Dial(callerCtx context.Context) error {
ctx, cancel, _ := r.AcquireQueryCtx(callerCtx, r.rpcTimeout)
defer cancel()
Expand Down
Loading
Loading