Allow ARRAY JOIN to be interleaved with other JOIN statements

[hermanschaaf]

Since opening #229, I've learned that it is also valid to interleave ARRAY JOIN statements with other JOIN statements. This PR implements a fix for this.

Example

Here is a minimal example that can be tested against both ClickHouse and the parser. Before this change, ClickHouse succeeds but the parser fails, as it assumes that ARRAY JOIN statements always comes first.

CREATE TABLE t1
(
    id String,
    props String
) ENGINE = Memory;

CREATE TABLE t2
(
    id String,
    props String
) ENGINE = Memory;

CREATE TABLE t3
(
    id String,
    props String
) ENGINE = Memory;

INSERT INTO t1 VALUES
(
    't1_1',
    '{"arr1":[{"id":"t2_1"}]}'
);

INSERT INTO t2 VALUES
(
    't2_1',
    '{"arr2":[{"id":"t3_1"}]}'
);

INSERT INTO t3 VALUES
(
    't3_1',
    '{"value":"hello"}'
);

SELECT
    JSONExtractString(t3.props, 'value') AS value
FROM t1
ARRAY JOIN JSONExtractArrayRaw(t1.props, 'arr1') AS a1
INNER JOIN t2
    ON t2.id = JSONExtractString(a1, 'id')
ARRAY JOIN JSONExtractArrayRaw(t2.props, 'arr2') AS a2
INNER JOIN t3
    ON t3.id = JSONExtractString(a2, 'id')
WHERE value != '';

[coveralls]

Pull Request Test Coverage Report for Build 21248785455

Details

• 34 of 44 (77.27%) changed or added relevant lines in 2 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage increased (+0.2%) to 51.351%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
parser/parser_query.go	27	37	72.97%

Files with Coverage Reduction	New Missed Lines	%
parser/walk.go	1	41.79%

Totals
Change from base Build 21194243629:	0.2%
Covered Lines:	8022
Relevant Lines:	15622

---

💛 - Coveralls

[chatgpt-codex-connector]

💡 Codex Review

clickhouse-sql-parser/parser/parser_query.go

Lines 203 to 213 in d563f06

	case p.matchKeyword(KeywordLeft), p.matchKeyword(KeywordRight):
	modifiers = append(modifiers, p.last().String)
	_ = p.lexer.consumeToken()
	if p.matchKeyword(KeywordOuter) {
	modifiers = append(modifiers, p.last().String)
	_ = p.lexer.consumeToken()
	}
	if p.matchKeyword(KeywordSemi) || p.matchKeyword(KeywordAnti) ||
	p.matchKeyword(KeywordAny) || p.matchKeyword(KeywordAll) ||
	p.matchKeyword(KeywordAsof) || p.matchKeyword(KeywordArray) {
	modifiers = append(modifiers, p.last().String)

Disallow RIGHT ARRAY JOIN in join modifiers

The new parseJoinOp branch for LEFT/RIGHT now accepts ARRAY as a modifier, which means inputs like RIGHT ARRAY JOIN ... will parse as valid. That widens the grammar compared to the existing array-join clause parser, which only allows LEFT/INNER before ARRAY JOIN (so RIGHT was previously rejected). This makes the parser accept SQL that ClickHouse doesn’t support and produces an AST for invalid queries; it’s especially likely to slip through formatting/analysis without errors. Consider restricting ARRAY to the LEFT/INNER cases only (or explicitly reject RIGHT ARRAY JOIN).

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hermanschaaf]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b1df3a0f8d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[git-hulk]

@hermanschaaf Thanks for your improvement, a few comments inline.

[hermanschaaf]

Thanks @git-hulk, please take another look.

I'm not sure how you handle breaking changes in this library, but I assume that removing ArrayJoin could be a breaking change, at least to some users.

Also there's some error with the linter now, I'm not sure what the issue is, any tips would be appreciated.

[hermanschaaf]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91b3cdef30

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[git-hulk]

Also there's some error with the linter now, I'm not sure what the issue is, any tips would be appreciated.

It should be caused by the Go version being too low(1.18).

[git-hulk]

I'm not sure how you handle breaking changes in this library, but I assume that removing ArrayJoin could be a breaking change, at least to some users.

Thanks for pointing that out. We would absolutely like to keep compatibility if possible, but this is a design issue in the old implementation. So, I prefer to use your implementation and just highlight this in our release notes.

[hermanschaaf]

btw there's a typo in the breaking change label :)

[git-hulk]

https://github.com/AfterShip/clickhouse-sql-parser/blob/master/.github/workflows/ci.yaml#L18 @hermanschaaf Need to change the go version in ci.yaml.

[git-hulk]

@hermanschaaf Thank you!