hrw4u: Sync Python transpiler and add roundtrip tests#13058
hrw4u: Sync Python transpiler and add roundtrip tests#13058zwoop wants to merge 1 commit intoapache:masterfrom
Conversation
Extract hrw4u-specific changes from apache#12825. Fixes issues in the Python transpiler found during C++ parser development, normalizes autest .conf rules for bidirectional conversion, and adds roundtrip tests verifying hrw4u<->conf equivalence.
There was a problem hiding this comment.
Pull request overview
Synchronizes the Python HRW4U transpiler / inverse transpiler with recent header_rewrite behavior, normalizes autest rule corpora for stable bidirectional conversion, and adds tests to verify hrw4u↔conf roundtrips against the gold_tests rules.
Changes:
- Add pytest-based validation that hrw4u output matches the autest
.confrule files, plus reverse conversion tests. - Extend hrw4u/u4wrh symbol tables and inverse visitor logic (including new ops and section/condition handling) and expose
--no-merge-sectionsinu4wrh. - Normalize and add autest HRW4U source files and update corresponding
.conffiles for consistent quoting/indentation/modifiers.
Reviewed changes
Copilot reviewed 61 out of 62 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/hrw4u/tests/test_autest_rules.py | New forward (hrw4u→conf) tests against autest gold rules. |
| tools/hrw4u/tests/test_autest_rules_reverse.py | New reverse (conf→hrw4u) tests against autest gold rules. |
| tools/hrw4u/tests/data/ops/skip-remap.input.txt | Update skip-remap test to PRE_REMAP section. |
| tools/hrw4u/tests/data/ops/skip-remap.output.txt | Update expected hook mapping for PRE_REMAP. |
| tools/hrw4u/tests/data/ops/skip-remap.ast.txt | Update AST expected for PRE_REMAP section name. |
| tools/hrw4u/tests/data/ops/set-effective-address.input.txt | Add forward test input for set-effective-address. |
| tools/hrw4u/tests/data/ops/set-effective-address.output.txt | Add expected conf output for set-effective-address. |
| tools/hrw4u/tests/data/ops/set-effective-address.ast.txt | Add expected AST for set-effective-address. |
| tools/hrw4u/tests/data/ops/set-cc-alg.input.txt | Add forward test input for set-cc-alg. |
| tools/hrw4u/tests/data/ops/set-cc-alg.output.txt | Add expected conf output for set-cc-alg. |
| tools/hrw4u/tests/data/ops/set-cc-alg.ast.txt | Add expected AST for set-cc-alg. |
| tools/hrw4u/src/types.py | Add magic string identifiers for new operators. |
| tools/hrw4u/src/tables.py | Update symbol/section restrictions and add new statement function mappings. |
| tools/hrw4u/src/hrw_visitor.py | Adjust inverse conversion behavior (section merging, if-chain handling, [L] handling, preamble insertion). |
| tools/hrw4u/src/hrw_symbols.py | Improve set parsing for quoted commas and remove unused logic. |
| tools/hrw4u/scripts/u4wrh | Add --no-merge-sections CLI flag plumbing. |
| tools/hrw4u/README.md | Document C++ native parser build requirements and setup notes. |
| tools/hrw4u/pyproject.toml | Add autest pytest marker. |
| tests/gold_tests/pluginTest/header_rewrite/rules/set_redirect.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/set_redirect.conf | Normalize conf formatting/quoting for roundtrip stability. |
| tests/gold_tests/pluginTest/header_rewrite/rules/run_plugin.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/run_plugin.conf | Normalize conf formatting/quoting for roundtrip stability. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule.conf | Normalize conf indentation for if/elif chain. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_header_after_ssn_txn_count.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_header_after_ssn_txn_count.conf | Normalize conf quoting/indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_status.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_status.conf | Normalize hook lines and quoting. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_from_remap.hrw4u | Add HRW4U source version of autest rule (multiple same hook sections). |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_from_remap.conf | Normalize hook lines, quoting, and condition formatting. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_from_plugin.hrw4u | Add HRW4U source version of autest rule (multiple same hook sections). |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_from_plugin.conf | Normalize hook lines, quoting, and condition formatting. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_session_vars.hrw4u | Add HRW4U source version of autest rule (SESSION_VARS). |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_session_vars.conf | Normalize hook lines and indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_server_conditions.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_server_conditions.conf | Normalize hook lines and indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_procedures.hrw4u | Add HRW4U source version of autest rule (procedure usage). |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_l_value.hrw4u | Add HRW4U source version of autest rule (break/[L] semantics). |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_l_value.conf | Normalize [L] representation and indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_empty_body.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_empty_body.conf | Normalize hook lines and quoting. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_effective_address.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_effective_address.conf | Normalize hook lines and quoting. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_cond_method.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_cond_method.conf | Normalize hook lines and remove [L] in favor of hrw4u control-flow. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_client.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_client.conf | Normalize hook lines/modifier ordering/indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_add_cache_result_header.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/rule_add_cache_result_header.conf | Normalize hook lines and quoting. |
| tests/gold_tests/pluginTest/header_rewrite/rules/regex_tests.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/regex_tests.conf | Normalize hook lines and indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/query_sub_key.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/query_sub_key.conf | Normalize hook lines and indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/procs/test/stamp.hrw4u | Add procedure definition used by procedure autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/nested_ifs.hrw4u | Add HRW4U source version of nested-if autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/nested_ifs.conf | Normalize nested if/elif/endif indentation and spacing. |
| tests/gold_tests/pluginTest/header_rewrite/rules/implicit_hook.hrw4u | Add HRW4U source version of implicit-hook autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/implicit_hook.conf | Normalize hook lines and indentation. |
| tests/gold_tests/pluginTest/header_rewrite/rules/glob_set_redirect.hrw4u | Add HRW4U source version of autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/glob_set_redirect.conf | Normalize hook lines and quoting. |
| tests/gold_tests/pluginTest/header_rewrite/rules/complex_logics.hrw4u | Add HRW4U source version of complex logic/grouping autest rule. |
| tests/gold_tests/pluginTest/header_rewrite/rules/complex_logics.conf | Normalize hook lines/modifiers/indentation for complex grouping cases. |
| def _close_if_chain_for_new_rule(self) -> None: | ||
| """Close if-else chain when a new rule starts without elif/else.""" | ||
| expecting_nested_if = self._expecting_if_cond | ||
| self._expecting_if_cond = False | ||
|
|
||
| if (self._if_depth > 0 and not self._in_elif_mode and not self._pending_terms and not expecting_nested_if and | ||
| not self._in_group): | ||
| self.debug("new rule detected - closing if chain") | ||
| self._start_new_section(SectionType.REMAP) |
There was a problem hiding this comment.
_close_if_chain_for_new_rule() hard-codes SectionType.REMAP when it decides to start a new section. This ignores the visitor's configured default section_label and can produce incorrect output when the inverse visitor is used with a non-REMAP default hook (e.g., global header_rewrite configs default to READ_RESPONSE). Consider starting the new section using the configured default (e.g., a stored _default_section_label captured from the ctor arg), rather than always REMAP.
| @pytest.mark.parametrize("conf_file,hrw4u_file", _collect_autest_pairs()) | ||
| def test_hrw4u_to_conf(conf_file: Path, hrw4u_file: Path) -> None: | ||
| """Test that hrw4u -> conf output matches the .conf file.""" |
There was a problem hiding this comment.
pyproject.toml adds an autest pytest marker, but this test isn't marked with it. Adding @pytest.mark.autest (alongside the parametrize) would let CI/devs include/exclude these integration-style tests explicitly (they depend on the gold_tests rule corpus).
| @pytest.mark.parametrize("conf_file,hrw4u_file", _collect_autest_pairs()) | ||
| def test_conf_to_hrw4u(conf_file: Path, hrw4u_file: Path) -> None: | ||
| """Test that conf -> hrw4u output matches the .hrw4u file.""" |
There was a problem hiding this comment.
pyproject.toml adds an autest pytest marker, but this test isn't marked with it. Adding @pytest.mark.autest would make it easier to run these roundtrip tests selectively (they depend on the gold_tests rule corpus).
| visitor = HRWInverseVisitor(filename=str(conf_file), merge_sections=False) | ||
| result = visitor.visit(tree) | ||
|
|
||
| assert result is not None, f"u4wrh produced no output for {conf_file.name}" | ||
| assert len(result) > 0, f"u4wrh produced empty output for {conf_file.name}" |
There was a problem hiding this comment.
This reverse test doesn't pass an ErrorCollector into HRWInverseVisitor, so any semantic issues won't be reported in a structured way (and warnings can't be asserted) compared to other tests that use ErrorCollector. Consider constructing an ErrorCollector here, passing it via error_collector=..., and asserting it has no errors before comparing output.
Extract hrw4u-specific changes from #12825. Fixes issues in the Python transpiler found during C++ parser development, normalizes autest .conf rules for bidirectional conversion, and adds roundtrip tests verifying hrw4u<->conf equivalence.