Skip to content

Port pycparserext to pycparser 3.0#102

Open
Copilot wants to merge 5 commits intomainfrom
copilot/port-pycparserext-to-pycparser-3-0
Open

Port pycparserext to pycparser 3.0#102
Copilot wants to merge 5 commits intomainfrom
copilot/port-pycparserext-to-pycparser-3-0

Conversation

Copy link
Copy Markdown

Copilot AI commented Mar 20, 2026

  • Explore repository and understand pycparser 3.0 API changes
  • Rewrite ext_c_lexer.py: replace PLY-based lexer with pycparser 3.0 CLexer subclass
  • Rewrite ext_c_parser.py: replace PLY grammar rules with recursive descent overrides
  • Update pyproject.toml: require pycparser>=3.0, requires-python~=3.10
  • All 41 tests pass
  • Fix _parse_attributes_opt: use None coord instead of string when no token present
  • Fix _parse_designator: pass [ token coord to RangeExpression for consistent location reporting
Original prompt

This section details on the original issue you should resolve

<issue_title>pycparser 3.0 compatibiilty</issue_title>
<issue_description>Is there interest in making pycparser-ext work with pycparser 3.0 ?

From #99 (comment) it seems this will require some work, but unclear if it is desirable to have this done or if the long term aim is to remain on pycparser < 3 </issue_description>

<agent_instructions>Port pycparserext to be compatible with pycparser 3.0. Preserve as much of the existing interface as possible, and pass the existing test suite.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@inducer I don't use pycparserext myself for anything at the moment, so it's hard for me to justify sinking time into even reviewing such a thing, given the anticipated scale. As such, I think a pycparser 3.0-compatible pycparserext would require a bigger break from the existing setup. I would be happy for someone who actually cares to take over maintainership. Alternatively, the project can be forked, too, but I can do a bit of work to help and avoid that if it helps provide a smoother upgrade path for users.

Btw: I had no idea this was used in a Chapel-adjacent tool, neat! 👋 Hi @bradcray, nice to (virtually?) see you!</comment_new>


🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Copilot AI changed the title [WIP] Port pycparserext to be compatible with pycparser 3.0 Port pycparserext to pycparser 3.0 Mar 20, 2026
Copilot AI requested a review from inducer March 20, 2026 16:43
Copilot AI and others added 2 commits March 20, 2026 12:22
…tests pass

Co-authored-by: inducer <352067+inducer@users.noreply.github.com>
Co-authored-by: inducer <352067+inducer@users.noreply.github.com>
@inducer inducer force-pushed the copilot/port-pycparserext-to-pycparser-3-0 branch from a1db7c3 to 16c9b71 Compare March 20, 2026 17:22
@inducer inducer force-pushed the copilot/port-pycparserext-to-pycparser-3-0 branch from 16c9b71 to 26b54a6 Compare March 20, 2026 17:28
@inducer inducer marked this pull request as ready for review March 29, 2026 18:26
Copilot AI review requested due to automatic review settings March 29, 2026 18:26
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Ports pycparserext to work with pycparser 3.x by replacing the old PLY-based lexer/parser integration with pycparser 3.0’s CLexer + recursive-descent CParser extension points.

Changes:

  • Bump runtime requirements to python~=3.10 and pycparser>=3.0, removing ply.
  • Rewrite GNU/OpenCL lexers as thin CLexer subclasses that remap extension keywords to custom token types.
  • Rewrite GNU/OpenCL parsers as recursive-descent overrides to support attributes, asm labels/statements, GNU/OpenCL qualifiers, and GNU case ranges.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
pyproject.toml Updates Python and dependency requirements for pycparser 3.x.
pycparserext/ext_c_lexer.py Replaces PLY token rules with CLexer keyword remapping and OpenCL line-comment handling.
pycparserext/ext_c_parser.py Reimplements extension parsing via recursive-descent overrides and AST extension nodes.
.github/workflows/ci.yml Updates CI Python versions to align with the new minimum supported Python.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +1059 to +1065
self._expect("RPAREN")
return compound
return super()._parse_primary_expression()

def _parse_statement(self):
tok_type = self._peek_type()
if self._is_asm_keyword_token(tok_type):
Copy link

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RangeExpression nodes created for GNU designator case ranges are missing coord information here (RangeExpression(expr, last) is called without coord), while other RangeExpression uses in this file do set coord. This results in coord=None for these nodes and makes location reporting inconsistent. Pass an appropriate coord (e.g., from the '[' token) when constructing RangeExpression in this branch.

Copilot uses AI. Check for mistakes.
Comment on lines +321 to +324
exprs = [self._parse_attribute()]
while self._accept("COMMA"):
exprs.append(self._parse_attribute())
coord = exprs[0].coord if exprs else None
Copy link

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In _parse_attributes_opt, when tok is None the ExprList coord is set to self.clex.filename (a string). pycparser AST coords are expected to be Coord objects (or None), so this can break downstream code that assumes coord has file/line attributes. Consider using a Coord (e.g., self._coord(0) / self._tok_coord of the last token) or simply None for the empty list case.

Copilot uses AI. Check for mistakes.
@inducer
Copy link
Copy Markdown
Owner

inducer commented Mar 29, 2026

@copilot apply changes based on the comments in this thread

Copy link
Copy Markdown
Author

Copilot AI commented Mar 29, 2026

@copilot apply changes based on the comments in this thread

Applied both fixes in 9e5df04:

  • _parse_attributes_opt: changed coord fallback from self.clex.filename (a string) to None when no token is present
  • _parse_designator: captures the [ token from _accept("LBRACKET") and passes its coord to RangeExpression

Copilot AI requested a review from inducer March 29, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pycparser 3.0 compatibiilty

3 participants