Add support for abi3.abi3t tag from PEP 803

[ngoldbaum]

This updates the tags logic to handle abi3t from PEP 803. Also adds tests to validate everything.

Opening as a draft for visibility and to aid end-to-end testing of PEP 803. See https://github.com/Quansight-Labs/stable-abi-testing.

[pfmoore]

The tags returned here should all be simple tags, not compound ones (i.e., no dots in the components).

[pfmoore]

Same here - no dots in returned tags.

[ngoldbaum]

@pfmoore, thank you! I also pushed an accompanying fix to my setuptools PR that allows setuptools to correctly support compressed ABI tags without incorrectly returning compressed tags from cpython_tags here. See the newest commit in pypa/setuptools#5193.

[pfmoore]

Just to be 100% clear here, when you returned abi3.abi3t, you did mean that the environment will work with either an abi3 or an abi3t wheel, didn't you?

You seem now to be saying that free threaded builds only support abi3t.

I think one of us is confused here 🙁

[ngoldbaum]

You seem now to be saying that free threaded builds only support abi3t.

Take a look at the test I added that checks the output of tags.cpython_tags((3, 16), ["cp316"], ["platform"] and note how there are abi3t tags for both 3.15 and 3.16.

https://github.com/pypa/packaging/pull/1099/changes#diff-c2a8158c2ea75e325dc5d418fe408374d215b771f40be4a907dca380ad4a6701R1060-R1080

It turns out setuptools doesn't currently support compressed ABI tags unless I make packaging produce them, which is why I thought that was necessary initially. I also had to slightly patch how setuptools checks whether tags are compatible by calling tags.parse_tags. See the change I made to the assert here: https://github.com/pypa/setuptools/pull/5193/changes#diff-5f0d2f159393c18a35b50b6a97212ace4cb44ef75df0c85dd28edb23dce9a782L356

[pfmoore]

OK, some thoughts. But I'm getting way out of my depth here, and I'm concerned that we're not going off the PEP here but rather trying to define behaviours that aren't clearly stated in the PEP. It's quite likely we need to get the PEP clarified before making decisions here.

For example, why is cp315-abi3t so low in the priority order in the cp316 example? Surely a cp315-abi3t wheel should be preferred over a cp310-abi3 one? Or is the ordering only implemented in sys_tags() and the low-level functions don't need to yield their values in preference order? This is where I don't understand the packaging implementation well enough to know what's acceptable - but I'm 99.9% sure that pip calls cpython_tags directly and relies on the values being returned in "most preferred first" order...

I can't comment on the setuptools point. But I do know that pip won't work if packaging returns a compressed tag.

I feel that this might need to be brought back to Discourse. At the very least, I think we need more eyes on the problem.

[ngoldbaum]

For example, why is cp315-abi3t so low in the priority order in the cp316 example?

Because it simply didn't occur to me that sys_tags is in priority order. But also it says it right there in the docstring:

packaging/src/packaging/tags.py

Lines 654 to 660 in cd1520c

	def sys_tags(*, warn: bool = False) -> Iterator[Tag]:
	"""
	Returns the sequence of tag triples for the running interpreter.
	The order of the sequence corresponds to priority order for the
	interpreter, from most to least important.
	"""

I adjusted the logic in my latest commit.

I feel that this might need to be brought back to Discourse. At the very least, I think we need more eyes on the problem.

I agree, let's ask Petr to clarify the tag priority order for the GIL-enabled build, since it's ambiguous in PEP 803 right now.

[ngoldbaum]

And after trying to draft a message about this, I think I finally understand what needs to happen here.

Installers should only install wheels on the free-threaded build with an abi3t tag, that means packaging should only produce abi3t tags on the free-threaded build.

I was confused in my original implementation, because I thought that packaging was responsible for generating compressed tag sets, but that's wrong. Build tools do that.

I was also confused by the fact that setuptools didn't handle compressed tag sets in abi tags. I'll raise this issue with the setuptools maintainers separately from PEP 803 support.

I agree that this is a little "sparely" described in the PEP. I'll post a followup to discourse about this.

[encukou]

Suggested change

	if use_abi3t and minor_version > 14:
	if use_abi3t:

[edit: practically this could be minor_version >= 13, but it's not worth the special case.]

[pfmoore]

Given the clarification in the Discourse thread, I'd strongly recommend adding some sort of test(s) here to validate that if you change the Python implementation from 316 to 316t, the list of tags remains unchanged except for abi3 changing to abi3t and cp316 changing to cp316t. And probably vice versa, for good measure 🙂

[pfmoore]

The Discourse discussion suggests that cp314-abi3t, cp313-abi3t, cp312-abi3t, ... should also be returned here...

[ngoldbaum]

I think the last push responds to all comments. Thanks for helping me to understand this better!

[henryiii]

This looks reasonable to me.

[dstufft]

This presumably should be >= (3, 15)?

[ngoldbaum]

No, see e.g. #1099 (comment)

The idea here is that in principle there could be abi3t builds targeting older Python versions. In practice someone would only try this with 3.14. Petr specifically asked not to preclude that possibility.

[dstufft]

I know that there's ongoing discussion around this, so I just want to follow up on this with a couple of thoughts:

• If we want to support abi3t prior to 3.15, then that's fine, but then it should probably be >= (3, 14) or >= (3, 13). The concept of abi3t on Python 3.2 is meaningless and is just generating extra tags to live in memory for no reason.
• Depending on how the discussion around PEP 803 goes, if abi3t is supported on a GIL Python, then this should probably not be gated against threading at all, and it's just that abi3t is supported on Python >= 3.15 (or 3.14, or 3.13, whatever) regardless.
• If the version check is not 3.15, we should add a comment about why when the the doc string clearly states that PEP 803 was first implemented in 3.15 (there's a comment down later, but I think it'd be better to do it here).

[henryiii]

I believe the plan was that you would have to specifically tag it with both tags. So if you want to load it on both, it needs abi3t.abi3. That leads the option later to drop GIL python, whereas otherwise you'd be stuck having to support both forever.

Agree making it 3.13+ seems reasonable, if we really want to leave the possibility for backporting support.

[dstufft]

AFAICT the plan is sometimes trying to treat abi3 and abi3t as compatible and sometimes not, which makes it hard to actually understand what's going on. Though I don't think having abi3t be supported on GIL Python means that GIL Python needs to be supported forever, rather CPython would have to support abi3t on GIL Python, for as long as GIL Python exists.

The abi3.abi3t compressed tag only currently works in PEP 803 because FT Python is pretending to understand abi3 at the filesystem level when it actually doesn't. I think that's wrong, and either CPython needs to commit to maintaining abi3t support for GIL Python, or an abi3.abi3t wheel would need to ship a .abi3.so and a .abi3t.so.

[dstufft]

It is wrong though, and it doesn't work. The only reason we have the tagged .so filenames at all is to support multiple versions of Python using the same directory via PEP 3149. Reusing .abi3.so to mean abi3t means that we're implicitly breaking that.

If we don't want to support multiple versions of Python using the same directory, then we should stop tagging the module filenames all together.

And of course, the extension machinery still supports an untagged .so, so if you wanted to you could ship a single abi3.abi3t wheel that shipped a single foo.so instead of foo.abi3.so, which would be just as meaningful as redefining what .abi3.so means.

[dstufft]

It only appears to work because the people who are testing it are using systems that are ensuring that they're treating abi3 and abi3t as distinct, despite the fact that CPython currently is not. The wheel tags are currently papering over the incompatibility.

Sure, that's a pragmatic choice (to avoid having to go change all build systems)

The build systems already have to be changed to understand how to produce abi3t builds to begin with, they have to be updated to understand the Py_GIL_DISABLED and Py_TARGET_ABI3T macros so that they can produce abi3t wheels. Otherwise they wouldn't use (or know about) that the existence of the abi3t tag at all.

Besides, there's what, less than 10 build systems in popular use? Breaking abi3 to prevent having to update ~10 projects just feels inherently wrong to me.

[rgommers]

Okay, the Debian use case discussion seems to belong on https://discuss.python.org/t/pep-803-round-two-abi3t-stable-abi-for-free-threaded-builds/106181/22. It is orthogonal to this PR though, which is purely about tags.

If we don't want to support multiple versions of Python using the same directory, then we should stop tagging the module filenames all together.

Well this keeps going wrong because it's apparently completely untested in CPython, and only half-supported. PEP 739 has the same issue - currently under discussion, because no one even noticed until after the feature had already shipped in Python 3.14.

If the outcome of discussing that is that build systems have to start producing .abi3t.so or a plain .so, I don't think it affects this PR at all?

[dstufft]

It is orthogonal to this PR though, which is purely about tags.

That's why my original comment was "Depending on how the discussion around PEP 803 goes" :)

If that discussion goes such that abi3t and abi3 are treated as wholly distinct and abi3t is supported by GIL Python, then the logic should be updated so abi3t is supported on both GIL and FT Python. I was just adding a marker to say that particular logic might want to change once that thread gets resolved one way or another.

Well this keeps going wrong because it's apparently completely untested in CPython, and only half-supported. PEP 739 has the same issue - currently under discussion, because no one even noticed until after the feature had already shipped in Python 3.14.

One mistakenly broken thing doesn't excuse purposefully breaking a thing :) And TBH maybe the right answer is to remove support for PEP 3149, I don't personally have a strong opinion on that other than if we do that, it should be done explicitly.

[pfmoore]

I mostly agree with @dstufft here, to the extent that I'm not willing to accept the packaging related side of PEP 803 until there's clarity on .so file naming and how that impacts what will get shipped in .abi3, .abi3t and .abi3.abi3t wheels.

I'm not enough of an expert in Linux ABI issues to have an opinion on what the correct approach should be, but I do think the PEP should be able to describe the solution clearly enough that a non-expert like me, interested only in the packaging aspects of the choice, can understand it.

Let's move this discussion to Discourse, as it affects the PEP itself, not just the implementation in packaging.