Feedback on Mastodon

[mattgodbolt]

Some excellent feedback on the example I posted from unlambda here: https://hachyderm.io/@unlambda/115249107176306941

Tracked also in compiler-explorer/compiler-explorer#8121

[mattgodbolt]

Conversation thread, edited slightly for brevity, with permission:

---

It's easy to try something like this out, and see it producing something that you might think useful at a quick glance, but I find that actually evaluating the utility of these beyond the novelty "hey, it's pretty cool that it can do this" is quite difficult, because it's hard to take into account all of the partial failure modes.

One of the things I'd like to see, especially from something like Compiler Explorer which is all about digging into the details of why something is doing what it's doing, is a copy of the input you sent to the LLM. For instance, on the linked example, one of my initial questions was "did it figure out that this was -O3 just from looking at the ASM, or were the compiler options provided in the context?" Looking at the source (https://github.com/compiler-explorer/explain/blob/main/app/prompt.py#L192-L201), it looks like we provide that in the context given to the LLM, but that wasn't immediately obvious, it would be great if I could view the request that was sent to the LLM with the full prompt and all of the context.

Then looking in more detail of the actual output. I'm looking at the example you linked; hopefully you see the same thing I do due to it being cached, but I'll try to quote specific relevant part.

It provides a bit of helpful context for a beginner: an explanation of the register calling conventions, which is helpful as a beginner trying to interpret this. But it immediately makes a confusing mistake: it says that rdi is the string view's data pointer, and rsi is the string view's length, but but in the actual ASM it's the other way around

For reference, here's what I"m seeing in the ASM:

cmp rdi, 16
jne .LBB0_1
movzx edx, byte ptr [rsi]

And the explanation:

rdi contains the first function argument (string view's data pointer)
rsi contains the second argument (string view's length)
eax will hold the return value (boolean result)

Then it explains the algorithm, to a degree. It explains "Subtract 48 (ASCII '0') from each character" which is good, helpful context. Then it says "Check if result is within a specific range" without really saying much about what that range means. Then "Use bit test (bt) instruction for fast character set membership" and it never explains or expands on this, it just comments the constant as "Precomputed bit mask" and says "This uses a clever bit manipulation technique to quickly check if a character is in the valid set (0-9, a-f)."

It also later discusses optimization observations, where it says:

This is highly optimized -O3 code
Uses minimal branching
Performs character validation in a tight, predictable loop
Avoids function calls by inlining validation logic

The first is trivially observed from the inputs.
The second and fourth, sure those happen, but also pretty much happen in any optimization, they aren't all that interesting. The third point is simply wrong, there is no loop after optimization, it has been fully unrolled.

And one more thing that it never explains, which would be helpful at the beginner level; that "al" and "eax" are the same register, just different ways of referring to it.

So, basically, it provides a little bit of useful context (the registers, the ASCII character that it's subtracting) but gets several details wrong, and it fails to explain the most interesting parts of the algorithm.

Is this helpful? Overall, not that much. The parts that are somewhat useful could probably be done in a deterministic way that would be always correct (for instance, providing some kind of tooltip hovers over registers that says what part of the calling convention they are, and maybe uses debuginfo to tell you what struct fields they correspond to).

The parts that are wrong are misleading.

And a lot of the rest is just kind of fluff.

I find this mix of somewhat useful, fluff, misleading, and plain wrong much more frustrating than nothing at all, because when you really need it and aren't paying attention, it's hard to filter out the misleading and wrong.

I would rather more effort put into exact, precise tools, that provide the kind of context that I would want, rather than a statistical text generator that produces a lot of misinformation.

[mattgodbolt-molty]

The feedback from this thread has been addressed — see compiler-explorer/compiler-explorer#8121 for details.

Key changes: model upgrade (Haiku 3.5 → Sonnet 4.6), prompt cleanup, and a correctness testing framework. The specific is_valid_id example that was critiqued now produces an accurate, detailed explanation.

Merged but needs an infra deploy to go live. This can likely be closed once deployed.

(I'm Molty, an AI assistant acting on behalf of @mattgodbolt)

[mattgodbolt-molty]

Closing — the feedback has been addressed in merged PRs #10 and #11. Needs infra deploy to go live.

(I'm Molty, an AI assistant acting on behalf of @mattgodbolt)