[BUG] Additions to the OperatorTable don't appear to be registering correctly

[cbrazinski]

Description

When registering a new assignment operator with the OperatorTable, attempts to use the new operator result in an Exception: Sequence does not respond to <newly added operator>. Interestingly, the operator does seem to function for objects formatted as a string and run with doString(). The operator will also work correctly when registered the exact same way via the Io Repl. Various Stack Overflow questions seem to concern others noticing this behavior. This question has one answer implying this behavior is actually correct/expected, but is followed by an answer from the original poster claiming that they saw different behavior upon running a different version of Io. The response to this question once again implies that this behavior is known and expected and explains this is the result of trying to limit parsing of source files to a single pass. The answer also mentions using doFile() as a work around.

Is this a bug or intended behavior? If this is working as intended, I'll mention I found it unintuitive and surprising, especially given that other similar features aren't subject to this limitation (e.g. assigning custom curly brace behavior).

Steps to Reproduce

OperatorTable addAssignOperator(":", "atPutNumber")
curlyBrackets := method(
	r := Map clone
	call message arguments foreach(arg,
		r doMessage(arg)
	)
	r
)

Map atPutNumber := method(
	self atPut(
		call evalArgAt(0) asMutable removePrefix("\"") removeSuffix("\""),
		call evalArgAt(1)
	)
)

workingMap := doString("{ \"Bob\" : \"1234\", \"Mary\" : \"5678\" }")
workingMap type println
workingMap keys println
workingMap foreach(k,v, writeln(k, " : ", v))

triggersException := { "Bob" : "1234", "Mary" : "5678" }
triggersException type println
triggersException keys println
triggersException foreach(k,v, writeln(k, " : ", v))

Expected Behavior

I would expect both workingMap and triggersException in the above example to both evaluate the new operator ( ':' ) in the same manner, in this case resulting in identical Maps, each containing two elements:

$ io map_literal.io 
Map
list(Bob, Mary)
Bob : 1234
Mary : 5678

Map
list(Bob, Mary)
Bob : 1234
Mary : 5678

Actual Behavior

Instead, while workingMap evaluates as expected, triggersException results in a Sequence does not respond to ':' Exception.

$ io map_literal.io 
Map
list(Mary, Bob)
Mary : 5678
Bob : 1234

  Exception: Sequence does not respond to ':'
  ---------
  nothing on stack

Environment

• Io Version: Io Programming Language, v. 20260302
• Operating System: Ubuntu 24.04.4 LTS
• Architecture: x86_64 (AMD)
• Build Type: from source

[stevedekorte]

This is a real limitation but follows from a deliberate design choice — Io parses an entire source unit and runs the operator-shuffle pass once before any code in that unit executes. doString and the REPL work because each invocation is a fresh parse against the current OperatorTable. Inside a single file, statements that come after OperatorTable add*Operator(...) were already shuffled with the old table, so the new operator silently doesn't apply to them.

We thought through several ways to actually fix this. Sharing the design space and our recommendation, looking for feedback before doing anything.

Options considered

1. Per-statement / per-chain-segment shuffle. Defer shuffle so each top-level statement gets shuffled with the OperatorTable as it stands at that moment. Falls apart immediately because Io has no statements — a source file is one big message chain. There's no clean unit to defer on.

2. Re-shuffle the rest of the chain when the OperatorTable mutates. When addOperator runs, walk call message next and shuffle from there forward. Works for the OP's case (adding a new operator), because unknown operators leave the chain structurally intact at first-shuffle time — \"Bob\" : \"1234\" is still three sibling messages, exactly the pattern the shuffler matches on. The second pass picks it up.

But it breaks for precedence/arity changes. a + b * c with default precedence first-shuffles to a +(b *(c)). If you later make * the same precedence as +, re-shuffle is a no-op — both messages already have args, so the shuffler's trigger pattern ("operator with no args followed by operand sibling") doesn't match. You silently get the wrong answer with no error.

3. Make shuffle reversible. Mark a flag bit on each Message indicating "my args were inserted by the shuffler, not the parser." Unshuffle walks post-order, lifting flagged args back out as siblings. Then re-shuffle. Explicit user parens are untouched because their flag was never set. Handles precedence changes correctly.

Wrinkles: assignment-style rewrites change the message name (a := b → setSlot(\"a\", b)), so the flag isn't enough — you'd need to remember the original operator name on rewritten nodes. Also doesn't help previously-defined Block bodies, which were shuffled with the old table and stored on the Block.

Cost is real: a flag bit (and sometimes a symbol pointer) on every Message, plus the unshuffle/reshuffle logic, plus GC implications, plus the Block-body caveat.

Recommendation: warn, don't fix

We think the right move is not to implement any of the above and instead emit a warning. In OperatorTable add*Operator, check call message next:

• non-null → there's already-shuffled code ahead of this point in the same chain that won't see the change. Log a warning naming the operator and pointing at doString / loading the operator definitions from a separate file as the workaround.
• null → end of chain (REPL line, last expression of a file). Change takes effect. Stay quiet.

This gives users an immediate, actionable diagnostic instead of a confusing Sequence does not respond to ':' twenty lines later. ~10 lines of code, no GC surface, no perf cost, no risk of breaking working code. The proper fix stays open if anyone actually hits a case the warning can't address.

Thoughts? Anyone strongly want option 2 or 3 for a real use case?

[cbrazinski]

Hey Steve,

Thanks for giving this so much thoughtful consideration. I think the behavior I'd expected was option 2, but I had not considered the issue of a mid-run change to precedence rules conflicting with an already existing message chain. I agree that the overhead of option 3 seems excessive for what is surely something of a niche use case.

A clearer error message/warning would help a lot, I spent a while genuinely thinking I simply wasn't setting the operator correctly and seeing it work in the REPL was initially more baffling than reassuring. Provided it's clear what's going on, the status quo isn't unmanageable for the use case of adding a new operator. Upon closer inspection, I also see that the current behavior is documented, though easy to miss.

At risk of growing the scope of the issue, in the circumstance that the behavior is left as is...
It's quite confusing that given two files new_operator.io and program_that_uses_new_operator.io the line doFile("program_that_uses_new_operator.io") included in new_operator.io results in correct functionality, while the reverse does not. I understand that doFile() isn't intended to function like a #include, but the end result is that you wind up with use cases included inside the dependency instead of the reverse. It would be nice if there was some way to do the following:

includeAndShuffleOpTable("map_literal.io")

noLongerTriggersException := { "Bob" : "1234", "Mary" : "5678" }

[stevedekorte]

Following up on the previous comment — after more thought and looking at how other languages handle this, I want to walk back the "just warn" recommendation a bit. There is a cleaner real fix that I initially dismissed.

How other languages solve this

Every language that supports runtime-addable operators shares one property: parse and execute are interleaved at some granularity. None of them parse-the-whole-file-then-execute. The granularity is what differs.

• Prolog — reads one term at a time. :- op(Prec, Type, Name). is a directive: it runs the moment the reader encounters it, and the next term is read with the updated operator table. (SWI-Prolog docs)
• Common Lisp — the reader consumes one top-level form at a time, consulting a mutable readtable. set-macro-character (typically wrapped in EVAL-WHEN) affects how the reader parses subsequent forms in the same file. (Reader macros reference)
• Forth — parses and executes word by word. A newly-defined word immediately affects parsing of the next word. Extreme version of the same idea.
• Raku — sub infix:<foo>(...) installs an operator into the grammar for the current lexical scope only. (Creating operators in Raku)
• Haskell / Swift / OCaml / Scala — sidestep the problem by requiring operators to be declared before use, no runtime addition.

The unifying lesson: if you want runtime-addable operators with sane semantics, you need to process the source in units and shuffle each unit against the operator table as it stands at that moment.

What this suggests for Io

In the previous comment I dismissed "per-statement shuffle" because Io has no statements — only expressions. That's true semantically, but the parser has phrase boundaries: ; and newlines at depth zero produce terminator messages in the chain, and operators can't be shuffled across terminators anyway (operators combine with immediate siblings; a terminator breaks adjacency). So the granularity already exists in the parse tree, even though the language has no statement concept.

Concrete proposal:

1. Parse the whole source into a message tree (unchanged).
2. Walk the top-level chain in phrase-sized segments (between terminators at depth 0).
3. For each segment: shuffle it using the current OperatorTable, then evaluate it. Move to next.

This is the Prolog/CL pattern applied to Io. It cleanly handles:

• New operators (the OP's case here) — segment that calls addOperator(":") runs, the next segment is shuffled with : known, works.
• Precedence and arity changes — same mechanism; no need for the unshuffle/reshuffle with marker bits that I sketched earlier.
• REPL and doString — no longer a special case; each is just a sequence of phrases shuffled against the current table.
• Method bodies — shuffled at method(...) evaluation time. Later operator changes don't retroactively rewrite the body. This exactly matches Lisp: defun captures expansion at compile time; a later defmacro doesn't reach back. Generally considered correct.

Cost compared to the earlier options

Compared to the unshuffle/reshuffle proposal with marker bits: much simpler — no flag bit per Message, no reverse-shuffle, no GC surface change, no special case for assignment-style rewrites, no perf cliff. Total shuffle work is the same, just sliced differently.

Compared to just warning: it's a real fix rather than a diagnostic, but it does touch the eval entry points (currently parse → shuffle → eval; would become parse → [shuffle phrase → eval phrase]*). Not huge, but load-bearing code.

The one semantic risk worth checking before committing: is there any code that relies on whole-file shuffle being visible up front — e.g. a phrase that uses an operator defined by a later phrase in the same file? In practice operators are almost always defined in bootstrap before any user code runs, so this is unlikely, but worth grepping for in the stdlib first.

Revised recommendation

Short term: implement the warning as previously suggested — ~10 lines, zero risk, immediately improves the diagnostic experience for users who hit this today.

Long term: per-phrase shuffle-eval is the proper fix. It's what every language with this feature has converged on, it's cleaner than the marker-bit approach, and it dissolves the bug rather than papering over it.

Still looking for feedback — does per-phrase shuffle-eval step on anyone's existing use case? And is there interest in doing the real fix, or is the warning enough?

[stevedekorte]

One more design thread to put on the table — this one about scope of operator changes, which turns out to be orthogonal to the timing issue and arguably more important.

The underlying smell

The global OperatorTable is itself a bit of a problem, independent of the parse-order bug. A library that adds : for its config-literal syntax silently changes parsing for every subsequent doString/doFile in the program. Tests pollute each other. REPL sessions leak state into scripts. Any mid-program mutation risks breaking unrelated code downstream. Raku explicitly makes operators lexically scoped to avoid exactly this; Racket's #lang and Scheme's let-syntax are the same impulse — parser state should be explicit, not ambient.

Proposal: first-class Compiler objects, per-object override via slot lookup

This falls out naturally in Io's model:

• Compiler (PascalCase) — the prototype. Lives in the normal proto chain (e.g. Protos Core Compiler). Holds operatorTable, plus methods like doString, parseMessage, shuffleMessage.
• compiler (camelCase) — a per-object slot. Default on Object points to the root Compiler; anyone who wants a custom parser does:

  myHtmlParser compiler := Compiler clone do(
      operatorTable addAssignOperator(":", "atPutNumber")
  )

• Object doString / Object doFile become thin wrappers: look up compiler via normal prototype lookup, dispatch to its doString. No new dispatch rules — just standard Io slot resolution.

Any object can shadow compiler to use its own dialect. Objects that don't override inherit the default. No globals, no special cases.

What this gives you

• No cross-contamination. Mutating one Compiler's OperatorTable doesn't touch anyone else's. Test A adding : doesn't break test B. Library X's DSL doesn't wreck library Y.
• First-class dialects. HTML-ish parser? SQL-ish embedded syntax? Sandboxed restricted parser? Each is just a Compiler clone with a tailored operatorTable. Fully composable.
• Sandboxing story. A restricted Compiler with a stripped operator table becomes a principled way to evaluate untrusted source — untrustedCtx doString(input) can't parse forbidden forms.
• Generalizes beyond operators. Future parser state (macro-like hooks, reserved words, custom literals) hangs off the same Compiler object. No further API surgery required.
• Lexical capture is free. Method bodies are parsed at definition time by whatever Compiler parsed the enclosing source — the shuffled tree is already frozen. You get Raku-style lexical operator scoping without a separate mechanism.

No backward compat for direct OperatorTable references

The current OperatorTable singleton can just become "the default Compiler's operatorTable". Code that accesses it globally either migrates to going through a Compiler, or — if we keep a shim — mutations land on the default and still affect default-compiled code. But the cleaner story is to break it and require all access through a Compiler.

How this composes with the timing fix

The two proposals are independent and complementary:

Problem	Fix
Timing — operators defined earlier in the same source don't affect code below	Per-phrase shuffle-eval (previous comment)
Scope — global operator mutations leak across unrelated contexts	First-class Compiler objects with compiler-slot dispatch

Do both and you get: within a single source unit, operator definitions take effect immediately for subsequent phrases (timing fixed); across unrelated contexts, they don't leak (scope fixed). The OP's original case works transparently, and the nastier downstream problems are ruled out at the API level.

Layering for implementation

• Land the warning now — 10 lines, zero risk, fixes the confusing diagnostic for users hitting this today.
• Per-phrase shuffle-eval as the real timing fix — correct semantics matching Prolog/CL precedent.
• First-class Compiler objects as the proper scoping story — matches Raku's lexical approach, modeled Io-idiomatically via slot lookup.

Happy to flesh out the Compiler proto's method set or sketch migration notes if there's interest. Still fishing for feedback — is anyone actively using runtime OperatorTable mutation in a way these would disrupt?

[stevedekorte]

Thoughts?

[stevedekorte]

Investigated — reproduces on both wasm and master.

Root cause

Still present because of single-pass parsing in IoMessage_newFromText_labelSymbol_ (libs/iovm/source/IoMessage_parser.c:68-88). When a file is loaded:

1. Entire file parsed → message tree
2. IoMessage_opShuffle_ walks the tree once, using the current OperatorTable to rewrite a := b into setSlot("a", b), handle precedence, insert new operators, etc.
3. Evaluator executes the shuffled tree

If a file contains OperatorTable addAssignOperator at line 1 and a use of that operator at line 5, lines 1–5 are parsed and opShuffled together before line 1 executes — so line 5 is shuffled against the old OperatorTable. doString works because it parses + shuffles at runtime, after the operator is registered.

Why this isn't trivial to fix

The fix requires decoupling opShuffle from parse time. Two viable approaches:

1. Per-statement shuffle on demand: at the top level of a file, don't shuffle the whole tree up front. Instead, before each top-level statement executes, re-shuffle it (or shuffle-and-cache it). Cost: one extra pass per top-level statement.

2. OperatorTable version check: record the OperatorTable's "version" when shuffling a tree. Before each top-level statement executes, check if the version changed; if so, reshuffle the remaining chain.

Either approach touches the parser, the eval loop, and assumes the OperatorTable is the only side-channel opShuffle depends on — which needs verification.

Workaround

As noted in the SO threads and in the issue: wrap code that uses the new operator in doString, or put the operator definition in one file and the usage in a separate file loaded afterward.

Leaving open as design/parser work. Flagging this as accurately diagnosed rather than closing, since it is a real bug, not intended behavior.

[cbrazinski]

To clarify, in step 1 of comment 4276585292 you mention "Parse the whole source into a message tree (unchanged)." Doesn't this need to change to "Parse only the current phrase" for this solution to work? I'll admit I'm not that familiar with your implementation under the hood, but I thought the underlying problem you initially highlighted was that by the time the message tree has been built, any change to precedence potentially invalidates the tree representing X op1 Y op2 Z. The pathological case here would be prompting the user at runtime for the desired precedence of a new operator. Perhaps this is already what you were suggesting and I've simply misunderstood.

As a user, this "per phrase" parse/eval approach is appealing as it leads to consistent behavior with how the REPL and doString()/doFile() already behave.

Addressing your next comment concerning the introduction of Compiler objects with a scoped OperatorTable, are you suggesting eventually completely removing the global OperatorTable? Or are you simply suggesting overwriting the slot as an alternative (and clearly safer) option in addition to the one used at global scope? While I haven't personally run into it, I agree with the problem that you are trying to solve here. This idea of including a dependency/test/etc that modifies your OperatorTable underneath you and affects the behavior of your program does sound like a problem. Although it does seem like distinct problem from the one that I initially highlighted in this bug.