Feat/gin#177
Merged
Merged
Conversation
Unify generics and bare-name references under a new AliasType and introduce scope-based resolution (LocalScope / TypeScope). Types now accept an optional scope parameter on resolution-touching APIs (valid, parse, encode, compatible, like, simplify, props/get/call/init, etc.) and AliasType.resolve consults the caller-supplied scope before its captured scope. The commit adds types/alias.ts, updates README to document the new AliasType/TypeScope model and simplify(), removes the old generic/ref substitution model, and updates numerous tests and internals to use r.alias(...) and LocalScope instead of bind/substitute. Several test expectations were adjusted to reflect lazy alias resolution and stable toJSON behavior (call-site bindings do not mutate source definitions).
Add truncation for oversized validation messages and improve strict-mode schema handling.
- Introduce truncateValidationError (default cap 4096) and validationErrorMaxLength on PromptInput; apply truncation to JSON parse, schema parse, validation errors, and tool-arg parse paths. Pass the option into newToolExecution so tool-related errors are also bounded.
- Replace fragile z.codec usage for ZodRecord with z.preprocess to robustly normalize array-of-{key,value} inputs into records. Add a preprocess for ZodTuple to accept object-with-numeric-keys and convert to arrays before validation.
- In convertSchema, emit strict-mode tuples as objects with numeric-string keys to preserve per-position types (with fallbacks for rest/variadic tuples).
- Update many tests and fixtures to use the updated 'obj' extension name (replacing previous 'object' references) and include various ginny prompt/tool updates (new/modified ginny tool files and prompt changes).
These changes avoid massive zod error payloads that consume model context and make strict-mode conversions more robust for recursive and union schemas.
Enforce generic constraints at call-sites, improve object compatibility rules, and add editor tooling and tests. Changes include: - Add tests: generic-constraints.test.ts and obj-compatible-widening.test.ts; adjust person.test expected rendering for empty args. - Validate generics on CallStep.callSiteScope: parse bindings in called scope, check against declared constraints (skip self-referential alias as "no constraint"), and throw on violations; empty-args render as () in Path.toCode. - FnType: treat declared generics as constraint types (not bound placeholders) when parsing; use any() for unconstrained params; adjust compatibility semantics (args treated bivariantly for pragmatic structural checks). - ObjType.compatible: allow this-type optional fields to be absent on other, support exact mode to reject extras, and refine semantics used by edit-compat tooling. - Prop/getter evaluation and Path getter: catch ReturnSignal so saved fn/method bodies using flow:'return' unwind to their call boundary; import ReturnSignal where needed. - Expr schemas: remove spurious comment fields on lightweight expressions (flow/get/new) so strict-mode rejects noise. - Analysis/Engine: add optional ValidateContext to validate entry points so callers can mark root as already inside a loop or lambda. - Ginny natives: update generic constraints for ask/fetch/llm to reflect intended constraints (any / text|obj / any) instead of implying defaults. - Add edit_fn tool in designer: allows safe editing of saved functions with backwards-compat checks (args contravariant, returns covariant), spawns inner programmer for body authoring, and persists on success. Add edit-type tool to programmer prompt and expand programmer guidance (plan-and-approve workflow) and generic-binding docs. Rationale: these changes make generic parameters enforceable and explicit (constraints vs defaults), fix object compatibility to support edit/replace workflows, improve control-flow handling for saved bodies, and provide tools for safe on-disk function edits.
Introduce Registry.augment to let callers add props/get/call/init to named types (merged across calls; props are additive, get/call/init are first-wins). Wire augmentation into Type.props so augmented props are visible at runtime/static analysis and code rendering.
Reshape the loop `yield` into a path-shaped callable that accepts a single args object {key, value}. Update runLoop to provide a Value-wrapped callable and add natives.helpers.setupYield to efficiently build per-iteration callers with correct key/value types. Replace ad-hoc yield usage across native iterators (list, map, obj, tuple, text, num) to use setupYield and preserve accurate argument types, avoiding per-iteration type allocations.
Add extensive tests: empirical parallel/concurrency checks (gaps-parallel), loop coverage across map/obj/text (loop-coverage), and registry augmentation behavior (registry-augment) including init/call/get augmentation, schema shaping, and merging semantics.
Completely rewrites packages/gin/README.md to replace the previous quick-start and terse overview with a comprehensive, restructured reference. New content documents the type system surfaces (props, get, call, init), generics and TypeScope resolution, type compatibility rules, extensions vs augmentations, the 12 expression kinds, parsing model, the Registry API, and a full built-in type catalog. Updated example shows extension, augmentation, native registration, and engine.run usage. Removes older short examples and test/dump commands in favor of a longer, developer-oriented guide and sample code.
Add higher-level docs for @aeye/gin and @aeye/ginny to README; include install/usage notes and feature summaries. Bump @aeye/gin package to v0.3.8 and move puppeteer to optionalDependencies in package metadata. Tighten several gin expression schemas by removing comment/baseExprFields allowances (GetExpr, NewExpr, TemplateExpr) to enforce stricter shapes. Add GPL-3.0 LICENSE for ginny and introduce new runtime signal utilities (runtime-signal.ts, signal-utils.ts) along with multiple ginny source updates (ai, index, natives, tools, web content, esbuild config, package.json). Updated lockfile to reflect dependency changes.
Add strict identifier validation for type 'name', 'extends' and 'satisfies' (zod schema + registry.parse) to catch LLM-emitted junk early. Improve docs/descriptions for text/num type options to discourage over-specification. Harden runtime AbortSignal behaviour by raising listener limits (esbuild banner + CLI setMaxListeners) and patching AbortController so long-running libs/fetch don't hit listener caps. Revamp fns.fetch: add convert modes (markdown|raw), HEADLESS re-rendering for HTML via puppeteer, better content-type handling (binary/text/markdown), user-agent header, and separate typed-JSON path. Introduce genId() and wire 6-char ids into retry logging, LLM parse errors, tool errors, write validation messages, and progress logging so short user-facing one-liners link to full ginny.log context. Improve EventDisplay streaming semantics (stream termination, producedText flag) and make runRequest abort/cleanup behaviour friendlier. Update prompts/docs to describe the three fns.fetch modes, sequencing rules for create_new_fn, and guidance for not embedding JSON programs in prose.
Add auto-call support for prop methods that have no required args (runtime, type inference, and validate walk) and include tests (path-auto-call.test). Introduce isAutoCallable and synthesize empty-call behavior in Path.evaluate/typeOf/validate. Improve error messages by rendering full type code (safeTypeCode) and clarify if-condition warnings. Harden logging: cap serialized log output (~1MB), truncate long strings, and replace noisy full payload dumps with summarized provider hooks (with GIN_LOG_FULL_PAYLOAD to opt into full dumps). Hoist LLM prompt creation (use LlmInput) to avoid per-call prompt allocation and pass output schema per call. Also clean up prompt metadata typings (remove unnecessary as any).
Make template params optional and implement scope-fallback for placeholders: TemplateExpr now accepts an optional params Expr, evaluates params if present, and falls back to scope lookups for unresolved placeholders. Validator now errors on unresolved literal placeholders, respects typed params (including opaque any), and improves toCode rendering (inline literal fields, multi-line wrapping, and omission of with()). Improve define/type checks: skip strict declared-type checks when the inferred type is a universal placeholder and add an alias hint for unbound generics. Prop / set handling: path validation now distinguishes computed (read-only) props (error code set.prop.computed) while allowing method-typed props to pass. Prop read/write delegate to Type.propGet/propSet; default propGet/propSet implementations added to Type to support structural reads/writes (and clearer errors for non-object raws). Tests updated/added to cover template behaviors and set validation. Small ObjType and schema tweaks applied. Add a streaming Markdown renderer (MarkdownStream) and integrate it into ginny's EventDisplay to render streamed markdown chunks in the terminal.
Add Call and Prop type imports in path.ts and replace inline import('./type') references with the imported types to simplify type annotations. In registry.ts, give parseValue a default scope (TypeScope = this) so callers can omit the scope and the registry is used by default.
Add fine-grained JSONCode rendering for Type definitions and implement surface validation for embedded Expr bodies. Key changes: - New tests: type-jsoncode-demo.test.ts, type-jsoncode.test.ts, type-validate-surface.test.ts to exercise JSONCode spans and surface validation behavior. - Extension: merge registry augmentations into props/get/call/init resolution so augment(...) entries are visible alongside base and local definitions. - Registry: add Registry.validate(engine) to sweep and aggregate Problems for all named types and augmented built-ins. - Type: replace coarse toJSONCode with renderTypeDefJSONCode that emits spans per structural slot (props/get/call/init/constraint/etc.) and delegate embedded Expr rendering to parsed Expr.toJSONCode. - Type.validate now performs round-trip parse checks and a deep walk of the type surface (validateTypeSurface) to parse & validate embedded ExprDefs (using walkValidate), comparing inferred types against declared slots and emitting Problems. - Add helper functions for rendering type JSON, validating embedded expressions and building scopes for prop/call/init slots. These changes make formatter/validator tooling able to underline exact ranges inside type definitions and surface type-surface issues at registry validation time.
Introduce memory instrumentation and improve error/validation diagnostics across the codebase. Highlights: - packages/ginny: add logger.mem snapshots, mem probes, and per-turn/stream/tool/subagent memory markers to help leak hunting and surface payload sizes; track stream chunk/byte counts and emit periodic stream memory lines; avoid duplicating large problem blocks in model-visible tool results (keep full JSON in ginny.log). - packages/ginny/logger.ts: implement mem() with v8 heap limit, peak RSS detection, WARN flags, and probe support; register probes; simplify logObject to preserve full payloads (remove aggressive truncation logic). - packages/ginny/event-display.ts: sample streamed partials for memory snapshots, count chunks/bytes, log stream start/end and tool start/done mem markers. - packages/ginny/ai.ts: wire before/after/onError hooks to log mem snapshots and request/response payload sizes; register memory probes for engine globals and registry/session counters. - packages/ginny/progress.ts: add subagent start/done memory snapshots. - packages/gin/src/code.ts: deduplicate underline rendering by collapsing identical ranges and using severity ordering so the most severe color wins. - packages/core/src/prompt.ts: improve tool-argument parsing to treat empty string as missing args (produce clearer validation message) and avoid sending full JSON schema in some validation errors (remove toJSONSchema usage in one message). - packages/core/src/schema.ts: remove unused getInputSchema helper and extend JSONSchemaFormat to include 'anthropic' and 'google'. - packages/ginny/tools/write.ts: avoid duplicating large validation blocks into the model-visible tool result (keep concise TS snippet for model and full JSON in ginny.log), reducing serialized conversation history and resident memory. Why: these changes add fine-grained, low-overhead memory logging and diagnostics to make it easier to find streaming or history-related memory growth, produce clearer parsing/validation errors for the model, and reduce unnecessary duplication of large payloads in model-visible responses to mitigate OOMs.
Introduce end-to-end strict-mode handling across AI packages: docs, types, registry, APIs, providers, and tests. Key changes: - Docs: add Strict Mode sections to packages/ai and packages/core READMEs and docs/guides. - Types: add 'toolsStrict' capability, 'toolsStrict' parameter, and ModelInfo.strictFormat field to declare/describe strict JSON-Schema dialects. - Registry: implement resolveStrictFormat, auto-derive toolsStrict from explicit strictFormat overrides, propagate strictFormat through model merging, and add getStrictFormat to pick a FormatDescriptor or LENIENT fallback. - API layer: BaseAPI now tracks optional capabilities/parameters; ChatAPI promotes per-tool strict flags into required (true) vs optional (numeric/default) capabilities/parameters. - Providers: AWS Bedrock support for strict: pick model info from context, build schema budgets, pick and attach descriptors per-tool, strictify tool schemas, and thread ctx into request conversion. Added supportedStrictFamilies and buildSchemaBudget helper. - Models/usage: expose strictSupport from @aeye/models in the example (cletus) so curated overrides opt models into toolsStrict. - Tests & mocks: add strict-related unit tests (chat-strict, strict-support), update provider mock embedding usage shape, and adjust core tests to snapshot mutated messages and reasoning shape. - Exports: export './common' from packages/ai index. Overall this enables per-model strict dialect resolution, marks strict-capable models via curated overrides, treats numeric/omitted strict as a preference (optional) while true is a hard requirement, and ensures providers can emit the proper strict wire shape with graceful lenient fallback.
Add pairing guarantees and synthesis for unpaired tool_calls. Introduces a toolsComplete flag (default true) that synthesizes placeholder tool results ([aborted:], [interrupted], [error: …], etc.) for any tool_call left without a real result due to aborts or ToolInterrupts, while preserving PromptSuspend semantics (suspended tools remain unpaired). Implements abort-aware dispatch to avoid starting further tools once the signal is aborted, returns early from the main loop on aborted signal, and moves/truncates validation error logic into a protected Prompt.truncateValidationError for overrides. Adds Prompt.synthesizeUnpairedResult to generate placeholders and threads a truncator into newToolExecution. Also adds comprehensive tests (packages/core/src/__tests__/prompt-tool-result-pairing.test.ts) covering parallel/sequential aborts, ToolInterrupt, opt-out behavior (toolsComplete: false), and suspend semantics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.