Implement GMPL → MUIO conversion pipeline (ClickSAND / UTOPIA to OSeMOSYS UI JSON)

[parthdagia05]

Feature: OSeMOSYS GMPL to MUIO JSON Transformation Pipeline ⚙️

Closes: #1

Summary

This PR implements a complete, structured pipeline for transforming standard OSeMOSYS GMPL models (e.g., ClickSAND / UTOPIA format) into MUIO-compatible JSON case structures. It automates the conversion of standard OSeMOSYS .dat files into the structured JSON format required by the OSeMOSYS UI and Cloud platforms.

The implementation follows a modular 3-layer architecture to ensure clean separation between syntax parsing, semantic expansion, and schema transformation.

---

Motivation

Currently, two primary data presentation formats exist within the ecosystem:

1. Standard OSeMOSYS GMPL: Used by ClickSAND and starter data kits.
2. MUIO / OSeMOSYS Cloud: ZIP archives containing structured JSON records.

This PR bridges the gap by introducing a Python-based transformation engine capable of parsing standard GMPL, interpreting wildcard slice semantics, and generating exact MUIO JSON schemas.

---

Architecture

The pipeline flows through three distinct phases:

GMPL (.dat / .txt) → Phase 1: GMPLParser → Phase 2: SliceInterpreter → Phase 3: MuioTransformer → MUIO JSON

Phase 1 — GMPLParser (Syntax Layer)

Focuses on pure structural extraction of GMPL content without applying transformation logic.

• Capabilities: Handles set definitions, multi-slice parameters, headerless tables, and mixed whitespace.
• Robustness: Manages := placement variations, comment stripping (#), and end; terminations.
• Dialects: Supports both Dense (UTOPIA-style) and Sparse (MUIO-style) formatting.

Phase 2 — SliceInterpreter (Semantic Layer)

Expands GMPL slice notation into normalized tuples.

• Expansion: Converts wildcard slices (e.g., [REGION,*,FUEL,MODE,*]) into normalized mapping: (region, tech, fuel, mode, year) → value.
• Features: Numeric coercion to float, default-value filtering, and sparse tuple representation.
• Registry: Uses an explicit dimension registry to avoid length inference errors.

Phase 3 — MuioTransformer (Application Layer)

Transforms normalized tuples into the final MUIO JSON schema.

• Mapping: Renames FUEL to COMMODITY and injects MUIO-specific sets (STORAGEINTRADAY, UDC, etc.).
• Determinism: Generates deterministic IDs (e.g., TECHNOLOGY → T_0, COMMODITY → C_0).
• Metadata: Generates genData.json and creates the scenario envelope (SC_0).

---

Validation & Testing

Comprehensive validation scripts are included to ensure transformation accuracy:

• validate_parser.py | validate_interpreter.py | validate_transformer.py

Testing Fixtures:

• UTOPIA: Verified against the dense canonical OSeMOSYS model.
• MUIO-Sparse: Verified against sparse GMPL samples.

Assertions Covered: Correct slice expansion, ID determinism, set injection, and JSON record shape completeness. All checks pass.

---

Impact & Compatibility

• Backward Compatible: No changes to existing solver execution or case workflows.
• Purely Additive: Functions as a standalone engine for interoperability.
• Integration Ready: Provides the foundation for future CLI wrappers or automated cloud upload tooling.

---

Future Extensions (Optional)

While the core engine is complete, a small wrapper can be added to:

1. Write JSON files directly to disk.
2. Package output into a standardized ZIP case archive.

---

Feedback on architecture, mapping completeness, or integration strategy is welcome.