Polish JSON ingestion extension APIs and docs

AGENTS.md

@@ -85,9 +85,10 @@ All gems follow the pattern: `elasticgraph-[name]/` containing:
 - `elasticgraph-elasticsearch`: Elasticsearch client wrapper
 - `elasticgraph-opensearch`: OpenSearch client wrapper
 
-**Extensions** (5 gems): Optional functionality
+**Extensions** (6 gems): Optional functionality plus the default JSON Schema ingestion serializer
 - `elasticgraph-apollo`: Apollo Federation support
 - `elasticgraph-health_check`: Health checks
+- `elasticgraph-json_ingestion`: JSON Schema ingestion serializer
 - `elasticgraph-query_interceptor`: Query interception
 - `elasticgraph-query_registry`: Source-controlled query registry
 - `elasticgraph-warehouse`: Data warehouse ingestion

CODEBASE_OVERVIEW.md

@@ -197,11 +197,12 @@ graph LR;
 
 ### Extensions (6 gems)
 
-These libraries extend ElasticGraph to provide optional but commonly needed functionality.
+These libraries extend ElasticGraph to provide commonly needed functionality. Most are optional; `elasticgraph-json_ingestion`
+is the default schema-definition ingestion serializer and is wired into generated ElasticGraph projects as an explicit extension.
 
 * [elasticgraph-apollo](elasticgraph-apollo/README.md): Transforms an ElasticGraph project into an Apollo subgraph.
 * [elasticgraph-health_check](elasticgraph-health_check/README.md): Provides a health check for high availability ElasticGraph deployments.
-* [elasticgraph-json_ingestion](elasticgraph-json_ingestion/README.md): JSON Schema ingestion support for ElasticGraph.
+* [elasticgraph-json_ingestion](elasticgraph-json_ingestion/README.md): Default JSON Schema ingestion serializer for ElasticGraph.
 * [elasticgraph-query_interceptor](elasticgraph-query_interceptor/README.md): Intercepts ElasticGraph datastore queries.
 * [elasticgraph-query_registry](elasticgraph-query_registry/README.md): Provides a source-controlled query registry for ElasticGraph applications.
 * [elasticgraph-warehouse](elasticgraph-warehouse/README.md): Extends ElasticGraph to support ingestion into a data warehouse.
@@ -336,4 +337,3 @@ graph LR;
     click aws-sdk-s3 href "https://rubygems.org/gems/aws-sdk-s3" "Open on RubyGems.org" _blank;
     click faraday_middleware-aws-sigv4 href "https://rubygems.org/gems/faraday_middleware-aws-sigv4" "Open on RubyGems.org" _blank;
 ```
-

elasticgraph-apollo/README.md

@@ -49,7 +49,7 @@ index 4a5ef1e..5c16c2b 100644
 
 ```
 
-Finally, update `Rakefile` so that `ElasticGraph::GraphQL::Apollo::SchemaDefinition::APIExtension` is
+Finally, update `Rakefile` so that `ElasticGraph::Apollo::SchemaDefinition::APIExtension` is
 used as one of the `extension_modules`:
 
 ```diff
@@ -67,8 +67,8 @@ index 2943335..26633c3 100644
 @@ -12,6 +13,8 @@ ElasticGraph::Local::RakeTasks.new(
    local_config_yaml: settings_file,
    path_to_schema: "#{project_root}/config/schema.rb"
- ) do |tasks|
-+  tasks.schema_definition_extension_modules = [ElasticGraph::Apollo::SchemaDefinition::APIExtension]
+) do |tasks|
++  tasks.schema_definition_extension_modules += [ElasticGraph::Apollo::SchemaDefinition::APIExtension]
 
 ```
 

elasticgraph-apollo/lib/elastic_graph/apollo/schema_definition/api_extension.rb

@@ -35,7 +35,7 @@ module ElasticGraph
   #     local_config_yaml: "config/settings/local.yaml",
   #     path_to_schema: "config/schema.rb"
   #   ) do |tasks|
-  #     tasks.schema_definition_extension_modules = [ElasticGraph::Apollo::SchemaDefinition::APIExtension]
+  #     tasks.schema_definition_extension_modules += [ElasticGraph::Apollo::SchemaDefinition::APIExtension]
   #   end
   module Apollo
     # Namespace for all Apollo schema definition support.
@@ -55,7 +55,7 @@ module SchemaDefinition
       #     local_config_yaml: "config/settings/local.yaml",
       #     path_to_schema: "config/schema.rb"
       #   ) do |tasks|
-      #     tasks.schema_definition_extension_modules = [ElasticGraph::Apollo::SchemaDefinition::APIExtension]
+      #     tasks.schema_definition_extension_modules += [ElasticGraph::Apollo::SchemaDefinition::APIExtension]
       #   end
       module APIExtension
         # Applies an apollo tag to built-in types so that they are included in the Apollo contract schema.

elasticgraph-json_ingestion/README.md

@@ -1,9 +1,79 @@
 # ElasticGraph::JSONIngestion
 
-JSON Schema ingestion support for ElasticGraph.
+Default JSON Schema ingestion support for ElasticGraph.
 
 This gem provides the schema-definition extension that generates JSON Schema artifacts for indexing
-events and validates JSON-ingestion-specific schema options.
+events and validates JSON-ingestion-specific schema options. It is installed with
+`elasticgraph-schema_definition`; applications enable it by adding
+`ElasticGraph::JSONIngestion::SchemaDefinition::APIExtension` to the schema-definition extension modules.
+
+## Schema Definition APIs
+
+Use `schema.json_schema_version` to identify the current JSON schema artifact. Every change that affects
+the JSON schema should increment this version so publishers and indexers can safely evolve independently.
+
+```ruby
+ElasticGraph.define_schema do |schema|
+  schema.json_schema_version 1
+end
+```
+
+Use `schema.json_schema_strictness` to configure whether indexing events may omit nullable fields or include
+extra fields. We recommend enabling at most one of these options, because enabling both can hide misspelled
+event fields.
+
+```ruby
+ElasticGraph.define_schema do |schema|
+  schema.json_schema_strictness allow_omitted_fields: true, allow_extra_fields: false
+end
+```
+
+Custom scalar types must declare how they are represented in JSON Schema:
+
+```ruby
+ElasticGraph.define_schema do |schema|
+  schema.scalar_type "URL" do |t|
+    t.mapping type: "keyword"
+    t.json_schema type: "string", format: "uri"
+  end
+end
+```
+
+Fields and object/interface types can add JSON Schema validations. Use field-level validations sparingly:
+they run while indexing events, so violations can send otherwise valid source-system data to the dead letter
+queue. They are best reserved for constraints that ElasticGraph needs in order to index correctly.
+
+```ruby
+ElasticGraph.define_schema do |schema|
+  schema.object_type "Card" do |t|
+    t.field "id", "ID!"
+
+    t.field "expYear", "Int" do |f|
+      f.json_schema minimum: 2000, maximum: 2099
+    end
+
+    t.field "expMonth", "Int" do |f|
+      f.json_schema minimum: 1, maximum: 12
+    end
+
+    t.index "cards"
+  end
+end
+```
+
+On fields, `nullable: false` disallows `null` in indexing events while keeping the GraphQL field nullable:
+
+```ruby
+ElasticGraph.define_schema do |schema|
+  schema.object_type "Widget" do |t|
+    t.field "name", "String" do |f|
+      f.json_schema nullable: false
+    end
+
+    t.index "widgets"
+  end
+end
+```
 
 ## Dependency Diagram
 

elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/api_extension.rb

@@ -9,15 +9,7 @@
 require "elastic_graph/constants"
 require "elastic_graph/graphql/scalar_coercion_adapters/valid_time_zones"
 require "elastic_graph/json_ingestion/schema_definition/factory_extension"
-require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/enum_extension"
-require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/object_extension"
-require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/scalar_extension"
-require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/union_extension"
 require "elastic_graph/json_ingestion/schema_definition/state_extension"
-require "elastic_graph/schema_definition/indexing/field_type/enum"
-require "elastic_graph/schema_definition/indexing/field_type/object"
-require "elastic_graph/schema_definition/indexing/field_type/scalar"
-require "elastic_graph/schema_definition/indexing/field_type/union"
 
 module ElasticGraph
   module JSONIngestion
@@ -47,20 +39,12 @@ module APIExtension
           "LongString" => {type: "integer", minimum: LONG_STRING_MIN, maximum: LONG_STRING_MAX}
         }.freeze
 
-        # Wires up the factory extension when this module is extended onto an API instance.
+        # Wires up the JSON ingestion extensions when this module is extended onto an API instance.
         #
         # @param api [ElasticGraph::SchemaDefinition::API] the API instance to extend
         # @return [void]
         # @api private
         def self.extended(api)
-          # Prepend our indexing-field-type extensions onto the core classes so they participate in
-          # `to_json_schema` / `format_field_json_schema_customizations` / `json_schema_field_metadata_by_field_name`.
-          # Guarded so re-extending an already-extended API instance is a no-op.
-          ElasticGraph::SchemaDefinition::Indexing::FieldType::Enum.prepend(Indexing::FieldType::EnumExtension) unless ElasticGraph::SchemaDefinition::Indexing::FieldType::Enum < Indexing::FieldType::EnumExtension
-          ElasticGraph::SchemaDefinition::Indexing::FieldType::Object.prepend(Indexing::FieldType::ObjectExtension) unless ElasticGraph::SchemaDefinition::Indexing::FieldType::Object < Indexing::FieldType::ObjectExtension
-          ElasticGraph::SchemaDefinition::Indexing::FieldType::Scalar.prepend(Indexing::FieldType::ScalarExtension) unless ElasticGraph::SchemaDefinition::Indexing::FieldType::Scalar < Indexing::FieldType::ScalarExtension
-          ElasticGraph::SchemaDefinition::Indexing::FieldType::Union.prepend(Indexing::FieldType::UnionExtension) unless ElasticGraph::SchemaDefinition::Indexing::FieldType::Union < Indexing::FieldType::UnionExtension
-
           state = api.state.extend(StateExtension) # : ElasticGraph::SchemaDefinition::State & StateExtension
           state.reserved_type_names << EVENT_ENVELOPE_JSON_SCHEMA_NAME
           api.factory.extend FactoryExtension

elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/factory_extension.rb

@@ -14,6 +14,7 @@
 require "elastic_graph/json_ingestion/schema_definition/schema_elements/object_interface_extension"
 require "elastic_graph/json_ingestion/schema_definition/schema_elements/scalar_type_extension"
 require "elastic_graph/json_ingestion/schema_definition/schema_elements/type_reference_extension"
+require "elastic_graph/json_ingestion/schema_definition/schema_elements/union_type_extension"
 
 module ElasticGraph
   module JSONIngestion
@@ -78,6 +79,14 @@ def new_type_reference(name)
           super(name).extend(SchemaElements::TypeReferenceExtension)
         end
 
+        # @private
+        def new_union_type(name)
+          super(name) do |type|
+            type.extend SchemaElements::UnionTypeExtension
+            yield type if block_given?
+          end
+        end
+
         # Creates a new Results instance with JSON Schema extensions.
         #
         # @return [ElasticGraph::SchemaDefinition::Results] the created results instance

elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/indexing/field_type/enum.rb

@@ -0,0 +1,31 @@
+# Copyright 2024 - 2026 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/enum_extension"
+require "elastic_graph/schema_definition/indexing/field_type/enum"
+
+module ElasticGraph
+  module JSONIngestion
+    module SchemaDefinition
+      module Indexing
+        module FieldType
+          # JSON-ingestion indexing field type for enums.
+          #
+          # @private
+          class Enum < ::ElasticGraph::SchemaDefinition::Indexing::FieldType::Enum
+            include EnumExtension
+
+            def self.wrap(field_type)
+              new(field_type.enum_value_names)
+            end
+          end
+        end
+      end
+    end
+  end
+end

elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/indexing/field_type/object.rb

@@ -0,0 +1,37 @@
+# Copyright 2024 - 2026 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/object_extension"
+require "elastic_graph/schema_definition/indexing/field_type/object"
+
+module ElasticGraph
+  module JSONIngestion
+    module SchemaDefinition
+      module Indexing
+        module FieldType
+          # JSON-ingestion indexing field type for objects.
+          #
+          # @private
+          class Object < ::ElasticGraph::SchemaDefinition::Indexing::FieldType::Object
+            include ObjectExtension
+
+            def self.wrap(field_type, json_schema_options:)
+              new(
+                schema_def_state: field_type.schema_def_state,
+                type_name: field_type.type_name,
+                subfields: field_type.subfields,
+                mapping_options: field_type.mapping_options,
+                doc_comment: field_type.doc_comment
+              ).with_json_schema_options(json_schema_options)
+            end
+          end
+        end
+      end
+    end
+  end
+end

elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/indexing/field_type/scalar.rb

@@ -0,0 +1,31 @@
+# Copyright 2024 - 2026 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/scalar_extension"
+require "elastic_graph/schema_definition/indexing/field_type/scalar"
+
+module ElasticGraph
+  module JSONIngestion
+    module SchemaDefinition
+      module Indexing
+        module FieldType
+          # JSON-ingestion indexing field type for scalars.
+          #
+          # @private
+          class Scalar < ::ElasticGraph::SchemaDefinition::Indexing::FieldType::Scalar
+            include ScalarExtension
+
+            def self.wrap(field_type)
+              new(scalar_type: field_type.scalar_type)
+            end
+          end
+        end
+      end
+    end
+  end
+end

elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/indexing/field_type/union.rb

@@ -0,0 +1,31 @@
+# Copyright 2024 - 2026 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+require "elastic_graph/json_ingestion/schema_definition/indexing/field_type/union_extension"
+require "elastic_graph/schema_definition/indexing/field_type/union"
+
+module ElasticGraph
+  module JSONIngestion
+    module SchemaDefinition
+      module Indexing
+        module FieldType
+          # JSON-ingestion indexing field type for unions.
+          #
+          # @private
+          class Union < ::ElasticGraph::SchemaDefinition::Indexing::FieldType::Union
+            include UnionExtension
+
+            def self.wrap(field_type)
+              new(field_type.subtypes_by_name)
+            end
+          end
+        end
+      end
+    end
+  end
+end