docs update (#319)

* update readme and spec

* add custom xo docs

* update

Author: danicc097

README.md

@@ -124,11 +124,69 @@ bin/deploy-instrumentation
 
 ## Notes on code generation
 
-Docs WIP:
+### `xo`
+
+Db struct mappings can be extended with SQL column comments, joined with ` && `:
+- `properties`:`<p1>,<p2>,...`
+  - `private`: exclude a field from JSON.
+  - `not-required`: make a schema field not required.
+  - `hidden`: exclude field from OpenAPI generation.
+  - `refs-ignore`: generate a field whose constraints are ignored by the referenced table,
+  i.e. no joins will be generated.
+  - `share-ref-constraints`: for a FK column, it will generate the same M2O and M2M join fields the ref column has.
+- `type`: override the type annotation with an existing spec schema. Also allows
+  complex JSON columns to be encoded and decoded thanks to `pgx`.
+- `cardinality`:`<O2O|M2O|M2M>` to generate/override joins explicitly. Only O2O is inferred.
+- `tags`: append literal struct tag strings.
+
+### `rest` package
+
+Structs in `models.spec.go` will be generated via `openapi-go` and replaced in
+the spec. Make sure they don't clash with existing `models` names or db tables.
+
+### `models` package
+
+Combines db codegen via a custom `xo` alongside spec schemas codegen for ease of use.
+
+### CRUD + tests generation
+
+See `project test.create-crud-gen`'s implementation for an example workflow to create a
+variety of implemented CRUD endpoints and relevant tests for a given database
+table.
+This is a one-off script per table.
+
+### OpenAPI spec
+
+#### Paths
+
+Endpoints are exclusively managed through the spec and created/kept up to date
+in their respective files aggregated by `tags` (limited to one)
+via codegen and AST modifications. This ensures the server always implements the
+spec.
+
+For endpoint implementation, only method bodies are to be modified.
+
+#### Vendor extensions
+
+Component schemas:
+  - `x-gen-struct` generates a schema from a struct and keeps it up to date,
+    which can be tagged with OpenAPI fields (via openapi-go).
+    Possible structs to generate from are listed in `internal/codegen/structs.gen.go`
+    indexed by `x-gen-struct`.
+    Packages to generate structs from are added adhoc (currently rest and models packages only)
+    Note that rest package structs are generated without prefix since they
+    are the base of the spec - else prefix is the pascal cased package name.
+    The value of `x-gen-struct` must match the schema name, to prevent duplicates and
+    confusing definitions.
+    To use a yet to be generated `rest` package struct in spec `paths`, simply use the `$ref`
+    without creating any schema and it will be autogenerated later if it doesn't exist
+    (only for `rest` package structs, since any other should not be used as request/response in the
+    first place).
+  - `x-spec-rest-type`: set to true to mark rest types (i.e. request, responses, query params...) that are manually defined in the spec instead of structs in `models.spec.go`.
+Path operations:
+  - `x-required-scopes` defines required scopes for a request.
+  - `x-required-role` defines the minimum required role for a request.
 
-- Backend generation pipeline
-- Frontend generation pipeline
-- External tooling summary and upgrades
 
 <!-- xo custom templates with cardinality, property comments for join and public model generation for embedding, schema from structs, spec sync -->
 
@@ -138,14 +196,6 @@ Simplified:
 
 ![](.github/system-diagram.png)
 
-## Changelog
-
-- `v0.2`: discard `OpenAPITools/openapi-generator` with custom postgeneration for `deepmap/oapi-codegen`. Any
-  change to fix broken generator functionality requires opening a PR or a disturbing
-  amount of postgeneration code. Templates getting out of hand and also require
-  a PR for custom functions. `oapi-codegen` much more extensible and idiomatic
-  being already Go and properly maintained.
-
 ## Known issues
 
 - Nested functions in `project`'s `x` functions will break automatic

bin/project

@@ -897,10 +897,6 @@ x.gen.client-server() {
     go-utils.find_all_types all_types "$REST_MODELS"
     rest_types_list=$(join_by "," ${all_types[*]})
 
-    # TODO should work for references that are not used in, e.g. Topics
-    mapfile -t server_types < <(yq e '.components.schemas[] | select(.x-use-server-type == true) | key' $SPEC_PATH)
-    server_types_list=$(join_by "," ${server_types[*]})
-
     mapfile -t spec_rest_types < <(yq e '.components.schemas[] | select(.x-spec-rest-type == true) | key' $SPEC_PATH)
     spec_rest_types_list=$(join_by "," ${spec_rest_types[*]})
 
@@ -972,7 +968,7 @@ EOF
     # TODO: we already have structs.gen.go indexed by "Models(models struct)" or plain rest type...
     # however this might be faster if the build is cached (templates unchanged). could construct internally as map set anyway
     oapi-codegen --config <(echo "$models_config") --types "$rest_types_list" "$SPEC_PATH" || err "Failed types generation" &
-    oapi-codegen --config <(echo "$server_config") --spec-rest-types "$spec_rest_types_list" --models-pkg models --server-types "$server_types_list" --types "$rest_types_list" "$paths_file" || err "Failed server generation" &
+    oapi-codegen --config <(echo "$server_config") --spec-rest-types "$spec_rest_types_list" --types "$rest_types_list" "$paths_file" || err "Failed server generation" &
     oapi-codegen --config <(echo "$test_client_config") --types "$rest_types_list" --spec-rest-types "$spec_rest_types_list" "$SPEC_PATH" || err "Failed client generation" &
     # not used now. may be useful for a cli at some point
     # oapi-codegen --config internal/client/oapi-codegen-client.yaml "$SPEC_PATH" || err "Failed client generation" &

frontend/src/client-validator/gen/dereferenced-schema.json

@@ -3735,7 +3735,6 @@
           "x-is-generated": true
         }
       ],
-      "x-use-server-type": true,
       "x-spec-rest-type": true
     },
     "Scopes": {
@@ -5866,8 +5865,7 @@
             "filterMode"
           ]
         }
-      ],
-      "x-use-server-type": true
+      ]
     },
     "Pagination": {
       "type": "object",
@@ -5952,8 +5950,7 @@
                 "filterMode"
               ]
             }
-          ],
-          "x-use-server-type": true
+          ]
         },
         "sort": {
           "type": "string",
@@ -5962,8 +5959,7 @@
             "desc"
           ]
         }
-      },
-      "x-use-server-type": true
+      }
     },
     "PaginationItems": {
       "type": "object",
@@ -6051,8 +6047,7 @@
                   "filterMode"
                 ]
               }
-            ],
-            "x-use-server-type": true
+            ]
           },
           "sort": {
             "type": "string",
@@ -6061,8 +6056,7 @@
               "desc"
             ]
           }
-        },
-        "x-use-server-type": true
+        }
       }
     },
     "PaginationCursor": {
@@ -6190,8 +6184,7 @@
                       "filterMode"
                     ]
                   }
-                ],
-                "x-use-server-type": true
+                ]
               },
               "sort": {
                 "type": "string",
@@ -6200,8 +6193,7 @@
                   "desc"
                 ]
               }
-            },
-            "x-use-server-type": true
+            }
           }
         }
       },
@@ -11130,7 +11122,6 @@
           "x-is-generated": true
         }
       ],
-      "x-use-server-type": true,
       "x-spec-rest-type": true
     },
     "Scopes": {
@@ -13261,8 +13252,7 @@
             "filterMode"
           ]
         }
-      ],
-      "x-use-server-type": true
+      ]
     },
     "Pagination": {
       "type": "object",
@@ -13347,8 +13337,7 @@
                 "filterMode"
               ]
             }
-          ],
-          "x-use-server-type": true
+          ]
         },
         "sort": {
           "type": "string",
@@ -13357,8 +13346,7 @@
             "desc"
           ]
         }
-      },
-      "x-use-server-type": true
+      }
     },
     "PaginationItems": {
       "type": "object",
@@ -13446,8 +13434,7 @@
                   "filterMode"
                 ]
               }
-            ],
-            "x-use-server-type": true
+            ]
           },
           "sort": {
             "type": "string",
@@ -13456,8 +13443,7 @@
               "desc"
             ]
           }
-        },
-        "x-use-server-type": true
+        }
       }
     },
     "PaginationCursor": {
@@ -13585,8 +13571,7 @@
                       "filterMode"
                     ]
                   }
-                ],
-                "x-use-server-type": true
+                ]
               },
               "sort": {
                 "type": "string",
@@ -13595,8 +13580,7 @@
                   "desc"
                 ]
               }
-            },
-            "x-use-server-type": true
+            }
           }
         }
       },

frontend/src/client-validator/gen/schema.json

@@ -1479,7 +1479,6 @@
           "$ref": "#/definitions/DemoTwoWorkItemResponse"
         }
       ],
-      "x-use-server-type": true,
       "x-spec-rest-type": true
     },
     "Scopes": {
@@ -2089,8 +2088,7 @@
         {
           "$ref": "#/definitions/PaginationFilterArray"
         }
-      ],
-      "x-use-server-type": true
+      ]
     },
     "Pagination": {
       "type": "object",
@@ -2101,8 +2099,7 @@
         "sort": {
           "$ref": "#/definitions/Direction"
         }
-      },
-      "x-use-server-type": true
+      }
     },
     "PaginationItems": {
       "type": "object",