FM: design for diagnosis inputs/analysis preparation phase

[hawkw]

RFD 603 kind of punted on designing this but we need to figure out how we are going to handle determining what inputs need to be passed into the FM analysis code and how they are collected and accessed if we are going to actually perform FM analysis.

Some notes:

• The current inventory is an input, but that's one of the easier bits (I hope)
  • We probably need something that guards against doing FM analysis using an older inventory collection than the one in the parent sitrep?
• FM-driven observations of health endpoints/other states
  • Eventually the sitrep will have to be able to request that something goes and polls a health endpoint, in response to things like data loss ereports (and checking up on ongoing cases)
  • Requesting an observation is an output from one sitrep and the result of the observation is an input to a subsequent one
  • Querying Oximeter could probably fit in here unless it makes more sense for it not to?
  • We gotta figure out how to model this part eventually...
• Ereports
  • We need some way to be aware of which ereports are new since the current sitrep
    • This could just means tracking which ones aren't in cases, but that could become an expensive query
    • RFD 603 suggested doing this by tracking the last seen ENA for each reporter restart ID, and querying anything newer than that or with new restart IDs.
    • That could work but would require some finagling. In particular we want to be able to not keep around "old" restart IDs when a thing has restarted again, so that whatever tracks this doesn't grow unbounded.
    • Or, perhaps we could just track this in memory while we are loading a sitrep --- as we load the ereports, also track the latest ENA from each reporter. This way we are stuffing less state in the database forever...
  • But, we will eventually also want some way to be able to look back at historical ereports that we already saw but which are still in the db
    • The easiest way to do this is to "just query the database if you want to look at older ereports", but we would like to decouple the analysis logic from CRDB, and we would really like analysis to be a pure function of inputs to outputs...
    • @smklein suggested that perhaps we model querying historical ereports as a type of observation request, as discussed above.
      • This feels conceptually very tidy but means we have to figure out observation requests before we can query old ereports. Which is maybe fine...
    • This may not be needed immediately so we can probably figure it out after we figure out some of the other stuff

[hawkw]

• Or, perhaps we could just track this in memory while we are loading a sitrep --- as we load the ereports, also track the latest ENA from each reporter. This way we are stuffing less state in the database forever...

The more I think about it, I actually quite like this approach. I think we will also want to be constructing an (again, in memory) understanding of how each restart ID maps to the topology of reporters (i.e. for each sled, what is the latest SP and host OS restart ID?). I may start trying to put something about this together.

[hawkw]

I think that really, the more general thing I'm getting at here is that we are gonna end up building a bunch of in-memory indices anyway and we can probably use them to determine what ereports are new in a way that can be used in preparation.

[hawkw]

In order to do this, I think we may need to add a table mapping reporter restart IDs (which are unordered UUIDs) to a generation number for that slot, where a not-yet-seen UUID is inserted by incrementing the slot's counter. I think this should be fairly possible but may require an annoying ish CTE. Or, perhaps this can be implemented by a materialized view on the ereport table?

[hawkw]

To be a little bit more concrete about what I'm imagining, I would really like to be establishing an ordering of restart IDs on a per-reporter-location basis, so that we can query for "all ereports from the last sitrep's newest restart ID with ENAs greater than its last seen ENA, or with restart IDs greater than the newest restart for that thing in the last sitrep".

I am presently imagining doing this with something to the effect of:

1. Add a table like

   column	type
   type	enum of { sled_sp, sled_host, psc, switch }
   slot	like a u16 or whatever
   restart_id	you know this one's gonna be a UUID
   restart_generation	some form of integer
   part_number	Option<String> because sometimes the VPD EEPROM is sad
   serial_number	Option<String> as above
   time_first_seen	a timestamptz but also i feel complicated about including this because it could have Weird Implications and i'd rather pretend it doesn't exist

   where the (type, slot, restart_id) is probably the PK, and restart_generation is UNIQUE

2. have a CTE that takes a type, slot, restart_id, part_number, and serial_number, does something along the lines of:

   • if the inserted restart ID already exists:
     • if the current part_number or serial_number is NULL and the provide one is non-NULL:
       • then update that thing to the new one
     • else if the existing and provided ones are both non-NULL:
       • if they are equal, then do nothing lol
       • else freak out, lie down, try not to cry, cry a lot?
   • then update the PN and SN if they are null and the provided ones are non-null, or if they are not null do nothing
   • else: insert a new record into the table with the restart_generation set to the max(restart_generation for this type/slot) + 1 and the other fields set to whatever it was that we were given

3. make the code for inserting a tranche of ereports do that before writing the ereports (it should be fine for this to occur outside of a transaction, actually, as long as it happens first --- if we die before writing the ereports, that's actually fine because they will still get inserted by another nexus, and we will just make sure it's okay to have a restart ID generation entry in the table with no corresponding ereports yet (???)

4. figure out a query to backfill this table from existing ereport records which i am both 100% certain is possible and equally certain will be Kinda Annoying

[hawkw]

@cbiffle pointed out that we could alternatively just compute this index based on the timestamps of ereports that we already have in the ereport table¹, and, in fact, we do have an OMDB command that kinda sorta does that already, but i feel a bit uncomfortable about making The Timestamp That Nexus Thought It Was Upon Seeing An Ereport be the canonical source of ordering for restart IDs --- i feel a lot comfortabler doing it based on "was it already in the table when someone saw it", but maybe i'm being paranoid because i have timestamp trauma and should go to timestamp exposure therapy?

Footnotes

1. which is unfortunately what a backfilling query in a migration would have to do... ↩

[hawkw]

To be a little bit more concrete about what I'm imagining, I would really like to be establishing an ordering of restart IDs on a per-reporter-location basis, so that we can query for "all ereports from the last sitrep's newest restart ID with ENAs greater than its last seen ENA, or with restart IDs greater than the newest restart for that thing in the last sitrep".

I am presently imagining doing this with something to the effect of:

1. Add a table like
   
   
   
   **column**
   **type**
   
   
   
   
   `type`
   enum of { sled_sp, sled_host, psc, switch }
   
   
   `slot`
   like a u16 or whatever
   
   
   `restart_id`
   you know this one's gonna be a UUID
   
   
   `restart_generation`
   some form of integer
   
   
   `part_number`
   `Option<String>` because sometimes the VPD EEPROM is sad
   
   
   `serial_number`
   `Option<String>` as above
   
   
   `time_first_seen`
   a `timestamptz` but also i feel complicated about including this because it could have Weird Implications and i'd rather pretend it doesn't exist
   
   
   
   where the `(type, slot, restart_id)` is probably the PK, and `restart_generation` is `UNIQUE`

2. have a CTE that takes a `type`, `slot`, `restart_id`, `part_number`, and `serial_number`, does something along the lines of:
   
   * **if** the inserted restart ID already exists:
     
     * **if** the current `part_number` or `serial_number` **is** `NULL` **and** the provide one is non-`NULL`:
       
       * **then** update that thing to the new one
     * **else if** the existing and provided ones are both non-`NULL`:
       
       * **if** they are equal, **then** do nothing lol
       * **else** freak out, lie down, try not to cry, cry a lot?
   * **then** update the PN and SN if they are null and the provided ones are non-null, or if they are not null do nothing
   * **else**: insert a new record into the table with the `restart_generation` set to the `max(restart_generation for this type/slot) + 1` and the other fields set to whatever it was that we were given

3. make the code for inserting a tranche of ereports do that before writing the ereports (it should be fine for this to occur outside of a transaction, actually, as long as it happens first --- if we die before writing the ereports, that's actually fine because they will still get inserted by another nexus, and we will just make sure it's okay to have a restart ID generation entry in the table with no corresponding ereports yet (???)

4. figure out a query to backfill this table from existing ereport records which i am both 100% certain is possible and equally certain will be Kinda Annoying

@smklein has convinced me to not mess around with VPD stuff here, but I will otherwise go ahead and figure out a restart-ID-ordering table.

[smklein]

yup, just to confirm: I think the approach of tracking type/slot/restart_id/restart_generation makes total sense - and

where the (type, slot, restart_id) is probably the PK, and restart_generation is UNIQUE

all makes sense.

The big thing about dropping VPD info is that "we don't need it here" - it's stored in ereports, it may or may not be present, we still can create/use indexes on the ereports themselves to make that information accessible, etc.

From a database normalization point-of-view, it seems nice to avoid duplicating the info (e.g., risking it being inconsistent) where we can. Regardless, the "restart ID generation table" still can tell us which ereports to look for, hopefully without having any complicated CTE shenanigans to do back-filling of part/serial info.