Support for versioned resources

[MartinBernstorff]

Smile and HAPI-FHIR added support for the _includeHistory parameter to their bulk exporter in HAPI 8.6.0. Docs.

As far as I can tell from the docs:

• FHIRPath is typically agnostic to resource versions/history, except if you specify it as part of the canonical URL
• SQL on FHIR is likewise
• And so is Pathling

We have some use-cases where this could come in handy. Off the top of my head, for example "number of careplans closed by month".

Are there any plans for any of the abovementioned packages to support such a use-case and/or do they already support it somewhere I'm missing?

Perhaps somewhat related to #2519

FYI @jkiddo

[johngrimes]

Hi Martin,

Thanks for raising this. There are currently two approaches available in Pathling for working with temporal data:

Accessing meta element via FHIRPath

Pathling supports querying meta, meta.versionId and meta.lastUpdated, directly in view definitions. You can filter and group by these fields:

result = data.view(
  resource="CarePlan",
  select=[{
      "column": [
          {"path": "id", "name": "id"},
          {"path": "meta.lastUpdated", "name": "last_updated"},
          {"path": "status", "name": "status"},
      ]
  }],
)
# Use Spark to extract month from lastUpdated
from pyspark.sql.functions import date_format
result.withColumn("month", date_format("last_updated", "yyyy-MM"))

However, this only provides access to the current version of each resource—not historical versions.

Delta Lake time travel

If you load FHIR data into Delta tables incrementally (using merge operations), you can use Delta time travel to query previous states of the data. This allows you to compare snapshots and identify when status changes occurred:

# Get current state
current = spark.read.format("delta").load(path)

# Get state at a previous version
previous = spark.read.format("delta").option("versionAsOf", 0).load(path)

# Find resources that changed from active to completed
current.join(previous, "id").filter(
  (previous.status == "active") & (current.status == "completed")
)

To count care plans closed by month, you could:

1. Use meta.lastUpdated to determine when the status was set to "completed"
2. Group by month to count closures

Pathling (and SQL on FHIR generally) works with multiple technical versions of FHIR resources, but the source system needs to export all historical versions. Some implementations do this, e.g. _includeHistory in Smile CDR/HAPI FHIR bulk export.

Hope this helps!