Enable HOT updates for expression and partial indexes

Currently, PostgreSQL conservatively prevents HOT (Heap-Only Tuple)
updates whenever any indexed column changes, even if the indexed
portion of that column remains identical. This is overly restrictive
for expression indexes (where f(column) might not change even when
column changes) and partial indexes (where both old and new tuples
might fall outside the predicate).  Finally, index AMs play no role
in deciding when they need a new index entry on update, the rules
regarding that are based on binary equality and the HEAP's model for
MVCC and related HOT optimization.  Here we open that door a bit so
as to enable more nuanced control over the process.  This enables
index AMs that require binary equality (as is the case for nbtree)
to do that without disallowing type-specific equality checking for
other indexes.

This patch introduces several improvements to enable HOT updates in
these cases:

Add amcomparedatums() callback to IndexAmRoutine. This allows index
access methods like GIN to provide custom logic for comparing datums by
extracting and comparing index keys rather than comparing the raw
datums. GIN indexes now implement gincomparedatums() which extracts keys
from both datums and compares the resulting key sets.  Also, as
mentioned earlier nbtree implements this API and uses datumIsEqual() for
equality so that the manner in which it deduplicates TIDs on page split
doesn't have to change.  This is not a required API, when not
implemented the executor will compare TupleTableSlot datum for equality
using type-specific operators and take into account collation so that an
update from "Apple" to "APPLE" on a case insensitive index can now be
HOT.

ExecWhichIndexesRequireUpdates() is re-written to find the set of
modified indexed attributes that trigger new index tuples on updated.
For partial indexes, this checks whether both old and new tuples satisfy
or fail the predicate. For expression indexes, this uses type-specific
equality operators to compare computed values. For extraction-based
indexes (GIN/RUM) that implement amcomparedatums() it uses that.

Importantly, table access methods can still signal using TU_Update if
all, none, or only summarizing indexes should be updated.  While the
executor layer now owns determining what has changed due to an update
and is interested in only updating the minimum number of indexes
possible, the table AM can override that while performing
table_tuple_update(), which is what heap does.  While this signal is
very specific to how the heap implements MVCC and its HOT optimization,
we'll leave replacing that for another day.

This optimization trades off some new overhead for the potential for
more updates to use the HOT optimized path and avoid index and heap
bloat.  This should significantly improve update performance for tables
with expression indexes, partial indexes, and GIN/GiST indexes on
complex data types like JSONB and tsvector, while maintaining correct
index semantics.  Minimal additional overhead due to type-specific
equality checking should be washed out by the benefits of updating
indexes fewer times.

One notable trade-off is that there are more calls to FormIndexDatum()
as a result.  Caching these might reduce some of that overhead, but not
all.  This lead to the change in the frequency for expressions in the
spec update test to output notice messages, but does not impact
correctness.

Author: gburd

src/backend/access/brin/brin.c

@@ -290,6 +290,7 @@ brinhandler(PG_FUNCTION_ARGS)
 	amroutine->amproperty = NULL;
 	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = brinvalidate;
+	amroutine->amcomparedatums = NULL;
 	amroutine->amadjustmembers = NULL;
 	amroutine->ambeginscan = brinbeginscan;
 	amroutine->amrescan = brinrescan;

src/backend/access/gin/ginutil.c

@@ -26,6 +26,7 @@
 #include "storage/indexfsm.h"
 #include "utils/builtins.h"
 #include "utils/index_selfuncs.h"
+#include "utils/memutils.h"
 #include "utils/rel.h"
 #include "utils/typcache.h"
 
@@ -78,6 +79,7 @@ ginhandler(PG_FUNCTION_ARGS)
 	amroutine->amproperty = NULL;
 	amroutine->ambuildphasename = ginbuildphasename;
 	amroutine->amvalidate = ginvalidate;
+	amroutine->amcomparedatums = gincomparedatums;
 	amroutine->amadjustmembers = ginadjustmembers;
 	amroutine->ambeginscan = ginbeginscan;
 	amroutine->amrescan = ginrescan;
@@ -477,13 +479,6 @@ cmpEntries(const void *a, const void *b, void *arg)
 	return res;
 }
 
-
-/*
- * Extract the index key values from an indexable item
- *
- * The resulting key values are sorted, and any duplicates are removed.
- * This avoids generating redundant index entries.
- */
 Datum *
 ginExtractEntries(GinState *ginstate, OffsetNumber attnum,
 				  Datum value, bool isNull,
@@ -729,3 +724,86 @@ ginbuildphasename(int64 phasenum)
 			return NULL;
 	}
 }
+
+/*
+ * gincomparedatums - Compare datums to determine if they produce identical keys
+ *
+ * This function extracts keys from both old_datum and new_datum using the
+ * opclass's extractValue function, then compares the extracted key arrays.
+ * Returns true if the key sets are identical (same keys, same counts).
+ *
+ * This enables HOT updates for GIN indexes when the indexed portions of a
+ * value haven't changed, even if the value itself has changed.
+ *
+ * Example: JSONB column with GIN index. If an update changes a non-indexed
+ * key in the JSONB document, the extracted keys are identical and we can
+ * do a HOT update.
+ */
+bool
+gincomparedatums(Relation index, int attnum,
+				 Datum old_datum, bool old_isnull,
+				 Datum new_datum, bool new_isnull)
+{
+	GinState	ginstate;
+	Datum	   *old_keys;
+	Datum	   *new_keys;
+	GinNullCategory *old_categories;
+	GinNullCategory *new_categories;
+	int32		old_nkeys;
+	int32		new_nkeys;
+	MemoryContext tmpcontext;
+	MemoryContext oldcontext;
+	bool		result = true;
+
+	/* Handle NULL cases */
+	if (old_isnull != new_isnull)
+		return false;
+	if (old_isnull)
+		return true;
+
+	/* Create temporary context for extraction work */
+	tmpcontext = AllocSetContextCreate(CurrentMemoryContext,
+									   "GIN datum comparison",
+									   ALLOCSET_DEFAULT_SIZES);
+	oldcontext = MemoryContextSwitchTo(tmpcontext);
+
+	initGinState(&ginstate, index);
+
+	/* Extract keys from both datums using existing GIN infrastructure */
+	old_keys = ginExtractEntries(&ginstate, attnum, old_datum, old_isnull,
+								 &old_nkeys, &old_categories);
+	new_keys = ginExtractEntries(&ginstate, attnum, new_datum, new_isnull,
+								 &new_nkeys, &new_categories);
+
+	/* Different number of keys, definitely different */
+	if (old_nkeys != new_nkeys)
+	{
+		result = false;
+		goto cleanup;
+	}
+
+	/*
+	 * Compare the sorted key arrays element-by-element. Since both arrays are
+	 * already sorted by ginExtractEntries, we can do a simple O(n)
+	 * comparison.
+	 */
+	for (int i = 0; i < old_nkeys; i++)
+	{
+		int			cmp = ginCompareEntries(&ginstate, attnum,
+											old_keys[i], old_categories[i],
+											new_keys[i], new_categories[i]);
+
+		if (cmp != 0)
+		{
+			result = false;
+			break;
+		}
+	}
+
+cleanup:
+	/* Clean up */
+	MemoryContextSwitchTo(oldcontext);
+	MemoryContextDelete(tmpcontext);
+
+	return result;
+}

src/backend/access/hash/hash.c

@@ -50,6 +50,10 @@ static void hashbuildCallback(Relation index,
 							  void *state);
 
 
+static bool hashcomparedatums(Relation index, int attnum,
+							  Datum old_datum, bool old_isnull,
+							  Datum new_datum, bool new_isnull);
+
 /*
  * Hash handler function: return IndexAmRoutine with access method parameters
  * and callbacks.
@@ -98,6 +102,7 @@ hashhandler(PG_FUNCTION_ARGS)
 	amroutine->amproperty = NULL;
 	amroutine->ambuildphasename = NULL;
 	amroutine->amvalidate = hashvalidate;
+	amroutine->amcomparedatums = hashcomparedatums;
 	amroutine->amadjustmembers = hashadjustmembers;
 	amroutine->ambeginscan = hashbeginscan;
 	amroutine->amrescan = hashrescan;
@@ -944,3 +949,42 @@ hashtranslatecmptype(CompareType cmptype, Oid opfamily)
 		return HTEqualStrategyNumber;
 	return InvalidStrategy;
 }
+
+/*
+ * hashcomparedatums - Compare datums to determine if they produce identical keys
+ *
+ * Returns true if the hash values are identical (index doesn't need update).
+ */
+bool
+hashcomparedatums(Relation index, int attnum,
+				  Datum old_datum, bool old_isnull,
+				  Datum new_datum, bool new_isnull)
+{
+	uint32		old_hashkey;
+	uint32		new_hashkey;
+
+	/* If both are NULL, they're equal */
+	if (old_isnull && new_isnull)
+		return true;
+
+	/* If NULL status differs, they're not equal */
+	if (old_isnull != new_isnull)
+		return false;
+
+	/*
+	 * _hash_datum2hashkey() is used because we know this can't be a cross
+	 * type comparison.
+	 */
+	old_hashkey = _hash_datum2hashkey(index, old_datum);
+	new_hashkey = _hash_datum2hashkey(index, new_datum);
+
+	/*
+	 * If hash keys are identical, the index entry would be the same. Return
+	 * true to indicate no index update needed.
+	 *
+	 * Note: Hash collisions are rare but possible. If hash(x) == hash(y) but
+	 * x != y, the hash index still treats them identically, so we correctly
+	 * return true.
+	 */
+	return (old_hashkey == new_hashkey);
+}

src/backend/access/heap/heapam.c

@@ -3268,7 +3268,7 @@ heap_update(Relation relation, HeapTupleData *oldtup, HeapTuple newtup,
 			TM_FailureData *tmfd, LockTupleMode *lockmode,
 			Buffer buffer, Page page, BlockNumber block, ItemId lp,
 			Bitmapset *hot_attrs, Bitmapset *sum_attrs, Bitmapset *pk_attrs,
-			Bitmapset *rid_attrs, Bitmapset *mix_attrs, Buffer *vmbuffer,
+			Bitmapset *rid_attrs, const Bitmapset *mix_attrs, Buffer *vmbuffer,
 			bool rep_id_key_required, TU_UpdateIndexes *update_indexes)
 {
 	TM_Result	result;
@@ -4337,8 +4337,9 @@ HeapDetermineColumnsInfo(Relation relation,
  * This routine may be used to update a tuple when concurrent updates of the
  * target tuple are not expected (for example, because we have a lock on the
  * relation associated with the tuple).  Any failure is reported via ereport().
+ * Returns the set of modified indexed attributes.
  */
-void
+Bitmapset *
 simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tuple,
 				   TU_UpdateIndexes *update_indexes)
 {
@@ -4467,7 +4468,7 @@ simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tup
 
 		elog(ERROR, "tuple concurrently deleted");
 
-		return;
+		return NULL;
 	}
 
 	/*
@@ -4500,7 +4501,6 @@ simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tup
 	bms_free(sum_attrs);
 	bms_free(pk_attrs);
 	bms_free(rid_attrs);
-	bms_free(mix_attrs);
 	bms_free(idx_attrs);
 
 	switch (result)
@@ -4526,6 +4526,8 @@ simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tup
 			elog(ERROR, "unrecognized heap_update status: %u", result);
 			break;
 	}
+
+	return mix_attrs;
 }
 
 

src/backend/access/heap/heapam_handler.c

@@ -319,7 +319,7 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot,
 					Snapshot crosscheck, bool wait,
 					TM_FailureData *tmfd,
 					LockTupleMode *lockmode,
-					Bitmapset *mix_attrs,
+					const Bitmapset *mix_attrs,
 					TU_UpdateIndexes *update_indexes)
 {
 	bool		rep_id_key_required = false;
@@ -407,10 +407,6 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot,
 
 	Assert(ItemIdIsNormal(lp));
 
-	/*
-	 * Partially construct the oldtup for HeapDetermineColumnsInfo to work and
-	 * then pass that on to heap_update.
-	 */
 	oldtup.t_tableOid = RelationGetRelid(relation);
 	oldtup.t_data = (HeapTupleHeader) PageGetItem(page, lp);
 	oldtup.t_len = ItemIdGetLength(lp);

src/backend/access/nbtree/nbtree.c

@@ -155,6 +155,7 @@ bthandler(PG_FUNCTION_ARGS)
 	amroutine->amproperty = btproperty;
 	amroutine->ambuildphasename = btbuildphasename;
 	amroutine->amvalidate = btvalidate;
+	amroutine->amcomparedatums = NULL;
 	amroutine->amadjustmembers = btadjustmembers;
 	amroutine->ambeginscan = btbeginscan;
 	amroutine->amrescan = btrescan;

src/backend/access/table/tableam.c

@@ -336,7 +336,7 @@ void
 simple_table_tuple_update(Relation rel, ItemPointer otid,
 						  TupleTableSlot *slot,
 						  Snapshot snapshot,
-						  Bitmapset *modified_indexed_cols,
+						  const Bitmapset *mix_attrs,
 						  TU_UpdateIndexes *update_indexes)
 {
 	TM_Result	result;
@@ -348,7 +348,7 @@ simple_table_tuple_update(Relation rel, ItemPointer otid,
 								snapshot, InvalidSnapshot,
 								true /* wait for commit */ ,
 								&tmfd, &lockmode,
-								modified_indexed_cols,
+								mix_attrs,
 								update_indexes);
 
 	switch (result)

src/backend/bootstrap/bootstrap.c

@@ -961,10 +961,18 @@ index_register(Oid heap,
 	newind->il_info->ii_Expressions =
 		copyObject(indexInfo->ii_Expressions);
 	newind->il_info->ii_ExpressionsState = NIL;
+	/* expression attrs will likely be null, but may as well copy it */
+	newind->il_info->ii_ExpressionsAttrs =
+		copyObject(indexInfo->ii_ExpressionsAttrs);
 	/* predicate will likely be null, but may as well copy it */
 	newind->il_info->ii_Predicate =
 		copyObject(indexInfo->ii_Predicate);
 	newind->il_info->ii_PredicateState = NULL;
+	/* predicate attrs will likely be null, but may as well copy it */
+	newind->il_info->ii_PredicateAttrs =
+		copyObject(indexInfo->ii_PredicateAttrs);
+	newind->il_info->ii_CheckedPredicate = false;
+	newind->il_info->ii_PredicateSatisfied = false;
 	/* no exclusion constraints at bootstrap time, so no need to copy */
 	Assert(indexInfo->ii_ExclusionOps == NULL);
 	Assert(indexInfo->ii_ExclusionProcs == NULL);

src/backend/catalog/index.c

@@ -27,6 +27,7 @@
 #include "access/heapam.h"
 #include "access/multixact.h"
 #include "access/relscan.h"
+#include "access/sysattr.h"
 #include "access/tableam.h"
 #include "access/toast_compression.h"
 #include "access/transam.h"
@@ -58,6 +59,7 @@
 #include "commands/trigger.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
+#include "nodes/execnodes.h"
 #include "nodes/makefuncs.h"
 #include "nodes/nodeFuncs.h"
 #include "optimizer/optimizer.h"
@@ -2414,6 +2416,58 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode)
  * ----------------------------------------------------------------
  */
 
+/* ----------------
+ * BuildUpdateIndexInfo
+ *
+ * For expression indexes updates may not change the indexed value allowing
+ * for a HOT update.  Add information to the IndexInfo to allow for checking
+ * if the indexed value has changed.
+ *
+ * Do this processing here rather than in BuildIndexInfo() to not incur the
+ * overhead in the common non-expression cases.
+ * ----------------
+ */
+void
+BuildUpdateIndexInfo(ResultRelInfo *resultRelInfo)
+{
+	for (int j = 0; j < resultRelInfo->ri_NumIndices; j++)
+	{
+		int			i;
+		int			indnatts;
+		Bitmapset  *attrs = NULL;
+		IndexInfo  *ii = resultRelInfo->ri_IndexRelationInfo[j];
+
+		indnatts = ii->ii_NumIndexAttrs;
+
+		/* Collect key attributes used by the index, key and including */
+		for (i = 0; i < indnatts; i++)
+		{
+			AttrNumber	attnum = ii->ii_IndexAttrNumbers[i];
+
+			if (attnum != 0)
+				attrs = bms_add_member(attrs, attnum - FirstLowInvalidHeapAttributeNumber);
+		}
+
+		/* Collect attributes used in the expression */
+		if (ii->ii_Expressions)
+			pull_varattnos((Node *) ii->ii_Expressions,
+						   resultRelInfo->ri_RangeTableIndex,
+						   &ii->ii_ExpressionsAttrs);
+
+		/* Collect attributes used in the predicate */
+		if (ii->ii_Predicate)
+			pull_varattnos((Node *) ii->ii_Predicate,
+						   resultRelInfo->ri_RangeTableIndex,
+						   &ii->ii_PredicateAttrs);
+
+		/* Combine key, including, and expression attributes, but not predicate */
+		ii->ii_IndexedAttrs = bms_union(attrs, ii->ii_ExpressionsAttrs);
+
+		/* All indexes should index *something*! */
+		Assert(!bms_is_empty(ii->ii_IndexedAttrs));
+	}
+}
+
 /* ----------------
  *		BuildIndexInfo
  *			Construct an IndexInfo record for an open index