memberof: replace O(N²) dedup and diff with hash-based O(N) equivalents

[fcami]

Summary

This branch replaces two O(N²) algorithms in the memberof LDB module with O(N) hash-based equivalents and normalizes DN comparison to use ldb_dn_get_casefold (fully case-insensitive), correcting a semantic inconsistency with RFC 4514. The changes target environments with groups containing thousands of users, where the quadratic cost of the original algorithms dominates wall-clock time during cache writes.

This is meant as an experimental / thought-provoking research item, not an urgent branch to merge. There are drawbacks to the approach and obviously the possibility of regressions especially with case-sensitiveness has to be kept in mind.

Changes

mbof_append_addop: dedup via hash table

The previous implementation scanned a singly-linked list on each insertion to detect duplicates, yielding O(M²) comparisons over M insertions and O(M) per append (pointer walk to tail). The replacement uses a hash_table_t keyed by ldb_dn_get_casefold(entry_dn) for O(1) expected-time duplicate detection, and an add_list_tail pointer for O(1) append. The hash table and list are kept in sync: every hash_enter follows a successful link-in, and every hash_lookup precedes it. Space cost is O(M) for the table. The dedup tables are allocated with a fixed initial size of 1024 buckets; a smaller initial size could be considered to lower the allocation cost for operations on small groups.

mbof_mod_process_membel: set-difference via hash set

During LDB_FLAG_MOD_REPLACE operations, the module computes which members were added and which were removed by diffing the old and new member lists. The previous implementation used a doubly-nested loop with in-place array compaction: for each of A added entries, a linear scan of up to R removed entries, with shift-left compaction on match — O(A·R) total, O(N²) when A ≈ R ≈ N. The replacement is a three-pass algorithm: (1) build a hash set from the removed array, O(R); (2) probe and delete for each added entry, compacting the added array in place, O(A); (3) filter the removed array against the residual set, O(R). Total: O(A + R). The hash set is sized to removed->num * 2 buckets.

mbof_memberof_compute: function extraction

The @MEMBEROF-REBUILD recompute path previously had DN resolution and transitive-closure computation inline in the LDB_REPLY_DONE case of mbof_rcmp_grp_callback. This has been extracted into a standalone function mbof_memberof_compute that returns LDB_ERR_* codes, with the caller handling ldb_module_done. The transitive-closure fixpoint algorithm (relaxation via DLIST_DEMOTE/DLIST_PROMOTE with re-promotion on new memberOf discovery) is unchanged.

Casefold normalization in the recompute path

All hash table keys in the recompute path (user_table, group_table, memberofs) have been changed from ldb_dn_get_linearized (case-preserving) to ldb_dn_get_casefold (canonical lowercase). Lookups in the member-resolution loop now parse stored DN values via ldb_dn_from_ldb_val and casefold them before probing, rather than using raw byte strings as keys. This ensures correct matching regardless of how the member DN was stored. The output path (mbof_rcmp_update) recovers linearized DN strings via ldb_dn_get_linearized(parent->dn) from the mbof_member pointers stored as hash values (hash_values replacing hash_keys). Separate update_user_list/update_group_list pointers decouple the update phase from the compute phase's use of group_list. NULL checks have been added to all 9 ldb_dn_get_casefold call sites and to the ldb_dn_get_linearized call in the output loop.

mbof_member_iter: error handling fix

A hash_enter failure in mbof_member_iter previously returned LDB_ERR_OPERATIONS_ERROR (integer value 1) as a bool, which evaluated to true and caused hash_iterate to continue — silently swallowing the error. This is now mem->status = MBOF_ITER_ERROR; return false;, matching the error pattern used elsewhere in the same function.

Semantic change

The dedup in mbof_append_addop previously used sss_linearized_dn_match, which performs strcasecmp on the full DN but a case-sensitive strncmp on the name= value portion. The replacement uses ldb_dn_get_casefold, which is fully case-insensitive. This matches the LDAP DN comparison semantics defined in RFC 4514. The original code contained a FIXME: check if this is right comment on this comparison.

Test plan

• test_sysdb_memberof_store_large_group: store a group with 10,000 user members; verify memberOf on every user and memberuid on the group
• test_sysdb_memberof_replace_large_group: replace membership on a 10,000-member group (delta of 2); verify memberOf removed from dropped user, added to new user, and preserved on retained users
• test_sysdb_memberof_store_multi_group: store 12 groups of 10,000 users each; verify memberOf counts and memberuid counts
• test_sysdb_memberof_many_groups_same_users: store 50 groups of 2,000 users each; verify cumulative memberOf scaling
• test_sysdb_memberof_diamond_dedup: diamond topology (parent→A,B→C→10 users); verify each user has exactly 4 memberOf with no duplicates
• test_sysdb_memberof_mixed_case_dn: store group with mixed-case member DNs, then replace with canonical-case DNs; verify memberOf is matched correctly across the casing change
• test_sysdb_memberof_mixed_case_cross_domain: 3 domains (FILES, example.com, linux.example.com) with 3 distinct casing patterns; verify memberOf across all 6 users
• test_sysdb_memberof_rebuild: 3-group nested chain; corrupt memberOf and memberuid via SSSD_UPGRADE_DB bypass; trigger @MEMBEROF-REBUILD; verify counts and DN values via ldb_msg_find_val
• test_sysdb_memberof_rebuild_diamond: diamond topology; corrupt all memberOf and memberuid; trigger @MEMBEROF-REBUILD; verify counts, DN values, and no-duplicate invariant on all users and groups
• test_sysdb_memberof_restore_group: store group with 30 members, re-store with 5 dropped + 5 added; verify dropped users lose memberOf, kept users retain it, added users gain it
• test_sysdb_memberof_mixed_case_rebuild: store group with mixed-case member DNs; corrupt memberOf; trigger @MEMBEROF-REBUILD; verify recompute path resolves mixed-case DNs correctly

[gemini-code-assist]

Code Review

This pull request optimizes the memberof LDB module by replacing O(N) linked-list traversals and O(N*M) nested-loop set differences with hash-table-based lookups and operations. It also improves correctness by using ldb_dn_get_casefold for case-insensitive DN matching. A comprehensive suite of performance and regression tests has been added to verify these changes. Feedback was provided regarding a missing return value check for a hash table insertion in the set-difference logic, which could lead to database inconsistencies upon failure.

[gemini-code-assist]

The return value of hash_enter is not checked here. If the hash table insertion fails (e.g., due to memory exhaustion), the subsequent set-difference logic will be incorrect, potentially leading to inconsistent membership data in the database. All other hash_enter calls in this file correctly check for success.

                hret = hash_enter(old_set, &hkey, &hval);
                if (hret != HASH_SUCCESS) {
                    talloc_free(old_set);
                    return LDB_ERR_OPERATIONS_ERROR;
                }

[fcami]

This is fixed in be3de32