Parametrized collections

[jaskp]

Hi, I'm creating this discussion based on this discord conversation.

This library is very powerful, but I discovered a suboptimal DX in a fairly simple case, which is an API that requires you to provide some arguments (i.e. specify what IDs to fetch). I've constructed an example that should demonstrate it:

type ExternalAccountId = string & { __brand: "ExternalAccountId" };

interface Account {
  externalId: ExternalAccountId;
}
interface AccountExternalData {
  accountId: ExternalAccountId;
  data: object;
}

declare function fetchAccounts(): Promise<Account[]>;
declare function fetchAccountsExternalData(
  accountIds: ExternalAccountId[]
): Promise<AccountExternalData[]>;

const accountCollection = createCollection(
  queryCollectionOptions({
    queryKey: ["accounts"],
    queryFn: () => fetchAccounts(),
    getKey: (data) => data.externalId,
    queryClient,
  })
);

const accountExternalDataCollection = createCollection(
  queryCollectionOptions({
    syncMode: "on-demand",
    queryKey: ["accounts", "externalData"],
    queryFn: ({ meta }) => {
      const { filters } = parseLoadSubsetOptions(meta?.loadSubsetOptions);
      // We need to figure out which account ids to fetch external data for,
      // because the API requires a list of account ids.
      const accountIdFilter = filters.find(
        ({ operator, field }) =>
          operator === "in" && field.length === 1 && field[0] === "externlId"
      );
      // Fingers crossed teammates don't use this query without the filter.
      if (!accountIdFilter) throw Error("Must provide a filter for externalId");
      // We have to cast to ExternalAccountId[],
      // because LoadSubsetOptions has no way of knowing the type.
      const accountIds = accountIdFilter.value as ExternalAccountId[];

      return fetchAccountsExternalData(accountIds);
      // Whoops, nobody caught the typo in "externlId"!
    },
    getKey: (data) => data.accountId,
    queryClient,
  })
);

createCollection(
  liveQueryCollectionOptions({
    query: (q) =>
      q
        .from({ account: accountCollection })
        .join(
          { data: accountExternalDataCollection },
          ({ account, data }) => eq(account.externalId, data.accountId),
          "left"
        ),
    startSync: true,
  })
);

As can be seen, the issues are:

• no way to enforce that queries using accountExternalDataCollection are fetched with an array of externalIds
  i.e. this is invalid: useLiveQuery(q => q.from({data: accountExternalDataCollection}))
• loss of type information by using loadSubsetOptions, demonstrated by the typo in "externlId"

I'm wondering if there's a way this use-case can be satisfied. The tricky part is that arbitrary parametrization like this (imagine API) has a problem:

const accountExternalDataCollection = createCollection(
  queryCollectionOptions({
    syncMode: "on-demand",
    queryKey: ["accounts", "externalData"],
    // Imaginary API
    queryFn: (_, args: ExternalAccountId[]) => {
      return fetchAccountsExternalData(args);
    },
    getKey: (data) => data.accountId,
    queryClient,
  })
);

createCollection(
  liveQueryCollectionOptions({
    query: (q) =>
      q.from({
        data: accountExternalDataCollection,
        // Imaginary API
        args: q
          .from({ account: accountCollection })
          .select(({ account }) => account.externalId),
      }),
    startSync: true,
  })
);

The problem being that changing the parameter would likely have to delete all rows from the collection, because there is no clear relationship between the inputs and outputs, so client-side filtering can't work.

@samwillis seemed to be wiling to give this a thought and had some ideas with custom operators, so I'm hoping we can keep this discussion going.

Any ideas welcome, thanks!