Timestamp fingerprinting is slow for large glob patterns (e.g. **/*.gql)

[ItaiYosephi]

Description

When using method: timestamp with glob patterns like **/*.gql that match thousands of files across a large directory tree, the fingerprinting step is significantly slower than expected (~6.6s for ~5000 matching files in a monorepo).

What did you do?

Created a task with method: timestamp and source globs like:

sources:
  - /path/to/monorepo/schema/**/*.gql
  - /path/to/monorepo/app/src/**/*.gql
  - /path/to/monorepo/app/scripts/codegen/**/*
generates:
  - /tmp/output.txt

What did you expect to happen?

The up-to-date check should be fast since it only needs to compare file modification times.

What happened instead?

The check takes ~6.6s because:

1. Globs() uses execext.ExpandFields which shells out to expand glob patterns into a complete file list before any timestamp comparison. For **/*.gql patterns matching thousands of files, this is very expensive (~2.7s per call).

2. Value() is called redundantly: In compiledTask() (variables.go:184), the Value() call is not gated by evaluateShVars. Since RunTask calls both FastCompiledTask and CompiledTask sequentially (task.go:142 and task.go:159), Value() runs twice — adding ~2.7s of unnecessary overhead.

3. No short-circuiting in IsUpToDate: All globs are fully expanded into a file list before any timestamp comparison begins. For the common "up-to-date" case, this means stating every single matching file even though only one newer file would be enough to decide.

Version

Latest main (commit 54bdcba)

Operating system

macOS (Darwin 25.3.0), but the issue is platform-independent — it affects any large directory tree with ** glob patterns.

[trulede]

The succinct problem is, essentially (and because) files normally don't change in your example, that you want to reliably detect 1 changed file out of 5000 when often the are actually no changed files. This means, statistically, that most often you would have to scan a large portion of the files for a single change (50%), and if normally you don't expect files to change, actually you would have to scan all the files every time (100%).

To do that Task will have to glob all those files, and then compare the timestamps of each. The call to value() might be redundant, in some cases, and I believe there is a more generic strategy to deal with that - but its a longer term adjustment.

So, returning to the original (succinct) problem. You could possibly use a status command to check if any file has changed, that would prevent the unnecessary globing in most cases. Try a command like this, and see if it is significantly faster than Task. That would establish a useful benchmark.

find /your/directory -type f -name "*.txt" -newermt "2024-01-01"

With that benchmark you can determine if a status command would be of benefit.

[trulede]

@ItaiYosephi do you have any more feedback. Otherwise I would close the issue with a suggestion to use a status command to bypass the task mechanism, especially if you don't expect files to change.