ci: restrict number of jobs during build stage

[timsaucer]

Which issue does this PR close?

Related to #1429 but we need to verify if it resolves the issue.

Rationale for this change

We are getting OOM and this restricts the number of rustc invocations.

What changes are included in this PR?

CI parameter change.

Are there any user-facing changes?

No

[kevinjqliu]

I was looking into fixing this and testing on my fork,

adding swap helped
https://github.com/kevinjqliu/datafusion-python/pull/4/files#diff-5c3fa597431eda03ac3339ae6bf7f05e1a50d6fc7333679ec38e21b337cb6721R244-R255

we can add it to give us more buffer

[kevinjqliu]

this should help, we can even do CARGO_BUILD_JOBS=1 and also add it to the "debug" mode below on L241

[kevinjqliu]

Lets see if scoping it to substrait will help here.

I think we might need to bite the bullet and do this:

[profile.release]
lto = "thin"
codegen-units = 4

or override the options using env var just for that one job.

but note that this affects the final artifact

[kevinjqliu]

Claude on why profile.release.package.substrait might not work:
"""
That's a reasonable idea but it won't help with your specific OOM. The problem is that with lto = true (fat LTO), all crates get merged into a single IR blob inside rustc for the final optimization pass — the per-package opt-level and codegen-units overrides apply during the initial compilation of each crate, but by the time the LTO link step runs, rustc is holding everything together regardless.
"""

The priority order that minimizes artifact impact:

• More swap — zero artifact impact, just survival insurance (this worked when i tested on fork)
• mold linker — zero artifact impact, just more memory-efficient linking (trying this out https://github.com/kevinjqliu/datafusion-python/actions/runs/23552615526) EDIT this failed
• Set release option (lto = "thin", codegen-units = 4)