Composition rules
Everything in this package reduces to NestedThreading.with_thread_budget. The rules below are what make it safe to use from library code that does not know what its caller is doing.
Budgets are refcounted, process-wide
The registry keeps a multiset of every budget scope currently active anywhere in the process.
- On the transition from no scope to one scope, the current thread counts are snapshotted.
- While any scope is open, the applied count is
minimumover all active budgets. - On the transition back to no scope, the snapshot is restored.
Nesting only ever narrows
with_thread_budget(1) do
with_thread_budget(8) do
BLAS.get_num_threads() # == 1
end
endAn inner scope asking for more threads cannot widen a restriction an outer caller is relying on, because the applied value is the minimum. This matters for operator algebra: a batched operator whose body contains another batched operator, or an NFFT operator constructed with threaded = true that ends up being called from inside a saturated loop, both compose correctly without either site knowing about the other.
Concurrent scopes cannot corrupt each other
This is the failure mode the refcounting exists to prevent. With per-scope save/restore — even with a lock around each half — two tasks interleave:
| task A | task B | BLAS | |
|---|---|---|---|
| 1 | saves 8, applies 1 | 1 | |
| 2 | saves 1, applies 1 | 1 | |
| 3 | restores 8 | 8 | |
| 4 | restores 1 | 1 |
and the process is left permanently throttled to 1 with no scope active. A mutex does not fix this: it is the pairing of save and restore that interleaves, not the individual mutations. Snapshotting once and restoring once removes the failure entirely.
The cost: conservatism across unrelated tasks
BLAS and FFTW thread counts are process-global; the C libraries have no per-task scoping and nothing in Julia can give them any. Taking the minimum over all active scopes therefore means:
All-or-nothing libraries
Polyester and NFFT are switched off whenever the applied budget is below NestedThreading.capacity, not only when it is 1: a budget of 4 inside an 8-thread outer loop still means the machine is fully occupied, and a nested Polyester loop spawning its own workers on top of that is exactly the oversubscription being avoided. NFFT has no partial control at all; Polyester does, and it is deliberately not used — see the measurements below.
The one exception is a loop that is the Polyester consumer. @budgeted_batch passes exclude = (:polyester,) so that the @batch loop being budgeted is not disabled by its own restriction.
Whether budgeting helps depends almost entirely on how Threads.nthreads() compares to the machine's core count, and the difference is not a small factor. Both benchmarks from test/runtests.jl :benchmark, run on the same 48-core machine at two thread counts:
| benchmark | nthreads() | unbudgeted | budgeted | |
|---|---|---|---|---|
nt×GEMM(384) | 8 | 0.0280 s | 0.0190 s | 1.5x |
nt×GEMM(384) | 48 | 85.998 s | 0.102 s | 846x |
nt×(@batch over 16384) | 8 | 0.00256 s | 0.00405 s | 0.63x — loses |
nt×(@batch over 16384) | 48 | 1.103 s | 0.0135 s | 82x |
At nthreads() well below the core count there is spare hardware, nesting is free, and restricting only wastes threads — the Polyester row actually comes out negative there. At nthreads() near the core count the unbudgeted versions collapse: 86 seconds for work that takes a tenth of a second. Benchmarks run in the first regime will tell you this package is useless, and that conclusion does not transfer.
BenchHelpers.regime_warning in the test suite flags the mismatch when it sees one.
PolyesterWeave can reserve some worker slots, so an obvious refinement is to reserve only the capacity() ÷ budget the outer loop is using and leave the rest for nested loops. That was implemented and measured: 2.7x better at nthreads() = 8 with 2 outer iterations, and 320x worse at nthreads() = 48 with 2 outer iterations. At high thread counts a nested parallel loop is catastrophic however few workers it gets, so leaving any free reopens exactly the pathology being closed. All-or-nothing costs a bounded factor in the under-subscribed regime and avoids an unbounded one in the saturated regime, so that is what the Polyester guard does.
The benchmark suite asserts this rather than claiming it. Against a plain Threads.@threads loop, three controls all come out at 0.99-1.00x: a do-nothing closure wrapper, a budget scope with the :polyester pool excluded, and the Polyester guard wrapped around a loop with no @batch in it at all. Whatever budgeting costs or saves, none of it is overhead from this package.
Numbers from julia --project=test test/runtests.jl :benchmark; rerun them on your own machine and thread count. The benchmarks use BenchmarkTools with paired measurements — baseline and variant back-to-back within each round, median of the per-round ratios — because on a dual-socket machine the same loop is bimodal depending on how threads land across sockets, by far more than the effects being measured. Paired, the control numbers reproduce to within 1% run to run.
Guarded pools are decided at entry, counted pools are not
The two kinds of pool track the process-wide minimum differently, and the difference is observable.
A counted pool is a global that the registry writes on every entry and exit, so it always reflects the current minimum: if another task opens a narrower scope while yours is running, your BLAS and FFTW counts drop with it. A NestedThreading.GuardedPool has no global to write — the guard is a lexical wrapper around f(), and f() is already running. Whether a guard was entered is therefore decided once, from the budget at entry, and cannot be revised.
| task A | task B | BLAS (counted) | Polyester (guarded) | |
|---|---|---|---|---|
| 1 | opens a full-capacity scope | capacity() | enabled | |
| 2 | opens a scope at 1 | 1 | disabled for B only | |
| 3 | body runs | body runs | 1 | A's nested @batch still spawns workers |
So a scope that opens while nothing is restricted runs its body unguarded even if the process is restricted underneath it a moment later. Note what is not affected: the counted pools are correct throughout (row 3, BLAS is at 1 for both tasks), and the opposite direction is safe — a scope that entered restricted stays guarded for its whole body even after the other scope exits, which is conservative rather than oversubscribing.
The window only exists for a scope that opened unrestricted, and only for guarded libraries — today that is Polyester alone. It is not closable: disable_polyester_threads reserves worker slots around a call, and there is no way for another task to enter that reservation around a call already in flight. If it matters for your workload, open the restricting scope before spawning the tasks rather than from inside them, which is what the loop macros do — they wrap the loop, not the loop body.
A full budget raises, it does not merely permit
The applied budget is written to every counted pool unconditionally, so a scope that works out to NestedThreading.capacity — with_full_threads, or a loop with a single item — sets the pools up to that value, past a lower count the caller may have configured by hand:
BLAS.set_num_threads(2)
@budgeted_threads for i in 1:1 # budget == capacity()
BLAS.get_num_threads() # == capacity(), not 2
end
BLAS.get_num_threads() # == 2 againThis is deliberate: it is what lets with_full_threads turn a default-off library such as NFFT on. The snapshot/restore applies as usual, so the hand-set value comes back when the scope closes — but a hand-tuned count is not honoured inside a full-throttle scope.
Hard limits, soft defaults and grants
A hard limit is the right tool for a batch loop, and the wrong one for a computation whose calls into a library are mostly too small to thread but a few of which are large. An iterative solver is the typical case: its vector updates want serial BLAS, its occasional SVD wants every thread. Under a hard limit of 1 the SVD runs serial too, since nothing nested can widen it.
So there are three kinds of scope, and a pool's count is worked out from those open that apply to it:
| scope | kind | combines with its own kind by |
|---|---|---|
NestedThreading.with_thread_budget | hard limit | minimum |
NestedThreading.with_thread_default | soft default | minimum |
NestedThreading.with_thread_grant | grant | maximum |
The highest grant wins over every soft default, but never goes past the count the process runs the pool at with no scope open. Without a grant, the lowest soft default applies. Without either, the lowest hard limit applies. Whichever it is is then clamped to the lowest hard limit.
with_thread_default(1; only = (:blas,)) do
BLAS.get_num_threads() # 1
with_thread_grant(8; only = (:blas,)) do
BLAS.get_num_threads() # 8
end
with_thread_budget(1) do # a saturated batch worker
with_thread_grant(8; only = (:blas,)) do
BLAS.get_num_threads() # 1: a grant never beats a limit
end
end
endWhen a grant closes and a pool's count goes down, that pool's NestedThreading.ParkHooks run. The one this package registers shuts OpenBLAS's worker threads down, which would otherwise spin for about 0.1 s on the cores the caller wants next; see NestedThreading.park_openblas.
Worked example
With Threads.threadpoolsize() == 8:
| loop | trip count | budget | BLAS/FFTW inside | Polyester/NFFT inside |
|---|---|---|---|---|
@budgeted_threads for i in 1:32 | 32 | max(1, 8 ÷ 32) = 1 | 1 | off |
@budgeted_threads for i in 1:2 | 2 | 8 ÷ 2 = 4 | 4 | off |
@budgeted_threads for i in 1:1 | 1 | 8 ÷ 1 = 8 | 8 | on |
@budgeted_threads threads = false for i in 1:2 | — | 1 | 1 | off |