Composition rules

Everything in this package reduces to NestedThreading.with_thread_budget. The rules below are what make it safe to use from library code that does not know what its caller is doing.

Budgets are refcounted, process-wide

The registry keeps a multiset of every budget scope currently active anywhere in the process.

  • On the transition from no scope to one scope, the current thread counts are snapshotted.
  • While any scope is open, the applied count is minimum over all active budgets.
  • On the transition back to no scope, the snapshot is restored.

Nesting only ever narrows

with_thread_budget(1) do
    with_thread_budget(8) do
        BLAS.get_num_threads()   # == 1
    end
end

An inner scope asking for more threads cannot widen a restriction an outer caller is relying on, because the applied value is the minimum. This matters for operator algebra: a batched operator whose body contains another batched operator, or an NFFT operator constructed with threaded = true that ends up being called from inside a saturated loop, both compose correctly without either site knowing about the other.

Concurrent scopes cannot corrupt each other

This is the failure mode the refcounting exists to prevent. With per-scope save/restore — even with a lock around each half — two tasks interleave:

task Atask BBLAS
1saves 8, applies 11
2saves 1, applies 11
3restores 88
4restores 11

and the process is left permanently throttled to 1 with no scope active. A mutex does not fix this: it is the pairing of save and restore that interleaves, not the individual mutations. Snapshotting once and restoring once removes the failure entirely.

The cost: conservatism across unrelated tasks

BLAS and FFTW thread counts are process-global; the C libraries have no per-task scoping and nothing in Julia can give them any. Taking the minimum over all active scopes therefore means:

While any task is restricted, unrelated concurrent tasks are restricted too.

A long-lived restricted scope throttles the whole process for as long as it is open. This is deliberate — the alternative direction (letting the widest request win) oversubscribes, which is the problem this package exists to solve — but it is worth knowing before opening a budget scope around a long-running operation.

All-or-nothing libraries

Polyester and NFFT are switched off whenever the applied budget is below NestedThreading.capacity, not only when it is 1: a budget of 4 inside an 8-thread outer loop still means the machine is fully occupied, and a nested Polyester loop spawning its own workers on top of that is exactly the oversubscription being avoided. NFFT has no partial control at all; Polyester does, and it is deliberately not used — see the measurements below.

The one exception is a loop that is the Polyester consumer. @budgeted_batch passes exclude = (:polyester,) so that the @batch loop being budgeted is not disabled by its own restriction.

Measure in the regime you deploy in

Whether budgeting helps depends almost entirely on how Threads.nthreads() compares to the machine's core count, and the difference is not a small factor. Both benchmarks from test/runtests.jl :benchmark, run on the same 48-core machine at two thread counts:

benchmarknthreads()unbudgetedbudgeted
nt×GEMM(384)80.0280 s0.0190 s1.5x
nt×GEMM(384)4885.998 s0.102 s846x
nt×(@batch over 16384)80.00256 s0.00405 s0.63x — loses
nt×(@batch over 16384)481.103 s0.0135 s82x

At nthreads() well below the core count there is spare hardware, nesting is free, and restricting only wastes threads — the Polyester row actually comes out negative there. At nthreads() near the core count the unbudgeted versions collapse: 86 seconds for work that takes a tenth of a second. Benchmarks run in the first regime will tell you this package is useless, and that conclusion does not transfer.

BenchHelpers.regime_warning in the test suite flags the mismatch when it sees one.

Why guarded pools are all-or-nothing

PolyesterWeave can reserve some worker slots, so an obvious refinement is to reserve only the capacity() ÷ budget the outer loop is using and leave the rest for nested loops. That was implemented and measured: 2.7x better at nthreads() = 8 with 2 outer iterations, and 320x worse at nthreads() = 48 with 2 outer iterations. At high thread counts a nested parallel loop is catastrophic however few workers it gets, so leaving any free reopens exactly the pathology being closed. All-or-nothing costs a bounded factor in the under-subscribed regime and avoids an unbounded one in the saturated regime, so that is what the Polyester guard does.

Budget scopes themselves are free

The benchmark suite asserts this rather than claiming it. Against a plain Threads.@threads loop, three controls all come out at 0.99-1.00x: a do-nothing closure wrapper, a budget scope with the :polyester pool excluded, and the Polyester guard wrapped around a loop with no @batch in it at all. Whatever budgeting costs or saves, none of it is overhead from this package.

Numbers from julia --project=test test/runtests.jl :benchmark; rerun them on your own machine and thread count. The benchmarks use BenchmarkTools with paired measurements — baseline and variant back-to-back within each round, median of the per-round ratios — because on a dual-socket machine the same loop is bimodal depending on how threads land across sockets, by far more than the effects being measured. Paired, the control numbers reproduce to within 1% run to run.

Guarded pools are decided at entry, counted pools are not

The two kinds of pool track the process-wide minimum differently, and the difference is observable.

A counted pool is a global that the registry writes on every entry and exit, so it always reflects the current minimum: if another task opens a narrower scope while yours is running, your BLAS and FFTW counts drop with it. A NestedThreading.GuardedPool has no global to write — the guard is a lexical wrapper around f(), and f() is already running. Whether a guard was entered is therefore decided once, from the budget at entry, and cannot be revised.

task Atask BBLAS (counted)Polyester (guarded)
1opens a full-capacity scopecapacity()enabled
2opens a scope at 11disabled for B only
3body runsbody runs1A's nested @batch still spawns workers

So a scope that opens while nothing is restricted runs its body unguarded even if the process is restricted underneath it a moment later. Note what is not affected: the counted pools are correct throughout (row 3, BLAS is at 1 for both tasks), and the opposite direction is safe — a scope that entered restricted stays guarded for its whole body even after the other scope exits, which is conservative rather than oversubscribing.

The window only exists for a scope that opened unrestricted, and only for guarded libraries — today that is Polyester alone. It is not closable: disable_polyester_threads reserves worker slots around a call, and there is no way for another task to enter that reservation around a call already in flight. If it matters for your workload, open the restricting scope before spawning the tasks rather than from inside them, which is what the loop macros do — they wrap the loop, not the loop body.

A full budget raises, it does not merely permit

The applied budget is written to every counted pool unconditionally, so a scope that works out to NestedThreading.capacity — with_full_threads, or a loop with a single item — sets the pools up to that value, past a lower count the caller may have configured by hand:

BLAS.set_num_threads(2)
@budgeted_threads for i in 1:1        # budget == capacity()
    BLAS.get_num_threads()            # == capacity(), not 2
end
BLAS.get_num_threads()                # == 2 again

This is deliberate: it is what lets with_full_threads turn a default-off library such as NFFT on. The snapshot/restore applies as usual, so the hand-set value comes back when the scope closes — but a hand-tuned count is not honoured inside a full-throttle scope.

Hard limits, soft defaults and grants

A hard limit is the right tool for a batch loop, and the wrong one for a computation whose calls into a library are mostly too small to thread but a few of which are large. An iterative solver is the typical case: its vector updates want serial BLAS, its occasional SVD wants every thread. Under a hard limit of 1 the SVD runs serial too, since nothing nested can widen it.

So there are three kinds of scope, and a pool's count is worked out from those open that apply to it:

scopekindcombines with its own kind by
NestedThreading.with_thread_budgethard limitminimum
NestedThreading.with_thread_defaultsoft defaultminimum
NestedThreading.with_thread_grantgrantmaximum

The highest grant wins over every soft default, but never goes past the count the process runs the pool at with no scope open. Without a grant, the lowest soft default applies. Without either, the lowest hard limit applies. Whichever it is is then clamped to the lowest hard limit.

with_thread_default(1; only = (:blas,)) do
    BLAS.get_num_threads()                            # 1
    with_thread_grant(8; only = (:blas,)) do
        BLAS.get_num_threads()                        # 8
    end
    with_thread_budget(1) do                          # a saturated batch worker
        with_thread_grant(8; only = (:blas,)) do
            BLAS.get_num_threads()                    # 1: a grant never beats a limit
        end
    end
end

When a grant closes and a pool's count goes down, that pool's NestedThreading.ParkHooks run. The one this package registers shuts OpenBLAS's worker threads down, which would otherwise spin for about 0.1 s on the cores the caller wants next; see NestedThreading.park_openblas.

Worked example

With Threads.threadpoolsize() == 8:

looptrip countbudgetBLAS/FFTW insidePolyester/NFFT inside
@budgeted_threads for i in 1:3232max(1, 8 ÷ 32) = 11off
@budgeted_threads for i in 1:228 ÷ 2 = 44off
@budgeted_threads for i in 1:118 ÷ 1 = 88on
@budgeted_threads threads = false for i in 1:2—11off