Concurrent Task Dispatcher

Overview

The program generates 1,000 tasks (70% IO-bound, 30% CPU-bound) and runs them through three concurrent parts:

• Dispatcher (main thread): pushes tasks into a shared queue every 20 ms
• Worker pool (8 threads): each worker takes a task from the queue and runs it
• Monitor thread: snapshots CPU usage, active workers and queue length every 10 ms to produce the final metrics

Shared state lives in one struct wrapped in Arc: the task queue (Mutex<VecDeque>), a CPU usage counter, an active-worker count (AtomicU32) and a shutdown flag (AtomicBool). Runs use a fixed random seed, so every run gets the same workload.

Results

I compared two scheduling policies on the same workload: 1,000 tasks, 8 workers, a new task every 20 ms, 200 ms per task and a 95% CPU cap.

• FIFO: workers take tasks in arrival order, with no idea of task type or CPU load. This is the baseline.
• CPU-aware: workers check CPU usage first. With headroom, they pick CPU tasks so those don't pile up; when the CPU is loaded, they pick cheaper IO tasks.

FIFO kept almost all 8 workers busy, but CPU usage averaged 139%: with no resource awareness, it overcommits the CPU. CPU-aware matched FIFO's throughput, but CPU tasks waited longer whenever the CPU was busy, so the worst-case wait went from about 4.9 to 12.8 seconds.

My takeaway: in this simulation, tasks just sleep, so overcommitting the CPU costs nothing and the smarter policy can't show a speedup. On real hardware, that overcommitting causes contention, and that's where CPU-aware scheduling would pay off. Exact timings vary a little from run to run, since they depend on how the OS schedules threads.

Bugs I fixed

• Lock held too long: an early version kept the queue locked while a task ran, so only one worker could run at a time. Releasing the lock before running the task fixed it.
• Shutting down too early: workers exited while tasks were still queued. Now they only stop once the shutdown flag is set and the queue is empty.
• Floating-point drift: thousands of small additions and subtractions pushed the CPU counter slightly below zero, which confused the CPU-aware logic. A one-line clamp fixed it.

Challenge

An AI assistant suggested a CPU-aware policy where the dispatcher held CPU tasks back in a buffer. It made the optimized run significantly slower, because workers sat idle while tasks were held. I moved the scheduling decision into the workers instead: each worker scans the queue and picks the best available task, which kept utilization high.

AI use

I used Claude as an assistant for syntax, explanations and best practices. I accepted its suggestion to use an atomic shutdown flag for clean termination, and rejected the dispatcher-side buffer for the reason above.

What's next

The CPU-aware policy can still starve CPU tasks during long bursts of IO work. Adding aging, where a task's priority rises the longer it waits, would put a ceiling on how long any task can wait.