In multithreaded scenarios we might encounter a situation when the consumer threads have to wait until there is a minimum number of jobs available in the job queue/pool they are working on.
Consider the case when a consumer thread is instructed to do the work after fetching at least m jobs from a job pool. A typical approach would be to
- Lock a mutex
- Check if there are at least m jobs in the pool.
- If so, fetch them and do the work.
- Else, sleep for sometime and check again.
- How long should we sleep?
- If we sleep too much, we'll do less work.
- If we don't sleep enough, we might waste CPU cycles by waking up too early.
Assuming we have a 2 staged queues, the consumer thread can wait on the stage1 if there is nothing to be popped from it like this:
The corresponding publisher can wake up the consumers like this:
With this approach, there can be multiple stages and there can be thread pools for each stage. These stages can be tuned perfectly to match the performance or throughput or latency criteria that we need to meet. For instance, in the above example, only 1 job was popped off by the consumer. If the time to process the job is less than the time to pop the job, then that stage can be tuned to shed out at least m jobs (in a list or so).
Check this code for more elaborate example.