Sequential Versus Concurrent Requests

SIMULATED RESULT

Outcomes by client concurrency limit

What this simulation found

Matching the client limit to the eight available server workers completed all 48 requests in a median 1.834 seconds with no server queue. Issuing all 48 at once produced the same median service drain time, but created a 40-request queue and a median two client timeouts. Sequential execution completed every request but took a median 11.518 seconds.

Median trial value across 100 shared seeded workloads
Strategy	Completed	Timed out	Client outcome time	Server drain time	Peak in flight	Peak server queue	P95 queue wait
SequentialIssue one request and wait for its outcome before issuing the next.	48 / 48	0	11.518 s	11.518 s	1	0	0 ms
Bounded concurrency: 4Keep at most four requests awaiting a client outcome.	48 / 48	0	3.121 s	3.121 s	4	0	0 ms
Bounded concurrency: 8Match the client limit to the eight available server workers.	48 / 48	0	1.834 s	1.834 s	8	0	0 ms
Unbounded concurrencyIssue all 48 requests at simulated time zero.	46 / 48	2	1.500 s	1.834 s	48	40	1.188 s

Client outcome time ends when every request has either completed or timed out. Server drain time includes work that continued after a client timeout. A shorter client outcome time can therefore include failed requests.

Fixed simulation scenario

Each trial generates 48 independent request durations from the published seeded distribution. A service has eight workers and a FIFO queue. Each client request times out 1500 milliseconds after it is issued, while timed-out server work continues to completion.

Requests: 48 per workload
Server capacity: 8 workers
Client timeout: 1,500 ms from issue
Queue: FIFO, with timed-out work continuing to completion
Common duration: 75% from 80 through 240 milliseconds
Slower duration: 20% from 241 through 500 milliseconds
Long-tail duration: 5% from 800 through 1200 milliseconds
Workloads: Integers 1 through 100, using Mulberry32

Scenario SHA-256: bd32c090f67d357d42fcbdf0e0d68a05c7b4a56647c9c2be5f874fc3a2b9388e

Methodology

Measure how the client concurrency limit changes successful completion, timeout outcomes, elapsed simulated time, server queue depth, queue wait, and work performed after a client timeout under one fixed-capacity model.

Trials: 100 shared workloads per strategy
Statistic: Median trial metric, with minimum and maximum retained
Event order: Ascending simulated millisecond, service completion before timeout, then insertion sequence
Server model: Eight workers with a FIFO queue; timed-out work is not canceled
Timeout: A request without a completed response 1500 milliseconds after client issue time

1
Generate one 48-request workload for each seed from 1 through 100. Every strategy receives the same workload for a given seed and request durations remain in the same order.
2
Issue requests until the selected client concurrency limit is full. A client slot becomes available when that request succeeds or reaches its client timeout.
3
Start work immediately when one of eight server workers is free. Place additional issued requests into a FIFO server queue.
4
Record a successful client outcome when service completes before or exactly at the timeout. Record a timeout otherwise, while allowing the server work to continue.
5
Report the median trial value for every metric and retain the minimum and maximum across the 100 shared workloads.

Strategy details and variation

Each strategy ran against the same 100 workloads. The cards show the observed minimum and maximum where the workload durations changed the outcome.

Client limit: 1

Sequential

Issue one request and wait for its outcome before issuing the next.

Outcome time range: 8.059 s to 14.639 s
Timeout range: 0 to 0
Maximum queue wait: 0 ms median
Work after timeout: 0 ms median

Client limit: 4

Bounded concurrency: 4

Keep at most four requests awaiting a client outcome.

Outcome time range: 2.204 s to 4.080 s
Timeout range: 0 to 0
Maximum queue wait: 0 ms median
Work after timeout: 0 ms median

Client limit: 8

Bounded concurrency: 8

Match the client limit to the eight available server workers.

Outcome time range: 1.231 s to 2.511 s
Timeout range: 0 to 0
Maximum queue wait: 0 ms median
Work after timeout: 0 ms median

Client limit: 48

Unbounded concurrency

Issue all 48 requests at simulated time zero.

Outcome time range: 1.231 s to 1.500 s
Timeout range: 0 to 14
Maximum queue wait: 1.227 s median
Work after timeout: 510 ms median

Recorded environment

This runner advances simulated time through queued events. The processor does not determine the simulated durations, but the execution environment is recorded for reproducibility.

Runtime: Node.js 18.20.8
Operating system: Linux 5.14.0-687.5.4.el9_8.x86_64, x86_64
Processor: AMD EPYC 7713 64-Core Processor, 4 logical CPUs available
Performed: 2026-07-13

Observations

Sequential execution completed all 48 requests in every trial, but its median client outcome time was 11.518 seconds because it used only one of eight available server workers.
A client limit of four completed every request in a median 3.121 seconds with no server queue or timeout. It traded some elapsed time for a lower in-flight ceiling.
A client limit of eight matched the modeled service capacity. It completed every request in a median 1.834 seconds and created no server queue.
Unbounded issue did not improve the median 1.834-second service drain time because the server still had only eight workers. It placed 40 requests in the queue immediately.
The unbounded strategy had a median two timeouts, a median 1.188-second 95th-percentile queue wait, and a median 510 milliseconds of server work performed after client timeouts.
The unbounded client outcome time stopped at a median 1.500 seconds because that is the timeout boundary. That shorter number does not mean the batch completed successfully or the server stopped working.

Limitations

This is a discrete event simulation, not a live network, language runtime, database, or provider measurement. Simulated milliseconds are not wall-clock benchmark timings.
The service has exactly eight identical workers and one FIFO queue. Real services can have connection pools, multiple queues, rate limits, resource contention, load shedding, and changing capacity.
Request durations are independent seeded samples from the published distribution. They do not model shared caches, correlated slowdowns, payload sizes, dependency fan-out, or priority classes.
Timed-out work continues in this model. Systems that reliably cancel queued or active work can recover capacity differently.
The client has no connection-pool or transport limit beyond the selected strategy. A runtime or proxy might silently bound an application that attempts unbounded concurrency.
This benchmark does not include retries. Adding retries to timed-out requests could increase load and change every outcome.
The best client limit is not universally equal to a server worker count. Production limits should account for measured latency, provider quotas, available connections, memory, and downstream behavior.
The ranges belong to seeds 1 through 100 and this Mulberry32 implementation. Other workloads and random generators can produce different results.

Reproduce the simulation

The public runner generates all 100 workloads, simulates every client strategy, and prints the scenario, environment, median metrics, and complete trial ranges as JSON. It makes no network requests.

node sequential-concurrent-requests.mjs

Open simulation runner

Runner SHA-2563b59b6a74c7a1c47a3b4d1e92a51a33c5ed4fa2f69a27f74442e1ed20658a3d4

Background reading

Handling Overload Google Site Reliability Engineering
Using load shedding to avoid overload Amazon Builders' Library