When 1 Million Users Open App at 8AM: Designing a High‑Load Serverless App

/ Prime / By

When 1 Million Users Open Your App at 8AM: Designing a High‑Load Serverless App

Consumer apps live on a predictable heartbeat: morning peaks as commuters check news and messages, lunch dips, and promo-driven traffic spikes that can be 10–30x normal. If your backend isn’t designed for those surges, user experience craters exactly when your business needs it most. This guide explains how to plan for bursty traffic, architect for resilience and speed, and roll out changes safely—grounded in proven patterns and supported by authoritative references.

Why mornings (and promos) break apps

Spiky traffic is not an edge case; it’s a distribution reality. The Google SRE community emphasizes that reliability is measured at percentiles, not averages, and that overload protection is a first-class design concern for internet-scale services. See SRE guidance on SLIs/SLOs and overload handling for a deeper foundation (SLIs/SLOs, Handling Overload).

Promotions amplify these patterns. A push notification to millions can compress minutes of steady load into seconds of chaos. Without guardrails—rate limiting, queues, backpressure, caches, and replica-aware databases—the system cascades from elevated latency to timeouts, thundering herd effects, and data contention.

Set concrete performance targets

  • Availability SLO: 99.9% overall; 99.95% during declared peak windows.
  • Latency SLOs (steady state targets; peaks may be slightly higher but must remain stable):
    • Read APIs: p95 ≤ 300 ms, p99 ≤ 800 ms.
    • Write APIs: p95 ≤ 500–700 ms, p99 ≤ 1.2 s.
    • Background job enqueue: p95 ≤ 100 ms.
    • Queue time under surge: p99 ≤ 2 s; max acceptable queue time ≤ 30 s for non-interactive tasks.
  • Error budget policy: throttle or shed non-critical traffic to protect SLOs when budget burn accelerates.

Reference architecture for peak resilience

At a high level, a resilient, cost-efficient, serverless-first reference path looks like this:

  • Client → CDN (HTTP/3/QUIC, TLS termination, static/media edge cache)
  • CDN → WAF and rate limiter (per-IP, per-user, and global throttles)
  • WAF → API gateway (authN/Z, request shaping, canary routing)
  • API gateway → Stateless compute (Functions or containers), with circuit breakers
  • Compute ↔ Cache tier (in-memory, distributed)
  • Compute → Message queue (for burst smoothing and async work)
  • Compute → Database: primary for writes, replicas for reads; read/write splitting
  • Observability across all layers (OpenTelemetry traces, metrics, logs)

Key standards and resources: HTTP caching (RFC 9111), stale-while-revalidate and stale-if-error (RFC 5861), HTTP/3 (RFC 9114), QUIC transport (RFC 9000).

High-load systems

High-load systems must sustain high request rates and concurrency while preserving tail latencies. Design for elasticity first: stateless compute, idempotent operations, and horizontal scaling at every tier. Integrate overload protection—admission control, priority queues, and circuit breakers—so the system degrades gracefully under stress instead of failing catastrophically. Google’s SRE “Handling Overload” chapter outlines practical techniques such as load shedding and retry budgets that keep your most critical paths healthy during spikes.

Scalability

Scale horizontally, not just vertically. That means distributing reads to replicas, partitioning writes, and avoiding shared mutable state. Favor eventual consistency for non-critical counters and feeds, and design for idempotency to absorb client retries. Use rolling deployments with canaries to scale changes safely. When you must re-shard or re-index, do it under a traffic gate and behind queues to prevent write amplification during peak windows.

Caching

Cache hierarchies are your first line of defense against thundering herds:

  • Edge (CDN): Cache static assets and API GETs where legal. Use cache keys that include auth-related vary fields when needed, and enable stale-while-revalidate/stale-if-error to serve stable content during backend hiccups (RFC 5861). See primers from Cloudflare and others on CDN fundamentals (Cloudflare Learning).
  • Mid-tier/application cache: In-memory distributed caches (e.g., Redis, Memcached) for hot data. Prevent stampedes with request coalescing (single-flight) and jittered TTLs; use “serve stale” on cache-miss pressure. Monitor with a latency budget; Redis provides latency tooling (Redis latency monitor).
  • Database cache: Materialized views or read-optimized tables for expensive queries, refreshed asynchronously.

Rule of thumb: every cache layer should reduce origin traffic by at least 60–90% for eligible content during peaks, and cache-miss amplification must be bounded by coalescing.

Rate limiting

Rate limits keep the system stable and fair. Use token bucket or leaky bucket algorithms to allow short bursts but cap sustained rates. Enforce limits at the edge (cheapest) and again at the gateway for authenticated policies. On breach, return HTTP 429 Too Many Requests with a Retry-After header (RFC 6585). NGINX documents efficient request limiting modules (limit_req), and AWS covers throttling patterns for API Gateway (AWS docs).

Never forget the client side: apply exponential backoff with jitter to avoid synchronized retries during incidents (AWS Architecture Blog). For streaming or reactive systems, adopt backpressure-aware protocols (see Reactive Streams).

Capacity planning

Plan for surges by modeling arrival rates and concurrency explicitly. Start with baseline metrics—average RPS, 95th percentile peak RPS, payload sizes, and per-endpoint CPU/memory/IO cost—and simulate promo scenarios at 10–30x. Maintain headroom (e.g., 2–3x) on the hottest partitions and caches during peak windows.

  • Arrival model: If 1,000,000 users open the app within 10 minutes and 30% call a homepage API in the first 60 seconds, that’s 300,000 requests/60 ≈ 5,000 RPS to that endpoint, not counting retries and assets.
  • Concurrency: Concurrency ≈ arrival rate × average service time. At 5,000 RPS with p95 service time of 200 ms, you need ~1,000 concurrent workers for that endpoint alone.
  • Safety margins: Add a 20–30% buffer for cold starts, GC, and noisy neighbors in multi-tenant clouds.
  • Data tier: Provision read capacity across replicas to keep replica lag under 100–200 ms during bursts.

Refine these estimates with iterative tests and production telemetry before major promotions.

Load testing

Test like production, not a lab. Use a mix of traffic patterns:

  • Spike tests: 0 → peak in seconds to find bottlenecks and cold-start behavior.
  • Stress tests: Push until failure to validate overload protections and tail latencies.
  • Soak tests: Multi-hour runs at expected peaks to find memory leaks and slow drifts.
  • Chaos drills: Kill instances or inject latency to confirm graceful degradation.

Open-source tools like k6, Locust, and Gatling are ideal for scripting realistic user journeys. Instrument end-to-end with OpenTelemetry and propagate context using the W3C Trace Context standard to tie client, gateway, and backend spans together.

API gateway

The API gateway is the control plane for your edge. It should terminate TLS, authenticate tokens, apply per-tenant policies, shape requests (e.g., size limits), and direct traffic for canary releases. Keep the gateway stateless and highly available across zones/regions. Common capabilities include:

  • Policy enforcement: IP reputation, bot mitigation, and schema validation to fail fast.
  • Rate limits and quotas: Per-user/key, burst and sustained.
  • Resilience: Retries with jitter for idempotent reads, circuit breaking to downstreams, and request collapsing for identical cacheable GETs.
  • Routing: Canary and blue/green with fine-grained percentages.

Whether you use AWS API Gateway, Kong, or Envoy-based ingress, the patterns are consistent; see the AWS throttling docs for practical limits and defaults (AWS).

CDN

A globally distributed CDN absorbs static and semi-static load at the edge, accelerating your app via proximity and HTTP/3 while reducing origin traffic. CDNs like Cloudflare, Fastly, and Akamai provide PoPs in hundreds of cities worldwide (Cloudflare Network). For dynamic APIs, cache what’s safe: public GETs, feature flags, configuration, and personalized pages where headers and keys allow. Use cache-busting carefully—prefer content-addressed assets and immutable caching for static bundles, and set explicit TTLs for API responses with ETags.

Serverless

Serverless runtimes (functions, managed queues, event buses) scale rapidly with variable load and lower the cost of idle capacity. They’re excellent for bursty traffic when paired with good caching and queues. Two key considerations:

  • Concurrency and scaling: Understand cold starts and soft limits. AWS Lambda, for example, scales by increasing concurrent executions with regional concurrency quotas; provisioned concurrency can eliminate cold starts for critical paths (Lambda scaling, Lambda quotas).
  • Backpressure via queues: Decouple writes and heavy workloads with SQS or equivalent. Queue depth becomes your elastic buffer; keep processing latency within your UX budget, and autoscale consumers on lag. See SQS quotas and throughput guidance (AWS SQS).

Use event-driven patterns (e.g., outbox to queue, then function consumers) to protect the primary database during surge ingests.

Data tier: replicas and read/write splitting

Reads should go to replicas; writes should funnel to a primary. Postgres streaming replication and managed equivalents are proven building blocks (PostgreSQL docs; RDS Read Replicas). Design considerations:

  • Replica lag: Monitor and keep under 100–200 ms for interactive workloads. If lag spikes, degrade gracefully—serve slightly stale reads or pin read-after-write sessions to primary.
  • Read-your-writes: For critical flows (checkout, profile updates), use session consistency, version checks, or write-through caching.
  • Connection pooling: Pool at the app layer to avoid stampedes; serverless functions often need a managed pooler.

Queueing and backpressure

Queues are shock absorbers. Put them in front of non-critical writes (analytics events, email/SMS, image processing) and even critical but non-latency-bound tasks (payment reconciliation, feed fan-out). Backpressure signals should propagate upstream: when queue depth or processing lag exceeds thresholds, gateways and clients must slow down. This keeps the system inside stable operating limits rather than letting latency explode at the tail.

Rollout plan: load testing and canaries

Shipping a scaling change during peak hour without guardrails is a gamble. A safer plan:

  1. Stage with production-like data: Scrubbed datasets and realistic traffic scripts (payload sizes, think times, cache warm-ups).
  2. Benchmark baselines: Capture p50/p95/p99 latency, error rates, and resource saturation before changes.
  3. Introduce canaries: Route 1% of traffic to the new version; compare golden metrics to control using statistically significant windows (Google SRE: Canarying Releases).
  4. Scale to 5% → 25% → 50% → 100%: Hold at each step while monitoring.
  5. Game-day drills: During a controlled window, run a spike test and verify auto-scaling, queues, and limits behave as expected.

A brief, real-world pattern

Consider a consumer news app planning a promo blast at 7:55–8:05 AM. By fronting the app with a CDN that caches the homepage and article lists for 30–60 seconds, enabling stale-while-revalidate, and pushing personalized feeds behind an application cache, the team offloads 80–90% of read traffic from origin during the spike. An API gateway enforces per-user limits (e.g., 10 rps burst, 3 rps sustained) and a global concurrency cap. Writes (likes, save-for-later) go to a queue, then consumers update the primary database and invalidate cache keys. Postgres replicas handle reads, with a read-after-write pin to primary for 30 seconds on sessions that just updated data. The result: p95 latencies stayed under 300 ms at 8 AM despite a 20x surge, and the queue absorbed transient backlogs without users noticing.

Putting it all together for your stack

Before your next promo or product launch, align on a concrete plan:

  • Define SLOs and tail budgets for each critical endpoint.
  • Instrument tracing and metrics end-to-end.
  • Implement rate limits and backpressure at the edge and gateway.
  • Build a cache hierarchy (CDN → app cache → DB cache) with stampede mitigation.
  • Split reads/writes and provision replicas; monitor lag.
  • Adopt queues for surge absorption and asynchronous work.
  • Load test with spike, stress, and soak patterns; rehearse canary rollouts.

Work with a partner who has done this before

If you’re planning a major scale-up or expecting promo traffic, a focused assessment can de-risk your launch. Teyrex designs and builds high-load, secure web and mobile applications, from API gateways and CDNs to serverless compute and replica-aware data tiers. Explore our expertise in full‑stack development and Next.js, or contact us to schedule a scaling assessment.