Scale the cache

The page cache (revalidate) and the data cache (cacheData, cachedFetch) follow the same rules when traffic is high.

Concurrent requests share one run. While a page or a value for a key is rendering, every other request for that key waits for it instead of starting another render. Each request gets its own Response (or its own copy of the value), so one of them reading or changing its answer cannot reach the others. A failure is shared by the requests that were waiting and then forgotten: the next request renders again, nothing is cached. A request that leaves (the client hung up: its page request is answered 499; the signal you pass to cacheData or cachedFetch aborts) stops waiting at once, and the shared run goes on for the others. The run's own signal (event.signal in a loader of a cached page, the argument of the function you give cacheData) aborts only when every request that had a signal left, or when the run takes longer than its timeout, so pass it to fetch and to your driver.

The bounds protect the table that makes sharing possible. Past one of them, requests are not refused and not queued: they behave as they would without a cache, so the worst case is the load you had before the cache.

import type { HandlerOptions } from "@shadow-js/dusk/server";

export const cache: HandlerOptions["cache"] = {
  maxWaiters: 500,
  maxInFlight: 5_000,
  renderTimeoutMs: 10_000,
  backoffBaseMs: 2_000,
  backoffMaxMs: 120_000,
};

Stale-while-revalidate. A page past its revalidate seconds, within its cache.staleSeconds window, is served at once as x-dusk-cache: STALE while one background render refreshes it (concurrent requests find the refresh running and start no other). The same holds for a value of cacheData between revalidate and its expiry. The refresh is handed to the platform's waitUntil when the request has one (Workers, Vercel, Netlify: the instance stays alive until the refresh ends); elsewhere it runs on in the process, bounded by maxInFlight and the render timeout. If it fails, the stale copy keeps being served, the error is reported once to onError with the phase cache, and the next refresh of that key waits 1 s, then 2 s, 4 s, up to 60 s; a success forgets the key. A refresh that renders an error page (a 500, a cookie) counts as a failure for the wait; the cache reports nothing more for it. Once the stale window ends, requests wait for a fresh render, and a failure there reaches those requests as the error it is.

A page answers with the headers a shared cache in front needs to count the same way the server does:

A purge (revalidatePath, revalidateTag) records itself before it removes entries. A render that began before the purge does not store its page after it (the requests that waited for it still get it), and a cacheData run that began before a purge of one of its tags does not store its value.

Shared stores. The default page cache is a bounded memory store of the process. Behind several instances, give the handler a store they all reach. redisStore (@shadow-js/dusk/stores/redis) is a page cache store over a client of five commands, so it needs no Redis package: get, set with a millisecond expiry, del, sAdd, sRem, sMembers (and pExpire when you ask for tagTtlSeconds). Adapt the client you use in a few lines. The adapters below describe the documented call shapes of each client; check them against the version you run, because they were not run against a real server.

import { type RedisClient, redisStore } from "@shadow-js/dusk/stores/redis";

/** The ioredis methods the adapter calls; an `ioredis` instance has them. */
interface IoredisLike {
  get(key: string): Promise<string | null>;
  set(key: string, value: string): Promise<unknown>;
  set(key: string, value: string, mode: "PX", milliseconds: number): Promise<unknown>;
  del(key: string): Promise<number>;
  sadd(key: string, member: string): Promise<number>;
  srem(key: string, member: string): Promise<number>;
  smembers(key: string): Promise<string[]>;
  pexpire(key: string, milliseconds: number): Promise<number>;
}

export function ioredisClient(redis: IoredisLike): RedisClient {
  return {
    get: (key) => redis.get(key),
    set: (key, value, options) =>
      options?.pxExpire === undefined
        ? redis.set(key, value)
        : redis.set(key, value, "PX", options.pxExpire),
    del: (key) => redis.del(key),
    sAdd: (key, member) => redis.sadd(key, member),
    sRem: (key, member) => redis.srem(key, member),
    sMembers: (key) => redis.smembers(key),
    pExpire: (key, milliseconds) => redis.pexpire(key, milliseconds),
  };
}

export function pageCache(redis: IoredisLike) {
  return redisStore(ioredisClient(redis), { prefix: "myapp:isr:" });
}

import type { RedisClient } from "@shadow-js/dusk/stores/redis";

/** The node-redis methods the adapter calls; a `createClient()` client has them. */
interface NodeRedisLike {
  get(key: string): Promise<string | null>;
  set(key: string, value: string, options?: { PX: number }): Promise<unknown>;
  del(key: string): Promise<number>;
  sAdd(key: string, member: string): Promise<number>;
  sRem(key: string, member: string): Promise<number>;
  sMembers(key: string): Promise<string[]>;
  pExpire(key: string, milliseconds: number): Promise<unknown>;
}

export function nodeRedisClient(redis: NodeRedisLike): RedisClient {
  return {
    get: (key) => redis.get(key),
    set: (key, value, options) =>
      options?.pxExpire === undefined
        ? redis.set(key, value)
        : redis.set(key, value, { PX: options.pxExpire }),
    del: (key) => redis.del(key),
    sAdd: (key, member) => redis.sAdd(key, member),
    sRem: (key, member) => redis.sRem(key, member),
    sMembers: (key) => redis.sMembers(key),
    pExpire: (key, milliseconds) => redis.pExpire(key, milliseconds),
  };
}

import type { RedisClient } from "@shadow-js/dusk/stores/redis";

/** The Upstash REST client methods the adapter calls (`@upstash/redis`). */
interface UpstashLike {
  get(key: string): Promise<unknown>;
  set(key: string, value: string, options?: { px: number }): Promise<unknown>;
  del(key: string): Promise<number>;
  sadd(key: string, member: string): Promise<number>;
  srem(key: string, member: string): Promise<number>;
  smembers(key: string): Promise<string[]>;
  pexpire(key: string, milliseconds: number): Promise<unknown>;
}

export function upstashClient(redis: UpstashLike): RedisClient {
  return {
    // Upstash parses a JSON string it reads back; the store wants the string it wrote.
    get: async (key) => {
      const value = await redis.get(key);
      if (value === null || value === undefined) return null;
      return typeof value === "string" ? value : JSON.stringify(value);
    },
    set: (key, value, options) =>
      options?.pxExpire === undefined
        ? redis.set(key, value)
        : redis.set(key, value, { px: options.pxExpire }),
    del: (key) => redis.del(key),
    sAdd: (key, member) => redis.sadd(key, member),
    sRem: (key, member) => redis.srem(key, member),
    sMembers: (key) => redis.smembers(key),
    pExpire: (key, milliseconds) => redis.pexpire(key, milliseconds),
  };
}

Use the store with createHandler({ cache: { store } }). What it writes, for a prefix P (default dusk:isr:): P e:<key> is one string per page, {"v":1,"e":<entry>}, with a millisecond expiry at the end of the page's stale window, so Redis drops pages by itself; P t:<tag> is a set of the keys indexed under the tag. No command scans the keyspace: a purge reads one set. An entry written by another version, or not JSON, is a miss, never an error. An entry over maxEntryBytes (1 MiB) is refused with a RangeError (the cache reports it and serves the page). A tag index lives until its tag is purged or its members are found dead; with unbounded URLs (a query string nobody controls) set tagTtlSeconds to at least your longest stale window, and the index of an idle tag expires (each write renews it). A page that would outlive it, including a page that never goes stale, is then refused.

The other shared stores are the KV adapters over any KVStore (cache.kv, see above). A page key or tag too long for a KV key is stored under its digest.

What several instances get. The memory store keeps pages in each process; a shared store keeps them where every instance reads.