Async Generators for Data Streams in JavaScript

1. Why async generators are the right tool for data streams

An async generator is a function that does not return values all at once but produces them one after another, as soon as the consumer asks for the next value. The key difference from a regular async function that returns an array: the async generator does not need to load and hold all data completely in memory before the caller can start processing. For a paginated API with a hundred pages, that means processing can start after the first page, instead of waiting for all hundred responses.

This property makes async generators the natural tool for data streams of every kind: CSV exports with millions of rows, log files that keep growing, or database query results too large to fit fully into memory. Instead of a function that returns one huge array at the end, reserving the entire memory footprint upfront, an async generator delivers a continuous flow of values that the consumer works through at its own pace.

2. Syntax: async function* and yield in detail

An async generator is declared with the combination of async and function*, so async function* myGenerator() { }. Inside the function body, yield value produces the next value of the stream and pauses execution until the consumer requests the next value. Unlike a synchronous generator, an await can precede every yield, so asynchronous operations such as network requests or database queries fit seamlessly into the flow.

The return value of calling an async generator is neither a promise nor a direct value, but an AsyncGenerator object that also implements Symbol.asyncIterator. Every call to .next() on this object returns a promise that resolves to { value, done }. This structure is identical to that of a synchronous iterator, only every step happens asynchronously, which makes async generators compatible with any code that already understands the async iterator protocol.

// A minimal async generator that yields values over time
async function* countWithDelay(limit, delayMs) {
  for (let i = 1; i <= limit; i++) {
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    yield i; // pauses here until the consumer asks for the next value
  }
}

// Calling it does not execute the body yet — it returns an async iterator
const generator = countWithDelay(3, 500);
console.log(await generator.next()); // { value: 1, done: false }
console.log(await generator.next()); // { value: 2, done: false }

3. Consuming with for await...of

The idiomatic way to consume an async generator is the for await...of loop. It internally calls .next() repeatedly, waits for each promise, extracts the value, and automatically ends the loop as soon as done is true. To the code processing the data, this looks like an ordinary for...of loop over an array, even though every step runs asynchronously in the background and may only be triggered by a network request.

This symmetry with synchronous iteration is the real payoff of async generators over manual promise chaining or recursive then() calls. Error handling works with a normal try/catch around the loop, because a rejected promise inside the generator surfaces as a thrown exception exactly at the point in the loop where iteration currently stands.

async function processCountStream() {
  try {
    for await (const value of countWithDelay(5, 300)) {
      console.log('Received:', value);
    }
    console.log('Stream finished normally');
  } catch (error) {
    console.error('Stream failed:', error.message);
  }
}

4. Modeling paginated APIs as an async generator

The most common practical case for async generators is modeling paginated REST APIs. Instead of a function that loads all pages and returns a merged array at the end, an async generator delivers every loaded page, or even every individual item, as soon as it becomes available. The caller can start processing while the next page is already being fetched in the background, and does not have to wait until the API answers the last page.

Another advantage shows up with cancellation conditions: if the consumer has had enough after the first fifty items, say because a search match was already found, the loop can be exited with break without unnecessarily loading all remaining pages first. With a function that returns the complete array upfront, all requests would already have executed before the caller can even decide it needs no more.

// Async generator wrapping a paginated REST API
async function* fetchAllUsers(apiUrl) {
  let nextUrl = apiUrl;
  while (nextUrl) {
    const response = await fetch(nextUrl);
    const page = await response.json();
    for (const user of page.items) {
      yield user; // yield individual items, not whole pages
    }
    nextUrl = page.nextPageUrl ?? null;
  }
}

// Consumer stops early without loading remaining pages
for await (const user of fetchAllUsers('/api/users?limit=50')) {
  if (user.email.endsWith('@mironsoft.de')) {
    console.log('Found:', user);
    break; // remaining pages are never fetched
  }
}

5. Combining with ReadableStream and fetch body streaming

Async generators combine elegantly with the ReadableStream API that implements the body of a fetch response. The reader of a ReadableStream delivers chunks via .read(), which structurally matches the .next() call of an async generator. A thin wrapper function can turn a ReadableStream into an async generator, enabling the processing of large downloads, such as NDJSON or CSV responses, with the same for await...of syntax used for paginated APIs.

This unification is especially valuable when an application must process both paginated JSON APIs and streamed raw data. Both data sources can hide behind the same async generator interface, so the consuming code does not need to know whether the data comes from a paginated API or a continuous stream. That reduces the cognitive load for teams juggling multiple data sources with different transport mechanisms.

// Convert a fetch response body into lines via an async generator
async function* streamLines(response) {
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';

  try {
    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      buffer += decoder.decode(value, { stream: true });
      let newlineIndex;
      while ((newlineIndex = buffer.indexOf('\n')) >= 0) {
        yield buffer.slice(0, newlineIndex);
        buffer = buffer.slice(newlineIndex + 1);
      }
    }
    if (buffer) yield buffer; // trailing line without a newline
  } finally {
    reader.releaseLock();
  }
}

6. Backpressure and consumer driven iteration

Backpressure describes the problem of a fast producer flooding a slower consumer with data when no control mechanism is in place. Async generators solve this problem structurally, because the producer code only continues once the consumer actually calls .next(), either directly or implicitly via for await...of. There is no queue that grows without bound, because the generator genuinely pauses execution at every yield until demand exists.

That fundamentally distinguishes async generators from event emitter based streaming approaches, where data is emitted regardless of how fast the consumer processes it and, in the worst case, has to be buffered. Anyone pairing a slow database export with a fast consumer automatically benefits from this pull based control, without implementing custom buffering or throttling logic.

7. Early exit and cleanup with return() and throw()

When a for await...of loop over an async generator exits early via break, return, or a thrown exception, the runtime automatically calls .return() on the generator. That runs any open finally block inside the generator, which is essential for closing database cursors, file handles, or network connections. Without this mechanism, prematurely aborted iterations would leave resources open that nobody ever cleans up.

An explicit call to generator.return(value) is also possible manually, for example when a timeout should end a running iteration from the outside. Likewise, generator.throw(error) can be used to raise an exception at the generator's current pause point, allowing it to react in a controlled way inside its own try/catch block instead of terminating abruptly. These two methods make async generators full fledged, cooperatively cancellable resources.

async function* readDatabaseCursor(cursor) {
  try {
    while (await cursor.hasNext()) {
      yield await cursor.next();
    }
  } finally {
    // Runs on normal completion, early break, or thrown error
    await cursor.close();
    console.log('Cursor closed');
  }
}

for await (const row of readDatabaseCursor(cursor)) {
  if (row.id === targetId) break; // triggers cursor.close() via finally
}

8. Composition: building generator pipelines

Async generators can be chained into pipelines, similar to synchronous array methods like map and filter, only without every stage producing a complete intermediate list. A transformation function accepts an async generator, itself iterates over it with for await...of, and returns another async generator object delivering the transformed values. Every stage of the pipeline processes only one element at a time, keeping memory usage constant regardless of the total size of the data stream.

This pattern allows complex data processing to be assembled from small, reusable async generator functions: one stage filters, a second transforms, a third batches items into groups. Every stage remains independently testable, because it only accepts an input async generator and returns an output async generator, without needing knowledge of the other stages in the pipeline.

// Reusable async generator transformation stages
async function* mapAsync(source, transform) {
  for await (const item of source) {
    yield transform(item);
  }
}

async function* filterAsync(source, predicate) {
  for await (const item of source) {
    if (predicate(item)) yield item;
  }
}

// Compose a pipeline: fetch -> filter -> transform
const activeUserNames = mapAsync(
  filterAsync(fetchAllUsers('/api/users'), (u) => u.active),
  (u) => u.name.toUpperCase()
);

for await (const name of activeUserNames) {
  console.log(name);
}

9. Array collection vs async generator compared

The following table compares the classic method of collecting all results in an array against the approach using async generators. The difference becomes especially visible with large datasets and in cancellation scenarios.

The comparison shows that for small, fixed datasets, a plain array is often the more pragmatic choice. Once data volume, streaming behavior, or cancellability become relevant, the async generator is the more robust foundation.

10. Summary

Async generators solve the problem of processing large or continuous amounts of data without buffering them completely in memory. With async function* and yield, paginated APIs, streamed HTTP responses, and database cursors can all hide behind the same for await...of syntax, making the consuming code independent of the concrete data source.

Backpressure is structurally built into async generators, because the producer only continues once the consumer actually signals demand. return() and throw() ensure reliable cleanup on early exit, and the chainability into pipelines allows complex data processing to be assembled from small, independently testable building blocks, without intermediate results ever landing as complete arrays in memory.

11. FAQ: Async Generators for Data Streams