REST API Performance: Payload Size, Projection, Streaming and Compression

1. The payload size problem: over-fetching and under-fetching

Over-fetching is the most common performance cause in REST APIs: an endpoint always returns every field of a resource, even when the client only needs three of them. A user object with 30 fields (including a large bio text field and a base64-encoded avatar) gets transferred in full even though a dropdown list only needs id, name, and email. With a hundred users in a list, the overhead multiplies accordingly. Under-fetching is the opposite problem: the client has to make multiple requests to get all the data it needs, because relationship data is never embedded.

Both problems have structural solutions: projection (field selection via query parameter) addresses over-fetching, while controlled embedding (embedding via parameter) addresses under-fetching. These mechanisms do not need to be implemented as fully as GraphQL field selection; even simple ?fields=id,name,email support reduces the payload in typical list queries by 70 to 90 percent. The implementation effort is small; the performance gain is measurable.

2. Implementing projection and sparse fieldsets

Sparse fieldsets following the JSON:API pattern use query parameters such as ?fields[products]=id,name,price for field selection. In a Symfony context this combines elegantly with serializer groups: instead of static groups, a dynamic group is built from the requested fields. The Symfony serializer supports passing serialization context per request, so field selection can be retrofitted without architectural changes.

Important: the allowed fields must be validated to prevent information disclosure. A client must not be able to use projection to access internal fields that are not part of the public API schema. An allowlist of valid fields per resource prevents this. Performance tip: if projection is implemented at the database layer (SELECT only the needed columns), you save not only network load but also database bandwidth and serialization time.

> $allowedFields Allowed fields per resource type
     */
    public function __construct(
        private readonly array $allowedFields,
    ) {}

    /**
     * Extracts and validates the requested fields from the query string.
     * Returns null if no projection is requested (return all allowed fields).
     *
     * @return list|null
     */
    public function resolve(Request $request, string $resourceType): ?array
    {
        $param = $request->query->get('fields');
        if ($param === null) {
            return null; // No projection, return all fields
        }

        $requested = array_filter(
            array_map('trim', explode(',', $param)),
            fn(string $f) => $f !== ''
        );

        $allowed = $this->allowedFields[$resourceType] ?? [];
        $valid = array_values(array_intersect($requested, $allowed));

        // Always include id for resource identification
        if (!in_array('id', $valid, true)) {
            array_unshift($valid, 'id');
        }

        return $valid ?: null;
    }

    /**
     * Builds Symfony Serializer context groups from allowed fields.
     *
     * @param list|null $fields
     * @return array
     */
    public function buildSerializerContext(?array $fields): array
    {
        if ($fields === null) {
            return ['groups' => ['api:read']];
        }

        // Map individual fields to groups: field "name" -> group "api:field:name"
        $groups = array_map(
            fn(string $f) => 'api:field:' . $f,
            $fields
        );

        return ['groups' => $groups];
    }
}

3. Pagination: cursor-based vs. offset for large datasets

Offset-based pagination (?page=5&per_page=20) is intuitive and simple to implement, but has fundamental performance problems with large datasets. A SELECT ... LIMIT 20 OFFSET 10000 has to scan the first 10,000 entries even though only 20 are returned. With a table of a million entries, page 50,000 is practically unqueryable. Offset pagination is also unstable: if an entry is inserted between page 3 and page 4, everything shifts by one position.

Cursor-based pagination solves both problems. The cursor is an opaque string (typically a base64-encoded primary or secondary index value) that identifies the last record seen. The query then becomes WHERE id > cursor LIMIT 20: no offset calculation, just an index scan. The response contains a next_cursor value for the next page. Cursors are immutable: inserts and deletes between requests do not affect pagination. The only limitation is that you cannot jump to an arbitrary page, only forward and backward (with the corresponding cursor).

4. HTTP streaming for large datasets

HTTP streaming makes it possible to transfer large datasets without keeping them fully in memory. Symfony's StreamedResponse transmits data as a stream: the connection stays open, data is sent in chunks, and the client can start processing immediately. This is especially valuable for export endpoints (CSV, JSON Lines) and for endpoints that need to stream database queries with very large result sets.

For JSON streaming, the JSON Lines format (application/x-ndjson) is a good fit: each line is a complete JSON object. The client can process each line immediately without waiting for the response to finish. Unlike a large JSON array, the client does not need to load the JSON Lines format into memory as a whole. For PHP that means while ($row = $query->fetchAssociative()) with a direct echo json_encode($row) . "\n" and a regular flush().

productRepository->getConnection();
                $stmt = $connection->executeQuery(
                    'SELECT id, name, sku, price, stock FROM products WHERE active = 1'
                );

                $count = 0;
                while ($row = $stmt->fetchAssociative()) {
                    echo json_encode($row, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n";
                    $count++;

                    // Flush buffer every 100 rows to avoid memory accumulation
                    if ($count % 100 === 0) {
                        ob_flush();
                        flush();
                    }
                }
            },
            status: 200,
            headers: [
                'Content-Type' => 'application/x-ndjson',
                'X-Accel-Buffering' => 'no',      // Disable Nginx buffering
                'Cache-Control' => 'no-store',
                'Transfer-Encoding' => 'chunked',
            ]
        );
    }
}

5. HTTP compression: Gzip, Brotli and Content-Encoding

HTTP compression is one of the simplest and most effective performance optimizations for REST APIs. JSON is highly compressible, typically down to 15 to 30 percent of its uncompressed size. A 100 KB JSON response shrinks to 15 to 30 KB. For small responses under 1 KB, the compression overhead dominates; above 1 KB, compression is almost always worthwhile. The client signals support via the Accept-Encoding header: Accept-Encoding: br, gzip, deflate. Brotli (br) is more modern and achieves 15 to 25 percent better compression ratios than gzip at comparable decompression speed.

For Symfony: Nginx or Apache already compress before PHP execution when configured correctly. In the Nginx config: gzip on; gzip_types application/json application/problem+json; brotli on;. That is more efficient than PHP-side compression with ob_gzhandler. Important: when using HTTP caching, the Vary: Accept-Encoding header must be set so caches keep separate entries for gzip and non-gzip clients.

6. HTTP caching: ETag, Cache-Control and conditional requests

HTTP caching eliminates redundant requests entirely. If a client already has the current version of a resource and it has not changed, the server responds with 304 Not Modified, without a response body. That saves bandwidth and server processing time. Two mechanisms are involved: ETag (entity tag) is a fingerprint of the response content. Last-Modified is a timestamp of the last change. On follow-up requests, the client sends If-None-Match: "etag-value" or If-Modified-Since: Tue, 09 May 2026 10:00:00 GMT, and the server replies with either 304 or the current version.

Cache-Control directives control how long and by whom something is cached: Cache-Control: public, max-age=300 allows public caches (CDN, proxies) to cache for 5 minutes. Cache-Control: private, max-age=60 allows only browser caching for 1 minute. Cache-Control: no-store prevents any caching (for sensitive data). s-maxage overrides max-age for shared caches (CDN), useful when browser and CDN TTL should differ. Symfony HttpFoundation makes it easy to set these headers correctly.

productRepository->find($id);
        if ($product === null) {
            return $this->json([
                'type' => 'https://mironsoft.de/errors/not-found',
                'title' => 'Product not found',
                'status' => 404,
            ], 404, ['Content-Type' => 'application/problem+json']);
        }

        // Build ETag from content hash, changes when product changes
        $etag = md5(serialize([
            $product->getId(),
            $product->getUpdatedAt()?->getTimestamp(),
        ]));

        // Check conditional request
        if ($request->getETags() && in_array('"' . $etag . '"', $request->getETags(), true)) {
            return new Response('', 304, [
                'ETag' => '"' . $etag . '"',
                'Cache-Control' => 'public, max-age=300, s-maxage=600',
            ]);
        }

        $data = [
            'id' => $product->getId(),
            'name' => $product->getName(),
            'price' => $product->getPrice(),
            'sku' => $product->getSku(),
        ];

        return new JsonResponse($data, 200, [
            'ETag' => '"' . $etag . '"',
            'Cache-Control' => 'public, max-age=300, s-maxage=600',
            'Vary' => 'Accept-Encoding, Accept-Language',
            'Last-Modified' => $product->getUpdatedAt()?->format('D, d M Y H:i:s') . ' GMT',
        ]);
    }
}

7. The N+1 problem and eager loading in API responses

The N+1 problem is the most common database performance cause in API endpoints: one query returns N products, then separate queries are run for each product's category, images, and pricing rules, for a total of 1 + N*3 queries. At N=100 that is 301 database queries for a single API response. Doctrine ORM makes this problem invisible through lazy loading: relationships are automatically loaded on access, without the code explicitly requesting it.

The solution is eager loading: Doctrine queries with ->leftJoin()->addSelect() load all needed relationships in a single query. The Symfony Profiler and the Doctrine SQL logger show how many queries an endpoint executes. More than 5 queries for one API response is a warning sign. Redis caching at the aggregate level (complete product data including relationships) can further reduce database access, but it requires a clean cache invalidation strategy for changes.

8. Comparing the performance strategies

Every performance strategy has its optimal area of use. The choice depends on the use case, the data volume, and the implementation complexity you can accept.

9. Summary

REST API performance is primarily a problem of transferred payload size, not server processing time. Projection reduces over-fetching through field selection via query parameter: simple to implement, immediately measurable. HTTP compression cuts network load roughly in half without any code change, provided Nginx is configured correctly. HTTP caching with ETag eliminates redundant transfers entirely for unchanged resources. Cursor-based pagination scales to millions of entries without offset degradation. Streaming enables export endpoints without memory overhead. The N+1 problem is detected and eliminated through eager loading and query profiling.

The practical path: measure first (Symfony Profiler, HTTP analyzer), then optimize. The most common wins are compression (Nginx config) and ETags (3 lines of code), which have an immediate large impact. Projection and cursor pagination require more implementation work but are indispensable in the long run for scalable APIs.

10. FAQ: REST API Performance