Log File Analysis: How Google Actually Crawls Your Store

1. Why Server Log Files Show More Than Google Search Console

Google Search Console delivers valuable signals in the "Crawl stats" report and under "Page indexing," but it is built on samples and aggregated values. For every status group like "Crawled, currently not indexed," Google shows at most around 1,000 example URLs, regardless of whether a store has ten or two hundred thousand affected pages. Raw server log files know no such limit: every single HTTP request Googlebot makes to the server is logged, complete with an exact timestamp, the requested URL, the status code, and the user agent. Anyone who wants to know what Google actually does, rather than what Google shows in a condensed interface, cannot avoid this raw data source.

The second crucial difference is timeliness. Search Console data is one to three days old depending on the report and sometimes further smoothed, while log files are available immediately after the request, as soon as log rotation kicks in. For acute problems like a sudden crawl spike after a deployment or a drop in crawl frequency after a server migration, this delay is decisive: anyone waiting on Search Console data reacts, in the worst case, only after the damage is already done. Log file analysis closes this gap by tapping the data source directly at its root, right where Googlebot actually arrives.

2. What a Log Line Actually Contains: Structure and Relevant Fields

A typical log line in the Nginx or Apache "combined log format" contains seven core fields: the requesting IP address, the timestamp, the full request line with HTTP method and path, the returned status code, the number of bytes transferred, the referrer, and the user agent string. For SEO-relevant crawling analysis, three fields matter most: the IP address, because it can be verified via reverse DNS, the requested path including the query string, because it shows exactly which URL variant was crawled, and the status code, because it reveals whether a request was answered with a 200, a redirect, or an error.

The timestamp additionally provides the basis for frequency analysis: how often does Googlebot visit a given URL per week, and does that frequency change after a content update? The bytes transferred and, if logged, the response time in turn show how much load individual crawling waves actually place on the server, which quickly becomes relevant for stores with thousands of product variants and facet URLs. Anyone who consistently evaluates these fields understands not just what is being crawled, but also with what priority and at what cost in server load.

# Nginx combined log format: a genuine Googlebot request
66.249.66.1 - - [11/Jul/2026:03:14:22 +0200] "GET /catalog/product/view/id/482 HTTP/1.1" 200 48213 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

# Same user-agent string, but the IP does not belong to Google
185.220.101.47 - - [11/Jul/2026:03:15:07 +0200] "GET /catalog/product/view/id/482?color=42&size=99&sort=price_desc HTTP/1.1" 200 48210 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

# Redirect and error responses are just as important as 200s
66.249.66.3 - - [11/Jul/2026:03:16:41 +0200] "GET /catalogsearch/result/?q=old-product HTTP/1.1" 404 512 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

3. Setting Up Log Analysis: Access, Rotation, and Retention

The first step is getting access to the raw data itself. In the Mark Shust Docker setup, the relevant access logs live inside the container and can be read via the wrapper command bin/log; in production, access usually sits with the hosting provider or in central log management. It is important to configure log rotation from the start so that at least 30, ideally 90 days of raw data are retained, because crawling patterns and frequency changes only become visible over several weeks. Retention that is too short makes the entire analysis worthless, since seasonal or deployment-related effects simply can no longer be traced.

Once access is in place, it is worth separating bot and user traffic early, directly in the ingestion pipeline, for example via a regex filter on the user agent string, to keep the data volume manageable for the actual analysis. It is also important to think about the CDN or WAF layer, such as Cloudflare: requests answered or cached there never reach the origin server log and must be pulled in separately via the CDN's own log exports, otherwise the picture of actual crawling volume ends up distorted.

4. Spotting Wasted Crawl Budget: Patterns in Raw Logs

Crawl budget describes the limited number of requests Googlebot allows itself against a server in a given period, depending on server capacity and the perceived importance of the domain. For large Magento catalogs with faceted navigation, wasting this budget is a common but, in Search Console, barely visible problem: combinations of color, size, and sort order theoretically generate thousands of URL variants per category, the overwhelming majority of which show the same content in a different order. In raw logs this pattern becomes visible immediately once requests are grouped by URL path and counted by frequency.

Other typical waste patterns that only show up in the raw log: session IDs or cache-busting parameters in the URL that make every request look like a "new" page to Google, redirect chains that get crawled repeatedly instead of resolved directly, and soft 404 pages that technically respond with status code 200 even though the content is missing. A simple breakdown of top URL patterns by request frequency reliably reveals which handful of path patterns consume the bulk of daily crawl budget while genuinely indexable product pages get visited only rarely.

# Filter Googlebot user-agent hits and count status codes
awk -F'"' '$6 ~ /Googlebot/ {print $0}' /var/log/nginx/access.log \
  | awk '{print $9}' \
  | sort | uniq -c | sort -rn

# Top 20 URL paths requested most often by Googlebot
grep 'Googlebot' /var/log/nginx/access.log \
  | awk -F'"' '{print $2}' | awk '{print $2}' \
  | sort | uniq -c | sort -rn | head -20

5. Telling Real Googlebot Apart from Spoofed User Agents

The user agent string Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) can be faked by any scraper or bad bot at will, with no technical barrier whatsoever. Anyone running crawl budget analysis based on the user agent alone risks attributing fake bot traffic to Google and drawing the wrong conclusions from it, for example that Google is excessively crawling a certain URL when in reality a scraper or a competitor's tool is behind it. In many log files, this fake traffic makes up a surprisingly high share of supposed Googlebot hits.

The official and reliable verification method is a two-way DNS check: first, a reverse DNS lookup determines which hostname the requesting IP address belongs to. A genuine Googlebot always resolves to a subdomain of googlebot.com or google.com. That hostname is then resolved again via a forward DNS lookup to confirm it leads back to the original IP address. Only when both directions match is the request a verified Googlebot. This check can easily be automated and should be a fixed part of every crawl budget analysis before any content conclusions are drawn.

#!/bin/bash
# Verify a claimed Googlebot IP via reverse and forward DNS lookup
IP="66.249.66.1"

HOSTNAME=$(dig +short -x "$IP")
echo "Reverse DNS: $HOSTNAME"

if [[ "$HOSTNAME" == *".googlebot.com."* || "$HOSTNAME" == *".google.com."* ]]; then
  RESOLVED_IP=$(dig +short "$HOSTNAME" | tail -1)
  if [[ "$RESOLVED_IP" == "$IP" ]]; then
    echo "VERIFIED: $IP is a real Googlebot"
  else
    echo "SPOOFED: forward lookup mismatch ($RESOLVED_IP != $IP)"
  fi
else
  echo "SPOOFED: hostname does not belong to google.com or googlebot.com"
fi

6. Crawled but Not Indexed: Diagnosis Through Log File Matching

The Search Console status "Crawled, currently not indexed" is one of the most common yet least explained messages for Magento stores with large catalogs. Google does not reveal why an already crawled page was not added to the index, but matching it against raw logs often provides decisive clues: if the URL was visited only once, weeks ago, and never again since, that points to low priority in Google's internal queue. If it was crawled repeatedly instead, but the content never changed, that points to a quality or duplicate content problem rather than a pure crawling issue.

In practice, a join between log data, prepared by last crawl date, frequency, status code, and response size per URL, and a complete URL inventory from the product catalog proves useful. This makes it possible to systematically detect whether affected pages respond unusually slowly, are unusually small, which points to thin content, or sit in a cluster of very similar URLs. This combination of log data and content signals delivers far more concrete clues than the plain status message in Search Console.

-- Crawl frequency grouped by URL pattern (BigQuery, logs exported via Cloud Logging)
SELECT
  REGEXP_EXTRACT(request_path, r'^(/catalogsearch|/catalog/category|/catalog/product)') AS url_pattern,
  COUNT(*) AS crawl_hits,
  COUNTIF(status_code = 200) AS hits_200,
  COUNTIF(status_code >= 300 AND status_code < 400) AS hits_redirect,
  COUNTIF(status_code >= 400) AS hits_error,
  APPROX_QUANTILES(response_time_ms, 2)[OFFSET(1)] AS median_response_ms
FROM `project.logs.googlebot_requests`
WHERE request_date BETWEEN '2026-06-11' AND '2026-07-11'
GROUP BY url_pattern
ORDER BY crawl_hits DESC;

7. Tools for Log Analysis at Scale

For small to medium stores, the Screaming Frog Log File Analyser is the most pragmatic entry point: raw logs are imported via drag and drop, the tool automatically verifies Googlebot requests via reverse DNS, and it immediately produces readable overviews of crawl frequency, status codes, and URLs never crawled, matched against the sitemap. For data volumes in the low millions of lines, this is entirely sufficient and requires no dedicated infrastructure.

Once a store runs multiple server instances, sits behind a CDN, and generates log volume in the tens of gigabytes per month, a self-hosted ELK stack made of Elasticsearch, Logstash, and Kibana becomes worthwhile, allowing custom dashboards, longer retention, and flexible queries. For enterprise setups with very high volume, BigQuery is the most scalable option: logs are exported via Cloud Logging or Fluentd, and queries across billions of rows can be joined against sitemap exports or crawl stats data with SQL in seconds. The Search Console Crawl Stats API usefully complements all three approaches but does not replace raw data, since it too only delivers aggregated values.

8. Magento and Hyvä-Specific Log Considerations

Magento stores generate a few recurring, technically rooted sources of crawl budget waste. Faceted navigation in the category layer theoretically allows an unlimited number of filter combinations per category, the built-in catalog search generates additional crawlable URLs via the q= parameter, and multi-website or multi-store-view setups frequently produce near-duplicate URLs for the same content across different store views. REST and GraphQL endpoints are occasionally crawled too, if they are not cleanly excluded via robots.txt, even though they provide no value for organic search whatsoever.

The Hyvä Theme itself has no direct influence on these server-side causes, since crawl budget waste occurs at the URL and response level, not in frontend JavaScript. An indirect but measurable effect comes through the full page cache: pages served quickly from cache cost Googlebot less time per request, allowing more URLs to be crawled within the same time budget. In raw logs this shows up clearly as a cluster of very short response times for FPC hits versus noticeably slower, uncached responses for cache misses, a pattern that can be evaluated specifically to identify cache gaps.

# robots.txt: block low-value, crawl-budget-wasting URL patterns
User-agent: *
Disallow: /catalogsearch/result/
Disallow: /*?*color=
Disallow: /*?*sort=
Disallow: /*?*p=
Disallow: /checkout/
Disallow: /customer/
Disallow: /catalog/product_compare/
Allow: /

Sitemap: https://mironsoft.de/sitemap.xml

9. Turning Findings into Action: The Optimization Loop

Findings from log analysis only create value once they are prioritized into concrete action. The most sensible starting point is sorting the identified URL patterns by wasted request volume: the handful of patterns that consume the largest share of daily crawl budget get addressed first via robots.txt disallow rules, canonical tags, or noindex meta tags, instead of tackling every problem at once. Soft 404 pages should be consistently switched to genuine 404 or 410 status codes so Google correctly classifies them as no longer existing, instead of repeatedly crawling them without success.

Log analysis is not a one-time audit but a recurring process: a monthly or at least quarterly log review uncovers new waste patterns before they take hold, for example after introducing new filter attributes or a new marketing tracking parameter. The insights gained should also flow back into internal linking and XML sitemap maintenance: reducing links to low-value filtered URLs and focusing the sitemap on genuinely indexable pages reinforces the effect of the technical blocks. The loop closes because saved crawl budget measurably reaches the pages that were previously undercrawled more often.

Search Console Compared to Server Log Files

The following overview summarizes where server log files offer a concrete analytical advantage over Search Console.

In practice, both data sources complement each other: Search Console provides a quick overview of trends and status groups, while server log files provide the granular raw data for solid prioritization decisions. Anyone who combines both sources and reviews them regularly makes crawl budget decisions based on facts rather than guesswork.

10. Summary

Log file analysis answers a question that Search Console structurally cannot answer: what does Googlebot actually do, unfiltered and unsampled, on every single URL of a Magento store? Raw server log files show the exact request, the status code, and the timestamp of every single crawling event, making patterns visible that disappear in aggregated reports: wasted crawl budget from facet URLs, spoofed Googlebot access, and the silent causes behind "Crawled, currently not indexed."

The effort of a clean setup, from log rotation through reverse DNS bot verification to choosing the right analysis tool among Screaming Frog, ELK, and BigQuery, pays off with more precise decisions than any Search Console interpretation alone can deliver. Anyone who consistently feeds results back into robots.txt rules, internal linking, and sitemap maintenance ensures that Google spends its limited crawl budget where it actually matters for visibility and revenue.

11. FAQ: Log File Analysis for Magento Stores