TECHNICAL · SEO GLOSSARY
Log File Analysis
Analysing a web server's access logs to understand exactly which URLs Googlebot crawled, how often, and with what response codes — the most direct way to diagnose crawl budget problems.
Definition
Log file analysis is the process of examining the raw HTTP access logs that a web server generates for every request. For SEO purposes, the relevant entries are those from Googlebot (and other crawlers) — showing which URLs were fetched, when, how frequently, and what HTTP status code was returned. Log analysis answers questions that no third-party tool can: Does Googlebot crawl my most important pages daily? Is it wasting crawl budget on parameter URLs? Are 404 errors being served at significant URLs? How long after a deploy does Google discover the change? The most common tools: Screaming Frog Log File Analyser, JetOctopus, or custom Python scripts on nginx/Apache log files.
Why it matters for SEO
Log file analysis is the ground truth of how Googlebot actually behaves on your site. Common findings: (1) important pages are crawled only monthly while low-value parameter pages are crawled daily; (2) noindexed pages are still being crawled, wasting crawl budget; (3) a redesign broke URLs that Googlebot is still trying to fetch (404s on formerly-popular URLs with no redirects); (4) the homepage is crawled 50× more than the deepest product pages — classic crawl depth problem. These findings directly inform decisions: which pages to move to shallower IA, which redirects to add, which parameter paths to Disallow in robots.txt.
How DeepSEOAnalysis checks this
The DeepSEOAnalysis audit does not read server log files — those are private by definition and require server access. The audit approximates log-file findings using crawl data: it maps which URLs it reached, notes crawl depth, identifies orphan pages and parameter chains, and flags broken internal links. For full log file analysis, you need to download your server logs and use a dedicated log analysis tool. The audit flags situations where log file analysis is strongly recommended — large sites with parameter URLs, sites post-migration, and sites where GSC shows unexpected coverage drops.
Useful tools and resources
GLOSSARY
Related terms
technical
Crawl Budget
The number of pages Googlebot will crawl on a site within a given timeframe — determined by crawl rate limit and crawl demand.
Read definition →technical
Crawl Depth
The number of clicks (links) a crawler must follow from the homepage to reach a given page — pages deeper than 4 clicks receive less frequent crawling and less internal PageRank.
Read definition →technical
Crawlability
Whether search engine crawlers can successfully access, fetch, and parse a page — the prerequisite for indexing and ranking.
Read definition →technical
URL Parameter
A key-value pair appended to a URL after a `?` character — such as `?sort=price` or `?page=2` — that can create thousands of duplicate-content URLs if not handled correctly.
Read definition →technical
Faceted Navigation
Filter-and-sort UI on category pages (color, size, price, brand) that generates a combinatorial explosion of parameter URLs — the most common source of crawl budget waste on e-commerce sites.
Read definition →technical
Google Search Console
Google's free tool for monitoring how your site appears in Google Search — showing impressions, clicks, CTR, average position, index coverage, and crawl errors.
Read definition →See how your site scores on Log File Analysis.
The free DeepSEOAnalysis audit checks log file analysis and 100+ other signals. Full report, no signup.
Run a free audit →