Web server log files can yield a wealth of valuable information for the savvy SEO.
In this episode of our popular Here’s Why digital marketing video series, Perficient Digital’s Eric Enge explains why log files are so valuable for SEO analysis.
Don’t miss a single episode of Here’s Why. Click the subscribe button below to be notified via email each time a new video is published.
Resources
- The Value of Log File Analysis
- 15 Crucial Elements of an Informative SEO Audit
- See all of our Here’s Why Videos | Subscribe to our YouTube Channel
Transcript
Mark: Okay, Eric. Say, I’m a site owner.
Eric: Okay. You’re a site owner.
Mark: Say, I’m a site owner and I’ve done all the basic tasks of an SEO audit. Is there anything people commonly overlook?
Eric: Well, often in my experience, Mark, site owners neglect to check their log files on a regular basis.
Mark: Their log files?
What Is a Log File?
Eric: A log file is a record maintained on most web servers that records all the incoming activity from humans, bots, and anything else that might have access to or even attempt to access your site. What’s particularly valuable about those logs for SEO purposes is the fact that they record every visit from a search engine to your site.
Mark: And you can learn valuable things from that?
Eric: Oh, yes. For example, over time, you can spot patterns of search engine crawler behavior on your site. Is a crawler’s visiting frequency tending up or down? Each could be communicating a message about how that search engine views your site.
[Tweet “A site’s log file reveals how search engine crawlers treat your site. Why is that important for #SEO?”]
Mark: Well, that’s interesting because that’s something you can’t see in your analytics, right?
Eric: That’s right. Another thing you can’t do in your analytics is go back in time any further than before when you installed the analytics in the first place. So, in contrast, with log files you can go back to the beginning of the site time since you set it up, provided that your server host preserves log files. And most do. And if they don’t, you can ask them to do that for you.
Mark: Good. Well, that sounds handy for investigating problems that might have been building up over a long time. Does log file analysis have any other benefits for SEO?
Eric: Sure. For one thing, it’s not a drag on page speed like JavaScript for analytics can be. And you don’t have to worry about messing it up by applying the code incorrectly.
Mark: Okay, well those are some of the advantages of log files. Now let’s get specific about what we can discover by using them.
What Can SEOs Discover in a Log File?
Eric: There are a number of useful things you can find out by examining your log files. One is knowing when and how often crawlers access your site. Of course, that means you can also see who might be trying to scrape your site or identify possible DOS, or denial of service, malicious attacks.
Mark: Well, what about the issue of crawl budget and priority?
Eric: Ah, yes. Search engine crawlers will only crawl so much of a site on any given day. We call how much effort they will give to a particular site its crawl budget. Knowing that, you want to direct as much as possible of that crawl budget to the pages that are highest priority for you to make sure that search crawlers are getting to your most important pages first and most often. Your log files help you determine all of this. And then, there are things you could do to redirect crawl priority to where you want it.
[Tweet “Your site’s log files can help you determine the search engine crawl budget for your site. More at”]
Mark: To learn more about how to do that and about some other uses of log file analysis, see Eric’s in-depth article on the topic on our blog. We’ll give you a link in the episode notes.
Don’t miss a single episode of Here’s Why. Click the subscribe button below to be notified via email each time a new video is published.
See all of our Here’s Why Videos | Subscribe to our YouTube Channel
I use GoDaddy for hosting, but they only show me half of my logs for the month. Is there another hosting provider I should look at that is relatively affordable so I can get the whole month?
Is this file really important for SEO?
I use Pair.com, which is more expensive, but I have access to all my log files. You can also ask GoDaddy if they can give you access to more.
As noted in my article, I think you can learn a lot that will help you with SEO. The log files themselves don’t have any direct impact on SEO.
I did ask GoDaddy yesterday, but they were unable to create a Cron Job for me to get the daily log file data. They only give monthly summary files to download that oddly enough only contains half the month’s logs
I’ve used http://www.loggly.com in the past, what log viewer do you use?
We use our own proprietary tool here at Perficient Digital. Our web crawler crawls sites, and also looks at the log files at the same time.
Good Article For Newbies Eric. Thanks For Sharing this.