Why SEO Needs Better Testing – Here’s Why #90

Real scientists know they have to set up tests and experiments with extreme care if they want to get meaningful results. SEO is both an art and a science and for the science part of it, we’re highly dependent on findings from studies and tests. So they deserve the same care that real scientists give their experiments.

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

Don’t miss a single episode of Here’s Why. Click the subscribe button below to be notified via email each time a new video is published.

Subscribe to Here’s Why

Resources Mentioned

• Are Links Still a Powerful Ranking Factor? (New Study).
• Becoming Better SEO Scientists – Whiteboard Friday
• Perficient Digital’s Social Media & SEO Data Studies
• All of our Here’s Why videos

Transcript

Mark: Eric, you’re the lead co-author for a book titled “The Art of SEO.” But I know you agree that while there is an art to doing great SEO, there’s a certain amount of creativity and intuition based on experience that goes into it, that SEO is also a science, right?
Eric: Absolutely. We’re working with data in an environment, and that environment constantly evolves over time. We can devise hypotheses about what causes it to evolve in certain ways, then we can set up studies and experiments to test those hypotheses.
Mark: I like to think about what scientists do, the way we usually think about them. We can draw analogies to what we try to do here at Perficient Digital, right? So scientists work within a world, our world. The cosmos and all of experience confirms to us that the world operates by fixed laws. However, we don’t yet know what all those laws are or how they all interact. So part of the world is known in the laws we already know and have proven to work through consistent results for many experiments and part we don’t know yet. For the latter, we can often see the effects in our world. And we know that there must be laws controlling those effects. But we don’t yet know what those laws are.
Eric: An example would be what scientists call dark matter. No one has seen dark matter. But we hypothesize that it must be there because we see effects in our universe that we can’t otherwise explain. If dark matter exists, then it could be exerting forces that would account for those effects. So scientists keep setting up tests and experiments to try to confirm or disprove the existence of dark matter.
Mark: It is a great example because, in our world of SEO, we could think of our universe, our world as the search results pages of search engines like Google, and all the things known and unknown that might affect those results. Now just as with our larger real world, some of the laws of search are known and accepted as proven while others remain darker and more mysterious.
Eric: An example of the laws of SEO we already know with a high degree of certainty is that links from other sites have a major effect on search rankings.
Mark: In the case of links, the only thing under dispute over time is just how much of an effect those links have in the mix of all the things that affect rankings. In fact, Eric conducted a very careful study on just that question which can be found here: Are Links Still a Powerful Ranking Factor? (New Study).
Eric: Okay, that’s all good background. But I happen to know that you came into the studio today with a pet peeve bugging you on this topic of SEO testing. So let’s hear it.
Mark: It’s true. I’ll do my best to stay out of rant mode. Anyway, one of my pet peeves in general is junk science. What I mean by that is either scientific testing that is not conducted carefully or that is reported by media in ways that misunderstand or misrepresent the findings, or sometimes, it’s just that the current findings are only preliminary. They need more testing. They get announced as “Science has now proven whatever.”
Eric: And I take it you see that in the world of SEO as well.
Mark: Sadly, yes. All too often, we see published studies or tests that either show very tentative preliminary results that need to be confirmed, or that are just plain invalid due to either faulty methodology or unwarranted conclusions.
Eric: Do you have an example?
Mark: I do. Now, I’m not going to call out the author or link to the article because it’s not my purpose to embarrass anyone here. But recently, I saw published in one of the most respected and widely read online publications in our industry, a test about the effect of click-through rate on rankings that made me groan in pain.
Eric: Why?
Mark: Well, for one thing, the testing results presented were one-off and anecdotal. That is, the author did one test with only one input and saw what he judged to be a positive response and stopped there. Now, maybe what he saw was interesting, maybe not. Without further tests, there is no way to know how probable it is that the result was directly caused by the input.
Eric: So one principle of good testing is don’t put too much stock in a one-off test or an anecdotal experience.
Mark: Or to put it positively, it’s important to test many incidences to see if the effect happens consistently. It can also be valuable to have others try to replicate the experiment with their own inputs to see if they get similar results. Now, the other thing that bothered me in this test was the lack of any control.
Eric: Explain what a control is in a science experiment.
Mark: Okay, put simply, controlling an experiment means two things: first, you should try to eliminate as much as possible the possibility of other variables that could affect your results. Now, in the case of the CTR test I saw, I could come up with probably a dozen things known and unknown that could have accounted for the person’s results without any involvement from click through.
Now, second, in most experiments, it’s good to have a specific control. That’s a part of the test where you have similar inputs but you do not do whatever you’re doing to the experimental group. If you see the same results in your control group as in your experimental group, then probably what you did to your experimental group isn’t the direct cause of what happened there.
Eric: So anything else you wanna say about such SEO testing before we wrap up?
Mark: Yes, just one more thing. I actually think it’s fine to present incomplete or unverified test or study results as long as you are crystal clear that’s what they are. On the positive side, such data can be useful to stimulate further or better experiments that will give us true, actionable results. Now, it’s valuable for raising questions that we should be testing more deeply. But it goes negative for me when as happens all too often, someone presents a preliminary or faulty test and then actually recommends that others take action based on their findings.
Eric: Thanks, Mark. There’s a lot more to valid SEO testing than what Mark has covered here. I recommend that you watch his Moz Whiteboard Friday video called “Becoming Better SEO Scientists”.
Mark: Thanks, Eric. And here at Perficient Digital, we’ve invested a lot into doing careful scientifically valid tests and studies. If you’d like to see what you could learn from our efforts, you can view our studies in one listing here.
Don’t miss a single episode of Here’s Why. Click the subscribe button below to be notified via email each time a new video is published.

Subscribe to Here’s Why