Be The Search Engine  

Often times the best way to diagnose an SEO-related problem with your site is view the site as a search engine would see it. Today I’m going give you a few tips that will help you do just that. Shall we?

Step 1 – Download the Firefox browser, if you haven’t already

I can’t begin to tell you all the ways that Firefox is superior to Internet Explorer. For the sake of this article however, we’ll be leveraging its extensive list of “Add-ons.” Add-ons are little widgets you can add to your browser that help you to perform all kinds of useful tasks. The best part? They’re usually free.

Step 2 – Become the Googlebot

Googlebot is the name of the crawler or “User-Agent” that visits your website and tells Google all about what’s going on. There’s a really helpful Add-on that allows you to view the site as if your browser was a Googlebot crawler. Here’s how:

  1. Visit in Firefox, download the add-on and install it.
  2. You’ll likely need to restart the browser. But when you do, you should now be able to find this set of menus under “Tools”:
    Switch User Agent
  3. Select which Search Robots you’d like to crawl around as, and then go to your website. You’ll now be viewing the site like a crawler!

Why is this important?

Sometimes developers or web-designers create a different experience for the crawlers than they do for regular users. This is called “cloaking” and it most assuredly will eventually bring a penalty from the search engines. If you think you’re site has been penalized, this one way you can find out why. Try turning the plug-in on and off to see if the experience changes dramatically. If it does, you may have a problem.

Step 3 – See your site like a search engine would

Search engine crawlers are not as adept at crawling the web as our browsers are. They have limitations as to what they can do and see. Two examples of this are Cookies and JavaScript.

Cookies are little piece of data that get stored on your computer as you travel the web. These Cookies provide information to websites and they’ll use this data to make your visit a little more unique or useful. A good example of this is remembering if you were logged-in or remembering what you may have previously searched for on the site.

JavaScript is a kind of language used to make more robust and interactive experiences on a website. Programmers use it on a multitude of things from site navigation, to animations, to banner ad placements.

Search engines do not currently accept Cookies and they don’t perform JavaScript code. Here’s how you can easily disable Cookies and JavaScript so you can view your site like a search engine:

  1. Visit in Firefox and install this Add-on. After you restart the browser, you should see a new toolbar appear.
  2. Disable JavasScript by selecting the following:
    Disable JavaScript
  3. Disable Cookies by selecting the following:
    Disable Cookies
  4. Now reload the site you want to view and you’ll be visiting without JavaScript or Cookies enabled.

Why is this important?

If you’re not sure why a certain page isn’t ranking as well as you’d like, one cause could be that the crawlers have no way to find it because all the navigation is being handled by JavaScript or the site just doesn’t work unless you have Cookies enabled. You’ll be able to diagnose that right away by disabling both of those and visiting your site. This happens more often than you would think.

There you go! You’re on your way to acting just like a search engine and diagnosing SEO like a pro. There certainly are more tools out there you can use to diagnose even more problems. Which ones do you use? I’d love to hear about what works for you. If you’d like to hear more about the other ones I use, just let me know in the comments below and maybe we’ll have a part-two for this article. Now get out there and start crawling your sites!


This entry was posted in SEO. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>