Find Duplicate Tags Using Google Webmaster Tools

picture of alfonso bedoya from Treasure of the Sierra Madre

Just throw those duplicate title tags over here and we’ll be on our way.

Sometimes this is easy, sometimes not so much.  The first and foremost rule of SEO is to make sure each of your unique pages has a unique title tag.  That’s because the first and foremost rule of Google (or at least sometimes it seems that way) is to avoid the dreaded “duplicate content” monster.

Now the purpose of this SEO tip is not to show you what a title tag is, or how to manipulate it.  I’ll leave that to other pages that are already online.  (At the bottom of this post is a link to one that you might find of interest.)

Instead of covering the basics of title tags right here, I want to point you to an easy way to determine where to look for duplicate tags in your site.  The best tool I know of, and it’s free, is of course Google Webmaster Tools.  If you haven’t already created an account with Google, or tied your Google account into, Webmaster Tools, you’ll need to get that done first (learn more about Google Webmaster Tools here).  Then if you visit Google Webmaster Tools, the following illustration will make sense to you:


Webmaster Tools

OK, so we’re done.  Right?

Eliminate “Duplicate” Content with Robots.txt and Wild Cards

Not so fast, hombre.  (Or mujer, as the case may be.)  Depending on the type of platform you’re working on, you might have a challenge on your hands.  For example, one of my sites is an ecommerce website built on top of a Magento platform.  I noticed that the duplicate title tags for this site numbered in the thousands and thousands, yet I had carefully optimized the website title tags.

After a bit of searching I found that the primary problem related to the way Magento was allowing Google to find many pages using different URL’s.  For example, the following two URL’s take a shopper to the exact same item:

Google will view these addresses, not as one page with duplicate addresses, but separate pages with duplicate content.  The mighty G doesn’t like that.  It’s so untidy.

So what do we do about that?  Allow me to introduce my friend, robots.txt.  Allow me to introduce my other friend, wild cards.  By using a robots.txt file (which Google allows you to create in Webmaster Tools) and some creative use of wild cards, we can make sure that Google only looks at this jersey in one of the two places that Magento would normally allow it to.

Here’s how to get to the robots.txt tab in the dashboard.

Robots txt Testing Tool in Webmaster Tools

A robots.txt file basically tells Google what to ignore.  So, let’s use robots.txt to tell Google to ignore any content that appears with a category of “mens-s-jerseys” (you can see that in the first url up above).  The syntax for that would look like this:

Disallow: /*men-s-jerseys*

Now that you have the principle in hand, slow down.  If you put a disallow line in your robots.txt file, and if you get it wrong, you could block Google from accessing your entire site.  Yikes!

But I didn’t say stop, just slow down.  Fortunately Google builds a handy little test mechanism into Webmaster tools.  Here’s how you use it:

  1. I add the line I want to test in the robots.txt
  2. I input the two different url’s that I want to test.
  3. I click the test button
  4. I examine the results…and the results are what I want: one url is blocked, one is allowed.  Duplicate content has been eliminated.  Hurray for me!

See the screenshot below to see what that 1-4 process looks like on screen.

 screen shot of testing robots.txt in Google webmaster tools

Well, I have no illusions that this process is a bit on the advanced side of SEO.  But it’s certainly not beyond your reach, you starry-eyed dreamer.  Just use the test tool, ok?  Otherwise I’ll worry.

Footnote: Want the basics on title tags? I did a quick Google search (handy tool, this Google thang).  I found this fairly useful beginners introduction to title tags (and a few other tags as well).  In the near future I’ll also post a video tutorial on updating title tags in WordPress, the platform I prefer to work with for the websites I manage.

Free SEO Website Analysis

Subscribe to our blog graphic