Blog

Crawl Traps: The Definitive Guide

When a particular website has structural issues that cause crawlers to find a virtually infinite number of irrelevant links, this might lead crawlers to get stuck at the one part of a website and it won’t complete crawling those irrelevant links this is where the “crawl traps” occurs.

We know that these crawlers are essential for crawling a website, indexing our content and ultimately displaying it to the audiences. If a certain website structure doesn’t allow a crawler to move through it seamlessly, the crawler will reach its limit of crawling allowance and move on to the next website.

The site’s speed will be downgraded as well as its rank compared to the site’s competitor and the site will never make it to the SERP.

Four main types of common crawler traps:

Mix and Match Trap:

Similar information provided in many ways.

Session ID Trap:

Almost duplicates with pages that differ by some infinite detail.

Calendar Trap:

Pages that are technically unique but provide no useful information (e.g. an event calendar that goes thousands of years into the future).

Never-Ending URL:

Infinite different URLs that point to the same page with duplicated content.

A crawl trap should be avoided at all costs as it decreases your site’s ability to be crawled and indexed, which in turn will greatly impact your overall organic visibility and rankings.

Forcing search engines to waste most of their crawl budget loading useless, near-duplicate pages. As a result, the search engines are often so busy with this that they never get around to loading all the real pages that might otherwise rank well.

  • Track down and fix the malformed link(s) that are creating the extra directory levels.
  • Add rules to the server config to limit rewrites to URLs with a specific number of slashes in them. Any URL with the wrong number of slashes should not be rewritten. This will cause malformed relative links to return a 404 error (as they should).

Block the trap URLs using robots.txt.

  •  
  •  
  •  
  •  

Leave a Reply

Your email address will not be published. Required fields are marked *