Do you need to perform a rapid crawl of a website? Do you want to visualize the insights to determine what to do next? This article will show you how to use Screaming Frog SEO Spider to do a rapid, high-level crawl of a website.
Screaming Frog’s SEO Spider is a versatile tool that you can use for a variety of tasks related to SEO. One of the most important functions of SEO Spider is its ability to crawl a website. This identifies the potential issues that may be negatively impacting search engine visibility. Moreover, the software includes a number of different crawl visualization options. You can use them to gain insights into the structure and content of a site. Users can more effectively use the Screaming Frog SEO Spider to improve their website’s ranking on search engine results pages.
Visualizations: An Overview
The interactive website visualizations provided by the Screaming Frog SEO Spider are of two types. They are – crawl visualizations and directory tree visualizations.
The two sorts of visualizations are distinct and serve different purposes in comprehending the design of a location. This tutorial will explain the distinctions and show how each might be used.
Each visualization also includes two formats: a force-directed diagram and a tree graph. Format type is a matter of personal preference. Although, the best format to use will also depend on the size of the site and the scenario.
While visualizations are appealing and entertaining, they do not add to the information provided in a crawl. They aren’t always the greatest technique to diagnose problems, and they don’t always give the complete picture.
However, visualizations provide perspective, communicate ideas, and reveal underlying patterns. These are difficult to detect in data and spreadsheets.
Let’s look at the distinctions between the two sorts of visualizations.
Crawl Visualization
The ‘force-directed crawl diagram’ and ‘crawl tree graph’ visualizations show how the SEO Spider explored the site. This is done by following the quickest path to a page. They display the quickest path to a page from the starting URL. They don’t reveal every internal link. Since it makes visualizations difficult to scale and incomprehensible.
An indexed page with many shortest paths displays the link crawled first. As a result, it is the first found in the source code.
In the crawl visualizations, crawl depth is shown by a line connecting URLs, which indicates the quickest path. Thus, crawl visualizations are valuable for analyzing site design and internal connectivity.
Force-Directed Crawl Diagram
The force-directed crawl diagrams resemble heat maps. The start URL is indicated by the darkest green, biggest node (the circles) in the center. The homepage is usually the starting point of the crawl. The lines (known as ‘edges’) reflect the link between two URLs (by the shortest path if you’ve been paying attention).
Linked together, smaller nodes represent the next crawl depth level. And as they grow further away, the nodes become smaller and lighter as the crawl depth increases.
Green nodes represent indexable pages. Whereas pastel red nodes highlight non-indexable URLs. This makes it very simple to identify problematic portions or pages of a website.
Crawl Tree Graph
The tree graph style is distinct from the force-directed crawl diagram. Still, they represent websites the same way, with URLs represented by circles and the shortest path represented by hyperlinks as the lines joining them.
By default, they are hierarchical by crawl depth from left to right, but you may turn them over to bottom (or right to left, bottom to top, etc.).
The crawl tree graphs will also display up to 10k URLs in the browser, but you can right-click and ‘focus’ on certain portions of a site to see additional URLs in that location (up to another 10k URLs at a time).
Directory Tree Visualization
The ‘force-directed directory tree diagram’ and ‘directory tree graph’ visualizations depict a website’s URL architecture. URL components such as protocol, host and path organize the Nodes (circles). Unlike crawl visualizations, nodes do not always indicate resolving URLs. , the lines do not always reflect resolving URLs.
The visualizations of the directory trees are hierarchical by URL component and path. The lines between URLs reflect the directory path. This makes directory tree visualizations valuable for analyzing URL structure and website information architecture.
This view also makes it simpler to discover issues because URLs share the same directory generally have the same template.
Force-Directed Directory Tree Diagram
The force-directed directory tree diagrams are similar to a heat map, with the protocol(s) represented by the darkest green, biggest node (the circles) often hanging from the very center node, which is the homepage (the hostname).
The lines (‘edges’) reflect the URL path. Thus protocol nodes link to the host, which connects to the first directory, path, etc.
By increasing the route, the nodes on the webpage become smaller and lighter as they grow farther away from the largest node.
Directory Tree Graph
The directory tree graph shows websites like the force-directed directory tree diagram, with nodes representing URL components and lines linking them representing the URL route.
By default, they are hierarchical by URL path from left to right, but you may turn them over to be top to bottom (or right to left, bottom to top, etc.).
Scaling Force-Directed Diagram
You may use different metrics to scale the ‘force-directed crawl diagram’ and the ‘force-directed directory tree diagram’ (by clicking on the cog icon). As previously stated, the size and color of the nodes are scaled by crawl depth or URL route.
There are, however, other factors that can affect the scale of both diagrams. Such as unique inlinks, Link Score (PageRank), or external metrics such as Moz Page Authority or Ahrefs URL Rating.
As an example, one can set Ahrefs URL Rating diagrams to visualize the URLs that have the greatest authority (including external links, not just internal links).
In the case of only internal links, Link Score indicates internal PageRank on the entire website.
Both of the preceding examples can assist in visualizing key pages and areas of a site with the highest authority and those that could benefit from the improved linkage.
Final Thoughts
Screaming Frog is an all-in-one tool with awesome functionalities that can benefit various aspects of SEO. These crawl visualizations are absolutely crucial not just to fix crawl errors, but also to improve interlinking and even to find new content opportunities.
