JS Rendering with Screaming Frog and Python

JS Rendering with Screaming Frog and Python

In this blogging session, you will get to know how to test rendering with Screaming Frog and Python. Follow along with the step-by-step guides to perform the operation.

js rendering through python

Prerequisites:

  • Need to install Python and Anaconda consoles on Windows/Mac
  • Need to have a licensed Screaming Frog tool to test

How to Test JavaScript Rendering on a Large Scale? (Step-By-Step)

Now it is time to put our website’s JavaScript (JS) to the test.

What we want to do is to:

Make two crawls with Screaming Frog, one with “Text Only” rendering and the other with “JavaScript” rendering.

  1. Export the Data in CSV
  2. Load the Crawl Data Using Python
  3. Combine the Crawls Into One Data Frame
  4. Check Differences Between Crawls
  5. Make a Report With Excel

Text Only Rendered Crawl

First, let’s crawl our website like Googlebot would do in its first wave before it renders the JS.

Go to Screaming frog > Configuration > Spider > Rendering > Text Only as shown here.

Then click on Start to crawl

JavaScript Rendered Crawl

Now, let’s crawl our website including rendered results. This will mimic which link Google will find in its second wave, where it renders the JS content after it has available resources.

Go to Screaming frog > Configuration > Spider > Rendering > JavaScript

Make sure to deselect the checkbox for Enable Rendered Page Screenshots

Then click on Start to crawl

Now that your crawl is complete, you will want to export the Data to CSV.

Go in Screaming Frog > Export

Now you have two different CSV files ready for your site, one for text-only version crawl and the other one is for JS rendered crawl data.

Now we run the crawls to Pandas to perform the test rendering

We only consider the following metrics:

  • Address
  • Status Code
  • Word Count
  • Outlinks
  • Unique Outlinks
  • Inlinks
  • Unique Inlinks

Check Differences Between Crawls

Because it will mean that a lot of content is hidden behind JavaScript and can’t be accessed from Google’s first wave crawling.

##Check the differences in each crawl

df['Diff Wordcount'] = df['Word Count_y'] - df['Word Count_x']
df['Diff Outlinks'] = df['Outlinks_y'] - df['Outlinks_x']
df['Diff Unique Outlinks'] = df['Unique Outlinks_y'] - df['Unique Outlinks_x']
df['Diff Inlinks'] = df['Unique Inlinks_y'] - df['Unique Inlinks_x']

##Check if canonical links are equivalent

## Need NumPy library

import numpy as np

df["Canonicals are equal"] = np.where((df["Canonical Link Element 1_y"] == df["Canonical Link Element 1_x"]), "yes", "no")

In this way, you will find out all the rendered JS files with the help of Screaming Frog and Python.