⭐️What is an SEO Competitor Analysis
An SEO competitor analysis is a process of evaluating and analyzing the SEO strategies and performance of your competitors to gain insights and identify opportunities to improve your own search engine optimization efforts through programmatic SEO approach. It involves studying the websites and online presence of your competitors to understand their organic search rankings, keyword targeting, content strategy, backlink profile, and overall SEO tactics.
How to perform an SEO competitor analysis using Python
Set up a custom search engine
- Create a new search engine
https://programmablesearchengine.google.com/controlpanel/create
Enter a Name for your search engine
- New search engine has been created
Here is a link to the Programmable search engine homepage and within a few clicks, you should be able to set up your own programmable search engine.
https://programmablesearchengine.google.com/controlpanel/all
Then go Customise
- Go to Google Developer Console and click the Get a Key button.
https://developers.google.com/custom-search/v1/overview#api_key
Select your search engine and go Next.
- Competitor Analysis – SERP heatmap Master.ipynb
https://colab.research.google.com/drive/1SqVPj7nYynXFHX8alYkAIp-Pm2ArUNQY?usp=sharing#scrollTo=6d0c64b0
Run and Wait.
Add Keywords and your Search Engine ID & API Key
Run the serp_goog function
View the results of the serp_rankings output
5. Run & Export Data to CSV
7. Run the SERP Heatmap Function
8. Call or Invoke the serp_heatmap function
9. View your SERP Heatmap Data Visualization
⭐️Perform A Cookie Audit of the Website
Cookie Audit Introduction:
It’s essential today to understand the presence and purpose of cookies on a website. The data privacy landscape is continuing to evolve and there are various laws and regulations in place throughout the world, where cookie compliance is almost always included in some way.
1) Select ‘Cookies’ For Extraction
Open up the SEO Spider, go to ‘Config > Spider > Extraction’ and select ‘Cookies’ under ‘URL Details’.
This means the SEO Spider will now store all cookies discovered.
Please note, when you choose to store cookies, the auto exclusion performed by the SEO Spider for Google Analytics tracking tags is disabled to provide an accurate view of all cookies issued.
This means it will affect your analytics reporting, unless you choose to exclude any tracking scripts from firing by using the exclude configuration (‘Config > Exclude’).
2) Enable JavaScript Rendering
Click ‘Config > Spider > Rendering’ and select ‘JavaScript’. This means the SEO Spider will open each web page in a headless Chrome browser behind the scenes.
This is an important step, as it allows cookies that are loaded using JavaScript or pixel image tags to be discovered.
The window size is automatically set to Googlebot Smartphone, but this can be adjusted to desktop if there are any differences in the way the site issues cookies.
3) Set The User-Agent To Chrome
To replicate a regular user, switch the user-agent to a browser, such as Chrome via ‘Config > User-agent’.
Some websites conditionally set cookies based upon user-agent and crawling as the ‘Screaming Frog SEO Spider’ may not always give a true picture otherwise.
4) Ignore robots.txt
Click ‘Configuration > robots.txt > Settings’ and select ‘Ignore robots.txt’ or ‘Ignore robots.txt but report status’.
Cookies can be loaded from URLs which are available to users but not bots, such as pages or resources blocked via robots.txt. Therefore it’s important we allow all resources to be loaded. Any user set Include or Exclude functions can also affect this.
5) Crawl the Website
Open up the SEO Spider, type or copy in the website you wish to crawl in the ‘Enter URL to spider’ box and hit ‘Start’.
Wait until the crawl finishes and reaches 100%, but you can also view some details in real-time.
6) View Number of Cookies For Each URL
In the ‘Internal’ tab, there’s a ‘cookies’ column which displays the number of cookies discovered for each URL.
You will need to scroll over to the right to see it. This will help you identify where cookies are being discovered in the crawl.
7) View The Cookies Tab
Click on a URL in the top window, then the lower ‘Cookies’ tab to populate the lower window pane with more details on cookies discovered for each URL.
You can click on the above image to view a larger version. You’re able to see granular cookie data for every URL. You can also highlight multiple URLs at a time and view them together (the ‘address’ column on the right shows which URL each is on).
The columns listed in the Cookies tab include:
- Cookie Name – The name of the cookie.
- Cookie Value – The cookie value.
- Domain – The domain that issued the cookie. These can be either first or third party.
- Expiration Time – The cookie expiry time.
- Secure – Details of the cookie secure attribute. True means the ‘secure’ attribute is present.
- HttpOnly – Details of the cookie HttpOnly attribute. True means the ‘HttpOnly’ attribute is present.
- Address – The URL the cookie was set on.
8) View Aggregated Cookie Summary Report
Export an aggregated summary of cookies discovered by clicking ‘Reports > Cookies > Cookie Summary’.
This shows an aggregated view of unique cookies discovered during a crawl, considering their name, domain, expiry, secure and HttpOnly values. The number of URLs each unique cookie was issued on will also be displayed. The cookie value itself is discounted in this aggregation (as they are unique!).
⭐️How To Calculate Keyword Density
Purpose: The purpose of keyword density calculation is to determine the prominence of a specific keyword or phrase within a given text or webpage. Keyword density was historically used as a metric to gauge the relevance and importance of a keyword within the content. However, it is important to note that keyword density is not as significant a factor in search engine rankings as it once was.
While keyword density is still considered by some as a basic SEO (Search Engine Optimization) factor, it is generally recommended to focus on creating high-quality, relevant content that is valuable to users, rather than obsessing over keyword density. It is important to strike a balance between using keywords naturally and ensuring that the content is readable and provides a good user experience.
⭐️Keyword Density Formula in Classic Way:
Keyword Density = ( KR / ( TW – ( KR x ( NWK – 1 ) ) ) ) x 100
KR = how many times key-phrases are repeated
NWK = number of words in key-phrases
TW = total words in the analysed text
Number of Key Phrases are Repeated:
Here the keyword “ashwagandha’s anti-inflammatory properties” has repeated for 8 times
So, Number of Key Phrases are Repeated: 8
Number Of words In Key-phrases:
Here 4 words are present in the keyword
So, Number Of words In Key-phrases: 4
Total Words In The Analysed Text:
Here the word number of the web page needs to be counted.
The word number of the blog is 1187.
So, Total Words In The Analysed Text: 1187
Calculation Of Keyword Density:
Keyword Density = ( KR / ( TW – ( KR x ( NWK – 1 ) ) ) ) x 100
KR = 8
NWK = 4
TW = 1187
So,
( 8 / ( 1187 – ( 8 x ( 4 – 1 ) ) ) ) x 100
=( 8 / ( 1187 – ( 8 x 3 ) ) ) x 100
=( 8 / ( 1187 – 24 ) ) x 100
=( 8 / 1163) x 100
=0.687876 ~ 0.7
So, the Keyword density is 0.7(approx)
The ideal score are as follows:
- If the keyword density is between 0.5 – 1.5 = Good
- If the keyword density is 1.5 – 2 = Need To reduce keyword numbers
- If the keyword density is >2 = Keyword Stuffing
Now let us compare keyword density:
The score for this blog is 0.7(approx) which is within the zone of 0.5 – 1.5.Hence the keyword density of this current blog is 0.7, so it is Good.
