Robots.txt instruct search engine how to crawl pages on their website, Robots.txt files inform search engine crawlers how to interact with indexing your content.search engines tend to index as much high-quality information as they can & will assume that they can crawl everything unless you tell them otherwise.
Robots.txt can help prevent the appearance of duplicate content. Sometimes your website might purposefully need more than one copy of a piece of content.
Robots.txt follow pre-defined protocols:
User-agent: A means of identifying a specific crawler or set of crawlers.
Allow: All content may be crawled.
Disallow: No content may be crawled.
Create a Robots.txt File in WordPress:
Using Yoast SEO
A robots.txt file can be a powerful tool in any SEO’s usage as it’s a great way to control how search engine crawlers/bots access certain areas of your site. Keep in mind that you need to be sure you understand how the robots.txt file works or you will find yourself accidentally disallowing Googlebot or any other bot from crawling your entire site and not having it be found in the search results!