All search engine crawlers search the robots file first when they visit the site. Errors in this file almost always lead to poor indexing.

Create and configure the correct robots txt and sitemap for your site, which will allow your site to be indexed even faster, and forbidden pages for indexing will be hidden from search robots.

A robots file. txt is a text file located in the root directory of the site, in which special instructions for search robots are written. These instructions may prohibit some sections or pages on the site from being indexed, indicate proper "mirroring" of the domain, recommend the search robot to observe a certain time interval between downloading documents from the server, etc.