This is optional. If this option is ticked, generator will follow the Allow and Disallow instructions in general section of User-agent: *
«Personal» sections of User-agent: Google or User-agent: Yandex are considered when choosing the appropriate crawler identification type as search robot.
In addition, you may create section User-agent: Mysitemapgenerator, which will be taken into account when choosing the direct identification of our robots (Mysitemapgenerator-HTTP or Mysitemapgenerator-Mobile - for Sitemap Mobile).
Below is an example of a robots.txt file:
#No robots should visit any URL starting with /noindex-directory/ User-agent: * Disallow: /noindex-directory/ #Google does not need to visit a specific URL User-agent: Googlebot Disallow: /noindex-directory/disallow-google.html #Yandex does not need to visit URL starting with /noindex-directory/ #But allows to index a specific page User-agent: Yandex Disallow: /noindex-directory/ Allow: /noindex-directory/allow-yandex.html #Mysitemapgenerator does not need to visit URL starting with /noindex-directory/ #But allows to index pages with a specific extension User-agent: Mysitemapgenerator Disallow: /noindex-directory/ Allow: /noindex-directory/*.html