XML Generator Sitemap Generator
Рус Eng
Support Ticketing System
  • Support Ticketing System
Mon—Fri, 11:00—19:00 UTC+3:00
Indexing Cyrillic Domains

Will be the indexing limits, specified in robots.txt, taken into account during indexing?

This is optional. If this option is ticked, generator will follow the Allow and Disallow instructions in general section of User-agent: *
«Personal» sections of User-agent: Google or User-agent: Yandex are considered when choosing the appropriate crawler identification type as search robot.
In addition, you may create section User-agent: Mysitemapgenerator, which will be taken into account when choosing the direct identification of our robots (Mysitemapgenerator-HTTP or Mysitemapgenerator-Mobile - for Sitemap Mobile).
Below is an example of a robots.txt file:

    #No robots should visit any URL starting with /noindex-directory/
    User-agent: *
    Disallow: /noindex-directory/
    #Google does not need to visit a specific URL
    User-agent: Googlebot
    Disallow: /noindex-directory/disallow-google.html
    #Yandex does not need to visit URL starting with /noindex-directory/
    #But allows to index a specific page
    User-agent: Yandex
    Disallow: /noindex-directory/
    Allow: /noindex-directory/allow-yandex.html
    #Mysitemapgenerator does not need to visit URL starting with /noindex-directory/
    #But allows to index pages with a specific extension
    User-agent: Mysitemapgenerator
    Disallow: /noindex-directory/
    Allow: /noindex-directory/*.html

Отзывы пользователей генератора SitemapReviews All Reviews →
Add +
  • Ян, atleticomadrid.ru

    Спасибо сервису за карту для поисковиков. Очень удобно. Очень рад.

  • default, www.mebelinsochi.ru

    Пользуюсь сервисом не первый раз, очень удобно!! Спасибо!


Follow Us!

How can we help you? Open a Support Ticket Share Service!

About Us

©Sitemap Generator - Online Generator
VisaMasterCard Secure