For websites that work on the CMS with embedded systems to restrict access
Please note that the indexing process, the scanner sends a large number of requests to your website. Some of the CMS, in particular, is a popular product of 1C-Bitrix, with appropriate settings, can block our crawler requests for reasons of safety or load on a Web server. Recommend that you remove such protection at the time indexing of your site. If such an opportunity you do not - use a minimum speed of indexing, which will minimize the load on your server.
Selecting the optimum speed of indexing and the level of burden on your web server
In the options of the scanner are three levels of speed indexing, creating appropriate levels of load on the servers to be indexed:
Indexing hidden pages (Deep Web)
Deep Web – Web pages indexable by search engines due to the lack of hyperlinks on them with the available pages. For example - this page was generated through HTML-forms interface or the contents of frames.
If you want to find and include these pages in a Sitemap, check the options: