Technical documentation - Robots.txt
If SEO settings or SiteMap genaration is enabled, web viewer will automatically create a rotobs.txt.
Robots.txt generation is available in version 5.1.4 and later.
Ensure to have site map and SEO settings disabled on test sites, not to have test sites indexed. It is not good for getting high rank and may create unwanted traffic to test sites.
Overview
Robots.txt contains instructions to robots how to index the site. The robots.txt file will always be generated. Depending on if SEO settings or SiteMap is configured, sites on the server will get a reference to the location of the site map or disallowed.
Robots.txt will look something like this, here with one disallowed site and one sitemap reference.
User-agent: *
Disallow: /*/SampleSite/*
Sitemap: http://www.domain.com/Sitemap