{"id":2715,"date":"2018-08-23T09:37:19","date_gmt":"2018-08-23T09:37:19","guid":{"rendered":"https:\/\/www.searchviu.com\/?p=2715"},"modified":"2023-02-09T13:58:07","modified_gmt":"2023-02-09T13:58:07","slug":"robots-txt-staging-environment","status":"publish","type":"post","link":"https:\/\/www.searchviu.com\/en\/robots-txt-staging-environment\/","title":{"rendered":"Why you shouldn’t block your entire staging environment via robots.txt"},"content":{"rendered":"

It’s a common practice to block entire staging environments via robots.txt – We see roughly 50% of our website migration clients do it. Here’s why it’s not a good idea.<\/strong><\/p>\n

Webmasters, when preparing a website migration, often decide to add a robots.txt file with the following content to the staging environment of the new website:<\/p>\n

User-agent: *
\nDisallow: \/<\/code><\/p>\n

These instructions ask robots to stay away from all pages on the site. The idea behind such an implementation is to prevent the content of the staging environment from being indexed by search engines.<\/p>\n

Use the “real” robots.txt file<\/h2>\n

The short and simple robots.txt file we saw above obviously isn’t the one that will be needed once the new website goes live. The “real” robots.txt will be a bit more complex. It’s a good idea to add this final robots.txt to the staging environment already, for the following reasons:<\/p>\n