Today’s post is a quick tip that will help you keep your robots.txt after you update the site’s Drupal core with drush.
Most likely, you know the problem, too: each time you update the Drupal core (with drush, first of all), you lose all changes you may have introduced to robots.txt manually. A good idea is to always have a backup of this file handy, but every now and then you only have an outdated version of robots.txt and you learn that only when you check the updated site with webmaster tools.
A good solution for the problem is the RobotsTxt module, which works well with three latest releases of Drupal. When you have the module up and running, you can edit robots.txt in the backend and check it at a corresponding address. However, there is a minor but very annoying thing about the module: it does not want to work properly if you have your robots.txt stored at the site’s root folder. Coupled with admin’s forgetfulness, this leads to quite the same situation as with an outdated robots.txt.
After having to clean up the mess a couple dozen times, we decided to make a simple yet efficient solution for the persisting problem. Step by step, the solution transformed into the Custom robots.txt module by Alexey Zubenko.
The module draws a clear line between default and custom robots.txt settings. You can add anything to this file via the backend and it will go to the end of your robots.txt. There is also a cron routine to check for changes made to robots.txt by the admin, which means that even if you have lost your changes after the core update the module will add them back to robots.txt after the first scheduled check up while keeping the new default entries intact.
The module takes default robots.txt contents from cgit.drupalcode.org.
We made the Custom robots.txt module for Drupal 7, but if the idea proves worthy, we may code one for Drupal 8.