🦸‍♂️
KT's DevOps Notes
  • Hello World
  • Misc
    • Ubuntu Setup
    • Mac Os Setup
  • Database
    • Redis
      • How To Migrate Redis Data to a DigitalOcean Managed Database
      • How To Connect to a Managed Redis Instance over TLS with Stunnel and redis-cli
    • MariaDB
      • How to Setup MariaDB Galera Cluster on Ubuntu 18.04 with HAProxy
      • How to change innodb_thread_concurrency Read/Wirte Concurrency in MySQL
      • How to change innodb_write_io_threads in MySQL
      • How to change innodb_read_io_threads in MySQL
  • OCI Foundations 2020 Associate Certification | 1Z0–1085–20 | Preparation…
  • Security
  • Ubuntu
    • Apache Server Configuration
      • Configuring Let’s Encrypt SSL Cert for Apache on Ubuntu 18.04
      • Installing Apache, MySQL, PHP (LAMP) Stack on Ubuntu 18.04
      • Installing Apache on Ubuntu 18.04 with Multiple Domains
    • Nginx Server Configuration
      • Robots.txt disallow all with nginx
      • Installing Nginx, MySQL, PHP (LEMP) Stack on Ubuntu 18.04
      • Installing Nginx on Ubuntu 18.04 with Multiple Domains
      • Configuring Let’s Encrypt SSL Cert for Nginx on Ubuntu 18.04
    • Creating Swap Space on Ubuntu 18.04
    • Dell Inspiron 7559 Ubuntu Linux Guide
    • How to configure Postfix to use Gmail SMTP on Ubuntu 18.04 & 16.04 | LeetDev
  • Setting up SSL Certificates for HAProxy with certbot
  • PHP
  • PHP / Apache: Set Max File Upload and Post Size | LeetDev
  • Upgrade centos7 openssh & openssl to the latest version
  • Downgrading to php7.3 on Amazon Linux 2
Powered by GitBook
On this page

Was this helpful?

  1. Ubuntu
  2. Nginx Server Configuration

Robots.txt disallow all with nginx

PreviousNginx Server ConfigurationNextInstalling Nginx, MySQL, PHP (LEMP) Stack on Ubuntu 18.04

Last updated 5 years ago

Was this helpful?

If you’re managing an environment similar to a production and want to keep bots from indexing traffic, it’s customary to add a robots.txt file at the root of your website to . Instead of creating a two-line plain text file, you can do this with only nginx:

location = /robots.txt {
  add_header  Content-Type  text/plain;
  return 200 "User-agent: *\nDisallow: /\n";
}

Add this into your configuration management as determined by environment, or add it by hand, and no longer worry if Google might start broadcasting your dev site to the world.

disallow all