One of our consumers recently enquired about how to prevent crawling and indexing of their WordPress website by search engines. You may wish to prevent search engines from indexing your website or from displaying it in search results in a variety of situations. We’ll teach you how to prevent search engines from crawling a WordPress site in this post.
Who Would Want to Stop Search Engines and Why?
The majority of websites get the majority of their visitors from search engines. Why would anyone wish to ban search engines, you may wonder.
Many folks don’t know how to set up a local development environment or a staging site when they first start out. You probably don’t want Google to index your in-progress or maintenance mode page if you’re constructing your website live on a publicly accessible domain name.
Additionally, a lot of individuals use WordPress to build private blogs, and since such sites are private, they don’t want them to appear in search results.
Additionally, some individuals use WordPress as an intranet or project management tool, and you wouldn’t want your internal papers to be visible to the general public.
You probably don’t want search engines to index your website under any of the aforementioned circumstances.
It’s a prevalent myth that search engines won’t ever locate my website if there are no links going to it. This is not entirely accurate.
Search engines can locate a website that is connected elsewhere in a variety of ways. For instance:
- It’s possible that another person once had your domain name and that person still has some links pointing to your current website.
- Your link can appear in the results of some domain search engines.
- Your website might be listed on one of the literally hundreds of pages that contain a list of domain names.
On the internet, a lot of things happen that are mostly out of your control. You still have control over your website, so you may tell search engines not to index or follow it.
How to Stop Search Engines from Indexing and Crawling Your WordPress Site
A built-in feature of WordPress enables you to tell search engines not to index your website. The only thing you have to do is go to Settings » Reading and choose the option for Search Engine Visibility.
When this option is ticked, WordPress adds the following line to the header of your website:
|
1
|
<meta name='robots' content='noindex,follow' /> |
Additionally, WordPress edits the robots.txt file on your website, adding the following lines:
|
1
2
|
User-agent: *Disallow: / |
These words request that web crawlers (robots) not index your pages. Search engines have complete discretion over whether to comply with this request or not. Despite the fact that the majority of search engines follow this, a page or random picture from your website could get crawled.
How Can You Be Certain Your Website Won’t Show Up in Search Results?
WordPress’s default visibility settings tell crawlers not to index your website. But even so, a page, file, or image from your website can end up being crawled and indexed by search engines.
By password-protecting your whole WordPress site on the server level, you can ensure that search engines do not index or crawl your website at all.
This indicates that before reaching WordPress, visitors to your website are prompted for a login and password. Search engines are also included in this. They get a 401 error after a failed login attempt, and the bots leave. Here’s how to secure the entire WordPress site using a password.
Method 1: Using cPanel to password-protect the whole site
You may secure your entire site using cPanel if your WordPress hosting company provides access to it for managing your hosting account. Simply log in to your cPanel dashboard, go to the Security area, and select the ‘Password Protect Directories option.
Choosing the folder where WordPress is installed is the next step. Usually, the public html folder is where it is.
If your public html directory contains several WordPress sites, you must click the folder icon to browse and choose the folder containing the website you wish to password-protect.
You must enter a name for the protected directory and press the “Save” button on the following screen.
After saving your data, cPanel will refresh the page. The next step is to add a verified user by providing a username and password.
That’s all there is to adding password security to your WordPress website.
The login and password you generated previously will now be required every time a user or search engine accesses your website.
Method 2: Using a Plugin to Password Protect WordPress
You may occasionally not have access to cPanel if you are utilizing a managed WordPress hosting service.
In such a situation, you can choose to password-protect your website using a variety of WordPress plugins. The top two options are shown below:
The most popular coming soon and maintenance mode plugin for WordPress, SeedProd, is installed on more than 800,000 websites. You may utilize its comprehensive access control and permissions tools to keep your website hidden from search engine as well as from everyone else.
With only one password, you can easily password-protect your WordPress site with the Password Protected plugin (no user creation needed).
We wish you luck in your efforts to prevent search engines from indexing or crawling your WordPress website.
Comments are closed.