6 Effective Steps; To make Google Crawl My site ( Instant Indexing technique)

Published by Mrdecode on

How I Make Google Crawl My site

"how to make google crawl my site faster"

Do you want Google to index your website fast? 

As a content creator, my answer would be yes. Who does not want to show up their site in search results? Every content creator wants that.  

In this Step-by-step blog post, I will show you how to make Google crawl your website.

See, everyone wants traffic from Google (organic traffic). But do you know, if a page is not indexed by Google, there’s a 0% chance that it will receive organic traffic? Therefore, to make google index your site, you will have to submit a sitemap of your website in google search console.

So, to make these things simpler, I will explain to you what Google search console is. How it works, How to submit a sitemap and how to check if a site is indexed or not.

So Let’s dive directly into the tutorial;

Step-by-step Guide to make Google crawl my website

In this tutorial, I will show you how I make Google to crawl my website. But before we start let’s take a step back and look at what indexation actually is, and why it plays such a major role in getting a spot in search results.

What is Web Index?

In its basic form, an index is a list of information. Long before the internet, indexes existed in many forms, like the alphabetical indexes at the end of books.

A web index, then, is a database of information on the internet.

Search engines use these databases to store billions of pages of information. So when you use a search engine, you aren’t actually searching for everything that exists on the internet.

You’re searching that search engine’s index of stored pages and information.

Every time someone publishes content. Google’s bots or spiders crawl that new content and store them in an index based on their topic, relevance, authority, and more. So when people search for your keyword, google bots will show your indexed pages on the search results.

Why do you need to index your site?

First, the obvious answer.

If you want your site to show up in the search results at all, then it needs to be indexed.

However, you don’t want your site to be indexed just once. You want the search engines to keep re-indexing your site.

Search engines like Google don’t just update automatically.

They rely on spiders — little bits of computer code that each search engine sends out to “crawl” the web (hence, “spider”). You want an efficient, frequent crawl rate.

"Google Spider, Crawl your site"

The spider’s job is to look for new stuff on the web and update the already indexed version of your site. That “new stuff” can be a new page on an existing site, a change to an existing page, or an entirely new site or blog.

Once the spider finds a new site or page, it needs to figure out what that new site or page is about. It will then store the crawled pages and serve the indexed pages to million of google’s audience.

Therefore, it is very important to get your site indexed on google.

If Google doesn’t index your website, then you’re pretty much invisible. You won’t show up for any search queries, and you won’t get any organic traffic whatsoever

Here, are few key terms you should know;

  • Crawling: The process of following hyperlinks on the web to discover new content.
  • Indexing: The process of storing every web page in a vast database.
  • Web spider: A piece of software designed to carry out the crawling process at scale.
  • Googlebot: Google’s web spider.

Indexing is showing up for the race; ranking is winning.

You can’t win without showing up for the race in the first place.

1.Step; How to check if my site is indexed already?

Unless you’re starting a brand new site, your website is probably already indexed.

If you’re not sure, here’s how to find out.

Go to Google, then search for site:yourwebsite.com

" number of pages indexed on google"

This number shows roughly how many of your pages Google has indexed.

If you want to check the index status of a specific URL, use the same site:yourwebsite.com/web-page-slug operator.

If your site is already indexed, that’s great, but there is likely room for improvement. The rest of the steps in this guide will help you make sure that your site is indexed to its full potential.

No results will show up if the page isn’t indexed.

Now, it’s worth noting that if you’re a Google Search Console user, you can use the Coverage report to get a more accurate insight into the index status of your website. Just go to:

Google Search Console > Index > Coverage

2. Step; Install and setup google search console

If you aren’t familiar with GWT or Google Search Console, let’s head back to square one. Google Search Console is a free service that lets you learn a great deal of information about your website and the people who visit it. 

"google search console"

You can use it to find out things like how many people are visiting your site and how they are finding it, whether more people are visiting your site on a mobile device or desktop computer, and which pages on your site are the most popular. It can also help you find and fix website errors, submit a sitemap, and create and check a robots.txt file.

Sign Up for google search console

Click here to go to Google Search Console. Login with your Google account.

You’ll see this screen:

"Add property, google search console"

Type in your website URL.

You’ll then need to verify that you own that domain. There are a few ways to do this.

"verify ownership google search console"

By default, it may show you a verification option through your web host.

1.Sign In to Search console through your web host

Click on the dropdown to find your domain name provider.

"Domain name provider method for site verification on google search console"

If yours isn’t on the list, you can press Other (at the bottom).

Search Console then asks you to create a TXT record, which involves an edit to your domain configuration.

This step will vary, based on your DNS provider. In a nutshell, what you want to do is add the TXT record to your DNS server (being sure to choose TXT as the record type), and verify its addition.

If you still want to add a TXT record though, here’s how.

2. Verify your site ownership by uploading HTML file to your website

This verification method is best for users and site owners who have experience working with HTML code. This step requires direct access to your website’s server, which you can obtain via File Transfer Protocol (FTP) or cPanel File Manager.

Using an FTP client such as FileZilla, enter your credentials and log into your server. You’ll then need to upload the file to your site’s root directory (which contains the wp-content folder, the wp-config.php file, etc.):

Click and drag the file from wherever you saved it on your computer to your site’s root folder.

Now, you can return to the Google Search Console and click on Verify. This will complete the process!

3. Search console sign in using Google Analytics

If you already have the Google Analytics asynchronous tracking code installed at your site, you can easily verify your site using Google Analytics.

There’s no detailed step-by-step for this one. Just:

  1. Select the Google Analytics radio button
  2. Click Verify

4. Search console site verification using google tag manager

Just like Google Analytics, if you’re already using Google Tag Manager, you can verify your site with just a single click as long as you have the Google Tag Manager container snippet active on your site.

Again, there’s no detailed step-by-step necessary. Just:

  1. Go to the Alternate methods tab
  2. Select the Google Tag Manager radio button
  3. Click Verify

But if you are not familiar with such site configuration, or if you don’t like the manual methods. Then there is one WordPress plugin that is easier to use; Yoast SEO.

Yoast SEO Plugin to help with google site verification

If you’re using a WordPress SEO plugin, there’s a good chance that your plugin of choice offers an easy tool to help with Google site verification. We’ll show you how to do it with the most popular WordPress plugins – Yoast SEO.

5. (Yoast SEO) Search Console Verification Using Meta Tags or HTML tag

To complete the Google Search Console verification process with Yoast SEO, go to SEO → General in your WordPress dashboard and select the Webmaster Tools tab.

All you need to do is paste your Google verification code into the box:

To find your Google verification code:

  1. Go to the Alternate methods tab in the Google Search Console interface
  2. Select the HTML tag option
  3. You can copy the entire tag, the Yoast SEO plugin will automatically strip out the extra, leaving just the code.

Make sure to click Save Changes in the Yoast SEO interface when you’re done.

Hurray! Now that our site ownership is verified on search console. Let’s see how to submit a sitemap to let Google crawl our site faster.

How to make Google crawl my site ( 6 Effective steps)

Here, I will write down 6 effective steps that will make Google crawl your site faster. 

1. Submit your sitemap to Google Search Console

Sign in to Google search console

In the sidebar, select your website

Click on ‘Sitemaps‘.

The ‘Sitemaps’ menu is under the ‘Index’ section. If you do not see ‘Sitemaps’, click on ‘Index’ to expand the section.

Remove outdated or invalid sitemaps (if any) like sitemap.xml

Enter ‘sitemap_index.xml’ in the ‘Add a new sitemap’ field to complete the sitemap URL.

Note: A domain property will not show the domain for the field. Please use the full sitemap URL.

Click Submit.

Note; type https://www.yoursite.com/sitemap.xml In your browser to check your sitemap.

2. Use URL Inspection Tool

Google Search Console has a feature allowing you to input a URL for Google to “Fetch”. After submission, Googlebot will visit your page and index.

Here’s how to do it…

  • Log into Google Search Console
  • Navigate to “URL Inspection”
  • Take the URL you’d like indexed and paste it into the search bar
  • Click enter
  • After Google had found the URL, click request Indexing

Assuming the page is indexable, it will be picked up within a few hours.

Search engines crawl from page to page through HTML links.

We can use authority pages on your site to push equity to others. I like to use Ahrefs “best pages by links” report.

This report tells me the most authoritative pages on my site – I can simply add an internal link from here to a page that needs equity.

It’s important to note, the 2 interlinking pages need to be relevant – it’s not a good idea to link unrelated pages together.

4. Hight Traffic site make Google crawl my site faster

Sites like Reddit and Quora are popular sites that allow you to drop links. I make it a regular practice to promote recently published pages on Quora – it helps with indexation, but also can drive a ton of traffic.

5. Check that your pages aren’t orphaned

Orphan pages are those without internal links pointing to them.

Because Google discovers new content by crawling the web, they’re unable to discover orphan pages through that process. Website visitors won’t be able to find them either.

To check for orphan pages, crawl your site with Ahrefs’ Site Audit. Next, check the Incoming links report for “Orphan page (has no incoming internal links)” errors:

This shows all pages that are both indexable and present in your sitemap, yet have no internal links pointing to them.

IMPORTANT

This process only works when two things are true:

  • All the pages you want indexing are in your sitemaps
  • You checked the box to use the pages in your sitemaps as starting points for the crawl when setting up the project in Ahrefs’ Site Audit.

Not confident that all the pages you want to be indexed are in your sitemap? Try this:

  • Download a full list of pages on your site (via your CMS)
  • Crawl your website (using a tool like Ahrefs’ Site Audit)
  • Cross-reference the two lists of URLs

Any URLs not found during the crawl are orphan pages.

You can fix orphan pages in one of two ways:

  • If the page is unimportant, delete it and remove from your sitemap.
  • If the page is important, incorporate it into the internal link structure of your website.

6. Share your site; Google crawl social media sites regularly

One of the best ways to to get indexed is to submit or share your content on social media sites like. Facebook, Quora, Twitter, Reddit and Pinterest. Due to the massive amount of valuable information being shared on this social platform. 

Google bots work regularly on these sites to crawl the important link, data, entries, media etc., to keep search engine up to date. 

This way when you submit your webpage on a social platform, the spiders crawling that platform will index your pages faster.

Learn More;

If you liked this post, please make sure to share, subscribe and Leave a Comment.


0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *