Indexing a website on Google can seem like a pretty passive task. Just write content, optimize it for search engines, upload it to your site and wait for the magic to happen, right?
Not really. It requires more than just clicking the publish button. In order for your site to appear in search results, Google must “crawl” and “index” your content. This happens automatically in a few days or weeks, but if you are the site owner, you can manually submit your site to Google and speed up the process.
There are two ways to do this. But first, let’s take a brief look at how Google crawls and indexes your site’s content.
How Google Finds Your Content ?
Google says it uses a large number of computers to crawl billions of pages on the web. Known as Googlebot, this crawler basically starts with a list of web page URLs generated by previous crawlers, and then supplements those pages with sitemap data available on Google Search Console. During the crawling process, Googlebot (also known as a “spider”) looks for new sites, updates to existing pages, and broken links.
When you have new pages in your sitemap, Google discovers them, crawls the content, and may list the page in search results based on evaluating over 200 criteria.
Once the crawling process is complete, all results will be entered into the Google index, and any new sites or updated content will be listed accordingly. When processing results, Google looks at information on your page, such as title tags, meta descriptions, alt tags, and more. Note that if your page has good content, Googlebot may not be able to read it and crawl the default version. It is recommended that your default version be optimized for search.
Due to Google’s crawling, you may never have to submit your site, as it is automatically recognized. The downside to this approach has always been that it depends on the time frame Google crawls and indexes your site’s content, which may not be as fast as you’d like.
Would you like to check if a website is listed on Google? Just start your search with “site:sitename.com”. For example, Google shows the following for HubSpot.com:
If the site has had no content indexed yet, Google will tell you that your search returned no results.
If nothing is found, the next step should be to create a sitemap that can be submitted to Google.
If you want to learn more about how to create a sitemap, check out this article.
How long does it take for Google to index your content?
When publishing content without manually submitting an updated sitemap, Henry found that Google took 1,375 minutes to crawl and Yahoo took 1,773 minutes. In terms of these numbers, that’s about a full day to scrape your content.
If you’re launching a new website or adding a bunch of new pages to an existing domain, it might be worth submitting an updated sitemap. According to the same study, Henry found that after submitting an updated sitemap, bots took an average of 14 minutes to visit a page, compared to 245 minutes for Yahoo!. Your new page can generate organic traffic and conversions on the same day.
How do I submit my website to Google?
To submit your site to Google, you can either add an updated sitemap to your Google account, or submit a request for indexing of the desired URL via fetch as Google does. Note that both of these processes require site owners to register with Google Search Console.
Here are the details for each option:
If you have a brand-new website…
The first thing you should do when you launch your website for the first time is to verify that you own the domain name in Google Search Console. Then submit here – go to this page and select the “Submit Sitemap” option.
If you have an existing site and are starting a new page…
With an already activated web domain, you can still submit new pages to Google for indexing and ranking accordingly. Anyone used to be able to do this with a page to crawl, whether they owned it or not. Now, just like you start a brand-new website, you need to be the owner of the URL to ask Google to recrawl it.
If you want Google to recrawl a page, you have several options:
First, you can submit an updated sitemap to make sure it gets listed as soon as possible. To submit an updated Sitemap, log into Google Search Console and select “Add Attribute.”
There, you can submit your updated sitemap to Google for crawling as soon as possible.
Fetch as Google :
Fetch as Google allows you to display web pages on your website the way Google sees them. As a website owner, you can also use this tool to reindex individual URLs after they have been obtained. To do so, log into Google Search Console and select your current listing on Google.
From the left sidebar, select Crawl > Fetch as Google, as shown below:
This will open a form where you can enter the URL path after your domain name and “fetch” a specific page on your site.
And as you can see, two sample fetch requests are listed below the fetch column. Also, the status of these fetches is partial, but once these fetches are complete, Google will evaluate them for eligibility for re-indexing. If the URL you get is eligible, you will see a “Request Indexing” option with the red box in the screenshot above. Select this option and Google will start crawling and re-indexing the page again.
As mentioned earlier in this article, indexing requests can take anywhere from a day to a few weeks. Therefore, periodically check your crawl against a Google spreadsheet to see the status of your active indexing requests.
Based on the above instructions, you may be wondering if you need to submit an updated sitemap or indexing request every time a new page is published. So, if you’re updating extremely important content that you want Google to recognize quickly, you can of course do so. You need to keep in mind Google also re-crawls pages on its own, and it’s ok to let this process work in the background as you regularly create and update your content.