Site Structure and Search Engine Crawlers

An important aspect of search engine optimisation (SEO) is website structure. If your site is to be found and indexed by search engines, it needs to be structured in a manner which makes it easy to navigate.

Search engines like Google index web pages using ‘crawlers‘.  These crawlers search for, find and return information about web pages to the search engines, allowing the search engine to add the site to its index. This site can then be ranked and can be returned to search engine users. You can also submit your site URL directly to them to speed up the process. These links will take you to the submission form for three of the biggest search engines:

Web crawlers

Crawlers are often known as ‘spiders’ or ‘ants’, and this can be a good way to imagine them if you want to picture how they work. The crawlers head out into the web, where they crawl sites, navigating their way link by link and collecting information about the pages they find. In actuality, the crawlers do not go anywhere: they are static computers which search for information on the web in much the same way we do, but on a much larger scale, finding pages far quicker than we could and carrying out many searches at the same time.

Navigating your website

In order for the crawlers to properly navigate your site, it must be structured in a way which allows all the pages to be reached. If the crawler is unable to reach a page or gets stuck, it may not be able to add the content from that page to the search engines. As crawlers read text rather than images, there must be text links instead of, or in addition to, link button graphics. The home page should link to the site’s internal pages, which in turn should link to each other. Ideally, all pages should not be more than three clicks away from the home page, and where possible sites should consist of just one or two navigation levels. By following these structure rules, the crawlers should be able to easily find their way around your site and its content.

Customer navigation

It is also important to note that if the crawlers struggle to navigate your site, the chances are that your customers will find themselves with the same problem. For effective search engine optimisation and a great customer experience, ensure your site is laid out in an easy-to-follow manner.