Search engines

Search engines

Search engines are automated tools that send out ‘crawlers’ or ‘spiders’ to scan the web and index every website that they find by following hyperlinks. The submission process for a website is simply to provide the URL for a page and the search engine will visit the site and index the page content and HTML tags.

In contrast to directories, that are compiled manually by humans, the search engines are automated so tend to have a much greater coverage of the web and therefore provide better opportunities for a site to achieve a ranking across more search terms – although optimising a website for each search engine is crucial in order to get these results.

The following is a summary of the most popular UK search engines. They all operate differently and will use a variety of ‘algorithms’ to retrieve websites once a search has been made on them. More information is provided by links to pages on the history of the main sites and information about how to get a site listed and to achieve a high ranking on each of these main search tools used in the UK.

A late entrant in search engine terms, Google only launched in 1998 but has rapidly become the most popular search engine favoured by savvy web users due to the speed and accuracy of results served. It is now the undisputed leader in terms of search traffic and the most important search tool to achieve a top placement on, with Google results also now being used by AOL and Netscape Search, as well as portals like Wanadoo andBBCi.

Google is also one of the main search engine to use ‘link popularity’ as a key element impacting search rankings, whilst Google’s PageRank measure is a core indicator of how a site may perform in the search results, based on both internal and external factors on a website.

Back to top | A history of Google. | Google optimisation – getting a site ranked.

Microsoft’s own web portal MSN has always provided a search service to users – MSN Search – which was also set as the default search option from the Internet Explorer browser, so it tended to attract a reasonably high proportion of search traffic. However, Microsoft have been quite slow to recognise the potential of developing a search service themselves and therefore the MSN search function was provided by third parties, most notably Inktomi.

In the Spring of 2005 however, MSN finally launched its own search engine which was supported by a large advertising campaign. This new search tool is based on the Google model and provides some strong results, with a frequently updated database and a selection of advanced search functions. In the UK MSN Search is the second most popular search tool and benefits from its association with the main MSN portal and other online tools provided by Microsoft.

Back to top | A history of MSN Search. | MSN Search optimisation – getting a site ranked

Yahoo! was traditionally the largest and most important directory on the Internet and it developed into a leading online portal offering a range of services to users. Over the last few years, the Yahoo! directory has dropped its profile and as the size of the web has grown dramatically, the provision of a spidered search engine service became more important, with the introduction of Yahoo! Search.

The search engine service on Yahoo! was initially provided as a supplement to the directory and results were sourced from third parties – initially from Inktomi until this was replaced by Google as the primary search supplier in 2003. However, in 2004 Yahoo! went through a significant acquisition process, buying Overture (along withAltaVista and AlltheWeb which had previously just been taken over by Overture) and then Inktomi. Yahoo! Search has now established a stronger search offering by using the Inktomi search engine and developing this with technology from its other search companies, so that the reach of Yahoo! has grown, to hold the third largest share of the UK search market.

Back to top | A history of Yahoo! Search. | Yahoo! Search optimisation

Ask was formerly the Ask Jeeves search engine. Following a rebrand in February 2006, Jeeves ‘the butler’ was finally dropped and the name of this search engine was changed as the focus was revised to try to gain more market share. The original Ask Jeeves always tried to set itself apart as a different kind of search engine – the search tool for the masses, or those that aren’t very confident on using web search tools. Initially Ask Jeeves promoted itself as providing results to questions posed in plain English, so that users could type in a general question and receive relevant results. The information presented came from numerous sources – part search engine results, part directory, part paid links.

Although the new look Ask search engine still holds quite a small share of the search market, it remains a reasonably significant player and the company has grown through acquisition. Most notably, the purchase of Teoma – which was touted as the biggest challenge to Google when it launched in 2002 – has helped Ask to develop its own search engine and to try to compete in the mainstream search market by offering a range of powerful search tools.

Back to top | A history of Ask. | Ask optimisation – getting a site indexed and ranked.

Developed in Norway and launched in Spring 1999, AlltheWeb/FAST remained a relatively unknown search engine in the UK despite it providing results to the Lycos network. It was a well respected search index and retrieval system, offering a powerful range of search options and an accuracy that was considered second only to Google. AlltheWeb was acquired by Overture in 2003 and soon after by Yahoo!, so that this search engine is simply a satellite of Yahoo!, providing the same search engine results.

Back to top | A brief history of AlltheWeb / FAST.

Since its launch at the end of 1995, AltaVista became one of the largest and most popular search engines, providing a good collection of search tools and advanced search functions. However, it suffered financial and operational setbacks during 2001, consequently losing a large number of users to the ascendant Google. It relaunched its interface – again – and provided new search functionality at the end of 2002 but was acquired by Overture and then Yahoo! – suffering the same fate as AlltheWeb. Once the leading search engine on the web, AltaVista is now simply a replica of the Yahoo! search engine.

Back to top | A brief history of AltaVista.

Direct Hit was a unique search tool that measured ‘click-throughs’ and the time spent on websites to improve the rankings on its search results. It worked closely with key partners, such as Ask Jeeves and HotBot, to refine and develop these results. Owned by Ask Jeeves, this search tool was superceded by Teoma towards the end of 2002.

Back to top | A brief history of Direct Hit.

One of the original search engines, Excite became a popular general search tool and portal-based service. It acquired two competitors – Magellan and Webcrawler – in 1996, but sadly succumbed to financial losses at the end of 2001 by closing its own search database and many regional search sites. Excite.com currently provides metasearch results, including from Overture and Inktomi.

Back to top | A brief history of Excite.

Launched in Spring 1996, HotBot is now owned by Lycos and is a large index with a good range of advanced searching features. Having taken its search results from the Inktomi and Direct Hit databases for some years, it was another search tool to relaunch at the end of 2002 with a new, cleaner look. It now offers a more powerful search functionality with multiple search options allowing users to search through the databases from Google, FAST, Inktomi or Teoma.

Back to top | A brief history of HotBot.

Inktomi did not provide a search engine interface in its own right, but it spidered the web and provided an index to many key partners such as HotBot and MSN. During 2002 it lost a significant share of the search market by key partners such as AOL and Netscape moving to Google results. At the end of the year Inktomi was acquired by Yahoo! and the search technology has now been absorbed and rebranded as Yahoo! Search.

Back to top | A brief history of Inktomi.

One of the original search engines, Lycos still spiders the web but now places more emphasis on search results from FAST and on its directory format, taking data from the Open Directory Project (see the top Directories). Investment in this search tool has reduced its usage over recent years and more prominence is now given to its portal services.

Back to top | A brief history of Lycos.

Teoma is a relatively new search engine, founded in 2000 by a team from Rutgers University in the US. Using a highly regarded combination of criteria to rank relevant websites, this was seen as a potential new competitor to the dominance established by Google. During 2002Teoma was purchased by Ask Jeeves and used as the main search results on this tool. When Ask Jeeves relaunched in 2006 as just Ask, the Teoma search engine was fully absorbed into this new search engine and no longer exists as a standalone search tool.

Back to top | A brief history of Teoma.

Other types of specialised search engines

As the number of websites and pages of content continues to grow substantially, even the main search engines are finding it difficult to index everything and provide the most relevant or accurate results. We have provided a forecast of how search engines of the future may develop, but one area that is becoming more important in a search engine marketing campaign is to target more specialised search engines, such as:

• Shopping search engines – there are a number of price comparison search tools that can provide ecommerce websites with another channel to present their product details and attract traffic through to their site.
• Local search engines – regional search traffic is vital for many businesses that just want to attract traffic from a local area. The growth in the number and quality of local search tools now provide an excellent resource to reach this market.
• Image search engines – searching for pictures has been a function provided by the main search engines for some time and some companies can easily take advantage of this search option by optimising their sites to rank well within the image results.