Matt Cutts Teaches Us To Crawl

Cutts' latest blog post reviewed Bigdaddy's crawl-caching proxy in greater depth. He even provided helpful charts to illustrate the process.

As a webmaster, one may see numerous fetches from multiple Googlebots, each of them using some bandwidth while accomplishing their appointed rounds. It makes for a more accurate Google index, but the site impact has given some webmasters fits over the bandwidth usage.

The proxy used in the Bigdaddy infrastructure works like other proxies. It handles the effort of retrieving pages from websites, and fulfills requests from the various Google crawlers. Instead of multiple spiders hitting a website, they hit the cache instead.

Cutts breaks down the crawl caching in a summary during his post (spacing added; we like Matt, but we'd really like him to enjoy the Return key a bit more often :) :

So the crawl caching proxy work like this: if service X fetches a page, and then later service Y would have fetched the exact same page, Google will sometimes use the page from the caching proxy.

Joining service X (AdSense, blogsearch, News crawl, any Google service that uses a bot) doesn't queue up pages to be include in our main web index. Also, note that robots.txt rules still apply to each crawl service appropriately. If service X was allowed to fetch a page, but a robots.txt file prevents service Y from fetching the page, service Y wouldn't get the page from the caching proxy.

Finally, note that the crawl caching proxy is not the same thing as the cached page that you see when clicking on the "Cached" link by web results. Those cached pages are only updated when a new page is added to our index.

It's more accurate to think of the crawl caching proxy as a system that sits outside of webcrawl, and which can sometimes return pages without putting extra load on external sites.

The essential goal of the proxy, to reduce bandwidth, seems to have worked to Google's satisfaction. Cutts wrote that "it was working so smoothly that I didn't know it was live."

This article first appeared on the WebProNews website on April 24, 2006. It was written by WebProNews staff writer David A. Utter.

F-Shaped Pattern For Reading Web Content

There is more to marketing a website than great SEO. While a solid SEO strategy such as Total Optimizer Pro can help you achieve is key to your online success, so too is understanding what your visitors will do and see when they get to your site. Below you will find an interesting article by Jakob Neilson of Nielsen Norman Group.

For more information on Jakob and other inportant usabability issues visit his profile on the useit.com website at http://www.useit.com/jakob/.

F for fast. That's how users read your precious content. In a few seconds, their eyes move at amazing speeds across your website’s words in a pattern that's very different from what you learned in school.

In our new eyetracking study, we recorded how 232 users looked at thousands of Web pages. We found that users' main reading behavior was fairly consistent across many different sites and tasks. This dominant reading pattern looks somewhat like an F and has the following three components:

• Users first read in a horizontal movement, usually across the upper part of the content area. This initial element forms the F's top bar.
• Next, users move down the page a bit and then read across in a second horizontal movement that typically covers a shorter area than the previous movement. This additional element forms the F's lower bar.
• Finally, users scan the content's left side in a vertical movement. Sometimes this is a fairly slow and systematic scan that appears as a solid stripe on an eyetracking heatmap. Other times users move faster, creating a spottier heatmap. This last element forms the F's stem.

Obviously, users' scan patterns are not always comprised of exactly three parts. Sometimes users will read across a third part of the content, making the pattern look more like an E than an F. Other times they'll only read across once, making the pattern look like an inverted L (with the crossbar at the top). Generally, however, reading patterns roughly resemble an F, though the distance between the top and lower bar varies.

Heatmaps from user eyetracking studies of three websites. The areas where users looked the most are colored red; the yellow areas indicate fewer views, followed by the least-viewed blue areas. Gray areas didn't attract any fixations.

The above heatmaps show how users read three different types of Web pages:

• an article in the "about us" section of a corporate website (far left),
• a product page on an e-commerce site (center), and
• a search engine results page (SERP; far right).

If you squint and focus on the red (most-viewed) areas, all three heatmaps show the expected F pattern. Of course, there are some differences. The F viewing pattern is a rough, general shape rather than a uniform, pixel-perfect behavior.

On the e-commerce page (middle example), the second crossbar of the F is lower than usual because of the intervening product image. Users also allocated significant fixation time to a box in the upper right part of the page where the price and "add to cart" button are found.

On the SERP (right example), the second crossbar of the F is longer than the top crossbar, mainly because the second headline is longer than the first. In this case, both headlines proved equally interesting to users, though users typically read less of the second area they view on a page.

Implications of the F Pattern

The F pattern's implications for Web design are clear and show the importance of following the guidelines for writing for the Web instead of repurposing print content:

• Users won't read your text thoroughly in a word-by-word manner. Exhaustive reading is rare, especially when prospective customers are conducting their initial research to compile a shortlist of vendors. Yes, some people will read more, but most won't.
• The first two paragraphs must state the most important information. There's some hope that users will actually read this material, though they'll probably read more of the first paragraph than the second.
• Start subheads, paragraphs, and bullet points with information-carrying words that users will notice when scanning down the left side of your content in the final stem of their F-behavior. They'll read the third word on a line much less often than the first two words.

Detailed Scanning Behaviors

It's fascinating to watch the slow-motion replay of users' eye movements as they read and scan across a page. Every page has reading issues beyond the dominant F pattern I'm discussing here. For example, users scan in a different, more directed way when they're looking for prices or other numbers, and an interesting hot-potato behavior determines how users look at a list of search engine ads. We also have many findings on how people look at website images.

Searcher Behavior Research

By Chris Sherman, Executive Editor, SearchEngineWatch
April 11, 2006

As part of ongoing work conducted by Jupiter Research and sponsored by iProspect, "The iProspect Search Engine User Behavior Study" found that 62% of search engine users click on a search result within the first page of results, and a full 90% of users click on a result within the first three pages of search results.

These figures were just 48% and 81% in 2002, based on similar research iProspect did at the time.

Search marketers should take note of these findings, as they emphasize the importance of appearing on the first few pages of search results, whether in natural or sponsored listings. The message is clear: You can't simply rely on search engine optimization or search advertising if you want qualified prospects to find you. You must invest the time and resources in both types of search marketing, or risk being overlooked.

These findings, while interesting and important in themselves, raise several questions:

• Are searchers getting more sophisticated and demanding—or conversely, getting lazier and more easily satisfied?


• Have search engines improved so significantly over the past four years that most people find what they want on the first page of results?


• Apart from "quality" search results, how have user attitudes toward marketing and branding messages changed?

Partially answering the first question, results of the iProspect study suggest that at least a certain percentage of searchers are getting more sophisticated and demanding.

For example, 41% of search engine users who continue their search when they don't find satisfactory results on the first page do one of two things: Change engines or change search terms. Four years ago, just 28% did.

Even more determined are users who don't find what they're looking for at all on their first try. Fully 88% of these users change engines or change their search terms, up from 78% in 2002.

But these figures mask a somewhat paradoxical finding related to loyalty: 82% of search engine users re-launch an unsuccessful search using the same search engine used initially, adding more keywords to their query. Just 68% stayed with the same engine in 2002.

This suggests searchers are not only loyal, they're increasingly going out on the "long tail" using lengthier queries. For search marketers, this means if you're not targeting both simple keywords as well as lengthier keyword-rich phrases you're likely missing out on a significant amount of traffic that simply wasn't there a few years ago.

iProspect also concludes that with more searchers persisting with the same engine despite failed initial searches, that user loyalty has been earned. In other words, in staying with the same engine and using a different or longer query, searchers are implicitly saying that the problem is with their own search strategy, not with the search engine.

What about the branding aspect? The study found that 36% believe that companies whose websites are returned at the top of the search results are the top companies in their field. Slightly more (39%) felt neutral on this question. At the other end of the spectrum, just 25% said that top search engine rankings had nothing to do with market or brand leadership.

Net, these findings reinforce what search marketers have instinctively believed for years: If you're not ranking well for your desired search terms, brand names and other important key words and phrases, you're missing out on significant, highly qualified traffic.

A full copy of the iProspect Search Engine User Behavior Study can be downloaded from: http://www.iprospect.com/about/searchenginemarketingwhitepapers.htm (no registration required).

A UK Perspective on Searcher Behavior

In a separate study, UK based online marketing firm Harvest Digital surveyed "experienced" internet users about their attitudes toward search. Unsurprisingly, a majority of people reported using Google, but notably, only 24% reported using a single search engine. A full 20% said they regularly used four or more search engines.

Why use so many? UK users, despite relying heavily on search engines as a significant source of information, don't trust the results they get. Just 22% of users reported that they were confident that search engines would always give them the information that they needed.

But users blame themselves, not the engines. Just 8% said the problem was poor search engine performance. Many more said the problem was caused by their use of the engine, with 36% saying that they were not using correct terms and a further 32% said that they were looking for information that was too specialized.

The remaining 24% blamed search advertisers, though it's not clear whether this result related to sponsored links or whether survey respondents thought advertisers were buying their way into the top of natural search results (the study didn't attempt to answer this question).

Supporting but also in contrast to the iProspect findings, 43% of searchers said that the most important reason for clicking on a result was that it appeared on the first page, with just 8% saying that the brand name or website looked reputable. 32% said the relevance of the description was most important, with 17% saying that a result at the top of the first page was the most important criteria.

Credits:

This article was written by Chris Sherman, Executive Editor of SearchEngineWatch.com.

SEO News From Total Optimizer Pro

Total Optimizer Pro is pleased to announce the launch of it's new SEO blog. Here we will keep you updated on the latest goings on in the SEO world, information on the development of Total Optimizer Pro and other useful SEO tools and resources and of course, let you know when major events happen on the major search engines.

Add this page to your favorites and visit often to keep updated on the latest news to get your website ranking highly, and keep it there.