Update on MSN/Live and Yahoo Traffic

It's been almost a month since I started working on getting more traffic from Yahoo and MSN/Live.com search engines on one of my sites, but my efforts seem to be paying off. All the changes were on site, mainly changes to my robots.txt, sitemap.xml and pinging updates when the site ads new content.



Compare to the previous period, traffic improved across all major search engines. Google was already giving me a good amount of visitors but traffic from Google increased over 17%. Traffic from Yahoo! search increased 50% and MSN/Live.com traffic had the biggest improvement of over 180%. Ask.com traffic improved by over 90%.



Google still is my major source of search engine traffic by far but these results seem promising.



Live.com has indexed a lot more of this site as a result of the changes. Before the changes, Live.com had only indexed about 1% of the site, now it's over 10% of the site and MSNBot seems to be visiting more frequently. Every time I check the number increases and my domain score is 5 out of 5.



The increased index pages helped bring some more long tail keyword traffic from Microsoft's search engines. Since there are still 10 times as many pages that can be indexed by MSNBot that haven't, I'm hoping traffic from MSN and Live.com continue to grow. Yahoo seems to have indexed the pages well. While I'm expecting search traffic growth from Yahoo!, my expectations are less.



One thing to note about traffic from Live.com is that log analyzers such as AWStats may be misleading. Microsoft has a crawler they use to detect cloaking. It acts like a normal website visitor and downloads all images, stylesheets, javascript, and so on. The problem is that when it accesses a page, it uses a referrer header to make it appear is if it came from a search on live.com. While Microsoft claims to have fixed the problem, I still see this behavior across different sites and the invalid search engine keywords being affected. 



Google Anlytics seems to recognize these fake search engine referrers and doesn't report them as genuine search engine traffic.

Tell Googlebot to Crawl Faster

Google's Webmaster Tools has changed the interface that allows a webmaster to control how fast or how slow Googlebot crawls a site.

I just noticed the change today, in the past, there were 3 radio buttons for normal, faster and slower. Many webmasters could only set it to normal and slower because faster was disabled unless Google thought your site really needed to be updated faster and could handle it.

As you can see from the screenshot, the control has now been changed to a slider and presumably everyone can set it how they wish, but it's likely that Google may override the settings. Also, setting the crawl rate faster doesn't mean Googlebot will visit your site more frequently. The crawl rate just determines how spaced out one request will be from the other. If Google thinks your site should get 200 visits a day, you're going to get 200 visits a day. If you set a faster crawl rate, that just means those request could happen within the same hour.

One neat thing about the new interface is that it shows you what Google's recommended crawl rate for your site is. One one of sites the crawl rate was faster than the other. In the past, you could only judge by seeing your crawl stats. The top chart shows the number of visits, the second the number of kilobytes downloaded and the third the average time spent downloading a page.

As you can see from the charts, this site seems to be getting slower and the recommended crawl rate was set a little slower than on other sites. Probably because Google thought there was too much strain on the server. This wasn't the case. The drop in response time had to do with the use of a third-party api but the site had plenty of capacity. In this case, I instructed Googlebot to crawl faster.

But sometimes, you really do want to slow down your crawl rate. On another site that is fairly new, Googlebot visits the site 10 times more than real visitors do. It's not a site I'm actively promoting yet and while I don't mind getting the many pages crawled, it seems a waste to waste so much bandwidth on crawling compared to the actual site traffic. In that case I slowed it down.

If you want your site to be crawled less, setting a slower crawl rate can help accomplish that. For instance, if you only want Googlebot to visit your site 100 times a day, you would set your crawl rate so that Googlebot only visits ever 864 seconds.

To help figure out what you should set your crawl rate to take 86,400 and divide it by how often Googlebot should crawl your site. 86,400 is the number of seconds in a day.

If you're not familiar with the Google Webmaster Tools interface, these are the steps to make the change. The placement has also changed:

1. Log into Google Webmaster Tools
2. Click on the website you'd like to change in the Dashboard
3. Select Settings from the left menu bar
4. Scroll down to the Crawl rate section
5. Slide the slider either towards faster or slower
6. Click the Save button

Remember that these are just recommendations and Google may decide to do what it feels is best.