All savvy search engine marketers know how to create a search-friendly site that can be easily read by a search engine spider, or bot. But what happens when those bots go bad?

A special report from the Search Engine Strategies conference, August 7-10, 2006, San Jose, CA.

Many site owners tell horror stories of lost uptime, wasted bandwidth and stolen content that have resulted from rogue spiders invading their virtual domains. Several experts, including representative from search engines Google, Yahoo and Become.com, sat on a panel to discuss the issues of “bot obedience.”

Jon Glick of search engine Become.com started the session with a brief overview of the topic.

“Robots are great at finding links and content,” said Glick, “But they are dumb. They can’t process cookies or javascript.”

Glick went on to highlight how to create bot friendly sites using tactics such as hyper link navigation, or navigation that uses “pure” links with no javascript or Flash elements. Glick also suggested that web authors avoid excessive parameters in dynamic URLs when building bot friendly sites.

Dan Thies of SEO Research Labs talked about one of the darker consequences of untamed bots—duplicate content. Thies said that unscrupulous bots regularly target highly performing web pages, scrape their content and place it in thousands of other pages throughout the web. This can cause consumer confusion, along with the fact that duplicate content is eventually filtered out of the search engines main results and relegated to supplemental results at best. At worst, the content is booted out of the search engine index.

Thies said he has seen client’s sites lose “considerable revenue” after sites were scraped and unauthorized duplicate content was posted throughout the web.

“My client saw a drop of 50% drop in SEO referrals,” said Thies. This would be disastrous for most sites; however since [the client was] good at PPC and other online advertising, the effect was noticeable but not devastating.”

Bill Atchison of Crawlwall.com started off his presentation by getting the audience’s attention.

“Most bots steal, steal, steal from me,” said Atchison. “I’ve had bad bots hit my site so hard they took it down. Before I took action, my website was under constant attack.”

Atchison said that before he took steps to alleviate the problem, 10% of his site’s page views were from “bad bots,” or bots that ignored all instructions from robots.txt files or meta information. Atchison claimed that he has had copyrighted material stolen from his web site and his servers were overloaded and brought down.

“What was happening [with the bad bots] was unacceptable and had to be stopped,” said Atchison.

Atchison said the first step to Bot obedience is to differentiate good bots from bad bots. Below is a list of characteristics Atchison said he uses to identify if a bot is good or bad.

Good Bot Characteristics:

• Obey Internet standards like robots.txt
• Don’t crawl your server abusively fast
• Return to get fresh content in a reasonable timeframe
• Provide traffic in return for crawling your site

Bad Bot Characteristics:

• Will go to any length to get your content
• Ignore Internet standards like robots.txt
• Spoof ‘bot names used by major search engines
• Change the User Agent randomly to avoid filters
• Masquerade as humans (stealth) to completely bypass filters
• Crawl as fast as possible to avoid being stopped
• Crawl as slow as possible to slide under the radar
• Crawl from as many IPs as possible to avoid detection
• Return often to get your new content and get indexed first
• Violate your copyrights and repackage your site
• Hijack your search engine positions
• Provide no value in return for crawling

Atchison outlined several failed approaches that are commonly used to control rogue bots on web sites, including the “opt-out” approach. The opt-out approach involves using Robots.txt files, meta information and user-agent blacklists to direct bots on where they can and cannot go and keep blacklisted bots out. Atchison said this approach fails most of the time because rogue bots ignore this type of data and mask their true identity by falsifying their user-agent data. Rogue bots can also turn the tables on site owners by using the prohibited files information in robots.txt files as a road map for the places they want to go.

Atchison said he employs an “opt-in” approach. In this approach only bots known to be good are allowed into the site. All other bots are blocked. This approach is more risky, as some bots that do provide quality traffic may be blocked from accessing the site, but Atchison said the benefits outweigh the risk.

Both Rajat Mukherjee from Yahoo and Vanessa Fox from Google stated that the best way to control their respective bots is to use the robots.txt protocol.

“Slurp (the name for Yahoo’s bot) is an obedient bot,” Said Mukherjee. “I always recommend reading the information at in order to control slurp.”

Both Fox and Mukherjee said Yahoo and Google want to be contacted if their bots are found to be misbehaving.

Mukherjee also took some time to introduce the revamped Yahoo Site Explorer program. The Yahoo! Site Explorer program allows site owners to authenticate their sites and get more detailed information about how their sites are being viewed by Yahoo. The program also allows users to manage site feeds (other than paid inclusion feeds) and export information in .CSV format.

During the Q&A section of the presentation, an audience member suggested that all responsible bots implement an identity program to certify the bot is actually what its user-agent says it is, a suggestion that was met with enthusiasm from the panelists as well as the audience.

SOURCE: , Tony Wright is vice president of client services for Kinetic Results, LLC.

Bonnie Burns SEO Director

The world doesn’t care about you:-) — Well, at least 95% of the world doesn’t care about you.

A recent study showed that between 3% and 5% of the people who visit your webpage click on the “About Us” link to find out more about you. The good news is that the visitors who click on this link buy at a rate 30% higher than visitors who don’t click on the link. People like to know who they’re sending their money to.

But, looking at it another way, this study also showed that 95% to 97% of the visitors to your site couldn’t care less about you or what you named your dog.

What does all of this mean to you?

#1. Have an “About Us” link on your Home page. It helps make buyers out of 3% to 5% of your visitors and it doesn’t hurt sales to the people who don’t look at the “About Us” link.

#2. Don’t spend a lot of time on your Home page (or any other page) talking about yourself or what you have been doing or are planning to do. (Maybe 5% should be about YOU!)

Yes, working a little bit of information about yourself into your website will help sales, but it should pertain to your product expertise.

And for the 5% of you who give a rip –  After being in the SEO buisness for 12 years or so, I still, to this day, love the challenges it presents. It is far from boring, often frustrating, and more often then not, usually amazing.

Bonnie Burns. SEO Director

Less people use 1 word phrase in search engines according to OneStat.com

Amsterdam – July 24 2006 – OneStat.com ( ), the number one provider of real-time intelligence web analytics, today reported that most people use 2 word phrases in search engines. Of all the search phrases world wide, 28.91 percent of the people use 2 word phrases, 27.85 percent use 3 word phrases and 17.11 percent use 4 word phrases. Less and less people use now 1 keywordsince the last measurement in July 2005. 

“Search engines like Google, MSN and Yahoo can drive a lot of traffic to a web site. It is important that a webmaster or SEO expert knows what kind of search phrases they have to use to drive more traffic to a site. Our software is the ultimate solution to measure search phrases and search phrases by search engine. Each webmaster or SEO expert can analyse what kind of search engines the visitors use to find a web site,” said Niels Brinkman, co-founder of OneStat.com.

The 7 most used word phrases in search engines on the web are:

July 2006			Â	 July 2005
1.	2 word phrases	28.91%	Â	1.	2 word phrases	29.60%
2.	3 word phrase	27.85%	Â	2.	3 word phrase	27.55%
3.	4 word phrases	17.11%	Â	3.	4 word phrases	16.21%
4.	1 word phrases	11.43%	Â	4.	1 word phrases	13.42%
5.	5 word phrases	8.25%	Â	5.	5 word phrases	7.58%
6.	6 word phrases	3.68%	Â	6.	6 word phrases	3.21%
7.	7 word phrases	1.59%	Â	7.	7 word phrases	1.34%

 

USA
1.	3 word phrases	28.83%
2.	4 word phrase	22.28%
3.	2 word phrases	20.43%
4.	5 word phrases	11.97%
5.	1 word phrases	6.19%
6.	6 word phrases	5.76%
7.	7 word phrases	2.59%

 

Canada
1.	4 word phrases	24.02%
2.	3 word phrase	23.85%
3.	5 word phrases	16.77%
4.	2 word phrases	15.40%
5.	6 word phrases	8.88%
6.	1 word phrases	5.36%
7.	7 word phrases	3.64%

 

UK
1.	3 word phrases	29.38%
2.	2 word phrase	24.89%
3.	4 word phrases	18.36%
4.	5 word phrases	10.82%
5.	1 word phrases	8.91%
6.	6 word phrases	4.08%
7.	7 word phrases	1.91%

 

Australia download movie the chronicles of narnia: the voyage of the dawn treader
1.	3 word phrases	28.74%
2.	2 word phrase	23.21%
3.	4 word phrases	21.52%
4.	5 word phrases	11.90%
5.	1 word phrases	7.71%
6.	6 word phrases	4.64%
7.	7 word phrases	1.58%

 

Germany
1.	2 word phrases	40.05%
2.	1 word phrase	28.89%
3.	3 word phrases	20.94%
4.	4 word phrases	7.14%
5.	5 word phrases	2.12%
6.	6 word phrases	0.58%
7.	7 word phrases	0.15%

 

France
1.	2 word phrases	34.74%
2.	3 word phrase	26.84%
3.	4 word phrases	14.11%
4.	1 word phrases	10.44%
5.	5 word phrases	6.65%
6.	6 word phrases	3.63%
7.	7 word phrases	1.80%

 

Italy
1.	2 word phrases	33.64%
2.	3 word phrase	26.55%
3.	4 word phrases	17.41%
4.	1 word phrases	12.32%
5.	5 word phrases	6.05%
6.	6 word phrases	2.17%
7.	7 word phrases	1.09%

Belgium
1.	2 word phrases	34.74%
2.	3 word phrase	26.84%
3.	4 word phrases	14.11%
4.	1 word phrases	10.44%
5.	5 word phrases	6.55%
6.	6 word phrases	3.63%
7.	7 word phrases	1.80%

the Netherlands
1.	2 word phrases	35.35%
2.	3 word phrase	27.35%
3.	1 word phrases	16.11%
4.	4 word phrases	13.04%
5.	5 word phrases	5.07%
6.	6 word phrases	1.87%
7.	7 word phrases	0.72%

The OneStat.com solutions provide executives, marketers and webmasters with answers to critical e-business questions such as:  

·        Who is visiting my website?

·        How many pageviews, visits (sessions) and visitors are coming on a daily, weekly, monthly, quarterly or yearly basis?

·        What content, products, and services do my visitors prefer?

·        How many visitors return to the website and how often?

·        What kind of search engine do they use?

·        What kind of technology do your visitors use to view the website?

·        How much time do they spent on the website?   

OneStat.com is one of the largest providers of real-time intelligence web analytics with 50.000 subscribers in more than 100 countries.

Methodology: A global usage share of xx percent for search phrase Y means that xx percent of the visitors of Internet users arrived at sites that are using one of OneStat.com’s services by using the particular number of search phrases Y. All numbers mentioned in the research are averages and all measurements are normalised to the GMT timezone. Research is based on a sample of 2 million visitors divided into 20,000 visitors of 100 countries each day. 

Bonnie Burns – SEO Directory www.ecombuffet.com

Rick Butts, a colleague and friend of mine has just launched an amazing new tool.  For all you bloggers out there, Rick has created a tool that will save you time as you build multiple blogs for Adsense (or other purposes).  Rick can tell you better than I can the power of this tool, so check it out:  WordPress Cloner

Please note: it is not often that we recommend a tool, and if we do it, you can be sure it is one we use ourselves and have determined it is worth it! 

Click Here to check out WordPress Cloner

Happy Blogging,

Jenn Horowitz – Director of Marketing  

 

 

I’m back after a short break, and ready to continue talking rankings, traffic, and all those fun things. 

One of the most important and most overlooked factors in your website’s success is your conversion rate. 

You make the decision to get SEO done, you hire a great company, you get more rankings than you had hoped for…but you still come back to the SEO firm and tell them you have no ROI. 

If your SEO company truly knows their stuff, and cares about the clients they server, they should start addressing your concern by looking at your website statistics.

Your stats are a goldmine of information waiting to be discovered..  We can find out who comes to your site, what words they found to get there, what engines they used, we can find out who links to you, how long people stay on your site, what page they enter on, what page they exit on and so much more.

Think about how useful that information is to you.  It is safe to assume that if you have good rankings, and good keywords, you should expect to see increased traffic to your site.  If traffic increases, you would expect your opt-in rate or sales to increase.  If there is no increase, then something isn’t working.  People are finding your site, but they are leaving before they take any action.

If you start looking at your stats, you can determine which page people are leaving on most often – then you can spruce that page up.  If people are spending a long time on your site, but not buying – maybe you have too much information and they read a lot but are never compelled to order, so then you need to look at your calls-to-action in your text and see how it can be improved.

Another great thing to look at is how many 404 errors people are getting.  If they are trying to find pages that aren’t there, of course they are going to leave.  If your web stats show a lot of 404 errors, you need to find the problem on your site and fix it. 

I’ve only begun to touch on just a few marketing concepts and ideas you need to educate yourself about.  The key take away here is: Your website success is not only dependant on rankings or traffic.  That is only step one.  As soon as you commit to driving traffic to your site, you need to commit to your marketing so your site can handle converting the new found traffic.

You have some options here:

- Learn what you need to know and handle it yourself.  That only works if you can spare the time, and truly commit.  - Your other option is to hire help.  If you are lucky your SEO firm will look at the big picture and be able to help with marketing and conversion issues.  If not, you need to find a marketing consultant.

Just a little bit to get you thinking, I’ll be back soon with more tips and advice, as well as news and updates.