Wednesday, November 26, 2008

Search Engine Optimization (SEO): Science, Art or just Guesswork?

The 1 family. friendly. food. traffic from search engines was 12 visits in July, 22 in August, 45 in September, and 303 in October. My estimate for November 2008 was 485 visits coming from search engines. In that post I wrote "In November, I expect a 60% increase in search engine traffic due to more posts (and therefore more hits), better rankings, and adding the blog to Live Search and Yahoo! Search."

With 4 days left till the end of November, we are trending to have... drum roll... an amazing 102 visits. Yes, that's not a typo - it is a third of the October value, and less than a quarter of my November estimate. On the bright side, it is still more than July, August and September combined. Should I be worried? Glancing through the first chapter of ProBlogger's book, I've seen that he had a similar issue when he got started.

I have spent a lot of time this month, tracking the results almost daily, reading about search engine optimization (SEO) and trying to figure out what is going on.

My main conclusion so far is that while SEO consultants try to paint SEO as an art more than a science (and in the process ask for a lot of money to optimize your site for search engines), it is still more guesswork than art. It is probably were medicine was 150 years ago. We know of some rough correlations between certain actions and how it might impact search engine traffic, but it is not yet possible to accurately predict the results.

I think that there is a huge business potential here (I'm giving it away for free, but will be happy to talk if someone is interested). The difficulty of accurately predicting results stems from the fact that many factors are involved:
  • The overall search traffic might be down because of various factors such as the current recession.
  • The traffic for the specific web site type or industry might change seasonally or because of other factors unrelated to my specific site.
  • The number and quality of incoming links to a site affect its ranking.
  • The changing site structure affects the search engine's ability to effectively crawl and index it.
  • The changing site content will affect how various keywords are deemed relevant by the search engines.
  • Periodical changes in search engines' algorithms will impact the ranking as well.
  • The title and description of the site's pages will impact the likeliness of users to click on the page in the search results.
  • Paid ads to the site are of course affected by the marketing campaign budget and settings.
  • And the list goes one and on...

The 3 major search engines provide very similar tools to web masters to promote their web sites. They have some web master tools allowing some basic data (e.g. is the site in the index and when it was last crawled), but very little reasons. They have analytics which track usage of the site, but again no tools to understand why that is the traffic or to model who it will change. The best tools are related to ads (which makes some sense since that is where they get all the money from) which allow some better modeling (e.g. some historic data about keywords) but it is limited to paid ads and not good enough in my opinion.

Business potential: I think that if one of the 3 bug search engines (or some third party) would create a tool allowing people to model how various changes will impact their search traffic, it would be a huge differentiator (and will allow them to sell more ads indirectly). All the data I have listed in the bullets above exists or can be gathered by the search engines. This is data that is currently underutilized. Probably in 5-10 years we'll get there. Too bad...

So what happened?

Anyway - enough with day dreaming. Let's see what actually happened to 1 family. friendly. food. First, the data (click to enlarge, the y axis is hits per week):



We started October with about 8 hits per day, it went up to a peak of about 12 per day towards the end of October. And then it fell back to 2 hits per day by mid November, with a slight increase in the past few days.

The main thing I can attribute this change to is the fact that we got a new domain name at the end of October, and redirected the traffic from the old address (wflavors.blogspot.com) to the new domain (http://www.familyfriendlyfood.com/). My assumption from trying to use the Google web master tools and trying to search for the site in Google is that it took them much less time to figure out that the old address is just a redirect and remove it, then it took them to add the new site. Hopefully the increase in traffic in the past few days will continue and get back to the old levels. One option (that I haven't seen mentioned in the SEO articles I have read) is that their search ranking algorithm takes into account the age of the site (discounting new sites), and will gradually increase the ranking. If that's the case, then it is just a matter of waiting. Note that I did see that the site was crawled starting in late October, so Google knew about it immediately after it went online.

As you can see, there is a lot of guesswork involved (more below). I have done a bunch of other things to improve the search engine traffic. Here is the list:

I have added the site to Live Search and Yahoo! Search in September. We had the first hits from Live Search and Yahoo! Search in Mid November, and 3 more hits since. Still negligible compared to Google Search, and it took Microsoft and Yahoo many months to catch up (the site has been live since July), but it is progress.

I have created several landing pages on the site (outside of the blog) to cover specific topics. The idea was that each such page will talk about a certain subject that is relevant to the site, and thus have a certain concentration of keywords increasing the likelihood to be found relevant by a search to those keywords (and also redirects to relevant pages on this topic). There is an Index page, a page about Weekly Menu Plans, a page about Cookbook Reviews, and a page about Kids and Food and so on. It is still a work in progress. So far it hasn't been successful (only 1 hit to those pages in November), but I have only seen Google pick up and index those pages a few days ago, so we'll see how it will go in November.

I have add a meta description tag to all the above pages. That should help drive traffic and cause people to click on the links if they are in the search results. The first few pages that I have created didn't have this tag, and Google indexed them with its default description. The newer pages which I have created with the description have it in the Google search results (to see all the search results, use this link). It has been almost a month since and Google has not updated the description for the older pages. I hope it will do it soon.

I have also added the blog to a couple of directories (blogged.com and bloggapedia.com) and submitted to a few others, although it is not at all clear if this influenced the search results in a any way (click this link for a list of sites linking to 1 family. friendly. food., the directories don't appear there).

Bottom line - I plan to read and learn more about this topic, and continue to do some (educated) guesses to improve search engine traffic. Hopefully, it will get to a point when it is closer to a science.

Saturday, November 1, 2008

October 2008 Results

October was by far the best month for 1. Family. Friendly. Food. so far, with the number of visits going up 483% month over month to 1825 visits. There were 1480 unique visitors (up 492% from 250 in September). In November, I expect the number of visits to go up 50% to about 2750 (see detailed analysis below).

Here is the week by week change in number of visits since July (click on the image for more details).



The income was still $0.00, with no clicks on ads. Currently there are no ads in the web site itself, only in searches.

Below you will find the distribution of visits by category, a quick overview of the status in each category, and the expected number of visits in November.

Search. This is traffic coming from search engines. The number of visits from search in October was 303, up 573% from 45 in September. See this post for more details about search engine traffic. In November, I expect a 60% increase in search engine traffic due to more posts (and therefore more hits), better rankings, and adding the blog to Live Search and Yahoo! Search.

Social sites. This is traffic coming as a result of publishing the posts in sites such as facebook, digg, reddit, etc. In October, there were 909 visits in this category, up from only 26 in September. This was by far the largest category in October, accounting for half the visits. 97% of all hits are from reddit.com, and I'll publish a separate post about this topic. In November I expect an increase of 75% in this category, due to a planned marketing push to several of these sites.

Other Family Friendly Food properties. This is traffic from other blogs that my wife writes (such as her new blog at Seattle P-I and Good Food Bad Food) and this blog. Traffic was up 907% to 141 visits in October. In November, I expect an increase of 50% primarily due to traffic from the Seattle P-I blog (which only came online a few days ago).

Other referring blogs. This is traffic originating from other blogs, usually food blogs. There were 93 visits in this category, up from 12 in September. Bloggers try to increase their traffic by posting comments in other blogs with a link back to their blog. Nurit is not very active in this, so I only expect a modest increase next month.

Direct traffic. This is users directly navigating to the site, probably by clicking a link in an email. There were 258 direct visits in October, up from 102 in September, but I don't expect any increase in November. I am planning to see if there are ways to improve the instrumentation for this category.

Web Mail. This is users clicking on a link in mail programs such as Gmail, Hotmail and Yahoo! Mail. There were 39 visits in October, and a similar number (35) in September. I don't expect any change in November.

Portals. This is a new category including things such as My Yahoo!, iGoogle and web news readers (such as Google Reader). It only had 15 visits in October (up from 2 in September), so it is too early to have a good plan for it. I will track it closely in November.

In terms of the quality of the visits, the number of pages per visit and average time in each visit went up slightly in October, after going down in September.