Tuesday, 8 June 2010

A/B testing: is it worth it?

If you are thinking of running an A/B test it is worth checking in advance whether you have a plausible chance of getting any significant result before starting. If you have a small sample size and only expect a small change at best, it could take you an awful long time to see a statistically significant effect.

To make this task (of figuring out, roughly, whether test is worth doing) easier, I just made this quick table which shows the smallest performance uplift which would be statistically significant at different sample sizes and conversion rates:

Sample size
Existing conversion rate10050010005000
5%247%91%61%26%
10%149%59%40%17%
25%74%32%22%10%

So if you have 5,000 views of your chosen page, and you currently see 25% completion rate (of whatever goal you are defining, a signup, click, whatever) then you just need a 10% uplift from your experiment to be statistically significant (at 95% confidence interval, using Chi-Squared test). A 10% uplift could be acheivable, so worth doing the test.

But if you are looking in some dark corner of your site getting only 100 page views in your chosen time period and a 5% goal completion rate, you'll need a whopping 247% uplift from your experiment to conclude that your experiment is working. Getting a 247% uplift from a single change is unlikely, so you will either have to wait a long time (to get more data) or just go with your gut.

Looking at the above table, a good rule of thumb is that you need a sample size of at least 1,000 to make it worth doing a test (assuming ~20% conversion rate and ~20% uplift from your experiment).

Wednesday, 24 March 2010

New in KeepMeBooked: revenue and occupancy reporting

From the KeepMeBooked blog:

You'll find a new option under "Reports" this morning: Revenue & Occupancy report:

This gives you a quick summary of revenue and occupancy over a selected date range:



[Read the rest over on the KeepMeBooked blog ...]

******
Need to get some accounting data out of your reservation system?
Try KeepMeBooked: simple web-based reservation management

Tuesday, 23 March 2010

ECJ judgement on using trademarks in Adwords

Here's my very quick review of the implications of the ECJ's decision (press release here, full judgement here) on  Louis Vuitton v Google.

Google isn't 'using' the mark, so Google's role kind of irrelevant:

Para 58:
"... a referencing service provider [=Google] is not involved in use in the course of trade within the meaning of the abovementioned provisions of Directive 89/104 and of Regulation No 40/94"

But the advertiser is:

Para 73:
"It follows from all of the foregoing that use by an advertiser of a sign identical with a trade mark as a keyword ... falls within the concept of use ‘in relation to goods or services’ within the meaning of Article 5(1)(a) of Directive 89/104."

But trademark owner still needs to show adverse effect from use of the mark:

Para 76:
"It follows from that case-law that the proprietor of the mark cannot oppose the use of a ... mark if that use is not liable to cause detriment to any of the functions of that mark"

For example, confusing users about the origin of the goods would be adverse effect:

Para 84 & 88
"The function of indicating the origin of the mark is adversely affected if the ad does not enable normally informed and reasonably attentive internet users, ... to ascertain whether the goods or services ... originate from the proprietor of the trade mark ... or ... originate from a third party. ... (para 88) It is for the national court to assess ... whether [there exist] adverse effects"

So that's important for advertisers: if you are using a keyword to confuse users about the origin of the goods or services you sell, that's probably an infringement

Simply out-bidding the trademark owner might look like infringement, because it affects the "advertising function of the mark":

94 ... the proprietor of the mark must pay a higher price per click ... to ensure that its ad appears before those of those operators which have also selected its mark as a keyword. 

But it isn't:

95      Nevertheless, those repercussions of use by third parties of a sign identical with the trade mark do not of themselves constitute an adverse effect on the advertising function of the trade mark.

Because the mark owner probably gets top placement in the natural search results anyway:

97:  the advertising page of the proprietor of that mark will appear in the list of the natural results, usually in one of the highest positions on that list ... means that the visibility ...  is guaranteed, irrespective of whether or not that proprietor is successful in also securing the display ... of an ad under the heading ‘sponsored links’.

(That seems misguided and confused: being top of natural results could still leave you three spots below competing bidders.)

So: Google is off the hook, but advertisers aren't. As ever with this long-running issue, we need more case law to determine whether or not it is a trademark infringement if you bid on a competitors trademark to display your own wares.

Wednesday, 10 March 2010

New in KeepMeBooked: pricing by day of week if you need it

We added the ability to set your room prices by day of the week.

On the rooms screen, you could previously have prices vary over the course of the year (by setting up seasons in Settings -> Setup Seasons). Now you see these new options which allow you to vary prices over the course of the week:



As ever, we try not to trouble you with extra complexity if you don't need it ... [read the rest over on the KeepMeBooked blog]

******
Still using pen and paper to manage your guesthouse reservations?
Try KeepMeBooked: simple web-based reservation management

Wednesday, 24 February 2010

Browser choice screen on KeepMeBooked

Microsoft's browser choice screen deployment is timed perfectly, because over at KeepMeBooked we are about to deploy our own 'browser choice' message to encourage people to use Chrome or Firefox which run KeepMeBooked about three times faster than IE does:



(We are working on getting IE to display our pages faster, but it ain't easy. While I don't really approve of EU bureaucrats meddling with IE's market share, the internet will be a better place once more people start using Chrome or Firefox.)

Here it comes: Microsoft's new EU-compliant advertisement for Chrome and Firefox

Here it comes: Microsoft's new EU-compliant advertisement for Chrome and Firefox:


Gotta love those EU bureaucrats!

What happens if I don't download it? Does MS get fined for not complying with EU anti-trust law? What a load of nonsense.

Friday, 19 February 2010

KeepMeBooked remote usability testing with UserTesting.com

Just ran our first remote usability test using UserTesting.com. Details over on the KeepMeBooked blog.



******
Looking for reservation software that takes usability seriously?
Try KeepMeBooked: simple web-based reservation management