There’s yet another one of these problems that are inherently weaved deep into CubeCart that you just wouldn’t know about… It displays different versions of pages to search engines and to humans, namely–it disables the shopping basket and checkout functionality as well as the login and registration.

First of all, needs to be said that troubleshooting this and finding the reason for users being unable to purchase off a CubeCart site is an absolute nightmare. When a customer says they use ‘bog standard’ IE8 and report their ‘add to basket’ button doing nothing whatsoever, one tends to think ‘has the javascript handler gone wrong?’. You go on a wild goose chase, trying to reproduce the problem and failing despite of installing various javascript exception tracking modules, looking through logs and quizzing puzzled shoppers. All the while, you can’t help but wonder about the possible extent of the problem, how many users get this? And then – you catch a break by accidentally discovering customers unable to purchase also lack the login/register links – time to start connecting the dots…

From session.inc.php, which controls the login / register links in CubeCart:
if (!$cc_session->user_is_search_engine() || $config['sef'] == false)

The template does not get shown to search engines? That makes sense… having different versions of pages shown to users and to spiders is not only a bad practice (google really don’t appreciate “cloaking” techniques), there is just NO need for it whatsoever. In order to prevent spiders from indexing pages that are deemed irrelevant to e-commerce and spill page rank / relevance, they could have been disallowed from within the robots.txt file. A rel=”nofollow” could have been applied to links to such pages… but what have the clever folks at CubeCart done instead?

They created a boolean method into the sessions class that decides if the user is a bot or not, user_is_search_engine(). It does so by comparing the contents of a file called spiders.txt, filled with “known” extracts from the user agent strings of various spiders from around the web, against the user agent string of the visitor. To be fair, the original idea for this kind of testing comes from OS Commerce…

The problem is when a legitimate customer is being discarded from using the site’s e-commerce because their user agent string is customised in a ‘bad’ way. How does that work? The original CubeCart spiders.txt file has lines that reject things like:

‘googlebot’ but also just ‘google’,
‘msnbot’ but also just ‘msn’

And so forth… multiply by x number of toolbars and custom strings CubeCart may never know about and you have a HUGE problem. For instance, users with Google Desktop get their user agent string set to Mozilla/5.0 (compatible; Google Desktop) so they promptly get rejected. Luckily, that’s not a very popular application but the “MSN” string and the absolutely COUNTLESS numbers of people that have got a user agent string like this one: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Sky Broadband; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; InfoPath.2; .NET CLR 3.5.21022; MSN OptimizedIE8;ENGB) presents a VERY real problem. MSN optimized IE8? Not on my shop site, mate.
edit: check your spiders.txt version, if it predates august 2008, you are affected.

I have been looking at the user sessions table and have thus far found over 3000 genuine users that have been rejected by CubeCart’s loose user agent matching routine. That’s a lot of business to lose and the store owners are understandably upset. It’s not free software and at a testing time like the credit crunch we’re enjoying, having your own store work against you is far from ideal. The real frustration comes from the fact that people had reported an intermittent loss of shopping cart functionality on the CubeCart forums and on their bug / ticketing system. Reported and dismissed – apparently, too difficult to trace or unsupported due to store being customised. Every programmer makes mistakes, but being unable to rectify them and failing to provide support to your paying customers – it’s just bad business. I am sorry to say, CubeCart has failed to impress once again…

The fix to the CubeCart user agent problem:

1. apply nofollow to the links for login, register and checkout
2. empty the contents of spiders.txt in your cubecart root folder (don’t delete it)
or
2. change user_is_search_engine() to always return false.

To test if your store is affected, use FireFox and check this post on how to change your user agent string, set it to the one I put as an example above and visit your shopsite, then try to add a product to your basket.

update: I am being told that this problem is no longer to be found in current releases of cubecart. Well done, the team :) Now, how many existing customers on versions pre-dating august 2008 have been notified?