Follow me: @D_mitar

Most read posts recently



Jun 3rd 2010 × beware: googlebot understands javascript, mootools and ajax now

This came as a complete shock to me: due to the Google Mayday update, I have been monitoring how the bot accesses our sites in a futile search for clues as to the page rankings changes. Very surprised to discover googlebot fetching URLs that are _strictly_ available through Ajax only.

Here is a sample request from earlier today that fetched a small worker thread that produces additional product images:

66.249.65.25 - - [03/Jun/2010:11:25:43 +0100] "GET /angles.php?id=102257&version=Brown HTTP/1.1" 200 801 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

We have all spoken of Google’s increased capability of following and deciphering javascript logic but it goes beyond the scope of reasonable effort, in my opinion. Is the scraping done based upon URLs detected in the javascript blocks or does it actually evaluate, run and monitor the calls? If the scraping works based upon regex that catches URLs, would obfuscation / base64 encoding help? Does googlebot support XHR via POST or only via GET? Too many questions without an answer at this point, I will update this post as more info becomes available to me.

Such fetches can often be undesirable–certain AJAX calls we do are tied in with sessions and can cause errors / notifications when being requested w/o due cause, session or event. I am currently trying to find if there is a plausible way to apply a “nofollow” to such calls at all that does not involve a lot of editing and blocking of googlebot from any scripts that it has no place reading. If it comes to it, I will revert all ajax handling to a /xhr/ folder and disallow it in robots.txt, wonder what the best practice is…

As for the Mayday Update, the less said about it, the better. I have never seen Google SERPs get things so very wrong – for instance, ranking disabled products with 0 links anywhere on the net that redirect their traffic instead of what used to be the landing pages. Longtail relevance? With 2k organic in-links to the landing page, PR4 and quality content built over 2 years, this is not bizarre enough…


Mar 10th 2010 × Cross-domain AJAX calls via YQL as proxy and mootools JSONP

I wrote this a while back when I needed to do pseudo AJAX calls to a remote host. Due to XSS restrictions and security policies, a normal XMLHttpRequest (XHR) call is not allowed to work across domains or even sub-domains. But Yahoo’s YQL interface kindly lets you GET any URL (which also means being able to submit via GET), irregardless of whether the domain is local or not.

Here is how it works. You need mootools (also download mootools-more as this class will extend Request.JSONP which is a part of mootools-more).

Request.YQLajax = new Class({
    // gets basic info such as country and latitude data
    Extends: Request.JSONP,
    options: {
        log: !true,
        url: "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22{location}%22&format=xml"
    },
    initialize: function(location, options) {
        this.parent(options);
        if (!location)
            return;

        this.options.url = this.options.url.substitute({location: encodeURIComponent(location)});
    },
    success: function(data, script) {
        this.parent(data, script);
    }
});

// example use to fetch BBC's page
new Request.YQLajax("http://www.bbc.co.uk/", {
    onSuccess: function(data) {
        $("result").set("html", data.results);
    }
}).send();

You can see this in action on mooshell or play with it below. Happy cross-domain ajax-ing.


Apr 30th 2009 × javascript obscure exception trapping and reporting via AJAX for mootools

I’ve had this post in my submission bin for a while but a question of the digital point forums today prompted me to dig it out… First of all, what is the difference between a javascript exception and a javascript error? Well, my view on this is simplistic: an exception IS an error of a runtime nature that can be handled, whereas an error is typically to do with syntax and can’t be ‘caught’.

So why would we care of exception errors anyway?
Whilst programmers take every possible measure to release compliant and bug-free code for all possible browsers, the sad truth is that–for ajax-rich applications–no amount of testing can ensure things won’t break for somebody. There always will be the ‘daft’ people, their quirky old browsers (IE6, anyone?), odd plug-ins, proxies / anti-virus software, leap years and so forth–somehow managing to trigger unexpected javascript exceptions. The more traffic you get, the better the chances are for somebody to stumble upon such difficulties. What you don’t always get is people phoning up to tell you about it. Not unless it’s something critical – like the inability to add an item to a shopping basket or failure to checkout. And let me tell you, your customer services won’t be able to get any meaningful debug information.

A practical example?
Recently I got a report of a customer that was unable to checkout and for whom the ‘checkout’ button did nothing. The button itself had an onclick event assigned to it within the mooTools domready function… Surprisingly, it turned out the domready just consistently fired up earlier than expected for him, with the checkout.gif image just not being a part of the DOM at the time of the event assignment. This got me worried and made me want to try and discover other such potential deterrents to people’s browsing experience.

The solution is closer when you know your problem areas…
I created a javascript exception trapping handler and placed it around what was perceived to be the ‘dangerous / unsafe’ portions of the checkout code. The handler itself was set to use XHR (ajax) to send me the exception data via email, alongside of whatever additional data the PHP script was able to grab (browser, user info, basket contents and so forth). The results were… astonishing. This quirk was happening a fair bit more than we would have liked (1 in 30 customers). We also found out that our promo codes buttons were also being affected… The flaw in the mooTools 1.11 domready code aside, how did we fix it? By not assigning events until all the elements were available, by using load as opposed to domready for IE users and so forth.

Here is the refactored code for mootools 1.2+ instead:

var errorHandler = function(dbug) {
    // trap errors and send via ajax to a script that logs them
    new Request({
        url: "/errorTrap.php",
        data: dbug,
        onComplete: function() {
             $("errorOutput").set("html", this.response.text);
        } // comment this function out out and the comma on the line above. data will arrive via $_POST.
    }).send();
    return true;
}; // end errorHandler

$("makeException").addEvent("click", function(e) {
    new Event(e).stop();
    try {
        duffColours[2] = 0;
    } catch(er) {
        errorHandler(er); // install it.
    }
});

And here is an example of the errorTrap.php contents, I’d advise you to do data washing/escaping as well:

<?PHP
$body = "ERROR DETAILS FOLLOW:\n\n";

foreach($_REQUEST as $key => $value) {
    $body .= "$key: " . str_replace('"', '\"', $value) . "\n";
}

$body .= "\n----------------------------------------------------\nServer variables:\n\n";
foreach($_SERVER as $key => $value) {
    $body .= "$key: " . mysql_escape_string($value) . "\n";
}

mail("your@email.com", "JS exception on {$_SERVER['HTTP_REFERER']}", $body);
?>

To create an exception, click here – it will send it to a file called errorTrap.php, which iterates the $_GET and $_SERVER variables before dispatching me the email

Error output here


Jan 20th 2009 × getting fancy with AJAX: your first form submission and verification

Having frequented the Digital Point Forums as of recent, I have come to the conclusion that most people are barely coping with the concepts of javascript and AJAX. So much so that they are struggling to interpret anything that is more advanced than changing a few field names and including some javascript files in the page header.

Naturally – people wanting it to ‘just work’ are right, of course. There will be plenty of solutions out there for most things that can do this and achieve it in such a way that does not warrant an understanding of any javascript concept. But there comes a day when you go ‘hold on, what if I need to do this?’ and find you are stuck with a solution that you cannot comprehend…

I guess all my posts thus far have been aimed at a different type of person – one that is more ‘at home’ with javascript and mootools. Time to go back to basics…

*edit* if you’ve come here for the magical email regex that will cover _any_ RFC compliant email, here it is:
var validEmailRegex = /^(([^<>()[\]\\.,;:\s@\”]+(\.[^<>()[\]\\.,;:\s@\”]+)*)|(\”.+\”))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;

click here to view sample tests and code

The most common use for AJAX that seems to draw the crowds is still a form submission. It can be via GET or via POST, it may do some form processing for required fields and so forth. But it needs to submit without a page reload and it needs to output something back. So, how do you set about creating an AJAX form?

1. First of all, grab a copy of a javascript framework. A framework is simply a library of pre-set functions and methods you can use in your own pages. I tend to use mootools but anything will do – read my post on selecting a framework. In this example, we WILL use mootools, however.

2. Create your form page. As basic as this example is, we need to assume at you have at the very least the rudimentary skill of being able to compose a form in HTML. What you should try to do is to get your form working and submitting data to your PHP or ASP or CGI processor page on its own – and we will then convert your page into an AJAX submission instead. Here is an example form page:


<div id="feedback" style="background: #eee; padding: 2px; border: 1px solid #333; float: left">
    <form action="mooForm.php?a=send" method="POST" id="myform">
    <label for="customer_name">Your name:</label><p>
    <input type="text" class="required bordered" value="" name="customer_name" id="customer_name"/><p>
    <label for="customer_email">Your email:</label><p>
    <input type="text" class="required bordered" value="" name="customer_email" id="customer_email"/><p>
    <label for="customer_comment">Your message:</label><br />
    <textarea class="required" style="width: 360px" name="customer_comment" id="customer_email"/></textarea><br /><br />
    <input type="submit" value="Send data" /></form>
</div>

3. Create a wrapper element for the form where we will output the reply: when contact_us.php is ‘reached’ and the data is saved, it can send us feedback to this effect – something along the lines of: “Thank you for your submission”. To do so, just add <div id=”feedback”></div> around your form – or you can keep it to the side, up to you and your styling preferences. We are going to be ‘replacing’ the form with the reply, however. Here it is how it all looks and works ‘on-screen’:


4. Attach javascript to intercept the form submission: you need to take control of what happens once the user presses “Send data”. Do this by fetching a copy of mootools and uploading it to your server. Within the head section of your site, you’d add the following:

<script type="text/javascript" src="mootools-1.2.1core.js"></script>
<script type="text/javascript">
// in mootools, the 'onload' event is called 'domready' and it ensures
// the DOM has completed loading and elements can be modified safely.
window.addEvent("domready", function() {
    // now, add events for the form
    $("myform").addEvents({
        "submit": function(e) {
            // stop the event from propagation...
            e.preventDefault();

            new Request({
                url: this.get("action"),
                data: this,
                onComplete: function() {
                    $("feedback").set("html", this.response.text);
                }
            }).send();
        }
    });
}); // end domready
</script>

Although, you can leave it at that, to get a working AJAX form

5. Expand on your code to add some error checking. Of course, you can use PHP or ASP or whatever serverside language you like to inspect the data being submitted – and probably should do so anyway. But it helps if you can add some trapping clientside as well – on the off-chance it reduces errors and keeps your server load lower.


// extend elements to support warnings within values
Element.implement({
    fieldWarning: function(warningText, warningDelay) {
        // very basic - changes the value of the input to warningText
        // restores to "" within warningDelay in ms

        this.set({
            value: warningText,
            styles: {
                backgroundColor: "#fcc"
            },
            events: {
                focus: function() {
                    this.set({
                        value:  "",
                        styles: {
                            backgroundColor: "#fff"
                        }
                    }).removeEvents();
                }
            }
        });

        (function() {
            this.fireEvent("focus").removeEvents();
        }).delay(warningDelay, this);
    }
});

var validEmailRegex = /^(([^<>()[\]\\.,;:\s@\"]+(\.[^<>()[\]\\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;

// define common functions and variables
var isValid = function(email) {
    // boolean function that checks email string against allowed strings in accordance to RFCs
    return email.test(validEmailRegex);
},  // end isValid
cleanData = function(elementsArray) {
    // cleans passed elements with property 'value' from spaces and XSS
    elementsArray.each(function(el) {
        el.set("value", el.get("value").clean().stripScripts());
    }); // end each
} // end cleanData

window.addEvent("domready", function() {
    // now, add events for the form
    $("myform").addEvents({
        "submit": function(e) {
            // stop the event's default action from propagation...
            e.preventDefault(); // can also do new Event(e).stop();

            // clean all form fields from excess spaces and cross side scripting attacks
            cleanData($("myform").getElements("input,textarea"));

            var noErrors = true;

            // first, loop required elements
            $$("input.required,textarea.required").each(function(el) {
                var testedValue = el.get("value");
                if (testedValue.length == 0 || testedValue == "Required field") {
                    noErrors = false;
                    el.fieldWarning("Required field", 2000);
                }
            });

            if (noErrors) { // test email
                var testedEmail = $("customer_email").get("value");
                if (!isValid(testedEmail)) {
                    $("customer_email").fieldWarning("INVALID: " + testedEmail, 2000);
                    noErrors = false;
                }
            }

            // if no errors in the form, compose the XHTTP request manually.
            if (noErrors)
            new Request({
                url: this.get("action"),
                data: this,
                onComplete: function() {
                    $("feedback").set("html", this.response.text);
                    // since the form itself is contained within this layer
                    // it will 'self destroy' with whatever HTML we print
                    // from the PHP/ASP.
                }
            }).send();
        }
    });

});

What is taking place here? For the purposes of this small form, we are going to perform 2 types of javascript validations: a required field (which they all are) – a check to see if the length of the strings entered is greater than zero; and an email verification test – a check to see if the email string is compliant with RFCs, done via a regex test.

You can download the javascript above by clicking here and look at it at your own leisure.


Dec 9th 2008 × snippet time: mooRating, a star based ajax voting script


This is just a little snippet that can enable you to use star based rating on a page and interactively do something with the voting results. It is not a plug and play solution, you can attach whatever events you want to it and have complete control over the number of stars on show, the colour transitions and formatting.

window.addEvent("domready", function() {
    doRating(4.6, true, {
        targetObject: $("myrating"),
        clickEvent: function(ratingOptions) { // save the vote
            new Request({
                url: "ratinger.php",
                method: "get",
                onComplete: function() {
                    alert("saved via ajax!");
                }
            }).send("a=vote&ratingID="+ratingOptions.ratingID+"&rating="+ratingOptions.newRating);
        }
    });
}); // end domready

Here is an example use with defaults, start value of 4.5 and an empty layer called “myrating” to hold it all:

You can run more than one instance of a “ratinger” per page, naturally – but they do need distinct IDs in order to differentiate the click events when saving data via ajax. This time, something very, very different: a 6 star rating in the style of a retro hi-fi stereo output indicator (a new gif file for the background), with the start value of 3.6 (4 stars) and ‘tips’ enabled: here is myrating2:



The code behind the above vote instance (if you can’t be bothered to view the source):


doRating(3.6, true, {
    // use some default value you have read from the DB that represents your start / current rating
    targetObject: $("myrating2"),
    maxStars: 6,
    tipShown: true,
    background: '#010',
    colourBase: '#550000',
    colourTarget: '#00cc00',
    starWidth: 22,
    border: 0,
    tipPadding: 0,
    starSpacing: 1,
    starHeight: 10,
    imageURL: '/images/stereo.gif',
    clickEvent: function(ratingOptions) { // save the vote
        new Request({
            url: "ratinger.php",
            method: "get",
            onComplete: function() {
                alert("saved via ajax!");
            }
        }).send("a=vote&ratingID="+ratingOptions.ratingID+"&rating="+ratingOptions.newRating);
    }
});

How to use on your site
As usual, nothing you get from this site just works, it requires you to actually know what you are doing… Using this particular mootools vote snippet is very simple, however. First, download the mooRating javascript source and save it. Include on a page after your usual mootools init lines and within domready, you can instigate a ‘ratinger’ / ‘vote’ widget within any element as defined by your html / css. You will probably need to save the star.gif image I made or make your own with some transparency…

Here are the default options the mooRating function supports:


var doRating = function(rate, addevents, options) {
    // mootools star rating system by dimitar christoff
    // v2.1 for mootools 1.2.1
    // last modified: 09/12/2008 19:37:15

    // defaults
    var options = $merge({
        background: '#fff', // vote cell background
        colourBase: '#f4edaf', // start colour (left side)
        colourTarget: '#f9e526', // end colour (right side)
        maxStars: 10, // number of stars
        starWidth: 24, // box with
        starHeight: 22, // box height
        starSpacing: 0, // space between stars
        imageURL: "/images/star.gif", // url to box background image
        tipPadding: 8, // padding around the vote text
        tipSize: "12px", // size of text in tip
        tipShown: false, // toggle tips
        border: "1px solid #ffffff", // border around the "stars"
        clickEvent: $empty() // what to pass as a function on click
    }, options);
// .. and so on and so forth.

I hope it helps somebody to add easily votes or product ratings on their pages with plenty of control. NB: you need color.js compiled into your version of mootools or you need to take out the colour gradiency.