Minimap for Sahi Logs

to get a neat minimap for your Sahi Logs


simple, easy, works for playbacked logs
well.. boring but practical

Depends On

active internet connection
JavaScript

Change The Logs CSS To Include Javascript

Go to your 'SAHI_HOME/htdocs/spr/css' and change the content of the 'consolidated_log_format.css'. Just append at the end:

</style>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
<script type="text/javascript" src="jquery.fracs-0.11.min.js"></script>
<script type="text/javascript" src="jquery.outline-0.11.min.js"></script>
<script type="text/javascript">
$(document).ready(function() {
  $('body').append('<div style="position:fixed; top:20px; left:95%; width:100; height:500; background:infobackground;  ";> <canvas id="outline" width="300" height="500"></canvas> </div>');
  $("#outline").fracs("outline", {
    crop: true,
    styles: [{
      selector: '.INFO',
      fillStyle: 'rgb(220,220,220)'
    }, {
      selector: '.CUSTOM1',
      fillStyle: 'rgb(240,140,060)'
    }, {
      selector: '.error',
      fillStyle: 'rgb(255, 0, 0)'
    }, {
      selector: '.success',
      fillStyle: 'rgb(0, 128, 0)'
    }]
  });
  ;
});
</script>
<style>

Get Your Fracs 

Download: http://larsjung.de/release/jquery.fracs/jquery.fracs-0.11.zip
extract it and place 'jquery.fracs-0.11.min.js' and 'jquery.outline-0.11.min.js' in your SAHI_HOME/userdata/logs/playback.

Run A Test With HtmlLog 

Enjoy Your Log 

 Open your new log in a browser

Works on: 

sahi_20110719 - kubuntu - chromium browser

Won't work 

 On viewing suite logs...

Page Objects

The whole code better formated: http://paste.robbi5.com/648ced42

For a few weeks, we spoke about page objects. Dasspunk summarized his approach, but I'd like to give it a more object orientated feeling. As we are working with Sahi, some things might look kinda repetitive and awkward, but they are necessary. But the worst thing is, Sahi open sorce doesn't follow the ECMAScript 5 standard, so instead of using:


var $HomePage=Object.create(PagePrototype, {
  helloWorld: function(){}
});
we are forced to use

var $HomePage=new PagePrototype()
$HomePage.helloWorld=function(){};
This adds some code overhead, but we have to live with it for now.

To start from the scratch, I'd like to explain my test runner. It basically just takes a two dimensional array, where the first line is the property name of the testSubject which will be crated.

function runTestV2($subjects, $test, $setUp, $tearDown) {
  for (var $i=1; $i<$subjects.length; $i++){
  var $thisSubject={};
  for(var $j=0; $i<$subjects[$i].length; $j++) {
    $thisSubject[$subjects[0][$j]]=$subjects[$i][$j];
  }
  try{
    $setUp();
    $test($thisSubject);
    $tearDown();
  } catch ($e) {
    _logExceptionAsFailure($e);
    $tearDown();
  }
}

So when you define your test data like

$testData=[
  ["url", "name"],
  ["url1", "Joe"],
  ["url2", "Michi"]
];
you can easily write $testSubject.url in you script. This provides some data driven feeling while your test data is still in your script and you don't have to have multiple excel sheets opened. I believe, it's better to have your data right in front of you than having to switch tabs all the time. What's important here is the try-catch-block, where you can define general test error handling, log the page objects or just make sure your teardown gets executed.

When developing page objects, you might encounter generic error messages like "Object [object] has no function helloWorld" or "variable $url is not defined". When you have a hundreds of page objects, it is pretty hard to tell, which object raised that error. That is why I defined a page prototype which will help locating errors:

function Page($name, $uniqueElement){
  var $this=this;
  $this.$uniqueElement=$uniqueElement;
  $this.$name=$name;
}
Page.prototype.pagePresent = function() {
  var $this=this;
  return _condition(_exists($this.$uniqueElement));
};
Page.prototype.ifPagePresent = function ($action) {
  try{
    var $this = this;
    if ( $this.pagePresent() ) {
      $action();
    } else {
      unsupportedForThisPage($action, $this);
    }
  } catch ($e) {
    _log("Page Object: "+JSON.stringify($this)+"\nURL: "+document.location.href, "failure");
    throw $e;
  }
};
Page.prototype.toString = function(){
  var $this=this;
  return "Page "+ $this.$name;
}

The pages constructor will take a name, and an element. For many actions, the page object will check if this element exists, done by calling "$Page.ifPagePresent(//do something)". This makes sure, that errors on displaying the website are instantly noticed, but it adds some overhead, increasing the tests run time a little bit. It might also be difficult to find a unique element, then just use something generic like "_div(0)" or "document".
I override the Objects toString function to tell us the name of the object. When there is an error in you code, maybe a typo in your variable name, the testRunner will catch the exception and instead of " Cannot find function HelloWorld in Object [object]" we get a meaningful error.
Lets define a page:

var $HomePage=new Page('home', '_div("welcome")') ;
$HomePage.helloWorld=function(){};
and call helloWorld

$HomePage.HelloWorld(); //typo
Output: "Cannot find function HelloWorld in Page home". Now we know where to look.

You might wonder, what unsupportedForThisPage() does. For meaningful errors, when we have a mistake in our test where the code is valid, but the test steps are messed up. When we attempt to do something, which is not possible on the given site, an exception will be thrown. The log will tell us about the page objects name, the sites url and the code of the function, we wanted to execute here.

function unsupportedForThisPage($action){
throw "Wrong page for function: " + $action +"\n" +
"Url: " + document.location.href;
}

Thats all I want for debugging. and the beginning of an OO approach. I am currently figuring out how much sense there is in defining abstract element prototypes for reoccurring things like page navigation, lists etc

Oh No ... Not Again

yeah the page object pattern is working pretty well..
so well that I think about refactoring all my tests... again -.- as I haven't spend enough time already on making the code beautiful ( = fast to write and easy to read )

we are getting a volunteer soon, so I may outsource this frustrating work :)

I will summarize the lessons I learned from dasspunk soon.

Cleanup

I have finally been promoted to a Sahi forum admin. After getting the needed permission, I have deleted around 40 spam posts. Veeeeery satisfying :)

Narayan offered a t-shirt for the inconvenience. I really appreciate that and it would be hilariously funny, if it was labeld:

"I kicked 25 spammers and all I got was this stupid t-shirt"

Checklist: Automating Tests With Sahi

You might encounter the situation where you employer points at you with a grim expression on his face. His daemonic voice seems to come from deep within your head:

"Our Usability Tests are too expensive. You have to automate them!"

Whether this scenario is realistic or not, you might evaluate Sahi for test automation. Here is what you should think about, when considering the Open Source version of Sahi for automating the usability tests for you web application.

The Environment

The Application Environment

Sahi is operating system and browser independant. Most certainly, the application under test is running in a browser. While the Sahi homepage states, that Sahi supports every browser and AJAX and highly dynamic web applications and ..., you might encounter some problems with the following technologies:

• Your application is only accessible via VPN or a proxy: This does not seem to work in a few cases, most of the time due to bad configuration and you can get good support in the forum. Other times... it just will not work. 
• Internet Explorer: You may have expected it, the Internet Explorer is kind of special as usual. Lately, some users report difficulties with browsing pages or invoking the Sahi Controller. Some recorded scripts work well on every browser but the IE. 
• Some Dojo Widgets are difficult to test with Sahi. Reports in the forum mention problems with the dijit.form.DateTextBox and some workarounds. 
• The GWT DatePicker is not fully accessible. 
• Some Record/Playback difficulties exist when testing an Extjs application 
• Flash Support: Just forget about it. 

Developing Environment

Sahi works on many browsers and operating systems. The Sahi Controller provides an easy way to record and playback tests. For serious automation you should develop your tests in a proper IDE. Sahi supports syntax highlighting and content assist in Eclipse, Notepad++, Textpad and jEdit.
Choose some kind of source control.
Where is the testdata stored? Sahi is capable of reading excel sheets and csv files but doesn't provide any database connection (Sahi Pro does).
Choose a language to develop your tests. Testscripts can be written in Ruby, Java or JavaScript.

Custom Libraries

It may be beneficial to wrap the Sahi APIs into your own library to easily add further functionality. You may want to add some error checking or logging.

Reporting

Write Your Own Reporter!

The Sahi log html files are good for evaluating single scripts, but in the long run you will end up searching a long list of tests and a huge html file for a specific line. Writing your own reporter gets even more important, when someone is evaluating the test outcome, who has no understanding for the Sahi APIs. This person may tell if a test failed or not, but what has been tested and how, one can not tell. There should be a possibility to generate a business report which is is understandable, even if the reader does not know.. or just does not care about the codebase.
You have different options to do so:

• Develop a log writer within the test codebase like this XML fragment writer. This may not suffice for your needs and adds a lot of overhead as you have to get the "result" from within the test. 
• Parse the Sahi generated logs. Sahi generates Html and Xml logs and you could use some text parsing tool to extract all the information you need. But you are limited to the information these logs provide and you have no influence on the changes these logs may go through in the upcoming Sahi versions. 
• While Sahi Pro has the capability to report to a database, this feature is missing in Sahi OS. Write you own reporter package and build it into the Sahi source code, which is not that easy. The source is not modular and it looks pretty hard to add features. For a partial open source project, it should be easier to attach a different reporter. When you know the different locations, where to patch in your reporter, you have full access to all information within and after a testrun. This too may get you into trouble on new releases. 

Test Case Management

Somehow you have to keep track of the test coverage, which testcases are implemented and how long it will approximately take to reach the coverage you are aiming to. A wiki, an Excel Sheet or a ticketing system might not be the right way to document your testplan. Adding, changing and removing testcases, scheduling and keeping track of results is pretty difficult. A database may be easier to maintain but is better used in combination with a Test Management Tool. A list of Open Source Software can be found here. I did not have the time to find out, which tool possibly is the best in combination with Sahi, so I am not recommending any tool. If you have some experiences with test management tools which work well with Sahi, please leave a comment.
Thanks for reading.
When you encounter any typos, if I raised a question somewhere or if you want me to explain some things more detailed, feel free to leave a comment.

Entwicklungen

In technischer Hinsicht hat sich bei der Sahi Entwicklung einiges getan. Nach einigen Problemen mit ffmpeg und nachdem ich mich eingehender mit dem Sourcecode beschäftigt habe, sind wir jetzt auf dem Stand, regelmäßig automatische Usability Tests durchzuführen, Videomaterial und Screenshots aufzunehmen und auszuwerten. Dem - gelinde gesagt unübersichtlichen - Sourcecode habe ich eine Datenbankverbindung beigebracht und Spiele nun mit dem Gedanken, die Ergebnisse mit Hilfe von Solr im Browser darzustellen. Zuerst aber arbeite ich mit BIRT, womit Manu schon seine ambivalenten Erfahrungen gemacht hat.

Eine umfassende Abdeckung mit automatischen Test zu erreichen, war sehr umständlich, doch nachdem wir einen Sprint darauf verwendet haben Testscripte zu schreiben, ist - wenn man allein die Anzahl der Testfälle betrachtet - nur noch wenig zu tun. Dafür sind die verbleibenden Testfälle schlecht zu automatisieren, Sonderfälle und erfordern sehr viel Benutzerinteraktion die nicht simuliert werden kann. In einigen Fällen kann Sahi jedoch unterstützend wirken.

Wenn es beispielsweise mehrere Benutzer erfordert, einen Testfall zu überprüfen, so könnte Sahi den Part eines Benutzers automatisiert abspielen lassen. Mit der _confirm und _lastConfirm API von Sahi könnten Anweisungen für den menschlichen Tester dargestellt werden, die dieser ausführt und dann mit einem Klick die Testausführung fortsetzt. So kann man auch mit 'ungelernten' Testern halbautomatisch wesentlich schneller Testen als mit welchen, die das System kennen und alles manuell prüfen.

Nach langer Pause finde ich nun in meinen Blog zurück. In der postlosen Pause ist einiges passiert, was hier keiner Erwähnung bedarf und anderes, was nur bedingt erwähnenswert ist, wie die kleineren und größeren Änderungen an dem Blog. 

Zum Beispiel ist des Datum der Posts jetzt ISO 8601 konform, alles ist etwas aufgehellt und minimalistisch. Natürlich eine Vorlage, doch nimmt es auch zuviel Zeit in Anspruch ein eigenes Design zu gestalten.

Auf technischer Ebene ist im Zusammenhang mit Sahi einiges passiert, Details folgen.

desperate times call for desperate measures

Did anyone noticed before, that the search for porn on the web increases significantly around holidays (evidence: http://www.google.de/trends?q=porn ). Must be the same reason the suicide rate is increasing around Christmas Holidays or is this just an urban legend?

The Guard Puzzle

LösungsScript.txt

Updates on Sahi

Sahi gibt es nun schon eine Weile in der Version 3.5 Sahi on Sourcforge
Dabei sind unter Anderem zwei nette APIs hinzugekommen, die sich aller Wahrscheinlichkeit nach noch als sehr nützlich erweisen werden. Zunächst hätten wir da das schöne _collect(), was uns erlaubt, eine Menge von DOM Elementen durch eine Variable zu referenzieren. So könnte man z.B. alle Headlines auf einer Seite (oder einem Teil) referenzieren durch var $allHeadlines=_collect("_heading1","//") und kann die Elemente abarbeiten indem man  $allHeadlines wie ein übliches Array behandelt var $text = $allHeadlines[3].textContent (speichert den Text der 4. Überschrift in $text). _count() gibt - wer hätte es gedacht - die Anzahl von bestimmten Elementen zurück, also z.B. _count("_link", "//") sollte die Anzahl aller Links auf einer Seite zurückgeben.

Nun könnte doch alles so einfach sein, doch selbst die Beispiele die der Entwickler sebst angibt,  geben einen Fehler zurück: [Exception] TypeError: Cannot call method 'match' of undefined. Derzeit wartet man also auf eine Antwort im Forum, um dieses Problem zu klären. Anbei ein Zitat des Ausbilders:

"Das Problem ist bloß, dass das ein Problem ist"

Morgen geht es für den Azubi übrigens nicht ins Büro, denn er wird seinen Arbeitstag auf der Internetworld Messe verbringen.

[Edit] Die Fehler treten nur im Sahicontroller auf, die Scripts laufen mit dem Code ausgezeichnet.