7 hard earned lessons from a Single Page Application

My talk at London AJAX User group on some performance optimization techniques that can give a javascript rich Single Page Application big boost in terms of page load performance.

MyPicAJAXTalk

See the lecture here:

http://skillsmatter.com/podcast/ajax-ria/7-real-life-lessons-learnt-from-a-single-page-application/mh-6922

Here are the slides.

image

Fast Streaming Ajax Proxy with GET PUT POST DELETE

I have enhanced my streaming Ajax Proxy with POST, PUT and
DELETE features. Previously it supported only GET. Now it supports
all 4 popular methods for complete REST support. Using this proxy,
you can call REST API on external domain directly from your
website’s javascript code. You can test the proxy from this
link:


labs.omaralzabir.com/ajaxstreamingproxy/GetPutDeleteTest.aspx

The latest source code for the Ajax Proxy is available here:

http://code.google.com/p/fastajaxproxy/

You can find a detail CodeProject article that explains how the
streaming asynchronous aspect of this proxy works:

Fast, Scalable,
Streaming AJAX Proxy – continuously deliver data from across
domains

Here’s how the test UI looks like where you can test POST,
PUT and DELETE:


image

If you want to run the sample source code on your local IIS,
make sure you allow the POST, PUT, and DELETE headers on .ashx
extension from IIS properties:


image

The sample project shows how you can use the proxy to make calls
to external domains. You can directly hit any external URL and
perform POST or DELETE from your javascript code:

var proxyUrl = "StreamingProxy.ashx";
function download(method, proxyUrl, contentUrl, isJson, bodyContent, completeCallback) { var request = new Sys.Net.WebRequest(); if (method == "POST" || method == "PUT") request.set_httpVerb("POST"); else request.set_httpVerb("GET"); var url = proxyUrl + "?m=" + method +
(
isJson ? "&t=" + escape("application/json") : "") + "&u=" + escape(contentUrl); request.set_url(url); if (bodyContent.length > 0) { request.set_body(bodyContent); request.get_headers()["Content-Length"] = bodyContent.length; } var startTime = new Date().getTime(); request.add_completed(function(executor) { if (executor.get_responseAvailable()) { var content = executor.get_responseData(); var endTime = new Date().getTime(); var statistics = "Duration: " + (endTime - startTime) + "ms" + 'n' + "Length: " + content.length + " bytes" + 'n' + "Status Code: " + executor.get_statusCode() + " " + 'n' + "Status: [" + executor.get_statusText() + "]" + 'n'; appendStat(statistics); get('resultContent').value = content; completeCallback(); } }); var executor = new Sys.Net.XMLHttpExecutor(); request.set_executor(executor); executor.executeRequest(); }

I am using MS AJAX here. You can use jQuery to perform the same
test as well. All you need to do is hit the URL of the
StreamingProxy.ashx and pass the actual URL in query string
parameter “u” and pass the type of the http method in
query string parameter “m”. That’s it!

7 tips for for loading Javascript rich Web 2.0-like sites significantly faster

Introduction

When you create rich Ajax application, you use external
JavaScript frameworks and you have your own homemade code that
drives your application. The problem with well known JavaScript
framework is, they offer rich set of features which are not always
necessary in its entirety. You may end up using only 30% of jQuery
but you still download the full jQuery framework. So, you are
downloading 70% unnecessary scripts. Similarly, you might have
written your own javascripts which are not always used. There might
be features which are not used when the site loads
for the first time, resulting in unnecessary download during
initial load. Initial loading time is crucial – it can make
or break your website. We did some analysis and found that every
500ms we added to initial loading, we lost approx 30% traffic who
never wait for the whole page to load and just close browser or go
away. So, saving initial loading time, even by couple of hundred
milliseconds, is crucial for survival of a startup, especially if
it’s a Rich AJAX website.

You must have noticed Microsoft’s new tool Doloto
which helps solve the following problem:

Modern Web 2.0 applications, such as GMail, Live Maps, Facebook
and many others, use a combination of Dynamic HTML, JavaScript and
other Web browser technologies commonly referred as AJAX to push
page generation and content manipulation to the client web browser.
This improves the responsiveness of these network-bound
applications, but the shift of application execution from a
back-end server to the client also often dramatically increases the
amount of code that must first be downloaded to the browser. This
creates an unfortunate Catch-22: to create responsive distributed
Web 2.0 applications developers move code to the client, but for an
application to be responsive, the code must first be transferred
there, which takes time.

Microsoft Research looked at this problem and published
this
research paper in 2008
, where they showed how much improvement
can be achieved on initial loading if there was a way to split the
javascripts frameworks into two parts – one primary part
which is absolutely essential for initial rendering of the page and
one auxiliary part which is not essential for initial load and can
be downloaded later or on-demand when user does some action. They
looked at my earlier startup Pageflakes and reported:

2.2.2 Dynamic Loading: Pageflakes
A contrast to Bunny Hunt is the Pageflakes application, an
industrial-strength mashup page providing portal-like
functionality.
While the download size for Pageflakes is over 1 MB, its
initial
execution time appears to be quite fast. Examining network
activity
reveals that Pageflakes downloads only a small stub of code
with the initial page, and loads the rest of its code dynamically
in
the background. As illustrated by Pageflakes, developers today
can
use dynamic code loading to improve their web application’s
performance.
However, designing an application architecture that is
amenable to dynamic code loading requires careful consideration
of JavaScript language issues such as function closures,
scoping,
etc. Moreover, an optimal decomposition of code into
dynamically
loaded components often requires developers to set aside the
semantic
groupings of code and instead primarily consider the execution
order of functions. Of course, evolving code and changing
user workloads make both of these issues a software maintenance
nightmare.

Back in 2007, I was looking at ways to improve the initial load
time and reduce user dropout. The number of users who would not
wait for the page to load and go away was growing day by day as we
introduced new and cool features. It was a surprise. We thought new
features will keep more users on our site but the opposite
happened. Analysis concluded it was the initial loading time that
caused more dropout than it retained users. So, all our hard work
was essentially going to drain and we had to come up with something
ground breaking to solve the problem. Of course we had already
tried all the basic stuffs –
IIS compression
,
browser caching
, on-demand loading of JavaScript,
css and html
when user does something, deferred
JavaScript execution
– but nothing helped. The frameworks
and our own hand coded framework was just too large. So, the idea
tricked me, what if we could load functions inside a class in two
steps. First step will load the class with absolutely essential
functions and second step will inject more functions to the
existing classes.

I published a codeproject article which shows you 7 tricks to
significantly improve page load time even if you have large amount
of Javascript used on the page.

7
Tips for Loading JavaScript Rich Web 2.0-like Sites Significantly
Faster

  1. Use Doloto
  2. Split a Class into Multiple JavaScript Files
  3. Stub the Functions Which Aren’t Called During Initial Load
  4. JavaScript Code in Text
  5. Break UI Loading into Multiple Stages
  6. Always Grow Content from Top to Bottom, Never Shrink or
    Jump
  7. Deliver Browser Specific Script from Server

If you like these tricks, please vote for me!

ASP.NET AJAX testing made easy using Visual Studio 2008 Web Test

Visual Studio 2008 comes with rich Web Testing support, but
it’s not rich enough to test highly dynamic AJAX websites
where the page content is generated dynamically from database and
the same page output changes very frequently based on some external
data source e.g. RSS feed. Although you can use the Web Test Record
feature to record some browser actions by running a real browser
and then play it back. But if the page that you are testing changes
everytime you visit the page, then your recorded tests no longer
work as expected. The problem with recorded Web Test is that it
stores the generated ASP.NET Control ID, Form field names inside
the test. If the page is no longer producing the same ASP.NET
Control ID or same Form fields, then the recorded test no longer
works. A very simple example is in VS Web Test, you can say
“click the button with ID
ctrl00_UpdatePanel003_SubmitButton002”, but you cannot
say “click the 2nd Submit button inside the third
UpdatePanel”. Another key limitation is in Web Tests, you
cannot address Controls using the Server side Control ID like
“SubmitButton”. You have to always use the generated
Client ID which is something weird like
“ctrl_00_SomeControl001_SubmitButton”. Moreover, if you
are making AJAX calls where certain call returns some JSON or
updates some UpdatePanel and then based on the server
returned response, you want to make further AJAX calls or post the
refreshed UpdatePanel, then recorded tests don’t work
properly. You *do* have the option to write the tests hand coded
and write code to handle such scenario but it’s pretty
difficult to write hand coded tests when you are using
UpdatePanels because you have to keep track of the page
viewstates, form hidden variables etc across async post backs. So,
I have built a library that makes it significantly easier to test
dynamic AJAX websites and UpdatePanel rich web pages. There
are several ExtractionRule and ValidationRule
available in the library which makes testing Cookies, Response
Headers, JSON output, discovering all UpdatePanel in a page,
finding controls in the response body, finding controls inside some
UpdatePanel all very easy.

First, let me give you an example of what can be tested using
this library. My open source project Dropthings produces a Web
2.0 Start Page where the page is composed of widgets.


image

Each widget is composed of two UpdatePanel. There’s
a header area in each widget which is one UpdatePanel and
the body area is another UpdatePanel. Each widget is
rendered from database using the unique ID of the widget row, which
is an INT IDENTITY. Every page has unique widgets, with unique
ASP.NET Control ID. As a result, there’s no way you can
record a test and play it back because none of the ASP.NET Control
IDs are ever same for the same page on different visits. This is
where my library comes to the rescue.

See the web test I did:


image

This test simulates an anonymous user visit. When anonymous user
visits Dropthings for the first time, two pages are created with
some default widgets. You can also add new widgets on the page, you
can drag & drop widgets, you can delete a widget that you
don’t like.

This Web Test simulates these behaviors automatically:

  • Visit the homepage
  • Show the widget list which is an UpdatePanel. It checks
    if the UpdatePanel contains the BBC World widget.
  • Then it clicks on the “Edit” link of the “How
    to of the day” widget which brings up some options
    dynamically inside an UpdatePanel. Then it tries to change
    the Dropdown value inside the UpdatePanel to 10.
  • Adds a new widget from the Widget List. Ensures that the
    UpdatePanel postback successfully renders the new
    widget.
  • Deletes the newly added widget and ensures the widget is
    gone.
  • Logs user out.

If you want to learn details about the project, read my
codeproject article:

http://www.codeproject.com/KB/aspnet/aspnetajaxtesting.aspx

Please vote if you find this useful.

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

Dropthings
– my open
source
Web 2.0 Ajax Portal has gone through a technology
overhauling. Previously it was built using ASP.NET AJAX, a little
bit of Workflow Foundation and Linq to SQL. Now Dropthings boasts
full jQuery front-end combined with ASP.NET AJAX
UpdatePanel, Silverlight widget, full
Workflow Foundation implementation on the business
layer, 100% Linq to SQL Compiled Queries on the
data access layer, Dependency Injection and Inversion of Control
(IoC) using Microsoft Enterprise Library 4.1 and
Unity. It also has a ASP.NET AJAX Web Test
framework that makes it real easy to write Web Tests that simulates
real user actions on AJAX web pages. This article will walk you
through the challenges in getting these new technologies to work in
an ASP.NET website and how performance, scalability, extensibility
and maintainability has significantly improved by the new
technologies. Dropthings has been licensed for commercial use by
prominent companies including BT Business, Intel, Microsoft IS,
Denmark Government portal for Citizens; Startups like Limead and
many more. So, this is serious stuff! There’s a very cool
open source implementation of Dropthings framework available at
National
University of Singapore
portal.

Visit: http://dropthings.omaralzabir.com


Dropthings AJAX Portal

I have published a new article on this on CodeProject:

http://www.codeproject.com/KB/ajax/Web20Portal.aspx

Get the source code

Latest source code is hosted at Google code:

http://code.google.com/p/dropthings

There’s a CodePlex site for documentation and issue
tracking:

http://www.codeplex.com/dropthings

You will need Visual Studio 2008 Team Suite with Service Pack 1
and Silverlight 2 SDK in order to run all the projects. If you have
only Visual Studio 2008 Professional, then you will have to remove
the Dropthings.Test project.

New features introduced

Dropthings new release has the following features:

  • Template users – you can define a user who’s pages
    and widgets are used as a template for new users. Whatever you put
    in that template user’s pages, it will be copied for every
    new user. Thus this is an easier way to define the default pages
    and widgets for new users. Similarly you can do the same for a
    registered user. The template users can be defined in the
    web.config.
  • Widget-to-Widget communication – Widgets can send message
    to each other. Widgets can subscribe to an Event Broker and
    exchange messages using a Pub-Sub pattern.
  • WidgetZone – you can create any number of zones in any
    shape on the page. You can have widgets laid in horizontal layout,
    you can have zones on different places on the page and so on. With
    this zone model, you are no longer limited to the Page-Column model
    where you could only have N vertical columns.
  • Role based widgets – now widgets are mapped to roles so
    that you can allow different users to see different widget list
    using ManageWidgetPersmission.aspx.
  • Role based page setup – you can define page setup for
    different roles. For ex, Managers see different pages and widgets
    than Employees.
  • Widget maximize – you can maximize a widget to take full
    screen. Handy for widgets with lots of content.
  • Free form resize – you can freely resize widgets
    vertically.
  • Silverlight Widgets – You can now make widgets in
    Silverlight!

Why the technology overhauling

Performance, Scalability, Maintainability and Extensibility
– four key reasons for the overhauling. Each new technology
solved one of more of these problems.

First, jQuery was used to replace my personal hand-coded large
amount of Javascript code that offered the client side drag &
drop and other UI effects. jQuery already has a rich set of library
for Drag & Drop, Animations, Event handling, cross browser
javascript framework and so on. So, using jQuery means opening the
door to thousands of jQuery plugins to be offered on Dropthings.
This made Dropthings highly extensible on the client side.
Moreover, jQuery is very light. Unlike AJAX Control Toolkit jumbo
sized framework and heavy control extenders, jQuery is very lean.
So, total javascript size decreased significantly resulting in
improved page load time. In total, the jQuery framework, AJAX basic
framework, all my stuffs are total 395KB, sweet! Performance is
key; it makes or breaks a product.

Secondly, Linq to SQL queries are replaced with Compiled
Queries. Dropthings did not survive a load test when regular lambda
expressions were used to query database. I could only reach up to
12 Req/Sec using 20 concurrent users without burning up web server
CPU on a Quad Core DELL server.

Thirdly, Workflow Foundation is used to build operations that
require multiple Data Access Classes to perform together in a
single transaction. Instead of writing large functions with many
if…else conditions, for…loops, it’s better to
write them in a Workflow because you can visually see the flow of
execution and you can reuse Activities among different Workflows.
Best of all, architects can design workflows and developers can
fill-in code inside Activities. So, I could design a complex
operations in a workflow without writing the real code inside
Activities and then ask someone else to implement each Activity. It
is like handing over a design document to developers to implement
each unit module, only that here everything is strongly typed and
verified by compiler. If you strictly follow Single Responsibility
Principle for your Activities, which is a smart way of saying one
Activity does only one and very simple task, you end up with a
highly reusable and maintainable business layer and a very clean
code that’s easily extensible.

Fourthly, Unity
Dependency Injection (DI) framework is used to pave the path for
unit testing and dependency injection. It offers Inversion of
Control (IoC), which enables testing individual classes in
isolation. Moreover, it has a handy feature to control lifetime of
objects. Instead of creating instance of commonly used classes
several times within the same request, you can make instances
thread level, which means only one instance is created per thread
and subsequent calls reuse the same instance. Are these going over
your head? No worries, continue reading, I will explain later
on.

Fifthly, enabling API for Silverlight widgets allows more
interactive widgets to be built using Silverlight. HTML and
Javascripts still have limitations on smooth graphics and
continuous transmission of data from web server. Silverlight solves
all of these problems.

Read the article for details on how all these improvements were
done and how all these hot techs play together in a very useful
open source project for enterprises.

http://www.codeproject.com/KB/ajax/Web20Portal.aspx

Don’t forget to vote for me if you like it.

How to convince developers and management to use automated unit test for AJAX websites

Everyone agrees that unit testing is a good thing, we should all
write unit tests. We read articles and blogs to keep us up-to-date
on what’s going on in the unit test world so that we can
sound cool talking to peers at lunch. But when we really sit down
and try to write unit tests ourselves – “Naaah, this is
waste of time, let’s ask my QA to test it; that’s much
more reliable and guaranteed way to test this. What’s the
point testing these functions when there are so many other
functions that we should unit test first?” Had such moment
yourself or with someone else? Read on.

I had a conversation with our development lead Mike (using a
highly generic name since my
last post
caused some trouble), who runs “the show”
in our engineering team. As usual there was reservation in
introducing unit test to regular development schedule. Mike also
had valid points about lack of powerful tools for doing unit test
on AJAX websites. He also had confusion on ‘what’ and
‘how’ to unit test our code so that we aren’t
just testing database failures but real user actions that executes
both business and rendering logics. So, the discussion has a lot of
useful information, that will help you take the right decision when
you want to sell unit test to your ASP.NET and/or AJAX development
team and finally to higher management so that you can buy enough
time for the effort.

Friday, Jan 2007 – hallway
Omar
: Hey Mike, we need to start doing unit testing at
least on our web services. We are wasting way too much time on
manual QA. Since we are an AJAX shop, unit testing all our web
services should give us pretty well coverage.

Mike: Sure, that sounds fun. I will do some
feasibility check and see how can we chip this in into our next
sprint.

Friday, Feb 2007 – washroom
Omar: Hey Mike, let’s start doing unit
tests. I haven’t seen any tests last month. Can we start from
this sprint?

Mike: Sure, we can surely start from this
sprint. Let me find out which tool is the right one for us.

Friday, March 2007 – meeting room
Omar: Hey Mike, haven’t seen any unit tests
in the solution so far. Let’s seriously start writing unit
tests. Did you make any plan how you want to start unit testing the
webservices?

Mike: Yeah, I did some digging around and found
some tools. But most of them are for non-AJAX sites where you can
programmatically hit a URL or programmatically do HTTP POST on a
URL. You can also record button clicks and form posts from the
browser. There’s Visual
Studio’s Web Test
, which does pretty good job recording
regular ASP.NET site, but poor on AJAX sites. Moreover, you need to
buy Team Suite edition to get that Web Test feature. Besides,
recording tests and playing them back really does not help us
because all those tests contain hard coded data. We can’t
repeat a particular step many times with random data, at least not
using any off-the-shelf tools. We need to test things carefully and
systematically using random data set and sometimes use real data
from database. For example, a common scenario is loading 100 random
user accounts from database and programmatically log those users
into their portal and test whether the portal shows those
users’ personalized data. All these need to be done from
AJAX, without using any browser redirect or form post, because
there’s one page that allows user to login using Ajax call
and then dynamically renders the portal on the same page after
successful login. The UI is rendered by Javascript, so only a real
browser can render it and we have to test the output looking at the
browser.

Omar: I see, so you can’t use Visual
Studio Web Test to run unit tests on a browser because it does not
let you access the html that browser renders. You can only test the
html that’s returned by webserver. As we are AJAX website,
most of our stuffs are done by Javascripts – they call
Webservice and they render the UI. Hmm, thinking how we can do this
using VS. We can at least hit the webservices and see if they are
returning the right JSON. This way we can pretty much test the
entire webservice, business and data access layer. But it does not
really replace the need for manual QA since there’s a
lot of rendering logic in Javascript.

Mike: Now there’s a new project called
Watin that seems promising. You
can write C# code to instruct a browser to do stuffs like click on
a button, run some javascript and then you can check what the
browser rendered in its DOM and run your tests. But still,
it’s in its infancy. So, there’s really no good tool
for unit testing AJAX sites. Let’s stick to manual QA, which
is proven to be more accurate than anything developers can come up
with. We can handover a set of data to QA and ask them to enter and
check the result.

Omar: We definitely need to figure out ways to
reduce our dependency on manual QA. It simply does not scale. Every
sprint, we have to freeze code and then hand over to QA. They run
their gigantic test scripts for a whole day. Then next day, we get
bug reports to fix. If there’s severe regression bug we have
to either cancel sprint or work whole night to fix it and run
overnight QA to meet deployment date. For last one year, every
sprint we ended up having some bug that made dev and QA work over
night. We have to empower our developers with automated unit test
tool so that they can run the whole regression test script
automatically.

Mike: You are talking about a very long project
then. Writing so many unit tests for complete regression test is
going to be more than a month long project. We have to find the
right set of tool, plan what areas to unit test and how, then
engage both dev and QA to work together and prepare the right
tests. And then we have to keep the test suite up-to-date after
every sprint to catch the new bugs and features.

Omar: Yes, this is certainly a complex project.
We have to get to a stage that can empower a developer to run
automated unit tests and not ask QA to test every task for
regression bugs. In fact, we should have automated build that runs
all unit tests and does the regression test for us automatically
after every checkin.

Mike: We have automated build and deploy. So,
that’s done. We need to add automated unit test to it.
Seriously, given our product size, this is absolutely impossible to
engage in writing so many unit tests so that we can do the entire
regression test automatically. It’s not worth the time and
money. Our QA team is doing fine. They can take one day leave after
deployment when they do overnight work.

Omar: Actually QA team is at the edge of
quitting. They seem to have endless work load. After deployment,
they have to do manual regression test on production site to ensure
nothing broke on production. While they are at it, they have to
participate in sprint initiation meetings and write test plans.
When they are about to complete that, devs checkin stuffs and ask
for regression test of different modules. Before they can finish
that , we reach code freeze and they have to finish all those task
level tests as well as the entire regression test. So, they end up
working round-the-clock several days every sprint. They simply
can’t take it anymore.

Mike: How is it different than our life? After
spending sleepless night on the deployment date, next day we have
to attend 8 hours long sprint planning meeting. Then we have to
immediately start working on the tasks from the next day and have
to reach code-freeze within a week. Then QA comes up with so many
bugs at the last moment. We have to work round-the-clock last 3
days of sprint to get those bugs fixed. Then after a nerve wrecking
deployment day, we have to stay up at night to wait for QA to
report any critical bug and fix it immediately on production. We
are at the brink of destruction as well.

Omar: That’s understood. The whole team
is surely getting pushed to their limit. So, that’s why we
urgently need automated test so that it addresses the problems of
both dev and QA team. Dev will get tests done at a faster rate so
that they don’t get bug reports at the very end and then work
over-night to fix them. Similarly, we offload QA team’s
continuous overwork by letting the system do the bulk of their
test.

Mike: This is going to kill the team for sure.
We have so many product features and bug fixes to do every sprint.
Now, if we ask everyone to start writing unit tests for every task
they do, it’s a lot of burden. We can’t do both at the
same time.

Omar: Agree. We have to cut down product
features or bug fixes. We have to make room in every sprint to
write unit tests.

Mike: Good luck with that. Let’s see how
you convince product team.

Omar: First let me convince you. Are you
convinced that we should do it.

Mike: Not yet. I don’t really see its
fruit in near future, even after two months. There’s so many
features we have to do and so many customers to ship to, we just
can’t do enough unit tests that will really shed off QA load.
It’ll just be a distraction and delay in every sprint, heck,
in every task.

Omar: Let me show you a graph which I believe
is going to make an impact:


image

So, you see the more automates test we write, the less time
spent on Manual QA. That time can be spent on doing new tests or
task level tests and increase quality of every new feature shipped
and drastically reduce new bugs shipped to production. Thus we get
less and less bugs after every successful sprint.

Mike: Ya, I get it, you don’t need to
convince me for this. But I don’t see the benefit from
overall gain perspective. Are we shipping better product faster
over next two months? We aren’t. We are shipping less
features and bug fixes by spending a lot of time on writing unit
tests that has no impact on end-user.

Omar: Let me see if your assumption is
correct:


image

You see here, the more automated tests kick in, the more time QA
can spend on new features or new bugs. I agree that the speed of
testing new features/bugs decrease first one or two sprints, but
then they gradually get picked up and get even better. In the
beginning, there’s a big overhead of getting started with
automated test. But as sprints go by, the number of unit tests to
write gradually gets stable and soon it becomes proportional to new
features/bugs. No more time spent on writing tests for old stuff.
So, the number of unit tests you write after four sprints is
exactly what needed for the new tasks you did on that sprint.

Mike: Let’s see what if we just
don’t do any automated test and keep things manual. How does
the graph look like?


image

Omar: The future looks quite gloomy. We will be
spending so much time on regression test as we keep adding stuffs
to the product that at some point QA will end up doing regression
test full time. They will not spend time on new features and we
will end up having a lot more new bugs slipped from QA to
production due to lack of attention from QA.

Mike: OK, how do we start?

Omar: First step is to get the regression tests
done so that we can get rid of that 24 hour long marathon QA period
end of every sprint. Moreover, I see too many devs asking QA to do
regression test here and there after they commit some tasks. So, QA
is always doing regression tests from the beginning to the end of
each sprint. They should only test new things for which automated
test is not yet written and let the automated test do the existing
tests.

Mike: This will be hard to sell to management.
We are going to say “Look for next one month, we will be half
productive because we want to spend time automating our QA process
so that from second month, we can do tests automatically and QA can
have more free time.”

Omar: No, we say it like this, “We are
going to spend 50% of our time automating QA for next oen month so
that QA can spend 50% more time on testing new features. This will
prevent 50% new bugs from occurring every sprint. This will give
developers 50% more time to build new features after one
month.” We show them this graph:


image

Mike: Seems like this will sell. But for first
couple of sprints, we will be so dead slow that some of us might
get fired. Think about it, from management point of view, the
development team has suddenly become half productive. They
aren’t building only few new features and bugs are not
getting fixed as fast rate. Customer are screaming, investors
asking for money back. It’s going to get really dirty. Do you
want to take this risk?

Omar: I can see that this decision is a very
hard decision to take. I know what CEO will say, “We need to
be double productive from tomorrow, otherwise we might as well pack
our bags and go home. Tell me something that will make us double
productive from tomorrow, not half productive.” But you can
see what will happen after couple of months. Situation will be so
bad that doing this after couple of months will be out of question.
We won’t be in a position to even propose this. Now, at least
we can argue and they still have the mind to listen to long term
ideas. But in future, when our QA team is doing full time
regression test, new buggy features going to production, ratio of
new bugs increasing after every release, more customers screaming,
half baked features running on the production – we might have
to shut down the company to save our life.

Mike: We should have started doing automated
tests from day one.

Omar: Yes, unfortunately we haven’t and the more we delay,
the harder it is going to get. I am sure we will write automated
tests from day one in our next project, but we have to rescue this
project.

Mike: OK, I am sold. How do we start? We surely
need to unit test the business and data access layer. Do we start
writing unit test for every function in DAL and Business layer?

Omar: Writing unit test for DAL seems pointless
to me. Remember, we have very little time. We will get max two
sprints to automate unit tests. After that, we won’t get the
luxury to spend half of our time writing unit tests. We will have
to go back to our feature and bug fix mode. So, let’s spend
the time wisely. How about we only test the business layer
function?

Mike: So, we test functions like
CreateCustomer, EditCustomer, DeleteCustomer, AddNewOrder in
business layer?

Omar: Is that the final layer in business
layer? Is there another high level layer that aggregates such CRUD
like functions?

Mike: For many areas, it’s like CRUD, a
dumb wrapper on DAL with some minor validation and exception
handling. But there are places where there are complex functions
that do a lot of different DAL call. For example,
UpdateCustomerBalance – that calls a lot of DAL classes to
figure out customer’s current balance.

Omar: Does webservices call multiple business
classes? Do they act like another level that aggregates business
layer?

Mike: Yes, webservices are called mostly from
user actions and they generally call multiple business layer
classes to get the job done.

Omar: Where’s the caching done?

Mike: Webservice layer.

Omar: That sounds like a good place to start
unit testing. We will write small number of unit tests and still
test majority of business layer and data access classes and we
ensure validation, caching, exception handling code are working
fine.

Mike: But there are other tools and services
that call the business layer. For example, we have a windows
service running that directly calls the business layer.

Omar: Can we refactor it to call webservices
instead?

Mike: No, that’ll be like creating 10
more webservices. A lot more development effort.

Omar: OK, let’s write unit tests for
those business layer classes separately then. I suppose there will
be some overlap. Some webservice call will test those business
classes as well. But that’s fine. We *should be* unit testing
from business layer. But we don’t have time, so we are
starting from one level up. Webservices aren’t really
“unit” but you have to do what you have to do. At least
testing webservices will give us guarantee that we covered all user
actions under unit test.

Mike: Yes, testing webservices will at least
ensure user actions are tested. The background windows service is
not much of our headache. Now how do we test presentation logic? We
have ASP.NET pages and there’s all those Javascript rendering
code.

Omar: Let’s use Watin for that.

Mike: How to make that part of a unit test
suite?

Omar: Watin integrates nicely with NUnit,
mbUnit. mbUnit is pretty good. I used it before. It has more test
attributes and Assert functions than NUnit.

Mike: OK, so how do we unit test UI? A test
function will click on Login link, fill up the email, password box
and click “OK”. Then wait for one sec and then see if
Javascript has rendered the UI correctly?

Omar: Something like that. We can discuss later
exactly how we test it. But how do you test if UI is rendered
correctly?

Mike: We check from browser’s DOM for
user’s data like name, email, balance etc are available in
browser’s HTML.

Omar: Does that really test presentation logic?
What if the data is misplaced? What if due to CSS error, it does
not render correctly.

Mike: Well, there’s really no way to
figure it out if things are rendered correctly. We can ask the QA
guys to keep watching the UI while Watin runs the tests on the
browser. You can see on the browser what Watin is doing.

Omar: OK, that’s one way and certainly
faster than QA doing the whole step. But can it be done
automatically like matching browser’s screen with some
screenshot?

Mike: Yeah, we need AI for that.

Omar: Seriously, can we write a simple UI
capture and comparison tool? Say we take a screenshot of correct
output and then clear up some areas which can vary. Then Watin runs
the test, it takes the screenshot of current browser’s view
and then matches with some screenshot? Here’s the idea:


image

Say this is a template screenshot that we want to match with the
browser. We are testing Google’s search result page to ensure
the page always returns a particular result when we provide some
predefined query. So, when Watin runs the test and takes browser to
Google search result page, it takes a screenshot and ignores
whatever is on those gray area. Then it does a pixel by pixel match
on the rest of the template. So, no matter what the search query is
and no matter what ad Google serves on top of results, as long as
the first result is the one we are looking for, test passes.

Mike: As I said, this is AI stuff. Some highly
sophisticated being will be matching two screenshots to say, Yah,
they more or less match, test pass.

Omar: I think a pretty dumb bitmap matching
will work in many cases. Just an idea, think about it. This way we
can test if CSS is giving us pixel perfect result. QA takes a
screenshot of expected output and then let the automated test to
match with browser’s actual output.

Mike: OK, all good ideas. Let’s see how
much we can do. We will be starting from webservice unit testing.
Then we will gradually move to Watin based testing. Now it’s
time to sell this proposal to product team and then to management
team.

Omar: Yep, at least get the webservices tested,
that will catch a lot of bugs before QA spends time on testing.
Goal is to get as much testing done by developers, really fast,
automatically then letting QA spend time on them. Also we can
run those webservice unit tests in a load test suite and load test
the entire webservice layer. That’ll give us guaranty our
code is production quality and it can survive the high traffic.

Mike: Understood, see ya.

. . .

March 2008, Friday – The Code Freeze Day

Omar: Hey Mike, how are we doing this
sprint?

Mike: Pretty good. 3672 unit tests out of 3842
passed. We know why some of them failed. We can get them fixed
pretty soon and run the complete regression tests once during lunch
and once before we leave. QA has completed testing new features
pretty well yesterday and they can check again today. We got some
of the new features covered by unit tests as well. Rest we can
finish next sprint, no worries.

Omar: Excellent. Enjoy your weekend. See you on
Monday.

——————————

Suggested Reading:

Create REST API using ASP.NET MVC that speaks both Json and plain Xml

UPDATE: There’s a newer article on this that shows how to create a truly RESTful API and website using the same ASP.NET MVC code.

www.codeproject.com/KB/aspnet/aspnet_mvc_restapi.aspx

ASP.NET MVC Controllers can directly return objects and collections, without rendering a view, which makes it quite appealing for creating REST like API. The nice extensionless Url provided by MVC makes it handy to build REST services, which means you can create APIs with smart Url like “something.com/API/User/GetUserList”

There are some challenges to solve in order to expose REST API:

  • Based on who is calling your API, you need to be able to speak both Json and plain old Xml (POX). If the call comes from an AJAX front-end, you need to return objects serialized as Json. If it’s coming from some other client, say a PHP website, you need to return plain Xml.
  • Similarly you need to be able to understand REST, Json and plain Xml calls. Someone can hit you using REST url, someone can post a Json payload or someone can post Xml payload.

I have created an ObjectResult class which takes an object and generates Xml or Json output automatically looking at the Content-Type header of HttpRequest. AJAX calls send Content-Type=application/json. So, it generates Json as response in that case, but when Content-Type is something else, it does simple Xml Serialzation.

image

Here’s the ObjectResult that you can use from Controllers to return objects and it takes care of proper serialization method. Above shows the Json serialization, which is quite simple.XmlSerialization is a bit complex though:

image

Things to note here:

  • You have to force UTF8 encoding. Otherwise it produces UTF16 output.
  • XML Declaration is skipped because that’s not quite necessary. Wastes bandwidth. If you need it, turn it on.
  • I have turned on indenting for better readability. You can turn it off to save bandwidth.

Some of you might be boiling inside looking at my obscure coding style. I love this style! I am spoiled by jQuery. I wish there was a cQuery. I actually started writing one, but it never saw day light just like my hundred other open source attempts.

Now back to Object Serialization, we got the serialization done. Now you can return objects from Controller easily:

image

You can use the test web project to call these methods and see the result:

image

So far you have seen simple object and list serialization. A best practice is to return a common result object that has some status, message and then the real payload. It’s handy when you only need to return some error but no object or list. I use a common Result object that has three properties – ErrorCode (0 by default means success), Message (a string data type) andData which is the real object.

image

When you want to return only a result with error message, you can do this:

image

This produces a result like this:

image

No payload here. So, the return format is always consistent. Those who are consuming service can write a common Xml or Json parsing code to consume both success and failure response. Those who are building API for their website, I humbly request you to return consistent response for both success and failure. It makes our life so easier.

So, far we have only returned objects and lists. Now we need to accept Json and Xml payload, delivered via HTTP POST. Sometimes your client might want to upload a collection of objects in one shot for batch processing. So, they can upload objects using either Json or Xml format. There’s no native support in ASP.NET MVC to automatically parse posted Json or Xml and automatically map to Action parameters. So, I wrote a filter that does it.

image

This filter intercepts calls going to Action methods and checks whether client has posted Xml or Json. Based on what has been posted, it uses DataContractJsonSerializer or simpleXmlSerializer to convert the payload to objects or collections.

You use this attribute on Action methods like this:

image

The attribute expects a parameter name where it stores the deserialized object/collection. It also expects a root type that it needs to pass to the deserializer. If you are expecting a single object, specify typeof(SingeObject). If you are expecting a list of objects, specify an array of that object like typeof(SingleObject[])

You can test the project live at this URL:

http://labs.dropthings.com/MvcWebAPI

The code is also available at:

http://code.msdn.microsoft.com/MvcWebAPI

Enjoy!

————

Here’s an Eid gift for my believer brothers. Check out this amazing sitewww.quranexplorer.com/. You will get online recitation, translation – verse by verse. The recitation of Mishari Rashid is something you have to listen to to believe. Try these two recitations to see what I mean:

Sura 97 – Verse 1
Sura 114 – Verse 1

Press the “Play” icon at bottom left (hard to find).

kick it on DotNetKicks.com

Open Source ASP.NET 3.5 AJAX Portal – new and improved

Last week I released a new version of Dropthings, my open source
AJAX portal, that shows many fancy Web 2.0 features and showcases
extensive use of ASP.NET 3.5, Workflow Foundation, C# 3.0 new
language features, custom ASP.NET AJAX extenders, many performance
and scalability techniques. I have written
a book
on these topics as well.

The new version implements the following performance and
scalability improvement techniques:

Here’s how the new version looks:


Dropthings new version

Hope you like the new design and the performance and scalability
techniques that can significantly boost your ASP.NET website’s
quality. I highly recommend these techniques for ASP.NET websites.
These are easy to implement and makes a world of difference in
speed and smoothness for ASP.NET websites.

I am thinking about making an ASP.NET MVC version of this portal
using jQuery. Do you think it will be a hot area to explore?

Share this post :

kick it on DotNetKicks.com

ensure – Ensure relevant Javascript and HTML are loaded before using them

ensure allows you to load Javascript, HTML and CSS
on-demand, whenever they are needed. It saves you from writing a
gigantic Javascript framework up front so that you can ensure all
functions are available whenever they are needed. It also saves you
from delivering all possible html on your default page (e.g.
default.aspx) hoping that they might some day be needed on some
user action. Delivering Javascript, html fragments, CSS during
initial loading that is not immediately used on first view makes
initial loading slow. Moreover, browser operations get slower as
there are lots of stuff on the browser DOM to deal with. So,
ensure saves you from delivering unnecessary
javascript, html and CSS up front, instead load them on-demand.
Javascripts, html and CSS loaded by ensure remain in
the browser and next time when ensure is called with
the same Javascript, CSS or HTML, it does not reload them and thus
saves from repeated downloads.

Ensure supports jQuery,
Microsoft ASP.NET AJAX and
Prototype framework. This
means you can use it on any html, ASP.NET, PHP, JSP page that uses
any of the above framework.

For example, you can use ensure to download
Javascript on demand:

ensure( { js: "Some.js" }, function()
{
SomeJS(); // The function SomeJS is available in Some.js only
});

The above code ensures Some.js is available before executing the
code. If the SomeJS.js has already been loaded, it executes the
function write away. Otherwise it downloads Some.js, waits until it
is properly loaded and only then it executes the function. Thus it
saves you from deliverying Some.js upfront when you only need it
upon some user action.

Similarly you can wait for some HTML fragment to be available,
say a popup dialog box. There’s no need for you to deliver HTML for
all possible popup boxes that you will ever show to user on your
default web page. You can fetch the HTML whenever you need
them.

ensure( {html: "Popup.html"}, function()
{
// The element "Popup" is available only in Popup.html
document.getElementById("Popup").style.display = "";
});

The above code downloads the html from “Popup.html” and adds it
into the body of the document and then fires the function. So, you
code can safely use the UI element from that html.

You can mix match Javascript, html and CSS altogether in one
ensure call. For example,

ensure( { js: "popup.js", html: "popup.html", css: "popup.css" }, function()
{
PopupManager.show();
});

You can also specify multiple Javascripts, html or CSS files to
ensure all of them are made available before executing the
code:

ensure( { js: ["blockUI.js","popup.js"], html: ["popup.html", "blockUI.html"], css: ["blockUI.css", "popup.css"] }, function()
{
BlockUI.show();
PopupManager.show();
});

You might think you are going to end up writing a lot of
ensure code all over your Javascript code and result
in a larger Javascript file than before. In order to save you
javascript size, you can define shorthands for commonly used
files:

var JQUERY = { js: "jquery.js" };
var POPUP = { js: ["blockUI.js","popup.js"], html: ["popup.html", "blockUI.html"], css: ["blockUI.css", "popup.css"] };
...
...
ensure( JQUERY, POPUP, function() {
("DeleteConfirmPopupDIV").show();
});
...
...
ensure( POPUP, function()
{
("SaveConfirmationDIV").show();
);

While loading html, you can specify a container element where
ensure can inject the loaded HTML. For example, you can say load
HtmlSnippet.html and then inject the content inside a DIV named
“exampleDiv”

ensure( { html: ["popup.html", "blockUI.html"], parent: "exampleDiv"}, function(){});

You can also specify Javascript and CSS that will be loaded
along with the html.

How ensure works

The following CodeProject article explains in detail how ensure
it built. Be prepared for a high dose of Javascript techniques:

http://www.codeproject.com/KB/ajax/ensure.aspx

If you find ensure useful, please vote for me.

Download Code

Download latest source code from CodePlex: www.codeplex.com/ensure

Share this post :

kick it on DotNetKicks.com