Omar AL Zabir Blog

Browsers make 2 concurrent AJAX calls at a time to a domain. If
you make 5 AJAX calls, browser is going to make 2 calls first, then
wait for any one of them to complete and then make another call
until all remaining 4 calls are complete. Moreover, you cannot
expect calls to execute in the same order as you make the calls.
Here’s why:

Here you see, call 3’s response download is quite big and thus
takes longer than Call 5. So, Call 5 actually gets executed before
Call 3.

So, the world of HTTP is unpredictable.

If 2 http calls somehow get stuck for too long, those two bad
calls are going to make some good calls expire too which in the
meantime got queued. Here’s a nice example:

function timeoutTest()

{

PageMethods.Timeout( { timeoutInterval : 3000, onMethodTimeout:
function() { debug.dump(“Call 1 timed out”); } } );

PageMethods.Timeout( { timeoutInterval : 3000, onMethodTimeout:
function() { debug.dump(“Call 2 timed out”); } } );

PageMethods.DoSomething( ‘Call 1’, { timeoutInterval : 3000,
onMethodTimeout: function() { debug.dump(“DoSomething 1 timed
out”); } } );

PageMethods.DoSomething( ‘Call 2’, { timeoutInterval : 3000,
onMethodTimeout: function() { debug.dump(“DoSomething 2 timed
out”); } } );

PageMethods.DoSomething( ‘Call 3’, { timeoutInterval : 3000,
onMethodTimeout: function() { debug.dump(“DoSomething 3 timed
out”); } } );

}

I am calling a method named “Timeout” on the server which does
nothing but to wait for a long time so that the call gets timed
out. After that I am calling a method which does not timeout. But
guess what the output is:

Only one call succeeded “Do Something 1”. Try again and you
might see this:

Now two calls succeeded. So, if at any moment, browser’s two
connections get jammed, then you can expect other waiting calls are
going to timeout also.

In Pageflakes, we used to get nearly 400 to 600 timeout error
reports from users’ browsers. We could never find out how this can
happen. First we suspected slow internet connection. But that
cannot happen for so many users. Then we suspected something is
wrong with the hosting providers network. We did a lot of network
analysis to find out whether there’s any problem on the network.
But we could not detect any. We used SQL Profiler to see whether
there’s any long running query which times out ASP.NET request
execution time. But no luck. We finally discovered that, it mostly
happened due to some bad calls which got stuck and made the good
calls expire too. So, we modified the Atlas Runtime and introduce
automatic retry on it and the problem disappeared completely.
However, this auto retry requires a sophisticated open heart bypass
surgery on Atlas Runtime javascript code which you have to perform
again and again whenever Microsoft releases newer version of Atlas
Runtime. You also can no longer use the
tag which produces Atlas runtime references instead you have to
manually put links to Atlas runtime and compatibility javascript
files. So, you better do auto retry yourself in your own code from
Day 1. On the onMethodTimeout method, just make one retry all the
time to be on the safe side.

Atlas provides you batch
call feature which combines multiple web service calls into one
call. It works transparently, you won’t notice anything nor do you
need not write any special code. Once you turn on the Batch
feature, all web service calls made within a duration gets batched
into one call. Thus saves roundtrip time and total response
time.

The actual response time might be reduced but the perceived
delay is higher. If 3 web service calls are batched, the 1st call
does not finish first. All 3 calls finish at the same time. If you
are doing some UI updates upon completion of each WS calls, it does
not happen one by one. All of the calls complete in one shot and
then the UI gets updated in one shot. As a result, you do not see
incremental updates on the UI, instead a long delay before the UI
updates. If any of the call, say the 3rd call downloads a lot of
data, user sees nothing happening until all 3 calls complete. So,
the duration of the 1st call becomes nearly the duration of the sum
of all 3 calls. Although actual total duration is reduced, but the
perceived duration is higher. Batch calls are handy when each call
is transmitting small amount of data. Thus 3 small calls gets
executed in one roundtrip.

Let’s work on a scenario where 3 calls are made one by one.
Here’s how the calls actually get executed.

The second call takes a bit time to reach the server because
first call is eating up bandwidth. The same reason it takes longer
to download. Browsers open 2 simultaneous connections to the
server. So at a time, only 2 calls are made. Once the second/first
call completes, the third call is made.

When these 3 calls are batched into one:

Here the total download time is reduced (if IIS compression
enabled) and there’s only one network latency overhead. All 3 calls
get executed on the server in one shot and the combined response is
downloaded in one call. But to the user, the perceived speed is
slower because all the UI update happens after the entire batch
call completes. The total duration the batch call will take to
complete will always be higher than 2 calls. Moreover, if you do a
lot of UI update one after another, Internet Explorer freezes for a
while giving user a bad impression. Sometimes expensive update on
the UI makes the browser screen go blank and white. But Firefox and
Opera does not have this problem.

Batch call has some advantages too. Total download time is less
than downloading individual call responses because if you use gzip
compression in IIS, the total result is compressed instead of
individually compressing each result. So, generally batch call is
better for small calls. But if a call is going to send a large
amount of data or is going to return say 20KB of response, then
it’s better not to use batch. Another problem with batch call is,
say 2 calls are very small but the 3rd call is quite big. If these
3 call gets batched, the smaller calls are going to suffer from
long delay due to the 3rd larger call.

This is the first question everyone asks me when they see
Pageflakes. Why not
Protopage or Dojo library? Microsoft Atlas is a very promising AJAX
library. They are putting a lot of effort on Atlas, making lots of
reusable components that can really save you a lot of time and give
your web application a complete face lift at reasonably low effort
on changes. It integrated with ASP.NET v very well and it is
compatible with ASP.NET Membership and Profile provider.

When we first started developing Pageflakes, Atlas was in
infant stage. We were only able to use the Page Method and
Webservice Method call feature of Atlas. We had to make our own
drag & drop, component architecture, popups, collapse/expand
features etc. But now you can have all these from Atlas and thus
save a lot of development time. The web service proxy feature of
Atlas is a marvel. You can point a < script> tag to a .asmx
file and you get a javascript class generated right out of the web
service definition. The Javascript class contains the exact methods
that you have on the web service class. This makes it really easy
to add/remove new webservices, add/remove methods in webservices
which does not require any changes on the client side. It also
offers a lot of control over the AJAX calls and provides rich
exception trapping feature on the javascript. Server side
exceptions are nicely thrown to client side javascript code and you
can trap it and show nicely formatted error messages to the user.
Atlas works really well with ASP.NET 2.0 eliminating the
integration problem completely. You need not worry about
authentication and authorization on page methods and web service
methods. So, you save a lot of code on the client side (of course
Atlas Runtime is huge for this reason) and you can concentrate more
on your own code then building up all these framework related
codes.

Recent version of Atlas works nicely with ASP.NET Membership and
Profile services giving you login/logout features from Javascript
without requiring page postbacks and you can read/write Profile
object directly from Javascript. This comes very handy when you
heavily use ASP.NET membership and profile providers in your web
application which we do at Pageflakes.

On earlier versions of Atlas, there was no way to make HTTP GET
calls. All calls were HTTP POST and thus quite expensive calls. Now
you can say which calls should be HTTP GET. Once you have HTTP GET,
you can utilize Http response caching features which I will explain
soon.

I will be writing about lots of Atlas tips and tricks. I am
assuming you are familiar with Atlas and you have already tried
some quick start tutorials and you know the concepts of Page
Method, Web service Proxy, Script Manager etc.

Here’s a scenario. We use Email address as user name in ASP.NET
2.0 Membership provider. There were several places where we used to
create user accounts using this:

We did not notice what it was doing. After some days, users
started complaining. This is what users said whose account was
automatically created by the above code:

“Hi,

I got the email invitation. I went to your site. I tried login,
it said user name or password is wrong. So, I tried Signup. Signup
said user name already taken. Then I went to forgot password to
retrieve the password. It shows something is wrong and password
email cannot be sent.

I am stuck. Please help!”

Here’s the problem. When we use the above code, it creates a row
in aspnet_users table using the email address as user name. Fine no
problem. But in aspnet_membership table, the row it creates
contains Email is NULL. So, user cannot use “Forgot Password”
option to request the password because the email address is null.
Out database contained 908 of such unfortunate users, so we had to
run the following SQL to fix it:

The applicationID is something which you need to specify for
your own application. You can find the ID from aspnet_application
table.

Then we changed the code to create user accounts to this:

The 3rd parameter is the email address. We did not notice
this.

We were having very poor performance when we turned on
transaction log shipping on our SQL Server. We are using SQL Server
2005. The transaction log file was around 30 GB because the
database was in Full Recovery mode. The server became very slow,
every 15 mins when we were doing the log shipping, it used to
become very slow and sometimes nonresponsive. The event log was
getting full of SqlTimeout exceptions generated by the web site.
The web site started to show asp.net error page very frequently. We
could not use SQL Server Management Studio to login to SQL Server
so that we could do something about it.

Here’s how the connection time was reported from an external
monitoring site:

The peaks are 30 seconds which mean they timed out.

So, here’s what we did:

1. Turned off Log shipping
2. Restarted SQL Server.
3. Switched Database to Simple recovery model. Shrunk the log
   file. This made the log file come down to couple of megabytes.
4. Ran for some days. All looked ok.
5. Then switched DB to Full Recovery model and configured log
   shipping again.

So far running fine. But we go down for an hour every Saturday
when we run INDEX DEFRAG on the indexes. The log ships show around
5 or 6 log backups which are each 1 or 2 GB in size when the index
defrag happens.

Such queries will happily run on your development
environment:

Or you can get some user’s profile without any problem
using:

Even you can nicely update a user’s email in aspnet_membership
table like this:

But when you have a giant database on your production server,
running any of these will bring your server down. The reason is,
although these queries look like very obvious ones that you will be
using frequently, none of these are part of any index. So, all of
the above results in “Table Scan” (worst case for any query) on
millions of rows on respective tables.

Here’s what happened to us. We used such fields like UserName,
Email, UserID, IsAnonymous etc on lots of marketing reports at
Pageflakes. These are some
reports which only marketing team use, no one else. Now, the site
runs fine but several times a day marketing team and users used
call us and scream “Site is slow”, “Users are reporting extreme
slow performance”, “Some pages are getting timed out” etc. Usually
when they call us, we tell them “Hold on, checking right now” and
we check the site thoroughly. We use SQL profiler to see what’s
going wrong. But we cannot find any problem anywhere. Profile shows
queries running file. CPU load is within parameters. Site runs nice
and smooth. We tell them on the phone, “We can’t see any problem,
what’s wrong?”

So, why can’t we see any slowness when we try to investigate the
problem but the site becomes really slow several times throughout
the day when we are not investigating?

Marketing team sometimes run those reports several times per
day. Whenever they run any of those queries, as the fields are not
part of any index, it makes server IO go super high and CPU also
goes super high – something like this:

We have SCSI drives which have 15000 RPM, very expensive, very
fast. CPU is Dual core Dual Xeon 64bit. Both are very powerful
hardware of their kind. Still these queries bring us down due to
huge database size.

But this never happens when marketing team calls us and we keep
them on the phone and try to find out what’s wrong. Because when
they are calling us and talking to us, they are not running
any of the reports which brings the servers down. They are
working somewhere else on the site, mostly trying to do the same
things complaining users are doing.

Let’s look at the indexes:

Table: aspnet_users
Clustered Index = ApplicationID, LoweredUserName
NonClustered Index = ApplicationID, LastActivityDate
Primary Key = UserID

Table: aspnet_membership
Clustered
Index = ApplicationID, LoweredEmail
NonClustered = UserID

Table: aspnet_Profile
Clustered
Index = UserID

Most of the indexes have ApplicationID in it. Unless you put
Application=’…’ in the WHERE clause, it’s not going to use
any of the indexes. As a result, all the queries will suffer from
Table Scan. Just put ApplicationID in the where clause (Find your
applicationID from aspnet_Application table) and all the queries
will become blazingly fast.

DO NOT use Email or UserName fields in WHERE clause.
They are not part of the index instead LoweredUserName and
LoweredEmail fields are in conjunction with ApplicationID field.
All queries must have ApplicationID in the WHERE
clause.

Our Admin site which contains several of such reports and each
contains lots of such queries on aspnet_users, aspnet_membership
and aspnet_Profile tables. As a result, whenever marketing team
tried to generated reports, they took all the power of the CPU and
HDD and the rest of the site became very slow and sometimes
non-responsive.

Make sure you always cross check all your queries WHERE and JOIN
clauses with index configurations. Otherwise you are doomed for
sure when you go live.

Here’s an interesting way to calculate the execution time of a
code block:

You get an output like this:

6/14/2006
10:58:26 AM
4b1f6098-8c9d-44a5-93d8-e37394b6ef18
SomeFunction
9.578125

You can measure execution time of not only a function, but also
smaller blocks of code. Whatever is inside the “using” block, gets
logged.

Here’s how the TimedLog class do the work:

We are using Enterprise Library to do the logging. You can use
anything you like on the Dispose method.

The benefit of such log is, we get a tab delimited file which we
can use to do many types of analysis using MS Excel. For example,
we can generate graphs to see how the performance goes up and down
during peak hours and non peak hours. We can also see whether there
are high response times or not and what is the pattern. All these
gives us valuable indications where the bottle-neck is. You can
also find out which calls take most of the time by doing sort on
the duration column.