WCF does not support compression out of the box, so fix it

WCF service and client do not support HTTP Compression out of the box in .NET 3.5 even if you turn on Dynamic Compression in IIS 6 or 7. It has been fixed in .NET 4 but those who are stuck with .NET 3.5 for foreseeable future, you are out of luck.  First of all, it’s IIS fault that it does not enable http compression for SOAP messages even if you turn on Dynamic Compression in IIS 7. Secondly, it’s WCF’s fault that it does not send the Accept-Encoding: gzip, deflate header in http requests to the server, which tells IIS that the client supports compression. Thirdly, it’s again WCF fault that even if you make IIS to send back compressed response, WCF can’t process it since it does not know how to decompress it. So, you have to tweak IIS and System.Net factories to make compression work for WCF services. Compression is key for performance since it can dramatically reduce the data transfer from server to client and thus give significant performance improvement if you are exchanging medium to large data over WAN or internet.

There are two steps – first configure IIS, then configure System.Net. There’s no need to tweak anything in WCF like using some Message Interceptor to inject HTTP Headers as you find people trying to do here, here and here.

Configure IIS to support gzip on SOAP respones

After you have enabled Dynamic Compression on IIS 7 following the guide, you need to add the following block in the <dynamicTypes> section of applicationHost.config file inside C:WindowsSystem32inetsrvconfig folder. Be very careful about the space in mimeType. They need to be exactly the same as you find in response header of SOAP response generated by WCF services.

<add mimeType="application/soap+xml" enabled="true" />
<add mimeType="application/soap+xml; charset=utf-8" enabled="true" />
<add mimeType="application/soap+xml; charset=ISO-8895-1" enabled="true" />

After adding the block, the config file will look like this:

image

For IIS 6, first you need to first enable dynamic compression and then allow the .svc extension so that IIS compresses responses from WCF services.

Next you need to make WCF send the Accept-Encoding: gzip, deflate header as part of request and then support decompressing a compressed response.

Send proper request header in WCF requests

You need to override the System.Net default WebRequest creator to create HttpWebRequest with compression turned on. First you create a class like this:

public class CompressibleHttpRequestCreator : IWebRequestCreate
{
public CompressibleHttpRequestCreator()
{
}

WebRequest IWebRequestCreate.Create(Uri uri)
{
HttpWebRequest httpWebRequest =
Activator.CreateInstance(typeof(HttpWebRequest),
BindingFlags.CreateInstance | BindingFlags.Public |
BindingFlags.NonPublic | BindingFlags.Instance,
null, new object[] { uri, null }, null) as HttpWebRequest;

if (httpWebRequest == null)
{
return null;
}

httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip |
DecompressionMethods.Deflate;

return httpWebRequest;
}
}

Then on the WCF Client application’s app.config or web.config, you need to put this block inside system.net which tells system.net to use your factory instead of the default one.

<system.net>
<webRequestModules>
<remove prefix="http:"/>
<add prefix="http:" type="WcfHttpCompressionEnabler.CompressibleHttpRequestCreator, WcfHttpCompressionEnabler,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
/>
</webRequestModules>
</system.net>

That’s it.

I have uploaded a sample project which shows how all these works.

Safely deploying changes to production servers

When you deploy incremental changes on a production server, which is running and live all the time, you some times see error messages like “Compiler Error Message: The Type ‘XXX’ exists in both…”. Sometimes you find Application_Start event not firing although you shipped a new class, dll or web.config. Sometimes you find static variables not getting initialized and so on. There are so many weird things happen on webservers when you incrementally deploy changes to the server and the server has been up and running for several weeks. So, I came up with a full proof house keeping steps that we always do whenever we deploy some incremental change to our websites. These steps ensure that the web sites are properly recycled , cached are cleared, all the data stored at Application level is initialized.

First of all you should have multiple web servers behind a load balancer. This way you can take one server out of the production traffic, do your deployment and house keeping tasks like restarting IIS, and then put it back. Then you can do it for the second server and so on. This ensures there’s no outage for customer. If you can do it reasonable fast, hopefully customers won’t notice discrepancy between the servers some having new code and some having old code. You should only do this when your changes aren’t drastic. For ex, you aren’t delivering a complete revamped UI. In that case, some users hitting server1 with latest UI will suddenly get a completely different experience and then on next page refresh, they might hit server2 with old code and get a totally different experience. This works for incremental non-dramatic changes only.

image

During deployment you should follow these steps:

  • Take server X out of load balancer so that it does not get any traffic.
  • Stop all your .NET windows services on the server.
  • Stop IIS.
  • Delete the Temporary ASP.NET folders of all .NET versions incase you have multiple .NET versions running. You can follow this link.
  • Deploy the changes.
  • Flush any distributed cache you have, for ex, Velocity or Memcached.
  • Start IIS.
  • Start your .NET windows services on the server.
  • Warm up all websites by hitting major URLs on the websites. You should have some automated script to do this. You can use tinyget to hit some major URLs, especially pages that take a lot of time to compile. Read my post on keeping websites warm with zero coding.
  • Put server X back to load balancer so that it starts receiving traffic.

That’s it. It should give you a clean deployment and prevent unexpected errors. You should print these steps and hang on the desk of your deployment guys so that they never forget during deployment pressure.

Doing all these steps manually is risky. Under deployment time pressure, your production guys can make mistakes and screw up a server for good. So, I always prefer having a batch file that takes a server out and makes it ready for deploying code and then after the deployment is done, use another batch file to put the server back into load balancer traffic rotation after the server is warmed up.

Generally load balancers are configured to hit some page on your website and keep the server alive if that page returns a HTTP 200. If not, it assumes the server is dead and takes it our of rotation. For ex, say you have an alive.txt file on your website which is what load balancer is keeping an eye on. If it’s gone, the server is put out of the rotation. In that case, you can create some batch files that will take the server out, wait for couple of seconds to ensure the in-flight requests complete and then stop IIS, delete temporary ASP.NET files and make server ready to deploy stuff. Something like this:

serverout.bat
=====================
Ren alive.txt dead.txt
typeperf "ASP.NET Applications(__Total__)Requests Executing" -sc 30
iisreset /stop
rmdir /q /s "C:WINDOWSMicrosoft.NETFramework64v1.1.4322Temporary ASP.NET Files"
rmdir /q /s "C:WINDOWSMicrosoft.NETFramework64v2.0.50727Temporary ASP.NET Files"
md "C:WINDOWSMicrosoft.NETFramework64v1.1.4322Temporary ASP.NET Files"
md "C:WINDOWSMicrosoft.NETFramework64v2.0.50727Temporary ASP.NET Files"
xcacls "C:WINDOWSMicrosoft.NETFramework64v1.1.4322Temporary ASP.NET Files" /E /G MYMACHINEIIS_WPG:F /Q
xcacls "C:WINDOWSMicrosoft.NETFramework64v2.0.50727Temporary ASP.NET Files" /E /G MYMACHINEIIS_WPG:F /Q

Similarly you should have a batch file that starts IIS, warms up some pages, and then puts the server back into load balancer.

serverin.bat
============
SET TINYGET=C:Program Files (x86)IIS ResourcesTinyGettinyget.exe
iisreset /start"%TINYGET%" -srv:localhost -uri:http://localhost/ -status:200
ren dead.txt alive.txt
typeperf "ASP.NET Applications(__Total__)Requests Executing" -sc 30

Always try to automate this kind of admin chores. It’s difficult to do it right all the time manually under deployment pressure.

Quick ways to boost performance and scalability of ASP.NET, WCF and Desktop Clients

There are some simple configuration changes that you can make on machine.config and IIS to give your web applications significant performance boost. These are simple harmless changes but makes a lot of difference in terms of scalability. By tweaking system.net changes, you can increase the number of parallel calls that can be made from the services hosted on your servers as well as on desktop computers and thus increase scalability. By changing WCF throttling config you can increase number of simultaneous calls WCF can accept and thus make most use of your hardware power. By changing ASP.NET process model, you can increase number of concurrent requests that can be served by your website. And finally by turning on IIS caching and dynamic compression, you can dramatically increase the page download speed on browsers and and overall responsiveness of your applications.

Read the CodeProject article for more details.

http://www.codeproject.com/KB/webservices/quickwins.aspx

image

Please vote for me if you find the article useful.

Dynamically set WCF Endpoint in Silverlight

When you add a WCF service reference to a Silverlight Application, it generates the ServiceReference.ClientConfig file where the URL of the WCF endpoint is defined. When you add the WCF service reference on a development computer, the endpoint URL is on localhost. But when you deploy the Silverlight client and the WCF service on a production server, the endpoint URL no longer is on localhost instead on some domain. As a result, the Silverlight application fails to call the WCF services. You have to manually change the endpoint URL on the Silverlight config file to match the production URL before deploying live. Now if you are deploying the Silverlight application and the server side WCF service as a distributable application where customer install the service themselves on their own domain then you don’t know what will be the production URL. As a result, you can’t rely on the ServiceReference.ClientConfig. You have to dynamically find out on which domain the Silverlight application is running and what will be the endpoint URL of the WCF service. Here I will show you an approach to dynamically decide the endpoint URL.

First you add a typical service reference and generate a ServiceReference.ClientConfig that looks like this:

<configuration>
    <system.serviceModel>
        <bindings>
            <basicHttpBinding>
                <binding name="BasicHttpBinding_ProxyService" maxBufferSize="2147483647"
                    maxReceivedMessageSize="2147483647">
                    <security mode="None" />
                </binding>
                <binding name="BasicHttpBinding_WidgetService" maxBufferSize="2147483647"
                    maxReceivedMessageSize="2147483647">
                    <security mode="None" />
                </binding>
            </basicHttpBinding>
        </bindings>
        <client>
            <endpoint address="http://localhost:8000/Dropthings/API/Proxy.svc/pox"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_ProxyService"
                contract="DropthingsProxy.ProxyService" name="BasicHttpBinding_ProxyService" />
            <endpoint address="http://localhost:8000/Dropthings/API/Widget.svc/pox"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_WidgetService"
                contract="DropthingsWidgetService.WidgetService" name="BasicHttpBinding_WidgetService" />
        </client>
    </system.serviceModel>
</configuration>

As you see, all the URL are pointing to localhost, on my development environment. The Silverlight application now need to dynamically decide what URL the Silverlight app is running from and then resolve the endpoint URL dynamically.

I do this by creating a helper class that checks the URL of the Silverlight application and then decides what’s going to be the URL of the endpoint.

public class DynamicEndpointHelper
{
    // Put the development server site URL including the trailing slash
    // This should be same as what's set in the Dropthings web project's
    // properties as the URL of the site in development server
    private const string BaseUrl = "http://localhost:8000/Dropthings/";

    public static string ResolveEndpointUrl(string endpointUrl, string xapPath)
    {
        string baseUrl = xapPath.Substring(0, xapPath.IndexOf("ClientBin"));
        string relativeEndpointUrl = endpointUrl.Substring(BaseUrl.Length);
        string dynamicEndpointUrl = baseUrl + relativeEndpointUrl;
        return dynamicEndpointUrl;
    }
}

In the Silverlight app, I construct the Service Client this way:

private DropthingsProxy.ProxyServiceClient GetProxyService()
{
    DropthingsProxy.ProxyServiceClient service = new DropthingsProxy.ProxyServiceClient();
    service.Endpoint.Address = new EndpointAddress(
        DynamicEndpointHelper.ResolveEndpointUrl(service.Endpoint.Address.Uri.ToString(),
        App.Current.Host.Source.ToString()));
    return service;
}

After creating the service client with default setting, it changes the endpoint URL to the currently running website’s URL. This solution works when the WCF services are exposed from the same web application. If you have the WCF services hosted on a different domain and you are making cross domain calls to the WCF service then this will not work. In that case, you will have to find out what’s the domain of the WCF service and then use that instead of localhost.

Ten Caching Mistakes that Break your App

 

Caching frequently used objects, that are expensive to fetch from the source, makes application perform faster under high load. It helps scale an application under concurrent requests. But some hard to notice mistakes can lead the application to suffer under high load, let alone making it perform better, especially when you are using distributed caching where there’s separate cache server or cache application that stores the items. Moreover, code that works fine using in-memory cache can fail when the cache is made out-of-process. Here I will show you some common distributed caching mistakes that will help you make better decision when to cache and when not to cache.

Here are the top 10 mistakes I have seen:

  1. Relying on .NET’s default serializer.
  2. Storing large objects in a single cache item.
  3. Using cache to share objects between threads.
  4. Assuming items will be in cache immediately after storing it.
  5. Storing entire collection with nested objects.
  6. Storing parent-child objects together and also separately.
  7. Caching Configuration settings.
  8. Caching Live Objects that has open handle to stream, file, registry, or network.
  9. Storing same item using multiple keys.
  10. Not updating or deleting items in cache after updating or deleting them on persistent storage.

Let’s see what they are and how to avoid them.

http://www.codeproject.com/KB/web-cache/cachingmistakes.aspx

Please vote if you find this useful.

Dropthings now available from Microsoft/Web

Dropthings is now available on Microsoft/Web. You can now install it using the Web Platform Installer. I will soon write an article how to make an installer that can install a ASP.NET website, a SQL Server database, setup web.config files, setup directory permissions (eg App_Data) etc. It wasn’t straightforward and I learnt some best practices from the Microsoft/Web team. But for now, go ahead and download the app and build cool sites out of it.image

http://www.microsoft.com/web/gallery/Dropthings.aspx

Please vote for my app. It will help spread the open source effort. 

How to make screencasts in optimized animated GIF for free

I have been using animated GIFs to show short screencasts in my blogs and articles. Animated GIF is supported by all browsers and supports virtually any website in the world where even Flash is blocked. A picture is worth a thousand words, and an animation is worth a thousand multiplied by [frames in animation] words. So, I have been looking for a complete free solution to capturing screencasts and then converting it to animated GIF and then heavily compressing it.

First use CamStudio to capture the screenshot into an AVI. Before you capture, you need to set the CamStudio video recording setting to one frame per second, otherwise there will be too many frames in your animated GIF. You can set it so 2 or more frames per second if you are recording some frequent changes on the screen.

image

This will put one frame in animated GIF per second. Since animated GIF gets pretty large due to its lossless primitive compression, you need to put as little frames on it as possible.

Now you can record screenshot using CamStudio and save it in an AVI file.

Once you have the AVI file, you need to open the AVI using Microsoft GIF animator.

image

Then you need to click the “Select all” button and go to Image tab and put 100 on the Duration. This will set each frame delay to 1 second, exactly what you have set in the CamStudio Video Options. If you have set 2 frames per second in CamStudio, then you need to set 50 in Microsoft GIF Animator.

Now you can save the file as an animated GIF and use it wherever you like.

I would highly recommend you further optimize the animated GIF and eliminate duplicate frames and use some advanced compression. For this you can use the ImageMagick utility. You will find various ways to optimize animated GIF on this page. I just use the following command line and it gives me pretty good output:

c:Program Files (x86)ImageMagick-6.6.3-Q16>convert SourceImage.gif -layers OptimizePlus DestImage.gif

This optimizes animated GIFs pretty well. I have seen average 60% reduction on screen captures having white background and when there’s no translucent areas (eg Windows Vista/7 title bars).

Production Challenges of ASP.NET Website – recording of my talk

“It works in my PC”, the common dialog from Developers. But on production CPU burns out, disks go crazy, your site stops responding every now and then, you have to frequently recycle application pool, or even restart windows to bring things back to normal for a while.

As your site grows from thousands to millions of users, delivering new features and bug fixes becomes an increasingly complex process where you have to ensure the performance and stability of the site is not compromised. Let me tell you some stories from the trenches how I grew my hot startup Pageflakes (acquired by MySpace founder Brad Greenspan) from thousands to millions of visits within a year surviving through production software, hardware, process, people challenges and how you can avoid and solve them using my handy collection of articles and tools.

Watch the presentation I made in a LIDNUG session:

http://www.lidnug.org/Archives.aspx

Find the session with the title “Production Challenges of ASP.NET Website”.

Unfortunately they could not record the whole session. But whatever is there should be useful for you.

Building High Performance Queue in Database for storing Orders, Notifications, Tasks

We have Queues everywhere. There are queues for asynchronously sending notifications like email and SMS in most websites. E-Commerce sites have queues for storing orders, processing and dispatching them. Factory Assembly line automation systems have queues for running tasks in parallel, in a certain order. Queue is a widely used data structure that sometimes have to be created in a database instead of using specialized queue technologies like MSMQ. Running a high performance and highly scalable queue using database technologies is a big challenge and it’s hard to maintain when the queue starts to get millions of rows queued and dequeued per day. Let me show you some common design mistakes made in designing Queue-like tables and how to get maximum performance and scalability from a queue implemented using simple database features.

Queue

Let’s first identify the challenges you have in such queue tables:

  • The table is both read and write. Thus queuing and dequeuing impact each other and cause lock contention, transaction deadlocks, IO timeouts etc under heavy load.
  • When multiple receivers try to read from the same queue, they randomly get duplicate items picked out of the queue, thus resulting in duplicate processing. You need to implement some kind of high performance row lock on the queue so that same item never gets picked up by concurrent receivers.
  • The Queue table needs to store rows in certain order and read in certain order, which is an index design challenge. It’s not always first in and first out. Sometimes Orders have higher priority and need to be processed regardless of when they are queued.
  • The Queue table needs to store serialized objects in XML or binary form, which becomes a storage and index rebuild challenge. You can’t rebuild index on the Queue table because it contains text and/or binary fields. Thus the tables keep getting slower and slower every day and eventually queries start timing out until you take a downtime and rebuild the indexes.
  • During dequeue, a batch of rows are selected, updated and then returned for processing. You have a “State” column that defines the state of the items. During dequeue, you select items of certain state. Now State only has a small set of values eg PENDING, PROCESSING, PROCESSED, ARCHIVED. As a result, you cannot create index on “State” column because that does not give you enough selectivity. There can be thousands of rows having the same state. As a result, any dequeue operation results in a clustered index scan that’s both CPU and IO intensive and produces lock contention.
  • During dequeue, you cannot just remove the rows from table because that causes fragmentation in the table. Moreover, you need to retry orders/jobs/notification N times  incase they fail on first attempt. This means rows are stored for longer period, indexes keep growing and dequeue gets slower day by day.
  • You have to archive processed items from the Queue table to a different table or database, in order to keep the main Queue table small. That means moving large amount of rows of some particular status to another database. Such large data removal leaves the table highly defragmented causing poor queue/dequeue performance.
  • You have a 24×7 business. You have no maintenance window where you can take a downtime and archive large number of rows. This means you have to continuously archive rows without affecting production queue-dequeue traffic.

If you have implemented such queue tables, you might have suffered from one or more of the above challenges. Let me give you some tips on how to overcome these challenges and how to design and maintain a high performance queue table.

Read the article for details:

http://www.codeproject.com/KB/database/fastqueue.aspx

Please vote if you find this useful.