Prevent ASP.NET cookies from being sent on every css, js, image request

ASP.NET generates some large cookies if you are using ASP.NET membership provider. Especially if you are using the Anonymous provider, then a typical site will send the following cookies to every request when a user is logged in, whether the request is to a dynamic page or to any static resource:

.DBANON=w3kYczsH8Wvzs6MgryS4JYEF0N-8ZR6aLRSTU9KwVaGaydD6WwUHD7X9tN8vBgjgzKf3r3SJHusTYFjU85y
YfnunyCeuExcZs895JK9Fk1HS68ksGwm3QpxnRZvpDBAfJKEUKee2OTlND0gi43qwwtIPLeY1;
ASP.NET_SessionId=bmnbp155wilotk45gjhitoqg; DBAUTH12=2A848A8C200CB0E8E05C6EBA8059A0DBA228FC5F6EDD29401C249D2
37812344C15B3C5C57D6B776037FAA8F14017880E57BDC14A7963C58B0A0B30229
AF0123A6DF56601D814E75525E7DCA9AD4A0EF200832B39A1F35A5111092F0805B
0A8CD3D2FD5E3AB6176893D86AFBEB68F7EA42BE61E89537DEAA3279F3B576D0C
44BA00B9FA1D9DD3EE985F37B0A5A134ADC0EA9C548D

There are 517 bytes of worthless data being sent to every css, js and images from the browser to your webserver!

You might think 517 bytes is peanut. Do the math:

  • Avg page has 40 requests to server. 40 x 517 bytes = 20 KB per page view.
  • 1M page views = 20 GB
  • That’s 20GB of data getting uploaded to your server for just 1M page views. It does not take millions of users to produce 1M page views. Around 100k+ users using your site every day can produce 1M page views every day.

Here’s how to prevent this:

  • Setup a new website and map a different subdomain to it. If your main site is www.yoursite.com then map static.yoursite.com to it.
  • Manually change all the <link>, <script>, <img> css url(…) and prefix each resource with http://static.yoursite.com
  • If you don’t want to do it manually, use this solution I have done before.
  • Add a Global.asax and in the EndRequest do this trick:
    HttpContext context = HttpContext.Current;
    if (context.Request.Url.ToString.StartsWith("http://static.yoursite.com")
    {
      List<string> cookiesToClear = new List<string>();
      foreach (string cookieName in context.Request.Cookies)
      {
        HttpCookie cookie = context.Request.Cookies[cookieName];
        cookiesToClear.Add(cookie.Name);
      }
    
      foreach (string name in cookiesToClear)
      {
        HttpCookie cookie = new HttpCookie(name, string.Empty);
        cookie.Expires = DateTime.Today.AddYears(-1);
    
        context.Response.Cookies.Set(cookie);
      }
    }

    This code reads all the cookies it receives from request and expires them so that browser does not send those cookies again. If by any chance ASP.NET cookies get injected into the static.yoursite.com domain, this code will take care of removing them.

Build truly RESTful API and website using same ASP.NET MVC code

A truly RESTful API means you have unique URLs to uniquely represent entities and collections, and there is no verb/action on the URL. You cannot have URL like /Customers/Create or /Customers/John/Update, /Customers/John/Delete where the action is part of the URL that represents the entity. An URL can only represent the state of an entity, like /Customers/John represents the state of John, a customer, and allow GET, POST, PUT, DELETE on that very URL to perform CRUD operations. Same goes for a collection where /Customers returns list of customers and a POST to that URL adds new customer(s). Usually we create separate controllers to deal with API part of the website but I will show you how you can create both RESTful website and API using the same controller code working over the exact same URL that a browser can use to browse through the website and a client application can perform CRUD operations on the entities.

I have tried Scott Gu’s examples on creating RESTful routes, this MSDN Magazine article, Phil Haack’s REST SDK for ASP.NET MVC, and various other examples. But they have all made the same classic mistake – the action is part of the URL. You have to have URLs like http://localhost:8082/MovieApp/Home/Edit/5?format=Xml to edit a certain entity and define the format eg xml, that you need to support. They aren’t truly RESTful since the URL does not uniquely represent the state of an entity. The action is part of the URL. When you put the action on the URL, then it is straightforward to do it using ASP.NET MVC. Only when you take the action out of the URL and you have to support CRUD over the same URL, using three different formats – html, xml and json, it becomes tricky and you need some custom filters to do the job. It’s not very tricky though, you just need to keep in mind your controller actions are serving multiple formats and design your website in a certain way that makes it API friendly. You make the website URLs look like API URL.

The example code has a library of ActionFilterAttribute and ValurProvider that make it possible to serve and accept html, json and xml over the same URL. A regular browser gets html output, an AJAX call expecting json gets json response and an XmlHttp call gets xml response.

You might ask why not use WCF REST SDK? The idea is to reuse the same logic to retrieve models and emit html, json, xml all from the same code so that we do not have to duplicate logic in the website and then in the API. If we use WCF REST SDK, you have to create a WCF API layer that replicates the model handling logic in the controllers.

The example shown here offers the following RESTful URLs:

  • /Customers – returns a list of customers. A POST to this URL adds a new customer.
  • /Customers/C0001 – returns details of the customer having id C001. Update and Delete supported on the same URL.
  • /Customers/C0001/Orders – returns the orders of the specified customer. Post to this adds new order to the customer.
  • /Customers/C0001/Orders/O0001 – returns a specific order and allows update and delete on the same URL.

All these URLs support GET, POST, PUT, DELETE. Users can browse to these URLs and get html page rendered. Client apps can make AJAX calls to these URLs to perform CRUD on these. Thus making a truly RESTful API and website.

Customers

They also support verbs over POST in case you don’t have PUT, DELETE allowed on your webserver or through firewalls. They are usually disabled by default in most webservers and firewalls due to security common practices. In that case you can use POST and pass the verb as query string. For ex, /Customers/C0001?verb=Delete to delete the customer. This does not break the RESTfulness since the URL /Customers/C0001 is still uniquely identifying the entity. You are passing additional context on the URL. Query strings are also used to do filtering, sorting operations on REST URLs. For ex, /Customers?filter=John&sort=Location&limit=100 tells the server to return a filtered, sorted, and paged collection of customers.

Read my CodeProject article for details:

http://www.codeproject.com/KB/aspnet/aspnet_mvc_restapi.aspx

The source code is available here:

http://code.msdn.microsoft.com/Build-truly-RESTful-API-194a6253

Enjoy!

Automatic Javascript, CSS versioning to refresh browser cache

When you update javascript or css files that are already cached in users’ browsers, most likely many users won’t get that for some time because of the caching at the browser or intermediate proxy(s). You need some way to force browser and proxy(s) to download latest files. There’s no way to do that effectively across all browsers and proxies from the webserver by manipulating cache headers unless you change the file name or you change the URL of the files by introducing some unique query string so that browsers/proxies interpret them as new files. Most web developers use the query string approach and use a version suffix to send the new file to the browser. For example,

<script src="someJs.js?v=1001" ></script>
<link href="someCss.css?v=2001"></link>

In order to do this, developers have to go to all the html, aspx, ascx, master pages, find all references to static files that are changed, and then increase the version number. If you forget to do this on some page, that page may break because browser uses old cached script. So, it requires a lot of regression test effort to find out whether changing some css or js breaks something anywhere in the entire website.

Another approach is to run some build script that scans all files and updates the reference to the javascript and css files in each and every page in the website. But this approach does not work on dynamic pages where the javascript and css references are added at run-time, say using ScriptManager.

If you have no way to know what javascript and css will get added to the page at run-time, the only option is to analyze the page output at runtime and then change the javascript, css references on the fly.

Here’s an HttpFilter that can do that for you. This filter intercepts any ASPX hit and then it automatically appends the last modification date time of javascript and css files inside the emitted html. It does so without storing the whole generated html in memory nor doing any string operation because that will cause high memory and CPU consumption on webserver under high load. The code works with character buffers and response streams directly so that it’s as fast as possible. I have done enough load test to ensure even if you hit an aspx page million times per hour, it won’t add more than 50ms delay over each page response time.

First, you add set the filter called StaticContentFilter in the Global.asax file’s Application_BeginRequest event handler:

Response.Filter = new Dropthings.Web.Util.StaticContentFilter(
    Response,
    relativePath =>
      {
        if (Context.Cache[physicalPath] == null)
        {
          var physicalPath = Server.MapPath(relativePath);
          var version = "?v=" +
            new System.IO.FileInfo(physicalPath).LastWriteTime
            .ToString("yyyyMMddhhmmss");
          Context.Cache.Add(physicalPath, version, null,
            DateTime.Now.AddMinutes(1), TimeSpan.Zero,
            CacheItemPriority.Normal, null);
          Context.Cache[physicalPath] = version;
          return version;
        }
        else
        {
          return Context.Cache[physicalPath] as string;
        }
      },
    "http://images.mydomain.com/",
    "http://scripts.mydomain.com/",
    "http://styles.mydomain.com/",
    baseUrl,
    applicationPath,
    folderPath);
}

The only tricky part here is the delegate that is fired whenever the filter detects a script or css link and it asks you to return the version for the file. Whatever you return gets appended right after the original URL of the script or css. So, here the delegate is producing the version as “?v=yyyyMMddhhmmss” using the file’s last modified date time. It’s also caching the version for the file to make sure it does not make a File I/O request on each and every page view in order to get the file’s last modified date time.

For example, the following scripts and css in the html snippet:

<script type="text/javascript" src="scripts/jquery-1.4.1.min.js" ></script>
<script type="text/javascript" src="scripts/TestScript.js" ></script>
<link href="Styles/Stylesheet.css" rel="stylesheet" type="text/css" />

It will get emitted as:

<script type="text/javascript" src="scripts/jquery-1.4.1.min.js?v=20100319021342" ></script>
<script type="text/javascript" src="scripts/TestScript.js?v=20110522074353" ></script>
<link href="Styles/Stylesheet.css?v=20110522074829" rel="stylesheet" type="text/css" />

As you see there’s a query string generated with each of the file’s last modified date time. Good thing is you don’t have to worry about generating a sequential version number after changing a file. it will take the last modified date, which will change only when a file is changed.

The HttpFilter I will show you here can not only append version suffix, it can also prepend anything you want to add on image, css and link URLs. You can use this feature to load images from a different domain, or load scripts from a different domain and benefit from the parallel loading feature of browsers and increase the page load performance. For example, the following tags can have any URL prepended to them:

<script src="some.js" ></script>
<link href="some.css" />
<img src="some.png" />

They can be emitted as:

<script src="http://javascripts.mydomain.com/some.js" ></script>
<link href="http://styles.mydomain.com/some.css" />
<img src="http://images.mydomain.com/some.png" />

Loading javascripts, css and images from different domain can significantly improve your page load time since browsers can load only two files from a domain at a time. If you load javascripts, css and images from different subdomains and the page itself on www subdomain, you can load 8 files in parallel instead of only 2 files in parallel.

Read here to learn how this works:

http://www.codeproject.com/KB/aspnet/autojscssversion.aspx

Appreciate your feedback.

Safely deploying changes to production servers

When you deploy incremental changes on a production server, which is running and live all the time, you some times see error messages like “Compiler Error Message: The Type ‘XXX’ exists in both…”. Sometimes you find Application_Start event not firing although you shipped a new class, dll or web.config. Sometimes you find static variables not getting initialized and so on. There are so many weird things happen on webservers when you incrementally deploy changes to the server and the server has been up and running for several weeks. So, I came up with a full proof house keeping steps that we always do whenever we deploy some incremental change to our websites. These steps ensure that the web sites are properly recycled , cached are cleared, all the data stored at Application level is initialized.

First of all you should have multiple web servers behind a load balancer. This way you can take one server out of the production traffic, do your deployment and house keeping tasks like restarting IIS, and then put it back. Then you can do it for the second server and so on. This ensures there’s no outage for customer. If you can do it reasonable fast, hopefully customers won’t notice discrepancy between the servers some having new code and some having old code. You should only do this when your changes aren’t drastic. For ex, you aren’t delivering a complete revamped UI. In that case, some users hitting server1 with latest UI will suddenly get a completely different experience and then on next page refresh, they might hit server2 with old code and get a totally different experience. This works for incremental non-dramatic changes only.

image

During deployment you should follow these steps:

  • Take server X out of load balancer so that it does not get any traffic.
  • Stop all your .NET windows services on the server.
  • Stop IIS.
  • Delete the Temporary ASP.NET folders of all .NET versions incase you have multiple .NET versions running. You can follow this link.
  • Deploy the changes.
  • Flush any distributed cache you have, for ex, Velocity or Memcached.
  • Start IIS.
  • Start your .NET windows services on the server.
  • Warm up all websites by hitting major URLs on the websites. You should have some automated script to do this. You can use tinyget to hit some major URLs, especially pages that take a lot of time to compile. Read my post on keeping websites warm with zero coding.
  • Put server X back to load balancer so that it starts receiving traffic.

That’s it. It should give you a clean deployment and prevent unexpected errors. You should print these steps and hang on the desk of your deployment guys so that they never forget during deployment pressure.

Doing all these steps manually is risky. Under deployment time pressure, your production guys can make mistakes and screw up a server for good. So, I always prefer having a batch file that takes a server out and makes it ready for deploying code and then after the deployment is done, use another batch file to put the server back into load balancer traffic rotation after the server is warmed up.

Generally load balancers are configured to hit some page on your website and keep the server alive if that page returns a HTTP 200. If not, it assumes the server is dead and takes it our of rotation. For ex, say you have an alive.txt file on your website which is what load balancer is keeping an eye on. If it’s gone, the server is put out of the rotation. In that case, you can create some batch files that will take the server out, wait for couple of seconds to ensure the in-flight requests complete and then stop IIS, delete temporary ASP.NET files and make server ready to deploy stuff. Something like this:

serverout.bat
=====================
Ren alive.txt dead.txt
typeperf "ASP.NET Applications(__Total__)Requests Executing" -sc 30
iisreset /stop
rmdir /q /s "C:WINDOWSMicrosoft.NETFramework64v1.1.4322Temporary ASP.NET Files"
rmdir /q /s "C:WINDOWSMicrosoft.NETFramework64v2.0.50727Temporary ASP.NET Files"
md "C:WINDOWSMicrosoft.NETFramework64v1.1.4322Temporary ASP.NET Files"
md "C:WINDOWSMicrosoft.NETFramework64v2.0.50727Temporary ASP.NET Files"
xcacls "C:WINDOWSMicrosoft.NETFramework64v1.1.4322Temporary ASP.NET Files" /E /G MYMACHINEIIS_WPG:F /Q
xcacls "C:WINDOWSMicrosoft.NETFramework64v2.0.50727Temporary ASP.NET Files" /E /G MYMACHINEIIS_WPG:F /Q

Similarly you should have a batch file that starts IIS, warms up some pages, and then puts the server back into load balancer.

serverin.bat
============
SET TINYGET=C:Program Files (x86)IIS ResourcesTinyGettinyget.exe
iisreset /start"%TINYGET%" -srv:localhost -uri:http://localhost/ -status:200
ren dead.txt alive.txt
typeperf "ASP.NET Applications(__Total__)Requests Executing" -sc 30

Always try to automate this kind of admin chores. It’s difficult to do it right all the time manually under deployment pressure.

Quick ways to boost performance and scalability of ASP.NET, WCF and Desktop Clients

There are some simple configuration changes that you can make on machine.config and IIS to give your web applications significant performance boost. These are simple harmless changes but makes a lot of difference in terms of scalability. By tweaking system.net changes, you can increase the number of parallel calls that can be made from the services hosted on your servers as well as on desktop computers and thus increase scalability. By changing WCF throttling config you can increase number of simultaneous calls WCF can accept and thus make most use of your hardware power. By changing ASP.NET process model, you can increase number of concurrent requests that can be served by your website. And finally by turning on IIS caching and dynamic compression, you can dramatically increase the page download speed on browsers and and overall responsiveness of your applications.

Read the CodeProject article for more details.

http://www.codeproject.com/KB/webservices/quickwins.aspx

image

Please vote for me if you find the article useful.

Dynamically set WCF Endpoint in Silverlight

When you add a WCF service reference to a Silverlight Application, it generates the ServiceReference.ClientConfig file where the URL of the WCF endpoint is defined. When you add the WCF service reference on a development computer, the endpoint URL is on localhost. But when you deploy the Silverlight client and the WCF service on a production server, the endpoint URL no longer is on localhost instead on some domain. As a result, the Silverlight application fails to call the WCF services. You have to manually change the endpoint URL on the Silverlight config file to match the production URL before deploying live. Now if you are deploying the Silverlight application and the server side WCF service as a distributable application where customer install the service themselves on their own domain then you don’t know what will be the production URL. As a result, you can’t rely on the ServiceReference.ClientConfig. You have to dynamically find out on which domain the Silverlight application is running and what will be the endpoint URL of the WCF service. Here I will show you an approach to dynamically decide the endpoint URL.

First you add a typical service reference and generate a ServiceReference.ClientConfig that looks like this:

<configuration>
    <system.serviceModel>
        <bindings>
            <basicHttpBinding>
                <binding name="BasicHttpBinding_ProxyService" maxBufferSize="2147483647"
                    maxReceivedMessageSize="2147483647">
                    <security mode="None" />
                </binding>
                <binding name="BasicHttpBinding_WidgetService" maxBufferSize="2147483647"
                    maxReceivedMessageSize="2147483647">
                    <security mode="None" />
                </binding>
            </basicHttpBinding>
        </bindings>
        <client>
            <endpoint address="http://localhost:8000/Dropthings/API/Proxy.svc/pox"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_ProxyService"
                contract="DropthingsProxy.ProxyService" name="BasicHttpBinding_ProxyService" />
            <endpoint address="http://localhost:8000/Dropthings/API/Widget.svc/pox"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_WidgetService"
                contract="DropthingsWidgetService.WidgetService" name="BasicHttpBinding_WidgetService" />
        </client>
    </system.serviceModel>
</configuration>

As you see, all the URL are pointing to localhost, on my development environment. The Silverlight application now need to dynamically decide what URL the Silverlight app is running from and then resolve the endpoint URL dynamically.

I do this by creating a helper class that checks the URL of the Silverlight application and then decides what’s going to be the URL of the endpoint.

public class DynamicEndpointHelper
{
    // Put the development server site URL including the trailing slash
    // This should be same as what's set in the Dropthings web project's
    // properties as the URL of the site in development server
    private const string BaseUrl = "http://localhost:8000/Dropthings/";

    public static string ResolveEndpointUrl(string endpointUrl, string xapPath)
    {
        string baseUrl = xapPath.Substring(0, xapPath.IndexOf("ClientBin"));
        string relativeEndpointUrl = endpointUrl.Substring(BaseUrl.Length);
        string dynamicEndpointUrl = baseUrl + relativeEndpointUrl;
        return dynamicEndpointUrl;
    }
}

In the Silverlight app, I construct the Service Client this way:

private DropthingsProxy.ProxyServiceClient GetProxyService()
{
    DropthingsProxy.ProxyServiceClient service = new DropthingsProxy.ProxyServiceClient();
    service.Endpoint.Address = new EndpointAddress(
        DynamicEndpointHelper.ResolveEndpointUrl(service.Endpoint.Address.Uri.ToString(),
        App.Current.Host.Source.ToString()));
    return service;
}

After creating the service client with default setting, it changes the endpoint URL to the currently running website’s URL. This solution works when the WCF services are exposed from the same web application. If you have the WCF services hosted on a different domain and you are making cross domain calls to the WCF service then this will not work. In that case, you will have to find out what’s the domain of the WCF service and then use that instead of localhost.

Ten Caching Mistakes that Break your App

 

Caching frequently used objects, that are expensive to fetch from the source, makes application perform faster under high load. It helps scale an application under concurrent requests. But some hard to notice mistakes can lead the application to suffer under high load, let alone making it perform better, especially when you are using distributed caching where there’s separate cache server or cache application that stores the items. Moreover, code that works fine using in-memory cache can fail when the cache is made out-of-process. Here I will show you some common distributed caching mistakes that will help you make better decision when to cache and when not to cache.

Here are the top 10 mistakes I have seen:

  1. Relying on .NET’s default serializer.
  2. Storing large objects in a single cache item.
  3. Using cache to share objects between threads.
  4. Assuming items will be in cache immediately after storing it.
  5. Storing entire collection with nested objects.
  6. Storing parent-child objects together and also separately.
  7. Caching Configuration settings.
  8. Caching Live Objects that has open handle to stream, file, registry, or network.
  9. Storing same item using multiple keys.
  10. Not updating or deleting items in cache after updating or deleting them on persistent storage.

Let’s see what they are and how to avoid them.

http://www.codeproject.com/KB/web-cache/cachingmistakes.aspx

Please vote if you find this useful.

Dropthings now available from Microsoft/Web

Dropthings is now available on Microsoft/Web. You can now install it using the Web Platform Installer. I will soon write an article how to make an installer that can install a ASP.NET website, a SQL Server database, setup web.config files, setup directory permissions (eg App_Data) etc. It wasn’t straightforward and I learnt some best practices from the Microsoft/Web team. But for now, go ahead and download the app and build cool sites out of it.image

http://www.microsoft.com/web/gallery/Dropthings.aspx

Please vote for my app. It will help spread the open source effort. 

Production Challenges of ASP.NET Website – recording of my talk

“It works in my PC”, the common dialog from Developers. But on production CPU burns out, disks go crazy, your site stops responding every now and then, you have to frequently recycle application pool, or even restart windows to bring things back to normal for a while.

As your site grows from thousands to millions of users, delivering new features and bug fixes becomes an increasingly complex process where you have to ensure the performance and stability of the site is not compromised. Let me tell you some stories from the trenches how I grew my hot startup Pageflakes (acquired by MySpace founder Brad Greenspan) from thousands to millions of visits within a year surviving through production software, hardware, process, people challenges and how you can avoid and solve them using my handy collection of articles and tools.

Watch the presentation I made in a LIDNUG session:

http://www.lidnug.org/Archives.aspx

Find the session with the title “Production Challenges of ASP.NET Website”.

Unfortunately they could not record the whole session. But whatever is there should be useful for you.