Archive for Omar AL Zabir

jChaart – Web Dashboard Framework

Screenshot

A Javascript-only, web based Dashboard library, that you can use to make eye-catchy Charts showing Transactions stats for your application, Server CPU, Memory, IO graphs etc. It offers libraries to convert different types of delimited text files into Charts. Those delimited files are generated from various sources, eg running SQL queries against a Database, or running shell scripts to collect system stats, or Powershell scripts to process IIS logs. End result is a nice-looking Twitter Bootstrap powered, responsive Web Dashboard, that you can get up and running in no time, on any platform. Since it is HTML and Javascript, you can customize it to show exactly what you want, how you want. Quite handy for earning brownie points and wooing your customers.

Here’s an example how a Transaction Dashboard may look like:

 

Screenshot

 

And an example of a System Monitoring dashboard:

ServerHealth

The GitHub project is here:

https://github.com/oazabir/jChaart

You are welcome to join and participate.

Student Data Bank: An ASP.NET MVC, WebAPI, EF Codefirst, Task, ESB showcase

HighLevelArchitecture

Imagine this: you are a student of Oxford and you want to join a program in Cambridge. Cambridge wants to pull all the courses you have done in Oxford, along with your course results and automatically credit them, so that you don’t have to repeat the courses. Eventually when Cambridge decides to award you a degree, they can automatically check that you have done all the courses required by their program, either in Cambridge or in Oxford, Harvard, MIT or any other universities in the world. Would it not be nice to have a common system where all your courses and results from any university in the world, are securely stored and universities can exchange with each other what you have done, fully automatically? There would be no need to get transcripts printed, mailed, certified etc anymore. It would be done entirely online securely.

Let’s build such an imaginary system using ASP.NET MVC and WebAPI, and expose that over an ESB like WSO2 ESB. You don’t have to use any ESB. You can just run the ASP.NET app directly. ESB is there to offer secure, reliable exposure of the services.

HighLevelArchitecture

Let’s define what features we want on this system:

StudentDataBank.org (SDB – an imaginary org) is an organization that offers secure online storage and controlled exchange of student records between universities, colleges, schools and other types of Educational Institutes (EI). An EI can establish a secure interface with Student Data Bank and upload its Courses, Programs, Students and Student’s Course Results. Then another EI can request a Student’s Record to be fetched from the offering EI. Student Data Bank (SDB) takes care of securely validating the request and fetching Student Records from an EI’s systems via the secure interface that EI has already established with the SDB. Thus SDB works as a broker between multiple EIs, empowering each to share student records with each other in a secure and auditable manner.

An awarding EI can initiate the process of awarding a degree to a student and in the process automatically establish whether the awardee student has completed all the required courses from the awarding EI, as well as from other EIs that the student claims to have attended required courses from. Each EI has a way to define what requirements their courses meet, so that courses can be accepted from other EIs automatically while awarding a degree to a student.

The Web site looks like this:

Fetch Courses

Clicking on that Fetch Course triggers a workflow, which fetches the courses from the University, via the interface integration done with the University.

Sequence Diagram

Read the full CodeProject article how this is done:

http://www.codeproject.com/Articles/895310/Student-Data-Bank-An-ASP-NET-MVC-WebAPI-EF-Codefir

 

Build, deploy, anonymize config, zip package, git commit, push from a single command

gitautomate

gitautomateWhile working on open source projects, you have to frequently build your code, clean up all temporary files, remove your own passwords, connections strings from web.config, then create a binary deployment package in a zip format and then commit and git push to GitHub. Let’s automate all these using a configurable powershell script.

Here’s the full script.

First step, let’s define the parameters for the script with some default values:

param (
    [string]$solution = "OracleDashboard.sln",
    [string]$zipname = "OracleDashboard.zip",
    [string]$compressor = "c:\Program Files\7-Zip\7z.exe",
    [string]$folder = "OracleDashboard",
    [string]$deployPath = "..\Binary",
    [string]$commitFrom = "..",
    [Parameter(Mandatory=$true)][string]$comment
)

Some description of these parameters:

  • $solution = path of the solution file relative to the script location
  • $zipname = name of the zip file.
  • $compressor = 7-zip’s 7z.exe file path.
  • $folder = the folder that contains the code, which is zipped.
  • $deployPath = relative path where the zip file will be moved to.
  • $commitFrom = relative path where the script will run git commit and git push from.
  • $comment = A comment for the git commit.

The first thing the script does is look for the solution open in Visual Studio and close it. You can comment this section out if you want. But if Visual Studio is open, then /obj folder cannot be deleted.

# If visual studio has the solution open, close VS, as we can't delete obj folder while it is open
$windowTitle = $solution.Replace(".sln", "")
$vsProcess = get-process | where {$_.mainwindowtitle -match $windowTitle -and $_.ProcessName -eq "devenv"} 
if ($vsProcess.Length -gt 0) {
    Write-Host "Visual Studio has this solution open. Closing..."
    $vsProcess | ForEach-Object { $_.CloseMainWindow(); }
    Sleep 5
    Read-Host "Press ENTER to proceed if Visual Studio is closed"
    $vsProcess = get-process | where {$_.mainwindowtitle -match $windowTitle -and $_.ProcessName -eq "devenv"} 
    if ($vsProcess.Length -gt 0) {
        Write-Host "Visual Studio still has the solution open. Aborting."
        Return
    }
}

Next step is to do some spring cleaning:

Push-Location

if (Test-Path $zipname) { rm $zipname; }

# Clean up deploy folder 
rm $deployPath\*.* -Force -Recurse

First remember the current path. We have to come back to this path after we are done. Then remove the zip file if it already exists. Then cleanup the $deployPath. You can remove this if you want to keep old deployment packages. Then you have to handle generation of unique file names for the packages.

Now, let’s build and remove /obj folder:

# Build new version
msbuild /verbosity:minimal $solution

# Delete obj
if (Test-Path $folder\obj) { rm $folder\obj -Force -Recurse }

Next step: remove all sensitive information from the web.config, which includes connection strings, authorization block, appSettings entries etc. This is all up to you to customize:

# backup the web.config and remove sensitive entries before pushing to git, eg connectionString
[string]$filename = gi $folder\web.config 
[string]$backup = [System.IO.File]::ReadAllText($filename)
$xml =
$backup $xml.PreserveWhitespace = $true foreach($n in $xml.configuration.connectionStrings.add) { $n.ParentNode.RemoveChild($n); } # Anonymize any sensitive appSettings entry foreach($n in $xml.configuration.appSettings.add) { switch($n.key) { "Password" { $n.value = "Password" } } } # Remove authorization blocks $xml.configuration.'system.web'.authorization.RemoveAll() $xml.Save($filename)

Finally, let’s run some regular expression check to ensure the web.config does not have any sensitive information left accidentally. Again, this is all up to you to customize.

# verify if web.config still contains any sensitive info
[string]$config = gc $folder\web.config
if ( ($config -match 'connectionString="\w+') -or ($config -match 'users="\w+') ) {
    Write-Host "Configuration file is not cleaned."
    # Restore web.config
    [System.IO.File]::WriteAllText($filename, $backup)
    Exit
}

Now time to compress the source folder and create a zip file using 7-zip.

# Compress the solution folder and copy to deploy folder
cmd /c $compressor a -tzip $zipname $folder -r 
cmd /c copy $zipname $deployPath /Y
cmd /c del $zipname

Finally git commit and push:

# Commit and push to GitHub
cd $commitFrom
git pull
git add -A *.*
git commit -a -m $comment
git push 
Pop-Location

And last step is to restore your own web.config, that was anonymized:

# Restore web.config
[System.IO.File]::WriteAllText($filename, $backup)

That’s it. Now all you have to do is, just hit ./gitpush.ps1 from Powershell command line and you are done!

Real-time Oracle Database Monitoring Dashboard in ASP.NET

Dashboard - Full

Oracle Performance Dashboard (OPD) is a small ASP.NET website that shows you performance & problems of one or more Oracle instances in near real-time. It uses the Dynamic Performance Views (DPV) and runs some popular DBA scripts in order to get meaningful, easy to understand information out of the server. You can use it to quickly spot blocking queries, who is blocking who, expensive query that are consuming high CPU or disk, see if there’s unusual locks, very high disk activity and so on.

Dashboard - Full

Demo

You can see a live demo of this from here:
http://odp.omaralzabir.com

Get the code

The binaries are here, which you can just extract into an IIS folder, put the connection strings in the web.config file, and you are ready to roll. No need to install any Oracle client software on the server.
GitHub Project Binaries

You can get the source code from the GitHub project site:
https://github.com/oazabir/OraclePerformanceDashboard

Feature walkthrough

OPD comes with the following features in V1:

  • Summary of all your instances in one screen, showing important indicators on each instance. Quick way to check if all your databases are doing fine or not.
  • Instance Dashboard showing details of an instance:
    • CPU usage on the OS.
    • CPU consumed by each session
    • Important System Statistics like Buffer Gets, Parse to execute ratio which would indicate some common problems on the server.
    • Sessions and what the sessions are doing, how much resource they are consuming, whether they are hogging the disk or not.
    • Waits, Blocks, Locks, deadlocks that make database suffer.
    • Historical analysis on the databse showing you some very useful stuff:
      • Most expensive queries in terms of CPU and Buffer Get, which are immediate convern for your devs to sort out.
      • IO usage on data files. You can see if some data file is unusually loaded and getting hammered by physical IO.
      • Tablespace usage. Alerts you if some tablespace is over 85% full.
      • Stale stats on tables. You should always keep this clean.
      • Killer Indexes that will actually blow up your system and confuse Oracle query optimizer. You need to get rid of those indexes and rewrite queries that you thought will hit those indexes for better performance. They won’t. They will kill your database.

Web-based real-time SQL Server Performance Dashboard

Dashboard-600

SQL Server Performance Dashboard (SSPD) is a small open source web app that shows you performance & problems of one or more SQL Server instances and their databases in near real time. It uses the Dynamic Management Views (DMV) to gather useful data from the verbose output and combines them with utility stored procs in order to get meaningful, easy to understand information out of them. You can use it to quickly spot blocking queries, who is blocking who, expensive query that are consuming high CPU or disk, see if there’s unusual locks, very high disk activity and so on.

See a live demo: http://dashboard.omaralzabir.com/

You can read details about the tool from this CodeProject article: http://www.codeproject.com/Articles/799053/Web-based-real-time-SQL-Server-Performance-Dashboa

The binaries are here, which you can just extract into a IIS folder, put the connection strings in the web.config file and you are ready to roll:

SqlServerPerformanceDashboard GitHub Project Binaries

Or you can get the source code from the GitHub project site: https://github.com/oazabir/SQLServerDashboard

Utility to make important windows remain always on top

AlwaysOnTop

Do you sometimes fail to notice Outlook reminder window? Do you wish a chat window would remain always on top of other windows so that you never miss a message? Do you want to have a notepad always on top so that you can take quick notes anytime, while working on other apps?

We have an app for that.

It runs quietly on system tray and monitors open windows. Whenever it finds a window that you want to make always on top, it does that:

You can use this AlwaysOnTop utility to make any window remain always on top of other windows.

Get the tool and details here:

http://www.codeproject.com/Articles/794407/Utility-to-make-important-windows-remain-always-on

 

Sit back and relax, let Sharepoint remind and chase your team

EmailExample

Sharepoint Task List is a great place to record tasks for your team members. However, once you have recorded and assigned the tasks, then the fun begins. You have to remind your team members repeatedly about what they need to do today, what tasks are overdue, what’s coming this week and so on. Here’s an app that takes that chore away, and you can sit back and relax, while it will reminds your team members as many times as you want. Your team members will get an email like this, reminding them of their assigned tasks:

The email templates are fully customizable. You can define exactly how the email should look like. You can also include custom fields from sharepoint taks items to appear on the email.

Read the CodeProject article here for details:

http://www.codeproject.com/Articles/789418/Sit-back-and-relax-let-Sharepoint-remind-and-chase

Google Docs to clean html, good for WordPress posts, emails

GoogleDocScreenshot

Google docs is a great platform to write documents, especially when you compare it with the WordPress editor.  It would be good to have a clean way to export a Google doc to a wordpress post or generate nice looking emails. If you copy and paste a Google doc into a WordPress post, it loses many formatting and produces a bloated html with lots of inline style, CSS classes that do not go well with WordPress. So, here’s a solution that will generate a clean HTML from a Google Doc and email it to you so that you can copy and paste it a WordPress post or send to others via email.

For example, here’s a Google Doc:

GoogleDocScreenshot

Once you run the script, it will produce a nice clean email for you:

GoogleDocsEmail

Here’s how to do it:

  1. Open your Google Doc and go to Tools menu, select Script Editor. You should see a new window open with a nice code editor.
  2. Copy and paste the code from here: GoogleDocs2Html
  3. Then from the “Select Editor” menu, choose ConvertGoogleDocToCleanHtml
  4. Click the play button to run the script.
  5. You will get an email containing the HTML output of the Google Doc with inline images.
  6. You can easily forward that email to anyone or copy and paste in a WordPress post.

Here’s how the code works:

First it will loop through the elements (paragraph, images, lists) in the body:

function ConvertGoogleDocToCleanHtml() {
  var body = DocumentApp.getActiveDocument().getBody();
  var numChildren = body.getNumChildren();
  var output = [];
  var images = [];
  var listCounters = {};

  // Walk through all the child elements of the body.
  for (var i = 0; i < numChildren; i++) {
    var child = body.getChild(i);
    output.push(processItem(child, listCounters, images));
  }

  var html = output.join('\r');
  emailHtml(html, images);
  //createDocumentForHtml(html, images);
}

The processItem function takes care of generating proper html output from a Doc Element. The code for this function is long as it handles Paragraph, Text block, Image, Lists. Best to read through the code to see how it works. When the proper html is generated and the images are discovered, the emailHtml function generates a nice html email, with inline images and sends to your Gmail account:

function emailHtml(html, images) {
  var inlineImages = {};
  for (var j=0; j<images.length; j++) {
    inlineImages[[images[j].name]] = images[j].blob;
  }

  var name = DocumentApp.getActiveDocument().getName()+".html";

  MailApp.sendEmail({
     to: Session.getActiveUser().getEmail(),
     subject: name,
     htmlBody: html,
     inlineImages: inlineImages
   });
}

Enjoy!

Remember images in the email are inline images. If you copy and paste into WordPress, the images won’t get automatically uploaded to WordPress. You will have to manually download and upload each image to WordPress. It’s a pain. But that’s the problem with WordPress editor.

Special thanks to this GitHub project, that gave me many ideas: https://github.com/mangini/gdocs2md

 

Faster page rendering by downloading JS/CSS before server generates full page

Fast

When a dynamic page is executing on the server, browser is doing nothing but waiting for the html to come from the server. If your server-side code takes 5 seconds to perform database operations on the server, then for that 5 seconds, user is staring at a blank white screen. Why not take this opportunity to download the Javascript and CSS on the browser simultaneously? By the time server finishes doing its work, server will just send the dynamic content and browser will be able to render it right away. This optimization technique can improve the performance of any dynamic page that takes some time to finish its job on the server, and has some js and css to download.

When server and browser are working simulteneously, you get to render the page significantly faster:

Fast

Server and browser working simultaneously

 

Read the full article here:

http://www.codeproject.com/Articles/755672/Faster-page-rendering-by-downloading-JS-CSS-before