Showing posts with label c#. Show all posts
Showing posts with label c#. Show all posts

Friday, March 14, 2014

Global Alignment Problem

The solution to the Manhattan problem described here only returned us the maximum score, but did not give instructions on how exactly we should walk through Manhattan to achieve this score. To lay out the path (and to use the solution in bioinformatics to align aminoacid sequences) we need an additional step: keep the history of our walk so that we could "backtrack" from the last intersection all the way back to the start. Essentially, it is simple: when we arrive to a new node, we always come from somewhere. This is the node we need to remember.

We should also consider that, unlike Manhattan graph, where we only had streets going "west" or "south", we now have "diagonal" streets if we want to adapt the problem to align aminoacid sequences. Consider the image below:

Alignment graph matches

The red diagonal arrows show the places where one sequence matches the other. We can give these edges a higher "score" so the path that goes through most of such edges is the highest scored. In this case we have three ways to reach any node s[i,j]: we can come from s[i - 1,j] by moving down, from s[i,j - 1] by moving right, or from s[i - 1, j - 1] by moving diagonally. In the simplest case, we'll calculate anything other than a match as a 0, and a match as 1. Then the score for any s[i,j] node will be calculated using the following formula:

Global alignment formula

In more complex scenarios, we may use scoring matrices, where each combination of two aminoacids is given a certain score, depending on how biologically reasonable is this combination.

Therefore, the following algorithm is used to calculate the score matrix s, and at the same time fill the backtracking matrix backtrack.

AlignmentMatrix(v, w)
 for i ← 0 to |v|
  si, 0 ← 0
 for j ← 0 to |w| 
  s0, j ← 0
 for i ← 1 to |v|
  for j ← 1 to |w|
   si, j = max{si-1, j, si,j-1, si-1, j-1 + 1 (if vi = wj)}
   backtracki, j = ↓ if si, j = si-1, j ; → if si, j = si, j-1 ; ↘ if si, j = si-1, j-1 + 1
 return s|v|, |w| , backtrack

After both the scoring matrices are filled, we will use the following algorithm to backtrack from the s[i,j] all the way back to the starting node. Along the way we will output two string. In places where there is a match between the strings, a symbol will be placed, and in other places we will fill in a dash. Following is the backtracking algorithm. We are filling two strings, v and w. If we came to the point i,j by a diagonal edge, we have a match, and output the same symbol into both strings. If we came by a vertical or horisontal edge, one of the strings receives a symbol, and the other a "mismatch" - a "-".

OUTPUTALIGNMENT(backtrack, v, w, i, j)
 if i = 0 or j = 0
  return
 if backtracki, j = ↓
  output vi = "-"
  outpuv wi
  OUTPUTLCS(backtrack, v, i - 1, j)
 else if backtracki, j = →
  output vi
  output wi = "-"
  OUTPUTLCS(backtrack, v, i, j - 1)
 else
  OUTPUTLCS(backtrack, v, i - 1, j - 1)
  output vi
  output wi

This is the C# implementation of the algorithms.

public static void GlobalAlignment(string s1, string s2)
{
 int s1l = s1.Length;
 int s2l = s2.Length;

 int[,] matrix = new int[s1l + 1, s2l + 1];
 int[,] backtrack = new int[s1l + 1, s2l + 1];

 for (int i = 0; i <= s1l; i++) for (int j = 0; j <= s2l; j++) matrix[i, j] = 0;
 for (int i = 0; i <= s1l; i++) for (int j = 0; j <= s2l; j++) backtrack[i, j] = 0;

 for (int i = 0; i <= s1l; i++) matrix[i, 0] = 0;
 for (int j = 0; j <= s2l; j++) matrix[0, j] = 0;

 for (int i = 1; i <= s1l; i++)
 {
  for (int j = 1; j <= s2l; j++)
  {
   matrix[i, j] = Math.Max(Math.Max(matrix[i - 1, j], matrix[i, j - 1]), matrix[i - 1, j - 1] + 1);

   if (matrix[i, j] == matrix[i - 1, j]) backtrack[i, j] = -1;
   else if (matrix[i, j] == matrix[i, j - 1]) backtrack[i, j] = -2;
   else if (matrix[i, j] == matrix[i - 1, j - 1] + 1) backtrack[i, j] = 1;
  }
 }

 OUTPUTLCSALIGN(backtrack, s2, s1, s1l, s2l);

 string aligned1 = ReverseString(Result);
 string aligned2 = ReverseString(Result2);
}

public static void OUTPUTLCSALIGN(int[,] backtrack, string v, string w, int i, int j)
{
 if (i == 0 || j == 0)
  return;
 if (backtrack[i, j] == -1)
 {
  Result = Result + "-";
  Result2 = Result2 + w[i - 1];
  OUTPUTLCSALIGN(backtrack, v, w, i - 1, j);
 }
 else if (backtrack[i, j] == -2)
 {
  Result = Result + v[j - 1];
  Result2 = Result2 + "-";
  OUTPUTLCSALIGN(backtrack, v, w, i, j - 1);
 }
 else
 {
  Result = Result + v[j - 1];
  Result2 = Result2 + w[i - 1];
  OUTPUTLCSALIGN(backtrack, v, w, i - 1, j - 1);
 }
}

public static string ReverseString(string input)
{
 char[] charArray = input.ToCharArray();
 Array.Reverse(charArray);
 return new string(charArray);
}

A slightly modified version of the function takes into account a penalty for an insertion / deletion (-5 in this case) and uses a BLOSUM62 scoring matrix

public static void GlobalAlignment(string s1, string s2)
{
 int indelPenalty = -5;

 int s1l = s1.Length;
 int s2l = s2.Length;

 int[,] matrix = new int[s1l + 1, s2l + 1];
 int[,] backtrack = new int[s1l + 1, s2l + 1];

 for (int i = 0; i <= s1l; i++) for (int j = 0; j <= s2l; j++) matrix[i, j] = 0;
 for (int i = 0; i <= s1l; i++) for (int j = 0; j <= s2l; j++) backtrack[i, j] = 0;

 for (int i = 0; i <= s1l; i++) matrix[i, 0] = indelPenalty * i;
 for (int j = 0; j <= s2l; j++) matrix[0, j] = indelPenalty * j;

 for (int i = 1; i <= s1l; i++)
 {
  for (int j = 1; j <= s2l; j++)
  {
   //deletion
   int delCoef = matrix[i - 1, j] + indelPenalty;
   //insertion
   int insCoef = matrix[i, j - 1] + indelPenalty;
   //match / mismatch
   int mCoef = matrix[i - 1, j - 1] + BlosumCoeff(s1[i - 1], s2[j - 1]);

   matrix[i, j] = Math.Max(Math.Max(delCoef, insCoef), mCoef);

   if (matrix[i, j] == delCoef) backtrack[i, j] = -1; //
   else if (matrix[i, j] == insCoef) backtrack[i, j] = -2; //
   else if (matrix[i, j] == mCoef) backtrack[i, j] = 1;
  }
 }

 OUTPUTLCSALIGN(backtrack, s2, s1, s1l, s2l);

 string aligned1 = ReverseString(Result);
 string aligned2 = ReverseString(Result2);
}

Given the following two strings

PLEASANTLY
MEANLY

The following alignment will be returned

PLEASANTLY
-MEA--N-LY

by . Also posted on my website

Tuesday, March 11, 2014

Manhattan Tourist problem

An introductory exercise to aligning amino acid sequences is the Manhattan tourist problem. Suppose a tourist starts his route in the top left corner of the map and wants to visit as many attractions on the way as possible, and finish in the down right corner. There is a restriction however - he can only move either to the right or down.

A Simplified Map of Manhattan

To describe the problem in numbers and data structures, the map is converted to the directed grid shown below. A tourist can move along the edges of the grid and the score of 1 is added if he passes an attraction, which are assigned to the bold edges. Of all possible paths we are interested in the one with the highest score.

Map of Manhattan as a Graph

The solution to this problem is equivalent to a more general problem. This time every "street", or the edge of the graph, is assigned a score. The goal is to find a path which gains the highest score.

Generic Manhattan Problem

The problem can be brute forced, of course, by calculating the score for all possible paths. This is not practical for the larger graphs of course. A better approach is to calculate the highest possible score at every intersection, starting from the top left corner. Here is how the logic goes: If the tourist goes only towards the left in the grid, or only down, there are no choices for him. So it is easy to calculate the highest scores for the top and left row of intersections. Now we have these scores calculated:

Generic Manhattan Problem - First Row and Column Solved

Now when that's done, we can proceed to the next row. How can the tourist get to intersection (1, 1)? He can either go to (0, 1) first, and gain 3 + 0 = 3 score, or go to (1, 0) and gain 1 + 3 = 4 score. Therefore, the maximum score he can gain at (1, 1) is 4.

Generic Manhattan Problem - Cell (1,1) Solved

The generic formula to use is quite intuitive: To reach an intersection, you have to arrive from one of two possible previous locations, and traverse a corresponding edge. You choose the path where the sum of the score of a previous location, plus the score of the edge, is higher.

Generic Manhattan Formula

We continue calculations along the column all the way down. After that, we can move on to the next row.

Generic Manhattan Problem - Second Column Solved

Following this logic, it is easy to compute the maximum score for each intersection and find out the maximum score that can be gained while traversing Manhattan.

Generic Manhattan Problem - Fully Solved

Along the way the algorithm also becomes clear: the maximum score at any intersection is the maximum of the following two values:

MANHATTANTOURIST(n, m, down, right)
 s0, 0 ← 0
 for i ← 1 to n
  si, 0 ← si-1, 0 + downi, 0
 for j ← 1 to m
  s0, j ← s0, j−1 + right0, j
 for i ← 1 to n
  for j ← 1 to m
   si, j ← max{si - 1, j + downi, j, si, j - 1 + righti, j}
 return sn, m

The following C# function implements the algorithm and returns the highest possible score, assuming we input all the scores for the edges towards the right and all the edges pointing down in the form of two-dimension arrays.

public static int ManhattanProblem(int[,] RightMatrix, int[,] DownMatrix)
{
 int n = RightMatrix.GetLength(0) + 1;
 int m = DownMatrix.GetLength(1) + 1;
 int[,] ManhattanMatrix = new int[n, m];

 ManhattanMatrix[0, 0] = 0;

 for (int i = 1; i <= n; i++)
 {
  ManhattanMatrix[i, 0] = ManhattanMatrix[i - 1, 0] + DownMatrix[i - 1, 0];
 }

 for (int j = 1; j <= m; j++)
 {
  ManhattanMatrix[0, j] = ManhattanMatrix[0, j - 1] + RightMatrix[0, j - 1];
 }

 for (int i = 1; i <= n; i++)
 {
  for (int j = 1; j <= m; j++)
  {
   ManhattanMatrix[i, j] =
    Math.Max(ManhattanMatrix[i - 1, j] + DownMatrix[i - 1, j], 
    ManhattanMatrix[i, j - 1] + RightMatrix[i, j - 1]);
  }
 }

 return ManhattanMatrix.Cast<int>().Max();
}

Understanding this problem turns out to be important to understanding sequence alignment in bioinformatics.

by . Also posted on my website

Thursday, December 26, 2013

Minimal change problem

A popular problem to introduce dynamic programming is the minimal change problem. Suppose a cashier needs to give me a certain amount of change and wants to do it with the minimal amount of coins possible. The input is a set of denominations and the amount, and the output is the set of coins.

For example, I may need to give 45 cents change and available coins are 1, 5 and 20. A solution intuitively is 20 + 20 + 5, but what is the best way to achieve it?

A recursive solution may be the first to try. A minimal collection of coins definitely belongs to the following set:

  • minimal collection of coins totalling 44 cents, plus 1 cent coin
  • minimal collection of coins totalling 40 cents, plus 5 cent coin
  • minimal collection of coins totalling 25 cents, plus 20 cent coin

So on the next step we will apply the same logic to our new change amounts: 44, 40 and 25. This looks like a classic recursion problem, which can be illustrated by the following image

Solving Minimum Change with Recursion

and described by the following algorithm / pseudocode (where |coins| is the number of denominations available)

RECURSIVECHANGE(money, coins)
	if money = 0
		return 0
	MinNumCoins ← ∞
	for i ← 1 to |coins|
		if money ≥ coini
			NumCoins ← RECURSIVECHANGE(money − coini, coins)
			if NumCoins + 1 < MinNumCoins
				MinNumCoins** ← NumCoins + 1
	 output MinNumCoins*

This should work, but is there something wrong with this approach? Well, one can see that the recursive algorithm will calculate the full solution for 19 cents 6 times on the third step only, and it will only get worse on the following steps. If the input value is large enough, the memory and time required to compute the solution will be huge. So, this is a classic example of the benefits of dynamic programming. I came across dynamic programming a few times before, but just couldn't properly figure it out.

Finally I found a good explanation. It came as a part of a free course in Bioinformatics Algorithms. Here's how it goes:

The key to dynamic programming is to take a step that may seem counterintuitive. Instead of computing MinNumCoins(m) for every value of m from 45 downward toward m = 1 via recursive calls, we will invert our thinking and compute MinNumCoins(m) from m = 1 upward toward 45, storing all these values in an array so that we only need to compute MinNumCoins(m) once for each value of m. MinNumCoins(m) is still computed via the same recurrence relation.

MinNumCoins(m) = min{MinNumCoins(m − 20) + 1, MinNumCoins(m - 5) + 1, MinNumCoins(m - 1) + 1}

For example, assuming that we have already computed MinNumCoins(m) for m < 6, MinNumCoins(6) is equal to one more than the minimum of MinNumCoins(6 - 5) = 1 and MinNumCoins(6 - 1) = 5. Thus, MinNumCoins(6) is equal to 1 + 1 = 2.

This translates into the following algorithm / pseudocode

DPCHANGE(money, coins)
 MinNumCoins(0) ← 0
 for m ← 1 to money
        MinNumCoins(m) ← ∞
        for i ← 1 to |coins|
            if m ≥ coini
                if MinNumCoins(m - coini) + 1 < MinNumCoins(m)
                    MinNumCoins(m) ← MinNumCoins(m - coini) + 1
    output MinNumCoins(money)

And a further C# implementation, which takes a comma-separated string of denominations available, and the target amount.

public static void DPCHANGE(int val, string denoms)
{
	int[] idenoms = Array.ConvertAll(denoms.Split(','), int.Parse);
	Array.Sort(idenoms);
	int[] minNumCoins = new int[val + 1];

	minNumCoins[0] = 0;
	for (int m = 1; m <= val; m++)
	{
		minNumCoins[m] = Int32.MaxValue - 1;
		for (int i = 1; i <= idenoms.Count() - 1; i++)
		{
			if (m >= idenoms[i])
			{
				if (minNumCoins[m - idenoms[i]] + 1 < minNumCoins[m])
				{
					minNumCoins[m] = minNumCoins[m - idenoms[i]] + 1;
				}
			}
		}
	}
}

References

Bioinformatics Algorithms
An Introduction to Dynamic Programming: The Change Problem
by . Also posted on my website

Sunday, March 24, 2013

Implementing a simple logging engine with MVC 4

I was investigating the simple logging mechanism for the MVC application. First I came up with some requirements for the logging engine:

High-level requirements:

  • Create logging engine that can log exceptions and other messages and store them in the database
  • Display a filterable grid of all messages

Logging engine

  • Should allow logging of exceptions, including handled and unhandled
  • Should allow logging of custom messages

Filterable grid

  • Should allow paged display of all exceptions and log messages in the database
  • Should allow the option to filter messages based on the date logged and severity

I started with these simple classes that will allow handling of messages and exception:

//actual entry, each will correspond to a line in the grid
public class LogEntry
{
 public int Id { get; set; }
 public DateTime TimeStamp { get; set; }
 public string Path { get; set; }
 public string RawUrl { get; set; }
 public string Source { get; set; }
 public string Message { get; set; }
 public string StackTrace { get; set; }
 public string TargetSite { get; set; }

 public int TypeId { get; set; }
 public virtual LogType LogType { get; set; }
}

//a "helper" class for types like Warning, Information etc.
public class LogType
{
 public int LogTypeId { get; set; }
 public string Type { get; set; }
}

//finally, an enum of all supported log message types
public enum LogTypeNames
{
 All = 0,
 Info = 1,
 Warn = 2,
 Debug = 3,
 Error = 4,
 Fatal = 5,
 Exception = 6
}

These will be reflected in two database tables - the main table for saving all log messages, and a helper table to keep names of message severity levels.

public DbSet<LogType> LogTypes { get; set; }
public DbSet<LogEntry> LogEntries { get; set; }

Next, it is time to mention logging of handled and unhandled exceptions, which can be divided into handled and unhandled exceptions.

Develop mechanism for logging exceptions:

1. Log unhandled exceptions

Unhandled exceptions are, well, exceptions that are not handled in the source code. First, the site web.config must be modified to add a line in the section:

Here's how it works: a method is called in Global.asax file:

public static void RegisterGlobalFilters(GlobalFilterCollection filters)
{
    filters.Add(new HandleErrorAttribute());
}

It registers the HandleErrorAttribute as global action filter. The HandleErrorAttribute checks the customErrors mode, and if it is off, shows the "yellow screen of death". If it is on, the Error view is rendered and a Model is passed to it containing exceptions stack trace. Therefore, an Error.cshtml should be added under Views/Shared, and in its simplest form may look as follows

@using Recipes.Logging
@using Recipes.Models
@{
    Layout = null;
    ViewBag.Title = "Error";
    Logger.WriteEntry(Model.Exception);
}

<!DOCTYPE html>
<html>
<head>
    <title>Error</title>
</head>
<body>
    <h2>
        Sorry, an error occurred while processing your request. The details of the error were logged.
    </h2>
</body>
</html>

For simplicity, all log messages - exceptions, handled and unhandled, and all other custom messages - will be saved in a single database table.

2. Log handled exceptions

The handled exceptions are caught by code and handled directly. The following is the Logger class which handles exceptions and custom messages and saves them to the database:

public static class Logger
{
 public static void WriteEntry(Exception ex)
 {
  LogEntry entry = BuildExceptionLogEntry(ex);
  SaveLogEntry(entry);        
 }

 public static void WriteEntry(string mesage, string source, int logType)
 {
  LogEntry entry = BuildLogEntry(mesage, source, logType);
  SaveLogEntry(entry);
 }

 private static void SaveLogEntry(LogEntry entry)
 {
  using (RecipesEntities context = new RecipesEntities())
  {
   context.LogEntries.Add(entry);
   context.SaveChanges();
  }
 }

 private static LogEntry BuildLogEntry(string message, string source, int logType)
 {
  LogEntry entry = BuildLogEntryTemplate();

  entry.Message = message;
  entry.Source = source;
  entry.LogType = GetLogEntryType((LogTypeNames)logType);
  entry.TypeId = logType;

  return entry;
 }

 private static LogEntry BuildExceptionLogEntry(Exception x)
 {
  Exception logException = GetInnerExceptionIfExists(x);
  LogEntry entry = BuildLogEntryTemplate();

  entry.Message = logException.Message;
  entry.Source = logException.Source ?? string.Empty;
  entry.StackTrace = logException.StackTrace ?? string.Empty;
  entry.TargetSite = logException.TargetSite == null ? string.Empty : logException.TargetSite.ToString();
  entry.LogType = GetLogEntryType(LogTypeNames.Exception);
  entry.TypeId = (int) LogTypeNames.Exception;

  return entry;
 }

 private static LogEntry BuildLogEntryTemplate()
 {
  return new LogEntry
       {
        Path = HttpContext.Current.Request.Path,
        RawUrl = HttpContext.Current.Request.RawUrl,
        TimeStamp = DateTime.Now,
       };
 }

 public static string BuildExceptionMessage(Exception x)
 {
  Exception logException = GetInnerExceptionIfExists(x);

  string strErrorMsg = Environment.NewLine + "Error in Path :" + HttpContext.Current.Request.Path;
  // Get the QueryString along with the Virtual Path
  strErrorMsg += Environment.NewLine + "Raw Url :" + HttpContext.Current.Request.RawUrl;
  // Get the error message
  strErrorMsg += Environment.NewLine + "Message :" + logException.Message;
  // Source of the message
  strErrorMsg += Environment.NewLine + "Source :" + logException.Source;
  // Stack Trace of the error
  strErrorMsg += Environment.NewLine + "Stack Trace :" + logException.StackTrace;
  // Method where the error occurred
  strErrorMsg += Environment.NewLine + "TargetSite :" + logException.TargetSite;
  return strErrorMsg;
 }

 private static LogType GetLogEntryType(LogTypeNames name)
 {
  return new LogType{LogTypeId = (int)name, Type = name.ToString()};
 }

 private static Exception GetInnerExceptionIfExists(Exception x)
 {
  if (x.InnerException != null)
   return x.InnerException;
  return x;
 }
}

With this basic structure in place, I can start adding user interface for displaying the log. I decided to only have two views, Index for the main grid which contains all log messages, and a Details for a detailed information about a single message. Details will be linked from the line in a grid that corresponds to a log message.

Index view.

The view will have several main parts, wrapped in a form.

@using (Html.BeginForm("Index", "Logging", new { CurrentPageIndex = 1 }, FormMethod.Get, new { id = "myform" }))
{

}

First is the div that shows the number of records found and gives an option to choose how many records per page will be displayed.

<div class="grid-header">
    <div class="grid-results">
        <div class="inner">
            <span style="float: left">
                @string.Format("{0} records found. Page {1} of {2}", Model.LogEvents.TotalItemCount, Model.LogEvents.PageNumber, Model.LogEvents.PageCount)
            </span>

            <span style="float: right">
                Show @Html.DropDownListFor(model => model.PageSize, new SelectList(FormsHelper.PagingPageSizes, "Value", "Text", Model.PageSize), new { onchange = "document.getElementById('myform').submit()" }) results per page
            </span>
            
            <div style="clear: both"></div>
        </div> <!-- inner -->
    </div>  <!-- grid-results -->
 </div>  <!-- grid-header -->

The second allows to filter records by date logged and severity

 <div class="grid-filter">        
    <div class="inner">
        Level : @Html.DropDownList("LogLevel", new SelectList(FormsHelper.LogLevels, "Value", "Text"))

        For : @Html.DropDownList("Period", new SelectList(FormsHelper.CommonTimePeriods, "Value", "Text"))
        
        <input id="btnGo" name="btnGo" type="submit" value="Apply Filter" />                      
    </div>
 </div>   

Next is the "pager" div, which allows navigation if multiple pages are reqiured

  <div class="paging">
    <div class="pager">
        @Html.Pager(ViewData.Model.LogEvents.PageSize, ViewData.Model.LogEvents.PageNumber, ViewData.Model.LogEvents.TotalItemCount, new { LogType = ViewData["LogType"], Period = ViewData["Period"], PageSize = ViewData["PageSize"] })
    </div>
 </div>

Finally, the main part is the actual grid which displays the messages.

 @if (Model.LogEvents.Count() == 0)
 {
 <p>No results found</p>
 }
 else
 {
 <div class="grid-container">
 <table class="grid">
    <tr>
        <th></th>
        <th>#</th>
        <th>Source</th>
        <th>Date</th>
        <th style='white-space: nowrap;'>Time ago</th>
        <th>Message</th>
        <th>Type</th>
    </tr>

 @{int i = 0;}
     @foreach (var item in Model.LogEvents)
     {
     <tr class="@(i++ % 2 == 1 ? "alt" : "")">
     <td>
        @Html.ActionLink("Details", "Details", new { id = item.Id.ToString(), loggerProviderName = "Go To Details" /*item.LoggerProviderName*/ })
     </td>
     <td>
        @i.ToString()
     </td>
     <td>
        @item.Source
     </td>
     <td style='white-space: nowrap;'>
        @String.Format("{0:g}", item.TimeStamp.ToLocalTime())
     </td>
     <td style='white-space: nowrap;'>
        @item.TimeStamp.ToLocalTime().TimeAgoString()
     </td>
     <td>
        <pre>@item.Message.WordWrap(80)</pre>
     </td>
     <td>
        @item.LogType.Type
     </td>
     </tr>
     }

 </table>
 </div> <!-- grid-container -->
}

A few points of interest:

The Index method in the controller returns a ViewModel. By default, all configurable parameters (page size, time period, page number and log level) are not set, and all log messages are displayed with the default page size of 20 entries. When a value is set in the UI and the form is submitted, a corresponding parameter is passed to the controller.

public ActionResult Index(string Period = null, int? PageSize = null, int? page = null, string LogLevel = null)
{
 string defaultPeriod = Session["Period"] == null ? "All" : Session["Period"].ToString();
 string defaultLogLevel = Session["LogLevel"] == null ? "All" : Session["LogLevel"].ToString();

 LoggingIndexModel model = new LoggingIndexModel();

 model.Period = Period ?? defaultPeriod;
 model.LogLevel = LogLevel ?? defaultLogLevel;
 model.CurrentPageIndex = page.HasValue ? page.Value - 1 : 0;
 model.PageSize = PageSize.HasValue ? PageSize.Value : 20;

 TimePeriod timePeriod = TimePeriodHelper.GetUtcTimePeriod(model.Period);

 model.LogEvents = repository.GetByDateRangeAndType(model.CurrentPageIndex, model.PageSize, timePeriod.Start, timePeriod.End, model.LogLevel);

 ViewData["Period"] = model.Period;
 ViewData["LogLevel"] = model.LogLevel;
 ViewData["PageSize"] = model.PageSize;

 Session["Period"] = model.Period;
 Session["LogLevel"] = model.LogLevel;

 return View(model);
}

GetByDateRangeAndType does the work for selecting appropriate set of log messages from the database.

public IPagedList<LogEntry> GetByDateRangeAndType(int pageIndex, int pageSize, DateTime start, DateTime end, string logLevel)
{
 IQueryable<LogEntry> list;
 IPagedList<LogEntry> pagedList;

 list = db.LogEntries.Where(e =>
   (e.TimeStamp >= start && e.TimeStamp <= end));

 if (logLevel != LogTypeNames.All.ToString())
 {
  list = list.Where(e => e.LogType.Type.ToLower() == logLevel.ToLower());
 }

 list = list.OrderByDescending(e => e.TimeStamp);
 pagedList = new PagedList<LogEntry>(list, pageIndex, pageSize);
 return pagedList;
}

The data is returned in the form of a PagedList which is implemented as follows:

public interface IPagedList<T> : IList<T>
{
 int PageCount { get; }
 int TotalItemCount { get; }
 int PageIndex { get; }
 int PageNumber { get; }
 int PageSize { get; }

 bool HasPreviousPage { get; }
 bool HasNextPage { get; }
 bool IsFirstPage { get; }
 bool IsLastPage { get; }
}

Main part of the PagedList class:

public class PagedList<T> : List<T>, IPagedList<T>
{
 public PagedList(IEnumerable<T> source, int index, int pageSize)
  : this(source, index, pageSize, null)
 {
 }

 #region IPagedList Members

 public int PageCount { get; private set; }
 public int TotalItemCount { get; private set; }
 public int PageIndex { get; private set; }
 public int PageNumber { get { return PageIndex + 1; } }
 public int PageSize { get; private set; }
 public bool HasPreviousPage { get; private set; }
 public bool HasNextPage { get; private set; }
 public bool IsFirstPage { get; private set; }
 public bool IsLastPage { get; private set; }

 #endregion

 protected void Initialize(IQueryable<T> source, int index, int pageSize, int? totalCount)
 {
  //### argument checking
  if (index < 0)
  {
   throw new ArgumentOutOfRangeException("PageIndex cannot be below 0.");
  }
  if (pageSize < 1)
  {
   throw new ArgumentOutOfRangeException("PageSize cannot be less than 1.");
  }

  //### set source to blank list if source is null to prevent exceptions
  if (source == null)
  {
   source = new List<T>().AsQueryable();
  }

  //### set properties
  if (!totalCount.HasValue)
  {
   TotalItemCount = source.Count();
  }
  PageSize = pageSize;
  PageIndex = index;
  if (TotalItemCount > 0)
  {
   PageCount = (int)Math.Ceiling(TotalItemCount / (double)PageSize);
  }
  else
  {
   PageCount = 0;
  }
  HasPreviousPage = (PageIndex > 0);
  HasNextPage = (PageIndex < (PageCount - 1));
  IsFirstPage = (PageIndex <= 0);
  IsLastPage = (PageIndex >= (PageCount - 1));

  //### add items to internal list
  if (TotalItemCount > 0)
  {
   AddRange(source.Skip((index) * pageSize).Take(pageSize).ToList());
  }
 }
}

PagedList uses some helper methods from the Pager class to render HTML and generate links to other pages of the log.

public class Pager
{
 .....

 /// <summary>
 /// Rendes HTML to display a "pager" control (used at the top and bottom of the list of logged messages)
 /// </summary>
 /// <returns>String of HTML</returns>
 public string RenderHtml()
 {
  int pageCount = (int)Math.Ceiling(totalItemCount / (double)pageSize);
  const int nrOfPagesToDisplay = 10;

  var sb = new StringBuilder();

  // Previous
  if (currentPage > 1)
  {
   sb.Append(GeneratePageLink("<", this.currentPage - 1));
  }
  else
  {
   sb.Append("<span class=\"disabled\"><</span>");
  }

  int start = 1;
  int end = pageCount;

  if (pageCount > nrOfPagesToDisplay)
  {
   int middle = (int)Math.Ceiling(nrOfPagesToDisplay / 2d) - 1;
   int below = (currentPage - middle);
   int above = (currentPage + middle);

   if (below < 4)
   {
    above = nrOfPagesToDisplay;
    below = 1;
   }
   else if (above > (pageCount - 4))
   {
    above = pageCount;
    below = (pageCount - nrOfPagesToDisplay);
   }

   start = below;
   end = above;
  }

  if (start > 3)
  {
   sb.Append(GeneratePageLink("1", 1));
   sb.Append(GeneratePageLink("2", 2));
   sb.Append("...");
  }
  for (int i = start; i <= end; i++)
  {
   if (i == currentPage)
   {
    sb.AppendFormat("<span class=\"current\">{0}</span>", i);
   }
   else
   {
    sb.Append(GeneratePageLink(i.ToString(), i));
   }
  }
  if (end < (pageCount - 3))
  {
   sb.Append("...");
   sb.Append(GeneratePageLink((pageCount - 1).ToString(), pageCount - 1));
   sb.Append(GeneratePageLink(pageCount.ToString(), pageCount));
  }

  // Next
  if (currentPage < pageCount)
  {
   sb.Append(GeneratePageLink(">", (currentPage + 1)));
  }
  else
  {
   sb.Append("<span class=\"disabled\">></span>");
  }
  return sb.ToString();
 }

 /// <summary>
 /// Generates a link to a page
 /// </summary>
 /// <param name="linkText">Text displayed on the page</param>
 /// <param name="pageNumber">Number of the page the link leads to</param>
 /// <returns></returns>
 private string GeneratePageLink(string linkText, int pageNumber)
 {
  var pageLinkValueDictionary = new RouteValueDictionary(linkWithoutPageValuesDictionary) {{"page", pageNumber}};
  var virtualPathData = RouteTable.Routes.GetVirtualPath(this.viewContext.RequestContext, pageLinkValueDictionary);

  if (virtualPathData != null)
  {
   const string linkFormat = "<a href=\"{0}\">{1}</a>";
   return String.Format(linkFormat, virtualPathData.VirtualPath, linkText);
  }
  return null;
 }
}

Details view.

There is nothing special about the details view - it's a usual MVC view that displays entity data.

@model Recipes.Models.LogEntry
@{
    ViewBag.Title = "Details";
}

<link href="@Url.Content("~/Content/logging.css")" rel="stylesheet" type="text/css" />

<h2>Details</h2>

<p>        
    @Html.ActionLink("Back to List", "Index")
</p>

<fieldset>
    <legend>Fields</legend>
        
    <div class="display-label">Id</div>
    <div class="display-field">@Model.Id</div>
        
    <div class="display-label">LogDate</div>
    <div class="display-field">@String.Format("{0:g}", Model.TimeStamp)</div>
        
    <div class="display-label">Source</div>
    <div class="display-field">@Model.Source</div>
        
    <div class="display-label">Type</div>
    <div class="display-field">@Model.LogType.Type</div>
        
    <div class="display-label">Message</div>
    <div class="display-field">
        <pre>@Model.Message.WordWrap(80)</pre>
    </div>
        
    <div class="display-label">StackTrace</div>
    <div class="display-field">@Model.StackTrace</div>                      
        
</fieldset>

<p>        
    @Html.ActionLink("Back to List", "Index")
</p>

Details Controller

public ActionResult Details(string loggerProviderName, string id)
{
 LogEntry logEvent = repository.GetById(id);

 return View(logEvent);
}

The final result is represented in the image below:

MVC logging engine

References

Processing Unhandled Exceptions (C#)
Logging in MVC Part 1- Elmah (and other posts of the series)
MVC Basic Site: Step 2 - Exceptions Management
How is Error.cshtml called in ASP.NET MVC?
by . Also posted on my website

Wednesday, December 19, 2012

Implementing a Tree View - Small Case Study

Implementing the control that allows navigating my blog history could be roughly divided into 4 steps.

1. Select and group posts from the database

Here the LINQ grouping came handy. Starting with grouping posts by year published to create my top level in hierarchy, the query would look like this:

var results = from allPosts in db.Posts.OrderBy(p => p.DateCreated)
     group allPosts by allPosts.DateCreated.Year into postsByYear;

Here results is the enumeration of groups - in my case, groups of posts which were published in the certain year. Posts are grouped by the key, which is defined in the IGrouping interface.

Moving further, I want to create child groups, in my case - posts by the month. I have to add a nested query along these lines

var results = from allPosts in db.Posts.OrderBy(p => p.DateCreated)
     group allPosts by allPosts.DateCreated.Year into postsByYear

     select new
     {
      postsByYear.Key,
      SubGroups = from yearLevelPosts in postsByYear
         group yearLevelPosts by yearLevelPosts.DateCreated.Month into postsByMonth;
     };

This is still not too bad. The first level are posts by year. Each year has SubGroups property which stores the group of posts published in a certian month. Now I finally need to get all the posts published in a month. I end up with the following query:

var results = from allPosts in db.Posts.OrderBy(p => p.DateCreated)
     group allPosts by allPosts.DateCreated.Year into postsByYear

     select new
     {
      postsByYear.Key,
      SubGroups = from yearLevelPosts in postsByYear
         group yearLevelPosts by yearLevelPosts.DateCreated.Month into postsByMonth
         select new
         {
          postsByMonth.Key,
          SubGroups = from monthLevelPosts in postsByMonth
             group monthLevelPosts by monthLevelPosts.Title into post
             select post
         }
     };

It is fully functional and suits my purposes. It is on the edge of being unreadable, however, and if I had to add one more level of depth it would probably be beyond. Following the example from Mitsu Furuta's blog, I make the query generic. The GroupResult class holds the grouping key and the group items. The GroupByMany extension allows for an undefined number of group selectors. This is the code I need to make it work:

public static class MyEnumerableExtensions
{
 public static IEnumerable<GroupResult> GroupByMany<TElement>(this IEnumerable<TElement> elements, params Func<TElement, object>[] groupSelectors)
 {
  if (groupSelectors.Length > 0)
  {
   var selector = groupSelectors.First();

   //reduce the list recursively until zero
   var nextSelectors = groupSelectors.Skip(1).ToArray();
   return
    elements.GroupBy(selector).Select(
     g => new GroupResult
     {
      Key = g.Key,
      Items = g,
      SubGroups = g.GroupByMany(nextSelectors)
     });
  }
  else
   return null;
 }
}

public class GroupResult
{
 public object Key { get; set; }
 public IEnumerable Items { get; set; }
 public IEnumerable<GroupResult> SubGroups { get; set; }
}

And now I can rewrite my query in one line:

var results = db.Posts.OrderBy(p => p.DateCreated).GroupByMany(p => p.DateCreated.Year, p => p.DateCreated.Month);

2. Populate a tree structure that will be used to generate HTML

I used a complete solution suggested by Mark Tinderhold almost without changes.

The BlogEntry class has a Name, which will be rendered, and references to Children and Parent nodes.

public class BlogEntry : ITreeNode<BlogEntry>
{
 public BlogEntry()
 {
  Children = new List<BlogEntry>();
 }

 public string Name { get; set; }
 public BlogEntry Parent { get; set; }
 public List<BlogEntry> Children { get; set; }
}

A list of BlogEntry is populated from my query results

var entries = new List<BlogEntry>();

//years
foreach (var yearPosts in results)
{
 //create "year-level" item
 var year = new BlogEntry { Name = yearPosts.Key.ToString().ToLink(string.Empty) };
 entries.Add(year);

 //months
 foreach (var monthPosts in yearPosts.SubGroups)
 {
  var month = new BlogEntry { Name = new DateTime(2000, (int)monthPosts.Key, 1).ToString("MMMM").ToLink(string.Empty), Parent = year };
  year.Children.Add(month);

  foreach (var postEntry in monthPosts.Items)
  {
   //create "blog entry level" item
   var post = postEntry as Post;
   var blogEntry = new BlogEntry { Name = post.Title.ToLink("/Post/" + post.PostID + "/" + post.Title.ToSeoUrl()), Parent = month };
   month.Children.Add(blogEntry);
  }
 }
}

3. Use the tree structure to generate HTML

The TreeRenderer writes out HTML.

public interface ITreeNode<T>
{
 T Parent { get; }
 List<T> Children { get; }
}

public static class TreeRenderHtmlHelper
{
 public static string RenderTree<T>(this HtmlHelper htmlHelper, IEnumerable<T> rootLocations, Func<T, string> locationRenderer) where T : ITreeNode<T>
 {
  return new TreeRenderer<T>(rootLocations, locationRenderer).Render();
 }
}
public class TreeRenderer<T> where T : ITreeNode<T>
{
 private readonly Func<T, string> locationRenderer;
 private readonly IEnumerable<T> rootLocations;
 private HtmlTextWriter writer;
 public TreeRenderer(IEnumerable<T> rootLocations, Func<T, string> locationRenderer)
 {
  this.rootLocations = rootLocations;
  this.locationRenderer = locationRenderer;
 }
 public string Render()
 {
  writer = new HtmlTextWriter(new StringWriter());
  RenderLocations(rootLocations);
  return writer.InnerWriter.ToString();
 }
 /// <summary>
 /// Recursively walks the location tree outputting it as hierarchical UL/LI elements
 /// </summary>
 /// <param name="locations"></param>
 private void RenderLocations(IEnumerable<T> locations)
 {
  if (locations == null) return;
  if (locations.Count() == 0) return;
  InUl(() => locations.ForEach(location => InLi(() =>
  {
   writer.Write(locationRenderer(location));
   RenderLocations(location.Children);
  })));
 }
 private void InUl(Action action)
 {
  writer.WriteLine();
  writer.RenderBeginTag(HtmlTextWriterTag.Ul);
  action();
  writer.RenderEndTag();
  writer.WriteLine();
 }
 private void InLi(Action action)
 {
  writer.RenderBeginTag(HtmlTextWriterTag.Li);
  action();
  writer.RenderEndTag();
  writer.WriteLine();
 }
}

The renderer will be called the following way from the view:

<div id="treeview" class="treeview">
    @MvcHtmlString.Create(Html.RenderTree(Model.BlogEntries, x => x.Name))
</div>

4. Render the HTML on the webpage

After reviewing a couple of other options, I decided on a jsTree. It has rich capabilities, but to this point I only used the "default" options. I added the tree to the _Layout.cshtml by adding a line of code

@Html.Action("BlogResult", "BlogEntry")

This line calls the function in the BlogEntryController

public PartialViewResult BlogResult()
{
 var results = db.Posts.OrderBy(p => p.DateCreated).GroupByMany(p => p.DateCreated.Year, p => p.DateCreated.Month);

 entries = new List<BlogEntry>();
 
 //code that populates entries - see above

 BlogEntryViewModel model = new BlogEntryViewModel(entries);

 return PartialView(model);
}

The BlogEntryViewModel is extremely simple.

public class BlogEntryViewModel
{
 public List<BlogEntry> BlogEntries { get; set; }

 public BlogEntryViewModel(List<BlogEntry> blogEntries)
 {
  BlogEntries = blogEntries;
 }

 public BlogEntryViewModel()
 {
 }
}

Finally, the partial view that is rendered

@model Recipes.ViewModels.BlogEntryViewModel

@{ Layout = null; }

<link href="@Url.Content("~/Content/blogentry.css")" rel="stylesheet" type="text/css" />

<!-- Tree View jstree -->
<script src="@Url.Content("~/Scripts/jquery.js")" type="text/javascript"></script>
<script src="@Url.Content("~/Scripts/jquery.hotkeys.js")" type="text/javascript"></script>
<script src="@Url.Content("~/Scripts/jquery.cookie.js")" type="text/javascript"></script>
<script src="@Url.Content("~/Scripts/jquery.jstree.js")" type="text/javascript"></script>

<script type="text/javascript">
    jQuery(function ($) {
        $("#treeview").jstree({ "plugins": ["themes", "html_data"] });
    });
</script>

<div class="blogheader">
<h2>Blog Archives</h2>
</div>
<div id="treeview" class="treeview">
    @MvcHtmlString.Create(Html.RenderTree(Model.BlogEntries, x => x.Name))
</div>

What I had to make sure of to make it work:

And for information, this is the contents of blogentry.css

div.treeview, div.blogheader {
    width: 14em;
    background: #eee;
   overflow: hidden;
 text-overflow: ellipsis;
}

div.blogheader h2 
{
    font: bold 11px/16px arial, helvetica, sans-serif;
    display: block;
    border-width: 1px;
    border-style: solid;
    border-color: #ccc #888 #555 #bbb;
    margin: 0;
    padding: 2px 3px;
    
    color: #fff;
    background: #000;
    text-transform: uppercase;
}

The end result looks like that:

Resulting TreeView

Resulting Treeview

References:

How can I hierarchically group data using LINQ?
Playing with Linq grouping: GroupByMany ?
Rendering a tree view using the MVC Framework
jQuery Treeview Plugin Demo
jsTree – Part 1: Introduction
jsTree on GitHub
Best place to learn JStree
by . Also posted on my website

Sunday, December 9, 2012

SEO Basics: Friendly URLs

Implementing SEO-friendly URLs turned out to be much easier than I expected - MVC routing already takes care of the "heavy lifting". The developer only needs to provide a function that returnes the "friendly urls" from strings (product names, blog titles etc.) and to update action links.

1. Routing. A new route needs to be added. It has to be added above the default route so that MVC framework attempted to match it first. The seofriendly parameter can be pretty much anything that will satisfy valid url requirements.

routes.MapRoute(
 name: \"SEOFriendly\",
 url: \"{controller}/{action}/{id}/{seofriendly}\",
 defaults: new { controller = \"Home\", action = \"Index\", id = UrlParameter.Optional, seofriendly = \"\" }
);

2. Creating friendly urls. Here is an example I found on the web and added it "as is".

public static string ToSeoUrl(this string url)
{
 // make the url lowercase
 string encodedUrl = (url ?? \"\").ToLower();

 // replace & with and
 encodedUrl = Regex.Replace(encodedUrl, @\"\&+\", \"and\");

 // remove characters
 encodedUrl = encodedUrl.Replace(\"'\", \"\");

 // remove invalid characters
 encodedUrl = Regex.Replace(encodedUrl, @\"[^a-z0-9]\", \"-\");

 // remove duplicates
 encodedUrl = Regex.Replace(encodedUrl, @\"-+\", \"-\");

 // trim leading & trailing characters
 encodedUrl = encodedUrl.Trim('-');

 return encodedUrl;
}  

3. Making use of the friendly url. Just adding an extra parameter to the object.

Before:

<div class=\"display-button\">@Html.ActionLink(\"Edit\", \"Edit\", new { id=item.PostID }) </div>
<div class=\"display-button\">@Html.ActionLink(\"Details\", \"Details\", new { id = item.PostID }) </div>
<div class=\"display-button\">@Html.ActionLink(\"Delete\", \"Delete\", new { id = item.PostID }) </div>

After:

<div class=\"display-button\">@Html.ActionLink(\"Edit\", \"Edit\", new { id=item.PostID, seofriendly = item.Title.ToSeoUrl() }) </div>
<div class=\"display-button\">@Html.ActionLink(\"Details\", \"Details\", new { id = item.PostID, seofriendly = item.Title.ToSeoUrl() }) </div>
<div class=\"display-button\">@Html.ActionLink(\"Delete\", \"Delete\", new { id = item.PostID, seofriendly = item.Title.ToSeoUrl() }) </div>

References:

SEO-Friendly URLs in ASP.Net MVC 3
How can I create a friendly URL in ASP.NET MVC?
by . Also posted on my website

Wednesday, November 28, 2012

MVC and SEO basics: inject title, keywords and description into views

Almost by accident, I came across Google's starter guide for SEO optimisation and decided that it is a good idea to make some improvements I've been neglecting. Here's what I did so far and how I applied it to the MVC framework.

1. Create unique, accurate page titles

One way to do it with the MVC framework is to create a placeholder on the master page and then override it on the view page.

Master:


    
    
        <%=this.Page.Title%>
    

View:


       Home Page

For now, I chose the easier approach to set the title in the _Layout.cshtml

@ViewBag.Title

And assign it in each view separately

@{
    ViewBag.Title = "Main Page - The Stepping Stone Markov Chain Algorithm - MVC Stepping Stone Example";
}

2. Make use of the "description" and "keywords" meta tags

This takes a little more work. Here's my chosen approach: First, make sure each controller inherits from the BaseController. Then create two new classes, MetaDescriptionAttribute and MetaKeywordsAttribute, and inherit them from System.Attribute

public class MetaDescriptionAttribute : Attribute
{
 private readonly string _parameter;

 public MetaDescriptionAttribute(string parameter)
 {
  _parameter = parameter;
 }

 public string Parameter { get { return _parameter; } }
}

public class MetaKeywordsAttribute : Attribute
{
 private readonly string _parameter;

 public MetaKeywordsAttribute(string parameter)
 {
  _parameter = parameter;
 }

 public string Parameter { get { return _parameter; } }
}

In BaseController, override OnActionExecuting

protected override void OnActionExecuting(ActionExecutingContext filterContext)
{
 var keywords = filterContext.ActionDescriptor.GetCustomAttributes(typeof(MetaKeywordsAttribute), false);
 if (keywords.Length == 1)
  ViewData["MetaKeywords"] = ((MetaKeywordsAttribute)(keywords[0])).Parameter;

 var description = filterContext.ActionDescriptor.GetCustomAttributes(typeof(MetaDescriptionAttribute), false);
 if (description.Length == 1)
  ViewData["MetaDescription"] = ((MetaDescriptionAttribute)(description[0])).Parameter;

 base.OnActionExecuting(filterContext);
}

Decorate the appropriate controller method with newly created attributes

[MetaKeywords("C#, MVC, Markov Chain, Stepping Stone, Programming")]
[MetaDescription("Stepping Stone Markov Chain model is an example that has been used in the study of genetics. In this model we have an n-by-n array of squares, and each square is initially any one of k different colors. For each step, a square is chosen at random. This square then chooses one of its eight neighbors at random and assumes the color of that neighbor")]
public ActionResult Index()
{
 SteppingStoneHelpers.CreateNewTable();
 HtmlString table = new HtmlString(SteppingStoneHelpers.table.ToString());
 return View(table);
}

Finally, in the _Layout.cshtml, add the following


All done! There we go, the resulting html:




    
    
    Main Page - The Stepping Stone Markov Chain Algorithm - MVC Stepping Stone Example
    
    

References:

Google Starter Guide
ASP.NET MVC - View with master page, how to set title?
asp.net mvc - strategy for including SEO information such as meta keywords and descriptions
by . Also posted on my website

Monday, November 26, 2012

WebGrid: AJAX Updates, Server Sorting

This is a brief summary of the changes I did to implement the AJAX updates to the WebGrid and sorting behaviour. I plan to put more detailed notes and the source code on my website

To use AJAX and update grid content, firstly the grid needs to be placed in the div which has an id. The ajaxUpdateContainerId has to be specified in the WebGrid declaration. To put it simple, in my main view I have a div

<div id="wbgrid" style="float:left; min-width:500px;">
 @Html.Partial("_WebGrid", Model)
</div>

And in the partial view the div name wbgrid is specified as ajaxUpdateContainerId.

@{ var grid = new WebGrid<Recipes.Models.Yahoo.YahooData>(null, rowsPerPage: 5, defaultSort: "YahooSymbolName", ajaxUpdateContainerId: "wbgrid");
grid.Bind(Model.Datas, rowCount: Model.TotalRows, autoSortAndPage: false);
}

The link on the WebGrid column has the following format http://localhost/Yahoo/Index?sort=DateTime&sortdir=ASC

Therefore, the controller function can automatically receive those parameters with the following signature:

public ActionResult Index(int page = 1, string sort = "YahooSymbolName", string sortDir = "Ascending")

The parameters will be then passed over to the function that retrieves data

public List<YahooData> GetData(out int totalRecords, int pageSize, int pageIndex, string sort = "YahooSymbolName", SortDirection sortOrder = SortDirection.Ascending )
{
 IQueryable<YahooData> data = db.YahooData;
 totalRecords = data.Count();

 Func<IQueryable<YahooData>, bool, IOrderedQueryable<YahooData>> applyOrdering = _dataOrderings[sort];
 data = applyOrdering(data, sortOrder == SortDirection.Ascending);

 List<YahooData> result = data.ToList();
 if(pageSize > 0 && pageIndex >=0)
 {
  result = result.Skip(pageIndex*pageSize).Take(pageSize).ToList();
 }
 return result;
}

A couple of helper functions are utilized by GetData: CreateOrderingFunc and _dataOrderings

// helpers that take an IQueryable<Product> and a bool to indicate ascending/descending
// and apply that ordering to the IQueryable and return the result
private readonly IDictionary<string, Func<IQueryable<YahooData>, bool, IOrderedQueryable<YahooData>>>
 _dataOrderings = new Dictionary<string, Func<IQueryable<YahooData>, bool, IOrderedQueryable<YahooData>>>
       {
        {"YahooSymbolName", CreateOrderingFunc<YahooData, string>(p=>p.DataName)},
        {"Ask", CreateOrderingFunc<YahooData, decimal?>(p=>p.Ask)},
        {"Time", CreateOrderingFunc<YahooData, DateTime>(p=>p.DateTime)}
        // Add for more columns ...
       };

/// returns a Func that takes an IQueryable and a bool, and sorts the IQueryable (ascending or descending based on the bool).
/// The sort is performed on the property identified by the key selector.
private static Func<IQueryable<T>, bool, IOrderedQueryable<T>> CreateOrderingFunc<T, TKey>(Expression<Func<T, TKey>> keySelector)
{
 return
  (source, ascending) => ascending ? source.OrderBy(keySelector) : source.OrderByDescending(keySelector);
}

Finally, to complete the functional example, I added a jQuery dialog that displays the data that was retrieved from Yahoo. In the view, the RetrieveData function triggers the controller action AddDataToDB (which calls the Yahoo website and adds results to the database).

function RetrieveData() {
 $.post('@Url.Action("AddDataToDB","yahoo")',
 function (d) {
  ShowDialog(d.o);
 });
}

function ShowDialog(msg) {
 $('<div/>').dialog({ title: 'Retrieved the following data', width: 450, height: 250, close: function(){ location.reload(); }}).html(msg);
}

AddDataToDB returns a Json result, containing the html table.

public ActionResult AddDataToDB()
{
 List<YahooData> datas = GetSingleSet();
 datas.ForEach(d => db.YahooData.Add(d));
 db.SaveChanges();

 string s = "<table><thead><tr class=\"webgrid-header\"><th>Company</th><th>Time</th><th>LTP</th><th>Volume</th><th>Ask</th><th>Bid</th><th>High</th><th>Low</th></tr></thead><tbody>";

 foreach (var yahooData in datas)
 {
  s = s + "<tr class=\"webgrid-row-style\">" + 
   "<td class=\"company\">" + yahooData.DataName + "</td>" +
   "<td class=\"time\">" + yahooData.DateTime.ToString("dd/MM/yyyy hh:mm") + "</td>" +
   "<td class=\"ask\">" + yahooData.LTP + "</td>" +
   "<td class=\"volume\">" + yahooData.Volume + "</td>" +
   "<td class=\"ask\">" + yahooData.Ask + "</td>" +
   "<td class=\"ask\">" + yahooData.Bid + "</td>" +
   "<td class=\"ask\">" + yahooData.High + "</td>" +
   "<td class=\"ask\">" + yahooData.Low + "</td>" +
   "</tr>";
 }

 s = s + "</tbody></table>";

 return Json(new { o = s });
}

The result is then utilised by the ShowDialog function, that displays a jQuery dialog. When the user closes the dialog, the page is refreshed so that the WebGrid contents are updated with the latest data retrieved.

Complete example

References

Get the Most out of WebGrid in ASP.NET MVC
by . Also posted on my website

Sunday, November 25, 2012

WebGrid: Stronly Typed with Server Paging

Continuing with the WebGrid, I first made it strongly typed. To achieve that, I created a derived type WebGrid. The source came from the reference at the end of the post, and the code I used is displayed below too. While it does not seem to change much in terms of functionality, the main advantage is that the IntelliSense and compiler checking will work with the grid now.

Next I added server paging. Why wouldn't I use the built-in paging? Well, the database table behind WebGrid may have hundreds of records. I wouldn't want to pass it all to my ViewModel and then to the view just to display 5 or 10 of the records I actually need. It is handy that the WebGrid paging is in the form of http://website/Items/ShowAll?page=3. This way my controller knows which page is to be displayed and can preselect the data just for this page only.

To implement paging, I added the TotalRows to the model - that will specify the total number of records in the database table.

The controller method now looks as follows:

public ActionResult Index(int page=1)
{
 int totalRecords;
 List<YahooData> datas = GetData(out totalRecords, pageSize: 5, pageIndex: page - 1);
 List<YahooSymbol> symbols = db.YahooSymbols.ToList();
 YahooSymbol symbol = symbols.First();
 int id = symbol.YahooSymbolID;
 return View(new YahooViewModel(id, symbol, symbols, datas, totalRecords));
}

public List<YahooData> GetData(out int totalRecords, int pageSize, int pageIndex)
{
 List<YahooData> data = GetData();
 totalRecords = data.Count;
 if(pageSize > 0 && pageIndex >=0)
 {
  data = data.Skip(pageIndex*pageSize).Take(pageSize).ToList();
 }
 return data.ToList();
}

The concept is quite simple - get data from the database table, identify the records that will be displayed on the WebGrid page that is requested, and only pass those records to the view. The WebGrid part of the view now looks as follows

<div id="webgrid" style="float:left; min-width:500px;">
 @{ var grid = new WebGrid<ViewModels.YahooData>(null, rowsPerPage: 5, defaultSort: "YahooSymbolName");
    grid.Bind(Model.Datas, rowCount: Model.TotalRows, autoSortAndPage: false);
    @grid.GetHtml(columns: grid.Columns( 
    grid.Column("DataName", header:"Company", format:@<text>@Html.ActionLink((string)item.DataName, "Details", "Company", new {id=item.SymbolId}, null)</text>),
    grid.Column("DateTime", header:"Time", style:"time", format:@<text>@item.DateTime.ToString("dd/MM/yyyy hh:mm")</text>), 
    grid.Column("LTP"), grid.Column("Volume"), grid.Column("Ask"), grid.Column("Bid"), grid.Column("High"), grid.Column("Low")),
    tableStyle: "webGrid", headerStyle: "header", alternatingRowStyle: "alt");
  }
</div>

My plan from here is to implement AJAX updates to the WebGrid content.

The strongly typed WebGrid samples:

//Strongly Typed WebGrid
public class WebGrid<T> : WebGrid
{
 // Wrapper for System.Web.Helpers.WebGrid that preserves the item type from the data source
 public WebGrid(IEnumerable<T> source = null, IEnumerable<string> columnNames = null, string defaultSort = null, int rowsPerPage = 10, bool canPage = true, bool canSort = true, string ajaxUpdateContainerId = null, string ajaxUpdateCallback = null, string fieldNamePrefix = null, string pageFieldName = null, string selectionFieldName = null, string sortFieldName = null, string sortDirectionFieldName = null)
  : base(source.SafeCast<object>(), columnNames, defaultSort, rowsPerPage, canPage, canSort, ajaxUpdateContainerId, ajaxUpdateCallback, fieldNamePrefix, pageFieldName, selectionFieldName, sortFieldName, sortDirectionFieldName)
 {
 }
 public WebGridColumn Column(string columnName = null, string header = null, Func<T, object> format = null, string style = null, bool canSort = true)
 {
  Func<dynamic, object> wrappedFormat = null;
  if (format != null)
  {
   wrappedFormat = o => format((T)o.Value);
  }
  WebGridColumn column = base.Column(columnName, header, wrappedFormat, style, canSort);
  return column;
 }
 public WebGrid<T> Bind(IEnumerable<T> source, IEnumerable<string> columnNames = null, bool autoSortAndPage = true, int rowCount = -1)
 {
  base.Bind(source.SafeCast<object>(), columnNames, autoSortAndPage, rowCount);
  return this;
 }
}

public static class EnumerableExtensions
{
 public static IEnumerable<TTarget> SafeCast<TTarget>(this IEnumerable source)
 {
  return source == null ? null : source.Cast<TTarget>();
 }
}
//WebGrid extensions
public static class WebGridExtensions
{
 // Light-weight wrapper around the constructor for WebGrid so that we get take advantage of compiler type inference
 public static WebGrid<T> Grid<T>(this HtmlHelper htmlHelper, IEnumerable<T> source, IEnumerable<string> columnNames = null,
   string defaultSort = null, int rowsPerPage = 10, bool canPage = true, bool canSort = true,
   string ajaxUpdateContainerId = null, string ajaxUpdateCallback = null, string fieldNamePrefix = null,
   string pageFieldName = null, string selectionFieldName = null, string sortFieldName = null, string sortDirectionFieldName = null)
 {
  return new WebGrid<T>(source, columnNames, defaultSort, rowsPerPage,
    canPage, canSort, ajaxUpdateContainerId, ajaxUpdateCallback, fieldNamePrefix, 
    pageFieldName, selectionFieldName, sortFieldName, sortDirectionFieldName);
 }

 // Light-weight wrapper around the constructor for WebGrid so that we get take advantage of compiler type inference and to automatically call Bind to disable the automatic paging and sorting (use this for server-side paging)
 public static WebGrid<T> ServerPagedGrid<T>(this HtmlHelper htmlHelper, IEnumerable<T> source, int totalRows, IEnumerable<string> columnNames = null,
   string defaultSort = null, int rowsPerPage = 10, bool canPage = true, bool canSort = true, string ajaxUpdateContainerId = null, 
   string ajaxUpdateCallback = null, string fieldNamePrefix = null,
   string pageFieldName = null, string selectionFieldName = null, string sortFieldName = null, string sortDirectionFieldName = null)
 {
  var webGrid = new WebGrid<T>(null, columnNames, defaultSort, rowsPerPage, canPage,
    canSort, ajaxUpdateContainerId, ajaxUpdateCallback, fieldNamePrefix,
    pageFieldName, selectionFieldName, sortFieldName, sortDirectionFieldName);
  return webGrid.Bind(source, rowCount: totalRows, autoSortAndPage: false); ;
 }
}

References

Get the Most out of WebGrid in ASP.NET MVC
by . Also posted on my website

Tuesday, November 13, 2012

Starting with WebGrid

WebGrid is an HTML helper provided as part of the MVC framework to simplify rendering tabular data. It is actually very simple to start with WebGrid. The following is enough to create a complete working example:

@model YahooViewModel

...

@{ var grid = new WebGrid(Model.Datas);
   @grid.GetHtml();
 }

Here the "Datas" is my list of YahooData entities. This, however, looks a little ugly, so I'll spend a few minutes on styling straight away. The following is a basic style for a WebGrid

<style type="text/css">
    .webGrid {margin: 4px; border-collapse: collapse; width: 300px;}
    .header {background-color: #E8E8E8; font-weight: bold; color: #FFF;}
    .webGrid th, .webGrid td { border: 1px solid #C0C0C0; padding: 5px;}
    .alt {background-color: #E8E8E8; color: #000;}
</style>

The style is applied as follows

@{ var grid = new WebGrid(Model.Datas);
   @grid.GetHtml(tableStyle: "webGrid", headerStyle: "header", alternatingRowStyle: "alt");
 }

First WebGrid

I don't want to show each and every column to the user. I can rewrite the WebGrid specifying the actual columns to show. Only specified columns will be displayed. Also, now the order of the columns is the same as the order I define them.

@{ var grid = new WebGrid(Model.Datas, 
       columnNames: new[] {"DataName", "Date", "LTP", "Time", "Volume", "Ask", "Bid", "High", "Low"});
   @grid.GetHtml(tableStyle: "webGrid", headerStyle: "header", alternatingRowStyle: "alt");
 }

Specific columns

Another way to do it is to actually define columns explicitly. First advantage is that I can now specify a name for the header.

@{ var grid = new WebGrid(Model.Datas, columnNames: new[] {"DataName", "Date", "LTP", "Time", "Volume", "Ask", "Bid", "High", "Low"});
   @grid.GetHtml(columns: grid.Columns( grid.Column("DataName", header: "Company"), grid.Column("Date"), grid.Column("LTP"), grid.Column("Time"), grid.Column("Volume"), 
   grid.Column("Ask"), grid.Column("Bid"), grid.Column("High"), grid.Column("Low")),
   tableStyle: "webGrid", headerStyle: "header", alternatingRowStyle: "alt");
 }

Finally, let's assume I want to let the user click the Company name and navigate to the page that provides some more information about the company. I can use format parameter of the Column to display an ActionLink.

@{ var grid = new WebGrid(Model.Datas, columnNames: new[] {"DataName", "Date", "LTP", "Time", "Volume", "Ask", "Bid", "High", "Low"});
   @grid.GetHtml(columns: grid.Columns( 
   grid.Column("DataName", header:"Company", format:@<text>@Html.ActionLink((string)item.DataName, "Details", "Company", new {id=item.SymbolId}, null)</text>),
   grid.Column("Date"), grid.Column("LTP"), grid.Column("Time"), grid.Column("Volume"), 
   grid.Column("Ask"), grid.Column("Bid"), grid.Column("High"), grid.Column("Low")),
   tableStyle: "webGrid", headerStyle: "header", alternatingRowStyle: "alt");
 }

The ActionLink will be created in the following format: "http://localhost/Company/Details/1"

Finally (for today) I would like to combine Date and Time in a single column and format it. The last bit of code shows how to format the date in the column and how to apply the style to a specific column.

<style type="text/css">

...

    .time {width: 200px; font-weight:bold;}
</style>

@{ var grid = new WebGrid(Model.Datas, columnNames: new[] {"DataName", "Date", "LTP", "Time", "Volume", "Ask", "Bid", "High", "Low"});
   @grid.GetHtml(columns: grid.Columns( 
   grid.Column("DataName", header:"Company", format:@<text>@Html.ActionLink((string)item.DataName, "Details", "Company", new {id=item.SymbolId}, null)</text>),
   grid.Column("DateTime", header:"Time", style:"time", format:@<text>@item.DateTime.ToString("dd/MM/yyyy hh:mm")</text>), 
   grid.Column("LTP"), grid.Column("Volume"), grid.Column("Ask"), grid.Column("Bid"), grid.Column("High"), grid.Column("Low")),
   tableStyle: "webGrid", headerStyle: "header", alternatingRowStyle: "alt");
 }

Better formatting

The plan from here is to add server-side paging to reduce the stress on the view when the number of records is high.

References

Get the Most out of WebGrid in ASP.NET MVC
WebGrid WebHelper in ASP.NET MVC 3 RC
by . Also posted on my website

Wednesday, November 7, 2012

Yahoo Data Download

Stock data can be downloaded from http://finance.yahoo.com/d/quotes.csv?s=[stock symbol string]&f=[special tags]. Some tags are listed in the table at the end of the post, but that's not the point. I'll be using a static url for a code example, such as http://download.finance.yahoo.com/d/quotes.csv?s=GOOG+AAPL+MSFT+YHOO&f=snd1l1t1vb3b2hg which will return values for Symbol, Name, Last trade date, Last trade (price only), Last trade time, Volume, Bid (real-time), Ask (real-time), Day's High and Day's Low.

The plan is to have a list of symbols (configurable), to get the data from yahoo and dynamically load the data into the WebGrid control. Therefore, I started with the basic ViewModel that has two sets of entities - one for the symbols and one for the data itself. Eventually the list of symbols will be made configurable.

//ViewModel
public class YahooViewModel
{
 public List<YahooData> Datas { get; set; }
 public List<YahooSymbol> Symbols { get; set; }
 public YahooSymbol Symbol { get; set; }
 public int YahooSymbolID { get; set; }

 public YahooViewModel(int symbolid, YahooSymbol symbol, List<YahooSymbol> symbols, List<YahooData> datas)
 {
  Symbol = symbol;
  YahooSymbolID = symbolid;
  Symbols = symbols;
  Datas = datas;
 }
}

The controller requests and populates the data, and later the automatic authentication may be added as described in the previous post.

//Controller
public ActionResult Index()
{
 List<YahooData> datas = GetData();
 List<YahooSymbol> symbols = db.YahooSymbols.ToList();
 YahooSymbol symbol = symbols.First();
 int id = symbol.YahooSymbolID;
 return View(new YahooViewModel(id, symbol, symbols, datas));
}

public List<YahooData> GetData()
{
 List<YahooData> datas = new List<YahooData>();

 HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://download.finance.yahoo.com/d/quotes.csv?s=GOOG+AAPL+MSFT+YHOO&f=snd1l1t1vb3b2hg");
 HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

 using (StreamReader streamReader = new StreamReader(resp.GetResponseStream()))
 {
  string t = streamReader.ReadToEnd();
  string[] strings = t.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);
  datas = InsertData(strings);
 }
 return datas;
}

private List<YahooData> InsertData(string[] lines)
{
 List<YahooData> datas = new List<YahooData>();

 foreach (string line in lines)
 {
  if (!String.IsNullOrEmpty(line))
  {
   YahooData datum = GetDatum(line);
   datas.Add(datum);
  }
 }
 return datas;
}

private YahooData GetDatum(string line)
{
 var datum = new YahooData();
 string[] splitLine = line.Split(',');
 datum = new YahooData()
 {
  DataName = splitLine[1].Replace("\"", ""),
  Date = DateTime.ParseExact(splitLine[2].Replace("\"", ""), "MM/d/yyyy", CultureInfo.InvariantCulture),
  LTP = decimal.Parse(splitLine[3]),
  Time = DateTime.Parse(splitLine[4].Replace("\"", "")),
  Volume = decimal.Parse(splitLine[5]),
  Ask = decimal.Parse(splitLine[6]),
  Bid = decimal.Parse(splitLine[7]),
  High = decimal.Parse(splitLine[8]),
  Low = decimal.Parse(splitLine[9])
 };
 return datum;
}

The symbols are seeded initially, and may be later made configurable.

//Seeding database with initial values
public class SampleData : DropCreateDatabaseIfModelChanges<RecipesEntities>
{
 protected override void Seed(RecipesEntities context)
 {
  AddSymbols(context);
 }
}

public static void AddSymbols(RecipesEntities context)
{
 List<YahooSymbol> symbols = new List<YahooSymbol>
 {
  new YahooSymbol {YahooSymbolID = 1, YahooSymbolName = "GOOG"},
  new YahooSymbol {YahooSymbolID = 2, YahooSymbolName = "AAPL"},
  new YahooSymbol {YahooSymbolID = 3, YahooSymbolName = "MSFT"},
  new YahooSymbol {YahooSymbolID = 4, YahooSymbolName = "YHOO"}
 };

 symbols.ForEach(p => context.YahooSymbols.Add(p));
 context.SaveChanges();
}

Finally, the table of tags and their meanings - just for the interest.

a Ask a2 Average Daily Volume a5 Ask Size
b Bid b2 Ask (Real-time) b3 Bid (Real-time)
b4 Book Value b6 Bid Size c Change & Percent Change
c1 Change c3 Commission c6 Change (Real-time)
c8 After Hours Change (Real-time) d Dividend/Share d1 Last Trade Date
d2 Trade Date e Earnings/Share e1 Error Indication (returned for symbol changed / invalid)
e7 EPS Estimate Current Year e8 EPS Estimate Next Year e9 EPS Estimate Next Quarter
f6 Float Shares g Day's Low h Day's High
j 52-week Low k 52-week High g1 Holdings Gain Percent
g3 Annualized Gain g4 Holdings Gain g5 Holdings Gain Percent (Real-time)
g6 Holdings Gain (Real-time) i More Info i5 Order Book (Real-time)
j1 Market Capitalization j3 Market Cap (Real-time) j4 EBITDA
j5 Change From 52-week Low j6 Percent Change From 52-week Low k1 Last Trade (Real-time) With Time
k2 Change Percent (Real-time) k3 Last Trade Size k4 Change From 52-week High
k5 Percebt Change From 52-week High l Last Trade (With Time) l1 Last Trade (Price Only)
l2 High Limit l3 Low Limit m Day's Range
m2 Day's Range (Real-time) m3 50-day Moving Average m4 200-day Moving Average
m5 Change From 200-day Moving Average m6 Percent Change From 200-day Moving Average m7 Change From 50-day Moving Average
m8 Percent Change From 50-day Moving Average n Name n4 Notes
o Open p Previous Close p1 Price Paid
p2 Change in Percent p5 Price/Sales p6 Price/Book
q Ex-Dividend Date r P/E Ratio r1 Dividend Pay Date
r2 P/E Ratio (Real-time) r5 PEG Ratio r6 Price/EPS Estimate Current Year
r7 Price/EPS Estimate Next Year s Symbol s1 Shares Owned
s7 Short Ratio t1 Last Trade Time t6 Trade Links
t7 Ticker Trend t8 1 yr Target Price v Volume
v1 Holdings Value v7 Holdings Value (Real-time) w 52-week Range
w1 Day's Value Change w4 Day's Value Change (Real-time) x Stock Exchange
y Dividend Yield
by . Also posted on my website

Thursday, October 25, 2012

Automating Website Authentication

Recently I had to implement automated logging on the website. In my particular case, that was Yahoo.com website, so the code snippets will be specific to this site. It should not be hard to modify them for other purposes. I developed two separate ways to achieve that, the first one has more code and is more complex (have to subscribe to two events and make more logical checks), but I figured it out first. It makes use of the WebBrowser class.

Create an instance of the WebBrowser and subscribe to Navigated and DocumentCompleted events

_browser = new WebBrowser();
_browser.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(browser_DocumentCompleted);
_browser.Navigated += new WebBrowserNavigatedEventHandler(browser_Navigated);

On a timeline, first meaningful event that will be caught is browser_DocumentCompleted on the login.yahoo.com. The code then will analyse the controls on the page. For successful operation, I need to know actual names of the login and password input elements. I find them by name, and set the values to actual login and password. Then I simulate the click on the login button.

Next meaningful event is browser_Navigated on my.yahoo.com page - see below.

After that, I'll point the browser to the url of the document I want to read or download. I'll catch browser_DocumentCompleted again, on that page, and read the contents using the WebBrowser.Document.Body.InnerText (end of the code snippet).

void browser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
 //loaded the Yahoo login page
 if (_browser.Url.AbsoluteUri.Contains(LoginUrl))
 {
  if (_browser.Document != null)
  {
   //Find and fill the "username" textbox
   HtmlElementCollection collection = _browser.Document.GetElementsByTagName("input");
   foreach (HtmlElement element in collection)
   {
    string name = element.GetAttribute("id");
    if (name == "username")
    {
     element.SetAttribute("value", _login);
     break;
    }
   }

   //Find and fill the "password" field
   foreach (HtmlElement element in collection)
   {
    string name = element.GetAttribute("id");
    if (name == "passwd")
    {
     element.SetAttribute("value", _password);
     break;
    }
   }

   //Submit the form
   collection = _browser.Document.GetElementsByTagName("button");
   foreach (HtmlElement element in collection)
   {
    string name = element.GetAttribute("id");
    if (name == ".save")
    {
     element.InvokeMember("click");
     break;
    }
   }
  }
 }
 
 //downloaded "quote.csv"
 if(_browser.Url.AbsoluteUri.Contains(".csv"))
 {
  if (_browser.Document != null && _browser.Document.Body != null)
  {
   string s = _browser.Document.Body.InnerText;
  }
 }
}

Here I actually copy the cookies, but that is not necessary. The WebBrowser will keep them internally and use them. The purpose of this code is to check if the browser is redirected to "my.yahoo.com", which is the indication of successful login. Further logic may be applied from here.
void browser_Navigated(object sender, WebBrowserNavigatedEventArgs e)
{
 //Successful login takes to "my.yahoo.com"
 if (_browser.Url.AbsoluteUri.Contains(MyYahoo))
 {
  if (_browser.Document != null && !String.IsNullOrEmpty(_browser.Document.Cookie))
  {
   _cookies = _browser.Document.Cookie;
  }
 }
}

The second approach is shorted, but it took me longer to figure out. Here I have to explicitly use the CookieContainer to save the cookies "harvested" by the HttpWebRequest which does the login, and use them in the HttpWebRequest which asks for the file after authentication. Of course, I still need to know what are the names of login and password elements, because I'm sending the values in the POST data.

Step one - authentication

string strPostData = String.Format("login={0}&passwd={1}", _login, _password);

// Setup the http request.
HttpWebRequest wrWebRequest = WebRequest.Create(LoginUrl) as HttpWebRequest;
wrWebRequest.Method = "POST";
wrWebRequest.ContentLength = strPostData.Length;
wrWebRequest.ContentType = "application/x-www-form-urlencoded";
_yahooContainer = new CookieContainer();
wrWebRequest.CookieContainer = _yahooContainer;

// Post to the login form.
using (StreamWriter swRequestWriter = new StreamWriter(wrWebRequest.GetRequestStream()))
{
 swRequestWriter.Write(strPostData);
 swRequestWriter.Close();           
}

// Get the response.
HttpWebResponse hwrWebResponse = (HttpWebResponse)wrWebRequest.GetResponse();

Step two - accessing data using the cookies.

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(_downloadUrl);
req.CookieContainer = _yahooContainer;
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

using(StreamReader streamReader = new StreamReader(resp.GetResponseStream()))
{
 string t = streamReader.ReadToEnd();
}

References:

WebBrowser control
submit a form data from external address !
C# Login to Website via program
how to login to yahoo website programatically by . Also posted on my website

Sunday, October 14, 2012

Crystal Reports, C#, Object as Data Source

Based on the Ping example from one of the recent posts, I'm continuing it with the Crystal Reports example, because I have never used Crystal Reports until now. So consider that the following class was added to the solution.

public class PingResult
{
 public string sPacketsSent;
 public string sPacketsReceived;
 public string sPacketsLost;
 public string sPacketTime;
}

An instance of the class is created and populated with results when the ping command runs and its results are parsed. So I have one of the simplest possible objects to use as a source for a report. The next step is to add a Crystal Report to the application. Visual Studio 2010 has an item to add called "Crystal Report", but they are not installed.

Crystal Report Online Template

When I select this item, I'm prompted with a download screen.

Download Crystal Reports

Installation is simple - just following the instructions. I chose the standard version, and the download size is 288MB. After a few short hours, I have a Crystal Report called pingReport.mht in my solution. I have an option to configure my report using the wizard, which I'm doing by choosing the following options:

On the first page, Using the Report Wizard, and Standard layout.

Create a New Crystal Report Document

On the next page, I choose to populate my report from .NET Object in project data. My PingResult class is in the list, and I move it to the Selected Tables.

Choose the data you want to report on

Then I choose the fields to display, of which I select all.

Choose the information to display on the report

I skip Grouping, Report Selection and Report Style, leaving default values. Now I have my report editor. I only want to do a little change - make the headers human readable, so I edit them in the following manner

Edit text object

Now some tricks: when I build my solution, I get the following error:

Warning 1 The referenced assembly "CrystalDecisions.CrystalReports.Engine, Version=13.0.2000.0, Culture=neutral, PublicKeyToken=692fbea5521e1304, processorArchitecture=MSIL" could not be resolved because it has a dependency on "System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" which is not in the currently targeted framework ".NETFramework,Version=v4.0,Profile=Client". Please remove references to assemblies not in the targeted framework or consider retargeting your project. PingTest

This is quite obvious, I need to add a reference to System.Web, but to do that I need to first change the Target Framework default setting of .NET Framework 4 Client Profile to just .NET Framework 4. Now the project builds.

In my toolbox I now have the pingReport1 component, which I add to the form.

pingReport1

I also need a report viewer, which I also add

CrystalReportViewer

The final effort: connect the report with the object that contains data. Here's how:

pingReport1 myReport = new pingReport1();
myReport.SetDataSource(new[] { pingResult });
pingReportViewer1.ReportSource = myReport;

Looks simple, just note how the objects are wrapped into array. This is important.

And the final trick, when I run the application, I get the FileNotFound exception

FileNotFoundException

which is resolved by adding the useLegacyV2RuntimeActivationPolicy="true" parameter to the startup node of my app.config which now looks like the following

<?xml version="1.0"?>
<configuration>
<startup  useLegacyV2RuntimeActivationPolicy="true">
  <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
</startup>
</configuration>

At this point I consider my small example complete.

Complete

References

Creating Crystal Reports using C# with Datasets
Very Odd situation with CrystalReport and/or Visual studio 2010 I don't know maybe .Net Framework
Can crystal reports get data from an object data source? by . Also posted on my website