Oxford Computer Consultants and Infoshare selected to help detect fraud in Warwickshire

Warwickshire Counter Fraud Partnership select Oxford Computer Consultants (OCC) and Infoshare Limited to provide Data Matching Software for Fraud Prevention

Warwick January 2016:  Following a competitive tender, Warwickshire Counter Fraud Partnership WCFP are delighted to announce the award of the contract for Data Matching Software for Fraud Prevention jointly to Oxford Computer Consultants (OCC) and Infoshare as part of the Department for Communities and Local Government (DCLG) counter fraud initiative. By bringing together and matching a number of key datasets across all of the local authorities within Warwickshire, WCFP will improve the detection and prevention of fraud.

Oxford Computer Consultants (OCC) were selected to lead the project from the government list of suppliers approved to deliver systems improving data sharing, interoperability and cost efficiency.

Jointly building on Infoshare’s strong Single Citizen View solution, OCC will develop software for detecting fraud and for the tracking and management of investigations.  Comprehensive reporting from both companies will give WCFP important information in their fight against fraud. OCC also provide system integration, project management and first line support, drawing on their extensive experience delivering to Local Authorities. OCC have worked closely with Infoshare since 2008.

Infoshare’s ClearCore software underpins their Single Citizen View solution and provides evidence based data matching across the datasets from Warwickshire County Council and the five district and borough councils within the partnership. ClearCore enables data to be linked and shared across government agencies to create a robust and trusted single view of citizen’s data. This provides an unparalleled strong foundation from which to run fraud reporting and analysis.

The initial award is for a 24 months period with an option to extend beyond.


The security standards you need to consider when handling sensitive data

At OCC we have been building and hosting software that deals with sensitive public sector data for over a decade. But where do you start if you are embarking on a project/business that has sensitive data at its heart?


ISO 27001

The basic standard you need to look at for a company in this sector is ISO 27001:2013. You can purchase a copy of this standard online. If you go along this path the one thing I will say is that it’s easy to misunderstand what its purpose is and hence go overboard; do more than you need to. It is a standard that outlines a number of things you have to think about, it’s not a list of what you have to do. It is perfectly acceptable to say that you have risk assessed something and decided it’s not worth it, providing you can justify it. The standard is designed to make it clear to others what it is you do and to have assurance (via audits) that you have thought about it and that you do what you say.

As a start up a number of things can be radically simplified and make it more complex as you grow. You can get consultants to help you with this but it is feasible to do it yourself. To get this you need to create a series of policies and practices, a Statement of Applicability (which relates the standard back to the policies) and get audited by a registered body, we use SGS but there are many others.

N3 & PSN

If you want to connect to systems within the NHS network, there is also the N3 connectivity. This will be important if you need to support your system running inside the N3 network or if you need a connection to it.

For N3 I only know the basics: that is you complete a code of connection, which is a bit like ISO 27001 only more thorough and prescriptive, i.e. there are things you have to have, like two factor authentication and a certain grade of security devices.

For non-health organisations there is also the PSN or Public Services Network for which the same principles apply but just not quite as strict. This is the one we are currently working on for OCC, and we hope to go live with by the end of the year.

Penetration Testing

The other thing you cannot do without in the current climate, is to get your system externally penetration tested by a company that is CHECK or CREST registered and ask for a statement of opinion you can show (potential) customers. This seems pretty much expected as standard among our customers now.

These companies can cost anywhere between £800-1200 per day and we usually have to buy about 3 or 4 days for a system. How often you do this will depend on how often you change the system.

Where to find help

It is a bit of a mountain to climb but it’s not as bad once you get into it; take it in chunks and keep it in perspective (you will find purists that will easily go over the top). There are a number of consultants that can help in this area – many auditors have a side business of their own that they to sell consultancy and training services and the good ones will tell you what you don’t need as well as what you do. My auditors have nearly all given pointers during the audits as well.

In addition there are hosting companies that have experience connecting to PSN/N3 networks and systems that hold sensitive data. They are able to provide you with advice and, if handled correctly, can be useful free resource if there is a contract in it for them. Just keep an eye on the costs of the eventual contract you have with them.

We’ve covered the standards most relevant to the public sector, but of course there may be others related to your industry. The PCI DSS (Payment Card Industry Data Security Standard) is an obvious example, for those handling payment information. You should seek advice on other standards that apply to your business.

The other thing to remember is the landscape in this area continually changes, the levels of security and the type of things that were acceptable when we first started hosting systems are no longer acceptable and we’ve had to adapt and migrate to stay on top of the changes. So you are never truly done!

ContrOCC Hackday V – Part 2

Carrying on from our first post following the results of our developers’ adventures in the most recent ContrOCC hackday, here is the final set of projects:

Julian – Client Provisions

As anybody who has ever peeked under the bonnet of the Charging Engine will know, charges are based on Client Provisions. These are entirely derived from Care Package Line Items and there is a many to many relationship between the two. However, at many Local Authorities, Client Provisions have a one to one relationship with its corresponding Care Package Line Item in about 90% of cases.

To see whether there are any fundamental problems with imposing “1 CPLI = 1 CP = 1 CPLI”, I doctored code to only make one CP per CPLI and then ran the standard charging tests. This resulted in some errors being reported; those where the totals still agreed but were made up of different individual charges (i.e. correct) and those where the totals were different (incorrect). These will be reviewed soon.

Maciej – Alternative storage

The aim for my Hack Tuesday was to investigate alternative options for database storage especially for bulky but simple stuff like documents or audit data to improve some support related activities, like:

  • Backup and restore performance
  • Backup download time for purposes of investigation

The following options have been considered:

  • Split databases (i.e. a master database and auxiliary database for the purpose of blobs/audit storage)
  • Multiple file groups
  • FILESTREAM option

It turned out that none of the above really matches our needs. Having separated databases possess a risk of internal consistency loss; multiple file groups are really tempting (in fact these are being exploited by some customers already), but these suit slightly different purposes; the FILESTREAM option may improve performance of some blob-related objects queries, but it does not provide any additional benefit in terms of backup/restore activities.

I have found some clues pointing out that we could exploit the read-only file groups to store audit data (or, to be exact, to move the audit data from the current to the archive table on some regular basis), which may allow for reduced locking, and some improved performance and faster backups (as there is no need to backup read-only file groups frequently; although that would require a different backup strategy).

Matthew – WiX

This time I ported one of our installers from Windows Installer Projects (which is deprecated) to Windows Installer XML (WiX). I chose the ‘Product Package Installer’ installer.
WiX is MSBuild based and declarative, requiring you to explicitly list all files that are installed. However, a built in tool (Heat) can be used to auto create fragments that list the files in a directory (it runs as a Pre-Build Target to the WiXProj).

Screenshot of Matthew's WiXProj file

You can optionally apply an XSL to exclude certain files, such a PDBs, that you don’t want installed. This could also be achieved with Pre-Build copies but the XSL transform is much more reliable.

Screenshot of Matthew's XSL

The main ‘Product.wxs’ does the hard work, in the screeenshot below the Component Groups ‘Binaries’ and ‘Tags’ are WiX Fragments that were automatically created by Heat. Unfortunately, in order to have a Start Menu shortcut it is necessary to explicitly include the main executable file rather than letting Heat do the work, this is why ProductPackageInstaller.exe isn’t part of the ‘Binaries’ Component Group and is excluded from that Fragment by the XSL.

Screenshot of Matthew's WXS

Nathan – DB upgrades with F#

I set about replacing the ‘What is the Upgrade Path’ wiki page, which describes how to upgrade a ContrOCC database from one version to another, with a standalone console application. The console application is written in F#: partly because it’s a good language for rapid prototyping and partly because I’m keen to gain more experience with the language – but primarily because I’d already collated the required structured branch / release / merge information as part of my ongoing development of the stress testing tool (also written in F#). The implementation is nearly complete: just some loose ends to tidy up, plus I’d like to add some unit tests.

Screenshot of Nathan's db upgrade console app

Nigel – Generating test data

I attempted to create an application that would allow for TPA (our internal testing language) to be created just by ticking the types of data that you want in your database. The idea being that different combinations of data could be created without needing to hand code any TPA. I built a framework that sticks TPA together and only references TPA data if it exists (i.e. For Organisations the Organisation Ownership field is only populated if the Organisation Ownerships have been created.)

I also included a base date with a view to allowing the TPA to be generated for different dates but I did not find time to implement this feature. I did implement the feature of allowing a user to choose what “invalid” characters should be included in the test data. (i.e. for a CSV export file you want to include commas in your test data to make sure that the export can handle commas.) With the framework in place more TPA “bricks” can be added to the “TPA bricks” folder to extend what test data can be generated. As you can see from the screenshot below I have only implemented 5 test data items.

Screenshot of Nigel's test settings app

Steph – New documentation

Not having a particularly good long term memory, I have a habit of writing down any point of interest on completion of a task in my lab book. Notes can vary from where a particular feature lives the graphical user interface and how to use it, to schema diagrams or more specific development related reminders. As time has gone on, the earlier notes seem very obvious and I rarely refer to them personally, but on more than one occasion they have proved useful to a colleague and have been borrowed and even photocopied.

Since a lack of documentation is a common complaint amongst newer members of the team, I used hack day to start writing them up in a digital format which can be indexed and searched. This will be made available in a central location and hopefully added to over time.

Tom G – Automating deployment

I looked into trying to take some of the pain out of setting up the Remote Module for testing: with mixed results.

I ended up spending most of the day automating the tedious process of manually setting up the folders and scripts required for the module, which differs depending on whether you are doing an install or upgrade, as well as the version of ContrOCC you are testing. I ended up with a working C# command line application that fully automates this section of the process, albeit with a couple of limitations.

I also spent some time trying to programmatically hack IIS settings, but didn’t get too far. This is probably not worth the effort anyway as the changes we make in IIS for the Remote Synchronisation Service are quick and trivial. I was going to try modifying web.config files in C#, but I didn’t get around to this and again this is probably not worth the effort considering how minor the changes we make are.

Tom L – Automating component testing

I spent my hackday firstly trying to get my “C# component test runner” up and running on the latest branch. This tool is intended to build our C# solutions (all of them, for test purposes) and run our Visual Studio tests. Ideally once working this would be kicked off nightly. After banging my head against annoying MSBuild-from-the-command-line issues in the morning I made progress on actually running some tests in the afternoon, finally resulting in the ImportExport tests being run.

I have yet to see whether the results I’ve got from running the tests in debug mode against a WIP database bear any relation to the official results being seen by people running the tests the standard manual way.

Tomasz A – Web-based CSV editor

I tried recreating our reference data csv editor tool as web application to see if it’s feasible at all. I used MVC 5 + WebApi 2.0 + AngularJS + Bootstrap to create a single page application that could fetch, edit and update our reference information. Adding additional filtering to ease the navigation proved to be easy and effective though I ran out of time to implement actual updating of data in the database so at this point it’s just a fancy data viewer. With the potential to style quickly it could become quite useful and suit developer personal preferences. With the filtering it’s also much simpler to add/edit data required.

Screenshot of Tomasz A's CSV editor web app

Trevor – Parsing & Analysing T-SQL

I wanted to look at ways of parsing T-SQL. Third party tools for T-SQL seemed pretty ropey. The general consensus on the message boards was to use the C# ScriptDom (Microsoft.SqlServer.TransactSql.ScriptDom).

Having established this as the correct approach, I started writing a simple win32 app that allows a SQL script to be specified, and the SQL parser is then used to inspect the script and report errors. The parser uses the “Visitor” code pattern to collect SQL fragments which correspond to scenarios that need to be examined. These fragments can then be broken down by casting them to specific DOM object types, providing helpful properties which prevent the need to write complex regex code to isolate SQL elements.

However the behaviour of the parser is complex, reflecting the complexity of T-SQL syntax (especially when inspecting JOIN fragments). While I managed to code a few scenarios (e.g. missing collate statements on temporary table definitions and missing decimal column precision declarations), I ran out of time when looking at some of the more complex scenarios.

My conclusion is that the functionality does show promise for writing more elegant tooling, however it requires significantly more time than hack day allows to fully explore this. I will therefore keep this project on hold, and hopefully revisit this on the next hack day.

How to write a 5 year plan (and why it doesn’t matter if no one follows it)

The Lead Developer LogoTom Litt & I will be attending The Lead Developer conference in September – it’s a new conference with a great line up of speakers covering new and disruptive technologies (of course), tools, methodologies, and, because it is aimed at Leads, also managing teams, motivation and leadership.

To warm up I’ve written an article for the conference blog: How to write a 5 year plan (and why it doesn’t matter if no one follows it).

ContrOCC Hackday V – Part 1

Our ContrOCC hackdays give our developers a day to work on tweaks, gripes, improvements, or whole new features of their choosing and then sharing those with the rest of the team.

For all the thinking behind our product hackdays, have a look at the intro to the first ContrOCC hackday.

We have plenty of projects to talk about again this year so I have split this post in two; we’ll post the remaining projects soon. Here is the first set:

Adam and Tomasz B – Code analysis

We did some research on Visual Studio’s Code Analysis tool. The tool automatically checks all code for compliance with a customizable set of predefined rules. Those rules check for common programming mistakes, adherence to good design practices, etc. There’s a wide selection of Microsoft recommended rules already built into Visual Studio, which can be cherry-picked to form custom rulesets.

We focused our work on developing a few custom rules that would enforce some of the ContrOCC specific conventions, such as using correct prefixes for variable names or making sure all forms inherit from the right base form. We’ve been able to develop a few sample rules and run them against the ContrOCC desktop client code as a proof of concept.

Alan – Database schema documentation via metadata

I put together a small demonstration of adding extended properties to database schema objects (specifically, schema, tables or columns) so that we can add documentation to them which can then be viewed in Management Studio.  This is something I’d done in a previous hack day, but this time I put together ContrOCC-style infrastructure to support it, giving a rough idea of how I’d expected it to be done if we wanted to use this for real.

I also mocked up a representation of how the data in our system reference tables could be moved into the metadata file, rather than its current location in CSV files. This would mean we could ditch the CSV files for good, thus having one fewer place to find data. It would also mean not having to maintain the CSV column headers to keep them in line with schema changes, and the metadata representation of each table’s schema would be visible in the same file as the data.

Hopefully the data would also diff more easily in this form, and would be more human-readable when viewing diffs or editing files manually.  If we were to implement this, the Data Maintenance Tool would be updated to read from and write to the metadata file rather than the CSVs – so, editing the data would be no different from before, but behind the scenes it would be stored in a more sensible way.

Screenshot of Alan's work

Chris G – Upgrade AllTheThings to .NET 4.5

Upgrade AllTheThings™ to .NET 4.5. Then try to find a cool, user facing, reason to keep it.

The first part of my task seemed trivial, the latter less so. Microsoft has fallen out of love with WinForms in favour of WPF, so there was little in the way of eye candy to entice users to higher versions of .NET (unless you count EnableWindowsFormsHighDpiAutoResizing in .NET 4.5.2). So I turned my attention to making the UI a bit more “U” friendly.

My efforts concentrated on my annoyance with having to support screen resolutions you can draw with a crayon. I looked into screens for creating new entities flowing through (a bit like a wizard) so we don’t need to present so many controls on one form. I also made the navigation bar collapsible to free up some real estate, this even works with the new hot-keys for keyboard navigation.

Screenshot of Chris' work

Chris H – XML import/export definitions

Our current representation of imports & exports uses a system of four interlinked system reference (TRefSys) tables. This means that the reference data defining any single export is scattered across four CSV files and has to be tied together with IDs. I’m interested in replacing this with a single XML file defining each import or export. This would have the following advantages:

  • All data defining a single export is in one file so easier to comprehend
  • XML is more human readable than CSV
  • No need to manually type lots of IDs for sub items

For hack day, I focused on:

  • Generation of XML from our existing TRefSys tables using SQL Server’s surprisingly helpful SELECT FOR XML feature. This would be used in the one-off transition to the new functionality.
  • Writing temporary table functions which exactly reproduced the original contents of the TRefSys tables (including for now the IDs) this then enabled existing code to be updated to use the XML via these functions with a simple search & replace operation.
  • Conversion of one actuals import to XML format

This was a success, and I was able to get the relevant import test to pass with the TRefSys definition of the same import deleted. The next step (which I didn’t have time for) would be to eliminate the unwanted Database IDs from the format and rewrite the SQL which depends on them.

Chris P – Loading lists asynchronously

For hack day I have produced a working prototype which performs asynchronous population of lists which should reduce delays users experience with populating lists as while the overall time is unchanged the application becomes responsive quicker.  This also means if filter conditions are entered which lead to a long delay then the filter conditions can be immediately modified.

Changes made:

  • Return to UI immediately list population started
  • While SQL running:
    • Show “Loading…” in list  (instead of normal SQL wait dialog)
    • Disable list so no editing allowed
  • Populate list on completion of SQL
  • Kill executing SQL if no longer required:
    • Move to different content type (i.e. control disposed)
    • Filter conditions change (i.e. another call to mFillList)

Other ideas:

  • It might make sense to modify lazy loading so in background start loading one list at a time anyway.  Need to ensure loading for displayed lists are shown immediately
  • We might want to limit maximum number of SQL commands which can execute at a time: probably only really an issue for accounts screen where we have a lot of tabs

Damian and Pawel – Visualise database schema metadata

Aim: To help maintain metadata files when creating/editing new database tables, columns etc.

What we did:

  • Visualise ContrOCC Metadata.xml file in more readable form (TreeView)
  • Edit existing attributes (most of them)
  • Add new attributes
  • Add new instances of the nodes

Screenshot of Damian & Pawel's work

Near the end we realized that the problem is harder and more complex than we thought. There is still a lot of work to do, but it was a fun challenge!

Ian L – CSV column alignment tool

I chose to create a file pre-processor that formats a CSV file such that all the columns line up and a post-processor which removes all the padding added by the pre-processor.

These file processors could then be used by AraxisMerge, the file difference tool I use, to help make comparing our table CSV files from the Data directory easier to compare.

As an extra challenge, I used PowerShell to write the file processors.

Learning how to use PowerShell took more time that I expected and the Cmd Applets that I thought existed did not, which means I will have to write more script than I anticipated.

Jon – Generating database test scripts from UI interactions

My task was to explore providing a user-friendly UI for generating database test scripts from the ContrOCC UI. I spent most of the day getting to grips with the ContrOCC UI Framework which I’ve never had to do any serious work with before, so it was a great training day for me though not very productive.

I reached the point of being able to open a form from the Troubleshooting menu which contained lists of clients and cplis with tick-boxes for selection. Unfortunately before I committed Visual Studio crashed and scribbled all over my main .cs file so I have no screenshot or code. However, now I know what I’m doing it should be fairly quick to repair if I ever return to this.

Adding a text size widget to your site using CSS and Sass

A requirement we hear from many of our Government customers is that a sizable number of their users with sight impairment prefer to have a text size widget on-screen when they browse a website.

These accessibility widgets are tough to implement cleanly using HTML and CSS but the advent of CSS preprocessors such as Sass and LESS make the job much easier. In this post we’ll see how we can use Sass to create a text size widget.

What we’re aiming for is the standard row of “A” characters increasing in size to denote changing the text size on the page, like this: Screenshot showing three A characters of increasing size. – you can take a look at one in action on one of our sites.

What we will aim to do is add a class (small/medium/large) to the body element of the page, which we can then refer to in CSS to say: when the body has a class of small then the font-size should be x; when medium then y; large then z.

There are two elements to making the widget work:

  1. JavaScript that adds the widget to the page, records the setting in a cookie, and adds a class to the page body so that our CSS can react to it.
  2. CSS that sets the font-size based on the body class. A Sass mixin makes this much more manageable.


Here’s the code, and we’ll go through it bit-by-bit:

$(document).ready(function () {
    // Add in the widget
    $('#text-size-widget').append('<ul><li class="small"><a href="#" class="selected">A</a></li><li class="medium"><a href="#">A</a></li><li class="large"><a href="#">A</a></li></ul>');
    // Read the current text-size from a cookie
    if ($.cookie('TEXT_SIZE')) {
        var cookieTextSize = $.cookie('TEXT_SIZE');
        $('#text-size-widget a').removeClass('selected');
        $('#text-size-widget li.' + cookieTextSize + ' a').addClass('selected');
    // Add the resize event handler to the widget's links
    $('#text-size-widget a').click(function () {
        var $el = $(this);
        var textSize = $el.parent().attr('class');
            .removeClass('small medium large')
        $('#text-size-widget a').removeClass('selected');
        $.cookie('TEXT_SIZE', textSize, { path: '/', expires: 10000 });
        return false;

We’re using jQuery here so the first line sets this script to run when the page is loaded and ready.

We then add in the HTML for the widget, by inserting a list of links into any element on the page with an id of text-size-widget, so you’ll need to have at least one of those in your page. We insert this using JavaScript so that if a user doesn’t have JavaScript enabled they won’t find a non-functioning list of links on their page.

We then check to see if the user has a cookie called TEXT_SIZE, if they do we read it, add the appropriate class to our body element, and refresh which of our “A” links is selected. This makes sure that the user’s choice carries over from page to page on our site.

Finally we set what happens when the user clicks one of the “A” links:

  • We detect which link they clicked and store the small/medium/large class in a variable to use later
  • We remove whatever text size class might be on the body currently and set the new one
  • We update which of the “A” links is selected
  • We store the new text size in the TEXT_SIZE cookie
  • We return false so that the link does not perform its standard functionality (in this case adding “#” to the URL)

That’s the widget, cookie and body classes dealt with, now to tell the CSS how to handle it.


What we’re looking to end up with is along the lines of:

p { font-size: 10px; }
body.medium p { font-size: 12px; }
body.large p { font-size: 14px; }

Clearly it would be massively frustrating to have to write all three options out every time you wanted to set the font-size of something – this is where our Sass mixin comes in.

Mixins let you extract repetitive code out of your CSS into something that looks a bit like a function (you can even pass in values/settings) and call from your CSS whenever you need it.

Here’s what our mixin looks like:

@mixin font-size($baseFontSize, $mediumMultiplier: 1.2, $largeMultiplier: 1.4) {
    font-size: $baseFontSize + px;
    body.medium &    { font-size: ($baseFontSize * $mediumMultiplier) + px; }
    body.large &    { font-size: ($baseFontSize * $largeMultiplier) + px; }

The mixin is called font-size and you pass in the “base” font size (i.e. the size to use for the smallest setting, which is the default), and optionally you can set the multipliers we use to go up to the medium and large sizes.

We’ve then got a bit of Sass code that generates our CSS using the base font size and the multipliers.

Of course if you prefer to use ems or another unit you can replace the px with whatever you want.

To use this mixin all we need to write is:

p {
    @include font-size(10);

And when we compile the Sass it will generate exactly what we want. So all we need to do is replace any normal font-size declarations with a call to our new mixin and we’re away!

Compiling Sass with Gulp in Visual Studio

We love visual studio at OCC. It’s an incredible IDE for software and web development and Microsoft put amazing effort into keeping up with the direction developers find themselves moving.

In this case we’re talking front end web development and specifically:

  • Gulp.js – a JavaScript-powered automated build system that uses Node to perform the tasks you find yourself doing over and over
  • Sass – a language extending CSS (that is compiled to produce CSS) to give developers more power when creating stylesheets

For quite a while Visual Studio has supported Sass, specifically through the Web Essentials plugin, which has offered intellisense, CSS preview and compilation. But much of this functionality is being removed from the plugin because the broader developer community is moving to using build tools like Gulp (and it’s peers, such as Grunt) and Web Essentials wants us to use them too.

But how do we go about doing that?

Setting up Gulp in Visual Studio 2013


The first step is to install Node.js, the JavaScript server that Gulp uses to run its tasks. Importantly, you’ll need v0.10.28 (there’s an x64 folder in there, which most of us will want) as the more recent versions are not yet supported by the Gulp-Sass compiler plugin. It looks like this will be fixed very soon though, which is great!

For Sass compilation we’ll need to install two Node packages, so open a command prompt and run the following npm (Node Package Manager) commands:

  • npm install gulp -g
  • npm install gulp-sass -g

The -g is a flag to install globally, so you can use the modules anywhere.

Now we can set up Gulp for your solution. Create a file called package.json in the root of your solution and add in the following information:

    "name": "Project Name",
    "version": "1.0.0",
    "description": "Project Description",

    "devDependencies": {
        "gulp": "3.8.11",
        "gulp-sass": "1.3.2"

You’ll need to make sure those gulp and gulp-sass version numbers match those you have installed.

This will allow any other developer to come along and run npm install on this directory and Node will download and install everything they need!

Give Gulp some tasks

Create a gulpfile.js file in your solution and add the following JavaScript to get started:

var gulp = require('gulp');
var sass = require('gulp-sass');

gulp.task('sass-compile', function () {

gulp.task('watch-sass', function () {
    gulp.watch('./Content/*.scss', ['sass-compile']);

Here we have three sections

  1. First we create gulp and sass variables and use Require to ensure they’re good to go.
  2. Then we create a Gulp task called ‘sass-compile’ and tell to it find our Sass stylesheet, compile it and output the result to our directory.
  3. Finally we create a ‘watch-sass’ task that watches all Sass files in our folder and runs the sass-compile task whenever anything changes.

Now, if we ran gulp sass-compile at a command prompt in our project, it would compile our Sass. If we ran gulp watch-sass it would start watching. But running commands all the time isn’t much fun…

Task Runner Explorer

The answer is to install the Task Runner Explorer Visual Studio Extension, which will give us some nice UI for running and automating Gulp tasks from within Visual Studio.

Once you’ve installed (and restarted VS) when you right-click on your gulpfile you’ll see a new ‘Task Runner Explorer’ option, which will open up a panel showing a list of your Gulp tasks on the left and some binding options (or the output of whatever has been run) on the right.

Task Runner screenshot

If you double-click on sass-compile it will run and compile the Sass. If you double click on sass-watch it will start watching. Much better!

Task Runner execution results screenshot

The icing on the cake is that you can right-click on one of your tasks and bind it to one of the following Visual Studio events:

  • Before Build
  • After Build
  • Clean
  • Solution Open

Right-click on your sass-watch task and bind it to Solution Open. Now, every time you open your solution, Gulp will automatically start watching your Sass files and compiling them whenever it needs to – perfect!

What else can Gulp do?

Gulp doesn’t just do Sass compiling, it’s hugely powerful. Here are just a few of the tasks Gulp can help you with:

  • Minify your CSS, JavaScript, HTML or images
  • Combine your CSS files
  • Generate CSS sprites
  • Generate favicons
  • Rename files
  • Check if files have changed or not
  • Output messages/errors
  • Add CSS vendor prefixes
  • Check code quality with hinting

More good news as Visual Studio 2015 will come with first-class support for Gulp as standard, so these tools will be built right in to the IDE.

Search engine rankings for Social Care: the 001 Taxi Problem

If you’re more than 20 years old you will remember the annual delivery of your local telephone directory; the thud as it lands heavily in your hallway, often with a crumpled cover, a testament to the efforts of the delivery boy to fit the tightly published pages through your letter box. For me, this directory was my first exposure to search engine rankings, with its wonderfully named 001 Aardvark Taxis vying angrily with 001 Ace taxis for first place in the listings.

Pigot’s directories, published in 1814, were the first widely available English reference source for everything from tradesman to public houses. After Bell’s invention of the telephone in 1876, directories became common, first to the secretaries of the nobility and then to the middle classes.

Coming First

The 001 Taxi problem is as alive today as it was in phone directories of my youth. Today it manifests itself in the ‘war’ of search engine rankings – a continuous cut and thrust between businesses seeking to steal a march on their competitors whilst the search engine developers try to outwit the tricks of web site developers. For some businesses, failing to appear on the first page of Google’s search results means anonymity and business failure; paying for adwords is expensive.

I became acutely aware of this problem in our recent development of Social Care information and advice portals. These portals, legal cornerstone of the Government’s 2014 Care Act, require local authorities to establish and maintain an information and advice service in their area. At OCC, we have sought to develop a smart ranking system that works on keyword relevance (i.e. the prominence of the word in the service profile). However, service providers may list keywords several times in their description to optimise their profile which we don’t want. We wish to rank services by relevance so if the keyword (or thesaurus equivalent) is found in the description once it should get the same relevance points as finding it 10 times. Google’s famous Page Rank algorithm is not available to us in the Social Care space, so we need to find another way to help our users find the highest quality information for their search.

To display our results we have introduced a secondary search ranking facet based on service profile quality score. The idea is that we establish a scoring mechanism for service profiles similar to the relevance score; points are awarded for logo, accreditation, quality of description, images, contact details, etc.

However, this still leaves a very serious problem. Serious for service users, providers and Local Authorities, namely that if you type in the same keywords you will get exactly the same search results in the same order every time. This is a reality – most people will use the same words to search for services: words like ‘Home Care’ or ‘Meals on Wheels’. So the same providers will always come first in your list; the same providers last. This is bad for creating a diverse and competitive market and bad for citizen choice. The reality is that well over 20 appropriate residential care providers may all have scored somewhere between 98% and 100% in our ranking algorithms. The small percentage differences are vital for the order of results but we surmise that they are trivial in terms of the actual suitability of the service for the citizen.


One solution – randomization. We could take results that have scored almost the same (and we can define what ‘almost’ means in percentage terms) and then randomize the results. All those providers that have scored ‘almost the same’ are presented in a different order each time you search – the actual order being determined by a randomization algorithm.

This simple technique helps to create a diverse market; shares business across a range of suitable providers and helps ensure a vibrant and competitive market. There is no incentive for Providers to give themselves silly names; no more 001 Aardvark Care Homes. However, we’re worried about the impact on usability – how many search engines do you use where the results come out different each time for the same search? If you send your search link to a friend and tell them to look at result #2, they will see a different service in that position.

If we’re going to use randomization to help create a vibrant Social Care market we’re going to have to make sure everybody understands what we’re doing. Thoughts on a spare page of an old Yellow Pages please.

Asynchronous processing in SQL with Service Broker

SQL Server Service Broker was introduced in Microsoft SQL Server 2005 and this article suggests several uses, the most significant of which is probably the ability to do asynchronous processing. Service Broker might be seen as the database equivalent of Microsoft Message Queuing.

In this scenario, a synchronous process can put a message on a queue and then complete immediately (i.e. return control to the user). A separate process can then take this message off the queue and perform some processing based on it – this processing is effectively done “behind the scenes” as far as the user is concerned. Typically, it might be some relatively time-consuming work which isn’t on the user’s critical path.

As with all asynchronous processes, the problem of how to report errors is a tricky one, given that the user might have gone home for the evening by the time they arise. You might imagine keeping a log of such errors and generating notifications, all of which could also be done from SQL, using Database Mail for example.

Getting started with Service Broker

Service Broker’s implementation may seem unnecessarily complicated, involving not only queues but also things called services, contracts and message types. These have been introduced to allow the functionality to be used in as many weird and wonderful ways as realistically possible but they do present a bit of a steep learning curve. So here’s a very simple set of scripts to get you going and show how it works.

Begin by running the first script from any existing database on a SQL Server, version 2005 or later. It creates a new database called ServiceBrokerDemo and makes in it a Queue, Service, Conversation, Message Type and a couple of example stored procedures.

Once that has completed, run the second script:

This puts some messages on the queue by calling the Proc_Queue_Send_Message stored procedure (which completes quickly). It then calls the Proc_Queue_Process_Messages stored procedure; this reads messages off the queue and processes them, putting the results in the T_QueueProcessingResult table. Proc_Queue_Process_Messages mimics doing “hard work” via a WaitFor Delay '00:00:05‘ statement.

An alternative way of working is implemented in the third script, which should then be run:

Now, Proc_Queue_Process_Messages will be called automatically whenever a message is put on the queue – i.e. there’s no need to explicitly call it.

Finally, run the fourth script:

This will again put some messages on the queue. Once it has completed (quickly) the processing will have been started and can be monitored by re-running the final Select * From dbo.T_QueueProcessingResult With (NoLock) statement. Every ~5s, a new row should appear (5 in total).

An aside about Conversations

As in real life, striking up a “conversation” can be a time-consuming process so these are generally cached outside of the Service Broker framework. The TSys_ConversationHandle table is used to do this.

ContrOCC Hackday IV – Part 2

Carrying on from our first post following the results of our developers’ adventures in the most recent ContrOCC hackday, here is the final set of projects:

Julian Fletcher – Graphically presenting performance information to the lay developer

We are now amassing ever greater amounts of anonymous performance information from live Local Authority (LA) databases:

  • execution times
  • deadlocks
  • missing index hints.

This information is transferred, via CSV files, from all the LA databases to a shared OCC server. Here, several Stored Procedures may be used to analyse the figures. However, these output slightly impenetrable tables, so I set about presenting the information in a graphical format using SQL Server Reporting Services (SSRS).

Setting this up was fairly tedious – in particular, getting all the necessary permissions (even when everything was on my own machine). However, in summary, this is how it works:

  1. You set up a SQL Server to have SSRS. This results in a couple of new databases being created:
    Screenshot of Julian's hackday project
  2. Configure SSRS using the Configuration Manager:
    Screenshot of Julian's hackday project
  3. Install the Microsoft SQL Server Report Builder application and design your reports. These can obtain their data from a database on an entirely different server.
  4. Save them to your hard drive and then publish them the Report Server when they’re ready.
  5. The screenshot shows what this looks like. You can get the averages at all LAs for any one stored procedure:
    Screenshot of Julian's hackday project
  6. You can also drill through to get a by-version breakdown for a particular LA (not down – this is a slightly different concept):
    Screenshot of Julian's hackday project
  7. Just by chance, I’ve chosen a stored procedure which appears to have sped up in later versions!

As I mentioned above, getting the right permissions was a pain and I still haven’t made the Report Server available to other users but I hope to be able to soon.

Maciej Luszczynski – Cleaning up the developer documentation

I focused on providing How To documentation for new developers on the wiki. I started by cleaning up existing pages. The Home page is now greatly improved, although the Windows Topics section still requires clean-up.

I have also identified some topics that it might be useful to add to the wiki, such as:

  • how to remove a database object
  • how to remove a database table properly
  • how to commit a change to the SVN properly
  • how to check the imports flow

Matthew Clarke – CSV Merger

This tool improves merging of CSV data files which often cause SVN conflicts, for example when a column is added.

The tool can be set to be your default merge tool for CSVs in SVN.

Screenshot of Matthew's hackday project

When you would normally resolve conflicts CSVMerger now opens.

Screenshot of Matthew's hackday project

It will highlight conflicts and missing values for new columns:

Screenshot of Matthew's hackday project Screenshot of Matthew's hackday project

You can even resolve conflicts via the UI (needs polish).

Screenshot of Matthew's hackday project

So, you’ll be able to go home 5 hours early because the merge was done in no time.

Nathan-Madonna Byers – F#/C#

I looked at using various F# Type Providers (from the open-source FSharp.Data.SqlClient package) to interact with ContrOCC databases in a type-safe manner, such that many errors can be caught at compile-time rather than raising run-time exceptions. In particular, I investigated the potential of this approach for writing test data scripts and worked through the implications of exposing this functionality to C# code.

The first thing to note is that there’s little in my prototype that couldn’t have been achieved solely in C#. However, I find F# to be a more natural and expressive language for this kind of thing. In terms of F#/C# interoperability, there were a few head-scratching issues that took a while to figure out – but it all seems to work pretty smoothly. (Famous last words.)

A quick summary of the specific Type Providers:

  • SqlProgrammabilityProvider: This can be used when working with stored procedures. Unfortunately, it doesn’t currently support SPs with output parameters or explicit return values, which rules out many of our SPs.
  • SqlCommandProvider: This can be used when working with arbitrary SQL commands. I’ve used this in my prototype as a lightweight wrapper around SPs (e.g. rather than calling an _Insert SP directly, I can have a simple three-line SQL command: declare an @ID parameter; execute the SP [passing @ID as the output parameter]; select the @ID).
  • SqlEnumProvider: This can be used to create enumerations for (relatively) static lookup data, e.g. from TRefSys_ tables. This means that it may no longer be necessary to define these enumerations manually in our C# code (and then have to worry about keeping them synchronized with the database &c.).

In terms of extending the prototype, various factors – the use of the SqlCommandProvider and the succinctness of F# (plus type inference and auto-generics) – mean that the amount of hand-written “boilerplate” code required is fairly small. (It’s not dissimilar to the amount of hand-written SQL needed to do it the old way but should be quicker to write.) The thing that really takes time is the need to analyze the SPs in relation to the database schema, e.g. to decide: which parameters to expose to C#; which should be mandatory, etc.

Finally, a summary of the key benefits:

  • Static typing (and Type Provider “magic”) mean that many errors can be caught at compile-time.
  • Intellisense (including visibility of defaults for optional parameters) and auto-completion.
  • Optional parameters are truly optional: no more “counting commas”!
  • The scripts are (more) readable.
  • Debugging support and full stack traces for exceptions.
  • “Production” SPs – i.e. those used by the UI (and other components) – are always used.
  • Modularization, i.e. we can finally share portions of data setup across tests.

Nigel Palmer – ContrOCC version manager

I set out to create a version number manager to assist with the building of ContrOCC. Version numbers are currently set manually in a variety of different places and there is some risk of one of them being missed. This tool will allow the developer responsible for the build to set all version numbers in the SQL database and the C# code. This will improve the accuracy of the build process and prevent test failures due to incorrect version numbers.

The ContrOCC Version Manager tool has a simple user interface that allows the user to set the version number. This then updates all the required build files to have the new version number.

Screenshot of Nigel's hackday project

Patrick Donkor – An executable imports/exports specification

I spent my Hack Day looking at how I could use Specflow as a way of creating an executable specification for the development of new Imports/Exports.

The idea is take a specification using the Gherkin syntax to specify a top level view of an import/export. This specification would be agreed upon between an IM and the Client. The specification would be passed to the developer who would code and test against it.

To see if this could be done, I used an existing client payments import as my template. I spent the first part of the day setting up the specification for the import. This went well. The second half of the day was spent trying to implement automated tests to use against the specification. I did encounter a few minor issues that, given time, could be rectified. The screenshot below shows what a sample file looks like.

Screenshot of Patrick's hackday project

Steph Sharp – Improving code integrity checks

I spent the day trying to suppress output from one of our internal code integrity checks (sys_checkschema). This output is an unwanted side effect, and can be confusing, especially to newer members of the team. Although I did manage to prevent this output being displayed, I was unsuccessful as there were problems with expected errors no longer being reported and the addition of spurious new errors, which I was unable to resolve.

Tom Litt – Migration from within the ContrOCC UI

I took the brief “make the migration process at new customer sites a part of the ContrOCC UI” and tried to see whether it would be feasible to run migrations via a wizard available in the main desktop client, instead of the existing process where there is a separate tool. Why? If a customer had this, they wouldn’t have to send data off-site. They could re-run migration tables as often as they liked. Selling this to us, we could ditch the SQL Server Integration Services packages which are a merge nightmare.

In production the wizard, would:

  • allow editing of migration settings
  • allow selection of files to be migrated in, or perhaps just point at a directory full of migration CSVs
  • run through each file independently, or run them all in an unattended batch; depending on the level of reporting required
  • feedback on the level of success/failure for each file.

The file specifications and import code are already written, so this is a case of wrapping the code as stored procedures and putting UI on it.

In practice I got as far as establishing a wizard, selecting the client file, importing it and writing the existing HTML feedback to the wizard. This could be extended fairly easily to handle all files and report more nicely, and I think this is worthy of future effort.

Tomasz Agurciewicz – Taming our test scripts

ContrOCC’s testing meta language takes some time for developers to learn. That’s why I decided to finish/enhance a tool I made some time ago which makes it easier:

Screenshot of Tomasz's hackday project

The tool parses a test script and tries to present its contents in a sane fashion. All commands are listed in the Command dropdown, and selecting one of them lists available actions. So for example Command “S” (Service) has actions “Add” and “Load”.

Next, for the selected action it finds the appropriate sql part in the script and tries to list what each column stands for. It then lets you enter all the required data and form a single test script line that you can use later on when preparing tests or data for ContrOCC. It also prints the selected chunk in a separate tab, so you can look up if in doubt what each parameter does.

Trevor Hingley – SQL Server projects in VS2012

I looked at SQL Server projects in Visual Studio 2012, with a view to seeing how viable it would be as a tool for ContrOCC database development and deployment.

There are immediate issues in that the SQL Server Data Tools (SSDT) installed on the machine must be in line with Visual Studio. In my case creating a project in VS2012 immediately raised an error, as the SSDT were not compatible.

After installing the latest version of SSDT, I found the SQL Server project very easy to use. Database schema can be imported from a database, a selection of scripts or a .dacpac file very easily. Multiple projects can be used to handle separate version schemas, and the tooling makes comparison between projects/schema easy to view and update. Pre- and post-deployment scripts can also easily be added to the project.

Where the viability of the SQL Server project type fails is in deployment. Schemas can easily be deployed to databases which can be connected to, and scripts can be generated to create a new database or upgrade from a known schema to the latest schema. However, there is no versioning capability when generating scripts to be able to produce scripts capable of upgrading from multiple schema versions to the current version. A possible alternative is to generate .dacpac files which have versioning capability. However this was not found to be viable, as a .dacpac generated in VS2012 causes an xml schema error when attempting to import into a 2k8 SQL Server database.