MVVM async progress meter example using the Task Parallel Library

I’ve just posted up on GitHub a quick example WPF application that shows an MVVM-style progress meter dialog, using the .NET 4.5 Task Parallel Library (TPL). Specifically demonstrating:

  • A progress meter using .NET 4 Tasks (not BackgroundWorkers).
  • Async support.
  • Cancelling of Tasks via CancellationTokens.
  • Exception handling back to the original thread.
  • Reporting background progress updates via IProgress<T>.
  • Tasks can either be declared as void, or have a return value.
  • Progress can be reported from multiple tasks chained together.
  • An injectable, stateless IProgressDialogService, that can be injected for unit testing purposes.
  • Using an attached property to hide the close button in the progress dialog’s title bar using Win32 interop.
  • Using an attached property to implement a close window command using MVVM.

screenshot

It’s based off a blog post by Jürgen Bäurle but converted to use MVVM and the TPL.

DDD: Questions around ubiquitous language

What happens in situations where the developers and the business don’t agree on the terminology — the Ubiquitous Language — of a system?

This post was inspired by a blog post by Greg Young, which was in turn inspired by a question on the DDD mailing list. Here are some of my experiences where this has occurred.

Developers propose a more precise name

Sometimes in conversations you will find different users speaking about the same concept, but using different words. Alternatively, they might use a common slang term, or short-hand abbreviation. As part of the vocabulary of your system (ubiquitous language), you should settle on a single name, unabbreviated, that is clear to everyone what it means (including new developers).

Here’s a very simple example of this, a new developer’s thought process when looking at a class in a market risk system:

  • Ticker — okay, probably some sort of ticker symbol. But I know different exchanges and data providers use their own proprietary symbols and notation, so I better not make any assumptions.
  • Bloomberg Ticker — ah. It is a symbol ticker that came from Bloomberg. I know where to look it up, and also that it’s not necessarily referring a stock traded on an exchange (it could be a country for example).

This is an improvement that clarifies the concept using the most formally correct — but maybe less commonly used — terminology.

As well, expanding abbreviations may be supported by programming language conventions — for example, in .NET, abbreviations are usually discouraged in favour of longer identifiers, expressed using full words.

Dev team proposes using a term that has never been used before

Generally, you should not ever need to do this. Introducing your own terminology can be confusing, and if it is a well-established business area, they will already have a well-established vocabulary.

The only situation I can think of, where it might be acceptable for the development team to propose that the business should use a different term for something, is when building out software for a brand new business area, where a firm vocabulary has not yet been established — For example, in a software startup still figuring out their business model.

If you do go this route, you must ensure the new term has to be agreed by everyone (developers, UX people, and the business) before you start adopting it. And keep an eye/ear out — if people still keep using the old term, cut your losses and revert to the old term in your code too.

Product Owner says, “let’s just rename the label on screen”

The third situation where implications may be unclear is when a product owner drops you a quick note, asking you to please change the name of a field in the UI. Technically, this is a quick change, but does this mean the ubiquitous language has changed, we should rename it all through the code as well?

This is a situation I’ve certainly encountered more than once, and more often than not, it comes from a cost/schedule/risk background. Read between the lines — what is the Product Owner considering when he or she makes this request?

  • Renaming a field in the UI: a small that should be quick to implement, without much risk of breaking anything.
  • Renaming a field all the way down to persistent storage (e.g. database tables): this is a much bigger change that will require a lot more time (more $$$), may impact the schedule, and could cause many things to break.

Use your judgement here. Is this really just renaming a field on screen, or has the vocabulary of the business changed?

If you think you can rename all through the code/database quickly, and have good unit/integration test coverage to prove everything still works, you might as well do it. But if you think it will take longer, the most pragmatic thing would be to perform the UI label change as requested, but keep an eye/ear on the users to see whether the ubiquitous language really has changed.

If you don’t pay attention after, you run the risk of your code diverging from reality:

diverging_vocab_technical_debt

Changing terms in APIs and wire protocols

Another question is what to do in the case the term that is changing is part of an API signature or wire protocol, which you can’t change because it will break compatibility with other systems. I don’t think this is such a big concern, to leave the old name here:

  • This is only one place, at the very boundary of your system, with an inconsistent name.
  • APIs are typically well documented, including explanations, background and “gotchas” about every field. (Example here from the Twitter API).
  • In future versions of the API you will have a chance to rename it. Or, if accepting flexible formats like JSON, you can accept both forms (under the robustness principle).

So you want a rollback script?

So you’ve got a new release coming up, including upgrading your production SQL database. You’re writing great upgrade scripts, but now your DBAs are demanding rollback scripts as well. How do you approach this?

Do we really need them?

Rollback scripts are an expensive overhead to development because of the time and effort involved in writing and testing them. There are only two valid reasons you should ever need to provide them:

  1. Your database is very large, and quickly restoring a backup is not practical.
  2. Your have an agreed non-functional requirement to support downgrades.

For everyone else, just restore a backup.

The naive approach (don’t do this)

So we really do need a rollback script. Okay. Here’s one that might be generated by a tool:

-- upgrade.sqlALTER TABLE Customer DROP COLUMN CreditCardNumber;-- rollback.sqlALTER TABLE Customer ADD COLUMN CreditCardNumber VARCHAR(24);

What’s wrong with this? A rollback script is supposed to return you to a previous point in time. This example restores the schema version, but not the data — if you run it you’ll lose all the data in the CreditCardNumber column. It is not a proper rollback!

Rolling back the data

In order for data to be rolled back, it has to be kept somewhere as a backup. We have to detach and reattach it from our schema.

During the upgrade we need to:

  1. Detach any old table or columns from the schema. Rename it to e.g. “_backup_CreditCardNumber”, unlink any foreign keys, and remove any constraints or triggers. But don’t drop it — we need to keep it as a backup in case rollback is required.
  2. Apply new schema changes e.g. create new table and populate it from the backup table.

Your schema should now look something like this. The tables and columns are still present, if we need them, but no longer active.

To roll it back we need to reverse any changes and reattach the objects:

  1. Drop any new schema objects.
  2. Reattach each backup table or column to the schema. Rename it back to its original name, and recreate the old foreign keys, constraints and triggers.

Backup tables can then be dropped at any point after the release, when they are no longer required. This can be done as a DBA task, or during the following release.

Testing your Upgrade/Rollback Scripts

At work we have a build set up in TeamCity that replays all our upgrade and downgrade scripts against a test database every time one is checked in — we are continuously integrating and testing our SQL scripts. It performs the following steps, and fails if any SQL errors are raised:

  1. Create an empty database.
  2. Run all the upgrade scripts, from version 1 up to current.
  3. Run all the downgrade scripts, from current back down to version 1.
  4. Run all the upgrade scripts again.

The second upgrade pass is important because it detects any objects that were missed in a rollback script: if you forgot to drop something, it will fail with an “object already exists” error. This saves a lot of time, and reduces the chance of nasty surprises on release day.

We have a big market data risk analysis database at work and this detach/reattach rollback pattern (with continuous testing) has worked well for us so far. We expect it to continue scaling as we grow, although if inserting/updating hundreds of millions of rows becomes too slow for a single release window, we may need to use streaming to populate new tables/columns offline before switching them over during an outage. But the same attach/detach principles will still apply.

App.net: 12,000 evangelists and counting

App.net is a new alternative to Twitter that’s been getting a lot of coverage lately.

They have taken a unique approach to funding, where instead of depending on advertising revenue, they simply charge each user an up-front $50 annual subscription fee.

The idea behind this is to eliminate any risk (in future) of privacy and platform access being eroded when corporate/advertising goals start to diverge from those of the non-paying users — a problem Twitter has recently appeared to have experienced.

This sounds like a pretty solid idea, and has garnered a lot of support so far. But funding is only one problem facing a new social network, and now they have another big challenge: how to get more users?

Here is where (I think) App.net has executed a real stroke of genius. By requiring a full year’s payment just to join the alpha, every user has a real, quantifiable incentive to evangelise App.net to their friends. With fifty bucks on the line, every user has much more interest in the success of the platform than an indifferent take-it-or-leave-it free service like Google+.

This was a smart move, solving two problems, that could only have been done right on day one. Best of luck to the App.net guys — it will be interesting to see how it all goes over the next few months.

Dogfooding: how to build a great API

It’s common these days for web applications to provide some sort of RESTful API for third party integration, but too many of them are built after the fact, as a lacklustre afterthought — with holes and limitations that makes them useless for all but the most trivial applications. Here’s some common pitfalls I’ve seen (and been partially responsible for!) arising from this style of development:

  • Only a narrow set of features supported: The application may have a hundred different tasks a user can do in the GUI, but only a handful are supported through the API.
  • Impractical for real-world use: APIs that fail to consider things like minimizing the number of requests through and batching and bulk APIs. Just think, how you would implement GMail’s “mark all as read” button if you could only update one conversation at a time?
  • API features lag UI features: For example a new field might be added to the UI, but there is no way to access it in the API until someone requests it.
  • Weird/buggy APIs: Bad APIs don’t get as much developer attention if no one’s using them, making them a natural place for bugs and design quirks to harbour.

Instead of restricting developers to a limited, locked-down set of operations, the goal is to empower developers to be as creative and productive as possible with your API.

A better way to design an API

Don’t try to build the sort of API you think people will want. Flip the problem on its head and eat your own dog food: make a rule that, from now on, all new UI features can only be based on the public API. Abandon your traditional, internal back-end interfaces and start using the public API as an interface for all your front-end applications.

“All Google tools use the API. There is no backdoor. The web UI is built on Google App Engine, for example.”

This quote from High Scalability beautifully summarizes the design of their API. There is no backdoor, no privileged internal interface — Google developers have to use the exact same API as you at home.

Twitter is another great example of an application that dogfoods its own API – click view source on twitter.com and you’ll see, apart from tracking and advertising, it’s mostly AJAX calls to their public developer API – the very same API all the thousands of third party apps use.

Extracting files from an SVN repository backup

Extracting files from an SVN repository backup

Say you’ve got a backup of an old SVN repository. You want to grab a couple of files, but the backup looks like this:

C:myrepo_backup    format    README.txt    conf    dab    db    hooks    locks

You figure you probably need to run some fiddly import commands to restore it into a live SVN server so you can connect and browse it right? Wrong! SVN has a feature that makes this incredibly easy: the client can browse and check out directly from a backed up repository on your own local file system.

All you need to do is use the file:// URL scheme and point to your backup, for example:

svn checkout file:///C:/myrepo_backup myrepo

And yes if you’re using Windows, TortoiseSVN does support this — so you can very easily use Repo-browser to explore any SVN backup. Very handy!

Another argument in favour of TDD

An interesting observation came up at work the other day, while we were discussing developer working habits.

Say you’re adding a new feature to an application, that takes 2-3 days to develop. If you don’t write any unit tests, you have no way of knowing if any of your code works until you’ve finished everything, and you can start the application and see it live and working (or not). You can’t confidently check in until you’ve finished coding the entire feature.

If you’re doing test-driven development, however, you have lots of little checkpoints along the way. This class works, check it in. Red, green, check it in. You don’t need to wait for the whole feature to be finished before committing, because your tests prove that the individual parts work. You can confidently check in your changes much more frequently, significantly decreasing integration risk, and the chances of ugly merge conflicts.

Just another reason why you should be doing test-driven development!

Running Mocha browser tests in TeamCity

Mocha is a great javascript testing framework that supports TeamCity out-of-the-box for testing node.js-based apps on your build server. Here’s a quick guide on how to get it running in TeamCity for browser-based apps as well.

Configuring Mocha’s TeamCity reporter

First we need to configure Mocha to emit specially formatted messages to console.log() that TeamCity can detect and parse. This is easy because Mocha supports pluggable reporters for displaying test progress, and it provides a TeamCity reporter out of the box. Set this up in your mocha.html page:

<script lang="text/javascript">
	mocha.setup({
		ui: 'bdd',
		reporter: function(runner) {
			// Note I am registering both an HTML and a TeamCity reporter here
			// so I can use the same HTML page for local browser development and
			// TeamCity.
			new mocha.reporters.HTML(runner);
			new mocha.reporters.Teamcity(runner);
		},
		ignoreLeaks: true,
		timeout: 5000 // ms
	});

	$(function(){
		mocha.run();
	})
</script>

Executing Mocha tests from the command line

For running our tests we will a special browser, PhantomJS, which is “headless” and has no GUI — you simply invoke it from the command line and interact with it via javascript. Here’s how you can load an HTML page containing Mocha tests — from either the local file system or a web server — passing the URL as a command-line argument:

(function () {
    "use strict";
    var system = require("system");
    var url = system.args[1];

    phantom.viewportSize = {width: 800, height: 600};

    console.log("Opening " + url);

    var page = new WebPage();

    // This is required because PhantomJS sandboxes the website and it does not
    // show up the console messages form that page by default
    page.onConsoleMessage = function (msg) {
        console.log(msg);

        // Exit as soon as the last test finishes.
        if (msg && msg.indexOf("##teamcity[testSuiteFinished name='mocha.suite'") !== -1) {
            phantom.exit();
        }
    };

    page.open(url, function (status) {
        if (status !== 'success') {
            console.log('Unable to load the address!');
            phantom.exit(-1);
        } else {
            // Timeout - kill PhantomJS if still not done after 2 minutes.
            window.setTimeout(function () {
                phantom.exit();
            }, 120 * 1000);
        }
    });
}());

You can then invoke your tests from the command line, and you should see a bunch of TeamCity messages scroll past.

phantomjs.exe phantomjs-tests.js http://localhost:88/jstests/mocha.html

Note in my example I am running tests on a local web server on port 81, but PhantomJS also supports local file:// URLs.

Setting up a build step for PhantomJS

Finally the last thing we need to do is to set up a new Command Line build step in TeamCity to run PhantomJS.

And that’s it! You should now see your test passes and fails showing up in TeamCity:

Acknowledgements

Thanks to Dan Merino for his original article on running Jasmine tests under TeamCity, which formed the basis for most of this post.

Deferring starts with Castleÿs StartableFacility

If you’re using Castle Windsor as your IoC container, the StartableFacility is great simple way to start up services like timers, socket listeners, etc that run for the duration of your application and require two-step initialization — i.e., constructing them then starting them. All you have to do is implement an IStartable interface, with a Start() and Stop() method and Castle will do the rest for you.

The StartableFacility will start each service eagerly by default, as soon as all of its dependencies have been registered in the container. But what if your services take a long time to start up, or you want more control over the time at which they start?

One option is to use the facility’s built-in Deferred Mode that waits until all components have been registered before starting any services.

If that is too early however, you can register all your IStartable components, but not drop the StartableFacility into the container until you want to start them. It will immediately start them all.

public void RegisterServices(){    // register startable services    container.Register(Component.For...));}public void StartServices(){    // drop this in when you're ready to start them    container.AddFacility<StartableFacility>();}

If you need even more fine-grain control, don’t register the facility at all — just loop through all your IStartable components and call Start() on them manually.

Recording the git commit hash in your assembly with TeamCity

If you’re using SVN, you can very easily configure TeamCity to insert the commit revision number into the assembly by using the built-in AssemblyInfo Patcher with a pattern like 1.0.0.%build.vcs.number%. This works because SVN and .NET both use integers for their version numbers.

Git, on the other hand, uses a 40-character SHA-1 string as a unique ID for commits. As you can see, this will not fit in a .NET assembly version:

Example git commit: c9d183c8570143142ca61c555360e7f0732efc09Example git commit (short version): c9d183c857Example .NET assembly version: 1.2.3.4

One place it will fit is the AssemblyInformationalVersion attribute, which accepts any string (thanks Joshka for the tip!). Here’s a quick MSBuild script to do it (using the AssemblyInfo task from MSBuild Community Tasks).

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">  <PropertyGroup>    <MSBuildCommunityTasksPath>packagesMSBuildTasks.1.3.0.528libnet20</MSBuildCommunityTasksPath>        <!-- Use the short version of the git hash. -->    <GitCommitShortHash>$(build_vcs_number.Substring(0, 7))</GitCommitShortHash>  </PropertyGroup>  <Import Project="$(MSBuildCommunityTasksPath)MSBuild.Community.Tasks.Targets"/>  <Target Name="GenerateAssemblyInfo">    <AssemblyInfo CodeLanguage="CS"      OutputFile="MyAppPropertiesAssemblyInfo.cs"      AssemblyInformationalVersion="$(GitCommitShortHash)" />  </Target></Project>

Then in C# you can use something like the following to display it on startup or your About page:

public void Main(){    var gitCommit = Assembly.GetExecutingAssembly()        .GetCustomAttributes(inherit: false)        .OfType<AssemblyInformationalVersionAttribute>()        .Single().Configuration;    Console.WriteLine("My App starting up (git commit {0})", gitCommit);    ...}

(Note Josh Flanagan has an alternative approach where he finds a way to convert the git commit into an int so you can use it in the AssemblyVersion).