Concurrent Programming on Windows

This is probably the definitive book on concurrency in Windows, covering general principles and the relevant APIs across both native (Win32) and managed (.NET). It has a good balance of theoretical discussion and practical advice, with no shortage of references at the end of each chapter for those who feel inclined for some additional background reading. (For instance, the “Further Reading” section at the end of Chapter 10 “Memory Models and Lock Freedom” points to some light reading: AMD x86-64 Architecture Programmer’s Manual, Volumes 1–5 (!))

What makes this book truly valuable is the amount of information and knowledge that it aggregates, from obscure technical sources, academic papers, and even first-hand spelunking in the Windows source code to find answers to some undocumented behavioural details. It also provides plenty of practical advice garnered from years of experience.

For me, having mainly managed programming experience, this book provided a nice opportunity to understand more about the underlying obscurities of Win32, and how these relate to and contrast with what is exposed in .NET. Having that underlying knowledge has let me see how passing the Invalid Wait Handle value to some asynchronous methods can make them execute synchronously instead, and to understand that asynchronous I/O needs to be decided on when a file handle is opened — details that had previously eluded me in my journeys through managed-land.

Other things that were interesting to learn about included lock-free algorithms (with clever tricks like structuring a lock-free linked list such that it has a sentinel node when empty, cunningly avoiding the problem of updating two pointers when the list transitions between empty and non-empty), and the details of kernel-mode synchronisation primitives, with their limitless caveats (the abandoned mutex scenario was my favourite… when waiting on a named mutex it is possible that it would have been abandoned if another process exited before releasing it. Despite returning an error, the operation has succeeded in acquiring the mutex and you must still remember to release it! As if you didn’t have enough to think about by that point, with all the other complexities around alertable waits and pumping the message queue if you’re in an STA).

I previously said that, at a length of 736 pages, CLR via C# (2nd Edition) was the largest book I have ever read. But with a length of 930 pages, Concurrent Programming on Windows has surpassed this. Next up on the reading list: CLR via C# (3rd Edition).

Did you know that asynchronous file I/O in .NET is not just about calling FileStream.BeginRead() or BeginWrite() in place of Read() or Write()? You should also make sure that the FileStream is opened for asynchronous operations, otherwise you’ll quietly get less performant ‘mock’ async operations that just execute synchronous I/O on the thread pool, rather than using true overlapped I/O at the Win32 level.

Excuses, Excuses

The natural starting point for creating a FileStream is the static File.Open() method, the documentation for which mentions nothing about synchronicity of the FileStream that is created! Nor does it allow you to provide FileOptions (which are used to specify the magic FileOptions.Asynchronous flag).

Instead, the FileStream is created with FileOptions.None. Any asynchronous operations are quietly faked by the obliging implementation of the Stream base class, which merely wraps the corresponding synchronous method in a delegate and invokes it on the thread pool using the BeginInvoke() method.

This is a deviation from the usual ‘pit of success’ design philosophy, where everything in .NET seems to work as you think it would, without a need to closely read the documentation and/or gradually discover obscure catches and gotchas over time.

On Balance

Admittedly I’ve never actually used asynchronous file I/O (for the applications I’ve worked on have used databases, queues, and other remote data persistence rather than local files) or else I might have read the FileStream.BeginRead() and BeginWrite() documentation a little more closely:

FileStream provides two different modes of operation: synchronous I/O and asynchronous I/O. While either can be used, the underlying operating system resources might allow access in only one of these modes. By default, FileStream opens the operating system handle synchronously. In Windows, this slows down asynchronous methods. If asynchronous methods are used, use the FileStream(String, FileMode, FileAccess, FileShare, Int32, Boolean) constructor.

 

That last Boolean parameter to the FileStream constructor is called useAsync and, if true, results in FileOptions.Asynchronous being used (or you can also use the other constructor overload which takes FileOptions in the last parameter, and specify FileOptions.Asynchronous yourself).

The underlying Stream.BeginRead() and BeginWrite() methods also talk about synchronicity:

The default implementation of BeginRead on a stream calls the Read method synchronously, which means that Read might block on some streams. However, instances of classes such as FileStream and NetworkStream fully support asynchronous operations if the instances have been opened asynchronously. Therefore, calls to BeginRead will not block on those streams. You can override BeginRead (by using async delegates, for example) to provide asynchronous behavior.

I think this documentation is out of date, or at least a little unclear. The default implementation of BeginRead does not call Read synchronously — Reflector shows that it calls Read by wrapping it in a delegate and calling BeginInvoke, which would result in it being called on a thread pool thread. This is an asynchronous call (with respect to the caller of BeginRead).

Perhaps the documentation is out of date, since it also suggests "using async delegates" to implement your own asynchronous behaviour — what advantage would that give you over the default implementation which does just the same?

As ever, the truth lies in Reflector.

Conclusion

In summary, if you want to do asynchronous file I/O:

• Don’t use File.Open() to create your FileStream — it will be opened for synchronous I/O.
• Create the FileStream directly, specifying useAsync=true (or options=FileOptions.Asynchronous) — this will open the Win32 file handle for overlapped I/O.
• Use BeginRead() and BeginWrite() as normal — the framework will hide the details of overlapped operations behind the Asynchronous Programming Model.

Finally, if you’re using asynchronous I/O you must care about performance, so don’t forget to measure, measure, measure! And heed the warning hidden in the documentation of that useAsync parameter:

useAsync

Specifies whether to use asynchronous I/O or synchronous I/O. However, note that the underlying operating system might not support asynchronous I/O, so when specifying true, the handle might be opened synchronously depending on the platform. When opened asynchronously, the BeginRead and BeginWrite methods perform better on large reads or writes, but they might be much slower for small reads or writes. If the application is designed to take advantage of asynchronous I/O, set the useAsync parameter to true. Using asynchronous I/O correctly can speed up applications by as much as a factor of 10, but using it without redesigning the application for asynchronous I/O can decrease performance by as much as a factor of 10.

ILMerge is released as a console application but also exposes an API to allow you to use it in other applications. For example, I see there are some GUI applications to ease the burden of typing in all those command line switches. ILMerge is mysteriously missing from the community collections of MSBuild tasks, such as the SDC Tasks Library and MSBuild Extended Tasks, probably because it is perfectly feasible to invoke the ILMerge executable using the Exec task that is provided with MSBuild.

The Goal

The goal is to integrate ILMerge into MSBuild, such that it runs automagically every time the project is built (either within Visual Studio, or with MSBuild from the command line).

Unfortunately there are some interesting details to integrate smoothly into the build, such as making sure the task handles incremental builds properly (so that adding ILMerge to one project in a solution doesn’t force a re-build of that entire sub-tree every time you build!)

I’ve not been able to find an adequate pre-canned way to achieve this, but I’ve hacked something together starting from Jomo Fisher’s solution and addressing some of the shortcomings I found along the way.

The Solution

Hand-edit your MSBuild project (e.g. *.csproj) file to tag the referenced assemblies you’d like to merge with the ILMerge=True metadata, like this:

<Reference Include="DependencyLibrary, Version=1.0.0.0, Culture=neutral, processorArchitecture=MSIL">
  <SpecificVersion>False</SpecificVersion>
  <HintPath>Referenced Assemblies\DependencyLibrary.dll</HintPath>
  <ILMerge>True</ILMerge>
  <Private>False</Private>
</Reference>

(Note that it is not necessary to set CopyLocal=True for the target assemblies.)

Then, define the following targets and properties at the bottom of your MSBuild project (just above the </Project> tag):

<Target Name="AfterBuild" DependsOnTargets="ILMerge" />
<PropertyGroup>
  <ILMergeExecutable>"..\BuildTools\ILMerge\ILMerge.exe"</ILMergeExecutable>
  <KeyFile>"$(ProjectDir)MyApplication.snk"</KeyFile>
</PropertyGroup>
<Target Name="ILMerge" Inputs="@(IntermediateAssembly)"
        Outputs="@(MainAssembly -> '%(RelativeDir)%(Filename).ILMergeTrigger%(Extension)')">
  <CreateItem Include="@(ReferencePath)" Condition="'%(ReferencePath.ILMerge)'=='True'">
    <Output TaskParameter="Include" ItemName="ILMergeAssemblies" />
  </CreateItem>
  <Exec Command="$(ILMergeExecutable) /Closed /Internalize /Lib:$(OutputPath) /keyfile:$(KeyFile) /out:@(MainAssembly) &quot;@(IntermediateAssembly)&quot; @(ILMergeAssemblies->'&quot;%(FullPath)&quot;', ' ')" />
  <!-- Make a copy of the merged output DLL to use as a trigger for incremental builds -->
  <Copy
    SourceFiles="@(MainAssembly)"
    DestinationFiles="@(MainAssembly -> '%(RelativeDir)%(Filename).ILMergeTrigger%(Extension)')" />
</Target>

Here's the full wolking solution: ILMergeExperiments

There are a couple of hacks here to deal with the fact that we want our ILMerged assembly to have the same name as the original:

1. We are referencing the intermediate assembly from the ‘obj’ directory as input (because you can’t have the same file as both input and output to ILMerge)
2. A copy of the output file is saved to Foo.ILMergeTrigger.dll, and it is this file that is named as the output of the build target. This is in order to correctly participate in the dependency analysis that is used during incremental builds (because if your merged output assembly has the same name as the unmerged output assembly, then the standard build will overwrite your merged assembly and make it look ‘up to date’, and your ILMerge task will not be executed because its outputs are up to date!)

This is somewhat hacky, and I’m sure there must be a more cunning way to integrate into MSBuild; I’ll have to revisit this once I’ve read the book Inside the Microsoft Build Engine: Using MSBuild and Team Foundation Build.

Background

In Win32 programming, functions that return a HANDLE (such as CreateFile) may return INVALID_HANDLE_VALUE to indicate failure (sometimes). You can check for this return value and call GetLastError to find out why the operation failed.

In .NET functions typically indicate unexpected failure by throwing an exception. The endless dance of “Do something; Did it succeed? If not, why did it fail. Do something else; Did it succeed? …” is replaced by structured exception handling and constructs such as try-catch, which let you defer thinking about error scenarios until you want to, rather than thinking… about errors… at every… step… of… the… way.

So if .NET methods such as File.Open() will throw exceptions rather than returning  INVALID_HANDLE_VALUE we have no need to expose INVALID_HANDLE_VALUE in the .NET BCL, right? Not quite.

INVALID_HANDLE_VALUE as an input parameter

In addition to being used as a magic return value indicating failure, INVALID_HANDLE_VALUE also has some magic powers with methods that accept a HANDLE as a parameter. Now, you won’t get any useful behaviour from passing INVALID_HANDLE_VALUE to CloseHandle, however there is a group of functions that let you provide an event handle, do some asynchronous work, and then signal your event to let you know the work has been completed.

Functions such as UnregisterWaitEx and DeleteTimerQueueTimer will cancel any pending registered wait operation or a timer-queue timer, however if a callback has already been triggered this will still run to completion. If you need to clean up any resources used by your callback, to avoid pulling the rug out from under its feet, you must first ensure that your callback is not still executing. To avoid having to manually introduce control synchronisation in your callback, UnregisterWaitEx and DeleteTimerQueueTimer let you provide an event handle which will be signalled when any executing callbacks have returned.

If you don’t want the overhead of allocating another event and then registering a wait on it (in order to perform the clean-up asynchronously, when the event is signalled) you can tell the function to block and wait for any executing collback functions to complete before returning by providing INVALID_HANDLE_VALUE for the wait handle.

What about .NET?

Now the interesting part: Since we previously concluded that there is no need to expose INVALID_HANDLE_VALUE in the .NET BCL, how would we get this handy blocking behaviour from the .NET equivalents to the methods mentioned above: RegisteredWaitHandle.Unregister(WaitHandle) and System.Threading.Timer.Dispose(WaitHandle)?

The MSDN documentation makes no suggestion that this behaviour is even possible (not even in the preview documentation for .NET 4). I’m not sure if this is an oversight or an intentionally unsupported behaviour.

To work around this we can do a little poking around the BCL with Reflector:

• WaitHandle, the base class for all events in .NET, has a static readonly IntPtr field called InvalidHandle which is populated with the value of INVALID_HANDLE_VALUE; but
• Alas, WaitHandle.InvalidHandle is protected, rather than public! (Why, oh why?)
• Conveniently, the internal handle value is initialised by the default constructor of WaitHandle to InvalidHandle; but
• Alas, WaitHandle is marked abstract, so we can't instantiate it directly.

Conclusion

So the only way to get at INVALID_HANDLE_VALUE in .NET is to subclass WaitHandle. We don't actually need to do anything in our subclass, mind you:

public class InvalidWaitHandle : WaitHandle { }

So there you have it, pretty convoluted but works like a charm!

Here’s the full version, with documentation and a cached instance:

using System.Threading;

/// <summary>
/// An inert wait handle that can be used to avoid allocating a real event in
/// some situations.
/// </summary>
/// <remarks>
/// <para>
/// An <see cref="InvalidWaitHandle"/> can be provided to methods such as
/// <see cref="RegisteredWaitHandle.Unregister(WaitHandle)"/> and 
/// <see cref="Timer.Dispose(WaitHandle)"/>. In this case, the function waits
/// for all callback functions to complete before returning, rather than
/// returning immediately and signalling the provided wait handle
/// asynchronously.
/// </para>
/// <para>
/// Internally, this results in the use of INVALID_HANDLE_VALUE when calling
/// the underlying Win32 functions.
/// </para>
/// <para>
/// For further information, see "Concurrent Programming on Windows" (First
/// Edition, 2009) by Joe Duffy, p. 374, 377.
/// </para>
/// </remarks>
public class InvalidWaitHandle : WaitHandle
{
    static InvalidWaitHandle()
    {
        Instance = new InvalidWaitHandle();
    }

    /// <summary>
    /// Gets a shared instance of <see cref="InvalidWaitHandle"/> which may
    /// be re-used.
    /// </summary>
    /// <remarks>
    /// Using this field allows a single <see cref="InvalidWaitHandle"/> to
    /// be re-used as opposed to creating a <c>new</c> instance at every call
    /// site.
    /// </remarks>
    /// <value>A shared instance of <see cref="InvalidWaitHandle"/> which may
    /// be re-used.</value>
    public static InvalidWaitHandle Instance { get; private set; }
}

This was all fairly straightforward, until he linked to a mind-melting video of a country singer asserting that he is his own grandfather, complete with an accompanying family tree diagram that updates as the song progresses to help you keep track of the facts:

So I guess the moral of the story is that directed graphs should ideally be kept acyclic if at all possible?

I’m comforted to know that I’m not the only one to go to town on analysis of everyday things.

And now, back to our regular programming...

Drive-through optimisation

Last time, I set the scene of a restaurant drive-through, with the hopes of illustrating how innocent optimisations can introduce security vulnerabilities if we’re not careful.

We forgot to do our threat modelling! Nobody stopped to ask the question “what are we changing? what does this affect? how could this compromise our security?”

The “null-order”

Let’s consider the “null-order”: An in-store customer reaches the head of the queue, walks up to the counter, and realises that she doesn’t have any money. The order transaction is safely aborted and the customer leaves. There is no need for the restaurant employee to enter the “null-order” in their computer because there is no order to be made, duh! (Note that the customer realised the lack of wallet before placing the order.)

Now lets consider a “null-order” for the drive-through interface: A customer joins the back of the queue and is (often, but not always) in a single-lane one-way traffic system, where the only way out is to drive through the whole system: past the order point, the pay window, and the delivery window. If the drive-through is busy, there will be other cars occupying this space so the driver is stuck — they cannot escape the pipeline.

When they realise they have no money (again, before placing the order) they will place a “null-order”. The employee will not enter the “null-order” into the computer because there is no order being made. Duh! The customer wants to leave the queue at this point but she is stuck, so has to drive to the payment window. She must then execute a “null-payment” and explain to the cashier that no payment is necessary because no order was placed. Duh! No problem. Then the customer drives to the third window, staffed by an employee whose job is to deliver food to the cars as fast as possible, preferably with a smile.

The exploit

So what does all this have to do with security vulnerabilities of concurrent pipeline processing?

Well, one fun example of a non-software exploit is the fast-food drive-through free food scam:

1. Go to the drive-through of a fast-food restaurant and make sure there is someone behind you in the queue.
2. Don’t order anything (say you forgot to bring money).
3. Go to the pay window and say you didn’t order anything.
4. Proceed to the order pick-up window and collect the food that the car behind you is currently paying for.

Security specialist Bruce Schneier describes this in more detail and offers some solutions.

One solution

Continuing with my previous analogy: Since the order-point employee didn’t enter the “null-order” into the system (duh!) it didn’t get pushed onto the internal order queue. At that point the internal order queue was no longer synchronised with the physical queue of cars. Therefore, the cashier didn’t know she was expecting a “null-payment” from this customer, but this could be explained away by the customer. If the customer doesn’t also explain this at the order-delivery window they will be handed the order placed by the car behind (which is currently being paid for right behind them!)

By introducing this pipeline optimisation, we’ve suddenly introduced the need for the in-sore systems to be able to process a “null-order”, which sounded like a ludicrous idea for the in-store scenario, but is actually essential for keeping the drive-through queue synchronised with the internal order queue.

As I said before, security is a burden that you must carry through any refactoring, optimisations, or other changes; even those that seem unrelated.

The fast-food restaurant’s basic feature is the ability to go in, order some food, pay for it, and receive the order (preferably within a short period of time).

The fast food drive-through

At some point this wasn’t enough. Cars got popular, people were in a rush, and a new interface was dreamt up which could allow this workflow to be executed without having to park your car and enter the restaurant — the drive-through was born.

Presumably the naïve drive-through interface would mimic the user experience of the in-store interface: There are several windows with individual queues, you pick one and join the queue with your car, drive up to the window, place your order, pay, and receive food all at the same window, then leave.

This presents some problems because cars are not as small and agile as humans, so trying to mimic the same layout for cars as humans isn’t going to work very well. (You’d need a massive amount of room to let the car from each window drive off independently.)

Given the amount of room available at a typical restaurant, the drive-through would need to only have a single queue. You would join the back of the queue, eventually reach the window, place and order, pay, and receive food all at the same window.

Optimising throughput at the drive-through

This design is still as secure as the in-store experience, but because there is only one window and one queue it is a lot slower than the in-store experience, where there are multiple restaurant employees serving multiple customer queues concurrently.

So how do we make a multi-stage process at the lone drive-through window more efficient? We introduce parallelism through pipelining (as seen in most processor architectures these days). We split the multi-function order/payment/deliver window up into three separate windows (or a microphone and two windows) and specialise each window to a single task. (Fast-food restaurants are already good at efficiency through task specialisation.)

Now the order window (or microphone) can be ready to take the next order as soon as it is done taking the current one. The payment window can be taking payment for the first order whilst the second order is being dictated. Then the third window can be delivering the order whilst the second window is taking payment for the next order, and the order window (or microphone) is taking yet a third order.

Genius! We’ve improved the efficiency by allowing three customers to be served at once, whilst keeping to the space constraint of only having a single queue of cars at the drive-through.

Can you spot the security hole we’ve opened up with this optimisation?

If it looks like a duck, and quacks like a duck, then it must be a duck!

“In computer programming, duck typing is a style of dynamic typing in which an object's current set of methods and properties determines the valid semantics, rather than its inheritance from a particular class or implementation of a specific interface.” — Wikipedia

With duck-typing an interface implementation is implicit once you have implemented the relevant members. .NET does not currently have any broad support for this, however, with the emergent dynamic language features, I wouldn't be surprised if this were supported natively by the runtime in the near future.

In the mean time, you can synthesise duck-typing via reflection, with a library such as this, which would allow you to do a duck-typed cast like this:

IDoo myDoo = DuckTyping.Cast<IDoo>(myFoo)

Duck Typing Trivia

Interestingly, there is one small place where duck-typing is in use in C# today — the foreach operator. Krzysztof Cwalina states that in order to be enumerable by the foreach operator, a class must:

Provide a public method GetEnumerator that takes no parameters and returns a type that has two members: a) a method MoveMext that takes no parameters and return a Boolean, and b) a property Current with a getter that returns an Object.

Notice that he makes no mention of IEnumerable nor IEnumerator. Although it is common to implement these interfaces when creating an enumerable class, if you were to drop the interfaces but leave the implementation, your class would still be enumerable by foreach. Voila! Duck-typing!

Example Code

But don’t take my word for it. Here’s some demo code to prove it:

public class Program
{
    public static void Main()
    {
        foreach (int i in new DuckEnumerable())
            Console.WriteLine(i);
        Console.ReadKey();
    }
}

public class DuckEnumerable // Not IEnumerable
{
    public Duck GetEnumerator()
    {
        return new Duck();
    }
}

public class Duck // Not IEnumerator
{
    private int n = 0;

    public int Current
    {
        get { return this.n; }
    }

    public bool MoveNext()
    {
        return (this.n++ < 10);
    }
}

Overview

When generating an XML serializer for schemas that feature a free-form xs:any node, the deserialization behaviour is incorrect in some scenarios.

For example, if you want the capability to hold an arbitrary fragment of XML configuration that is specified in a client-specific schema (which is not known up front, and cannot be included in your schema) you might address this by including a “freestyle” configuration element like this.

<xs:element name="config" minOccurs="0">
  <xs:complexType mixed="true">
    <xs:sequence minOccurs="0">
      <xs:any processContents="skip" />
    </xs:sequence>
  </xs:complexType>
</xs:element>

If you then use the XmlSerializer to deserialize this configuration, (or the sgen utility, which uses XmlSerializer under the covers) you may run in to the problem detailed below.

Walkthrough

Consider the following pair of documents:

Document A

<Root>
  <FirstChild>
    <config></config>
  </FirstChild>
  <SecondChild/>
</Root>

Document B

<Root>
  <FirstChild>
    <config/>
  </FirstChild>
  <SecondChild/>
</Root>

These two documents should be equivalent. The only difference is between <config></config> and <config/>. Just to make sure I’m not going insane, the XML Specification says that “The representation of an empty element is either a start-tag immediately followed by an end-tag, or an empty-element tag.” So these two documents are slightly different representations of something which should be semantically identical. The output from XML deserialization should be the same for both.

However, this is not the case (at least not in .NET 2.0 SP 2, version 2.0.50727.3053).

Document A is deserialized successfully. The FirstNode and SecondNode elements are non-null in the resulting class instance.

Document B is also deserialized successfully, however the XmlSerializer instance raises its UnknownNode event for the SecondChild node. In the resulting deserialized class instance, the FirstNode is accessible, but the SecondNode element is null!

Root Cause

The reason for this appears to be a bug in the generated serializer class which causes the method responsible for deserializing FirstChild (method Read2_RootFirstChild in the repro solution, linked below) to overrun past the end of the FirstChild element, and consume the SecondChild element from the reader, reporting it as an unrecognised element. (See 2l5qpkfr.0.cs line 251 in the repro solution.)

Subsequently, the method responsible for deserializing SecondChild (method Read1_Object) is unable to deserialize this element because it has already been consumed!

Standalone Reproduction of the problem

Here is a Visual Studio solution with a standalone repro of the problem, including an annotated copy of the generated serialization classes, showing where I believe the bug to be:

XmlSerializationRepro

I was unable to find this bug on Microsoft Connect so I am not sure if this issue is known to Microsoft.

Update (12 July 2009): Reported this on Microsoft Connect.

(new XmlDocument()).Load((Stream)null);

Which results in the following exception:

System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Xml.XmlReader.CalcBufferSize(Stream input)
   at System.Xml.XmlTextReaderImpl.InitStreamInput(Uri baseUri, String baseUriStr, Stream stream, Byte[] bytes, Int32 byteCount, Encoding encoding)
   at System.Xml.XmlTextReaderImpl.InitStreamInput(Stream stream, Encoding encoding)
   at System.Xml.XmlTextReaderImpl..ctor(String url, Stream input, XmlNameTable nt)
   at System.Xml.XmlTextReader..ctor(Stream input, XmlNameTable nt)
   at System.Xml.XmlDocument.Load(Stream inStream)
   at Scratch.XmlDocumentNullCheckTest.CallLoadWithNullStream()

You can see why I was unsure of myself — the rogue null stream drills in through five levels of functions before reaching the innocent-enough CalcBufferSize() method, which asks if the null stream if it CanSeek.

What XmlDocument.Load(Stream) should really do is to trap my null input right at the front door, and throw the specialised exception which exists for this exact scenario, System.ArgumentNullException, instead of exposing its implementation internals to my rogue input value.

Another Culprit: XmlTextReader

The first thing XmlDocument.Load(Stream) does is to delegate the task to the XmlTextReader class. Since XmlTextReader is a public class, and its (Stream, XmlNameTable) constructor is also public, then XmlTextReader is also in violation. (Perhaps the author of XmlDocument.Load() felt it acceptable to not validate for a null stream because they were relying on the XmlTextReader constructor doing that check for them?)

[ These problems exist in System.Xml 2.0.0.0 (2.0.50727.4918) but have probably been fixed in the version that ships as part of .NET 4.0. ]

Static Analysis: CA1062

The other reason I refused to believe this could be true until I had reproduced it is because I know that there is a static code analysis rule built in for this exact scenario: CA1062: Validate arguments of public methods. This rule should trigger for both XmlDocument.Load(Stream) and XmlTextReader..ctor(Stream, XmlNameTable) to remind the developer to validate any reference arguments passed in to externally visible methods.

Not in VS2008

I tried to validate the behaviour of this rule when I was reminded that this static analysis rule (and a few others) were actually removed in Visual Studio 2008! One of the buggy and ill-performing static analysis engines was cut loose for VS2008 (along with the rules that depended upon it) as part of a longer-term strategic move to write a new data flow analysis engine based on MSR’s “Phoenix”.

Back for VS2010

The rules that were removed for 2008 were reinstated in the Visual Studio 2010 September CTP. When they talk about “8 New Data Flow rules” in the September CTP, I guess they really mean “8 old rules from VS2005, that were removed in VS 2008, are now back” — I confirmed this by correlating the new analysis rules listed in the VS2010 September CTP Walkthroughs document with the list of rules removed in VS 2008 due to removal of the data flow engine.

I was hoping that some new rules would also pop up in the Beta, but there’s been no mention of this on the Code Analysis Team Blog so maybe they only managed to reinstate the old rules that were removed for now. This is understandable, considering they had to re-write the data flow analysis engine.

Update (12 July 2009): Reported on Microsoft Connect.

Update (16 July 2009): This can’t be fixed because it might break backward compatibility for applications that are already trapping the NullReferenceException. Ah, one of the joys of strict framework versioning: Unfixable bugs!

So we’ve concluded that the naïve implementation of up_CompareAndSwap is woefully inadequate in achieving its main goal: performing an atomic compare-and-swap operation.

The Succinct Solution: Fancy Update Statements

One way to resolve this is to make our implementation more succinct. The programmatic CAS operation is implemented by a single compare-exchange instruction at the CPU level; maybe we can implement the core of up_CompareAndSwap with a single compare-update SQL statement?

This would mean combining the select-check-update stages of the original stored procedure into a single update statement. Something along these lines:

update KeyValuePairs
set [Value] = @p_value
where [Key] = @p_key
    and [Value] = @p_comparand

Then we can test @@ROWCOUNT to find out if the row was updated or not, but sadly this is not enough. Even in this simple scenario, we need more information to fulfil the return codes contract:

-- RETURNS: 0 if KeyValuePairs was updated;
--          1 if KeyValuePairs was not updated;
--          2 if the specified p_key was not found.

If @@ROWCOUNT is 0 we can’t know if this was because the Key comparison failed (in which case we should return 2) or if it was because the Value part of the comparison failed (in which case we should return 1). There isn’t enough information.

Fortunately it turns out that the set clause can do a lot more than just assign values to columns. We can go ahead and extract the current value of the column before we update it, right in the set clause:

-- Optimistically update KeyValuePairs
update KeyValuePairs
set @currentValue = [Value],
    [Value] = @p_value
where [Key] = @p_key

Instead of checking the current value against @p_comparand in the where clause, we go ahead and update it optimistically.

Now we know that if @@ROWCOUNT is 0, it must be because that Key was not found in the table, so we can return 2:

if (@@ROWCOUNT != 1)
begin
    raiserror('Key <%s> was not found.', 16, 0, @p_key)
    commit transaction
    return 2
end

If the Key was found, and we did update a row, we can retrospectively validate the previous value against @p_comparand:

-- Validate the old value against p_comparand and rollback if necessary
if (@previousValue != @p_comparand)
begin
    print 'Comparand does not match. Value was not updated.'
    rollback transaction
    return 1
end

If the values don’t match, we undo our update by rolling back the transaction, and return 1 to signal that the entry was not updated.

One downside to this approach is that our optimistic update may be visible to readers using READ UNCOMMITTED isolation (e.g. via the NOLOCK hint) before being rolled back. Such readers could observe a phantom value “that never was”, but then again, that is exactly what you bargain for when you start to ignore transaction isolation boundaries.

The Explicit Solution: Locking Hints

My final variant on this stored procedure is to explicitly decorate the initial select statement with a locking hint. Our problem before was that the select statement only acquires a ‘Shared’ lock for the duration of the statement, and then releases this lock and leaves the row open to change.

We can use the HOLDLOCK hint to ensure that the ‘Shared’ lock is held until the end of the transaction. This protects us from having someone else update the row after we’ve read it and before we’ve updated it. However, this is not enough: Another instance of this same procedure could run concurrently and read the current, unchanged, value. Its validation would pass, and it would attempt to update the value, only then would it be blocked and wait for the ‘Shared’ lock to be released. After we write our new value, this concurrently executing instance would immediately overwrite it with its new value! That’s no good.

So we need something stronger. The XLOCK hint will force our ‘Shared’ lock to be upgraded to an ‘Exclusive’ lock, which will prevent anyone else from reading the value. This solves the problem noted above, but is a little too restrictive. We don’t have any problem with anyone reading the value, so long as they’re not reading it with the intent of later updating it.

SQL Server has another lock mode called ‘Update’, which communicates this exact intent of “reading with an intent to update”. We can use the UPDLOCK hint to acquire this ‘Update’ lock as part of the select statement. This will prevent anyone else acquiring an ‘Exclusive’ or ‘Update’ lock in the interim between our select and update statements. (A concurrently executing instance of the same procedure would attempt to acquire an ‘Update’ lock when reading the data, and will therefore be blocked on its select statement until our transaction is complete.) Any other read-only process, however, is still free to read the current value during this time. Once we reach the update statement the ‘Update’ lock will automatically be upgraded to an ‘Exclusive’ lock, and at this point both readers and writers would be blocked from accessing that row until the transaction completes.

If we want to ensure that this ‘Update’ lock is only acquired on the unique row in question, rather than an entire data page from the table, we can additionally use the ROWLOCK hint. This will ensure that if there is high contention for updating the KeyValuePairs table we won’t be needlessly serialising updates to separate rows.

/* Procedure up_CompareAndSwap

 Compares the value stored for p_key in the KeyValuePairs table
 against the p_comparand value.
   * If they match, the new p_value is stored in KeyValuePairs.
   * If they do not match, KeyValuePairs is not updated.

 Returns: 0 if KeyValuePairs was updated;
          1 if KeyValuePairs was not updated;
          2 if the specified p_key was not found.
*/
create procedure up_CompareAndSwap 
    @p_key nvarchar(50),
    @p_value int,
    @p_comparand int
as
begin
    begin transaction

    -- Read current value and hold on to an Update lock on the row
    declare @currentValue int
    select @currentValue = Value
    from KeyValuePairs with (UPDLOCK, ROWLOCK)
    where [Key] = @p_key
    
    if (@@ROWCOUNT != 1)
    begin
        raiserror('Key <%s> was not found.', 16, 0, @p_key)
        commit transaction
        return 2
    end

    -- Compare to the expected value (and return if they don't match)
    if (@currentValue != @p_comparand)
    begin
        print 'Comparand does not match. Value was not updated.'
        commit transaction
        return 1
    end

    -- Update to the new value
    -- We hold an Update lock on this row, so it couldn't have changed
    -- since we read it. The Update lock will be upgraded to 'Exclusive'
    update KeyValuePairs
    set [Value] = @p_value
    where [Key] = @p_key

    print 'Value was updated successfully.'
    commit transaction
    return 0
end
GO

This solution has the advantage of being simpler to implement, and simpler to retrofit to the an existing procedure — all I had to change was the locking hints for the initial select statement. It also feels cleaner to have done the necessary checks before updating the row, rather than optimistically updating it ahead of the checks, and potentially rolling it back subsequently.

The End

I hope you’ve enjoyed learning a little bit about SQL Server with me. As you can probably tell, I’m a bit of a novice when it comes to this subject matter. If there are any mistakes in what I’ve said above (or in the previous posts in this series), please let me know in the comments. Thanks!

There are a couple of problems with the intentionally-buggy CompareAndSwap stored procedure I published yesterday. Let’s take another look:

up_CompareAndSwap

/* Procedure up_CompareAndSwap

 Compares the value stored for p_key in the KeyValuePairs table
 against the p_comparand value.
   * If they match, the new p_value is stored in KeyValuePairs.
   * If they do not match, KeyValuePairs is not updated.

 Returns: 0 if KeyValuePairs was updated;
          1 if KeyValuePairs was not updated;
          2 if the specified p_key was not found.
*/
create procedure up_CompareAndSwap 
    @p_key nvarchar(50),
    @p_value int,
    @p_comparand int
as
begin
    begin transaction

    -- Read current value
    declare @currentValue int
    select @currentValue = Value
    from KeyValuePairs
    where [Key] = @p_key
    
    if (@@ROWCOUNT != 1)
    begin
        raiserror('Key <%s> was not found.', 16, 0, @p_key)
        commit transaction
        return 2
    end

    -- Compare to the expected value (and return if they don't match)
    if (@currentValue != @p_comparand)
    begin
        print 'Comparand does not match. Value was not updated.'
        commit transaction
        return 1
    end

    -- Update to the new value
    update KeyValuePairs
    set [Value] = @p_value
    where [Key] = @p_key

    print 'Value was updated successfully.'
    commit transaction
    return 0
end
GO

What’s wrong with this stored procedure?

The Problem: Atomicity

If we compare the naïve CompareAndSwap stored procedure above to a typical CAS operation, the glaring shortfall is that a true CAS operation is Atomic; to the extent that it is actually implemented by a single compare-exchange instruction at the CPU level.

The naïve CompareAndSwap stored procedure, on the other hand, has a distinct read-modify-write flow, and despite being enclosed in a SQL Transaction, there are no guarantees of atomicity (at the default READ COMMITTED transaction isolation level).

The read operation will acquire a Shared lock on the particular row of the KeyValuePairs table only for the duration of the select statement. After this statement is complete the lock is relinquished, leaving other transactions (including a concurrently executing instance of the same stored procedure!) free to read and modify this row before the update statement has executed.

You can convince yourself of this by adding an artificially long delay between the select and subsequent update statement…

if (@p_slow = 1)
    waitfor delay '00:00:10'

…and then executing concurrent queries targeting the same key.

This is bad! We need some way to bring atomicity to the CompareAndSave stored procedure.

Read on in Compare-and-Swap in SQL: Part 3.

When calling the procedure you specify three things:

• A Key, which identifies a row in the table;
• The new Value that you would like to associate with that Key; and
• What you think the current Value associated with that Key is.

The procedure will swap in your new Value for that Key, but only if the current Value is still what you thought it was.

This allows you to read the value (e.g. 42), transform it in some way (e.g. increment it to 43), then store the new value, making sure that nobody else has changed the value while you were busy transforming it.

For example, if someone else changed the value to 123 while you were incrementing to 43, it would not be valid for you to store 43 as the incremented value because then you would be overwriting the change to 123, as if it never happened. What you would need to do is start over again, with 123, increment it to 124, and store that (so long as nobody has changed the 123 to yet another value in the mean time).

Here’s a naïve stored procedure that implements this logic:

-- KeyValuePairs table definition
create table dbo.KeyValuePairs (
    [Key] nvarchar(50) not null primary key,
    Value int not null)
GO

/* Procedure up_CompareAndSwap

 Compares the value stored for p_key in the KeyValuePairs table
 against the p_comparand value.
   * If they match, the new p_value is stored in KeyValuePairs.
   * If they do not match, KeyValuePairs is not updated.

 Returns: 0 if KeyValuePairs was updated;
          1 if KeyValuePairs was not updated;
          2 if the specified p_key was not found.
*/
create procedure up_CompareAndSwap 
    @p_key nvarchar(50),
    @p_value int,
    @p_comparand int
as
begin
    begin transaction

    -- Read current value
    declare @currentValue int
    select @currentValue = Value
    from KeyValuePairs
    where [Key] = @p_key
    
    if (@@ROWCOUNT != 1)
    begin
        raiserror('Key <%s> was not found.', 16, 0, @p_key)
        commit transaction
        return 2
    end

    -- Compare to the expected value (and return if they don't match)
    if (@currentValue != @p_comparand)
    begin
        print 'Comparand does not match. Value was not updated.'
        commit transaction
        return 1
    end

    -- Update to the new value
    update KeyValuePairs
    set [Value] = @p_value
    where [Key] = @p_key

    print 'Value was updated successfully.'
    commit transaction
    return 0
end
GO

So, what’s wrong with this stored procedure?

Specifically, what shortfall does it have when comparing it to the characteristics of CAS operations such as .NET’s Interlocked.CompareExchange()?

Read on in Compare-and-Swap in SQL: Part 2.

Consider the following MSBuild project file, which invokes a batch script:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Default">
    <Exec Command="batch.cmd" />
  </Target>
</Project>

@ECHO OFF
ECHO Doing stuff
ECHO Failed...
EXIT /B 42

The only problem is that the MSBuild script above will merrily announce that the build succeeded, prompting you to scratch your head a little and wonder.

You might, as I did, try it without the /B switch, and you'd see that it works — MSBuild traps the non-zero return and fails the build. And if you're the sort of person that doesn't ask many questions this might suffice. Only you'd be left with a rather rude batch file.

Let's review the documentation for the EXIT command:

C:\>exit /?
Quits the CMD.EXE program (command interpreter) or the current batch script.
 
EXIT [/B] [exitCode]

  /B          specifies to exit the current batch script instead of
              CMD.EXE.  If executed from outside a batch script, it
              will quit CMD.EXE

  exitCode    specifies a numeric number.  if /B is specified, sets
              ERRORLEVEL that number.  If quitting CMD.EXE, sets the process
              exit code with that number.

So the /B stands for "Behave". If you call EXIT 1 in the middle of your batch you immediately and unconditionally cause the command shell to exit with that error code.

• What if you were being called from another batch script? Maybe it wanted to do some cleanup? Or carry on regardless of your failure? Maybe it wanted to log the error? It can't do any of these things because you just pulled the plug on CMD.EXE.
• What if you were being called interactively from a command window, by a user? They are going to see their window disappear without a trace, and with little opportunity to see any error messages that were output to the screen.

So we want to be good, and have our script Behave, so we put back that /B switch, and scratch our head some more about why MSBuild is missing this.

Then we get bored of scratching and bust out .NET Reflector to go to town on the Exec task.

So it turns out that internally, Exec doesn't just call your command directly. Instead it wraps it in a batch script of its own! It generates a batch in your temp directory that looks something like this:

setlocal
set errorlevel=dummy
set errorlevel=
batch.cmd
exit %errorlevel%

Then it invokes that with CMD /C and quickly deletes the temporary file. I'm not quite sure why it goes through the trouble of this intermediate batch file, to be honest; what extra value does it add? Wouldn't CMD /C batch.cmd give the same outcome?

One thing that it does add is a little silent caveat. If you invoke a batch file from within another batch file, the first batch file will never resume execution unless you use the CALL command to invoke the second batch.

So what is happening here is that, because we had no idea of this undocumented batch file being created in the background by the Exec task, our command of "batch.cmd" is being plonked in the middle of this generated batch file, and then rudely prevents the generated batch from explicitly calling EXIT to return the error code to MSBuild!

The solution? If you're going to call a batch file with the Exec task, prefix it with a "CALL".

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Default">
    <Exec Command="CALL batch.cmd" />
  </Target>
</Project>

This will ensure that the outer script runs to completion, and exits CMD returning the appropriate errorlevel value.

(Alternatively, you could upgrade to a suitabley recent operating system, and forget everything you've just read. Somewhere between Windows XP and Windows 7, CMD.EXE has acquired the elegance of picking up the latest errorlevel value and using that as its return code, so that even if that last EXIT command isn't executed, it will still pick up the value set by the inner batch script.)

Here are my summary learnings:

• The problem of calling event handlers after they have been unsubscribed from the event is entirely separate from the null-protection problem.
  Eric highlights that there are two separate problems here, with separate responsibilities
  1. The event publisher must ensure they don’t cause a NullReferenceException by attempting to invoke an empty delegate chain after the last subscriber has unsubscribed from the event. (See my post on .NET Event invocation thread safety)
  2. The event subscriber must be able to deal with the fact that a handler may be called shortly after it has been unsubscribed from the event. (See next point.)
• “Event handlers are required to be robust in the face of being called even after the event has been unsubscribed”
  Although not explicitly documented anywhere (yet) this conclusion is inevitable for multi-threaded code, particularly in light of the Jon Skeet’s thought experiment: “Now suppose that the invocation list for that delegate has 1000 entries. It's perfectly possible that the action at the start of the list will have executed before another thread unsubscribes a handler near the end of the list. However, that handler will still be executed because it'll be a new list. (Delegates are immutable.) As far as I can see this is unavoidable.” 
• Juval Löwy’s concern that compiler may optimise out the temporary copy of the delegate chain is unfounded
  I avoided talking about this twist in the post on .NET Event invocation thread safety as it sounded a little too hypothetical. In his book, after describing the classic “Richter” take-a-copy-and-check-for-null pattern, Juval  postulates that the compiler may optimise away the temporary copy for you, and undermine the pattern. Jon Skeet states that “The JIT isn't allowed to perform the optimization you're talking about in the first part, because of the condition. I know this was raised as a spectre a while ago, but it's not valid. (I checked it with either Joe Duffy or Vance Morrison a while ago; I can't remember which.)” so that’s a comfort.

The main thrust of the StackOverflow question is now:

Why is explicit-null-check the "standard pattern"? The alternative, assigning the empty delegate, requires only = delegate {} to be added to the event declaration, and this eliminates those little piles of stinky ceremony from every place where the event is raised. It would be easy to make sure that the empty delegate is cheap to instantiate. Or am I still missing something?

I still have some reservations about this, and suspect that the likes of Joe Duffy might have something to say about the performance impact of this. Although my performance analysis certainly suggests that the invocation-time overhead of this is, in real terms, negligible, I haven’t done any testing on the initialisation-time overheads. Still, the impact of this is likely to be negligible, and as Earwicker points out “Why let the ugly way be the recommended way? If we wanted premature optimisation instead of clarity, we'd be using assembler”

So, should we all be replacing the copy-and-null-check pattern with the initialise-with-empty-delegate pattern?

One such article I recently read was in Dr. Dobb's, written by Herb Sutter, and entitled Sharing Is The Root Of All Contention. Herb talks about the obvious concurrency impediments that are directly visible in code, but he also talks in-depth about the more elusive concurrency impediments that are hidden in the software layers and underlying hardware.

Here is Herb's fantastic breakdown matrix of "contention penalties" which summarises the main thrust of his article quite well (I'm not sure why it was included as a blurry image rather than a table, but it's like that in the original article):

Check out the full article for some great detail on each of these areas: Sharing Is The Root Of All Contention, by Herb Sutter

A few years ago I was presenting a paper at the annual IEEE Engineering in Medicine and Biology Society conference in New York, and I attended a presentation about a pacemaker device which was remotely re-programmable. I think the idea was to allow tuning and refinements without invasive surgery. There was no mention of security during the presentation. We’re talking about a remotely programmable pacemaker here… I questioned security at the end but the presenter did not seem concerned. I guess it hadn’t been a problem yet. (And I’m not sure if medical devices are subject to threat modelling, yet.)

Security is a big issue on the internet because the interconnectedness that provides much of the value of the internet is also its Achilles’ heel in terms of large-scale automated attack and exploitation. In areas such as internal software development, it’s not such a big consideration because the software is deemed to be isolated enough from the internet by firewalls and other protection (more on insider attacks another day).

The problem is that security costs money, and requires expertise. It is a burden that you must carry through any refactoring, optimisations, or other changes to the software. Until lack of security becomes a problem, it is hard to trade off a hypothetical threat that most people don’t think about, against the potential goodness of not having to worry about security (be it faster completion, more features, smaller budget, etc.).

By the time security becomes a problem, it’s often too late. (Pacemaker-B-Gone, anyone?)  Introducing security late in the software development life-cycle is an uphill battle when it hasn’t been designed-in from the ground up. Introducing security to a pacemaker that has already been “installed” might be even more of a challenge.

In CLR Via C# Richter points out a few subtle points about event invocation in multi-threaded classes:

• A delegate chain is immutable; a new chain is created to replace the first.
• A delegate chain with zero subscribers is null.
• That means (if your event is public) it may transition from null to non-null and vice versa, at any time.

The Naive Approach: Not thread safe

So consider this invocation code which raises an event:

public static event EventHandler<EventArgs> NonThreadSafeEvent;

public static void OnNonThreadSafeEvent(EventArgs e)
{
    if (NonThreadSafeEvent != null)
    {
        // Event could still become null in this interim,
        // after the check but before the invocation
        NonThreadSafeEvent(null, e);
    }
}

On a class that is being accessed by multiple threads, this could lead to a NullReferenceException, despite the well-intentioned null guard.

The Solution Space

There are a number of ways to overcome this problem and guarantee that multi-threaded classes will never be exposed to the risk of a NullReferenceException when attempting to invoke a delegate or event.

The Classic Solution: Take a copy

This is the solution that Richter proposes to achieve thread safety:

public static event EventHandler<EventArgs> ClassicNullCheckedEvent;

public static void OnClassicNullCheckedEvent(EventArgs e)
{
    EventHandler<EventArgs> localCopy = ClassicNullCheckedEvent;
    if (localCopy != null)
    {
        // Nobody can change our local copy so we're sure it's not null
        localCopy(null, e);
    }
}

New-Age Solution: Pre-Initialise

I like to call this the Juval Löwy solution, because he proposes it in one of his books:

“You can ensure that the internal invocation list always has at least one member by initializing it with a do-nothing anonymous method. Because no external party can have a reference to the anonymous method, no external party can remove the method, so the delegate will never be null”
    — Programming .NET Components, 2nd Edition, by Juval Löwy

public static event EventHandler<EventArgs> PreInitializedEvent = delegate { };

public static void OnPreInitializedEvent(EventArgs e)
{
    // No check required - event will never be null because
    // we have subscribed an empty anonymous delegate which
    // can never be unsubscribed. (But causes some overhead.)
    PreInitializedEvent(null, e);
}

Of course then he immediately says that “initializing all delegates this way is impractical” yet without explaining why it is impractical. Seems fine to me! Certainly more practical than remembering to copy-and-check-for-null every time you want to raise an event.

Performance Implications

As always, there are some subtle performance implications to each approach (particularly the last one!)

Executing 50000000 iterations . . .
OnNonThreadSafeEvent took:      432ms
OnClassicNullCheckedEvent took: 490ms
OnPreInitializedEvent took:     614ms
Subscribing an empty delegate to each event . . .
Executing 50000000 iterations . . .
OnNonThreadSafeEvent took:      674ms
OnClassicNullCheckedEvent took: 674ms
OnPreInitializedEvent took:     2041ms
Subscribing another empty delegate to each event . . .
Executing 50000000 iterations . . .
OnNonThreadSafeEvent took:      2011ms
OnClassicNullCheckedEvent took: 2061ms
OnPreInitializedEvent took:     2246ms
Done

Though you probably needn’t worry about these until your performance testing turns up a bottleneck on event invocation. i.e. probably never. (Note that the test run is for 50 million iterations.)

(Code samples from this post are available as a VS2008 Solution.)

This is by far the most comprehensive and clear explanation of the rather in-depth field of type variance, and features lots of examples (and diagrams!) to help your understanding.

My favourite part of the article was this awesome fruit farmer metaphor:

 "This might go unnoticed if the farmer doesn’t enforce runtime fruit/vegetable type safety."
   — Bart De Smet 

Read the full article here:

C# 4.0 FEATURE FOCUS – PART 4 – CO- AND CONTRA-VARIANCE FOR GENERIC DELEGATE AND INTERFACE TYPES 

Here are the three main version-related assembly attributes:

// Assembly mscorlib, Version 2.0.0.0
[assembly: AssemblyFileVersion("2.0.50727.3521")]
[assembly: AssemblyInformationalVersion("2.0.50727.3521")]
[assembly: AssemblyVersion("2.0.0.0")]

By convention, the four parts of the version are referred to as the Major Version, Minor Version, Build, and Revision.

The AssemblyFileVersion is intended to uniquely identify a build of the individual assembly

Typically you’ll manually set the Major and Minor AssemblyFileVersion to reflect the version of the assembly, then increment the Build and/or Revision every time your build system compiles the assembly. The AssemblyFileVersion should allow you to uniquely identify a build of the assembly, so that you can use it as a starting point for debugging any problems.

On my current project we have the build server encode the changelist number from our source control repository into the Build and Revision parts of the AssemblyFileVersion. This allows us to map directly from an assembly to its source code, for any assembly generated by the build server (without having to use labels or branches in source control, or manually keeping any records of released versions).

This version number is stored in the Win32 version resource and can be seen when viewing the Windows Explorer property pages for the assembly.

The CLR does not care about nor examine the AssemblyFileVersion.

The AssemblyInformationalVersion is intended to represent the version of your entire product

The AssemblyInformationalVersion is intended to allow coherent versioning of the entire product, which may consist of many assemblies that are independently versioned, perhaps with differing versioning policies, and potentially developed by disparate teams.

“For example, version 2.0 of a product might contain several assemblies; one of these assemblies is marked as version 1.0 since it’s a new assembly that didn’t ship in version 1.0 of the same product. Typically, you set the major and minor parts of this version number to represent the public version of your product. Then you increment the build and revision parts each time you package a complete product with all its assemblies.”
        — Jeffrey Richter, CLR via C# (Second Edition) p. 57

The CLR does not care about nor examine the AssemblyInformationalVersion.

The AssemblyVersion is the only version the CLR cares about (but it cares about the entire AssemblyVersion)

The AssemblyVersion is used by the CLR to bind to strongly named assemblies. It is stored in the AssemblyDef manifest metadata table of the built assembly, and in the AssemblyRef table of any assembly that references it.

This is very important, because it means that when you reference a strongly named assembly, you are tightly bound to a specific AssemblyVersion of that assembly. The entire AssemblyVersion must be an exact match for the binding to succeed. For example, if you reference version 1.0.0.0 of a strongly named assembly at build-time, but only version 1.0.0.1 of that assembly is available at runtime, binding will fail! (You will then have to work around this using Assembly Binding Redirection.)

Confusion over whether the entire AssemblyVersion has to match. (Yes, it does.)

There is a little confusion around whether the entire AssemblyVersion has to be an exact match in order for an assembly to be loaded. Some people are under the false belief that only the Major and Minor parts of the AssemblyVersion have to match in order for binding to succeed. This is a sensible assumption, however it is ultimately incorrect (as of .NET 3.5), and it’s trivial to verify this for your version of the CLR. Just execute this sample code.

On my machine the second assembly load fails, and the last two lines of the fusion log make it perfectly clear why:

.NET Framework Version: 2.0.50727.3521
---
Attempting to load assembly: Rhino.Mocks, Version=3.5.0.1337, Culture=neutral, PublicKeyToken=0b3305902db7183f
Successfully loaded assembly: Rhino.Mocks, Version=3.5.0.1337, Culture=neutral, PublicKeyToken=0b3305902db7183f
---
Attempting to load assembly: Rhino.Mocks, Version=3.5.0.1336, Culture=neutral, PublicKeyToken=0b3305902db7183f
Assembly binding for  failed:
System.IO.FileLoadException: Could not load file or assembly 'Rhino.Mocks, Version=3.5.0.1336, Culture=neutral, 
PublicKeyToken=0b3305902db7183f' or one of its dependencies. The located assembly's manifest definition 
does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'Rhino.Mocks, Version=3.5.0.1336, Culture=neutral, PublicKeyToken=0b3305902db7183f'

=== Pre-bind state information ===
LOG: User = Phoenix\Dani
LOG: DisplayName = Rhino.Mocks, Version=3.5.0.1336, Culture=neutral, PublicKeyToken=0b3305902db7183f
 (Fully-specified)
LOG: Appbase = [...]
LOG: Initial PrivatePath = NULL
Calling assembly : AssemblyBinding, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: No application configuration file found.
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v2.0.50727\config\machine.config.
LOG: Post-policy reference: Rhino.Mocks, Version=3.5.0.1336, Culture=neutral, PublicKeyToken=0b3305902db7183f
LOG: Attempting download of new URL [...].
WRN: Comparing the assembly name resulted in the mismatch: Revision Number
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

I think the source of this confusion is probably because Microsoft originally intended to be a little more lenient on this strict matching of the full AssemblyVersion, by matching only on the Major and Minor version parts:

“When loading an assembly, the CLR will automatically find the latest installed servicing version that matches the major/minor version of the assembly being requested.”
        — Jeffrey Richter, CLR via C# (Second Edition) p. 56

This was the behaviour in Beta 1 of the 1.0 CLR, however this feature was removed before the 1.0 release, and hasn’t managed to re-surface in .NET 2.0:

“Note: I have just described how you should think of version numbers. Unfortunately, the CLR doesn’t treat version numbers this way. [In .NET 2.0], the CLR treats a version number as an opaque value, and if an assembly depends on version 1.2.3.4 of another assembly, the CLR tries to load version 1.2.3.4 only (unless a binding redirection is in place). However, Microsoft has plans to change the CLR’s loader in a future version so that it loads the latest build/revision for a given major/minor version of an assembly. For example, on a future version of the CLR, if the loader is trying to find version 1.2.3.4 of an assembly and version 1.2.5.0 exists, the loader with automatically pick up the latest servicing version. This will be a very welcome change to the CLR’s loader — I for one can’t wait.”
        — Jeffrey Richter, CLR via C# (Second Edition) p. 164 (Emphasis mine)

As this change still hasn’t been implemented, I think it’s safe to assume that Microsoft had back-tracked on this intent, and it is perhaps too late to change this now. I tried to search around the web to find out what happened with these plans, but I couldn’t find any answers. I still wanted to get to the bottom of it.

So I emailed Jeff Richter and asked him directly — I figured if anyone knew what happened, it would be him.

He replied  within 12 hours, on a Saturday morning no less, and clarified that the .NET 1.0 Beta 1 loader did implement this ‘automatic roll-forward’ mechanism of picking up the latest available Build and Revision of an assembly, but this behaviour was reverted before .NET 1.0 shipped. It was later intended to revive this but it didn’t make it in before the CLR 2.0 shipped. Then came Silverlight, which took priority for the CLR team, so this functionality got delayed further. In the meantime, most of the people who were around in the days of CLR 1.0 Beta 1 have since moved on, so it’s unlikely that this will see the light of day, despite all the hard work that had already been put into it.

The current behaviour, it seems, is here to stay.

It is also worth noting from my discussion with Jeff that AssemblyFileVersion was only added after the removal of the ‘automatic roll-forward’ mechanism — because after 1.0 Beta 1, any change to the AssemblyVersion was a breaking change for your customers, there was then nowhere to safely store your build number. AssemblyFileVersion is that safe haven, since it’s never automatically examined by the CLR. Maybe it’s clearer that way, having two separate version numbers, with separate meanings, rather than trying to make that separation between the Major/Minor (breaking) and the Build/Revision (non-breaking) parts of the AssemblyVersion.

The bottom line: Think carefully about when you change your AssemblyVersion

The moral is that if you’re shipping assemblies that other developers are going to be referencing, you need to be extremely careful about when you do (and don’t) change the AssemblyVersion of those assemblies. Any changes to the AssemblyVersion will mean that application developers will either have to re-compile against the new version (to update those AssemblyRef entries) or use assembly binding redirects to manually override the binding.

• Do not change the AssemblyVersion for a servicing release which is intended to be backwards compatible.
• Do change the AssemblyVersion for a release that you know has breaking changes.

Just take another look at the version attributes on mscorlib:

// Assembly mscorlib, Version 2.0.0.0
[assembly: AssemblyFileVersion("2.0.50727.3521")]
[assembly: AssemblyInformationalVersion("2.0.50727.3521")]
[assembly: AssemblyVersion("2.0.0.0")]

Note that it’s the AssemblyFileVersion that contains all the interesting servicing information (it’s the Revision part of this version that tells you what  Service Pack you’re on), meanwhile the AssemblyVersion is fixed at a boring old 2.0.0.0. Any change to the AssemblyVersion would force every .NET application referencing mscorlib.dll to re-compile against the new version!