Hi Folks,

 

Today I picked an issues from our bug tracking system that can kill your whole day. Although in the end it was really very interesting and you learned a lot of things, these are not my favourite kind of bugs I like to solve.

 

The description of the ticket was "Windows service consumes a lot memory over time." - Right, great description, but lets start with a brief overview of the project: We're running a lot of asynchronous tasks in our projects, especially in projects which make extensive use of web services to communicate with slow external services in the background. As we switched to WCF some months ago a lot of tasks became much easier. Some of these services are hosted inside IIS, others are windows services.

 

In a current project one of the external parties is a Java-based XML interface, which indeed makes use of SOAP for the transport of messages, but relies completely on "plain" XML inside the SOAP message. All the messages are based on a XSD but there are several "root level" messages that may appear in a request or response.

As I tried to deserialize the schema with WCF's internal utilities in the first place it stated that it was not able to, but I should instead use the xsd.exe, which creates code that must be used with the "older" XmlSerializer of .NET 2.0 instead of the "newer" DataContractSerializer (a.k.a. XmlFormatter) of .NET 3.0. So I did.

The generated .cs file looks like a typical file designed for use with XmlSerializer - several [XmlWhatever] Attributes and lots of classes with even more properties.

To create a WCF service that uses the XmlSerializer for serialization and de-serialization instead of the DataContractSerializer, one has to annotate the service interface with the attribute [XmlSerializerFormat] (and some additional parameters maybe).

 

At that point the problem was the following: I couldn't create a service interface that was strongly typed. Reason: The plain XML root node within the SOAP message could be one of several, but does not have any "real" type information in it. It was just named like the type that xsd.exe generated for me, certainly without the C# namespace. Because I couldn't get WCF to recognize the type correctly when I created a service interface that accepted and returned 'object', I decided to work with generics to not loose more time with the problem. So whenever I created a service client in code, I passed the request and the response type to the client and after that the WCF engine had enough information about the XML messages to (de-)serialize it. Worked great for the moment and I moved to the next part of the project.

 

But back to the issue: "Windows service consumes a lot memory over time."

 

I started by installing ProcessExplorer from Microsofts TechNet SysInternals website and took a look at the process. I was very suprised after discovering that the process had a huge "loader heap" and about 2500 assemblies loaded. So why on earth should a process have 2500 assemblies loaded? I also recognized that the windows service frequently created a subprocess which then called the C# compiler (csc.exe), so obviously there was some kind of code generation going on.

Because I worked with the XmlSerializer a lot in other projects, I knew that by calling XmlSerializer a specialized assembly was created for the type that was going to be (de-)serialized, but why should WCF create a new assembly every time a XML message was received or created?

 

After thinking about the problem for a couple of minutes I came to the conclusion that it must have something to do with the way our project uses WCF and that there may be a connection with the way we're using generics to create the "right kind" of client proxy at runtime. In fact it turned out that WCF calls the XmlSerializer with a special constrcutor and passes the ExtraTypes[] array as parameter which then results in a XmlSerializer object that can not be correctly cached by the framework.

 

So I decided to re-design the whole service library and to create my own "DTO aware" XmlSerializer. It should evaluate the right kind of internal object by taking a look at the root element of the incoming XML message, create the type by using reflection (or maybe later by using some kind of mapping for performance purposes) and of course use a cached version of XmlSerializer.

 

First of all let me say that WCF is a really a great example of software architecture. It's like LEGO - everything up and running in minutes but you can replace any pieace by your own code, configuration and extension.

 

To accomplish my goal I created my own implementation of IOperationBehavior and made it available to the my service library by creating it as a C# Attribute named [ProjectFormatter]. By annotating a [ServiceContract]'ed Interface with my [ProjectFormatter]-Attribute, the behavior replaces the default formatter with the one I created. This ProjectMessageFormatter implements IClientMessageFormatter and IDispatchMessageFormatter. There are only a few methods to override and some simple Singleton patterns and in the end I had my own project serializer that not only resolved my memory issue but also provided my team with a much more convenient way to use the WCF client.

 

In the end I have to say that I'am really happy having read so much about C#, the CLR, XML handling in .NET and the WCF in books, blogs, on MSDN and by watching so many webcasts - no wasted time at all. I really hope to help someone else with this blog entry to understand the details a little bit better and looking forward to resolve my next "level 3" issue. :-)