Julius Ganns . netzkern

A couple of days ago, I needed a minor customization in my Outlook 2007. Being a developer, I just opened the integrated development environment in Office and wrote some quick VBA code. I would have done it with C# and VisualStudio, but to be honest, every time I start to work with Office components, especially Outlook, I struggle with the unnecessary complexity and the low quality COM-based API. While I was coding, I began to think about why there seemed to be virtually no effort at Microsoft to rewrite and redesign important applications like Outlook in the near future. It virtually made no sense to somehow abandon a successful product like that.

A couple of hours later, I checked my RSS reader and found an interesting article about the upcoming Exchange 2010 (or "E14" for short). Suprinsingly, when I took a look at the screenshots, they answered my earlier question.

Microsoft has not only started to rewrite its most important applications, in fact, they have completely redesigned every single one of them. Following the vision of its Chief Software Architect Ray Ozzy, Microsoft has finally understand the possibilities and the importance of the web. The new Microsoft Exchange 2010 has an Outlook interface that should be able to really compete with the “old” desktop client and as we all know, this is only part of a complete Microsoft Office application suite running within the browser.

When we started to see the web as platform for our business back in 2004 when I took over as Chief Software Architect at netzkern and our company began to focus all efforts on web and service-oriented technologies, there were a lot of people who disagreed with me on that vision. Up until today, history has proven us right and we’re looking forward what the web will bring to us in the next 5 years.

When Web UI Component Frameworks like ASP.NET WebForms and JSF were invented in the early years of this decade, they were an honest attempt to bring an additional layer of abstraction to the web. This layer was supposed to improve a couple of things, especially in the areas of usability, developer productivity, extensibility and out-of-the-box features. They were also created to attract the old guard of desktop developers, that spend so many years to learn and master their craft.

To accomplish that mission, these frameworks were composed of hundreds of invisible components to form a feature-rich environment based on abstraction and simulation. This environment targeted one specific thing: hiding away as much of the original web infrastructure as possible. To accomplish that goal, Microsoft, Sun and others created complex workarounds to overcome some of the “serious limitations” of the web. Those features were given fancy names like Postback, ViewState and CodeBehind, which we all know and have to live with every day.

The problem with these techniques is that they are not only an additional layer of abstraction, they are actually based on some quite different paradigms. At some point that inevitably creates major problems if you want to create “real” web applications and you find yourself searching for solutions to problems that you might not even have if you hadn’t started to use those frameworks in the first place.

When some developers began to realize that these heavy-weight component-oriented frameworks started to decrease their productivity, they did a very clever thing. Instead of reverting completely back to old frameworks like the original Struts or ASP, they took all the great things that the Web UI Component Frameworks had brought to light, merged them with those frameworks and included new techniques and approaches that emerged over the last few years. The result of these efforts are frameworks like ASP.NET MVC, Ruby on Rails, Grails and Spring MVC that handle requests on the server-side much more naturally and effectively, and frameworks like JQuery, ASP.NET AJAX, Silverlight, Flex and Prototype, that handle user interaction on the client-side.

The way of web development has changed and as I wrote in my post about this topic about a year ago, it’s time to rethink now. Don’t miss this one, it is a big one.

“The largest thing on earth is the human ego.” is one of that statements that remain true no matter what – probably today more than ever. In the beginning of this year, I was quite active on various social networking sites like Twitter and Facebook. In the beginning, it was really a lot of fun, seeing all the things my friends were doing and what was going on around me. Unfortunately, that feeling has changed. When I open Facebook these days, the only feeling I get is disgust about the grandstanding and ego-casting that is happening there way to much. It is the same thing that happens if people get to much attention in the real world: They get supercilious, start to actually need all the attention and feel obligated to attract all the approval of the other people around them.

The sad thing about that development is that Twitter and Facebook could really bring value to our world. They could help to stay in contact, to bring people closer together over large distances and they can help this world to overcome some of the biggest problems we face today: Biased media, censorship and ignorance.

The good thing is: Publicity is one of the most volatile things there are, and therefore, I expect those self-exposing images to be gone a certain while from now. And if we get lucky, someone will fill that void with sustainability and balance by using those great ideas for something useful.

ADO.NET Data Services is Microsoft’s new services framework for accessing data from remote clients using HTTP and other disconnected protocols by sending either LINQ or URI’s queries to a specialized DataService<T>. DataService<T> is connected to the underlying data source and can perform read, update, delete and create operations against any data framework with a sufficient data context implementation, for example LINQ-to-SQL or the ADO.NET Entity Framework.

While it is possible to create components that intercept the data operations at the service level, ADO.NET Data Services has been specifically designed to support CRUD operations performed by remote clients. That means, for the time being, there is no support for accessing and invoking business logic components over DataService<T> or for returning custom response messages based on data structures that are not managed by the underlying data context. That in turn means that business logic has to reside on the client (maybe even written in JavaScript) and can easily be manipulated. It also means that arbitrary clients can run all operations against every exposed interfaces of DataService<T> (which is a matter of configuration).

WCF and the DataTransferObjectManager on the other hand allow you to selectively open your API to external clients and provide a consistent interface for business operations beyond create, read, update and delete scenarios, regardless of the underlying data structures and without any affiliation to a specific data framework. They also allow you to invoke service operations that access repositories and business logic components while still returning only the necessary parts of large object graphs to the client.

ADO.NET Data Services (and the upcoming .NET RIA Services) are great frameworks with a lot of potential value, but for the time being they do not solve the problem of accessing any kind of API call through service operations without having to worry about large and deeply interconnected object graphs. This is where DTOM and the ServiceToolkit.NET step in, which we will release in a couple of days on CodePlex.

In a previous post we took a look at an early preview version of the DataTransferObjectManager, a library we developed at netzkern to integrate WCF with O/RM tools. Today we will take a look at how nk.DTOM solves the problem of mapping updates from clients to entity objects. Here is how to do this:

// This object represents an update request from an external client.
HouseDTO updateHouseParam = new HouseDTO();
updateHouseParam.ID = “6”;
updateHouseParam.Name = “176 Wallstreet”;
updateHouseParam.Price = “10 Million Dollar”;

// Read the corresponding entity from the data source.
House houseEntity = houseRepository.GetHouse(updateHouseParam.ID);

// Map the properties of the DTO to the entity.
DataTransferObjectManager.Map(updateHouseParam, houseEntity);

// Update the data source.
houseRepository.Save(houseEntity);

Pretty straight forward, right? Remember, by default nk.DTOM only maps value type properties from the source object to the destination object, and that’s exactly what we need here.

Now, let’s consider our entity “houseEntity” contains a property that should be readonly for external clients. In our case, the price of our house is fix and we don’t want it to be changed. In this case, we use the support for selective updates in nk.DTOM. The “Map”-Method contains a fourth parameter that allows us to specify certain properties that we want to preserve in the destination object.

// Create an empty PropertyPathCollection as third parameter, as we do not want to include any references.
PropertyPathCollection includeSourceReferenceProperties = new PropertyPathCollection();

// Create a PropertyPathCollection containing all the PropertyPaths we want nk.DTOM to skip during the mapping, which means preserving the original value.
PropertyPathCollection skipDestinationProperties = new PropertyPathCollection();
skipDestinationProperties.Add(“Price”);

// Call Map() with all four parameters.
DataTransferObjectManager.Map(updateHouseParam, houseEntity, includeSourceReferenceProperties, skipDestinationProperties);

By combining the third and the fourth parameter, nk.DTOM even enables us to update certain values of complete graphs of entity objects. We just need to preserve the original reference to the entity object and maybe some other properties. Take a look:

// Update a referenced DTO on the client-side.
updateHouseParam.Owner.Name = “Some new guy”;

// Create a PropertyPathCollection on the server-side that allows the client to update the referenced entity “Person” in the property “Owner” as well.
PropertyPathCollection includeSourceReferenceProperties = new PropertyPathCollection();
includeSourceReferenceProperties.Add(“Owner”);

// Create a PropertyPathCollection containing all the PropertyPaths we want nk.DTOM to skip, in this case especially the property “Owner” (as the client is not allowed to change the Owner itself and we don’t want nk.DTOM to initialize the property with a new instance) the Owner’s ID.
PropertyPathCollection skipDestinationProperties = new PropertyPathCollection();
skipDestinationProperties.Add(“Price”);
skipDestinationProperties.Add(“Owner.ID”);

// Call Map().
DataTransferObjectManager.Map(updateHouseParam, houseEntity, includeSourceReferenceProperties, skipDestinationProperties);

The service developer can even decide which properties can be changed based on the identity of the caller, for example by relying on WCF authentication features.

We’re currently planning to release netzkern.DataTransferObjectManager as part of a small toolkit on CodePlex. We hope, you’re going to use and like it.

I love reading books. Usually that means one, sometimes two or three books per week. In the last four years I extended my personal library to several self-development and management books and I have to say, I’m glad that I read most of them. Below you can find my personal top ten list in those two categories:

Getting Things Done
The 7 Habits of Highly Effective People
Brain Rules
Slowing Down to the Speed of Life
How To Be Brilliant
Behind Closed Doors
Good to Great
Speed Reading for Professionals
First Things First
Mind Performance Hacks

Additionally I own several other books that I haven’t read yet but that might make it to this list, maybe even replacing one or two of the ones listed above. Here are those I’m eager to read:

Think And Grow Rich
How To Win Friends And Influence Others
Your Money or Your Life
The Power Of Less

For a complete list of all my books, please visit my website at www.juliusganns.com.

(Version 1.1 – March 20th, 2009)

Developing with today’s platforms, a developer can utilize great tools and libraries to increase development speed and software quality. Two very common in the .NET space are WCF and LINQ to SQL, besides various other O/RM tools like NHibernate and the ADO.NET Entity Framework.

Unfortunately, using a messaging framework like WCF that has to perform quite a lot serialization work in combination with a tool to automatically retrieve data from data sources and create deeply nested object graphs like LINQ to SQL creates some pretty hard challenges. The reason for this is that WCF has to transform the graph into a “flat” message representation. If the graph contains cyclic references, WCF has two options to avoid running in circles (and kill your App) when it encounters an object it already has processed:

1. Abort the whole serialization. This is the default behavior and it is very reasonable to throw an exception here.
2. “Mark” that part of the message as “reference” to an earlier serialized part. This behavior can be introduced using your own WCF components (see the links at the bottom).

For service developers that want to use object-relational mapping tools to automate database queries, neither of one is really sufficient. Although the latter part may be an option in a strict .NET to .NET scenario, it does not implement any standard (because there is none) and thereby makes it nearly impossible for other platforms to use the serialized message with reasonable effort.

Even worse, in the Web 2.0 area more and more WCF services create JSON and XML output for client-side components like ASP.NET AJAX, JQuery, Silverlight and Flex. Therefore it is essential to provide client-side developers with all necessary information in a strctured way without the need to blow up the server side development.

Taking a look at the .NET framework, Microsoft currently does not offer any solution for this in its primary frameworks WCF and LINQ.

After re-designing and overworking our Reference Architecture over the last week, we finally managed to develop an approach here at netzkern to integrate both technologies with each other in a pragmatic and pretty “clean” way. We created a utility called the DataTransferObjectManager that is able to dynamically fill DTOs with the values and copies of all requested references. The trick here is that it does only selectively chooses the right references by letting the client-side developer provide a configuration parameter that indicates which properties he really needs to do his work. All other references are ignored and not returned to the client, they are usually not even loaded from the database.

Let’s have a look. First, we create two entity classes as part of our model. They don’t need to be public as they are never used outside of the core assembly.

namespace App.Core
{
    class House
    {
        public string Name { get; set; }
        public Person Owner { get; set; }
    }

    class Person
    {
        public string Name { get; set; }
        public House Residence { get; set; }
    }
}

Let’s pretend that instances of these classes are being tracked by an O/RM manager component that automatically detects changes and performs updates. Now, let’s create the corresponding DTO’s:

namespace App.DTO
{
    public class HouseDTO
    {
        public string Name { get; set; }
        public PersonDTO Owner { get; set; }
    }

    public class PersonDTO
    {
        public string Name { get; set; }
        public HouseDTO Residence { get; set; }
    }
}

These classes are basically message definitions. In WCF you would use these as DataContracts and annotate them with the appropriate attributes. Instances of these classes are somewhat “dumb”, they only contain data.

Now let’s consider, we have the following object graph:

House h1 = new House();
h1.Name = “House No. 1”;
Person p1 = new Person();
p1.Name = “Person A”;
h1.Owner = p1;
p1.Residence = h1;

This creates a graph with just two objects and a cyclic reference. When you try to serialize “h1” using WCF’s default DataContractSerializer, you get the above mentioned exception. Now let’s use the netzkern.DataTransferObjectManager (or nk.DTOM for short) and see what it can do for us:

// Create an empty base DTO.
HouseDTO h2 = new HouseDTO();

// Call the nk.DTOM and copy the contents of h1 to h2.
DataTransferObjectManager.CreateDataTransferObject(h1, h2);

What we get from this is the following “filled” DTO message object:
h2.Name: “House No. 1”;
h2.Owner: null;

So far we created a flat copy of “h1” with all references excluded. Now let’s include a copy of “h1.Owner” as part of “h2”:

// Create an empty base DTO.
HouseDTO h2 = new HouseDTO();

// Tell the nk.DTOM to include the following properties.
PropertyPathCollection includePaths = new PropertyPathCollection();
includePaths.Add(“Owner”);

// Call the nk.DTOM and copy the contents of h1 to h2.
DataTransferObjectManager.CreateDataTransferObject(h1, h2, includePaths);

The result now includes a copy of “p1” that has been mapped to a PersonDTO type:
h2.Name: “House No. 1”;
h2.Owner: [PersonDTO];
h2.Owner.Name: “Person A”;
h2.Owner.Residence: null;

We can now even include “deeper” properties, that are being copied equally by the nk.DTOM:

// Create an empty base DTO.
HouseDTO h2 = new HouseDTO();

// Tell the nk.DTOM to include the following properties.
PropertyPathCollection includePaths = new PropertyPathCollection();
includePaths.Add(“Owner.Residence”);

// Call the nk.DTOM and copy the contents of h1 to h2.
DataTransferObjectManager.CreateDataTransferObject(h1, h2, includePaths);

That creates the following result:
h2.Name: “House No. 1”;
h2.Owner: [PersonDTO];
h2.Owner.Name: “Person A”;
h2.Owner.Residence: [HouseDTO];
h2.Owner.Residence.Name: “House No. 1”;
h2.Owner.Residence.Owner: null;

So, how do we use this in WCF? After you have successfully loaded entities in your service operation using an O/RM, you just call the nk.DTOM and return the DataTransferObject as result. The trick is to let the client(!) choose which parts of your internal object graph are necessary to do its work. A simple service call might look like this:

IHouseService service = new HouseServiceClient();
HouseRequest req = new HouseRequest();
req.HouseID = houseID;
req.IncludeProperties = “Owner.Residence”;
HouseResponse resp = service.GetHouse(req);

That’s basically it. By using the netzkern.DataTransferObjectManager, you can define your data transfer objects independently of your internal entity objects. You can even omit certain properties and nk.DTOM will ignore them. We will now test this component internally and make it available on CodePlex as soon as it reaches BETA status. We also would love to hear your thoughts, just send a mail to blog@juliusganns.com.

Update
I’m currently writing an article for a leading German .NET magazine about the above mentioned topic. The article will contain more details about the architecture and more samples how to use the library. I will post an update here when it has been published in a couple of weeks.

Update
Microsoft has just announced the .NET RIA Services framework which also targets parts of the above mentioned problems, but solves them in a different way. As far as I understand the library, it may be possible to combine both approaches. When more bits and bytes come available, I will definitely try to integrate those with our framework.