Loosely coupled - Part I

I’m working on a fairly large BizTalk implementation, which in many ways does not fit the classic BizTalk project pattern in that, whilst it has a large element of integration to it, it is even more a business process rich application.

One of the things that constantly keeps us busy is trying to strike the right balance between complexity/performance and maintenance, and a big part of that is finding just how loosely coupled the solution can and should be; this is something I find ourselves spending more and more time on.

There are two parts to this challenge for us – one is about ensuring our processes are not too tightly coupled with any of the many services we’re calling (internal or external) and the other is that different ‘areas’ of our solution, although implemented on BizTalk, and even in most cases within the same group, are treated as independent ‘services’ and are decoupled from each other; we’ve learnt the hard way the price of not getting good enough – mostly in terms of maintenance complexity and cost and are keen to find the right answers, if there are any.

In this part I would like to start tackling the first point – calling services -

In a recent post I ranted somewhat about the WCF’s adapter lack of support for multi-part messages, a topic I’m likely to come back to every now and then I suspect.

This was triggered by my attempt to prototype a way to call services from orchestrations in a loosely coupled way – ideally without the orchestration having to know about the service implementation (isn’t that the promise behind BizTalk Server?)

Naively, I started with a very simple plan – the orchestration would have its own representation for the service call, which it would publish to the message box; the send port, configured with the WCF adapter, would pick up the request based on some subscription, a map in the send port would convert the request to the service’s format and the service would be called; the response would be handled in a similar way.

I have documented that approach here

This worked pretty well in my little prototype, and I was rather pleased with myself for a short while, at least until Ben was tasked with taking this approach into our real world; it took very little time for it to break – all he had to do is call a real service - one that takes more than one parameter (how’s that for a lesson to not over simplify prototypes…)

To be fair, this needn’t be an issue, had we not used multipart messages in our processes, but – accustomed to using the SOAP adapter were using multi-part messages to represent the numerous parameters to the service, and – as I’ve hinted in my previous post – I do sincerely believe this to be the better approach.

That meant two things – 1. the WCF adapter would not be able to transmit the message, as it does not support multipart messages, and - 2. we would only be able to map the body part of the message.

These pretty much put a lid on my idea.

The WCF adapter’s lack of support for multi-part messages we could probably work around using a custom assembler or encoder – such a component could take a multipart message and convert it to a single part message using some pre-defined rules and/or part names, context properties, etc. but the map limitation really puts the lid on this approach.

What can we do then? well – we’d have to create a schema in our process that contains all the parts needed for the service calls, this does not have to look like the service contract, but it does have to contain all the information required to construct the service’s request and effectively means describing the multipart message in XSD; this makes me slightly uncomfortable as, in a sense, this schema exists specifically for this service; it also means we would have to work harder in the process to construct this message – on top of creating and deploying another schema, we would also need some map to convert the multiple messages containing the information to the single entity (or use some code to achieve the same), but it would work – once we have the single part message, we could publish it, map it to the service’s format in the port and deliver it to the service, in the exact same way that worked in my prototype.

At the moment, this looks to me as our only bet, but I’m looking for alternatives, if anyone has any suggestions?

A transform shape gotcha Update

A while ago I blogged about this problem we've found with the transform shape; what I should have done, and haven't (I believe) is blog about the fix Microsoft issued as a result of our bug report.

You can read the details here

Interestingly I think the scope of the problem was bigger than we originally thought, certainly bigger that what I blogged about anyway, in that even if you don't have the tranform shape problem (for exmample - if you've created the orchestration on a machine with the hotfix, but then opened it in a machine without it) you can have some issues -

In one case we had a map, where the input was composed of several schemas; as you probably know BizTalk effectively creates a schema that aggregate all the parts into a single schema; to achieve this it had to impot several schemas from our solution, all share the same namespace;

When we tried to build the project with the map, we received errors saying a certain element type could not be found, the cause was that BizTalk was looking for it in the wrong schema, as it had several schemas imported into the same namespace, it chose the wonrg one and could not find the type definition.

Once we've installed the fix, and with no other changes, the map compiled all right.

BizTalk WCF adapter and multipart messages

I wrote myself notes for this post months ago, but haven’t taken the time to write it down properly; a passionate discussion today with Rupert Benbrook prompted me to get back to this.

The WCF adapter does not support multi-part messages, and some would argue that’s a suggestion of the feature possibly being dropped in some future version of the product.

This small fact seems to have been glossed over by most – I, for one, haven’t heard this discussed anywhere, to the point that I’m left wondering if I’m really just “stuck in my ways”, but I honestly think that’s a poor decision.

Trying to search for some detail about this fact I found the following, rather old, but still very valid and informative, post; here are some of the key points made -

The reason support for multi-part message in WCF adapter is not being recommended is because WCF does not support it and the model proposed is that whoever wants to use something equivalent, need to aggregate the messages into one xml and send it as one single-part message.

(Tapas Nayak, Principal SDE, BizTalk Adapter Pack)

Multipart messages are a marginalized feature in BizTalk, the pipeline components like the flat-file and xml don’t understand them, same story for the majority of the adapters. So we decided to not support them in the WCF Adapter too

(John Taylor, Senior Technical Lead, BizTalk Server)

 

With regards to the first statement – I think that is somewhat down to interpretation – WCF did not change how web services work; WSDL is the same WSDL, specifications have not really changed; so – WCF is just a new (and of course – much better) implementation; this is an important basis for any discussion because it indicates that the WCF adapter could have taken the same path the SOAP adapter did.

Consider the following Service definition in WCF:

        [OperationContract]
        CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut);

or the equivalent web method in asmx:

        [WebMethod]
        public CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut)
        {
        }

the WSDL for both methods is pretty much identical, the operation definition looks as follows -

<wsdl:operation name="GetDataUsingDataContract">

<wsdl:input wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContract" message="tns:IService1_GetDataUsingDataContract_InputMessage"/>


<wsdl:output wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContractResponse" message="tns:IService1_GetDataUsingDataContract_OutputMessage"/>


</wsdl:operation>

the messages are defined as so -

<wsdl:message name="IService1_GetDataUsingDataContract_InputMessage">

<wsdl:part name="parameters" element="tns:GetDataUsingDataContract"/>


</wsdl:message>


<wsdl:message name="IService1_GetDataUsingDataContract_OutputMessage">


<wsdl:part name="parameters" element="tns:GetDataUsingDataContractResponse"/>


</wsdl:message>

And the schemas, in both cases, define the aggregated messages – with all the in/out parameters as required.

So – what is the difference? the way I see it is it is in the way the different adapters handle the messages as they pass through the adapter; the SOAP adapter has a layer in which, when receiving messages from the outside,  breaks them to individual parts, delivering a multi-part message to the message box; similarly, when delivering messages from BizTalk to the service, it composes them, aggregating the various parts in the request message, to a single SOAP envelope; the WCF adapter does not include such functionality – it will take the message BizTalk provides and will wrap it with the SOAP envelope pretty much as is.

All my grievances about the SOAP adapter aside – I think its approach is a good one; it allowed the process to focus about the flow of the business process and avoided the need to define and construct artificial entities in the form of aggregated schemas, which often make very little sense (as the various parts can be largely unrelated from a process perspective, but still necessary for the service to perform its operation).

In my discussion with Rupert the same point was raised – the fact that SOAP supports one message in and one message out (i.e. no multipart messages), but in my view that’s a problematic argument – again – the whole point of the BizTalk architecture is that the send port can mask the transport mechanisms; ok – the SOAP adapter is not doing a particularly good job at that, but that’s hardly an excuse to make matters worse – I don’t want to have to know that I need an aggregated message to call this particular message because it uses the WCF adapter;

Moving on - I particularly disagree with the second statement mentioned - I think multipart messages are a very important feature in BizTalk, and would be surprised if any seasoned BizTalk developer would tell me they prefer not to use them (do let me know if you think I’m wrong); to start with – it allows one to define a type once and then use it multiple times; this makes maintenance infinitely better when types need to be changed (I thought it ought to be this schema, but found that this other schema is better); ok – so you have to be careful about how and where you re-use, to not introduce unwanted dependencies, but if you have large orchestrations which follow several steps,  this is very useful.

Secondly, it allows one to easily link pieces of information that are only ‘somewhat-related’, especially important, in my view, when using pub/sub; having to pass several pieces of information from the publisher to the subscribers; the alternative, not surprisingly, would be exactly what is suggested above – create a schema defining the various pieces of information required and use that instead, but I think that’s a much larger maintenance overhead than a multi-part message definition; it is much easier to change a message definition that it is to change a schema; especially in large implementation with many different assemblies;

This was the point most passionately discussed with Rupert today – he argued that if a message is important enough to be published (with all its parts) it is important enough to create a schema for it; the point I was making, possibly not good enough, is that – to me, specifically in this area, the two are the same – I can define my aggregated message in a schema, that would have, say – 2 elements, each using a type defined in my other, existing schemas, or I could use a multi-part message, with two parts, each pointing at existing schemas; there’s very little difference in principal – I’m defining something – strongly typed and well defined,  that is composed of two other, well known things, and I’m passing it around.

Obviously interoperability can only be achieved using standards like XSD, but that’s not a consideration at this point I’m publishing a message from BizTalk, which would be subscribed to by BizTalk, what’s wrong with describing it using BizTalk terms where conceptually there’s virtually no difference?

If there is no difference – why do I care? because multi-part messages are easier to create than aggregated schemas, and there’s less deployed artifact; I can simply put an assign shape and assign existing messages to the relevant parts; if I was to go down the aggregated schema route I would have to define the schema and create a map that would aggregate the messages; I could use code, or I could use xpath to assign the parts to the various elements, but I’d have to use a map or something to create the shell of the message first; seems like a lot of overhead for simply passing a couple of messages between processes.

I agree that many adapters don’t support multipart messages (the file and ftp adapters are the most prominent examples, outside the WCF adapter), and in many cases this makes perfect sense, but some do – SMTP and POP adapters, for example, use it very well; so does the SOAP adapter, of course.

There was one piece where we were in mostly in consensus, about the benefits of a single schema versus multi-part messages – when you wish to have  a map in the port, I believe BizTalk would only consider the body part of the message, but again – to me that’s a shortfall in the product, not a proof that the existing feature is not useful; for both this and the pub/sub scenario I would have lover to be able to treat a multi-part message type in the same way a single-part message is treated – with a specific type that you can subscribe on (I can always set the message type manually, I guess, but would have been great to get out of the box support)

Similarly I agree that the built in xml and FF disassemblers don’t support multipart messages ( only in the sense that they will not disassemble them), but the model does – you can easily access, and parse, multipart messages from within a pipeline component, even a custom disassembler, so I can’t help wondering if the answer is in the following statement made by John Taylor, as quoted in that blog post -

The feature set for the WCF Adapter was potentially very large indeed and we were under some pressure to make cuts where we could.

and I think it’s a big shame.

I, for one, would be very disappointed if multi-part messages stop being supported, and I’m slightly worried – WCF is obviously taking more and more front stage, generally and with BizTalk specifically (even more so with the LOB adapter SDK and the BizTalk adapter pack that makes use of it), and as it does not support multi-part messages, it may well set the tone for future BizTalk developments.