Why Integrate Dynamics CRM with BPM Tools?

I came across this article a few days ago about integration between Dynamics CRM and Business Process Management (BPM) tools. As much as I believe there is a case to be made on integration between CRM and BPM tools, I felt like the examples/scenario presented in that article weren’t strong enough.

For example, if we take the scenario were “once a lead record has been created, it is assigned to the appropriate owner on the sales team, and that person can take the necessary steps to nurture the lead towards an opportunity“, it is simple enough to be handled directly in Dynamics CRM. Same for a case management solution, when a “case has been created in Dynamics CRM with all of its details,” how do we “ensure that the proper support team member receives it, handles it according to defined steps, possibly escalates it, and ensures that it is completed accurately in an optimal manner?“. From these simple lines, depending on the specs, Dynamics CRM offers enough with its Windows Workflow Foundation based workflow engine. In the CRM integration context, the different BPM products out there are essentially about managing workflows with a third party software that does what CRM Workflows (WF) can do and more.

So why would one use a BPM tool when CRM already offers some workflow capabilities? Here are a few reasons that I think are key to help deciders make a sound decision on whether to invest in a BPM tool to integrate with CRM or not…

Workflow Complexity

When building very complex and long workflows, the Microsoft Dynamics CRM workflow editor is very limited and it can become very frustrating and reduce productivity:

  • The user interface is sometimes slow to respond (especially when there is a lot of custom code)
  • You can only go (I think) 5 or 6 levels deep and create child WF if you want to go deeper
  • It’s not easy to read at all

BPM tools often provide a better workflow design experience by using visual editors, either custom built or on top of other applications (e.g. MS Visio, MS Visual Studio etc.). That makes the workflows easier to build and to read. It also takes away the “levels” issue, removing the need to create child workflows when you have more than 5 deepness levels.

Workflow Systems Integration/Complex Workflow Operations

If you need your workflows to talk to other systems (web service calls, external connectors etc.) or just to perform some more complicated operations (retrieve multiple, bulk delete/update etc.), it’s all possible in CRM by creating custom workflow activities (.NET Code). These would have the logic within them to make the necessary web services/external calls/complex operations. Some BPM tools provide with the ability to integrate web services calls or make complex operations with simple configuration (no code). A few good reasons to use a BPM tools in this context would be:

  • You don’t have the Dynamics CRM expertise in house to build the custom pieces (custom workflow activities)
  • You have so many different external calls/integration to do within the workflow that it’ll create too much custom code resulting in more difficult debugging and costly maintenance in the future

Workflow Version Management

Depending on what the WF does, you may need to keep a detailed history of WF execution, including which version of WFs have been triggered for specific records. Also what CRM is lacking is a workflow “live update” feature. Say you trigger a WF on record A and it gets to some wait mode. At which point you update the WF in the CRM solution. Well, the version of the WF that is executing against record A is still the “old version” of the WF and you cannot change that. Some BPM tools have this feature built in.

Workflow Looping Mechanism

CRM Workflows don’t really allow looping. You can create a mechanism where a WF keeps calling itself as a child but that has a very negative impact on performance and eventually, CRM kills the loop thinking it’s an infinite loop (managing resource consumption). BPM tools have the capability to use loops sometimes very easily and with a much lover impact on system performance. Something to think about if you have plenty of WF in waiting and looping mode in your system at the same time.

These are a few elements that can help you decide if you want and/or need to use a BPM tool with your CRM. From my experience, depending on what you are building, BPM add-ons can be costly so before going down that road, it is important to assess the need and selected the right solution based on these needs. On the flip side, if what you really need is  a BPM tool to manage your workflow/processes, do you really need CRM? Food for thoughts… There are plenty of options out there, including AgileXRM, K2, PNMSoft and many more. They all come with different features so just checking them out is an interesting exercise!

Cheers!

Advertisements

Making Cross Domain AJAX calls from CRM Form JScript

One of the cool things about JScript in CRM 2011 is that you can now use the jQuery $.ajax method to call, most notably the CRM REST endpoint. Now, one of the things that is sometimes required in customers’ solution is the ability to call other REST endpoints from a CRM form. An example of that use case would be to have a button that calls a web service in order to perform some sort of operation (e.g. modify your current record, or just send data over another systems for synchronization etc.) in a synchronous and interactive way. Another example would be to automatically fill fields on a form with information coming from another system (retrieve them via web service and set the values on the form). There are two main issues with doing that :

  • First, some browsers will simply not allow jQuery AJAX requests to run if they are made to a domain different than that of the running web application. The reason behind is that not all browsers can create XMLHttpRequest objects that have a withCredentials property. That is used to specify that we want to include the user credentials in a cross domain request. To always have that working, jQuery has the solution for us. Use the following statement in the code before making the AJAX call: jQuery.support.cors = true;
  • The other more complicated issue is that most browsers simply won’t allow JavaScript to make a cross domain request for security reasons. In most contexts, this sounds normal and reasonable.

Here are a few workarounds if you are facing the second issue.

  1. Try to transfer your logic to the server side. Can you do what you need by using a dialog or a workflow with a custom activity (server side code)? Can your logic be performed on a synchronous plugin (server side code)? If one of these solutions is acceptable, then this is what you should do.Applies to: CRM Online, On Premise, ADFS / IFD
  2. Use IIS URL Rewrite rules on your server. To illustrate, let’s say your CRM server URL is http://MyCrmServer/MyOrg and your external REST endpoint is http://RestEndpoint/RestService?params. You can create a URL rewrite rule that will transform all requests coming in as http://MyCrmServer/RESTENDPOINT/RestService?params to http://RestEndpoint/RestService?params. In other words, whenever IIS sees the string MyCrmServer/RESTENDPOINT in a URL it replaces it with RestEndpoint. It is simple to configure with Regular Expressions.

    This is a good solution because it handles the request on the server side. From a Jscript perspective, it is as if you are querying the CRM server and the request is transformed once it reaches the server. You can read more about IIS URL Rewrite here. The downside of this approach is that every single page that is opened from the CRM web app triggers a URL rewrite rule evaluation. Microsoft says the algorithm are optimized for performance and I have never experienced visible delays due to using the rewriter. Also worth noting that obviously, this is not supported for CRM Online since we don’t have access to Microsoft’s CRM Servers.

    Applies to: CRM On Premise, ADFS / IFD (if provider gives access to CRM Server)

  3. Enable Cross Domain browsing on the browser. In IE, this is blocked by default. You can change the settings by going to Internet Options Security and Custom Level Settings. In most case, this solution is not acceptable for an enterprise’s perspective for security reasons.


    Applies to: CRM On Premise, ADFS / IFD, (CRM Online? Not confirmed but it should work)

I personally think making requests to a cross domain web site via JavaScript should not be done if there are acceptable other options simply because it adds overhead somewhere based on the solution you go with. (e.g. if you go with solution 2 on premises, you have to make sure all your CRM servers have rewrite rules create and enable etc.).

[*** Update | 13/06/2014 ***] If you are using CRM 2013, see this article on how to use Action processes to get around this challenge.

Hope this helps!

Microsoft Dynamics CRM Data Archiving Solutions

As Dynamics CRM consultants, one of the questions that we often get from clients is “how do we archive CRM data when it’s no longer needed”? I’ve seen companies that had to keep all of their data for a certain amount of years for legal reasons. In any case, if you are using Dynamics CRM as a service management tool for a busy call center for example, chances are you are creating a LOT of records at a very fast pace. Imagine having to retain all of the data you created in the past 5 to 10 years on the same system. The size of the database can grow to become extremely large, making every single query a long one and impacting directly the users experience and their trust in the system.

Before I go into details about the available options that I can talk to, here is a quick recap of what Microsoft officially said a while ago about archiving Dynamics CRM data.

  1. Archiving implies moving records from one location, storing those records for possible retrieval in another location, and deleting the archived records from the original location.” Given the complexity of the relationships in CRM, archiving as defined in the previous sentence can be very risky and must imply extensive thought, design and knowledge of the CRM application and database.
  2. An arching solution should not be implemented with the goal of improving system performance.
  3. Instead of deleting records, it is recommended to deactivate them when they are no longer required.
  4. The Dynamics CRM application can support very large databases, it is important to make sure that the hardware and database are configured in order to get optimal performance before looking at Archiving.

Here are some possible approaches that can be used to archive your Microsoft Dynamics CRM data. Before going down that path, make sure this is something you really need, don’t just build an archiving solution because you want to improve your system performance. Do your homework first!

Archiving Method How it work Pros Cons
1 – Create a copy of the database It requires a strong DBA with knowledge of the CRM database to make a copy of the DB, copy all the records that need to be archived to the new DB and finally delete them from their original location. Sounds easy for a DB expert I assume (I’m not!)
  • It’s one of the less expensive solution as it only requires a strong DB developer with knowledge of CRM and an available SQL Server database
  • Not supported nor recommended by Microsoft
  • A DBA with knowledge of the Dynamics CRM database is always required to retrieve data from the archive database
2 – Create a new instance of Dynamics CRM
  • Create new organization or new CRM installation and install the same solution as the one in production
  • Create a mechanism to copy records from the production environment to the new archiving organization and delete them from their original location. This can be done with :
    • A custom .NET application using the CRM SDK
    • Scribe, Microsoft’s CRM 2011 Instance Adapter or other third party providers routines
    • Database scripting (unsupported)
  • The archived records are easy to retrieve (same UI as prod)
  • Access to these records can be controlled by the Dynamics CRM security model
  • Can be done in a supported way
  • Expensive solution that may requires additional hardware, server and/or user licenses for CRM or SQL Server
  • Company ends up maintaining more than one CRM organizations in the long run
3 – Deactivated unwanted records and tweak database Indexes
This is easily the number 1 choice from Microsoft’s perspective. To me, this should be one of the drivers in the CRM application design. The lifecycle of all records that are created (except for master data) should always be to:

  • create record (with active state obviously)
  • do work
  • deactivate record after the job is complete (set state to inactive)

In many systems that I have seen, some records just remain active forever, even though they are old and on longer required in the active records view.

Also, it is important to create SQL indexes to enable a faster retrieval of active records.

 

  • This method does not involve the deleting on any record from the CRM database which is what MS recommends
  • It forces the application to be designed with best practice for records lifecycle management
I could write down that the down side is that records remain in the CRM database but again, you are not archiving simply because you want to reduce the size of your database…
4 – Create and store custom archive reports This is an interesting one. I’ve personally recommended using this solution to capture the history of successful workflows execution prior to deleting them from the system.
The idea is to create custom reports containing information from the records that are no longer needed in the system. Once the reports have been created, they can be stored in a document management system or sent by email. After that, then the records can be deleted from the CRM database.
There are many ways to create reports in CRM so I won’t spend time discussion that topic here. A simple Google (or should I say Bing searchJ) will give you plenty of starting points.
  • Lots of options for reports creation and execution (3rd party software, SSRS, scheduling jobs, custom workflow activities etc.)
  • Can be done in a supported way
  • What used to be CRM records are now lines of text in different reports
  • Records must be saved in a managed location (example: doc management system with redundancy instead of external hard drive for the system admin…)
  • Access to the reports must be controlled
  • Ability to search for archived information can be limited and/or very painful in flat files
  • Building the reports can be expensive :
    • Report design can take time
    • Development cost depends on the tool used (SSRS, 3rd party…)
5 – Create a custom Archive Database
This is very similar to option 2. The difference would be instead of creating a new instance of CRM, you can create a custom database where you would only keep the information that you need. This database would be populated by a custom .NET app, third party data migration tool routine or database scripting.
Optionally, a custom UI can be developed to enable administrators to search through the custom database.
  • Tailored archiving solution, enables for more granularity and focus on retaining only the information that is required
  • If done well and if CRM solution(s) doesn’t evolve too much over time, it’s a one effort and maintenance should be easier than maintaining 2 CRM instances
  • Can be done in a supported way
  • Expensive to build (requires deep analysis and development time)
  • Custom database must be maintained
  • Security is no longer controlled by the Dynamics CRM application

I personally like solution 5 better but it’s only because I’m a developer and I’d enjoy having to design the new database and the archiving tool. Of course, every company has its own preferences and resources with different set of skills. This should all be taken into consideration prior to deciding what solution to use. These are only the ones that I came up with, there are certainly other options out there.

Hope this helps!