Recurring Jobs Strategies for Dynamics CRM/365

With the rise of the cloud, I have seen lots of small businesses building their technology strategy with a very little amount of local infrastructure. Similarly, I also see larger enterprises migrating to the cloud and trying to remove most of their local infrastructure. In Dynamics CRM/365 implementations, we often need to run recurring jobs that executes business logic (data integration or synchronization, complex calculated field etc.). This article talks about the different places where you can have these Dynamics 365/CRM connected jobs running periodically.

Windows Executable or SSIS Packages

Windows Executable are a simple and very common approach. The principle is to write an Console Application and use CRM SDK to connect to CRM and C#/.NET and other APIs to connect to various systems as required to do all sorts of data processing. After the console app is coded and tested, it is typically deployed on a Windows Server machine as a Scheduled Task. Using a Window Services is a similar approach, the main difference is that the recurrence of the execution needs to be managed at the application level (in the Window Service code) instead of leveraging the Task Scheduler built-in feature. This strategy gives a lot of flexibility:

  • no constrains on how long the process can run for
  • you provide the hardware required to handle your activities
  • you leverage internal resources if they have .NET/C#/CRM SDK knowledge
  • no need to build a UI which saves a lot of time on the development side

If your business operates mostly in the cloud and you don’t have available servers to install and run you application(s), you can create a Virtual Machine in the cloud and deploy your executables on it (as long as your CRM/Dyn365 organizations and other integrated systems are accessible from the VM).

Using SQL Server Integration Services (SSIS) Packages is pretty much identical to the Windows Executable approach. The difference is that you use SSIS, combined with a third-party tool for your CRUD operations with CRM/Dyn365 such as KingswaySoft, to contain your business logic and manipulate your systems data. You can then schedule SQL Jobs to run SSIS packages on a recurring basis. This technique is used when there is stronger knowledge on the SQL/SSIS side within an enterprise. Again, if you do not have a local infrastructure, you will have to create a Virtual Machine in the cloud with the required SQL Server components to create SQL Jobs and execute SSIS packages.

Recurring CRM Workflow(s)

While Dynamics CRM/365 does not have true task scheduler functionality, out of the box processes (workflows) can be used to achieve similar type of functionality. This is done by using the “Process Timeout” operation within a CRM Workflow. Check this article from PowerObjects on exactly how to setup recurring workflows in CRM 2016.

In order to run custom business logic, data integration or calculation, you can create Custom Workflow Activities and call them from within the recurring workflow. The custom activity is native C#/.NET Code so you can do all sorts of operations from there. I love this approach because it allows you to keep your recurring processes mechanism inside the CRM solution. It makes the deployment lighter and simpler, you don’t have to deploy a package on a SQL Server, you don’t need to deploy an application on a server and configure it. That being said, this approach has quite a few limitations:

  1. It is not simple to reference libraries other than .NET and CRM libraries, fellow MVP Gonzalo Ruiz has a good article on how to achieve this here. If your custom business logic makes references to many external assemblies, it will be a bit more challenging to integration in a Custom Workflow activity (it’s a good idea to target web services instead).
  2. When your custom workflow activity as running is isolation mode or Sandboxed (this is always the case in cloud-base Dynamics CRM/365 Online tenants), there are limitations on what you can in the custom workflow activity (e.g. how to call external web applications, no IO operations, cannot access registry etc.). There is also a 2-minute timeout limit for any process running in Sandbox.

If your business logic is long to execute (more than 2 minutes), this is not a viable option.

Using Microsoft Flow

Flow is relatively new to the Microsoft Cloud. It is a configurable workflow engine that allows to automated operations related to a large variety of cloud applications. The workflows have triggers that originate from cloud applications (e.g. email received from a specific sender, document created in OneDrive, or Contact created in Dynamics 365) or timer based.

Flow offers a connector to Dynamics 365 in the cloud with a few trigger operations (record created, updated, deleted) and basic actions on Dynamics 365 such as creating, retrieving, updating or deleting a record or retrieving a list of records. This is a detailed article from Wayne Walton that perfectly summarized what Flow is and how it can be used, you can read it here.

While Flows can run on a schedule (every x seconds, minutes, days, hours), there is a limit to the systems you can connect to from flow (last time I checked, there was integration built with just over 80 SaaS apps). If you want flow to execute custom business logic, you can develop and expose a Web API to the web and configure Flow to call it with contextual parameters. Custom operations can then be processed from within the Web API.

The biggest advantage of using Flow is the ease of configuration. Power users with enough knowledge of Flow and Dynamics 365 can easily configure workflows. However, depending on the complexity of the operations that need to be executed, you might find yourself limited with the basic operations available. This could force you to write a custom Web API for additional processing. Overkill if you ask me, but there are situations in which this could make sense.

Other facts to take into consideration:

  • Flow can only connect to Dynamics 365/CRM organization in the Microsoft Cloud. If you are running on-premises with IFD, it will not work.
  • The License for flow is based on a number of runs per month (4500 with Plan 1 and 15000 with Plan 2).

Azure Functions

If you are not familiar with Azure Functions, you really should do that soon: “Azure Functions is a solution for easily running small pieces of code, or “functions,” in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it” (reference here).

You can write functions in the development language of your choice (C#, F#, Node.js, PHP or Python) that connect to various external services or systems, including Dynamics 365. See this walkthrough if interested in knowing what’s involved. Azure Functions support events based on a configurable timer.

What I particularly like about Azure Functions is that it is a pure serverless architecture to run custom code. If you had a small console application deployed as a scheduled task, you can easily bring your code into Azure Functions and do the same operations.

If you are a company that does not want to deal with the overhead of having and maintaining servers internally to run your recurring processes, this is definitely the way to go. The billing calculation is complicated, but in short you only pay for the resources your code consumes when it runs. That means if you don’t run anything for a period of time, you have nothing to pay. Another positive about Azure Functions is that we can reference the CRM SDK which means should be able to connect to IFD-configured on premise installations (I haven’t tried this but I am pretty confident).

One of the downside of using Azure Functions is the development experience. While this is changing, at the beginning you could only write your code in the Azure Functions code editor which makes you lose the magic of Visual Studio.

 

Of course, these are a few existing options for recurring processes as it relates to Dynamics 365/CRM. Ultimately, the decision for each enterprise comes from its strategy, its people and its willingness to make investment.

Hope this helps !


Dealing with Multi-Language Lookups

Very often, CRM entities are used as reference data tables, for example to keep a list of countries, states or provinces or other business/industry specific data. For some businesses I have seen entities to keep a list of distributors, list of business roles, regions to only name a few. When used that way, CRM entities provide a lot of great features that cannot easily be met with option sets such as the ability to manage large reference tables, lookup search, lookup filtering, ease of adding/editing/modifying data by power users without a deployment.

One of the issues with using CRM entities for reference data is that they is no concept of multi-language lookup in the Dynamics 365 / CRM platform. Lookups will always display the value of the primary field by default. This can cause an issue in places where you must have a fully multilingual application. In this article, I provide a few possible solutions to solve this issue.

As an example, we’ll use the context of a task for which we need to track its type. The list of the available task activity types is stored as records in an entity called “Task Activity“. The Task entity has a lookup to the Task Activity entity. The information needs to be stored in English and French


1 – Task Form with Lookup to Task Activity


2 – List of Task Activities

Resolution Option 1 – Concatenate multiple languages in one field with a separator

You will be disappointed, this is not a fancy solution. In the Task Activity entity, we have one field for the name in both languages and use concatenate both field values in the primary field using a workflow or plugin.

  • Name English (Single line of text – 100)
  • Name French (Single line of text – 100)
  • Name (Single line of text – 203) – read only for users, populated with “Name English | French Name”


3 – Task Activity Form

This is the most common approach that I have seen when the number of languages is small (2 languages). This has a disadvantage of sometimes creating long name values that are not fully visible in the views and on the forms, but it’s cheap and you keep the ability to search using lookup, and display the columns in French or English the views if you need to.

Resolution Option 2 – Plugins on Retrieve & Retrieve Multiple

This solution is a little more interesting, but risky. In the Task Activity, we still have one field for the name in both languages and we still concatenate both field values in the primary field.

  • Name English (Single line of text – 100)
  • Name French (Single line of text – 100)
  • Name (Single line of text – 203) – read only for users, populated with “Name English | French Name”

The principle is to write plugins on the Retrieve and Retrieve Multiple events of the Task Activity. In both of these plugins, you need to retrieve the connected user’s language (query the user settings table), and then replace the text being returned in the Name field by the value in the user’s language. This value can be obtained by querying the task activity record and retrieving the name in French or English, or simply splitting the Name field (primary field) with the separator and return the part in the desired user’s language. One plugin will handle the lookup column in list views (Retrieve Multiple), and the other will handle the form views (Retrieve).

Generally, it is not recommended to write plugins on the Retrieve and Retrieve Multiple events for performance reason. If the operations executed in those plugins are simple and optimized, it might be a viable solution. This is a solution that can scale well if you are dealing with more than 2 languages because in all cases, users see only the value in their selected language and the multiple values are transparent to them.

Stay tuned, I have an upcoming post where I provide some metrics about the impact of a plugin on Retrieve and Retrieve Multiple on performance.

Resolution Option 3 – Automated mapping of Option Set with Reference Entity Records

This is a bit of a complex solution, by far the fanciest. The idea is to use an option set instead of a lookup to reference the task activities, but the option set values will be “controlled” with records from the Task Activity CRM entity. It goes like this:

  1. Create a global option set named Task Activity
  2. Create your Task Activity entity with a primary “Name” field. Put the name of task activities in the primary language (language of the CRM org)
  3. When a record is created in the Task Activity entity, use a plugin to create an option set value in the global option set
  4. When a record is updated in the Task Activity entity, use a plugin to update the corresponding option set value in the global option set
  5. When a Task Activity record is deleted/deactivated, use a plugin to update the corresponding global option set value by putting brackets around the name for example, and also pushing the value to the bottom of the option set list
  6. You can then get the CRM Translation file and get the Task Activities values translated as part of the global solution.


4 – Records to Option Set Value mapping

The outcome is that for entities that need to capture the task activity information, there will be an option set field as opposed to the lookup field:


5 – Task Form with Option Set

While this has low impact on performance and leverages the out of the box language-aware option sets, it requires a serious time investment to define the development framework for each entity that required this mechanism to be implemented. It also requires a translator to update the CRM translation on a regular basis (every time there is a deployment). This is a fancy solution that requires a lot of coding and maintenance. In addition to that, you lose the ability to search and filter the content easily like you would do with lookups. You should make sure yours lists are not very long if you don’t want to end up with Option Set lists that are very long which will result in poor user experience.

I have rarely seen companies making such large investments to circumvent the lack of multi-language lookup in CRM. This is usually seen when there are laws that force you to have a system running fully in multiple languages.

Resolution Option 4 – Custom Screen for Lookup views & search

This is another fancy one for which I unfortunately don’t have any screenshot. We want to leverage option sets to “overwrite” lookup values and selection process with the following steps:

  1. Create a set of standardized Web Resources for Lookup Display and lookup value selection
  2. Display the web resources on the CRM Forms and hide the lookup controls
  3. The web resources will have built-in logic to display the value in the language of the current user, as well as a mechanism to allow searches (could be auto-complete based)

Writing a standard web resource control for that purpose is relatively simple. However, you might have additional work to do if you want to take advantage of filtering based on other fields, or custom filters. Also, this solves the issue on the form in the sense that you will see the values in the right language on the form, but for list views, reports etc. the problem will still exist so you’ll need to find another solution there.

Closing Thoughts

As you can see there is no perfect solution. Each organization has to decide the level of investment and risk they want to take to make sure they have multi-language option sets. Living in Canada where we have two official languages, this is a challenge that we often see in public sector implementations because having fully bilingual system is mandated by law. There are very few countries where this is the case (which is probably why Microsoft has not made investments in this area). Most private sector companies will usually impose a primary language for the entire organization.

Hope this helps!

Dynamics CRM Solution Architecture

Hello readers!

I just completed a series of very animated presentations on Solution Architecture for Dynamics CRM in a few different cities in the east of Canada. I thought I would share the slide deck.

As a summary, I complied a few thoughts about the CRM Solution Architect’s role and typical design consideration. Enjoy the content and feel free to reach out to me with questions and comments.

Dynamics CRM and multi-tenancy

It’s a topic that is often discussed and after Microsoft released its whitepaper on it late last year, I thought it would be interesting to write a short summary of some of the enterprise challenges that it addresses.

Definition

Let’s take a look at a few definitions of the term “Multitenancy

Multitenancy refers to a principle in software architecture where a single instance of the software runs on a server, serving multiple client-organizations (tenants).” – Wikipedia

Multi-tenancy is an architecture in which a single instance of a software application serves multiple customers. Each customer is called a tenant. Tenants may be given the ability to customize some parts of the application, such as color of the user interface (UI) or business rules, but they cannot customize the application’s code.” – WhatIs.com

Multitenancy is a reference to the mode of operation of software where multiple independent instances of one or multiple applications operate in a shared environment. The instances (tenants) are logically isolated, but physically integrated” – Gartner IT Glossary

Relating these definitions to Microsoft Dynamics CRM, we can view the CRM Server installation as the single instance of the application and the CRM Organizations created on the CRM server as tenants. The tenants are technical groupings within the application, such as separate databases within SQL Server.

Common Enterprise Challenges and Solutions based on Multi-tenancy

Function Localization

  • Description: Provide a common core of functionality, but with local functional variations – with “proper” delegation
  • Example: Multi-national company with business models that vary based on market size, legal compliance or other factors
  • Solution: Using multi-tenancy or multiple instances allows each business area or local region to have an independent implementation with its own local variations.

Master data management

  • Description: Provide a consistent, managed core of information, perhaps distributed only Selectively and enable smooth transitions without big bang replacements
  • Example: You may have an organization that operates in multiple areas and the terms/rules differ from one place to another. These different areas may need to manage their master data independently from one another. In these types of scenarios, it is important to maintain the data that is common across the different components and particularly critical is managing changes to that data. While there are different approaches for accomplishing this goal, many scenarios benefit by having a “master” for certain data sets because it provides for change management through that central master data source (master data management, or MDM)
  • Solution: This approach requires that the central master data be synchronized to all instances so that each instance has access to the latest version of the core information

Physical distribution

  • Description: Mitigate network latency
  • Example: For business solutions that support users that are physically distributed over large distances (global deployments), using a single instance may not be suitable because of the implications (such as WAN latency) associated with the infrastructure over which the users connect.
  • Solution: Distributing instances to provide users with more local access can reduce or overcome WAN-related issues, as the access occurs over shorter network connections

Security/privacy

  • Description: Accommodate legislative/national differences (e.g. patient confidentiality, Swiss banking, third-party use)
  • Example: This is usually a resulting of some sort of legal compliance. For example in Canada, healthcare patient information cannot be transferred from one province to another. If there was a national platform, there would have to be a way to prevent people from access data from patients living in other provinces.
  • Solution: In these types of scenarios, some or all of the data is stored locally, and potentially some of the data is stored centrally

Scalability

  • Description: Accommodate extreme volumes and/or extensive use of Service Scheduling; Provide for isolation of workloads (e.g. web site, customer kiosks)
  • Example: While a single instance of Microsoft Dynamics CRM can scale up and out to support the growth of a customer’s business, with very high data volumes or levels of complexity, there are additional considerations.
  • Solution: for scenarios in which groups of users work independently of each other in operational terms, it may be possible to host the groups on separate Microsoft Dynamics CRM instances and to use reporting to combine results across business areas for management oversight

Multi-tenancy Challenges and Solutions Patterns

The whitepapers has a full detailed section on possible solutions to the challenges introduced by multi-tenancy. These challenges come from the fact that the separation has to be managed. You may want to synchronize your metadata, or expose data from on CRM organization within a different organization.

Silverlight, HTML5 and Dynamics CRM

The Glory Days of Silverlight

Silverlight is Microsoft’s plugin for web-browsers that enables running rich Internet applications, with features and purposes similar to those of Adobe Flash such as multimedia, graphics, animations etc. Shortly after Silverlight was introduced to the market in 2007, Microsoft quickly started to build knowledge around how to write and deploy rich applications and integrate them with Microsoft Dynamics CRM starting at version 4.0.

When you thinking about it, the need for custom UI integration with Dynamics CRM has always been there since the earlier versions. As CRM solution providers, most of us have been in situations where we can come up with a complex data model and we feel like the out of the box UI capabilities won’t be user friendly enough to drive user adoption. Situations like that created the need to build more user friendly interfaces to simply make people’s lives easier when they start using the CRM/XRM application.

To better illustrate, here are a few examples of when we would want to write a custom UI components in CRM

  • Display a tree view of related records with parent/child relationship.
  • Display complex search results: I’m thinking about PowerSearch which is a Global Search add-on for CRM. If you want to search for multiple entity types at once, a custom UI is required to display all results on a single page/view
  • Display timesheets, Gantt project management charts in Professional Services Automation solutions such as Assistance PSA and XRM1 (view screenshots below)

The Decline of Silverlight and the Rise of HTML5

With the emergence of HTML5, it seems we are headed towards a future in which browsers will support HTML5 tags natively thus enable rich content without the need of plugins like Silverlight or Adobe Flash. If some of us as individuals don’t believe this is true, Microsoft and Adobe seem to believe it is since they both dropped or significantly slowed down the evolution of their platforms. Silverlight’s latest major release (version 5) came out in 2011 in a world in which we see companies releasing software solutions at a very fast pace. The emergence of HTML5 have been well documented over the past year. There are still a lot of skeptics out there and that is understandable given how long it’s taking for the HTML5 standard to be completely defined and made available in all browsers.

Dynamics CRM: HTML5/JavaScript or Silverlight

What does this all mean for us Dynamics CRM integrators? The need to have custom UI controls is still existent and it will not go away even with all the new flexibility that we get with CRM 2013. Some data models will always be complex enough to require a better UI to give the solution its best chance of being used. In addition to that, there are still plenty of CRM add-ons built by Microsoft Partners that still use Silverlight 5 as a key piece of their solution. Below is a decision matrix that I came up with for us CRM Solutions providers going forward in making a technology decision when building new UI pieces for MSCRM.

What about the upgrade question? You have Silverlight controls and are wondering if you should built new controls in HTML5 and JavaScript. It’s your decision. Silverlight is not dead, Microsoft is still supporting it and it will for a long time. If you want to learn the new technology and have the time and money to do so, then go for the HTML5 remodeling. Keep in mind that it is a risk given that we have no idea what the lifespan of HTML5 will be.

What about buying an add-on that heavily relies on Silverlight controls? I don’t have a problem with that as long as it’s OK for you to install Silverlight on all client computers. Moving controls from Silverlight to HTML5/Javascript is A LOT of work and represents a significant amount of work for the add-on solution providers. They will upgrade when the time is right for them to do so (hopefully).