With the rise of the cloud, I have seen lots of small businesses building their technology strategy with a very little amount of local infrastructure. Similarly, I also see larger enterprises migrating to the cloud and trying to remove most of their local infrastructure. In Dynamics CRM/365 implementations, we often need to run recurring jobs that executes business logic (data integration or synchronization, complex calculated field etc.). This article talks about the different places where you can have these Dynamics 365/CRM connected jobs running periodically.
Windows Executable or SSIS Packages
Windows Executable are a simple and very common approach. The principle is to write an Console Application and use CRM SDK to connect to CRM and C#/.NET and other APIs to connect to various systems as required to do all sorts of data processing. After the console app is coded and tested, it is typically deployed on a Windows Server machine as a Scheduled Task. Using a Window Services is a similar approach, the main difference is that the recurrence of the execution needs to be managed at the application level (in the Window Service code) instead of leveraging the Task Scheduler built-in feature. This strategy gives a lot of flexibility:
- no constrains on how long the process can run for
- you provide the hardware required to handle your activities
- you leverage internal resources if they have .NET/C#/CRM SDK knowledge
- no need to build a UI which saves a lot of time on the development side
If your business operates mostly in the cloud and you don’t have available servers to install and run you application(s), you can create a Virtual Machine in the cloud and deploy your executables on it (as long as your CRM/Dyn365 organizations and other integrated systems are accessible from the VM).
Using SQL Server Integration Services (SSIS) Packages is pretty much identical to the Windows Executable approach. The difference is that you use SSIS, combined with a third-party tool for your CRUD operations with CRM/Dyn365 such as KingswaySoft, to contain your business logic and manipulate your systems data. You can then schedule SQL Jobs to run SSIS packages on a recurring basis. This technique is used when there is stronger knowledge on the SQL/SSIS side within an enterprise. Again, if you do not have a local infrastructure, you will have to create a Virtual Machine in the cloud with the required SQL Server components to create SQL Jobs and execute SSIS packages.
Recurring CRM Workflow(s)
While Dynamics CRM/365 does not have true task scheduler functionality, out of the box processes (workflows) can be used to achieve similar type of functionality. This is done by using the “Process Timeout” operation within a CRM Workflow. Check this article from PowerObjects on exactly how to setup recurring workflows in CRM 2016.
In order to run custom business logic, data integration or calculation, you can create Custom Workflow Activities and call them from within the recurring workflow. The custom activity is native C#/.NET Code so you can do all sorts of operations from there. I love this approach because it allows you to keep your recurring processes mechanism inside the CRM solution. It makes the deployment lighter and simpler, you don’t have to deploy a package on a SQL Server, you don’t need to deploy an application on a server and configure it. That being said, this approach has quite a few limitations:
- It is not simple to reference libraries other than .NET and CRM libraries, fellow MVP Gonzalo Ruiz has a good article on how to achieve this here. If your custom business logic makes references to many external assemblies, it will be a bit more challenging to integration in a Custom Workflow activity (it’s a good idea to target web services instead).
- When your custom workflow activity as running is isolation mode or Sandboxed (this is always the case in cloud-base Dynamics CRM/365 Online tenants), there are limitations on what you can in the custom workflow activity (e.g. how to call external web applications, no IO operations, cannot access registry etc.). There is also a 2-minute timeout limit for any process running in Sandbox.
If your business logic is long to execute (more than 2 minutes), this is not a viable option.
Using Microsoft Flow
Flow is relatively new to the Microsoft Cloud. It is a configurable workflow engine that allows to automated operations related to a large variety of cloud applications. The workflows have triggers that originate from cloud applications (e.g. email received from a specific sender, document created in OneDrive, or Contact created in Dynamics 365) or timer based.
Flow offers a connector to Dynamics 365 in the cloud with a few trigger operations (record created, updated, deleted) and basic actions on Dynamics 365 such as creating, retrieving, updating or deleting a record or retrieving a list of records. This is a detailed article from Wayne Walton that perfectly summarized what Flow is and how it can be used, you can read it here.
While Flows can run on a schedule (every x seconds, minutes, days, hours), there is a limit to the systems you can connect to from flow (last time I checked, there was integration built with just over 80 SaaS apps). If you want flow to execute custom business logic, you can develop and expose a Web API to the web and configure Flow to call it with contextual parameters. Custom operations can then be processed from within the Web API.
The biggest advantage of using Flow is the ease of configuration. Power users with enough knowledge of Flow and Dynamics 365 can easily configure workflows. However, depending on the complexity of the operations that need to be executed, you might find yourself limited with the basic operations available. This could force you to write a custom Web API for additional processing. Overkill if you ask me, but there are situations in which this could make sense.
Other facts to take into consideration:
- Flow can only connect to Dynamics 365/CRM organization in the Microsoft Cloud. If you are running on-premises with IFD, it will not work.
- The License for flow is based on a number of runs per month (4500 with Plan 1 and 15000 with Plan 2).
If you are not familiar with Azure Functions, you really should do that soon: “Azure Functions is a solution for easily running small pieces of code, or “functions,” in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it” (reference here).
You can write functions in the development language of your choice (C#, F#, Node.js, PHP or Python) that connect to various external services or systems, including Dynamics 365. See this walkthrough if interested in knowing what’s involved. Azure Functions support events based on a configurable timer.
What I particularly like about Azure Functions is that it is a pure serverless architecture to run custom code. If you had a small console application deployed as a scheduled task, you can easily bring your code into Azure Functions and do the same operations.
If you are a company that does not want to deal with the overhead of having and maintaining servers internally to run your recurring processes, this is definitely the way to go. The billing calculation is complicated, but in short you only pay for the resources your code consumes when it runs. That means if you don’t run anything for a period of time, you have nothing to pay. Another positive about Azure Functions is that we can reference the CRM SDK which means should be able to connect to IFD-configured on premise installations (I haven’t tried this but I am pretty confident).
One of the downside of using Azure Functions is the development experience. While this is changing, at the beginning you could only write your code in the Azure Functions code editor which makes you lose the magic of Visual Studio.
Of course, these are a few existing options for recurring processes as it relates to Dynamics 365/CRM. Ultimately, the decision for each enterprise comes from its strategy, its people and its willingness to make investment.
Hope this helps !