Products and Prices synchronization between Dynamics CRM/365 and ERP

When Dynamics 365 is used as a Sales force platform, there is often a need to identify the products that customers are buying in order to determine information such as sales price, support eligibility or other more business and industry specific information. Dynamics 365/CRM offers a set of entities and processes to manage the sales cycles. However, in most cases, the sales have to be recorded in a Finance system, often referred to as ERP (Enterprise Resource Planning) software. This article talks about what Dynamics 365/CRM has to offer on the sales process, product and price management side, what the limitations are then it comes to integrating with an an ERP and discusses some possible solutions.

Managing Sales Process and Pricing in Dynamics 365/CRM

Dynamics 365/CRM comes with a pre-defined sales process that can take you from a lead to a sales order and an invoice. It’s up to each organization to decide what entities they want to use in the process and configure it to meet their needs. The image below shows the sales cycles as defines by Out of the Box Dynamics 365 for Sales.


During the sales process, there is a level of interaction with products and prices at almost every step.

  • In the Opportunity, you enter the list of products and prices that your prospect is interested in.
  • In the quotes, you can copy the products selected in the related opportunity and update them, change the prices, rework the quote until it is accepted by the customer
  • The Order uses the products and prices defined in the accepted quote and the Invoice has the same information as the order

Prior to being made available for selection, the products and prices have to be defined and published in the system. The principle (at a very high level) is illustrated in the image below:



I’m purposely not going over details related to the Price List setup (e.g. volume discounts setup) or Opportunity (write-in products, percentage or amount based manual discount etc.). There is much more to it but I just want to give the big picture here. To sell products in CRM, you must have pre-defined Price List per currency. If you sell in a lot of regions, have a lot of different bundles with lots of product types and variances, then you might end up with a system that is loaded with products and price lists. There is no issue with this model as long as there is an established process to manage those products and price lists in your CRM system.

What are the challenges when ERP comes into play?

There are a few challenges that we see in the field as it relates to the process and structure above.

The out of the box sales process ends with the creation of a sales order and an invoice in Dynamics CRM/365. The reality is that a lot of businesses have these steps in an ERP or have to synchronize what CRM produces to a Finance system. This can be complex because the details about the products selected in the Quote/Order/Invoice have to have matching elements in the ERP system (usually SKU). This means that your product and pricing information has to be synchronized between your CRM and your ERP. This is often a challenge.

ERPs typically have more advanced ways to manage products and prices. They are not as rigid as the price-list based Dynamics 365/CRM structure. As an example, you can have product masters that serve as models for variants (e.g. product is a t-shirt, variants can be size, color, drawing etc.) and prices are automatically calculated based on the selected variables, the region and various other configurable parameters. These mechanisms present in ERP systems are in many cases more adopted for complex products and pricing structures because they allow for more flexibility and result with a model that is easier to maintain. The direct consequence for synchronizing product and price lists with Dynamics CRM/365 is that the data structure of the ERP either has to be converted into a price-list based model, or CRM has to be customized to match the ERP system’s product and pricing structure. Here are a few suggestions and guidelines when you are faced with this type of decision:

  1. If the ERP data model is not too far from the CRM model, you can simply make adjustments as required in CRM and synchronize the data between the 2 systems as needed.
  2. If the ERP data model is complex (e.g. uses product variants), ask yourself the following questions
    • Is it even possible to adapt the CRM data model to match the complex ERP structure?
    • What is the effort (data modeling, data transformation) to keep the CRM and ERP products and prices in sync? (Some ERP systems have ways to automatically generate a price-list based data structure for extraction)
    • Considering that the data model and sync issues are resolved, does the structure and volume of the data (number of products and price lists and data model) allow for a good user experience on the CRM sales side? (i.e. can users still easily find their products and price lists in CRM while working with opportunities or orders etc.). – you might have to create a custom user interface to provide your users with a better experience in order to drive adoption.
  3. Consider using a Configure Price Quote (CPQ) solution as an add-on to Dynamics CRM/365. Some of them are extremely well designed and provide the ability to configure complex product pricing structures while taking care of the user interface to build quotes (add products and automatically calculate prices etc.) and being integrated with Dynamics 365 for Sales/CRM and some ERP products.

A question is often asked: if ERPs are more flexible for pricing, why not have your sales people directly working in ERP? And the answer is simple for Dynamics 365. First, the cost, it is generally much cheaper to buy a Dynamics 365 for Sales app license than to buy an ERP license. Second, Dynamics 365/CRM is much more configurable to adapt to the business processes (creating fields, forms, views, relationships, integrated reports, sales suggestions etc.). I could go further but I won’t. Don’t have your sales people working out of your ERP, it’s generally a bad idea.

Conclusion

    Product management in Dynamics CRM/365 is a relatively rigid process. When you have a complex scenario where the products and pricing info comes from another system, it is important to carefully analyse the gap with the Dynamics 365 structure and to decide how you want to close that gap.

Happy CRM’ing!


Recurring Jobs Strategies for Dynamics CRM/365

With the rise of the cloud, I have seen lots of small businesses building their technology strategy with a very little amount of local infrastructure. Similarly, I also see larger enterprises migrating to the cloud and trying to remove most of their local infrastructure. In Dynamics CRM/365 implementations, we often need to run recurring jobs that executes business logic (data integration or synchronization, complex calculated field etc.). This article talks about the different places where you can have these Dynamics 365/CRM connected jobs running periodically.

Windows Executable or SSIS Packages

Windows Executable are a simple and very common approach. The principle is to write an Console Application and use CRM SDK to connect to CRM and C#/.NET and other APIs to connect to various systems as required to do all sorts of data processing. After the console app is coded and tested, it is typically deployed on a Windows Server machine as a Scheduled Task. Using a Window Services is a similar approach, the main difference is that the recurrence of the execution needs to be managed at the application level (in the Window Service code) instead of leveraging the Task Scheduler built-in feature. This strategy gives a lot of flexibility:

  • no constrains on how long the process can run for
  • you provide the hardware required to handle your activities
  • you leverage internal resources if they have .NET/C#/CRM SDK knowledge
  • no need to build a UI which saves a lot of time on the development side

If your business operates mostly in the cloud and you don’t have available servers to install and run you application(s), you can create a Virtual Machine in the cloud and deploy your executables on it (as long as your CRM/Dyn365 organizations and other integrated systems are accessible from the VM).

Using SQL Server Integration Services (SSIS) Packages is pretty much identical to the Windows Executable approach. The difference is that you use SSIS, combined with a third-party tool for your CRUD operations with CRM/Dyn365 such as KingswaySoft, to contain your business logic and manipulate your systems data. You can then schedule SQL Jobs to run SSIS packages on a recurring basis. This technique is used when there is stronger knowledge on the SQL/SSIS side within an enterprise. Again, if you do not have a local infrastructure, you will have to create a Virtual Machine in the cloud with the required SQL Server components to create SQL Jobs and execute SSIS packages.

Recurring CRM Workflow(s)

While Dynamics CRM/365 does not have true task scheduler functionality, out of the box processes (workflows) can be used to achieve similar type of functionality. This is done by using the “Process Timeout” operation within a CRM Workflow. Check this article from PowerObjects on exactly how to setup recurring workflows in CRM 2016.

In order to run custom business logic, data integration or calculation, you can create Custom Workflow Activities and call them from within the recurring workflow. The custom activity is native C#/.NET Code so you can do all sorts of operations from there. I love this approach because it allows you to keep your recurring processes mechanism inside the CRM solution. It makes the deployment lighter and simpler, you don’t have to deploy a package on a SQL Server, you don’t need to deploy an application on a server and configure it. That being said, this approach has quite a few limitations:

  1. It is not simple to reference libraries other than .NET and CRM libraries, fellow MVP Gonzalo Ruiz has a good article on how to achieve this here. If your custom business logic makes references to many external assemblies, it will be a bit more challenging to integration in a Custom Workflow activity (it’s a good idea to target web services instead).
  2. When your custom workflow activity as running is isolation mode or Sandboxed (this is always the case in cloud-base Dynamics CRM/365 Online tenants), there are limitations on what you can in the custom workflow activity (e.g. how to call external web applications, no IO operations, cannot access registry etc.). There is also a 2-minute timeout limit for any process running in Sandbox.

If your business logic is long to execute (more than 2 minutes), this is not a viable option.

Using Microsoft Flow

Flow is relatively new to the Microsoft Cloud. It is a configurable workflow engine that allows to automated operations related to a large variety of cloud applications. The workflows have triggers that originate from cloud applications (e.g. email received from a specific sender, document created in OneDrive, or Contact created in Dynamics 365) or timer based.

Flow offers a connector to Dynamics 365 in the cloud with a few trigger operations (record created, updated, deleted) and basic actions on Dynamics 365 such as creating, retrieving, updating or deleting a record or retrieving a list of records. This is a detailed article from Wayne Walton that perfectly summarized what Flow is and how it can be used, you can read it here.

While Flows can run on a schedule (every x seconds, minutes, days, hours), there is a limit to the systems you can connect to from flow (last time I checked, there was integration built with just over 80 SaaS apps). If you want flow to execute custom business logic, you can develop and expose a Web API to the web and configure Flow to call it with contextual parameters. Custom operations can then be processed from within the Web API.

The biggest advantage of using Flow is the ease of configuration. Power users with enough knowledge of Flow and Dynamics 365 can easily configure workflows. However, depending on the complexity of the operations that need to be executed, you might find yourself limited with the basic operations available. This could force you to write a custom Web API for additional processing. Overkill if you ask me, but there are situations in which this could make sense.

Other facts to take into consideration:

  • Flow can only connect to Dynamics 365/CRM organization in the Microsoft Cloud. If you are running on-premises with IFD, it will not work.
  • The License for flow is based on a number of runs per month (4500 with Plan 1 and 15000 with Plan 2).

Azure Functions

If you are not familiar with Azure Functions, you really should do that soon: “Azure Functions is a solution for easily running small pieces of code, or “functions,” in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it” (reference here).

You can write functions in the development language of your choice (C#, F#, Node.js, PHP or Python) that connect to various external services or systems, including Dynamics 365. See this walkthrough if interested in knowing what’s involved. Azure Functions support events based on a configurable timer.

What I particularly like about Azure Functions is that it is a pure serverless architecture to run custom code. If you had a small console application deployed as a scheduled task, you can easily bring your code into Azure Functions and do the same operations.

If you are a company that does not want to deal with the overhead of having and maintaining servers internally to run your recurring processes, this is definitely the way to go. The billing calculation is complicated, but in short you only pay for the resources your code consumes when it runs. That means if you don’t run anything for a period of time, you have nothing to pay. Another positive about Azure Functions is that we can reference the CRM SDK which means should be able to connect to IFD-configured on premise installations (I haven’t tried this but I am pretty confident).

One of the downside of using Azure Functions is the development experience. While this is changing, at the beginning you could only write your code in the Azure Functions code editor which makes you lose the magic of Visual Studio.

 

Of course, these are a few existing options for recurring processes as it relates to Dynamics 365/CRM. Ultimately, the decision for each enterprise comes from its strategy, its people and its willingness to make investment.

Hope this helps !


Plugin on Retrieve and Retrieve Multiple – How bad is it?

I have managed to be in the Dynamics CRM/365 world for over 7 years without having to write a single plugin on Retrieve and Retrieve Multiple. The recommendation that I give is to stay away from those. The reason is simple, it sounds horrible from a performance standpoint, and even people from Microsoft have recommended against it in many scenarios. Faced with an issue recently where we had to really consider it, I did some research and testing to try to measure the impact of such plugins on system performance. This article provides some background as to why we recommend against these types of plugins, and it also provides some of our finding after we tested for performance.

Why are Plugins on Retrieve Multiple scary?

When looking at the event execution pipeline for Dynamics CRM/365, we need to consider that there are a lot of steps involved as part of every CRM transactions. To do anything, we need to go through the CRM web service APIs which will start the chain of events in the pipeline (pre-validate, pre-event, core action/database access and then post-event).


This means that in general, it is a good practice to build everything for optimized performance to give your user base a good experience. It’s not like having a custom database where you can create store procedures easily, add triggers and so on, taking advantage of the SQL Server features and infrastructure.

Now, back to Retrieve and Retrieve Multiple plugins.

Retrieve Multiple: Think about it this way, you retrieve a list of 200 accounts, your plugin on Retrieve Multiple fires once and gives you the list of accounts being returned to the screen with all columns being retrieved. For each of these row, you do some type of operation. It doesn’t too sound good, does it?

Retrieve: You double click on an account from a list view. As the account columns are being retrieved to display the account form on the screen, you plugin on Retrieve fires and gives you the account object. At that point you can modify the content of the columns being retured as required before they are returned to the screen for the user. This really doesn’t sound too bad.

Some Findings on the impact on Performance

To provide some context into what we were trying to do, I wrote about multi-language lookup in a previous blog post. One of the solution that we have considered for one of our client is to use a plugin on Retrieve and Retrieve Multiple in order to change the value of the lookup primary fields in order to display a value in the user’s current language. The method we used is similar to what Aileen Gusni does here and almost identical to what Scott Durow does here.

We store a Region in a custom Entity. Accounts have a lookup that indicates its region.

Scenario 1:

The Region’s Primary Field contains a concatenation of the English and French region names with a relatively safe separator (we use “|”). When you load the list view of accounts, we have a plugin on Retrieve Multiple that looks at the columns being returned. If the Region column is returned, we retrieve the user’s language, we split the name of the region with the separator (the name is available in the entity reference) and we replace its value in the target object by the region name in the user’s language. The plugin on Retrieve does the same operation on the single role being retrieved.

The average execution time for the Retrieve Multiple plugin when loading 150 rows was 15.43 milliseconds so 0.01543 seconds.
The average execution time for the Retrieve plugin to load one rows was too little for the system to return a value (we got 0 milliseconds every time).

Scenario 2:

The Region entity has English Name and French Name attributes. When you load the list view of accounts, we have a plugin on retrieve multiple that looks at the columns being returned. If the Region column is returned, we retrieve the user’s language, we then retrieve the English or French name from the Region entity and we replace its value in the target object by the region name in the user’s language. The plugin on Retrieve does the same operation on the single role being retrieved.

The average execution time for the Retrieve Multiple plugin to load 150 rows was 1003.94 milliseconds so 1.00394 seconds.
The average execution time for the Retrieve plugin to load one rows was 16.02 milliseconds so 0.01602 seconds.

What should you read into this?

While these numbers don’t look too crazy at all, especially in the first scenario, there are a lot of factors to take into consideration that are not really showing here and that will vary in almost any scenario.

  • What is the infrastructure you are running on? The faster your severs and networks, the better the performance will be.
  • What you do in these plugins matters a great deal. You should avoid or limit the number of read/writes to the database during the execution of those plugins.
  • Our tests were made with a low level of users in the system, it is critical to scale up and see what these numbers look like at peak time of your system.

With all this said, I still recommend against it. Use with a great deal of caution! If/when possible, use calculated fields instead of writing plugins on these messages. This will also keep you away from limitation such as this one.

Hope this helps!

Dealing with Multi-Language Lookups

Very often, CRM entities are used as reference data tables, for example to keep a list of countries, states or provinces or other business/industry specific data. For some businesses I have seen entities to keep a list of distributors, list of business roles, regions to only name a few. When used that way, CRM entities provide a lot of great features that cannot easily be met with option sets such as the ability to manage large reference tables, lookup search, lookup filtering, ease of adding/editing/modifying data by power users without a deployment.

One of the issues with using CRM entities for reference data is that they is no concept of multi-language lookup in the Dynamics 365 / CRM platform. Lookups will always display the value of the primary field by default. This can cause an issue in places where you must have a fully multilingual application. In this article, I provide a few possible solutions to solve this issue.

As an example, we’ll use the context of a task for which we need to track its type. The list of the available task activity types is stored as records in an entity called “Task Activity“. The Task entity has a lookup to the Task Activity entity. The information needs to be stored in English and French


1 – Task Form with Lookup to Task Activity


2 – List of Task Activities

Resolution Option 1 – Concatenate multiple languages in one field with a separator

You will be disappointed, this is not a fancy solution. In the Task Activity entity, we have one field for the name in both languages and use concatenate both field values in the primary field using a workflow or plugin.

  • Name English (Single line of text – 100)
  • Name French (Single line of text – 100)
  • Name (Single line of text – 203) – read only for users, populated with “Name English | French Name”


3 – Task Activity Form

This is the most common approach that I have seen when the number of languages is small (2 languages). This has a disadvantage of sometimes creating long name values that are not fully visible in the views and on the forms, but it’s cheap and you keep the ability to search using lookup, and display the columns in French or English the views if you need to.

Resolution Option 2 – Plugins on Retrieve & Retrieve Multiple

This solution is a little more interesting, but risky. In the Task Activity, we still have one field for the name in both languages and we still concatenate both field values in the primary field.

  • Name English (Single line of text – 100)
  • Name French (Single line of text – 100)
  • Name (Single line of text – 203) – read only for users, populated with “Name English | French Name”

The principle is to write plugins on the Retrieve and Retrieve Multiple events of the Task Activity. In both of these plugins, you need to retrieve the connected user’s language (query the user settings table), and then replace the text being returned in the Name field by the value in the user’s language. This value can be obtained by querying the task activity record and retrieving the name in French or English, or simply splitting the Name field (primary field) with the separator and return the part in the desired user’s language. One plugin will handle the lookup column in list views (Retrieve Multiple), and the other will handle the form views (Retrieve).

Generally, it is not recommended to write plugins on the Retrieve and Retrieve Multiple events for performance reason. If the operations executed in those plugins are simple and optimized, it might be a viable solution. This is a solution that can scale well if you are dealing with more than 2 languages because in all cases, users see only the value in their selected language and the multiple values are transparent to them.

Stay tuned, I have an upcoming post where I provide some metrics about the impact of a plugin on Retrieve and Retrieve Multiple on performance.

Resolution Option 3 – Automated mapping of Option Set with Reference Entity Records

This is a bit of a complex solution, by far the fanciest. The idea is to use an option set instead of a lookup to reference the task activities, but the option set values will be “controlled” with records from the Task Activity CRM entity. It goes like this:

  1. Create a global option set named Task Activity
  2. Create your Task Activity entity with a primary “Name” field. Put the name of task activities in the primary language (language of the CRM org)
  3. When a record is created in the Task Activity entity, use a plugin to create an option set value in the global option set
  4. When a record is updated in the Task Activity entity, use a plugin to update the corresponding option set value in the global option set
  5. When a Task Activity record is deleted/deactivated, use a plugin to update the corresponding global option set value by putting brackets around the name for example, and also pushing the value to the bottom of the option set list
  6. You can then get the CRM Translation file and get the Task Activities values translated as part of the global solution.


4 – Records to Option Set Value mapping

The outcome is that for entities that need to capture the task activity information, there will be an option set field as opposed to the lookup field:


5 – Task Form with Option Set

While this has low impact on performance and leverages the out of the box language-aware option sets, it requires a serious time investment to define the development framework for each entity that required this mechanism to be implemented. It also requires a translator to update the CRM translation on a regular basis (every time there is a deployment). This is a fancy solution that requires a lot of coding and maintenance. In addition to that, you lose the ability to search and filter the content easily like you would do with lookups. You should make sure yours lists are not very long if you don’t want to end up with Option Set lists that are very long which will result in poor user experience.

I have rarely seen companies making such large investments to circumvent the lack of multi-language lookup in CRM. This is usually seen when there are laws that force you to have a system running fully in multiple languages.

Resolution Option 4 – Custom Screen for Lookup views & search

This is another fancy one for which I unfortunately don’t have any screenshot. We want to leverage option sets to “overwrite” lookup values and selection process with the following steps:

  1. Create a set of standardized Web Resources for Lookup Display and lookup value selection
  2. Display the web resources on the CRM Forms and hide the lookup controls
  3. The web resources will have built-in logic to display the value in the language of the current user, as well as a mechanism to allow searches (could be auto-complete based)

Writing a standard web resource control for that purpose is relatively simple. However, you might have additional work to do if you want to take advantage of filtering based on other fields, or custom filters. Also, this solves the issue on the form in the sense that you will see the values in the right language on the form, but for list views, reports etc. the problem will still exist so you’ll need to find another solution there.

Closing Thoughts

As you can see there is no perfect solution. Each organization has to decide the level of investment and risk they want to take to make sure they have multi-language option sets. Living in Canada where we have two official languages, this is a challenge that we often see in public sector implementations because having fully bilingual system is mandated by law. There are very few countries where this is the case (which is probably why Microsoft has not made investments in this area). Most private sector companies will usually impose a primary language for the entire organization.

Hope this helps!

Where to store configuration data in Dynamics 365/CRM?

In almost all the complex systems that I’ve worked with, there has always been a need to store some configuration information. It could be URLs to external APIs or web sites, connection strings to a database for integration purposes, application specific parameters that drive business logic such as Security Roles, Accounts or Contacts. In Dynamics CRM/365, there is often a need to store the same type of information so that plugins, workflow and other integrated applications read them and perform business operation accordingly. In this article, I share the most common options and share some pros and cons associated to each of them.

Key Value Pair Entity

IF you’ve done some application design, you know what the concept of a key value pair table is. In the Dynamics CRM/365 context, it is an entity that contains two required text fields: one for the key and the other for the value. The key represents the name of your configuration variable and the value is, well, the value for the key. For each configuration element that needs to be stored in your system, you create a new row with a key and its value. The images below provide an example of a Key Value Pair entity in CRM in which we stored information to connect to an external web service and the ID of a Security Role, all as text values.

bp-1

1 – List of Key Value Pairs in Dynamics CRM/365

bp-2

2 – Key Value Pair Entity for Example

Pros

  • Very simple data structure
  • Easy to add/remove configuration values
  • Code to read and use the variables does not change over time
  • Retrieving a config element is a fast operation (one row with two columns)

Cons

  • Data type for the value is a text field (not practical for lookups or other data types as you may have to store GUIDs for example)
  • Inability to set default values
  • Inability to use FLS on specific config elements
  • The Keys must be hard-coded in code and/or documented and maintained somewhere

If you are planning to use a Key Value pair type of configuration table, my recommendation is to have one key field as text, and configuration value type field (option set with the type of field – example text, two options) and multiple value columns of different types (e.g. Value (lookup 1), Value (lookup 2), Value (text), Value (two option)). As a bonus, you can add some business rules to prevent the selection of the wrong data type based on the selected configuration value type.

Configuration Entity

Here, the idea is to have an entity in CRM with one field for each configuration element that needs to be stored. In the example below, we have a table that contains information to connect to an ERP web service as well as credentials, and a lookup to a System Admin role (similar information as above). It in this case, there should only one row in the configuration table.

bp-3

3 – System Configuration List View (only one row available)

bp-4

4 – System Configuration Entity form (shows all the configuration fields)

Pros

  • No need to have a list of configuration key names (use the field names enforced at the database level)
  • Each configuration element has the appropriate type (e.g Lookup, text field, two option, option set etc.)
  • Ability to enable Field Level Security on specific parameters
  • Allows for default values for certain data types
  • Easier to setup by an end user (create one row, set values as opposed to create multiple rows with key and value)

Cons

  • Schema change + Code update required anytime a new configuration element is needed
  • You need to ensure there is only one record for the entity (plugin validation on create)
  • Configuration table grows horizontally and not vertically over time

Wrap up

I have used both models extensively and they both work well. For a stable system with not a lot of moving piece, I tend to like the Configuration entity better. For system where things change all the time and new config items need to be added on a regular basis, using the key value pair entity is often more cost-effective. There is also the possibility to use an XML web resource for parameters or the plugin secure and unsecure configuration fields.

Regardless of the method you use, consider caching the configuration data when possible to increase your system’s performance. On the Client side with JavaScript, you can use a few different mechanisms for caching (local storage, cookie). On the server side for plugins, I often use a static cache but this only works for plugins executed in full trust mode (in other words, if you are in CRM Online, no caching on the server side).

Cheers!