Plugin on Retrieve and Retrieve Multiple – How bad is it?

I have managed to be in the Dynamics CRM/365 world for over 7 years without having to write a single plugin on Retrieve and Retrieve Multiple. The recommendation that I give is to stay away from those. The reason is simple, it sounds horrible from a performance standpoint, and even people from Microsoft have recommended against it in many scenarios. Faced with an issue recently where we had to really consider it, I did some research and testing to try to measure the impact of such plugins on system performance. This article provides some background as to why we recommend against these types of plugins, and it also provides some of our finding after we tested for performance.

Why are Plugins on Retrieve Multiple scary?

When looking at the event execution pipeline for Dynamics CRM/365, we need to consider that there are a lot of steps involved as part of every CRM transactions. To do anything, we need to go through the CRM web service APIs which will start the chain of events in the pipeline (pre-validate, pre-event, core action/database access and then post-event).


This means that in general, it is a good practice to build everything for optimized performance to give your user base a good experience. It’s not like having a custom database where you can create store procedures easily, add triggers and so on, taking advantage of the SQL Server features and infrastructure.

Now, back to Retrieve and Retrieve Multiple plugins.

Retrieve Multiple: Think about it this way, you retrieve a list of 200 accounts, your plugin on Retrieve Multiple fires once and gives you the list of accounts being returned to the screen with all columns being retrieved. For each of these row, you do some type of operation. It doesn’t too sound good, does it?

Retrieve: You double click on an account from a list view. As the account columns are being retrieved to display the account form on the screen, you plugin on Retrieve fires and gives you the account object. At that point you can modify the content of the columns being retured as required before they are returned to the screen for the user. This really doesn’t sound too bad.

Some Findings on the impact on Performance

To provide some context into what we were trying to do, I wrote about multi-language lookup in a previous blog post. One of the solution that we have considered for one of our client is to use a plugin on Retrieve and Retrieve Multiple in order to change the value of the lookup primary fields in order to display a value in the user’s current language. The method we used is similar to what Aileen Gusni does here and almost identical to what Scott Durow does here.

We store a Region in a custom Entity. Accounts have a lookup that indicates its region.

Scenario 1:

The Region’s Primary Field contains a concatenation of the English and French region names with a relatively safe separator (we use “|”). When you load the list view of accounts, we have a plugin on Retrieve Multiple that looks at the columns being returned. If the Region column is returned, we retrieve the user’s language, we split the name of the region with the separator (the name is available in the entity reference) and we replace its value in the target object by the region name in the user’s language. The plugin on Retrieve does the same operation on the single role being retrieved.

The average execution time for the Retrieve Multiple plugin when loading 150 rows was 15.43 milliseconds so 0.01543 seconds.
The average execution time for the Retrieve plugin to load one rows was too little for the system to return a value (we got 0 milliseconds every time).

Scenario 2:

The Region entity has English Name and French Name attributes. When you load the list view of accounts, we have a plugin on retrieve multiple that looks at the columns being returned. If the Region column is returned, we retrieve the user’s language, we then retrieve the English or French name from the Region entity and we replace its value in the target object by the region name in the user’s language. The plugin on Retrieve does the same operation on the single role being retrieved.

The average execution time for the Retrieve Multiple plugin to load 150 rows was 1003.94 milliseconds so 1.00394 seconds.
The average execution time for the Retrieve plugin to load one rows was 16.02 milliseconds so 0.01602 seconds.

What should you read into this?

While these numbers don’t look too crazy at all, especially in the first scenario, there are a lot of factors to take into consideration that are not really showing here and that will vary in almost any scenario.

  • What is the infrastructure you are running on? The faster your severs and networks, the better the performance will be.
  • What you do in these plugins matters a great deal. You should avoid or limit the number of read/writes to the database during the execution of those plugins.
  • Our tests were made with a low level of users in the system, it is critical to scale up and see what these numbers look like at peak time of your system.

With all this said, I still recommend against it. Use with a great deal of caution! If/when possible, use calculated fields instead of writing plugins on these messages. This will also keep you away from limitation such as this one.

Hope this helps!

Dealing with Multi-Language Lookups

Very often, CRM entities are used as reference data tables, for example to keep a list of countries, states or provinces or other business/industry specific data. For some businesses I have seen entities to keep a list of distributors, list of business roles, regions to only name a few. When used that way, CRM entities provide a lot of great features that cannot easily be met with option sets such as the ability to manage large reference tables, lookup search, lookup filtering, ease of adding/editing/modifying data by power users without a deployment.

One of the issues with using CRM entities for reference data is that they is no concept of multi-language lookup in the Dynamics 365 / CRM platform. Lookups will always display the value of the primary field by default. This can cause an issue in places where you must have a fully multilingual application. In this article, I provide a few possible solutions to solve this issue.

As an example, we’ll use the context of a task for which we need to track its type. The list of the available task activity types is stored as records in an entity called “Task Activity“. The Task entity has a lookup to the Task Activity entity. The information needs to be stored in English and French


1 – Task Form with Lookup to Task Activity


2 – List of Task Activities

Resolution Option 1 – Concatenate multiple languages in one field with a separator

You will be disappointed, this is not a fancy solution. In the Task Activity entity, we have one field for the name in both languages and use concatenate both field values in the primary field using a workflow or plugin.

  • Name English (Single line of text – 100)
  • Name French (Single line of text – 100)
  • Name (Single line of text – 203) – read only for users, populated with “Name English | French Name”


3 – Task Activity Form

This is the most common approach that I have seen when the number of languages is small (2 languages). This has a disadvantage of sometimes creating long name values that are not fully visible in the views and on the forms, but it’s cheap and you keep the ability to search using lookup, and display the columns in French or English the views if you need to.

Resolution Option 2 – Plugins on Retrieve & Retrieve Multiple

This solution is a little more interesting, but risky. In the Task Activity, we still have one field for the name in both languages and we still concatenate both field values in the primary field.

  • Name English (Single line of text – 100)
  • Name French (Single line of text – 100)
  • Name (Single line of text – 203) – read only for users, populated with “Name English | French Name”

The principle is to write plugins on the Retrieve and Retrieve Multiple events of the Task Activity. In both of these plugins, you need to retrieve the connected user’s language (query the user settings table), and then replace the text being returned in the Name field by the value in the user’s language. This value can be obtained by querying the task activity record and retrieving the name in French or English, or simply splitting the Name field (primary field) with the separator and return the part in the desired user’s language. One plugin will handle the lookup column in list views (Retrieve Multiple), and the other will handle the form views (Retrieve).

Generally, it is not recommended to write plugins on the Retrieve and Retrieve Multiple events for performance reason. If the operations executed in those plugins are simple and optimized, it might be a viable solution. This is a solution that can scale well if you are dealing with more than 2 languages because in all cases, users see only the value in their selected language and the multiple values are transparent to them.

Stay tuned, I have an upcoming post where I provide some metrics about the impact of a plugin on Retrieve and Retrieve Multiple on performance.

Resolution Option 3 – Automated mapping of Option Set with Reference Entity Records

This is a bit of a complex solution, by far the fanciest. The idea is to use an option set instead of a lookup to reference the task activities, but the option set values will be “controlled” with records from the Task Activity CRM entity. It goes like this:

  1. Create a global option set named Task Activity
  2. Create your Task Activity entity with a primary “Name” field. Put the name of task activities in the primary language (language of the CRM org)
  3. When a record is created in the Task Activity entity, use a plugin to create an option set value in the global option set
  4. When a record is updated in the Task Activity entity, use a plugin to update the corresponding option set value in the global option set
  5. When a Task Activity record is deleted/deactivated, use a plugin to update the corresponding global option set value by putting brackets around the name for example, and also pushing the value to the bottom of the option set list
  6. You can then get the CRM Translation file and get the Task Activities values translated as part of the global solution.


4 – Records to Option Set Value mapping

The outcome is that for entities that need to capture the task activity information, there will be an option set field as opposed to the lookup field:


5 – Task Form with Option Set

While this has low impact on performance and leverages the out of the box language-aware option sets, it requires a serious time investment to define the development framework for each entity that required this mechanism to be implemented. It also requires a translator to update the CRM translation on a regular basis (every time there is a deployment). This is a fancy solution that requires a lot of coding and maintenance. In addition to that, you lose the ability to search and filter the content easily like you would do with lookups. You should make sure yours lists are not very long if you don’t want to end up with Option Set lists that are very long which will result in poor user experience.

I have rarely seen companies making such large investments to circumvent the lack of multi-language lookup in CRM. This is usually seen when there are laws that force you to have a system running fully in multiple languages.

Resolution Option 4 – Custom Screen for Lookup views & search

This is another fancy one for which I unfortunately don’t have any screenshot. We want to leverage option sets to “overwrite” lookup values and selection process with the following steps:

  1. Create a set of standardized Web Resources for Lookup Display and lookup value selection
  2. Display the web resources on the CRM Forms and hide the lookup controls
  3. The web resources will have built-in logic to display the value in the language of the current user, as well as a mechanism to allow searches (could be auto-complete based)

Writing a standard web resource control for that purpose is relatively simple. However, you might have additional work to do if you want to take advantage of filtering based on other fields, or custom filters. Also, this solves the issue on the form in the sense that you will see the values in the right language on the form, but for list views, reports etc. the problem will still exist so you’ll need to find another solution there.

Closing Thoughts

As you can see there is no perfect solution. Each organization has to decide the level of investment and risk they want to take to make sure they have multi-language option sets. Living in Canada where we have two official languages, this is a challenge that we often see in public sector implementations because having fully bilingual system is mandated by law. There are very few countries where this is the case (which is probably why Microsoft has not made investments in this area). Most private sector companies will usually impose a primary language for the entire organization.

Hope this helps!

Selecting your Dynamics CRM developer(s) setup

If you are ever involved in a very large CRM implementation project, one of the decision that you need to make early is how to setup/structure your development environments and workstations. Not that I think it’s not as import on smaller project, but it is more critical in larger project.

First let’s start by defining what I mean by a large project (this is a completely personal):

  • Large development team (4+ people)
  • Multiple complex customizations pieces (plugins/custom activities/JavaScript/HTML) and/or integration points
  • Project length: variable, usually 6-8+ months

Last year, I wrote about how to setup your Dynamics CRM/Source Control environment properly for Dynamics CRM projects. Today, I want to focus more on the Development Machines setup and what I have found to be useful for decision making process to set them up.

In the typical CRM project, you will find a Dev CRM organization used for doing system configuration and basic customization. Then, for plugin/JavaScript development, we typically recommend to have either a dedicated VM with CRM installed for each developer, or a dedicated CRM Organization. The reason why we make this recommendation is very simple: development Isolation.

When you are developing say a web site or a mobile app, you don’t deploy your content on the test server or device that everyone uses until it’s ready to be tested. The same applies for CRM. If you have a primary development organization, why would you want developers to deploy work in progress on a regular basis? Plus, if they are working on the same module, they might overwrite each other’s code all the time. The loss of productivity will be at two levels

  • Deployed code that’s not finalized will cause bugs and people won’t be able to continue working, test in the CRM organization normally
  • Developers will likely overwrite each other if they are working on similar or dependent tasks which will result in a major loss of time and productivity

Now that I have sold you on the development isolation idea, here is a list of pros and cons that will help you decide whether you should go the individual CRM Organization way or the individual VM + CRM route.

Individual CRM Organization

  • Pros:
    • Easy to setup
    • (possibly) Part of your corporate domain which makes the access easier for developer and possibly other people if needed
    • Provides isolation for development
  • Cons
    • Developers lack control over the servers (i.e. can’t do IIS reset or restart services etc.)
    • Depending on how powerful your infrastructure is, sharing CRM/SQL Servers with multiple users, the servers might be overloaded quickly and you find yourself with an issue related to sharing your server resources
    • Removes the ability to debug the old fashion way by attaching to IIS/sandbox/async processes
    • Ongoing maintenance of the CRM/SQL server(s) hosting the organizations

clip_image002

Individual VM with CRM/SQL installed

  • Pros:
    • Provides developer with great control over the environment in which they develop
    • Provide isolation for development
    • Provide isolation for Hardware utilization (depending on how VMs are setup)
    • Easy to setup (build a VM once, create a capture and provision on need)
    • All development tools can be installed on the VM and isolated from workstation
    • Easy to do plugin/custom activities debugging by attaching to IIS/sandbox/async processes
    • Easier to perform integration test in an isolated VM
    • Relatively light use hardware resource if the VMs are hosted on developers’ workstations
  • Cons:
    • Possibly Windows OS Server license issue (each developer needs one license)
    • Network configuration can be tricky depending on your environment or Enterprise context (e.g. accessing TFS from VM, expose VM to the internet)
    • Heavy use of hardware resources (especially if you have VM Host server(s))
    • Ongoing maintenance of the VM host server and VMs

SNAGHTMLec1ea7

It’s worth noting that individual VMs can be created on a host server or on developers’ local workstations (laptops or desktop). You may or may not decide to join the VMs to the corporate domain, you could even setup your development environment in the cloud on a Virtual Network which we have done multiple times including for my company. 

Lastly, as noted by Darly Labar, if your project is with CRM Online, running VMs locally will prevent you from having a same version of Dynamics CRM in your developer VMs (CRM Online and On-prem are very similary, but not the same and not always in sync). The same applies for the individual developer CRM organization unless you have a CRM Online Sandbox for each developer or other instances of CRM Online they can use for development purpose.

There you have it. The great thing is that you always have options. From there, it’s making sure you select the one that is appropriate for your needs.

If you need help and guidance throughout this process, feel free to reach out to us at SADAX Technology. We offer “Hour Bucket” packages from 5 to 30 hours for Dynamics CRM and Azure consulting services. Request a quote here!

 

Setting up your Dynamics CRM Development & Source Control Environment

Every time I meet people trying to get CRM projects going, I get plenty of questions about how to setup environments and source control properly. Unfortunately, there is no easy answer. However I thought I would share a model that I’ve used on many occasions and that works well with a combination of a defined methodology, appropriate tools, training, commitment and discipline from the project team members. It is mostly relevant for large projects with multiple developers.

The proposed setup is based on the flowchart below, and it is inspired by fellow MVP Gonzalo Ruiz’s article on the same topic.

 

Development machines (CRM DEV)

The development machines can be a developer’s laptop or work station with CRM development tools + a dedicated CRM organization for the developer (CRM Online or on premise). It can also be a full server (or VM) with CRM installed and dedicated to a single developer. These organizations are setup with a master configuration solution (more on this later) and smaller transport solutions to bring required components for development.

Developers use the various development tools to write code and to debug (attach to CRM services, attach to IIS, using Profiler, browsers debugging tools or using tracing).

When development items are completed, developers build temporary “transport solutions” containing the components they worked on and they push them to the Master Configuration organization. Developers must be careful and not make schema changes on their own development organizations. As the code is deployed to the master configuration, it should also be archived in the source control.

Master Configuration CRM Organization

There is only one Master Configuration environment. It contains the schema configuration and all schema changes should be made there. There is usually one or multiple “Master” solutions sitting in this organization based on how you decide to structure your deployments. You may want to have multiple solutions for various types of customizations (e.g. one for schema, one for plugins, one for web resources etc.), or based on functionality grouping.

When developers set up their organizations, they can pull the entire master solutions from the master configuration organization as unmanaged and install them in their development. They can also create temporary “transport solutions” to pull only the customizations they need.

A process (manual or automated) should be put in place to archive the master solutions on a regular basis (daily is recommended). Archiving could be saving the CRM solution files in source control, or even creating snapshots of the server and keep them around for a reasonable period of time.

Integration / Test CRM Organization

This organization receives the master solutions and is used to perform tests. It can be combined with the master configuration environment. The choice to go with a managed or unmanaged solution when going from master configuration to integration/UAT/Prod environment is a business and technical decision. It is a separate discussion that we could write an entire blog post about J.

UAT / PRE PROD / PROD

These environments receive the master solution(s) after they have been tested and validated on the Integration/Test CRM organization.

Wrap Up

Putting this type of environment in place is not easy to do. It takes a lot of process implementation, tools and automation build and more important, training and discipline from the project team members. When working on small project (small number of developer, low amount of customizations and code), going with this approach is overkill. It applies best for big projects when you have multiple developers and heavy customization requirements.

 

This was fun. Hope it helps!

Dynamics CRM Archiving Solutions Revisited

I wrote about Archiving solutions for Microsoft Dynamics CRM about 2 years ago. I still get a lot of questions from the article and from customers in the field about what they should do when it comes to archiving their Dynamics CRM data. While there is no obvious answer, my generic approach is to make sure my clients understand that they shouldn’t try to build an archiving solution before their system has become slower. They should see it more as an opportunity to perform a thorough review of their implementation and isolate the bottleneck(s) and pain point(s) and see if those can be fixed.

A few months ago, I gave a presentation on the topic that you can see below. I go from the reasons to archive data to recommendations. Hope this helps!

“Only Secure Content is displayed” warning message when viewing CRM forms with IFRAME

After configuring an existing CRM solution to display some external content inside of IFRAMEs, we had to move the CRM web site from port 80 to SSL. After completing the configuration, we ran into a constant warning message in the browsers:

  • Internet Explorer: “Only Secure Content is displayed” with a button to allow all the content to be loaded
  • Chrome: “This page includes script from unauthenticated sources” with a link to load the unsafe script

The warning is actually indicating an obvious point: when you are running a web site configured for HTTPS/SSL, it makes sense from a security perspective to not display frames or other content that is not secured. The resolution is extremely simple, just move your custom content to a secured web site. The warnings will go away and the content of the frame will be displayed.

On the flip side, you can display secured content even if your CRM website is not configured for HTTPS.

Common CRM Solution Package Components

As a CRM Solution Architect, one thing that I often try to do is keep the complete solution package with a minimal amount of components in order to simplify the deployment process. In an ideal world, we want to limit a deployment to a solution deployment. In reality, this never happens on complex implementations. The purpose of this article is to point out some of common elements that are typically deployed as part of a CRM solution deployment.

  • CRM Solution: This is the .ZIP file extracted from your development CRM server that contains the system configuration and customizations. As part of a deployment, you typically always have one or multiple CRM solutions to apply to the target environment
    • Steps: Import CRM solution(s) in your target environment
    • Can be automated: Yes, use CRM SDK
  • CSV Files for data import: There is often a need to load master data on the system. For example, if you have entities for “Country” and “State”, you may want to populate them with data before people start using the system. This is easily done with a CRM data import wizard.
    • Steps: Import CSV or XML files in your target environment
    • Can be automated: Yes, use CRM SDK
  • Run Workflow(s) against a specific set of record: This is a technique that is often used to fix data issues. Examples of use: change record ownership, populate new fields created as part of the current release, delete records (data clean-up).
    • Steps: Define the query that will return the records you need to run your workflow against, find them (Advanced find or FetchXml) and execute the workflow against all records
    • Can be automated: Yes, use CRM SDK
  • Perform data load from external source: Loading data is usually done by external system (SSIS, Scribe, custom integration etc.) that have connectors to CRM. As part of a CRM deployment, you’ll often have to run data load/update tasks after configuring your integration service to point to your production server. This is used to do data migration or to enable on-going integration between multiple information systems.
    • Steps:
      • Deploy integration components and/or configure them to execute on your target environment (e.g. install SSIS packages, configure Scribe etc.);
      • Turn on integration jobs
    • Can be automated: Depends on the integration tools used
  • Deploy custom web services and/or web sites: To provide extended integration with CRM, we often need to create our own web services that expose custom operations involving CRM data. We also frequently see the need to build custom web applications outside of CRM to display complex and/or integrated data within CRM screens or to create external portals. These web services and web apps become part of the release cycle of your CRM solution.
    • Steps: Package web services and web sites; deploy them on target IIS server(s) and configure CRM endpoint(s) in Configuration files as needed
    • Can be automated: Yes, depending on how web apps is built and packaged

When delivering enterprise solution deployment packages, one best practice is to always plan to automate the deployment of all components that be can be automated. It shortens the solution package deployment process, limits the possibilities of human errors and helps make deployment administrators happier which results in them having a better relationship with the solution and product (I speak from experience J).