CH Upstream Integrations part 4: Useful Building Blocks for building Cloud Integrations with Sitecore Content Hub

Intro

A collection of my notes describing some of the readily available building blocks, provided by Microsoft Azure, which I found to be very useful in building integration flows with Content Hub using Azure Logic Apps and some of the readily available building blocks in Azure.

Read Sergey's blog posts at xcentium.com

The following is a collection of my own notes describing some of the readily available building blocks, provided by Microsoft Azure, which I found to be very useful in building integration flows with Content Hub using Azure Logic Apps and some of the readily available building blocks in Azure. Microsoft and Sitecore provide very good documentation on the below-described Content Hub integration hooks, Azure Functions, Logic Apps, Service Bus Queues, and API Management - so I won't attempt to repeat any of that information here. My aim is to provide some hints and hopefully useful ideas on how Azure features and building blocks can be used in Content Hub integrations.

---

This post is the 1st part in a 4-piece series, describing an integration approach that allows to. Content Hub and connect Sitecore with pretty much any external system. I'm composing a number of Azure Functions, both provided by Azure and my custom-built ones to build integration flows using Azure Logic Apps. Upstream integrations pull data from one or more external systems, extracted, transform and process as needed, and pushed changes into Sitecore Content Hub via its APIs. I chose to use Content Hub Web Client SDK, which is a .NET abstraction on top of the Sitecore Content Hub REST API because it simplifies the development of .NET client code and helps to deal with CH API throttling, and does a few more helpful things.

I believe Logic Apps is a good way to visually orchestrate various building blocks (Azure Functions) together with very little to no code required: easy to build, and easy to change, but of course, this isn't the only way. I'm sharing all source code here, so others can use it for building the custom integration solutions for Sitecore Content Hub.

All posts in this series:

• Part 1: Describes the UpsertEntity function, which will update or create a new entity from the payload data
• Part 2: Describes the UpsertRelation Azure Function, which looks up two Entities to be associated in Content Hub and creates or updates a relation between them.
• Part 3: Example Azure Logic App, combining above functions into example integration flow, which updates and/or creates entities and relations between them in Content Hub as source data is changing (in close to the real-time manner)
• Part 4 (this post): Useful Azure Building Blocks for building Cloud Integrations with Sitecore Content Hub

It's worth noting Sitecore that recently announced Sitecore Connect along with other new great products, so consider using Sitecore Connect before implementing your custom solution.

---

Content Hub Building Blocks

Custom actions and Integration Hooks

Content Hub provides scripts, actions, and triggers, which are often used together to extend the system behavior in response to events and user actions. Those can be used to either run some additional processing inside Content Hub or as integration hooks to get Content Hub to talk to external systems. Various downstream integrations, where Content Hub would "publish" its content to any 3rd party system in a close to real-time manner would usually leverage such integration hooks.

Scripts

Content Hub Scripts are pieces of .NET code, which can be executed in Content Hub to implement small bits of custom business logic. Inside a script, various context properties, such as information about the entity or user being processed, can be accessed. Here is a good collection of scripting use cases with sample code.

Any serious, resource-heavy processing better be offloaded to an external system or processing pipeline, e.g. Azure Logic Apps, Azure Batch, or any 3rd party app or service, which would receive a message and/or data from Content Hub and perform necessary processing.

Actions

Content Hub Actions are like functions or rather... tasks, units of work, that can do something - e.g. execute a script, invoke API, send messages to the Service Bus queue, and more. Combined with Triggers they can be configured to run in response to specific events. Actions can be very useful in downstream integration scenarios, where Content Hub would push content or a message to 3rd party system or a publishing destination.

One of the action types is to trigger the script, which in conjunction with triggers can be used to run custom code in response to pre-defined events in Content Hub.

Another type of action, worth mentioning is Azure Service Bus Action, which allows sending messages to Service Bus for pickup and further processing by other apps and services in a decoupled manager. Various connectors like Sitecore Connect for Content Hub and custom integrations use Service Bus to receive messages about content changes in Content Hub.

Triggers

Content Hub triggers are similar to database triggers in a way they are meant to trigger specific events or actions in response to something happening in the system, e.g. an entity had been updated/created/deleted.

Triggers can be run in process for the fastest response time or out of the process to decouple event processing from the event and prevent delays on the original event thread. Unless absolutely necessary, the out-of-process mode is recommended.

A few more useful bits...

Content Hub Throttle Limits

Similarly to other Cloud apps and services, Content Hub does throttle its API endpoints to protect its backend from being overwhelmed by high-volume external calls. Currently, the throttle rate limit is 15 calls per second, and requests going above this limit will return the 429 error code.

Content Hub WebSDK has an internal logic, which automatically deals with throttling and retries calls as needed behind the scenes. For high-volume integrations consider below bulk import and/or using Azure Service Bus.

Bulk Import/Export with Excel

Excel import allows feeding large amounts of content into Sitecore Content Hub without hitting throttling and at a much higher speed compared to sending it via Content Hub API.

Content Hub also provides an endpoint to export any kind of entity into Excel - following is an example call to download all entities of type M.PCM.Product to Excel. Once the request is submitted and accepted by your Content Hub instance - give it some time to process then log in with the credentials of the user on which behalf this request was submitted, head to downloads, and download the Excel file from there.

POST the below JSON payload to https://{{hostname}}/api/commands/portal/download.excel

{
    "query": "Definition.Name=='M.PCM.Product'",
    "cultures": [
        "en-US"
    ],
    "type": "Data",
    "request_uri": "https://yourinstance.cloud/",
    "filename": "M.PCM.Product",
    "userFriendlyColumnHeaders": true,
    "userFriendlyCellValues": false
}

---

Azure Building Blocks

Azure Service Bus

Azure Service Bus allows enabling reliable and elastic messaging between systems in a decoupled manner. As I mentioned above Content Hub has built-in integration hooks for Azure Service Bus, which comes in very handy in downstream integration scenarios. Microsoft provides extensive developer documentation on Azure Service Bus.

Service Bus Explorers

One helpful hint on exploring the contents of the Service bus. There are multiple options, of which I ended up mostly using the following two:

• Service Bus Explorer in Azure Portal is not generally available and is the simplest option, which is good enough, provided you have access to a given Service Bus Queue in Azure portal
• Service Bus Explorer app for Windows. This one is a bit old, but may still be a viable alternative in case there's no access to the Azure portal.

Azure Functions

Azure Functions is a serverless solution that allows you to write less code, maintain less infrastructure, and save on costs.

I described a couple of custom Azure functions for Sitecore in post 1 and post 2. Here are some useful tips on Azure Functions:

• Develop Azure Functions using Visual Studio
• Azure Function configuration and management
• Logging and monitoring
• Log streaming

Azure Logic Apps

"Azure Logic Apps is a cloud platform where you can create and run automated workflows with little to no code. By using the visual designer and selecting from prebuilt operations, you can quickly build a workflow that integrates and manages your apps, data, services, and systems"

I described an example Logic App in this post, following are the few additional things I found to be useful and thought are worth (re)sharing here.

Triggers

Azure Logic Apps come with a large variety of ways to initiate and execute. In Content Hub integration scenarios I mostly used Service Bus, HTTP(s) request, and API Management to trigger the execution of Logic Apps.

Connectors

Connectors provide a way for a Logic App to connect and interact with external apps and services. HTTP call, Custom Azure Functions, for example, UpsertEntity and UpsertRelation, and myriads of built in and 3-rd party connectors.

Azure Function Execution parameters, error handling and retry logic

Azure Logic Apps allow configuring error handling and retrying logic without writing extra code, which comes very handy in many comon scenarios where simple error-handling and retry logic are needed. Following is an example configuration of my custom UpserEntity function, executed by Example Logic App.

Logic App Plans: Consumption Standard vs Consumption

Azure Logic Apps come in two flavors/pricing tiers: Consumption and Standard. For some reason, the visual editor editing experience on the standard tier is different and, in my opinion, inferior to a lower tier Consumption one, as it is faster and seems much more reliable. There's also no simple way of migrating Consumption-based Logic Apps into the Standard tier, for which I had to resort to tweaking the JSON of the logic app (taken from its code view) to make it match the expected format.

Azure API Management

Azure API Management service allows you to connect different platforms and apps without having to write code to handle format differences, authentication, throttling, request transformations, and much more. I found this to be a very useful tool in both upstream and downstream integration scenarios with Content Hub and other apps and services. Following is an example of API integration, where the est URL is getting rewritten, request segments are copied into headers, and throttle limits are getting applied to incoming requests. There are many more useful features, which can be enabled through various settings and policies...