Adding Custom Payload to Content Hub’s Queue Messages

Introduction

Sitecore Connect for Content Hub allows to sync of Content Hub entities from Content Hub to Sitecore XM/XP by listening to the update notifications in the Message Bus Queue and acting on each notification by syncing the affected entity from Content Hub to Sitecore CMS. Out of the box, those messages in the Service Bus queue don’t have much information about the entity being updated or created so the listening client needs to read entity information from Content Hub via its REST API.

mb1

Challenge

Such an approach allows for great flexibility but comes with one big downside: Content Hub APIs impose a 15 requests per-second throttle limit, which leads to this read process becoming a bottleneck when the number of items to sync grows large (tens of thousands of items).

Solution

One way to solve this is to perform a mass batch sync, which I have described here in this 3-part series:

This approach works well in situations where sync delays are acceptable.

Embedding Content Hub data into Service Bus Messages

For those situations where something closer to real-time sync is desired, I would suggest an alternative approach: embed all needed data right into the message payload so that the listening client can retrieve all content that it needs from the Service Bus message itself, in which case there will be no need to read anything from the Content Hub API.

In order to make this work, the message payload would need to be custom-created on the Content Hub side before the message is sent out to the Service Bus queue.

Let me outline this solution, which I actually implemented for one of our clients.

Here’s how this works:

An Entity of a given type is created or updated in the Content Hub.
An in-process trigger gets triggered to run custom code, which collects & computes needed data and saves it into the above entity record. Think of it as a computed field, which is computed on every change.
1. In-process trigger configuration
1. Triggered action configuration (see below for details on the script linked to it)
The second trigger, this time out-of-process (so that it runs after the 1st one is done) gets fired and triggers the Service Bus message to be sent to the message bus queue. This message’s payload will have all additional fields added as shown below. Note how payload fields are configured to be added to the Service Bus message as name-value pairs where values in parenthesis are the actual entity fields. Please refer to this blog post for more details on adding custom fields to the message payload: How to Send Messages from Content Hub to Azure Service Bus. **
The listener on the client side of the Service Bus queue retrieves the message, reads all the data, and processes it as needed.

A custom code example to compute custom field values.

This is just an example, which I am providing for illustration purposes. Please feel free to use this as a starting point in your custom implementation. In this case, I’m retrieving the values I need from some of the related entities and then saving them back into a given entity so those field values can then be added to the Service Bus message payload, as described above.

The following is an entire script file with many comments to help make it clear.

using System.Linq;
using System.Threading.Tasks;
using System.Collections.Generic;
using Stylelabs.M.Base.Querying;
using Stylelabs.M.Base.Querying.Filters;
using Stylelabs.M.Framework.Essentials.LoadConfigurations;
using Stylelabs.M.Framework.Essentials.LoadOptions;
using Stylelabs.M.Sdk.Contracts.Base;
using Newtonsoft.Json;
using System.Globalization;

public class AssetDetails
{
    public long Id;
    public string Title;
    public string Description;
    public Dictionary<string, string> PublicLinks = new Dictionary<string, string>();
}

MClient.Logger.Info($"Start Extracting tags for MH.AssetGallery.");

var deliveryHost = "https://mhc-q-002.sitecorecontenthub.cloud";
var renditionName = "downloadOriginal";

var targetEntity = Context.Target as IEntity;
if (targetEntity == null)
{
    MClient.Logger.Debug($"Error extracting info for Asset Gallery: MH.AssetGallery is null.");
    return;
}

// Ensure the following members are loaded  
await targetEntity.LoadMembersAsync(new PropertyLoadOption("Title"), RelationLoadOption.All);

var title = targetEntity.GetProperty<ICultureInsensitiveProperty>("Title");
MClient.Logger.Debug("targetEntity title: " + title);
var titleValue = title?.GetValue<string>();
MClient.Logger.Debug($"titleValue value: {titleValue}");

var galleriesToMetroAreasRelation = targetEntity.GetRelation("GalleriesToMetroAreas");
var galleriesToMetroAreasIDs = galleriesToMetroAreasRelation.GetIds();
var metroAreaIDs = string.Join("|", galleriesToMetroAreasIDs);
targetEntity.SetPropertyValue("MetroAreaIDs", metroAreaIDs);
MClient.Logger.Debug($"MetroAreaIDs value: {metroAreaIDs}");

var galleriesToCommunitiesRelation = targetEntity.GetRelation("GalleriesToCommunities");
var galleriesToCommunitiesIDs = galleriesToCommunitiesRelation.GetIds();
var communityIDs = string.Join("|", galleriesToCommunitiesIDs);
targetEntity.SetPropertyValue("CommunityIDs", communityIDs);
MClient.Logger.Debug($"CommunityIDs value: {communityIDs}");

var galleriesToFloorPlansRelation = targetEntity.GetRelation("GalleriesToFloorPlans");
var galleriesToFloorPlansIDs = galleriesToFloorPlansRelation.GetIds();
var floorPlanIDs = string.Join("|", galleriesToFloorPlansIDs);
targetEntity.SetPropertyValue("FloorPlanIDs", floorPlanIDs);
MClient.Logger.Debug($"FloorPlanIDs value: {floorPlanIDs}");

var galleriesToLotsRelation = targetEntity.GetRelation("GalleriesToLots");
var galleriesToLotsIDs = galleriesToLotsRelation.GetIds();
var lotIDs = string.Join("|", galleriesToLotsIDs);
targetEntity.SetPropertyValue("LotIDs", lotIDs);
MClient.Logger.Debug($"LotIDs value: {lotIDs}");

//Load Asset public links
var galleriesToAssetsRelation = targetEntity.GetRelation("GalleriesToAssets");
var galleriesToAssetsIDs = galleriesToAssetsRelation.GetIds();
var assetIDs = string.Join("|", galleriesToAssetsIDs);
targetEntity.SetPropertyValue("AssetIDs", assetIDs);
MClient.Logger.Debug($"AssetIDs value: {assetIDs}");

var assetPublicLinks = new List<string>();
var assetObjects = new List<AssetDetails>();
foreach(var assetID in galleriesToAssetsIDs)
{
    var asset = await MClient.Entities.GetAsync(assetID);
    if(asset != null)
    { 

        var assetDetails = new AssetDetails();
        assetDetails.Id = asset.Id != null ? asset.Id.Value : -1;
        assetDetails.Title = asset.GetPropertyValue<string>("Title");
        CultureInfo enUs = CultureInfo.GetCultureInfo("en-US");
        assetDetails.Description = asset.GetPropertyValue<string>("Description", enUs);
        //IParentToManyChildrenRelation publicLinksRelation = asset.GetRelation<IParentToManyChildrenRelation>("AssetToPublicLink");

        var renditionFallback = string.IsNullOrEmpty(renditionName) ? "downloadoriginal" : renditionName.ToLower();
        var query = new Query()
        {
            Filter = new CompositeQueryFilter()
            {
                Children = new QueryFilter[] {
                    new DefinitionQueryFilter() {
                        Name = "M.PublicLink"
                    },
                    new RelationQueryFilter() {
                        ParentId = asset.Id,
                        Relation = "AssetToPublicLink"
                    },
                    new PropertyQueryFilter() {
                        Property = "Resource",
                        Value = renditionFallback
                    }
                }
            }
        };

        var relatedEntityItems = (await MClient.Querying.QueryAsync(query, EntityLoadConfiguration.Full).ConfigureAwait(false))?.Items;

        var publicLink = relatedEntityItems?.FirstOrDefault();
        
        if(publicLink != null)
        {             
            var resource = publicLink.GetPropertyValue<string>("Resource");       
            var relativeUrl = publicLink.GetPropertyValue<string>("RelativeUrl");             
            var versionHash = publicLink.GetPropertyValue<string>("VersionHash");    
            var publicLinkUrl = $"{deliveryHost}/api/public/content/{relativeUrl}?v={versionHash}";
            assetDetails.PublicLinks[resource] = publicLinkUrl;
        }

        assetObjects.Add(assetDetails);
    }
    
}

var assetJson = JsonConvert.SerializeObject( assetObjects );
targetEntity.SetPropertyValue("AssetPublicLinks", assetJson);
MClient.Logger.Debug($"AssetPublicLinks value: {assetJson}");

await MClient.Entities.SaveAsync(targetEntity);

Azure

Sergey Yatsenko