Fixing Memory Leak Inside of DatabaseCacheDisabler

Introduction

When dealing with large updates to Sitecore items, typically seen in batch process like imports or exports, it is common to see memory (RAM) go way up. This is because as items are retrieved from the database, Sitecore adds them to the prefetch cache, which is stored in memory. While the caching is ideal in most scenarios, large batch operations rarely operate on the same item multiple times and can be optimized by disabling this cache. In our case, we were working with a client who had a global rollout, 10 sites, and content authors logging in from around the world. The batch process that managed all the data, which ran periodically throughout the day on our CM instance, had a hefty job of updating a catalog with 100k+ individual Sitecore items. We were running Sitecore 9.3, but it still seems to be an issue in 10.2, and likely also the case for older versions.

Problem

Performance and efficiency were top priorities during our batch process. After seeing RAM ramp way up during the process and crashing our CM server, we intuitively added the DatabaseCacheDisabler. While it did help, we saw that memory would always crawl right back up until the server would crash. After looking at the problem from many different angles, all our memory dumps always pointed right back to the Prefetch cache growing out of control.

So we started digging and found that the data provider Sitecore.Data.DataProviders.Sql stored the items in an ItemsContext if the DatabaseCacheDisabler was used.

The problem here is that the internal cache was never cleared. As a result, these internal cache items would continue to grow, and if you’re processing thousands of items regularly, you will continue to consume more and more RAM until the server’s resources were completely exhausted.

Here is a view of how the memory (sky blue line) was stable at ~50% and then started to climb throughout the day, getting stuck at the high 90s, and eventually crashing the server. Even when it didn’t crash the server, it would cause a terrible content authoring experience.

Solution

We reached out to Sitecore to get their feedback on our findings and they provided two options:

1.      Remove DatabaseCacheDisabler

2.      Reset the inner cache

Removing the DatabaseCacheDisabler was not an option for us, as we added it specifically to reduce memory usage on our CM instance.

The second option, resetting the inner cache, was the solution for us. After the DatabaseCacheDisabler block, we can reach into Context.Items, get the reference to the internal dictionary, and clear it.

using (new Sitecore.Data.DatabaseCacheDisabler())
{
...
}
var dictionary = Sitecore.Context.Items["itemInnerCache"] as Dictionary<Sitecore.Data.ID, Sitecore.Data.DataProviders.PrefetchData>;
dictionary.Clear();

Once we implemented the reset of the inner cache, all of our memory leaks were resolved, and the servers have been healthy ever since!

Here is how our memory was doing after the change

Now, we never pass out typical 50-70% thresholds and the CM instance always remains responsive.

Conclusion

Optimizing a memory-consuming process can be tricky. Even when implementing the methods Sitecore provides, like DatabaseCacheDisabler, you can run into unintuitive issues. Luckily, we were able to track down the problem, and with the help of Sitecore, find a solution to keep our process optimized and servers running healthy. The best part was that the solution didn’t require any infrastructure changes, so it saved the client both money and time.