OrderCloud CSV Importer

Introduction

As a headless API designed to be streamlined and extensible , OrderCloud brings many features to the table that makes integration as fast and simple as possible for developers. That said, while OrderCloud has numerous features that fulfill most use cases, there are some features some developers might find the platform lacks. One important feature is the ability to send bulk requests to the API.

While most of OrderCloud’s POST and PUT related requests demonstrate performant results, it’s still very limited to only sending requests one at a time. This creates a problem for important use cases like updating a large catalog with thousands of products. Luckily, OrderCloud provides a .NET package that makes a workaround easier.

In this blog, we’ll be implementing a performant solution for the importation of large product catalogs stored in CSV files.

This example requires some NuGet Packages, including:

OrderCloud.SDK

ordercloud-dotnet-catalyst

CsvHelper

A basic CSV file will need to represent OrderCloud’s product model and look something like this:

In this example, each header field (OwnerID, DefaultPriceScheduleID, etc.) matches exactly with OrderCloud’s model properties. In cases where more advanced mapping options are needed, CsvHelper’s documentation provides many examples.

For our first step, we’ll need to parse the incoming CSV form from within the .NET application. We can do so by using CsvHelper’s model mapping features.

In an MVC based .NET project, create a controller with a form action.

Example form action PUT request with CsvHelper parsing:

HomeController.cs

[HttpPut("UpdateProducts")]
public async Task UpdateProducts(IFormFile formFile)
{
     var products = new List<Product>();
     using (var reader = new StreamReader(formFile.OpenReadStream()))
     using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))
     {
        products = csv.GetRecords<Product>().ToList();
     }
}

Next, we’ll create a function that will also save the resulting products by making the corresponding API calls to OrderCloud.

HomeController.cs

private async Task SaveProduct(Product product)
{
    try
    {
        await _context.Products.SaveAsync<Product>(product.ID, product);
    }
    catch
    {
        // Print exceptions here.
    }
}

We’ll now finish the UpdateProducts function using the Throttler from OrderCloud.Catalyst.

[HttpPut("UpdateProducts")]
public async Task UpdateProducts(IFormFile formFile)
{
    var products = new List<Product>();
    using (var reader = new StreamReader(formFile.OpenReadStream()))
    using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))
    {
        products = csv.GetRecords<Product>().ToList();
    }   
   await Throttler.RunAsync(products, minPause: 100, maxConcurrent: 20,
       async pricing => await SaveProduct(pricing));
}

Catalyst’s Throttler class ensures that the number of requests won’t trigger OrderCloud’s server throttler. If we had used the default .NET parallel Tasks library without any kind of throttler monitoring, the number of requests hitting OrderCloud’s server would trigger the throttler and slow down the update to a crawl. If the number of products is large enough, it may throw a stack overflow exception because of the number of tasks waiting to finish.

If needed, the minimum pause time (in milliseconds) and maximum concurrent threads may be adjusted to increase performance. You will need to play around with the settings to find the fastest option for your use case.