Enable CBOR within ASP.NET Core OData

Introduction

CBOR, which stands for Concise Binary Object Representation, is a data format whose design goals include the possibility of extremely small code size, small message size, and extensibility without the need for version negotiation (from cbor.io). CBOR is based on the wildly successful JSON (aka, JavaScript Object Notation) data model, and is a binary data representation of JSON. Here’s an example of a simple JSON value in plain text and binary representation.

OData library, by default, uses the plain text JSON representation for OData requests and response data serialization (writing) and deserialization (reading). But it’s also designed to enable developers to customize other JSON representations. I’d use this post to share with you the process to enable CBOR representation serialization and deserialization using ASP.NET Core OData. Let’s get started.

Prerequisites

Let’s start to create an ASP.NET Core Web API application named ODataCborExample with Microsoft.AspNetCore.OData (version-8.0.12) installed. For simplicity, I reuse the ‘Book’ entity and its controller from this post. I’m omitting those codes in this post and please refer to here for detailed implementations.

I need a library to finish the CBOR binary data reading and writing. You can find various implementations of CBOR for most computer languages from here. In this post, I choose to use System.Formats.Cbor, since it’s the built-in solution for .Net and everyone can install/consume this package from Nuget.org. Here’s ODataCborExample project configuration:

CBOR serialization

System.Formats.Cbor provides CborWriter to finish CBOR encoded JSON binary writing. To inject CborWriter into OData JSON serialization, we should customize an OData JSON writer to use it.

CBOR OData JSON writer

The OData JSON serialization process relies on two interfaces:

• public interface IJsonWriter {}
• public interface IJsonWriterAsync {}

Let’s create the following class named CborODataWriter which implements the above two interfaces as below:

using System.Formats.Cbor;

public partial class CborODataWriter : IJsonWriter, IJsonWriterAsync
{
    private CborWriter _cborWriter;
    private Stream _stream;
    private bool _isIeee754Compatible;
    private Encoding _encoding;

    public CborODataWriter(Stream stream, bool isIeee754Compatible, Encoding encoding)
    {
        _stream = stream;
        _isIeee754Compatible = isIeee754Compatible;
        _encoding = encoding;
        _cborWriter = new CborWriter();
    }
}

The constructor of CborODataWriter accepts the writing stream, an encoding and Ieee754Compatible Boolean value (the last two are not used in this post for simplicity) and creates an instance of System.Formats.Cbor.CborWriter.

We must implement the methods defined in the OData JSON writer interface. The implementation is straightforward as:

public partial class CborODataWriter : IJsonWriter, IJsonWriterAsync
{
    // ……
    public void StartArrayScope()
    {
        _cborWriter.WriteStartArray(null);
    }

    public void StartObjectScope()
    {
        _cborWriter.WriteStartMap(null);
    }

    public void EndArrayScope()
    {
        _cborWriter.WriteEndArray();
    }

    public void EndObjectScope()
    {
        _cborWriter.WriteEndMap();
    }

    public void WriteValue(bool value)
    {
        _cborWriter.WriteBoolean(value);
    }

    // …… Omit other writing methods.
    public Task StartArrayScopeAsync()
    {
        StartArrayScope();
        return Task.CompletedTask;
    }

    public Task StartObjectScopeAsync()
    {
        StartObjectScope();
        return Task.CompletedTask;
    }

    // …… omit other async methods.
}

We simply delegate each writing action in CborODataWriter to the corresponding writing method in CborWriter since CBOR is the binary representation of JSON.

Here’s a simple mapping table for your reference between OData JSON writing methods and CBOR writing methods:

OData JSON Writer	CborWriter	Things to write
StartArrayScope()	WriteStartArray()	[
EndArrayScope()	WriteEndArray()	]
StartObjectScope()	WriteStartMap()	{
EndObjectScope()	WriteEndMap()	}
WriteValue(bool value)	WriteBoolean(value)	true/false
…	…	…

For the async methods, let’s simply call the synchronous methods respectively.

CBOR OData JSON Writer factory

Once we have CborODataWriter, we must register it into the OData JSON serialization process. OData library uses the Factory pattern to inject JSON writer. So far, the OData library has the following three factories to “build” JSON writer.

• public interface IJsonWriterFactory
• public interface IJsonWriterFactoryAsync
• public interface IStreamBasedJsonWriterFactory

I’d use IStreamBasedJsonWriterFactory to build the CBOR OData JSON writer factory since it is consumed prior to the other two, and most importantly, it provides the Stream which is needed in CborODataWriter.

Here’s my CBOR OData JSON writer factory:

public class CborODataJsonWriterFactory : IStreamBasedJsonWriterFactory
{
    private ODataMessageInfo _messageInfo;
    private IJsonWriterFactoryAsync _jsonWriterFactory;

    public CborODataJsonWriterFactory(ODataMessageInfo messageInfo, IJsonWriterFactoryAsync jsonWriterFactory)
    {
        _messageInfo = messageInfo;
        _jsonWriterFactory = jsonWriterFactory;
    }

    public IJsonWriter CreateJsonWriter(Stream stream, bool isIeee754Compatible, Encoding encoding)
        => throw new NotImplementedException();

    public IJsonWriterAsync CreateAsynchronousJsonWriter(Stream stream, bool isIeee754Compatible, Encoding encoding)
    {
        if (_messageInfo.MediaType.Type == "application" && _messageInfo.MediaType.SubType == "cbor")
        {
            return new CborODataWriter(stream, isIeee754Compatible, encoding);
        }

        // delegate to default JSON writer
        TextWriter textWriter = new StreamWriter(stream, encoding);
        return _jsonWriterFactory.CreateAsynchronousJsonWriter(textWriter, isIeee754Compatible);
    }
}

Where,

1. We only need to implement async interface method ‘CreateAsynchronousJsonWriter’
2. An ODataMessageInfo is injected into the constructor to provide the media type which is used to create plain text JSON writer or CBOR JSON writer.
3. An IJsonWriterFactoryAsync is also injected into the constructor to create plain text JSON writer if the media type is not application/cbor.

We can inject this factory into the service container at startup as:

builder.Services.AddControllers().
    AddOData(opt =>
        opt.AddRouteComponents("odata", EdmModelBuilder.GetEdmModel(),
           services => services.AddScoped<IStreamBasedJsonWriterFactory, CborODataJsonWriterFactory>());

CBOR Media Type resolver

A media type resolver is needed to resolve ODataFormat based on the application/cbor media type. Here’s an implementation for your reference:

public class CborMediaTypeResolver : ODataMediaTypeResolver
{
    private readonly ODataMediaTypeFormat[] _mediaTypeFormats =
    {
        new ODataMediaTypeFormat(new ODataMediaType("application", "cbor"), ODataFormat.Json)
    };

    public override IEnumerable<ODataMediaTypeFormat> GetMediaTypeFormats(ODataPayloadKind payloadKind)
    {
        if (payloadKind == ODataPayloadKind.Resource || payloadKind == ODataPayloadKind.ResourceSet)
        {
            return _mediaTypeFormats.Concat(base.GetMediaTypeFormats(payloadKind));
        }

        return base.GetMediaTypeFormats(payloadKind);
    }
}

Where we build a mapping between application/cbor and OData JSON format.

We also need to inject it into the service container at startup as:

builder.Services.AddControllers().
    AddOData(opt =>
        opt.AddRouteComponents("odata", EdmModelBuilder.GetEdmModel(), services =>
            services.AddScoped<IStreamBasedJsonWriterFactory, CborODataJsonWriterFactory>()
                    .AddSingleton<ODataMediaTypeResolver>(sp => new CborMediaTypeResolver());

We register the writer factor using “Scoped” because the factory needs the ODataMediaType which is scoped.

We also need to let ODataOutputFormatter to understand application/cbor media type as:

builder.Services.AddControllers(opt =>
{
    var odataFormatter = opt.OutputFormatters.OfType<ODataOutputFormatter>().First();
    odataFormatter.SupportedMediaTypes.Add("application/cbor");
});

CBOR writing test

Now, let’s run our sample to test/verify the OData CBOR serialization.

By default, we send “GET http://localhost:5015/odata/books/1” without request header setting, we can get plain text JSON response payload as:

Let’s resend “GET http://localhost:5015/odata/books/1” with request header “Accept=application/cbor”, we can get:

The response body is unreadable since it’s binary data. You can use any other technique to parse the binary data from the response body.

Here’s a trick just for your reference. We can change the Flush() method in CborODataWriter to output the binary data as base64 string as:

public void Flush()
{
    if (!_cborWriter.IsWriteCompleted)
    {
        return;
    }

    var encode = _cborWriter.Encode();

    // if you want to get the Base64 string, use the following codes
    TextWriter sw = new StreamWriter(_stream);
    sw.Write(Convert.ToBase64String(encode));
    sw.Flush();

    // Be noted, if you write it as base64, please comment out the following codes.
    // _stream.Write(encode, 0, encode.Length);
    // _stream.Flush();

    _cborWriter.Reset();
}

Now, send the request and we can get the following base64 string:

v25Ab2RhdGEuY29udGV4dHgzaHR0cDovL2xvY2FsaG9zdDo1MDE1L29kYXRhLyRtZXRhZGF0YSNCb29rcy8kZW50aXR5YklkAWVUaXRsZWQxOTg0ZkF1dGhvcm1HZW9yZ2UgT3J3ZWxsZElTQk5xOTc4LTAtNDUxLTUyNDkzLTVlUGFnZXMZAQz/

Decode the base64 string, we can get the byte array data as:

bf 6e 40 6f 64 61 74 61 2e 63 6f 6e 74 65 78 74 78 33 68 74 74 70 3a 2f 2f 6c 6f 63 61 6c 68 6f 73 74 3a 35 30 31 35 2f 6f 64 61 74 61 2f 24 6d 65 74 61 64 61 74 61 23 42 6f 6f 6b 73 2f 24 65 6e 74 69 74 79 62 49 64 01 65 54 69 74 6c 65 64 31 39 38 34 66 41 75 74 68 6f 72 6d 47 65 6f 72 67 65 20 4f 72 77 65 6c 6c 64 49 53 42 4e 71 39 37 38 2d 30 2d 34 35 31 2d 35 32 34 39 33 2d 35 65 50 61 67 65 73 19 01 0c ff

Using any CBOR viewer tooling to decode the above byte array data, we can get the following plain text JSON, it’s the same as the default OData plain text JSON representation.

{
  "@odata.context": "http://localhost:5015/odata/$metadata#Books/$entity",
  "Id": 1,
  "Title": "1984",
  "Author": "George Orwell",
  "ISBN": "978-0-451-52493-5",
  "Pages": 268
}

Here’s the response data size comparison:

We can save ’20B’ even for such a simple entity, decreasing 12% the message size around.

CBOR deserialization

Same as CBOR writing, System.Formats.Cbor provides CborReader to finish CBOR encoded JSON binary reading. To inject CborReader into OData JSON deserialization, we should customize an OData JSON reader to use it.

CBOR OData JSON reader

The OData JSON deserialization process relies on two interfaces:

• public interface IJsonReader {}
• public interface IJsonReaderAsync {}

Let’s create the following class named CborODataReader which implements the above two interfaces as below:

using System.Formats.Cbor;
public partial class CborODataReader : IJsonReader, IJsonReaderAsync
{
    private Stack<JsonNodeType> _scopes;
    private CborReader _cborReader;
    public CborODataReader(Stream stream)
    {
        byte[] data = ReadAllBytes(stream);
        _cborReader = new CborReader(new ReadOnlyMemory<byte>(data));
        _scopes = new Stack<JsonNodeType>();
    }

    public object Value { get; private set; }

    public JsonNodeType NodeType { get; private set; } = JsonNodeType.None;

    public bool Read()
    {
        // Omit here, see below
    }

    public Task<object> GetValueAsync()
    {
        return Task.FromResult(Value);
    }

    public Task<bool> ReadAsync()
    {
        bool result = Read();
        return Task.FromResult(result);
    }

    private static byte[] ReadAllBytes(Stream instream)
    {
        // …… omit codes
    }
}

Where, a stack of JsonNodeType is used to identify which node should be marked as “property” within a JSON object.

OData JSON reader looks like a state machine. OData reading process calls Read() method every time when querying current JsonNodeType and its corresponding value, then move the reader to the next state until hit the end of input. For simplicity, we can implement the Read() method based on the CborReaderState as:

public bool Read()
{
    CborReaderState readerState = _cborReader.PeekState();
    switch (readerState)
    {
        case CborReaderState.Finished:
            NodeType = JsonNodeType.EndOfInput;
            return false;

        case CborReaderState.StartArray:
            _cborReader.ReadStartArray();
            NodeType = JsonNodeType.StartArray;
            _scopes.Push(JsonNodeType.StartArray);
            return true;

        case CborReaderState.EndArray:
            _cborReader.ReadEndArray();
            NodeType = JsonNodeType.EndArray;
            _scopes.Pop();
            return true;

        case CborReaderState.StartMap:
            _cborReader.ReadStartMap();
            NodeType = JsonNodeType.StartObject;
            _scopes.Push(JsonNodeType.StartObject);
            return true;

        case CborReaderState.EndMap:
            _cborReader.ReadEndMap();
            NodeType = JsonNodeType.EndObject;
            _scopes.Pop();
            return true;

        case CborReaderState.UnsignedInteger:
            NodeType = ParsePrimitiveValueNodeType();
            Value = _cborReader.ReadUInt32();
            return true;

        case CborReaderState.TextString:
            NodeType = ParsePrimitiveValueNodeType();
            Value = _cborReader.ReadTextString();
            return true;

        case CborReaderState.Boolean:
            NodeType = ParsePrimitiveValueNodeType();
            Value = _cborReader.ReadBoolean();
            return true;

        case CborReaderState.ByteString:
            NodeType = ParsePrimitiveValueNodeType();
            Value = _cborReader.ReadByteString();
            return true;

        default:
            throw new NotImplementedException($"Not implemented, please add more case to handle {readerState}");
    }
}

Where, ParsePrimitiveValueNodeType is used to identify the JsonNodeType.Property if the primitive value is first within a JSON object, See the sample here for implementation.

Here’s the mapping between CborReaderStart and OData JsonNodeType:

CborReaderState	OData JSON Node Type	Note
StartArray	StartArray	[
EndArray	EndArray	]
StartMap	StartObject	{
EndMap	EndObject	}
UnsignedInteger ByteString TextString ……	Property	If it’s first within JSON Object
	PrimitiveValue	Others
Finished	EndOfInput

For the async methods, let’s simply call the synchronous methods respectively.

CBOR OData JSON Reader factory

Same as the writer factory, the OData library also uses the Factory pattern to inject JSON reader. So far, the OData library has the following two factories to “build” JSON reader.

• public interface IJsonReaderFactory
• public interface IJsonReaderFactoryAsync

I’d use IJsonReaderFactory to build the CBOR OData JSON reader factory (The async version has not fully finished in the current OData library). Here it is.

public class CborODataJsonReaderFactory : IJsonReaderFactory
{
    private ODataMessageInfo _messageInfo;
    private IJsonReaderFactory _innerFactory;

    public CborODataJsonReaderFactory(ODataMessageInfo messageInfo, IJsonReaderFactory innerFactory)
    {
        _messageInfo = messageInfo;
        _innerFactory = innerFactory;
    }

    public IJsonReader CreateJsonReader(TextReader textReader, bool isIeee754Compatible)
    {
        if (_messageInfo.MediaType.Type == "application" && _messageInfo.MediaType.SubType == "cbor")
        {
            StreamReader reader = textReader as StreamReader;
            return new CborODataReader(reader.BaseStream);
        }

        return _innerFactory.CreateJsonReader(textReader, isIeee754Compatible);
    }
}

Where,

1. We only need to implement the synchronous interface method CreateJsonReader.
2. An ODataMessageInfo is also injected into the constructor to provide the media type which is used to create plain text JSON reader or CBOR JSON reader.
3. An IJsonReaderFactory is also injected into the constructor to create a default JSON reader if the media type is not application/cbor.

We can inject this factory into the service container at startup. However, as mentioned, we need IJsonReaderFactory in CborODataJsonReaderFactory to create the default JSON reader if the media type is not CBOR. So, we need more codes to inject the CBOR JSON reader factory into the service container as below:

builder.Services.AddControllers().
    AddOData(opt =>
        opt.EnableQueryFeatures()
           .AddRouteComponents("odata", EdmModelBuilder.GetEdmModel(),
              services =>
              {
                  // for JSON Reader factory
                  var selector = services.First(s => s.ServiceType == typeof(IJsonReaderFactory));
                  services.Remove(selector);
                  services.Add(new ServiceDescriptor(selector.ImplementationType, implementationType: selector.ImplementationType, lifetime: Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton));
                  services.AddScoped<IJsonReaderFactory>(s =>
                  {
                       return new CborODataJsonReaderFactory(
                           s.GetRequiredService<ODataMessageInfo>(),
                           (IJsonReaderFactory)s.GetRequiredService(selector.ImplementationType));
                  });

                // For JSON Writer factory
                services
                    .AddScoped<IStreamBasedJsonWriterFactory, CborODataJsonWriterFactory>()
                    .AddSingleton<ODataMediaTypeResolver>(sp => new CborMediaTypeResolver());
            }
    ));

CBOR reader test

That’s all for the CBOR OData reading implementation. Let’s run and test it.

I want to send a POST request to the “books” endpoint to create a new book. By default, we can use application/json to send the following POST request with plain text JSON.

POST http://localhost:5015/odata/books

BODY (application/json)

{
  "Title": "My Story",
  "Author": "Sam Xu",
  "ISBN": "978-0-111-52493-5",
  "Pages": 9527
}

To test the CBOR JSON reader, we need the CBOR binary data representation for the above plain text JSON. Let’s covert the plain text JSON to CBOR binary array and save it into a file and name it newbook.dat. Here’s the file viewer:

List the byte data as text below:

A4 65 54 69 74 6C 65 68 4D 79 20 53 74 6F 72 79 66 41 75 74 68 6F 72 66 53 61 6D 20 58 75 64 49 53 42 4E 71 39 37 38 2D 30 2D 31 31 31 2D 35 32 34 39 33 2D 35 65 50 61 67 65 73 19 25 37

Now, we can use Postman to send this binary file to http://localhost:5015/odata/books as below:

Where:

1. Specify binary as the request body media type and use the “newbook.dat” created above.
2. Set “Content-Type=application/cbor” in the request header.

Let’s debug and click send button in the Postman, we can hit the following breaking point with correct data in the new “book” object as below:

That’s all.

Summary

This post went through the steps to customize CBOR writing and reading within ASP.NET Core OData 8.x. CBOR, as the binary representation of JSON, can provide a small code size without losing any JSON benefits. Hope the contents and implementations in this post can help. Again, please do not hesitate to leave your comments below or let me know your thoughts through saxu@microsoft.com. Thanks.

I uploaded the whole project to this repository. If you find any bug or issue, please file them here.