MASES.KNet.Serialization.Protobuf 2.6.6

title: Serialization usage of .NET suite for Apache Kafka _description: Describes how to use serialization of .NET suite for Apache Kafka

KNet: Serializer/Deserializer

KNet comes with a base set of serializer and deserializer. Most of them are usable with primitives types (bool, int, etc) and array of bytes.

If the user wants to use structures types there are two ways:

1. Creates types in Java and reflects them in C#
2. Creates types in C# and extend the available serializer/deserializer

KNet suite offers some ready made serializer/deserializer usable with the specific APIs (KNetProducer/KNetConsumer).

The current available packages are:

• MASES.KNet.Serialization.Avro: it is a serdes which uses AVRO; till now is not ready.
• MASES.KNet.Serialization.Json: it is a serdes which uses Json; till now it is at its first stage and it is based on general purpose API from:
  • .NET Framework: it uses Newtonsoft.Json package
  • .NET 6/7: it uses the Json which comes with the frameworks
• MASES.KNet.Serialization.MessagePack: it is a serdes which uses MessagePack; till now it is at its first stage and it is based on general purpose API from MessagePack package
• MASES.KNet.Serialization.Protobuf: it is a serdes which uses Google.Protobuf; till now it is at its first stage and it is based on general purpose API from Google.Protobuf package

Let consider a type defined like the following one:

public class TestType
{
    public TestType(int i)
    {
        name = description = value = i.ToString();
    }

    public string name;
    public string description;
    public string value;

    public override string ToString()
    {
        return $"name {name} - description {description} - value {value}";
    }
}

To manage it within C#, without create TestType in Java, an user can create:

• serializer (the body must be updated with the user serializer):

KNetSerDes<TestType> serializer = new KNetSerDes<TestType>((topic, type) => { return new byte[0]; });

• deserializer (the body must be updated with the user deserializer):

KNetSerDes<TestType> deserializer = new KNetSerDes<TestType>((topic, data) => { return new TestType(0); });

Otherwise the user can use a ready made class like in the following snippet:

KNetSerDes<TestType> serdes = new JsonSerDes<TestType>();

A single JsonSerDes can be used in serialization and deserialization, and produce Json serialized data.

Specific cases

Some kind of serializers extension have specific needs will be listed below.

Avro serializer

The Avro serializer is based on Apache.Avro package. The types managed are:

• Avro types managed using the Avro library are Avro records which:
  • Shall have a parameterless constructor
  • Shall conform to ISpecificRecord

NOTE: simple types (the one that have an Apche Kafka default serializer) are not managed and will be refused

MessagePack serializer

The MessagePack serializer is based on MessagePack package. The types managed are:

• MessagePack types managed using the MessagePack library shall be MessagePack types.

NOTE: simple types (the one that have an Apche Kafka default serializer) are not managed and will be refused

Protobuf serializer

The Protobuf serializer is based on Google.Protobuf package. The types managed are:

• Protobuf types managed using the Protobuf library shall be messages types which:
  • Shall have a parameterless constructor
  • Shall conform to IMessage<T>

NOTE: simple types (the one that have an Apche Kafka default serializer) are not managed and will be refused