Module id.kineticstreamer

module id.kineticstreamer
kineticstreamer - Java module to do (de)serialization of Java objects into any type of streams. It provides default implementations for object conversion to stream of bytes and CSV files. Users can add support for any other format as well.

It parses object tree and allows to get control over how types are going to be (de)serialized.

Why not Java standard ObjectOutputStream

By default ObjectOutputStream adds to stream ObjectStreamConstants.TC_BLOCKDATA: 


 var fos = new FileOutputStream("t.tmp");
 var oos = new ObjectOutputStream(fos) {
     @Override
     protected void writeStreamHeader() throws java.io.IOException {
         // ignore header
     }
 };

 oos.writeInt(0xffff);
 oos.close();

Will put to stream: 

TC_BLOCKDATA|0x0000ffff

It is possible to use ObjectOutputStream default constructor to a avoid this but it requires to reimplement writeObject which does all work with class tree serialization (behind private method writeObject0 which you cannot access from writeObject because it is final).

Stream

In terms of kineticstreamer 'stream' represents sequence of any sort (sequence of bytes, words, numbers, etc). The type of elements in such sequence don't need to be the same. Streams have flat schemas opposite to JSON, XML, etc.

Example of streams are flat byte formats, CSV files, etc.

Kinetic stream

Kinetic stream is an abstraction which is defined by two interfaces:

kineticstreamer relies on them for Java objects (de)serialization.

Kinetic streams allow kineticstreamer to operate with any type of streams without even knowing anything about their format.

Every type of kinetic stream must have its own implementation of InputKineticStream (for reading objects from it) and OutputKineticStream for writing them).

kineticstreamer comes with some predefined kinetic streams (see id.kineticstreamer.streams). To add support for custom types of streams you need to implement kinetic stream interfaces for them.

Streamed classes

Streamed classes are classes which objects can be (de)serialized. These are also know as data transfer objects (DTO)

Streamed fields - fields of a streamed classes which are going to be (de)serialized.

Field types

kineticstreamer divides all types into two categories:
  • kinetic stream types - these are all Java primitive types plus any other types for which there are separate method definition in InputKineticStream and OutputKineticStream
  • foreign types - all others types (mostly these are classes which are composed from fields of kinetic stream types or other foreign types).

Understanding this separation is important for creating streamed classes and controllers.

Defining streamed classes

Streamed classes must satisfy following requirements:

  • Streamed class must have default constructor
  • Streamed fields of foreign types must not be null (unless you implement controllers and handle (de)serialization of such types manually). This is required for deserialization purposes. By default during deserialization KineticStreamReader expects all streamed fields of foreign types be present in the stream. It is a good practice to initialize such fields with their default constructor.

KineticStreamController

Controller is responsible for finding all streamed fields of a class and deciding in which order they must be (de)serialized (see StreamedFieldsProvider). By default KineticStreamController is using PublicStreamedFieldsProvider which considers all public fields of a class as streamed fields. Any static, private, final, transient fields are ignored. Users can change this behavior by implementing their own StreamedFieldsProvider.

Additionally controller helps in case of complex serialization logic of certain foreign field types. For example:

  • it allows to support polymorphic fields which type is undefined until certain flags are read from the stream
  • when certain fields may be present in the stream or not based on data which was previously read from it

Users can extend KineticStreamController to provide custom logic and inject into KineticStreamReader and KineticStreamWriter correspondingly.

Examples

Streamed class: 


 public class StringMessage {

     public String data;

     // streamed classes require default ctor
     public StringMessage() {
     }

     public StringMessage(String data) {
         this.data = data;
     }
 }
Streaming StringMessage object to stream of bytes and back: 

 // write
 var tmpFile = Files.createTempFile("", "kineticstream");
 var fos = new FileOutputStream(tmpFile.toFile());
 var dos = new ByteOutputKineticStream(new DataOutputStream(fos));
 var ksw = new KineticStreamWriter(dos);
 ksw.write(new StringMessage("hello kineticstreamer"));

 // read back
 var fis = new FileInputStream(tmpFile.toFile());
 var dis = new ByteInputKineticStream(new DataInputStream(fis));
 var ksr = new KineticStreamReader(dis);
 StringMessage actual = (StringMessage) ksr.read(StringMessage.class);
 System.out.println(actual.data);

Java records support

Currently kineticstreamer does not support records (de)serialization.

The reason behind it is based on observation that users are not using records as streamed classes (DTO).

Here is an example of a streamed class definition MarkerMessage which is used to communicate with Robot Operating System.

This class has certain fields which need to be set only in certain cases, otherwise they should contain default values:

  • text - Only used for text markers
  • mesh_resource - Only used for MESH_RESOURCE markers

Additionally some fields are assigned default values, so they don't need to be populated by the user each time.

  • lifetime - default is forever

To convert MarkerMessage to record and not loose all the specifics given above there are two ways:

  • define multiple overloaded constructors for all possible ways MarkerMessage can be created, like:
    • MarkerMessage(..., Duration lifetime) - user provided duration
    • MarkerMessage(...) - default duration
    • ...
    Which is time consuming and error prone.
  • define a separate builder which will look like current MarkerMessage except at the end it will produce record version of MarkerMessage. This is again time consuming and error prone.

To summarize: possibly one reason for records not being widely used as DTO is because they lack any builders support.

Useful links:

See Also: