Topics & Events

Components

Topic

A Kafka topic is a named, partitioned, and immutable log of messages where producers send data, and consumers read from, allowing for organized and scalable data streaming in Kafka.

In KSTD diagrams, topics are commonly used to show sources and sinks.

Three different variants of topic components are provided. The default content included serves as a guideline. Choose the one you like and add details such as topic config parameters when they are essential to the topology design.

Topic 'Simple'.
The most basic topic component. Icon and topic name.

Topic the Apache Kafka® logo.
In addition to the topic name, additional configuration details may be included in a 2nd line in parenthesis.

Topic the Apache Kafka® logo (2).
An alternative choice to the previous component.
More topic config options can be included, listed line-by-line.

Note

Internal topics in Kafka Streams are automatically managed Kafka topics used to store intermediate data, such as state store changelogs and repartitioned data, to support fault tolerance and scalability within stream processing applications.

As per our standard and design principles, changelog & repartition topics are not included in the diagram. They usually don’t provide additional value and only clutter the diagram.

But remember, it’s up to you. Draw what you feel is right and works best for you.

Key-Value Record

In a Kafka Streams topology, data flows as records, each consisting of a key, value, and associated metadata (like timestamps). These records can undergo multiple transformations, and their data types for both keys and values may change between operations such as map(), flatMap(), or groupBy().

Kafka Streams uses SerDes (Serializer/Deserializer) to convert data types between their in-memory representations (e.g., Java objects) and their serialized form (e.g., JSON, Avro, or bytes) for transport within Kafka.

The ‘Key-Value Record’ component usually is added to describe the type and content of your data throughout the topology. It’s commonly added for sources & sinks, and whenever it changes. Further, it is also used in combination with state stores.

Key-Value Record.

Describes the record key (k) and value (v). For generic types, it can be the name of an identifier. For complex objects, such as Avro or Protobuf, the record name is usually a good choice.

Compound string keys can be described like this: {departureDate}_{flightNumber}

Key-Value Record Typed.
Includes information on the SerDes and the resulting data types.

Note

SerDes may change throughout the topology as different operations are applied, ensuring correct serialization at each step.

• Initial SerDes: The SerDes for the source topic might, for example, deserialize a record key as a String and value as an Avro object.
• Transformations: If the topology applies a transformation (e.g., converting a ProductEvent into a ProductView), the SerDes can change accordingly, requiring a new SerDe for the transformed object.
• Repartitioning: Operations like groupBy() or selectKey() change the key type, often requiring a repartitioning topic with its own SerDe for the new key-value structure.
• State Stores: If stateful processing is used (e.g., aggregations), internal changelog topics will store serialized state data using SerDes that correspond to the in-memory structure of the state store.