Skip to content

Topics & Events

Components

Topic

A Kafka topic is a named, partitioned, and immutable log of messages where producers send data, and consumers read from, allowing for organized and scalable data streaming in Kafka.

In KSTD diagrams, topics are commonly used to show sources and sinks.

Three different variants of topic components are provided. The default content included serves as a guideline. Choose the one you like and add details such as topic config parameters when they are essential to the topology design.

The 'topic-1' component of the KSTD Excalidraw library
Topic 'Simple'.
The most basic topic component. Icon and topic name.
The 'topic-2' component of the KSTD Excalidraw library
Topic the Apache Kafka® logo.
In addition to the topic name, additional configuration details may be included in a 2nd line in parenthesis.
The 'topic-3' component of the KSTD Excalidraw library
Topic the Apache Kafka® logo (2).
An alternative choice to the previous component.
More topic config options can be included, listed line-by-line.

Key-Value Record

In a Kafka Streams topology, data flows as records, each consisting of a key, value, and associated metadata (like timestamps). These records can undergo multiple transformations, and their data types for both keys and values may change between operations such as map(), flatMap(), or groupBy().

Kafka Streams uses SerDes (Serializer/Deserializer) to convert data types between their in-memory representations (e.g., Java objects) and their serialized form (e.g., JSON, Avro, or bytes) for transport within Kafka.

The ‘Key-Value Record’ component usually is added to describe the type and content of your data throughout the topology. It’s commonly added for sources & sinks, and whenever it changes. Further, it is also used in combination with state stores.

The 'record-kv' component of the KSTD Excalidraw library
Key-Value Record.

Describes the record key (k) and value (v). For generic types, it can be the name of an identifier. For complex objects, such as Avro or Protobuf, the record name is usually a good choice.

Compound string keys can be described like this: {departureDate}_{flightNumber}

The 'record-kv-typed' component of the KSTD Excalidraw library
Key-Value Record Typed.
Includes information on the SerDes and the resulting data types.