KSTD Standard

The KSTD Standard outlines a comprehensive approach to designing Kafka Streams Topology Diagrams. This page defines the essential layout components, building blocks, and structure to follow, as well as naming conventions, which are crucial for maintaining clarity.

A consistent and intuitive design approach ensures that complex Kafka Streams topologies remain maintainable and aligned with business logic.

Layout and Building Blocks

When designing Kafka Streams Topology Diagrams, the key layout blocks are:

• Header: Title, diagram type, date, and version. (ref Architecture Diagram - Best Practices)
• Main Block: (left to right)
  • Source Topics
  • Topology: The detailed design of the Kafka Streams application, the heart of the diagram.
  • Sink Topics
• Footer: (Optional) Place additional description, such as key responsibilities and processing logic for steps and operators.

Note: Typically, a single topology is depicted per diagram, though in some cases, smaller sidecar topologies may accompany the main topology.

Topology Layout and Internals

The topology is the core of the entire diagram. While there are no strict rules governing the internal structure of a topology, there are important considerations:

• Sub-topologies: When relevant to the design, especially in more advanced cases, it’s recommended to include sub-topologies in your diagram. However, keep in mind that Kafka Streams automatically resolves sub-topologies, meaning they operate beyond the developer’s direct control.
• Grouped PAPI Elements: For complex use cases—such as those involving the Processor API, State Stores, Interactive Queries, or Punctuators — it can be useful to group related elements within an additional box. This provides a clear visual distinction of their logical association within the topology.
• Descriptions: A general rule of thub, when it comes to adding descriptions to the diagram, is to use inline descriptions sparingly. If there’s a sufficient level of complexity and therefore descriptions to add, consider placing them outside the Topology in the Footer block.

TODO: provide pictures with the ‘Topology Layout and Internals’ highlighted.

Swimlanes. (TODO)

Orientation

TLDR: ‘left to right’

KSTD recommends, but does not mandate a certain orientation or direction, but the library was created to work best for a ‘left to right’ direction of data flows.

All examples provided are created in landscape orientation and show data flows from left (sources) to the right (sinks).

Naming Conventions

While the KSTD standard does not mandate naming the elements in your diagrams, it is highly recommended to enhance clarity and comprehension.

Consistently use the names defined in your designs during implementation. Explore the benefits of this approach below.

Topology Name

Recommended to name after the APPLICATION_ID (to be) used by the application.

Named Processing Steps

Naming processors and processing steps in a Kafka Streams topology is more than just a best practice; it’s a critical step in making your stream processing application maintainable, debuggable, and observable. It helps to ensure that your topology is understandable, both for humans and for the tools that interact with it, throughout the application’s lifecycle.

1. Debugging and Monitoring: Named processors make logs, metrics, and error messages clearer, helping to trace issues and bottlenecks effectively.
2. Observability: Custom metrics and visualisation tools use processor names, enhancing insights into performance and data flow.
3. Code Clarity: Meaningful names improve code readability, making it easier to understand and maintain complex topologies.
4. Refactoring: Named steps simplify topology evolution and refactoring without affecting other parts.
5. Error Handling: Enables more targeted exception handling.
6. Testing: Facilitates focused and efficient testing of specific topology components.
7. Collaboration: Improves communication between teams by providing clear references to specific steps.
8. Business Alignment: Names can reflect business logic, making the topology more intuitive and aligned with business processes.

Remember, this also includes sources and sinks.

Named State Stores

State stores in Kafka Streams don’t always have to be named, but doing so is often beneficial, especially in more complex applications, for interactive queries, and for improving maintainability and observability.