Skip to content
Kafka Streams Topology Design

The standard for designing and visualising Kafka Streams Topologies

Avoid Mistakes. Save Time. Build Trust.
Create High-Quality Designs and Applications!

What is KSTD?

Design Standard

An open standard for describing and visualising Kafka Stream Topologies.

Clear guidelines on crafting diagrams that are descriptive, and self-explanatory.

It offers best practices for structuring topologies and providing essential context, such as the responsibilities of individual processing steps.

Excalidraw Library

A comprehensive component library for the popular drawing tool Excalidraw, known for creating hand-drawn style diagrams and visualisations.

It offers full coverage of the Kafka Streams DSL, along with essential infrastructure and architectural building blocks, licensed under GPL-3.0.

Uses and benefits

The structured representation of the data flow and processing logic…

Proactive Issue Resolution. Helps identify potential issues, gaps, and design flaws, enabling proactive resolution and more informed decision-making.

Enhanced Creative Process. Accelerates the creative design process, allowing for rapid progress and effective remote collaboration in whiteboard sessions.

Shared Understanding. Creates a common ground for teams, fostering shared understanding and alignment, promoting consensus on design decisions.

Improved Communication. Facilitates communication within and outside development/product teams, and streamlines the onboarding process for new team members.

Complete Project Lifecycle. From requirements engineering and brainstorming, through technical design, refinement, implementation and testing, to documentation, support, and maintenance.

Manageable Components. Breaking down complex systems into manageable components makes it easier to understand and work on individual parts. This allows for better planning, estimation, and work splitting.

Introducing

Kafka Streams Topology Design

Software architecture is the blueprint that defines the structure of a software system, defining its components, their interactions, and guiding principles to meet technical and business requirements.

”Diagrams must be self descriptive, consistent, accurate enough and connected to the code. That’s why it is important that every architect or software engineer rely on several guidelines when creating architectural diagrams, since they are the common ground of communicating the application’s architecture over time”

Tools like UML, ArchiMate, and C4 Model are used to describe software architecture, providing clear visual representations that improve understanding, communication, and alignment among stakeholders.

Surprisingly, there are few, if any, standards and tools available for designing and visually describing stream processing applications.

A search of the web as of August 2024 highlights this gap. You can count the number of diagrams (publicly) available describing a Kafka Streams topology on one hand.

What you’ll most often find are basic component integration diagrams and flowchart-derived visuals, typically created with standard diagramming tools.

As Simon Brown, the creator of the C4 model, aptly calls them, these are often just “boxes and lines” diagrams.

KSTD Standard

The “Kafka Streams Topology Design” open standard was created to provide guidelines and tools for designing and visualizing Kafka Streams topologies. It includes a comprehensive component library for Excalidraw, a drawing tool, allowing users to create clear, self-explanatory diagrams that accurately represent the structure and processing logic of stream processing applications.

Examples

Inspire Your Designs: Explore Real-World Kafka Streams Topology Examples.

You can find more details related to these examples in the documentation.

Features

What you get with KSTD

From concept to clarity: Kafka Streams development made easy.
Fast whiteboarding sessions

Quickly create clean, visually appealing diagrams, perfect for brainstorming sessions, meetings, and design sprints. Capture and refine ideas swiftly, facilitating efficient and productive creative processes.

Ease of use

Built on the popular collaborative whiteboard tool Excalidraw, provides an intuitive and user-friendly experience that requires no steep learning curve. You’ll find it easy to start creating and collaborating right away.

Designated library components

The Excalidraw library provides ready-to-use components to describe all aspects of a Kafka Streams Topology. Utilize specialized layout items to add additional information and descriptions of processing logic, enhancing the clarity and functionality of your diagrams.

Holistic design standard

Ensure your diagrams are comprehensive by including all relevant information. Detail topic specifics, record key-value schemata, and functional description of processor nodes within your designs to provide a full picture.

What Excalidraw provides

Excalidraw is a virtual collaborative whiteboard tool that lets you easily sketch diagrams that have a hand-drawn feel to them.

test

No Installation Required

Access Excalidraw directly through your web browser without the need for installation. Once loaded, it can function offline, providing flexibility and convenience for users on the go.

Secure Live Collaboration

Collaborate in real time with multiple users on the same drawing. Ideal for team brainstorming sessions and remote work, with end-to-end encrypted sessions ensuring your data remains secure.

test

Save/Open in Native Excalidraw Format

Manage your work efficiently by saving and opening diagrams in the native Excalidraw format. This feature supports version control, making collaboration and project management seamless.

Various Export Formats

Export your diagrams in multiple formats such as PNG, SVG, and others. This versatility ensures that you can easily incorporate your diagrams into presentations, documents, or share them with colleagues and stakeholders.

Privacy and Security

As an open-source tool that can be self-hosted, Excalidraw offers enhanced control over your data. This ensures greater privacy and security, allowing you to safeguard sensitive information.

Boost your productivity.
Start using KS Topology Design today.

Avoid Mistakes. Save Time. Build Trust.
Create High-Quality Designs & Applications!

Frequently Asked Questions

Have a different question and can’t find the answer you’re looking for? Please feel free to ask your question on GitHub Discussions.
What's the inspiration behind the KSTD standard?

Kafka Streams Topology Design is a Thriving.dev project created by Hartmut Armbruster , a software developer and architect, to craft and communicate Kafka Streams topologies.

Over the years, it has evolved and been refined through extensive use across various industries, proving to be an effective and valuable tool.

From experience, many teams had previously relied on only high-level diagrams (if any), often jumping into code without fully grasping the implications and risks, leading to time-consuming refactoring or even project failure.

Visualizing ideas and creating descriptive designs for stream processing applications is no trivial task. The KSTD standard aims to provide valuable guidelines and tools to the Kafka Streams community in overcoming these challenges.

We have multiple Kafka Streams apps; Should we put them in one Excalidraw board, or keep them separate?

The KSTD standard does not prohibit or restrict the inclusion of multiple topologies or applications within a single diagram.

However, clarity should always be your priority. Focus on presenting what’s most important to the viewer. If it’s crucial to display multiple topologies side-by-side to illustrate how they work together and provide a complete picture, then feel free to do so.

From a technical perspective, Excalidraw allows you to manage multiple iterations of your topology design or even maintain several different topologies within a single scene (whiteboard). You can export individual diagrams as needed. For additional guidance, check out the recommendations under Tips and Tricks.

Is KSTD used during the design phase, or for documenting existing applications?

Both!

“Software architecture diagrams are a fantastic way to communicate how you are planning to build a software system (up-front design) or how an existing software system works (retrospective documentation, knowledge sharing, and learning).”
Source: InfoQ - C4 Architecture Model

For the design phase, Excalidraw is a good choice if you want to design quickly. The hand-drawn aesthetic encourages fast, freeform sketching, which is perfect for getting ideas down quickly without worrying about precision.

When combined with the KSTD library, it enables rapid and efficient progress, allows to concentrate on designing the topology and refining the stream processing architecture.

In the later phases of the project lifecycle, once the topology is established, it becomes invaluable for guiding the implementation process. It serves as a go-to documentation resource when discussing changes or introducing new features, and facilitates the efficient onboarding of new team members.

What level of detail should we include in a diagram?

Classic answer: “It depends…”

Are you in the process of sketching, evaluating ideas - or creating architecture documentation?

Crafting & maintaining good diagrams takes time and effort.

Prefer less, or “just enough”, over risk of fragmentation, outdated diagrams. Define a weight for a certain diagram; the level of importance. This determines the time (and money) to invest.

As a rule of thumb: Start simple, go with the flow. Only add details to the topology design that affect or change the behaviour of the stream processing.

As an example: There’s no need to draw a state store to each KTable or Aggregation. It is though on the other hand when you have a ‘Global’ ‘InMemory’ store that is accessed via interactive queries by Processors.

Keeping a balance between cluttered vs. too vague is essential. Add enough text so it hardly requires additional explanation. If it raises more questions than answers, the diagram is not well created.

You can find more recommendations on design principles and architecture diagrams in the documentation.

OK, Kafka Streams. What about other stream processing technilogies?

The library has been created while designing and implementing applications with Kafka Streams. So the milestone for version 1.0.0 naturally was to focus on Kafka Streams.

Can the library be used to describe e.g. Apache Flink or Apache Spark applications? Simple answer: while no one’s stopping you, but it’s not supported right now.

You want to adapt and create something similar to KSTD for other technologies? Please get in touch to discuss this further!

Is KSTD free? It's licensed under GPL-3.0, what does that mean?

Yes, the KSTD standard and the Excalidraw library are free for personal, as well as for commercial use.

GPL-3.0 is a free software license that allows users to freely use, modify, and share software, ensuring it stays free and open for everyone.

The source code is openly available on GitHub.

Can I propose new ideas, features, or contribute to KSTD?

Absolutely! Contributions to improve the documentation or are always welcome. If you spot something incorrect or missing, feel free to create an issue or submit a pull request.

For discussing new ideas on library items or posting feature requests, please start a discussion!

Credits

Similar Projects

As the author, I want to acknowledge the resources and individuals who have provided valuable insights and inspiration along the way.
References
  • Flink Kafka Stream Implementation. While not specifically focused on visualizing topologies, the diagrams in the repository were created independently of KSTD, driven by the shared vision of Adam Souquières and Ludovic Boutros. There seems to be a natural connection between our approaches.
  • Designing Kafka Streams Applications. The team at Michelin published an article on the concept of ‘Master Topologies’ and ‘Micro Topologies’ they established over the past years designing and building Kafka Streams applications.

    The methodology puts more focus on modelling business requirements, and goes less detailed into Kafka Streams. But the blog post emphasises the very same need for comprehensive and granular understanding of our end-to-end communication of solutions.

  • zz85/kafka-streams-viz by Joshua Koo. A tool for visualizing stream topologies by generating diagrams from a kafka stream topology descriptions.
Special Thanks
  • C4 Model by Simon Brown. Simon Brown has had a significant impact on the software engineering community by improving how teams approach and communicate software architecture.

    Through his creation of the C4 Model, he introduced a clear, accessible way to document systems, making architectural diagrams more practical and widely understood. His work has helped standardize architecture visualization, fostering better collaboration and understanding across development teams and organizations.

    Simon’s work has been a source of inspiration and a positive example, motivating me to create and share this standard with the community.

  • Excalidraw has been my go-to tool for visualising Software Architecture for years. Its ease of use and support for creating reusable libraries allow me to sketch and create technical designs at a speed that does not hinder the creative process. I can brainstorm, bring my ideas and solutions “on paper”, and live collaborate with a remote team. Big thank you, Excalidraw!
Next Steps

KSTD Modeling Language

Coming soon…