The standard for designing and visualising Kafka Streams Topologies
Create High-Quality Designs and Applications!
What is KSTD?
An open standard for describing and visualising Kafka Stream Topologies.
Clear guidelines on crafting diagrams that are descriptive, and self-explanatory.
It offers best practices for structuring topologies and providing essential context, such as the responsibilities of individual processing steps.
A comprehensive component library for the popular drawing tool Excalidraw, known for creating hand-drawn style diagrams and visualisations.
It offers full coverage of the Kafka Streams DSL, along with essential infrastructure and architectural building blocks, licensed under GPL-3.0.
Uses and benefits
Proactive Issue Resolution. Helps identify potential issues, gaps, and design flaws, enabling proactive resolution and more informed decision-making.
Enhanced Creative Process. Accelerates the creative design process, allowing for rapid progress and effective remote collaboration in whiteboard sessions.
Shared Understanding. Creates a common ground for teams, fostering shared understanding and alignment, promoting consensus on design decisions.
Improved Communication. Facilitates communication within and outside development/product teams, and streamlines the onboarding process for new team members.
Complete Project Lifecycle. From requirements engineering and brainstorming, through technical design, refinement, implementation and testing, to documentation, support, and maintenance.
Manageable Components. Breaking down complex systems into manageable components makes it easier to understand and work on individual parts. This allows for better planning, estimation, and work splitting.
Kafka Streams Topology Design
Software architecture is the blueprint that defines the structure of a software system, defining its components, their interactions, and guiding principles to meet technical and business requirements.
”Diagrams must be self descriptive, consistent, accurate enough and connected to the code. That’s why it is important that every architect or software engineer rely on several guidelines when creating architectural diagrams, since they are the common ground of communicating the application’s architecture over time”Ionut Balosin
Tools like UML, ArchiMate, and C4 Model are used to describe software architecture, providing clear visual representations that improve understanding, communication, and alignment among stakeholders.
Surprisingly, there are few, if any, standards and tools available for designing and visually describing stream processing applications.
A search of the web as of August 2024 highlights this gap. You can count the number of diagrams (publicly) available describing a Kafka Streams topology on one hand.
What you’ll most often find are basic component integration diagrams and flowchart-derived visuals, typically created with standard diagramming tools.
As Simon Brown, the creator of the C4 model, aptly calls them, these are often just “boxes and lines” diagrams.
KSTD Standard
The “Kafka Streams Topology Design” open standard was created to provide guidelines and tools for designing and visualizing Kafka Streams topologies. It includes a comprehensive component library for Excalidraw, a drawing tool, allowing users to create clear, self-explanatory diagrams that accurately represent the structure and processing logic of stream processing applications.
Examples
You can find more details related to these examples in the documentation.
What you get with KSTD
Quickly create clean, visually appealing diagrams, perfect for brainstorming sessions, meetings, and design sprints. Capture and refine ideas swiftly, facilitating efficient and productive creative processes.
Built on the popular collaborative whiteboard tool Excalidraw, provides an intuitive and user-friendly experience that requires no steep learning curve. You’ll find it easy to start creating and collaborating right away.
The Excalidraw library provides ready-to-use components to describe all aspects of a Kafka Streams Topology. Utilize specialized layout items to add additional information and descriptions of processing logic, enhancing the clarity and functionality of your diagrams.
Ensure your diagrams are comprehensive by including all relevant information. Detail topic specifics, record key-value schemata, and functional description of processor nodes within your designs to provide a full picture.
What Excalidraw provides
Access Excalidraw directly through your web browser without the need for installation. Once loaded, it can function offline, providing flexibility and convenience for users on the go.
Collaborate in real time with multiple users on the same drawing. Ideal for team brainstorming sessions and remote work, with end-to-end encrypted sessions ensuring your data remains secure.
Manage your work efficiently by saving and opening diagrams in the native Excalidraw format. This feature supports version control, making collaboration and project management seamless.
Export your diagrams in multiple formats such as PNG, SVG, and others. This versatility ensures that you can easily incorporate your diagrams into presentations, documents, or share them with colleagues and stakeholders.
As an open-source tool that can be self-hosted, Excalidraw offers enhanced control over your data. This ensures greater privacy and security, allowing you to safeguard sensitive information.
Boost your productivity.
Start using KS Topology Design today.
Create High-Quality Designs & Applications!
Frequently Asked Questions
Kafka Streams Topology Design is a Thriving.dev project created by Hartmut Armbruster , a software developer and architect, to craft and communicate Kafka Streams topologies.
Over the years, it has evolved and been refined through extensive use across various industries, proving to be an effective and valuable tool.
From experience, many teams had previously relied on only high-level diagrams (if any), often jumping into code without fully grasping the implications and risks, leading to time-consuming refactoring or even project failure.
Visualizing ideas and creating descriptive designs for stream processing applications is no trivial task. The KSTD standard aims to provide valuable guidelines and tools to the Kafka Streams community in overcoming these challenges.
The KSTD standard does not prohibit or restrict the inclusion of multiple topologies or applications within a single diagram.
However, clarity should always be your priority. Focus on presenting what’s most important to the viewer. If it’s crucial to display multiple topologies side-by-side to illustrate how they work together and provide a complete picture, then feel free to do so.
From a technical perspective, Excalidraw allows you to manage multiple iterations of your topology design or even maintain several different topologies within a single scene (whiteboard). You can export individual diagrams as needed. For additional guidance, check out the recommendations under Tips and Tricks.
Both!
“Software architecture diagrams are a fantastic way to communicate how you are planning to build a software system (up-front design) or how an existing software system works (retrospective documentation, knowledge sharing, and learning).”
Source: InfoQ - C4 Architecture Model
For the design phase, Excalidraw is a good choice if you want to design quickly. The hand-drawn aesthetic encourages fast, freeform sketching, which is perfect for getting ideas down quickly without worrying about precision.
When combined with the KSTD library, it enables rapid and efficient progress, allows to concentrate on designing the topology and refining the stream processing architecture.
In the later phases of the project lifecycle, once the topology is established, it becomes invaluable for guiding the implementation process. It serves as a go-to documentation resource when discussing changes or introducing new features, and facilitates the efficient onboarding of new team members.
Classic answer: “It depends…”
Are you in the process of sketching, evaluating ideas - or creating architecture documentation?
Crafting & maintaining good diagrams takes time and effort.
Prefer less, or “just enough”, over risk of fragmentation, outdated diagrams. Define a weight for a certain diagram; the level of importance. This determines the time (and money) to invest.
As a rule of thumb: Start simple, go with the flow. Only add details to the topology design that affect or change the behaviour of the stream processing.
As an example: There’s no need to draw a state store to each KTable or Aggregation. It is though on the other hand when you have a ‘Global’ ‘InMemory’ store that is accessed via interactive queries by Processors.
Keeping a balance between cluttered vs. too vague is essential. Add enough text so it hardly requires additional explanation. If it raises more questions than answers, the diagram is not well created.
You can find more recommendations on design principles and architecture diagrams in the documentation.
The library has been created while designing and implementing applications with Kafka Streams. So the milestone for version 1.0.0 naturally was to focus on Kafka Streams.
Can the library be used to describe e.g. Apache Flink or Apache Spark applications? Simple answer: while no one’s stopping you, but it’s not supported right now.
You want to adapt and create something similar to KSTD for other technologies? Please get in touch to discuss this further!
Yes, the KSTD standard and the Excalidraw library are free for personal, as well as for commercial use.
GPL-3.0 is a free software license that allows users to freely use, modify, and share software, ensuring it stays free and open for everyone.
The source code is openly available on GitHub.
Absolutely! Contributions to improve the documentation or are always welcome. If you spot something incorrect or missing, feel free to create an issue or submit a pull request.
For discussing new ideas on library items or posting feature requests, please start a discussion!
Similar Projects
- Flink Kafka Stream Implementation. While not specifically focused on visualizing topologies, the diagrams in the repository were created independently of KSTD, driven by the shared vision of Adam Souquières and Ludovic Boutros. There seems to be a natural connection between our approaches.
-
Designing Kafka Streams Applications. The team at Michelin published an article on the concept of ‘Master Topologies’ and ‘Micro Topologies’ they established over the past years designing and building Kafka Streams applications.
The methodology puts more focus on modelling business requirements, and goes less detailed into Kafka Streams. But the blog post emphasises the very same need for comprehensive and granular understanding of our end-to-end communication of solutions.
- zz85/kafka-streams-viz by Joshua Koo. A tool for visualizing stream topologies by generating diagrams from a kafka stream topology descriptions.
-
C4 Model by Simon Brown. Simon Brown has had a significant impact on the software engineering community by improving how teams approach and communicate software architecture.
Through his creation of the C4 Model, he introduced a clear, accessible way to document systems, making architectural diagrams more practical and widely understood. His work has helped standardize architecture visualization, fostering better collaboration and understanding across development teams and organizations.
Simon’s work has been a source of inspiration and a positive example, motivating me to create and share this standard with the community.
- Excalidraw has been my go-to tool for visualising Software Architecture for years. Its ease of use and support for creating reusable libraries allow me to sketch and create technical designs at a speed that does not hinder the creative process. I can brainstorm, bring my ideas and solutions “on paper”, and live collaborate with a remote team. Big thank you, Excalidraw!