Change Data Capture
Boltic is a no-code/low-code big ops data platform that helps users solve complex data problems, automate workflows, build & share reports at scale by connecting data from multiple sources, transforming it, and sending it to desired destinations.
Change Data Capture (CDC) is a technique used in computer systems to track and capture changes made to a database or data source. In simple terms, it helps keep a record of every modification, addition, or deletion that happens to the data.
Think of CDC as a way to keep an eye on a flowing river of data. Whenever something changes in the database, CDC notices it and makes a note of what specifically changed, like a log or a record. This could include changes as simple as updating a customer's address, adding a new product to an inventory, or deleting a record from a database tab.
It helps organizations stay connected, make informed decisions based on real-time data, ensure data accuracy, and meet regulatory requirements. It enhances the efficiency and effectiveness of data integration, analytics, and overall data management processes.
To build and incorporate CDC into Boltic, I began with secondary research which involved understanding the backend process, steps in a CDC flow, methods to perform CDC and a competitive analysis of how other platforms have implemented it.
After understanding the major features required and the frictions I faced as a non-tech user while using these platforms, I conducted a primary research which involved various data analysts, IT admins, product managers, and data engineers I derived that incorporating CDC with a no-code integration and easy-to-use features into Boltic can transform the way users work with data. It can enhance user productivity, reduce barriers to entry, and empower a broader range of users to leverage the full potential of Boltic.
After understanding the key insights I derived during user research, based on our existing persona, I tried to empathize with their preferences and workflows. By understanding their typical workflows and data consumption patterns, I was able to define triggers that align with their needs. For example, business analysts heavily rely on real-time reporting, capturing data changes as they occur or on a scheduled basis would be crucial for them.
I then created multiple iterations of User Flows to create a CDC Pipeline. The final flow begins with the initial setup, where users connect data sources and define data capture rules. Then, subsequent steps are destination configuration, pipeline scheduling and monitoring. This approach ensures a comprehensive understanding of the entire workflow and allows for a detailed examination of each stage.
After finalising the User flow with stakeholders, I began wireframing the layouts to communicate design intent, assist in identifying key components, assess information hierarchy, and enable early-stage testing and validation with the stakeholder involved. Each process in the step has a dedicated page and the header above indicates the present step, previous completed step and the incomplete subsequent steps that the user needs to complete to create a pipeline. The layout used is similar to that of bolt creation in each page to maintain design legacy and ensure relatability for users.
After finalising the wireframes and the knitty-gritty of each functionality I designed the final screens of Figma which included pipe Creation, listing page and overview page. I’ve used Boltic’s Design System and legacy designs to create the final frames with some modification. The challenges I faced were that Bolt Creation’s process is created with a waterfall design method whereas in Pipe Creation the process is more horizontal with a lot of interdependencies, so I retained the cards to indicate each step in the respective process. The design also had to be scalable since there are many more features that had to be built in the future, so later on these features can be added as a part of the process independently as a horizontal step.
The overview page required a bar graph component to present the real-time rows being processed to the user. This bar graph has a hover state for each bar where they can see the rows changed during that miniscule period of time which allows a deeper insight to the user. I’ve also included a new page to the overview page called ‘Schema Mapper’, this will allow user to select or deselect any tables that are being mapped to the final destination. This is a part of pipe creation as well, but as I understand the frequency of editing specific tables is higher, so it’s important to provide the option to perform this action faster and with ease of use.
By successfully incorporating CDC into Boltic, I have learnt that the application of design process is iterative in nature. It's not a linear process, after each development I found myself going back to my previous steps and refining that thought. The biggest challenged I faced was that of a knowledge gap between me and the development team, since the feature require heavy knowledge of the data space as well as some SQL knowledge, I had to collaborate and communicate more effectively with them to deliver a feature that truly works for users at both the extremes of the spectrum. This was a great learning experience for me.