nDimensional, a leading data analytics and AI company that helps companies leverage their data to get to a deeper understanding of what's going on in their business. They needed a scalable data lake that could handle the size and complexity of a massive, growing data set. They turned to Knoldus to guide them in designing and developing a technology architecture that would be cost effective, efficient, and yet flexible enough to meet their current and future needs.

nD wanted to design and develop a scalable data lake that supports both Batch & Stream data that works along with their platform. Also, the objective was to make a self-containing application that can be packaged into an image (e.g.: Docker) and deployed on any cloud service or on-premise.

Although, the basic flow for the application was very simple but the key challenges were:

• Storing data in a unified format so that for the user there is no difference between batch and stream data.

• Accessibility of data should be quick, i.e., based on user query.

• Processing Batch/Stream using a plugable logic supplied by the user.

Knoldus worked on two key parts -

• Fast API - An interface that provides various actions for processing Batch and streaming data. Also, they help in connecting information from Kafka & other static resources like Files, DB(s), etc.

• Microservice Service (based on Lagom) for Background Tasks - It was developed to take load off the processing engine for saving data to the data lake (S3 and Cassandra) and assign it to a separate node. Since the service was built using Lagom, it was easily re-sizeable and deployable on different anywhere.

The technology stack and architecture met the SLAs which were required for the platform. The application was best suited for reactiveness and the entire Lightbend reactive technology stack helped achieving it. Currently this platform is in Beta version and works like a charm.

The new platform and technologies recommended and architected by Knoldus helped nD successfully do the following:

• Created streaming and batch pipelines along with workflows, in the same platform.

• Handled different sources of data like files, tables, streams (Kafka, Kinesis, etc) in the same platform.

• Supported 100,000 streams with zero latency.

• Dramatically increase processing speed.

• Substantially increase storage capacity.

• Save money while scaling their business.