Data Lake Infrastructure Setup on AWS Cloud Platform
Key Challenges
Key Results
Overview
Data Lake Infrastructure Setup on AWS Cloud Platform
Profectus Capital is a Reserve Bank of India registered non-banking financial services firm (NBFC–SI–ND). It provides term loans, working capital loans, and machinery loans for the manufacturing and service sectors. Other services include Dealers Supply Chain Financing solutions in industries such as pharmaceuticals, consumer durables, and automobiles. The client currently lends to eleven clusters across 15 branch sites and is working to be the preferred lending partner for thousands of SMEs across the world.
Challenges
Currently, the customer has multiple structured and unstructured data sources. The customer wants to combine all these data and run big data analysis, real-time analytics, and machine learning to create dashboards and visualizations for informed decision-making, collecting and storing any type or any scale of data at low costs, securing all data stored in the central repository, a quick analysis of new types of data and querying by defining the data’s structure at the time of use
Solution
- Ankercloud architected the data lake solution on the AWS Cloud Platform. For the same Ankercloud created AWS infrastructure for:
-Ingesting data from different data sources
-Processing data on the data lake
-Making processed data available for downstream systems and reporting purposes.
- Based on the problem statement, Ankercloud team has worked with the client-side technical team to build data lake solutions.
- By using AWS Database Migration Service, Ankercloud migrated data from data sources like PostgreSQL, MySQL, and SAP Hana and stored all ingested data on a centralized repository i.e. AWS S3.
- Using AWS Glue, the team has performed different ETL actions on the data by writing complex SparkSQL queries and transferring processed data to AWS Redshift (Data warehouse)
- By configuring Athena, customer teams query the data in S3 buckets and write data modeling jobs based on the KRA’s given by the client.
- Finally, all processed data is fed to NanoBi where customer side teams access the processed data and create different reports for visualization.
Business Outcome
- With secured data lake implementation on AWS cloud platform, all the finance data is completely secured and easily available for all the downstream system, making all the data operations faster.
- Data-centric decision-making was a huge aid in client’s business growth. Teams were able to make independent, data-oriented decisions by pulling, analyzing, and visualizing data themselves.
- The analysis platform allowed the team to focus on strengthening analytic capabilities, benefitting client’s performance.
