Data Analytics Pipeline Setup
.png)
Key Challenges
Unosecur needed to process real-time CloudTrail logs without duplication, handle large data volumes, and ensure cost-efficiency while transforming data for insights. Real-time updates every 4 hours and cost optimization were key challenges.
Key Results
The AWS EMR-based pipeline processed data efficiently, meeting the 4-hour update requirement. Spot instances reduced costs, while AWS Glue ensured seamless schema management. Redshift dashboards provided real-time insights, and CloudWatch alerts ensured minimal downtime, delivering a scalable and cost-effective solution.
Overview
Unosecur's mission is to simplify cloud security by addressing challenges associated with managing excessive permissions and identity risks in cloud environments. To provide businesses with actionable insights, Unosecur required a robust solution for processing and analyzing large volumes of log data.
The data, primarily in the form of AWS CloudTrail logs, was continuously generated and stored in an S3 bucket. The goal was to build an automated and scalable pipeline to process these logs, transform the data, and make it accessible for analysis and visualization on a dashboard updated every 4 hours.
Challenges
Continuous Data Ingestion:
CloudTrail logs were generated in real-time and stored in S3, requiring a system capable of processing incremental data without duplication.
Fetching the data in real time for the services or activity which does not come under the event-based pattern.
Data Transformation:
The log data needed to be filtered, transformed, and structured before loading into a data warehouse.
Handling Large Data Volumes:
The client required a scalable solution to efficiently process and analyze large volumes of data generated across multiple AWS accounts.
Cost Optimization:
The solution needed to minimize infrastructure costs while maintaining performance.
Real-Time Updates:
Insights from the data had to be available within 4-hour intervals, necessitating frequent data processing cycles.
Architecture:

Solution
The solution utilized Amazon EMR and AWS Glue Data Catalog, Glue job for a scalable, cost-effective data analytics pipeline:
Data Ingestion and Processing:
A glue job and Spark application were developed to process the CloudTrail logs and real-time data from the Application deployed on an EC2 instance to fetch the information regarding the AWS account and store it in an S3 bucket. The Spark application performed transformations such as filtering logs, restructuring data, and extracting relevant fields.
Persistent EMR Clusters:
An EMR cluster was deployed in Persistent Mode. The cluster executed the Spark streaming application reading the data in real-time from S3 and performing the operation and writing to another layer of S3.
Instance Configuration:
The EMR cluster used memory-optimized EC2 instances for efficient processing:
1 Master Node: r5.xlarge
2 Core Nodes: r5.xlarge
2 Task Nodes: r5.2xlarge (configured as spot instances, with fallback to on-demand instances when needed).
Metadata Management:
AWS Glue Data Catalog was used to manage unstructured data from CloudTrail logs. Glue crawlers were configured to update schemas dynamically as new fields were introduced.
Batch Job Configuration:-
We used Glue to read the data from MongoDB daily, calculate the metrics required for the analytics, and write them down in the s3 bucket.
Incremental Data Loading:
Checkpointing was implemented using Spark to ensure that only new logs were processed during each cycle. The processed data was loaded into a Redshift data warehouse.
Data Visualization:
Dashboards powered by Redshift allowed users to access insights in near real-time.
The data warehouse enabled efficient querying for analytics.
Monitoring and Alerting:
CloudWatch alerts were set up to monitor the EMR cluster and notify the team of any issues.
Key Features
Scalability:
The EMR cluster’s auto-scaling and transient nature ensured efficient processing of large data volumes without unnecessary resource allocation.
Cost Efficiency:
Spot instances reduced infrastructure costs significantly. On-demand instances were used only when spot capacity was unavailable.
Schema Management:
AWS Glue Data Catalog provided dynamic schema updates, allowing seamless handling of evolving data structures.
Incremental Processing:
Checkpointing prevented duplicate data processing and ensured that only new logs were loaded into Redshift.
Streamlined Integration:
The pipeline integrated seamlessly with S3, EMR, Glue, and Redshift, creating an end-to-end solution.
Dashboards and Alerts:
The solution provided real-time insights and proactive monitoring of potential issues.
Efficient Data Processing:
Spark on EMR enabled high-performance data transformations, meeting the 4-hour update requirement.
Cost Optimization:
The use of transient EMR clusters and spot instances minimized operational expenses.
Improved Data Management:
AWS Glue Data Catalog facilitated seamless metadata handling and schema updates.
Real-Time Insights:
Dashboards powered by Redshift provided timely and actionable insights for Unosecur and its customers.
Enhanced Monitoring:
CloudWatch alerts ensured prompt detection and resolution of issues, reducing downtime.
Business Outcome
Efficient Data Processing:
The Spark application on EMR allowed the high-performance transformation of CloudTrail logs, ensuring the data was processed within the required 4-hour intervals.
Cost Optimization:
Transient EMR clusters and the use of spot instances for task nodes significantly reduced operational costs. On-demand instances served as a reliable fallback when spot capacity was unavailable, balancing performance and cost.
Improved Data Management:
AWS Glue Data Catalog streamlined schema management, adapting to changes in the log data structure without requiring manual intervention. The metadata was consistently updated, enabling effective querying in Redshift.
Real-Time Insights:
Dashboards powered by Redshift provided stakeholders with actionable insights, enabling timely decision-making. The pipeline’s efficient design ensured minimal delay between log generation and data availability.
Enhanced Monitoring:
CloudWatch alerts ensured that any issues with the EMR cluster or data pipeline were promptly detected and addressed. The monitoring setup minimized downtime and maintained operational reliability.
Scalable and Future-Ready Architecture:
The EMR-based solution was designed to easily handle growing data volumes, ensuring long-term scalability and reliability for Unosecur’s evolving requirements.
Conclusion
This AWS EMR-based data analytics pipeline enabled Unosecur to process CloudTrail logs efficiently, delivering real-time insights to stakeholders. By combining scalability, cost-efficiency, and seamless integration, the solution supported Unosecur’s mission to enhance security and decision-making in cloud environments.
This architecture is future-ready, capable of scaling as Unosecur’s data volumes grow, ensuring reliable and efficient data analytics for years to come.