From Manual to Automated: Transforming Deployment and Enhancing Security
Key Challenges
Crimson is having difficulties with its release process, which is mostly caused by problems with autoscaling. Site downtime during deployments is happening too often and is hurting business operations. The current deployment process is manual, which causes several problems, such as deployment delays, human mistakes that cause sites to go down, a lack of visibility into past deployments, and more problems as the organization grows.
Key Results
Crimson improved deployment efficiency by 40% and eliminated manual errors through automated CI/CD pipelines with Jenkins. Implementing EC2 auto-scaling enhanced system scalability, while AWS WAF and GuardDuty provided robust security. These changes led to better operational efficiency, reliability, and business continuity.
Overview
Crimson is having difficulties with the release process, issues with autoscaling. Site down during deployment is hurting business operations. The current deployment process is manual, causing issues, such as deployment delays, human errors that cause site down, lack of visibility into past deployments.
Challenges
Manual Deployment Legacy: Initial resistance and adaptation challenges in transitioning from manual deployment processes to automated CI/CD pipelines. Overcoming organizational inertia and ensuring buy-in from stakeholders for embracing DevOps practices.
Complexity in Autoscaling Strategies: Configuring and fine-tuning EC2 auto-scaling policies and strategies to optimize resource utilization without over-provisioning or under-provisioning.
Security Configuration and Management: Ensuring proper configuration of AWS WAF rules and AWS GuardDuty alerts to minimize false positives while effectively identifying and mitigating security threats.
Managing and responding to security incidents promptly based on alerts generated by monitoring tools.
Monitoring and Alert Fatigue: Handling a large volume of monitoring alerts and ensuring that critical alerts are prioritized and responded to promptly, while non-critical alerts are managed effectively.
Continuous Improvement and Adaptation: Continuous refinement of deployment pipelines and monitoring configurations to align with evolving business requirements and technological advancements. Balancing between innovation and stability while adopting new AWS services and integrating them into existing infrastructure.
Addressing these challenges required a combination of technical expertise, strategic planning, and collaborative efforts across teams to achieve the outlined key results and enhance Crimson's operational efficiency and reliability.
Solution
We set up EC2 auto-scaling to make sure that we had the right number of machines to handle the traffic. We made the Auto Scaling groups and put the appropriate servers in them. Here, we set the minimum and highest number of instances in the Autoscaling group to keep the limit. And added the strategy to shut down the old instances once the new ones are up and running and taking care of requests.
We set up automatic deployment by building the job in different environments with Jenkins Pipeline. Jenkins helps automate the Crimson deployment process so that resources are always made available in a safe way. This process meets the requirements of not making any manual changes in production and using only automated methods. When code is added to GitLab, it starts our Jenkins CI/CD workflow. We have set up different groups for different environments, like UAT and Production. Once deployment and changes have been made smoothly on UAT and the developers have given their approval, Production jobs can begin.
Once the changes are committed to the gitlab source, the developer will start the job. During the build, comments are run on the Jenkins server, and then the build files are copied to the target machine. Also, we integrated Jenkins to Slack so that when a deployment happens, we get a message with the progress and the amount of time it took to finish the job.
AWS WAF is configured to protect from common web exploits and attacks, like SQL injection, DDoS attacks, and cross-site scripting (XSS) attacks. We use AWS Managed Rules and Custom Rules to keep an eye on requests and either let them through or block them based on the conditions we put in the rules. This helps us understand the threats and see if the logs show any known requests.
We turned on AWS Config so that we could keep track of how resources have been set up and what changes have been made to their control over time. It tells us about the collection of resources and lets us know when the configuration of a resource changes. We connected AWS CloudTrail to Amazon S3 so that we could record and store logs and histories of setup changes.
Amazon GuardDuty is set up for finding threats to give full security results that can be seen and fixed. It keeps an eye on our AWS processes, like EC2, S3, EKS, Lambda, and RDS protection. By using AWS Inspector, we were able to automate our assessments so that our application could be scanned constantly to find security holes and network exposure across all of our accounts. All the monitoring metrics are configured with alarms for all thresholds.
Business Outcome
- Enhanced Scalability and Availability by implementation of auto-scaling and rolling updates.
- Streamlined and automated deployment across environments reduced the time by 40% eliminating risk of manual errors.
- Real time monitoring and security by configuring AWS WAF, Guard duty.
