Problem

DDoS attackers use the same IPs to send many HTTP requests once the AWS WAF rate limit rule removes the block. The default block lasts only for a definite time, so attacks repeat again. We need a solution that makes the block time for harmful IPs last indefinitely, keeping them blocked until the attack persists. 

Solution Workflow

1. CloudFormation: Use the predefined CFT template to set custom block time for harmful IPs. Adjust by how severe the attack is.
2. EventBridge & Lambda: Let EventBridge call a Lambda function every minute. The function checks AWS WAF’s rate rule for blocked IPs.
3. Store in S3: Save blocked IPs in an S3 bucket with timestamps for records.
4. Update WAF Custom IP Sets: Lambda revises WAF custom IP sets by keeping IPs within block time. It also drops IPs that passed the block period.
5. Regular Updates: Run the process every minute to keep only harmful IPs blocked and avoid an outdated, heavy block list.

Deploying the Solution

1. Download the CloudFormation Template:
   Download the customized AWS CloudFormation template (customized-block-period-template.yaml) from the solution’s GitHub repository.
2. Create a Stack in CloudFormation Console:
   Open the AWS CloudFormation console, then create a new stack with the downloaded template. Check the CloudFormation User Guide for detailed instructions for stack creation.

3. Specify Stack Details:
   On the Specify Stack Details page, type a unique stack name. Enter the required parameters, such as blocking duration and configuration settings listed in the prerequisites.

4. Provisioning Resources:
   
   

The template provisions several AWS resources, including:

• AWS WAF IP Sets, which store the blocked IPs.
• An Amazon EventBridge Rule that triggers the Lambda function at regular intervals.
• Amazon S3 Buckets to store the blocked IP addresses and their timestamps.
• AWS IAM Roles with permissions to allow Lambda functions to query AWS WAF and access other required resources.
• The AWS Lambda function itself, which performs the logic for tracking and updating the blocked IP addresses.

5. Deploy and Apply the WAF Rule:
   Deployment takes under 15 minutes. When the stack shows CREATE_COMPLETE, build a custom AWS WAF rule to apply custom IP sets and block the malicious IPs.

6. Reviewing IPs that are Blocked:

Go to the IP Sets section on the AWS WAF console. Choose the set named with the prefix "IPv4-IPset." You can check the list of IPs blocked by the rate limit rule in the set produced by the stack.

7. Whitelisting or Removing Specific IPs from the Blocked List

To remove an IP from the blocked list, merely deleting it from the IP set in the AWS WAF console does not work. This is because the IP set updates every minute with a JSON file stored in an S3 bucket (controlled by the CloudFormation template).

To remove an IP properly, delete it from the JSON file; then upload the revised file to the S3 bucket. You may use a Lambda script to automate this process. The script lets you choose the IP to remove; it completes each required step.

You can find the environment variable details and the Python code for the script here:

 https://rentry.co/ew84t8tu

Blocking Requests Originating from Referrer URLs

Problem Statement: 

Third-party websites might copy images or content from your site and use them on their platforms. These requests come via referrer URLs.

Solution:

To block such requests, follow these steps:

1. Identify the Referrer URL:

• Open the site suspected of scraping your content in a browser.
• Right-click on the page and select Inspect to open the developer tools.
• Navigate to the Network tab and reload the page.
• Look for requests made to your site. For example, if the site https://www.webpagetest.org/ is scraping your images, you might find requests to your domain in the list.
• Identify the image being used (e.g., twitter.svg), and click on the request.

2. Retrieve the Referrer URL:

• In the request details on the right panel, locate the Headers section.
• Scroll to find the Referer value. This will show the URL of the site making the request (e.g., https://www.webpagetest.org/).

3. Block the Referrer in AWS WAF:

• Open the AWS WAF console and create a new Custom Rule.
• Set the Inspect field to Single Header.
• Use Referer as the Header Field Name.
• Set Match Type to Exactly matches string.
• Enter the referrer URL (e.g., https://www.webpagetest.org/) in the String to Match field.
• Set the Action to Block. You can optionally configure a custom response code for blocked requests.

Outcome

By enforcing this rule, you can block requests from specific referrer URLs stopping site mirroring and web scraping by third-party sites.