According to research done by the AWS Shield Threat Research Team, up to 51% of traffic heading into typical web applications originates from scripts running on machines, also known as bots. A wide variety of bots – some wanted, some unwanted – are hitting your endpoints.
Wanted bots are crawling your sites to index them and make them discoverable by your customers; others are monitoring your site availability or performance. But most of the bot traffic is generated by unwanted bots: scripts probing for vulnerabilities, or copying your content to replicate it somewhere else without your consent. In addition to the security risk, serving this traffic causes unnecessary pressure on, and costs for, your infrastructure.
Protecting your website from this unwanted traffic is time-consuming and error-prone. Managing a set of rules is complex, with risks of blocking good traffic or authorizing traffic that should be blocked.
Introducing AWS WAF Bot Control
Today, we are introducing AWS WAF Bot Control to identify, raise visibility of, and take action against common bot traffic. AWS WAF Bot Control is integrated into AWS Web Application Firewall and can be managed centrally using AWS Firewall Manager for large enterprise use cases.
Bot Control analyzes request metadata such as TLS handshakes, HTTP attributes, and IP addresses to identify the source and purpose of a bot. It categorizes bot types such as scraper, SEO, crawler, or site monitor.
Once Bot Control recognizes the bot, you can block traffic coming from unwanted bots. You can simply accept the default action to block unwanted bot traffic as part of your WAF configuration, or you can customize the configuration. For example, you can use the custom response capability to return a tailored response according to bot identification, or flag the request by inserting a new header. Integration with AWS WAF allows you to visualize the extent of bot traffic to your applications and control this traffic via WAF rules.
Bot Control uses two new functionalities that we are adding to AWS WAF Managed Rule Groups today: labeling and scope down statements. AWS WAF labels are metadata added to the request as the result of a matching rule statement. These labels can be used in future rule statements. You can think of WAF labels like a variable in which you can temporarily store the result of a rule action and use it in a subsequent rule. In addition, AWS WAF labels emit CloudWatch metrics and show up in AWS WAF logs. AWS WAF labels can be useful for evaluating multiple statements with a Count action and then taking action based on the labels, or reusing logic across multiple rules, among other examples. AWS WAF Bot Control uses labels to emit various bot-related signals, allowing you to customize the behavior that suits your need.
Some application resources are less likely to be subject to bot traffic or to need protection. Today, we are also introducing the concept of scope down statements. Scope down statements allow you to define under which conditions the managed rule group will execute. This is similar to the scope down functionality provided for rate based rules in AWS WAF today. You may want to include a ScopeDownStatement to reduce costs on paid managed rule groups to limit evaluation to specific parts of your application, to avoid false positives, or to avoid latency impact for specific paths, among other use cases.
Using a combination of managed rule group configuration, labels and scope down statements, you can customize how you process requests that originated from bots.
AWS WAF Bot Control Benefits
Using AWS WAF Bot Control brings you three key benefits:
- Bot Control gives you free visibility into bot traffic activities. When you are using AWS WAF, you get pre-built dashboards showing which applications have high levels of bot activity based on sampled data.
- Bot Control reduces operational and infrastructure costs by reducing the traffic generated by scrapers, scanners, and crawlers. Bot Control blocks unwanted bot traffic at the edge before it can increase your application processing costs or negatively impact application performance.
- Bot Control is easy to deploy. You can easily add bot protection to Amazon CloudFront, Application Load Balancer, Amazon API Gateway, or AWS AppSync just by adding an AWS managed rule group to a web access control list (web ACL).
Let’s See How AWS WAF Bot Control Works
Adding AWS WAF Bot Control works the same as adding an AWS WAF Managed Rule; you can start with just a few clicks. Let’s see an example and connect to the AWS WAF console.
On the left part of the screen, you notice a new Bot Control menu that provides an overview of bot-related traffic seen on your web ACL, as well as a summary of which web ACL has Bot Control enabled. All AWS customers get these bot activity metrics as part of the AWS WAF free tier: the split between bot and non-bot requests, the number of blocked bot requests and the categories of bots.
For this walkthrough, I decide to protect one of my endpoints. I select Web ACLs on the left menu and click Create web ACL:
I enter the detail of my Web ACL and click Next at the bottom of the page:
Under Add rules and rule groups, I open Add rules and select Add managed rule groups:
On the Add managed rule groups screen, I expand AWS Managed rule groups and turn on Bot Control, Add to web ACL. At the bottom of the page (not shown below), I click Add rules.
Finally, I choose the default action for requests that do not match rules and click Next.
I keep all the default values on subsequent screens, I click Next three times and, finally, I click Create web ACL.
Bot Control is similar to the Web ACL you already used: when selecting a specific set of rules, I can see the number of matching requests and a group of samples.
When I select the Bot Control tab on the top, I now have access to bot-specific data.
Pricing and Availability
AWS WAF Bot Control is available today in all AWS Regions where AWS WAF is available. Just like other AWS WAF rules, AWS WAF Bot Control can filter traffic hitting your Amazon CloudFront distributions, your Application Load Balancer, Amazon API Gateway, and AWS AppSync.
Bot Control is a paid AWS Managed Rule that can be added to your web ACL. You will be charged $10 / month (prorated by the hour) for each time Bot Control is added to your web ACL. In addition, you will be charged $1 per million requests processed by Bot Control. Bot Control charges are in addition to the AWS WAF fees.
Bot Control free usage tier includes 10M free requests processed by Bot Control per month.
Learn more about AWS WAF Bot Control and add it to your AWS WAF today.
