Michael Buckbee
28 Apr 2023
If you've ever sat and watched your incoming traffic for any publicly accessible application, you'll definitely find someone out there is going through your site looking for vulnerabilities. Unfortunately, hackers are constantly probing apps looking for any vulnerability they can exploit. Here is a sample of the logs from one of my applications:
Apr 07 12:59:14 my-heroku-app heroku/router at=info method=GET path="/.git/config" host=example.com request_id=c086ebd5-9cc6-4511-9127-3792e2d21a1b fwd="185.220.101.206" dyno=web.1 connect=0ms service=10ms status=404 bytes=1966 protocol=https
Apr 08 12:33:58 my-heroku-app heroku/router at=info method=GET path="/.env" host=example.com request_id=4253e66b-d41e-4dd5-b29a-c15c33ed9288 fwd="52.15.212.3" dyno=web.1 connect=1ms service=7ms status=404 bytes=1966 protocol=https
Apr 11 07:02:13 my-heroku-app heroku/router at=info method=GET path="/admin" host=example.com request_id=ecd1e1bf-bb5f-4985-9fce-6c82d5c60401 fwd="13.236.36.245" dyno=web.1 connect=0ms service=10ms status=404 bytes=1902 protocol=http
Apr 11 15:04:35 my-heroku-app heroku/router at=info method=GET path="/mifs/.;/services/LogService" host=example.com request_id=07ba7400-e33c-40ea-b6dd-f9af7a8a3477 fwd="91.241.19.175" dyno=web.1 connect=3ms service=8ms status=404 bytes=1902 protocol=https
What is actually happening here? It's an automated bot looking for vulnerabilities within my Ruby on Rails application; they're testing my site to see if I've exposed anything sensitive or if I'm using an out-of-date version of a service. Developers often dismiss blocking probes like this as not worth their time, but they're typically the precursor to broader attacks. If they find something, they'll try to exploit it. If you're using Ruby on Rails, one of the most popular gems to help defend your application is Rack::Attack.
Rack is the middleware that sits between the raw HTTP requests hitting your application and your Rails application code. Rack runs alongside your Rails application on the same server hardware. It functions on a lower level of your stack, using fewer resources per inbound HTTP request. While you could develop similar functionality to Rack::Attack and put it in your Rails application, the performance benefits of stopping bad requests at a lower level of the stack are essential as even hobbyist attackers can generate millions of requests per hour.
We're big fans of Rack::Attack, but there's no denying that it can be hard to configure, near impossible to monitor and doesn't even attempt to help with more severe security issues.
We're working on something better: Wafris - an open source Web Application Firewall.
Rack::Attack is Rack middleware you add to your Rails application via a Ruby gem. Rack::Attack lets you specify IP addresses that should not be allowed to make requests of your application or to throttle IPs that exceed the rate-limiting rules that you set. It was released by Kickstarter back in 2012; since then, it has been downloaded over 33 million times. So it's safe to say it's been pretty battle-tested in large production environments.
We combine it with Redis, which is used as a datastore for remembering which IP addresses have been banned.
Rack::Attack helps stop attacks arising from abusive IP addresses. This takes two forms:
You identify specific IP addresses from your logs or elsewhere and stop them from accessing your application. Ex: Blocking all requests from 159.65.231.165 from interacting with your application. The downside of this is that you will need to manually add specific IP addresses to your configuration and maintain those over time.
You define rules around how often an IP address can perform requests. Ex: submitting a login request no more than 3 times a minute. This requires you to set up a dedicated Redis instance to hold the IP request information and track the throttling rules. Note: the above rules may be enough to stop an attack on your application but it's important to consider that even though Rack uses much fewer resources per request it's still
We've helped thousands of companies better secure their SAAS applications and enthusiastically recommend it to all of our Rails clients. So seriously, if you're an Expedited WAF customer and would like help configuring Rack::Attack please book a time for us to help you. However, it's important to note that while Rack::Attack is great at solving some specific security concerns, it is not a complete security solution. A few things to keep in mind:
The first step is to add it to your Gemfile:
$ bundle add rack-attack
Once you've done that, you'll need to configure it. You can do this by creating the file config/initializers/rack-attack.rb, and adding the rules to fit your needs. Their documentation has a really fantastic sample setup to get you started. The only thing I normally change is to make sure I can programmatically enable it via an ENV in development & test environments:
# config/initializers/rack-attack.rb
Rack::Attack.enabled = ENV['ENABLE_RACK_ATTACK'] || Rails.env.production?
# Fallback to memory if we don't have Redis present or we're in test mode
if !ENV['REDIS_URL'] || Rails.env.test?
Rack::Attack.cache.store = ActiveSupport::Cache::MemoryStore.new
end
# Your Custom Rules here
I'm also pragmatically setting the Rack::Attack.cache.store, so in production, I use Redis to persist the blocked IPs between app instances & reboots. In development & test environments, it's helpful to enable Rack::Attack to confirm it's running as expected.
Most Rails applications aren't served from a single-threaded application but as a set of back-end instances fronted by another server. This introduces a problem of configuration for Rack::Attack as you want your rules applied universally. Below, we list three options for handling this:
You can bake Rack::Attack rules directly into your application configuration. While this might seem unwieldy, there's a variety of scenarios where this can be advantageous.
If you don't need to enable dynamic IP throttling and need to block a handful of IP addresses, then setting which to block via an Environment Variable (or as Heroku calls them: Config Vars) In your settings, declare a new environment variable called BLOCKED_IPS
BLOCKED_IPS=31.13.114.65,204.15.20.0
In your initialization file split that variable into an array of IPs
# config/initializers/rack-attack.rb
bad_ips = ENV['BLOCKED_IPS'].split(',') # Pulled from a external API on app boot.
# Block these IPs
Rack::Attack.blocklist "Block IPs from Environment Variable" do |req|
bad_ips.include?(req.ip)
end
If you're under an actual full scale attack or want to use Rack::Attack as anything more than as a means to block a few misbehaving IP addresses you'll need to set up a new dedicated Redis instance for your Rack::Attack installation. Rack::Attack uses Redis to share blocked and throttled IP address information between instances. If you're using Heroku, I'd encourage you to use Heroku Redis as your provider. You should use a dedicated instance just for handling your Rack::Attack data. Otherwise, it is quite likely that you'll grind your application to a halt as multiple services fight for a limited pool of Redis connections.
# config/initializers/rack-attack.rb
Rack::Attack.cache.store = Redis.new(url: ENV['REDIS_FOR_RACK_ATTACK_URL']) if ENV['REDIS_FOR_RACK_ATTACK_URL']
Configuring Rack::Attack is pretty daunting, so one of the best ways to build confidence in your configuration is by writing a test around it. This is an approach I've been using in my applications. I use an Environmental Variable to dynamically enable Rack::Attack to conditionally use it while in development mode.
# config/initializers/rack-attack.rb
Rack::Attack.enabled = ENV['ENABLE_RACK_ATTACK'] || Rails.env.production?
# Limit this endpoint to 10 requests a minute
Rack::Attack.throttle("req/ip/comments-tries", limit: 10, period: 1.minute) do |req|
req.ip if req.path.include?('/comments') && req.post?
end
When testing via RSpec, I often explicitly enable Rack::Attack before the start of the test & disable it afterward. For the actual test, I will fire requests at an endpoint with the expectation that it blocks the IP address after the threshold has been reached.
# spec/requests/rack_attack_spec.rb
require "rails_helper"
RSpec.describe "Rack::Attack", type: :request do
# Include time helpers to allow use to use travel_to
# within RSpec
include ActiveSupport::Testing::TimeHelpers
before do
# Enable Rack::Attack for this test
Rack::Attack.enabled = true
Rack::Attack.reset!
end
after do
# Disable Rack::Attack for future tests so it doesn't
# interfere with the rest of our tests
Rack::Attack.enabled = false
end
describe "POST /comments" do
let(:valid_params) { {comment: {body: "Sample"}} }
# Set the headers, if you'd blocking specific IPs you can change
# this programmatically.
let(:headers) { {"REMOTE_ADDR" => "1.2.3.4"} }
it "successful for 10 requests, then blocks the user nicely" do
10.times do
post comments_path, params: valid_params, headers: headers
expect(response).to have_http_status(:found)
end
post comments_path, params: valid_params, headers: headers
expect(response.body).to include("Retry later")
expect(response).to have_http_status(:too_many_requests)
travel_to(10.minutes.from_now) do
post comments_path, params: valid_params, headers: headers
expect(response).to have_http_status(:found)
end
end
end
end
The sample configuration provided by Rack::Attack is a good starting point, but you'll probably have quite specific needs based on your application requirements. Here are a few I've collected over the last few years which might help you out.
If you rely on a 3rd party service for your application, it would be very awkward if you accidentally blocked their IP addresses. Most services (Such as Stripe) will list the IP addresses used for Webhooks in their documentation, so you can safelist these IPs. Often I allow these IPs to perform certain requests without any limits because they're trustworthy:
# config/initializers/rack-attack.rb
# Get the IP addresses of stripe as an array
stripe_ips_webhooks = Net::HTTP.get(URI('https://stripe.com/files/ips/ips_webhooks.txt')).split("\n")
# Allow those IP addresses to send us as many webhooks as they like
Rack::Attack.safelist('allow from Stripe (To Webhooks)') do |req|
req.base_url == "https://webhooks.example.com" && req.post? && stripe_ips_webhooks.include?(req.ip)
end
With this example, we're calling the safelist method, and whenever it's a request which looks like a webhook query and is from an IP address Stripe has told us is one of theirs, we make sure not to block it.
If you're very security conscious, there are services out there which collate known compromised IP addresses (Such as the IP Abuse Database). You can configure Rack::Attack to block these known IP Addresses when you start your app like this:
# config/initializers/rack-attack.rb
bad_ips = ['31.13.114.65', '204.15.20.0'] # Pulled from a external API on app boot.
# Block these IPs
Rack::Attack.blocklist "block known bad IP address" do |req|
bad_ips.include?(req.ip)
end
# spec/requests/rack_attack_spec.rb
describe "Rack::Attack.blocklist - IP set to 204.15.20.0", type: :request do
let(:headers) { {"REMOTE_ADDR" => "204.15.20.0"} }
before do
Rack::Attack.enabled = true
Rack::Attack.reset!
end
it "Does not allow access to blocked IP address" do
get comments_path, headers: headers
expect(response.body).to include("Forbidden")
expect(response).to have_http_status(:forbidden)
end
end
This is a very proactive technique, which, while you'll need to refresh the list often, will help reduce a lot of junk attacking your application. Pretty much if an IP is considered suspicious, it won't be allowed to access your application. The main downside to this approach is your application's boot-up time will be reduced, as it has to wait for the external database of IP addresses to be loaded. Plus, you'll have to consider how to handle the external service being unavailable (e.g. caching the IP addresses in your local Redis Store or adding a rescue block).
For endpoints where users are likely to use them, but bots are also expected to try to poke them, it's useful to use the Allow2Ban approach. It'll allow a certain amount of requests through, and then after too many suspicious requests, it'll block that IP address from accessing that endpoint for a certain amount of time.
# config/initializers/rack-attack.rb
# Lockout IP addresses that are hammering your login page.
# After 5 requests in 1 minute, block all requests from that IP for 1 hour.
Rack::Attack.blocklist('allow2ban login scrapers') do |req|
# `filter` returns false value if request is to your login page (but still
# increments the count) so request below the limit are not blocked until
# they hit the limit. At that point, filter will return true and block.
Rack::Attack::Allow2Ban.filter(req.ip, maxretry: 5, findtime: 1.minute, bantime: 1.hour) do
# The count for the IP is incremented if the return value is truthy.
(req.path == '/users/sessions' && req.post?)
end
end
# spec/requests/rack_attack_spec.rb
describe "Rack::Attack::Allow2Ban.filter", type: :request do
let(:headers) { {"REMOTE_ADDR" => "124.15.20.0"} }
before do
Rack::Attack.enabled = true
Rack::Attack.reset!
end
it "Allows 5 requests, then bans from the entire site" do
5.times do
post '/users/sessions', headers: headers
expect(response).to have_http_status(:success)
end
# Banned for posting again
post '/users/sessions', headers: headers
expect(response.body).to include("Forbidden")
expect(response).to have_http_status(:forbidden)
# Banned from the rest of the site
get comments_path, headers: headers
expect(response).to have_http_status(:forbidden)
# Resets in 60 minutes
travel_to(61.minutes.from_now) do
get comments_path, headers: headers
expect(response).to have_http_status(:success)
end
end
end
With the above approach, I'm limiting the number of post requests to the devise login endpoint. If a user posts more than five times, it'll block them from that endpoint for 1 hour. You'll need to decide on the proper limits for your application, but I've generally found that it's probably a bot if a user has failed to log in more than five times.
Sometimes you have endpoints you know will only be visited by bots looking to find security holes within your site. For these IPs, you want to block them from your site altogether. In this case, Fail2Ban is the best approach.
# Block suspicious requests for '/etc/password' or wordpress specific paths.
# After 3 blocked requests in 10 minutes, block all requests from that IP for 5 minutes.
Rack::Attack.blocklist('fail2ban pentesters') do |req|
# `filter` returns truthy value if request fails, or if it's from a previously banned IP
# so the request is blocked
Rack::Attack::Fail2Ban.filter("pentesters-#{req.ip}", maxretry: 3, findtime: 10.minutes, bantime: 30.minutes) do
# The count for the IP is incremented if the return value is truthy
CGI.unescape(req.query_string) =~ %r{/etc/passwd} ||
req.path.include?('/etc/passwd') ||
req.path.include?('wp-admin') ||
req.path.include?('wp-login')
end
end
# spec/requests/rack_attack_spec.rb
describe "Rack::Attack::Fail2Ban.filter", type: :request do
let(:headers) { {"REMOTE_ADDR" => "125.15.20.0"} }
before do
Rack::Attack.enabled = true
Rack::Attack.reset!
end
it "Forbids users from visiting Wordpress path, then blocks user from whole site for 30 minutes" do
3.times do
get '/wp-login.php', headers: headers
expect(response.body).to include("Forbidden")
expect(response).to have_http_status(:forbidden)
end
# Banned from the rest of the site
get comments_path, headers: headers
expect(response).to have_http_status(:forbidden)
# Resets in 30 minutes
travel_to(31.minutes.from_now) do
get comments_path, headers: headers
expect(response).to have_http_status(:success)
end
end
end
In this example, if an IP address touches an endpoint we consider highly suspicious, we'll block them from our entire site for 30 minutes. It should be enough time to let a bot know we're onto them, and hopefully, they'll move onto their next target. Fail2Ban looks quite similar to Allow2Ban, but the critical difference is Fail2Ban will not allow the suspicious request through to your application while Allow2Ban will. So Fail2Ban should be used for endpoints you're very confident aren't going to be used by real users.
You may have an expensive endpoint on your website you'd instead not have accessed more than a few times in a minute. So you may want to limit it to a few users at a time. For this use case, the throttle method is the right choice. You can either set up a limit per IP address:
# config/initializers/rack-attack.rb
# Limit this endpoint to 10 requests a minute per an IP
Rack::Attack.throttle("req/ip/comments-tries", limit: 10, period: 1.minute) do |req|
req.ip if req.path.include?('/comments') && req.post?
end
# spec/requests/rack_attack_spec.rb
describe "Rack::Attack.throttle - req/ip/comments-tries", type: :request do
let(:headers) { {"REMOTE_ADDR" => "115.15.20.0"} }
let(:valid_params) { {comment: {body: "Sample"}} }
before do
Rack::Attack.enabled = true
Rack::Attack.reset!
end
it "successful for 10 requests, then blocks the endpoint for that user" do
10.times do
post comments_path, params: valid_params, headers: headers
expect(response).to have_http_status(:found)
end
post comments_path, params: valid_params, headers: headers
expect(response.body).to include("Retry later")
expect(response).to have_http_status(:too_many_requests)
# We can visit other paths
get comments_path, headers: headers
expect(response).to have_http_status(:ok)
# Resets in 10 minutes
travel_to(10.minutes.from_now) do
post comments_path, params: valid_params, headers: headers
expect(response).to have_http_status(:found)
end
end
end
Alternatively, you can also set the limit more generally & not make it IP address specific:
# config/initializers/rack-attack.rb
# Limit this endpoint to 10 requests in a 5 minute period for everyone
Rack::Attack.throttle("expensive-endpoint/posts", limit: 10, period: 5.minute) do |req|
"posts-endpoint" if req.path.include?('/posts') && req.post?
end
This approach is pretty standard within the API world. An API might limit the amount of requests per key or limit the number of write requests a User can make. However, even outside of the API world, you might have endpoints where they perform many operations, so it'll be prudent to add a layer of protection to them & this solution is suitable for that.
You may have a use case where you'd like to log that a certain type of user is doing something, in particular, you'd like to track more closely. Perhaps you might want to add some kind of stats around a certain user agent, or track if an endpoint is being used. You can achieve this by using the track method:
# config/initializers/rack-attack.rb
Rack::Attack.track("old-endpoint") do |req|
req.path.include?('/old-endpoint')
end
# Track it using ActiveSupport::Notification
ActiveSupport::Notifications.subscribe("rack.attack") do |_name, _start, _finish, _request_id, payload|
if false && payload[:request].env['rack.attack.matched'] == "old-endpoint" && payload[:request].env['rack.attack.match_type'] == :track
Rails.logger.info "Old Endpoint Touched: #{payload.path}"
# Maybe also fire an alert to Slack.
end
end
# spec/requests/rack_attack_spec.rb
describe "Rack::Attack.track", type: :request do
let(:headers) { {"REMOTE_ADDR" => "115.15.20.0"} }
before do
Rack::Attack.enabled = true
Rack::Attack.reset!
end
it "Tracks access to /old-endpoint" do
expect(Rails.logger).to receive(:info).with("Old Endpoint Touched: /old-endpoint")
get '/old-endpoint'
end
end
In the above example, we're checking if someone is still using an old endpoint on our app which could be used to help us decide if it's safe to deprecate it.
I once woke up to a bunch of unread texts on my phone from my boss. He had accidentally locked himself out of our Admin panel because he forgot his password. As we were using Heroku, the fastest approach was to fire up the Rails Console in production and run:
# Clears all the Redis store
Rack::Attack.cache.store.flushdb
# Clears all the Rack::Attack Keys
Rack::Attack.reset!
This cleared all the Redis stores & unblocked everyone, which was OK. However, for all our future accidental blocks we came up with the approach of:
$ redis-cli -u redis://127.0.0.1:6379/1 # Replace this with your REDIS_URL from Heroku
$ keys *
# 1) "rack::attack:26990977:req/ip/comments-tries:1.2.3.4"
$ 127.0.0.1:6379[1]> keys *1.2.3.4:*
# 1) "rack::attack:26990977:req/ip/comments-tries:1.2.3.4"
$ 127.0.0.1:6379[1]> del rack::attack:26990977:req/ip/comments-tries:1.2.3.4
# (integer) 1
With this approach, we connected directly to our production Redis server, then listed all our keys, it was kind of a big list, so we then filtered by IP address. Once we had the list of places that the user was banned, we could delete the keys which contained the user's IP address.
Rack::Attack is essential for most Rails Applications. It adds a layer of security that just stops traffic from mischievous sources from wasting your CPU cycles. I think even if you have fairly basic setup without too much customization it's a good place to start.
Start blocking traffic in 4 minutes or less