Hi,
Was wondering if anyone has had success autoscaling Sidekiq worker instances based on queue depth? If so I’d love to hear about your experiences.
Looking at the Convox docs I’d assume that I’d need to use something like this gem to publish my queue metrics to CloudWatch and then setup the autoscaling targets in my convox.yml?
To answer my own question, yes, thats how you do it!
Adding the gem to my app got all the metrics feeding direct into CloudWatch, and right now I’m doing simple scaling based on queue depth, appears to be working well:
scale:
count: 1-10
targets:
custom:
Sidekiq/EnqueuedJobs:
aggregate: max
value: 500Hi Chris, thanks for sharing this, that’s awesome! I was just scaling my Sidekiq workers based on CPU usage, but this is a much better approach, so I’ll set this up as well. Thanks!
this is exactly what I was looking for, thank you!
Also thanks for writing up this blog post, it was very helpful! Autoscaling Sidekiq Workers on Convox
I finally got this set up and it seems to be working very well.
I’ve been running some tests on my staging server, where I enqueued a large number of jobs:
It was able to start all 10 workers (the maximum I set) and reached a steady state, then after 20 minutes it started to stop them one-by-one, and seemed to stop one every 2 minutes. So I think it’s working pretty well! I’m tempted to change some of the intervals, but I think I’ll just keep it like this for now since it seems like a reasonable default.
Just a heads up, this appears to have stopped working with a recent rack update.
Now when I go into ECS and look at the AutoScaling policy it has changed to be based on CPU utilization, and also appears to be ignoring the cool down params that have been set.
As a workaround, you’re going to need to go into CloudWatch and create 2 alerts on your EnqueuedJobs metric, one for scaling up, and the other for scaling down, then manually (or via Terraform) add 2 auto scaling policies based on them to scale your worker count up and down.
Is there a way that i could scale on a per queue basis, i have a queue named watermark so if the number of enqueued jobs goes over a threshold lets say 100 can i autoscale it ?
sidekiq-worker-watermark:
command: bundle exec sidekiq -q watermark
scale:
count: 1-5
targets:
custom:
Sidekiq/QueueSize:
aggregate: max
value: 100
dimensions:
QueueName: watermark
i tried this, but doest seem to be working
edit #1
its working for scale-up, but doesn’t seem to be scaling time even after the default cool-down period has been exceeded
