Elastic Beanstalk

Requests that take more than 60s to complete fails – ElasticBeanstalk

Puneet PandeyLeave a comment

This short post specifically targets:
– RoR as web development environment – Can be ignored as this is closely related to nginx configuration settings
– Nginx as web server

While working with ElasticBeanstalk environments, one of my requirement was to send a heavy job to Worker App that takes > 1 min (60 seconds) to complete and I had no configuration in place. The way beanstalk-app was handling that job was disappointing. That job never finishes within a minute and another job (duplicate) gets triggered by the app and this goes on till it reaches the max retry count.

I tried googling the solution and found that we have to increase the timeout for the web server which is handling the request. In my case it was `nginx`.

So I decided to write a config file under `.ebextensions/02_nginx_proxy.config` — I am not going into the details of `ebextensions` here as this is out of scope of this article.

Here’s how my config file looks like:


client_max_body_size 30M;
proxy_send_timeout 600;
proxy_read_timeout 1h;
send_timeout 600;
upstream backend {
server unix:///var/run/puma/my_app.sock ;
}
server {
listen 80;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
server_name *.cmgresearch.net;
large_client_header_buffers 8 32k;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_buffers 8 32k;
proxy_buffer_size 64k;
proxy_pass http://backend;
proxy_redirect off;
location /assets {
root /var/app/current/public;
}
# enables WS support
location /cable {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade "websocket";
proxy_set_header Connection "Upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}

view raw

nginx_proxy.config

hosted with ❤ by GitHub

Here if you focus on lines:
`proxy_send_timeout 600;`
`proxy_read_timeout 1h;`
`send_timeout 600;`

This will prevent sending multiple requests of same job type until the current one finishes in 1h.

If you need any inputs on what other lines are all about/doing, then leave your comment.

Happy Coding!

Advertisement

Configure Delayed Job with Elastic Beanstalk

Puneet PandeyLeave a comment

Recently, I got a chance to work with Elastic Beanstalk and trust me, it is not as friendly as Heroku and it took me a little while to get used to with Beanstalk deployments.

I am using delayed_job in my project and the requirement was very simple, i.e. to automate background jobs. On heroku we simply add a worker dyno and jobs start working immediately. But, same is not the case with Beanstalk. I came across with various article where people suggested to add a shell script inside `.ebextensions` but I did not find this a full proof solution. Then my colleague suggested me to try `active_elastic_job` @ https://github.com/tawan/active-elastic-job

I find this gem extremely helpful and the installation instructions are super simple.

In this article, I will not dig into how to configure, install and start using this gem (because if you visit the link, the steps are defined in simplest form and should not block you). My idea, is to highlight/cover the areas which are not mentioned there or missing.

1. When you create a worker environment, skip selecting the RDS section (which means you’ll create your worker environment without a RDS DB).

2. Now you’ll have to use the existing database (which your web environment is using). For that, copy all RDS environment variables from web environment to your worker environment.

3. Once above is done, deploy your the code to your worker environment. In case your deployment fails with reason:


PG::ConnectionBad: could not connect to server: Connection timed out

Then, go to Services ~> Elastic Beanstalk ~> You Web Application ~> Configuration ~> Instances ~> Modify and note down the Security Group there (using new UI of Beanstalk)

Now, go to Services ~> Elastic Beanstalk ~> Your worker application ~> Configuration ~> Instances ~> Modify and check the security group which your Web Application is using (Note that you can check as many as you want) and Save your changes. This should allow your worker app to use the same DB.

4. Once you have successfully deployed and configured your worker app and you get:


ActiveJob::QueueAdapters::ActiveElasticJobAdapter::SerializedJobTooBig
The job contains bytes in its serialzed form, which exceeds the allowed maximum of 262144 bytes imposed by Amazon SQS

That means you are not allowed to send more than 256KB of data to delayed job. This could prove a bottleneck and you’ll have to think of a workaround to deal with this error. Although, folks on internet suggested that we could use Amazon Extended Client Library for JAVA @ https://github.com/awslabs/amazon-sqs-java-extended-client-lib but I wasn’t able to use and apply this in my Rails app.

Having said that, if anyone knows how to use this in Rails, please drop a comment or as an alternate try to send data < 256KB to SOS queue

That’s all I have for now. See you soon! Happy Coding. 🙂