Requests that take more than 60s to complete fails – ElasticBeanstalk

This short post specifically targets:
– RoR as web development environment – Can be ignored as this is closely related to nginx configuration settings
Nginx as web server


While working with ElasticBeanstalk environments, one of my requirement was to send a heavy job to Worker App that takes > 1 min (60 seconds) to complete and I had no configuration in place. The way beanstalk-app was handling that job was disappointing. That job never finishes within a minute and another job (duplicate) gets triggered by the app and this goes on till it reaches the max retry count.

I tried googling the solution and found that we have to increase the timeout for the web server which is handling the request. In my case it was `nginx`.

So I decided to write a config file under `.ebextensions/02_nginx_proxy.config` — I am not going into the details of `ebextensions` here as this is out of scope of this article.

Here’s how my config file looks like:

files:
/etc/nginx/conf.d/proxy.conf:
mode: "000755"
owner: root
group: root
content: |
client_max_body_size 20M;
proxy_send_timeout 600;
proxy_read_timeout 1h;
send_timeout 600;
server_names_hash_bucket_size 128;
upstream backend {
server unix:///var/run/puma/my_app.sock;
}
map $uri $preferred_proto {
default "https";
~^/(assets|public|packs)/ "none";
~^/health_check "http";
}
server {
listen 80;
if ($preferred_proto = "http") {
set $http_x_forwarded_proto $preferred_proto;
}
if ($preferred_proto = "none") {
set $preferred_proto $http_x_forwarded_proto;
}
if ($preferred_proto != $http_x_forwarded_proto) {
rewrite ^(.*) $preferred_proto://$host$request_uri redirect;
}
location / {
proxy_pass http://backend;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-NginX-Proxy true;
location /packs {
root /var/app/current/public;
gzip_static on;
gzip on;
expires max;
add_header Cache-Control public;
}
}
location /assets {
alias /var/app/current/public/assets;
gzip_static on;
gzip on;
expires max;
add_header Cache-Control public;
}
location /public {
alias /var/app/current/public;
gzip_static on;
gzip on;
expires max;
add_header Cache-Control public;
}
}

view raw
nginx_proxy.config
hosted with ❤ by GitHub

Here if you focus on lines:
`proxy_send_timeout 600;`
`proxy_read_timeout 1h;`
`send_timeout 600;`

This will prevent sending multiple requests of same job type until the current one finishes in 1h.

If you need any inputs on what other lines are all about/doing, then leave your comment.

Happy Coding!

Configure Delayed Job with Elastic Beanstalk

Recently, I got a chance to work with Elastic Beanstalk and trust me, it is not as friendly as Heroku and it took me a little while to get used to with Beanstalk deployments.

I am using delayed_job in my project and the requirement was very simple, i.e. to automate background jobs. On heroku we simply add a worker dyno and jobs start working immediately. But, same is not the case with Beanstalk. I came across with various article where people suggested to add a shell script inside `.ebextensions` but I did not find this a full proof solution. Then my colleague suggested me to try `active_elastic_job` @ https://github.com/tawan/active-elastic-job

I find this gem extremely helpful and the installation instructions are super simple.

In this article, I will not dig into how to configure, install and start using this gem (because if you visit the link, the steps are defined in simplest form and should not block you). My idea, is to highlight/cover the areas which are not mentioned there or missing.

1. When you create a worker environment, skip selecting the RDS section (which means you’ll create your worker environment without a RDS DB).

2. Now you’ll have to use the existing database (which your web environment is using). For that, copy all RDS environment variables from web environment to your worker environment.

3. Once above is done, deploy your the code to your worker environment. In case your deployment fails with reason:


PG::ConnectionBad: could not connect to server: Connection timed out

Then, go to Services ~> Elastic Beanstalk ~> You Web Application ~> Configuration ~> Instances ~> Modify and note down the Security Group there (using new UI of Beanstalk)

Now, go to Services ~> Elastic Beanstalk ~> Your worker application ~> Configuration ~> Instances ~> Modify and check the security group which your Web Application is using (Note that you can check as many as you want) and Save your changes. This should allow your worker app to use the same DB.

4. Once you have successfully deployed and configured your worker app and you get:


ActiveJob::QueueAdapters::ActiveElasticJobAdapter::SerializedJobTooBig
The job contains bytes in its serialzed form, which exceeds the allowed maximum of 262144 bytes imposed by Amazon SQS

That means you are not allowed to send more than 256KB of data to delayed job. This could prove a bottleneck and you’ll have to think of a workaround to deal with this error. Although, folks on internet suggested that we could use Amazon Extended Client Library for JAVA @ https://github.com/awslabs/amazon-sqs-java-extended-client-lib but I wasn’t able to use and apply this in my Rails app.

Having said that, if anyone knows how to use this in Rails, please drop a comment or as an alternate try to send data < 256KB to SOS queue

That’s all I have for now. See you soon! Happy Coding. 🙂

Learning RSPEC with Rails 4 (Alpha)

Hi Folks,

Hope you all are fine and doing good! I am sure most of you, are using TDD for years now. And, once we think of TDD, the first thing comes into our mind is, RSPEC. We all agree that Rspec is a great framework of Test-Driven-Development and its integration with other libraries (like fixtures etc.) is seamless.

With rapid revolution happening in Rails community, Rspec too has evolved over the years, and my motto is to bring this newly developed RSPEC framework in such a way so that, those who are eager to learn TDD can take tips and take this as a learning material. And those, who are planning to migrate can refer to this!

Environments I am using:

  • Ruby 2.1.2
  • Rails 4.2.1
  • Rspec 3.2.0

Idea is to share small code snippets while learning these changes, where we see: what is new. So let’s explore this now:

1. In Rspec 3, the support for ‘should‘ (e.g. foo.should == bar) is deprecated and is no longer supported. Instead of ‘should’, rspec encourages us to use ‘expect‘. Let’s see how it works:

require 'spec_helper'
class User
def initialize(args = {}); end
def email
'someone@example.com'
end
def mobile
12021679918
end
end
describe User do
describe "#initialize with old should way" do
it "should set the email and mobile for user" do
u = User.new email: 'someone@example.com', mobile: 12021679918
u.email.should == 'someone@example.com'
u.mobile.should == 12021679918
end
end
describe "#initialize with new expect way" do
it "expects email and mobile param for user" do
u = User.new email: 'someone@example.com', mobile: 12021679918
expect(u.email).to eq('someone@example.com')
expect(u.mobile).to eq(12021679918)
end
end
end

view raw
user_spec.rb
hosted with ❤ by GitHub

If you try running this example, you’ll get this deprecation warning message:

Deprecation Warnings:

Using `should` from rspec-expectations‘ old `:should` syntax without explicitly enabling the syntax is deprecated. Use the new `:expect` syntax or explicitly enable `:should` with `config.expect_with(:rspec) { |c| c.syntax = :should }` instead

 

Now, let’s take an example of controller spec. Create a controller with scaffold and write the spec as below:

# Controller /app/controllers/posts_controller.rb
def index
@posts = Post.all
end
# Spec /spec/controllers/posts_controller_spec.rb
describe "GET #index" do
it "responds successfully with a HTTP 200 status code" do
get :index
expect(response).to be_success
expect(response).to have_http_status(200)
end
it "should render index template" do
get :index
expect(response).to render_template('index')
end
end
# Controller /app/controllers/posts_controller.rb
def create
@post = Post.new post_params
respond_to do |format|
if @post.save
format.html { redirect_to @post }
else
format.html { render action: :new }
end
end
end
# Spec /spec/controllers/posts_controller_spec.rb
describe "POST #create" do
it "should create new post with valid params" do
post :create, post: { title: 'My First Post', description: 'My first post description' }
expect(Post.count).to eq(1)
end
it "redirects to show page once the post is created successfully" do
post :create, post: { title: 'My First Post', description: 'My first post description' }
expect(response).to redirect_to(Post.first)
end
it "does not allow to create post without title" do
post :create, post: { description: 'My first post description' }
expect(Post.count).to eq(0) # This make sure validates :title, presence: true in post.rb
end
it "renders new template if failed to create post" do
post :create, post: { description: 'My first post description' }
expect(response).to render_template('new')
end
end
# Controller /app/controllers/posts_controller.rb
def update
@post = Post.find params[:id]
respond_to do |format|
if @post.update_attributes(post_params)
format.html { redirect_to @post }
else
format.html { render action: :edit }
end
end
end
# Spec /spec/controllers/posts_controller_spec.rb
describe "PUT update/:id" do
before(:each) do
@post = FactoryGirl.create(:post)
end
it "should render edit template if unable to create post" do
@attr = { title: nil, description: 'My first post description' }
put :update, id: @post.id, post: @attr
expect(response).to render_template('edit')
end
it "redirects user on show post page" do
@attr = { title: 'My First Post', description: 'My first post description' }
put :update, id: @post.id, post: @attr
expect(response).to have_http_status(302)
expect(response).to redirect_to(@post)
end
it "validates the title and description of post after updation" do
@attr = { title: 'My 1st Post', description: 'My 1st post description updated' }
put :update, id: @post.id, post: @attr
@post.reload
expect(@post.title).to eql(@attr[:title])
expect(@post.description).to eql(@attr[:description])
end
end

view raw
controller_spec.rb
hosted with ❤ by GitHub

I’ll cover model, helper and view spec very soon.

Happy Coding 🙂

Add duplicate rows in Excel using Ruby

Happy New Year Folks,

Few days back a friend of mine came to me and said, “I am a RUBY Programmer and I am stuck in a requirement, wherein I have to read an excel-sheet (that contains multiple records and each record can have multiple entries in it.)”

Think of an example like given in the link: timesheet. It says we have 2 columns: EmployeeID and WorkingHours. Each EmployeeID can have multiple working hours.

Now, I would like to write a small Ruby program that iterate over each record in the excel, combine EmployeeID’s that has multiple entries and add working hours.

Let’s code it then!

Requirements:

  • Ruby 1.9.2 or higher
  • Spreadsheet gem: to read Excel
  • Download timesheet excel from the link above
require 'spreadsheet'
Spreadsheet.client_encoding = 'UTF-8'
class TimeSheet
def calculate_total_working_hours
employeeHash = {}
book = Spreadsheet.open 'timesheet.xls'
default_sheet = book.worksheet 0 # If there are multiple worksheet it will take the 1rst.
default_sheet.each 1 do |row| # This will start from the 1st record.
employeeHash[row[0]] = employeeHash.has_key?(row[0]) ? employeeHash[row[0]] + row[1] : row[1]
end
puts "Timesheet entry***** #{employeeHash.inspect}"
end
end
time_sheet = TimeSheet.new
time_sheet.calculate_total_working_hours

view raw
time_sheet.rb
hosted with ❤ by GitHub

Happy Coding! 🙂

Ruby Cheat-Sheet

Hi Folks,

In this new tutorial, I am going to show you what most of developers/programmers already know, but not able to keep track of such things. I call it as ‘Ruby cheat-sheet‘, you can call it by the name you like.

This will also help those people who have just started learning ‘Ruby’.

IMPORTANT NOTE:

  • I will keep on updating this article frequently.
  • Also need your valuable response, if you feel there is a better/easy concept available.
  • If I am not sure of something or don’t know the concept, I am going to highlight that with ‘Red‘ color. Later, you can put your opinion in the comment section and I will update that with your name.
  1. Symbols (e.b :name, :age etc.) as just a convenient way of referencing a name.
  2. You don’t need to per-declare a symbol.
  3. A ‘Class‘ is a combination of state (for example, the name of the animal) and method that use that state (perhaps the method how animal eats).
  4. The standard ruby constructor is ‘new‘.
  5. Every object in Ruby has a unique ‘Object identifier‘ (called as Object ID).
  6. Class have instance methods. These instance methods have access to the object’s instance variables.
  7. Methods are defined with the keyword ‘def‘ followed by method name.
  8. The most common way to create String objects in Ruby is ‘string literals‘. (E.g. “I am a string”, ‘I am another string’)
  9. The logical difference between ‘Single quotes‘ and “Double quotes” in Ruby is the amount of processing Ruby does. In double quote case, Ruby first looks for substitution and replaces them with some binary value. (E.g. puts “Good Night, \nAuthor”)
  10. Global variables: starts with ($) sign: E.g. $name
  11. Instance variable: starts with (@) sign: E.g. @name
  12. Class variables: starts with (@@) sign: E.g. @@name
  13. Class name, module name and Constants: Starts with the uppercase letter: E.g class MyClass, module Login
  14. Array stores collection of object with a key to access them.
  15. An array can have objects of different types.
  16. Hash contains key, value pair.
  17. Ruby treats ‘nil’ as false in conditions.
  18. A regular expression is a way of specifying a pattern of characters to be matched in a string.
  19. The match operator (=~) can be used to match a regular expression against a string.
  20. Yield (yield) is a Ruby method
  21. In Ruby, nil is also an Object
  22. When you run Ruby programs you can actually pass arguments. This concept is called command-line-arguments. For e.g. ruby my_example.rb firstname lastname city.
  23. The initialize method in Ruby lets us set the state of each object. When we call ClassName.new to create a new object, Ruby allocates some memory to hold an UN-initialized object.
  24. When we pass an object to puts based on the class, it writes the name of the object’s class, followed by a colon and a unique identifier.
  25. Ruby gives flexibility to its programmers. You can over-write the default methods. (E.g. def to_s)
  26. attr_reader is the modified way of accessor methods.
  27. attr_accessor provides you to read and write methods. (E.g. attr_accessor :age)
  28. Virtual attributes**
  29. require‘, ‘require_relative‘ & ‘load
  30. Access Controls – ‘Public‘, ‘Private‘ & Protected
  31. In ‘Ruby’ a variable is not an Object. It is simply reference to an Object.
  32. initialize‘ is a private method in Ruby.
  33. Block, Procs and Lambda are ways of grouping code we want to run. These are examples of closure in Ruby.
  34. Both Proc and Lambda are Proc Objects.
  35. Lambda check the number of arguments and throws argument error, whereas Proc does not.
  36. In “Inheritance“, if class B inherits the property of class A, then all the methods of class A becomes the member of class B.
    • Assuming child = Child.new; then child.superclass => Parent
    • parent.superclass => Object
    • Object.superclass => BasicObject
    • BasicObject.superclass => nil
  37. Modules: is a way of grouping together Methods, Classes and Constants.
    • Provide namespaces and prevent name clashes.
    • Supports the mix-in functionality.
    • A Module is not a class.
    • A Module cannot have instances.
    • We can always include a module in a class definition – After doing that all module instance methods are suddenly becomes available as methods in the class. They get mix-in.
    • mix-in modules don’t use instance variable directly. They use accessors to retrieve data.
  38. self.method_name: Gives you access to the current object.
  39. begin-rescue blockUse for exception handling.
  40. begin
    # Code to be executed here
    # This part will be 'protected'
    raise # Use this to raise any message
    rescue
    # Exception Handling, like argument error, 404
    retry # use this if you want the begin block to be executed again, if execption occurs.
    ensure
    # This part will always get executed.
    end

    view raw
    block.rb
    hosted with ❤ by GitHub

INTEGRATE ORACLE WITH RUBY

Since most of the rails application is now using various databases, this article will show you how you can useORACLE database with rails.

For that, we need Oracle XE

I will update this post as soon as I can…. Apologize for the dealy. However, you can mail me any of your query if you are stuck in configuring Oracle with Rails.

DON’T HESITATE TO TRY OUT NEW THINGS IN RUBY

Since last couple of months, I was trying installing Rails-3 on my Windows PC.
When I heard, that Rails-3 requires ruby-1.8.7 or higher, I instantly removed Ruby-1.8.6 and all its components and installed 1.8.7 from the fresh point.

I was happy, as my new system has Rails-3 now… Wow, what else I need?

But later, I found that my few of my old applications, which requires ruby-1.8.6 have stopped working. WTH!! Can’t it work with Ruby-1.8.7 now? With the investigation, I observed that, there were few libraries which were specific to 1.8.6 version and are not in 1.8.7 now. Some of them, I restored from 186 to 187 but were not compatible. Urrrgghhh!!! I am disappointed :-(

Now, I need some sort of version Manager, which will handle multiple version of Ruby. From here my search starts!!

After searching a lot on Google, and suggested by one of my friend, I found ‘pik‘.
Short Intro about PikPik is a version manager for Ruby. You can handle/install multiple versions of Ruby and switch between them easily.

Basic Requirement: You must have atleast one version of Ruby, up and running on Windows.

Installation: ‘pik’ comes as a gem. To install pik, follow these steps:

Dependencies:
> my current plateform: ruby-1.8.6. From the command prompt, type:
gem install pik
> After the successful installation of pik, type:
pik_install C:\tools
> Add this to your Path Environmental variables
My Computer > Right Click > Properties > Advanced System Settings > Environmental Variables > Path

Once you’re done. Open a new command prompt and type:
pik list
It will add the current version of ruby into the list.
Now what’s so special about pik? How it can handle the multiple versions? To see that in action, in command prompt, type:
pik install ruby 1.8.7
and this will start downloading and once downloaded it will install 1.8.7, under:
C:\Users\PC-name\.pik\rubies\Ruby-187-p302

Awesome!! isn’t it?
pik list
It will show you the current version of ruby and the other ones.
pik use 187
Will use now ruby-187. Don’t believe me? :-x Then type:
ruby -v

I am super excited now :-D I want to install Ruby-1.9.2 now. Afraid? Still hesitate? Don’t be, just follow the instructions…
From the command window, type:
pik install ruby 1.9.2
Heights of happiness… It is installing 1.9.2 now!!
Path will remain the same, i.e:
C:\Users\PC-Name\.pik\rubies\Ruby-192p302

Do a pik list now and you’ll see:
<image comes here />

Now, I am running multiple versions of Ruby on my Windows m/c and I can select any version by:
pik use 187
gem install rails --include-dependencies
rails s -p 4000

and can perform numerous operations… Yuppy!!!!

Here comes the another pain. Few days back I thought to work with Rhodes. The Mobile Framework for Ruby.

To install it, I went through the Rhodes website and followed the tutorials. I did:
pik use 187
gem install rhodes

I got make error :-x Why windows won’t work like the Linux does and why do we need to troubleshootC-compiler problems?

Another day I spent to look around for a solution and finally came up with Devkit.

Short Intro about Devkit: Devkit is a windows executable program, which creates native extensions for your Ruby gems, which uses C-complier.

Installation: Download the Devkit (Development Kit) from Ruby installer website. After downloading, extract it to:
C:\devkit
After extraction, open your command, go inside C:\devkit prompt and type:
ruby dk.rb init
This will create the config.yml inside C:/devkit. Now you need to define all the Ruby versions inside that file, so that devkit will create native extensions for them. To do that, open that file and at the end of it, write these lines:

- C:\ruby
- C:\Users\PC-name\.pik\rubies\Ruby-187-p330
- C:\Users\PC-name\.pik\rubies\Ruby-192-p0

Once done save it and from the command prompt, type:
ruby dk.rb install
and you should see something like this:

Awesome, easy steps, isn’t it? Now, what else you’re looking for? Go ahead, install any gem for any version of ruby without any hesitations :-)

Please let me know if you’re stuck or need any help. Good comments will always motivate me to write more and more and help the Ruby Community keep growing.

References taken from:

A NEW WAY OF INSTALLING PIK ON WINDOWS

Hi Guys,

In my previous article, Don’t hesitate to try out new things in Ruby on Rails, you got to know, how to install and configure pik to support multiple versions of Ruby on Windows.

In this article, I will add another simpler way to install pik on windows and configuring it for multiple versions of ruby. For Devkit, you can follow the same old article.

NOTE: I will go with Ruby192 and on that Ruby187 and so on… You can accomplish it in a reverse mode as well.

Step1: Download Ruby193 from here. Do install the RubyInstaller version i.e .exe file.

Step2: After downloading, Install it (click on that checkbox which will say, add Ruby to your environmental variables path).

Step3: Once the installation is done, open a command window and type:
ruby -v
It will tell you that, current Ruby is 1.9.3

Step4: Install the pik gem. Type:
gem install pik

Step5: Type- pik_install C:/tools

Step6: Add this path to your Environmental variables
My Computer > Right Click > Properties > Advanced System Settings > Environmental Variables > Path

Step7: Now, go to RubyInstaller.org and download the 7-zip archive of Ruby187 i.e Ruby 1.8.7-p358.

Step8: Once the download is finished, extract all the content of that file inside C:\ folder, using 7-zip utility. After extracting, rename the folder to Ruby187.

Step9: Open a new command window and type:
pik add C:\Ruby187\bin

Yuppy!! it will add 187 into the existing versions of Ruby. From here you can install ‘N’ number of Ruby versions on Windows.

Advantages of doing this:

If using Netbeans, you can now easily configure/add/manage multiple versions of Ruby.
If you follow the old approach, your pik version of Ruby won’t able to include the DevKit, due to which you may face problems, while installing Native extension gems. For e.g consider this path: C:/Users/foo bar/.pik/rubies/. It contains a space between foo and bar. So devkit skips this many of the times.
Happy coding…
Your feebback/suggestions are always welcome.

Learn Core Ruby Concepts

Hello Everyone,

We all know basic concepts of Ruby, like classes, objects, methods etc. etc and we integrate this with the concept of rails beautifully.

In short I can say we all are very good at Ruby on Rails, and rails specially, but when it comes to Ruby, we are not that much confident tough.

In this Post of mine, we will cover all the important concepts of Ruby, some pure ruby code, and how we can apply this with Rails.

So let’s start learning the Core Ruby Concepts.

  • In RoR programming, most of the times we need to identify the type of string. We don’t know if the incoming string is a “Integer”, “Float”, “String”. To check that:
    y = Incoming String
    y.is_a?(Integer) => returns true or false
    y.is_a?(String) => returns the same
    y.is_a?(Fixnum) => -----#-------
    y.is_a?(Float) => -----#-------

    • We can also ask the variable exactly what class of variable it is using the class method:
      y = "Incoming String"
      y.class => returns String
      y = 10.25
      y.class => returns Float
      y = 10 => returns Fixnum
    • Sometimes we need to change the incoming string to say Integer (Note: It is for sure that in RoR application, when browser sends any parameter to any controller’s method, it will be string only), Float for further operations. To do that:
      x = "10.25"
      y = x.to_f
      p y.class => returns Float
      z = y.to_i
      p z.class => returns Fixnum
  • Variable type: Ruby has four types of variables:
    • Local variable [a-z] or _
    • Global Variable $
    • Instance variable @
    • Class variable @@

    To identify the type of variable, do:
    a = 10
    p defined?(a) => "local-variable"
    $b = "Puneet"
    p defined?($b) => "global-variable"

  • Metaprogramming: Metaprogramming is a technique in which code writes other code. The prefix Meta refers to Abstraction.
    • At a high level, metaprogramming is useful in working towards DRY principle (Don’t Repeat Yourself).
    • Metaprogramming is primarily about simplicity. One of the easiest ways to get a feel for metaprogramming is to look for repeated code and factor it out. Redundant code can be factored into functions; redundant functions or patterns can often be factored out through the use of metaprogramming.

Minimise URL in Ruby

Does it looks odd?? You have so many websites which allows you to minimise your url like tinyurl, bit.ly etc etc list is endless, but what if you get some API of those to work with so you can minimise the url with your rails application or as a stand-alone ruby program??

Bit.ly comes with that. It provides ruby programmers an interface by which they can minimise the url. Wondering How? see it in action..

I am creating a simple ruby program here, if you want you can use it in your application…

All you need is json and open-uri to finish it off.

require ‘open-uri’
require ‘json’

code=’https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=zpwhtygjntrz&scc=1&ltmpl=default&ltmplcache=2&hl=en&#8217;
user=’YOUR USERNAME’
apikey=’YOUR API KEY’
version=’2.0.1′
url = “http://api.bit.ly/shorten?version=#{version}&longUrl=#{code}&login=#{user}&apiKey=#{apikey}”
buffer = open(url, “UserAgent” => “Ruby-ExpandLink”).read
result = JSON.parse(buffer)
shorturl = result[‘results’]['shortUrl']

That is it. Run this program in console/command prompt, where ever you want to see the output.

Post your queries, suggestions
Puneet