Productive with Ruby on Rails (RoR)

Tuesday, July 19, 2011

Hosting Ruby on Rails on Amazon AWS with Ruby Enterprise Edition

Tutorial on hosting a web application (Ruby on Rails - RoR) on Amazon AWS with EC2, EBS, Ruby Enterprise Edition (REE) and Phusion Passenger (mod_rails).

Author: K. C. Ramakrishna (www.rknowsys.com)

License: GNU-FDL - http://www.gnu.org/licenses/fdl.html

Date: 18-Nov-08

Please mail comments/corrections to kcr AT rknowsys DOT com, kcr AT members DOT fsf DOT org,

Note: This is a follow up to the tutorial on hosting the site with mongrel.
This tutorial can be found here: http://docs.google.com/Doc?id=dcn2ckbh_20hk4kc4d4
You may have to read it for some optimisations of RoR app to use EBS.

Credits and Disclaimer: Quite a bit of stuff exists in various tutorials - especially the part where we configure mysql to use EBS - This excellent tutorial is on AWS resources page (I don't have the exact URL). We just made an effort to consolidate a lot of relevant stuff into one document with a consistent form. This is the general outline you need to follow for any app on AWS.

Background:

We built a tourism portal in RoR for one of our clients. You have a look at it - - www.tripladder.com

After building it, we were requested to host and manage it for them. Initially we went with knownhost which is OK but a production RoR application needs more RAM than what we get on most VPS plans - especially if we have image processing. We did consider AWS but at that time it did not have EBS and the client did not initially expect enough traffic to justify a 'scalr' managed cluster. We were looking for a replacement to a dedicated server. Once EBS was launched, we immediately decided to move the site to AWS. The Cost-benefit analysis is compelling.

Update:
tripladder.com is not live anymore, please look at www.vitalizeu.com but the below process is still good.

The following tutorial starts off after signing up with AWS and configuring your desktop/laptop to be able to connect to AWS and launch instances i.e. we assume that you have completed the 'Getting Started' section of AWS.

We have started with the stock Fedora image and modified it to our requirements. We could have used CentOS but Fedora-8 appeared at the top of the list and we went ahead with it.

The application hosting has the following steps.

Launching an instance.

Installing RoR, gems, plugins...We used rmagick, hence we had to install Imagemagick too.

Installing REE and Phusion (mod_rails)

Installing mysql.

Intalling the application (checkout from subversion).

Creating and attaching a EBS volume. Mysql with data on EBS

Modifying the RoR app to save user upload files to EBS.(http://docs.google.com/Doc?id=dcn2ckbh_20hk4kc4d4)

Installing and configuring a production level ferret server

Configuring Apache to serve the application, caching optimisations for performance.

Configuring permanent public IP (covered) and DNS (we have the domain parked with go daddy but this is not covered in this article)

Configuring smtp (email) support for RoR application.

Once we have the perfect server setup, save it to S3.

Periodic automated backups - Using Amazon snapshots.

1. Launching an instance and logging in:

guest@kc-laptop:~$ ec2-run-instances ami-2b5fba42 -k gsg-keypair -z us-east-1a

RESERVATION      r-a864bec1      526262918289    default

INSTANCE        i-xxxxxxxx      ami-2b5fba42                    pending gsg-keypair     0               m1.small        2008-11-09T10:19:29+0000        us-east-1a      aki-a71cf9ce    ari-a51cf9cc

guest@kc-laptop:~$ ec2-describe-instances i-xxxxxxxx

RESERVATION     r-a864bec1      526262918289    default

INSTANCE        i-xxxxxxxx      ami-2b5fba42    ec2-xx-xxx-49-xxx.compute-1.amazonaws.com       domU-12-31-39-00-68-93.compute-1.internal       running gsg-keypair     0               m1.small        2008-11-09T10:19:29+0000        us-east-1a      aki-a71cf9ce    ari-a51cf9cc

# Authorise access on port 22 for ssh
guest@kc-laptop:~$ ec2-authorize default -p 22
GROUP           default
PERMISSION              default ALLOWS tcp     22    22    FROM    CIDR    0.0.0.0/0

# Authorise access on port 80 for http
guest@kc-laptop:~$ ec2-authorize default -p 80
GROUP           default
PERMISSION              default ALLOWS tcp     80    80    FROM    CIDR    0.0.0.0/0

# Authorise access on port 3306 for mysql admin.
guest@kc-laptop:~$ ec2-authorize default -p 3306
GROUP           default
PERMISSION              default ALLOWS tcp     3306    3306    FROM    CIDR    0.0.0.0/0

1.1 Associating a permanent IP address with this instance.

guest@kc-laptop:~$ ec2-allocate-address
ADDRESS 75.101.158.167
guest@kc-laptop:~$ ec2-associate-address -i i-xxxxxxxx 75.101.158.167
ADDRESS 75.101.158.167 i-xxxxxxxx

1.2 Check by logging in

guest@kc-laptop:~$ ssh -i ~/AWS-do-not-delete/id_rsa-gsg-keypair root@ec2-xx-xxx-49-xxx.compute-1.amazonaws.com

OR

guest@kc-laptop:~$ ssh -i ~/AWS-do-not-delete/id_rsa-gsg-keypair root@75.101.158.167

Server is accessible;
Went to godaddy.com (or your registrar) and set the DNS records accordingly:
tripladder.com <==> 75.101.158.167

2. Updating the system and installing required software

2.1 Updating the system

[root@domU-12-31-39-00-68-93 ~]# yum update

Updated: fedora-release.noarch 0:8-6.transition

Complete!

2.2 Installing mysql

[root@domU-12-31-39-00-68-93 ~]# yum install mysql mysql-server mysql-devel

Installed: mysql-devel.i386 0:5.0.45-6.fc8 mysql-server.i386 0:5.0.45-6.fc8

Dependency Installed: device-mapper-devel.i386 0:1.02.22-1.fc8 e2fsprogs-devel.i386 0:1.40.4-2.fc8 keyutils-libs-devel.i386 0:1.2-2.fc6 krb5-devel.i386 0:1.6.2-14.fc8 libselinux-devel.i386 0:2.0.43-1.fc8 libsepol-devel.i386 0:2.0.15-1.fc8 mysql.i386 0:5.0.45-6.fc8 mysql-libs.i386 0:5.0.45-6.fc8 openssl-devel.i386 0:0.9.8b-17.fc8 perl-DBD-MySQL.i386 0:4.005-2.fc8.1 perl-DBI.i386 0:1.58-2.fc8 zlib-devel.i386 0:1.2.3-14.fc8

Complete!

2.3 Installing httpd server, tools required for installing REE and subversion (for checking out the code).

[root@domU-12-31-39-00-68-93 ~]# yum install gcc-c++ httpd httpd-devel subversion

Installed: gcc-c++.i386 0:4.1.2-33 httpd-devel.i386 0:2.2.9-1.fc8 subversion.i386 0:1.4.4-7

Dependency Installed: apr.i386 0:1.2.11-2 apr-devel.i386 0:1.2.11-2 apr-util.i386 0:1.2.10-2.fc8 apr-util-devel.i386 0:1.2.10-2.fc8 cyrus-sasl-devel.i386 0:2.1.22-8.fc8 db4-cxx.i386 0:4.6.21-2.fc8 db4-devel.i386 0:4.6.21-2.fc8 expat-devel.i386 0:2.0.1-2 httpd.i386 0:2.2.9-1.fc8 httpd-tools.i386 0:2.2.9-1.fc8 libstdc++-devel.i386 0:4.1.2-33 neon.i386 0:0.27.2-2 openldap-devel.i386 0:2.3.39-4.fc8 perl-URI.noarch 0:1.35-3.1

Updated: db4.i386 0:4.6.21-2.fc8 openldap.i386 0:2.3.39-4.fc8

Complete!

2.4 Installing REE

[root@domU-12-31-39-00-68-93 ~]# mkdir software

[root@domU-12-31-39-00-68-93 ~]# cd software/

[root@domU-12-31-39-00-68-93 software]# wget http://rubyforge.org/frs/download.php/41040/ruby-enterprise-1.8.6-20080810.tar.gz

05:28:36 (3.66 MB/s) - `ruby-enterprise-1.8.6-20080810.tar.gz' saved [6431918/6431918]

[root@domU-12-31-39-00-68-93 software]# tar xzf ruby-enterprise-1.8.6-20080810.tar.gz

[root@domU-12-31-39-00-68-93 software]# ./ruby-enterprise-1.8.6-20080810/installer

[/opt/ruby-enterprise-1.8.6-20080810] :

## kc - Choose a path - I chose the default as mentioned above. Type 'yes' to all questions.

## kc -    You will get a confirmation after REE has been successfully installed.

2.4.1 Uploading the keys to EC2 instance - this will enable you to run AWS commands right from the instance.

[root@domU-12-31-39-00-68-93 ~]# mkdir .ec2

# Now go back to a terminal on your laptop/desktop:

guest@kc-laptop:~$ scp -i ~/AWS-do-not-delete/id_rsa-gsg-keypair /home/guest/AWS-do-not-delete/EC2-keys/pk-IFMHVYZ47LTIHHLIHU4A5X6SVULFXV3D.pem root@tripladder.com:

pk-IFMHVYZ47LTIHHLIHU4A5X6SVULFXV3D.pem       100% 922     0.9KB/s   00:00

guest@kc-laptop:~$ scp -i ~/AWS-do-not-delete/id_rsa-gsg-keypair /home/guest/AWS-do-not-delete/EC2-keys/cert-IFMHVYZ47LTIHHLIHU4A5X6SVULFXV3D.pem root@tripladder.com:

cert-IFMHVYZ47LTIHHLIHU4A5X6SVULFXV3D.pem     100% 916     0.9KB/s   00:00

[root@domU-12-31-39-00-68-93 ~]# mv cert-IFMHVYZ47LTIHHLIHU4A5X6SVULFXV3D.pem .ec2/

[root@domU-12-31-39-00-68-93 ~]# mv pk-IFMHVYZ47LTIHHLIHU4A5X6SVULFXV3D.pem .ec2/

2.4.2 Install latest and full version of AWS tools on the instance

We need Java for the AWS APIs:
[root@domU-12-31-39-00-68-93 software]# wget <Java Dk - jdk-6u10-linux-i586.bin>
[root@domU-12-31-39-00-68-93 software]# bash jdk-6u10-linux-i586.bin

# kc - Java has been installed to /root/software/jdk1.6.0_10/
# - We need to put it into .bash_profile

[root@domU-12-31-39-00-68-93 ~]# cd software/

[root@domU-12-31-39-00-68-93 software]# wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip

[root@domU-12-31-39-00-68-93 software]# unzip -q ec2-api-tools.zip

2.4.3 Set some environment variables for convenience

# kc - Note: Because we installed REE and not the native (yum install ruby) we need to make a small modification for ruby-lib-paths

# We need to put in some stuff in the bash_profile for some things - these are indicated in the comments below:

[root@domU-12-31-39-00-68-93 ~]# vi .bash_profile

# kc start - 09Nov08 - for enterprise ruby

PATH=/opt/ruby-enterprise-1.8.6-20080810/bin:$PATH

#http://developer.amazonwebservices.com/connect/thread.jspa?threadID=12491

# Because we have REE instead of 'yum install ruby'

export RUBYLIB=/usr/lib/site_ruby

# kc end - 09Nov08 - for enterprise ruby

# kc start - 09Nov08 - for EC2 tools

export EC2_HOME=/root/software/ec2-api-tools-1.3-26369

export EC2_PRIVATE_KEY=~/.ec2/pk-5YPLCLZG2ZBEKNWCTWESOHEOMACLCZTN.pem

export EC2_CERT=~/.ec2/cert-5YPLCLZG2ZBEKNWCTWESOHEOMACLCZTN.pem

PATH=$PATH:$EC2_HOME/bin
export JAVA_HOME=/root/software/jdk1.6.0_10/jre

# kc start - 09Nov08 - for EC2 tools

[root@domU-12-31-39-00-68-93 ~]# source .bash_profile

[root@domU-12-31-39-00-68-93 software]# source ~/.bash_profile

2.4.4 Installing Passenger and configuring it for the previously installed http server

[root@domU-12-31-39-00-68-93 ~]# gem install --no-ri --no-rdoc passenger

Building native extensions. This could take a while...

Successfully installed passenger-2.0.3

1 gem installed

[root@domU-12-31-39-00-68-93 ~]# passenger-install-apache2-module

# kc - accept default values and let the installation finish - It will take some time so be patient.

2.4.5 Installing some ruby gems and related software

# - strictly speaking this is not necessary but some gems check for availablity of rails-2.0.2 before installing so we might as well get it installed. Especially ferret checks for rails 2.0.2 before it starts.

[root@domU-12-31-39-00-68-93 tripladder]# gem install --no-ri --no-rdoc rails -v=2.0.2

Bulk updating Gem source index for: http://gems.rubyforge.org

Successfully installed rails-2.0.2

Successfully installed rake-0.8.2

Successfully installed activesupport-2.0.2

Successfully installed activerecord-2.0.2

Successfully installed actionpack-2.0.2

Successfully installed actionmailer-2.0.2

Successfully installed activeresource-2.0.2

All below gems are required for our app - You may install based on your application:

[root@domU-12-31-39-00-68-93 ~]# gem install --no-rdoc --no-ri actionwebservice -v=1.2.6

Successfully installed activesupport-1.4.4

Successfully installed actionpack-1.13.6

Successfully installed activerecord-1.15.6

Successfully installed actionwebservice-1.2.6

4 gems installed

[root@domU-12-31-39-00-68-93 ~]# gem install --no-rdoc --no-ri ferret -v=0.11.4

Building native extensions. This could take a while...

Successfully installed ferret-0.11.4

1 gem installed

[root@domU-12-31-39-00-68-93 tripladder]# gem install --no-rdoc --no-ri acts_as_ferret -v=0.4.3

Successfully installed acts_as_ferret-0.4.3

1 gem installed

2.5.6 Installing Imagemagick and rmagick - This step gave me a lot of trouble - do these with a lot of care

# See this tutorial: http://rmagick.rubyforge.org/install-faq.html

[root@domU-12-31-39-00-68-93 software]# yum install ghostscript-devel ghostscript ghostscript-font

Installed: ghostscript-devel.i386 0:8.63-1.fc8

Dependency Installed: ghostscript.i386 0:8.63-1.fc8 ghostscript-fonts.noarch 0:5.50-18.fc8 jasper-libs.i386 0:1.900.1-7.fc8 libXfont.i386 0:1.3.1-2.fc8 libfontenc.i386 0:1.0.4-4.fc8 urw-fonts.noarch 0:2.4-3.fc8 xorg-x11-font-utils.i386 1:7.2-2.fc8

Complete!

[root@domU-12-31-39-00-68-93 ~]# yum install perl perl-devel *ghostscript* *freetype* *jpeg* *png* *wmf* *tiff* *ghostscript* *lcms* *libexif* * libxml* *zlib* *bzip*

# - There will be lot of activity - just relax and accept all options.

# - Now get the exact version of Imagemagick and copy it to your 'software' directory.

[root@domU-12-31-39-00-68-93 software]# scp root@<myserver.com>:software/ImageMagick-6.4.1-0.tar.gz ./

root@rknowsys.com's password:

ImageMagick-6.4.1-0.tar.gz                    100%   11MB   1.2MB/s   00:09

[root@domU-12-31-39-00-68-93 software]# tar xzf ImageMagick-6.4.1-0.tar.gz

[root@domU-12-31-39-00-68-93 software]# cd ImageMagick-6.4.1

[root@domU-12-31-39-00-68-93 ImageMagick-6.4.1]# ./configure

# kc - *Lots* of activity.....

[root@domU-12-31-39-00-68-93 ImageMagick-6.4.1]# make install

# kc - *Lots* of activity.....

[root@domU-12-31-39-00-68-93 ImageMagick-6.4.1]# ln -s /usr/local/lib/* /usr/lib/

ln: creating symbolic link `/usr/lib/pkgconfig': File exists

[root@domU-12-31-39-00-68-93 ImageMagick-6.4.1]# mv /usr/lib/pkgconfig/ /root/software/original_pkgcongi_09Nov08

[root@domU-12-31-39-00-68-93 ImageMagick-6.4.1]# ln -s /usr/local/lib/pkgconfig/ /usr/lib/

[root@domU-12-31-39-00-68-93 ImageMagick-6.4.1]# gem install --no-rdoc --no-ri rmagick -v=2.3.0

Building native extensions. This could take a while...

Successfully installed rmagick-2.3.0

1 gem installed

3. Creating and attaching external EBS volume and making it usable

3.1 Creating a volume

guest@kc-laptop:~$ ec2-create-volume -s 20 -z us-east-1a

VOLUME vol-<my-volume>    20              us-east-1a      creating        2008-09-22T10:29:36+0000

guest@kc-laptop:~$ ec2-describe-volumes vol-<my-volume>

VOLUME vol-<my-volume>    20              us-east-1a      available       2008-09-22T10:29:36+0000

3.2 Attaching the volume to our instance

guest@kc-laptop:~$ ec2-attach-volume vol-<my-volume> -i <my-instance> -d /dev/sdh

ATTACHMENT      vol-<my-volume>    <my-instance>      /dev/sdh        attaching       2008-09-22T10:32:38+0000

guest@kc-laptop:~$ ec2-describe-volumes

VOLUME vol-8105e0e8    20              us-east-1a      in-use 2008-09-22T10:29:36+0000

ATTACHMENT      vol-<my-volume>    <my-instance>      /dev/sdh        attached        2008-09-22T10:32:38+0000

3.3Making it usable

[root@domU-12-31-39-00-65-E3 ~]# yes | mkfs -t ext3 /dev/sdh

mke2fs 1.40.4 (31-Dec-2007)

/dev/sdh is entire device, not just one partition!

Proceed anyway? (y,n) Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

2621440 inodes, 5242880 blocks

262144 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=0

160 block groups

32768 blocks per group, 32768 fragments per group

16384 inodes per group

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        4096000

Writing inode tables: done

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 26 mounts or

180 days, whichever comes first. Use tune2fs -c or -i to override.

[root@domU-12-31-39-00-65-E3 ~]# mkdir -p /mnt/data-store

[root@domU-12-31-39-00-65-E3 ~]# mount /dev/sdh /mnt/data-store/

4. Ensuring that mysql uses EBS

[root@domU-12-31-39-00-68-93 tripladder]# rmdir /var/lib/mysql/

[root@domU-12-31-39-00-68-93 tripladder]# mkdir -p /mnt/data-store/new_mysql_store/

[root@domU-12-31-39-00-68-93 tripladder]# ln -s /mnt/data-store/new_mysql_store/ /var/lib/mysql/

[root@domU-12-31-39-02-68-78 tripladder]# chown -R mysql:mysql /mnt/data-store/new_mysql_store/

[root@domU-12-31-39-00-65-E3 ~]# service mysqld start

Initializing MySQL database: Installing MySQL system tables...

OK

...

[root@domU-12-31-39-00-68-93 tripladder]# ls /mnt/data-store/new_mysql_store/

ibdata1 ib_logfile0 ib_logfile1 mysql mysql.sock .........

4.1 secure your mysql installation

[root@domU-12-31-39-00-65-E3 ~]# mysql_secure_installation

4.2 Create your DB

# - Now create your DB, Db-user and access privileges to the db-user.

# - You will have to do something like this:
                [root@domU-12-31-39-00-68-93 tripladder]# mysql -u root -p
                mysql> create database <database name>;
               Query OK, 1 row affected (0.01 sec)

                mysql> grant all on <database name>.* to '<database-user>'@'localhost' identified by '<database-passwd>';
               Query OK, 0 rows affected (0.00 sec)

4.3 Creating a schema or restoring from backup. (Optional) You can even do this later.

# after creating the database, you will have to create the schema and seed data -
                You can do this: (Take care to edit the config/database.yml file as shown in the next section before you run rake)
                [root@domU-12-31-39-00-68-93 tripladder]# rake db:migrate RAILS_ENV=production

                                    OR

               [root@domU-12-31-39-00-68-93 tripladder]# mysql -u <database-user> -p <database name> < <Path to backup scripts>/backup.sql


4.4 Stop the mysqld

[root@domU-12-31-39-00-68-93 tripladder]# service mysqld stop

5. Installing the application

[root@domU-12-31-39-00-68-93 ~]# cd /var/www/html/

[root@domU-12-31-39-00-68-93 html]# svn co https://<servername>/svn/tripladder/

5.1 Configuring the database access credentials

[root@domU-12-31-39-00-68-93 html]# cd tripladder/

[root@domU-12-31-39-00-68-93 tripladder]# vi config/database.yml

production:

adapter: mysql

database: <database name>

username: <database user>

password: <passwd for database user>

host: localhost

5.2 Configuring some stuff in environment.rb

# kc - Note - these configs are spread across the whole file. I am listing out the sections that need changing.

# - kc Note - Tripladder uses google apps for Intranet and email.

[root@domU-12-31-39-00-68-93 tripladder]# vi config/environment.rb

ENV['RAILS_ENV'] ||= 'production'

RAILS_GEM_VERSION = '2.0.2' unless defined? RAILS_GEM_VERSION

# kc start - use google to send our email - 23Sep08

ActionMailer::Base.smtp_settings = {

    :tls => true,

    :address => "smtp.gmail.com",

    :port => "587",

    :domain => "tripladder.com",

    :authentication => :plain,

    :user_name => "<mail-user>@tripladder.com",

    :password => "<mail-user-passwd>"

}

# kc end - use google to send our email - 23Sep08

5.3 Configuring stuff for ferret

# kc - Ferret will give some trouble unless all related directories have the right ownerships/permissions. You may need to iterate here a bit.

5.3.1 Ferret configurations

[root@domU-12-31-39-00-68-93 tripladder]# vi config/ferret_server.yml

production:

host: localhost

port: 9010

pid_file: log/ferret.pid

log_file: log/ferret_server.log

log_level: warn

# kc - Note - Also comment out rest of the file.

[root@domU-12-31-39-00-68-93 tripladder]# vi config/environments/ferret_environment.rb

INDEX = Index::Index.new(:path => '/var/www/html/tripladder/index')

# - kc - Now we will commit all these files to subversion so that we don't have to keep worrying about this during svn updates (Developers will have to take care to change relevant sections.

[root@domU-12-31-39-00-68-93 tripladder]# svn commit config -m "From production server - kc"

Sending        config/database.yml

Sending        config/environment.rb

Sending        config/environments/ferret_environment.rb

Sending        config/ferret_server.yml

Transmitting file data ....

Committed revision 997.

5.3.2 Configuring ferret init scripts

# kc - Create this file and paste the contents (make changes as per your setup)

[root@domU-12-31-39-00-68-93 tripladder]# vi script/ferret_init_production

# kc - end of content for the file - Do not copy this line

#!/bin/bash

#

# This script starts and stops the ferret DRb server

# chkconfig: 2345 89 36

# description: Ferret search engine for ruby apps.

#

# save the current directory

CURDIR=`pwd`

PATH=/usr/local/bin:/opt/ruby-enterprise-1.8.6-20080810/bin:$PATH

RORPATH="/var/www/html/tripladder"

case "$1" in

start)

     cd $RORPATH

     echo "Starting ferret DRb server."

     FERRET_USE_LOCAL_INDEX=1 \

                script/ferret_server -e production start

                chmod a+rwx /var/www/html/tripladder/log/ferret.pid # we need this for to allow some manipulations

     ;;

stop)

     cd $RORPATH

     echo "Stopping ferret DRb server."

     FERRET_USE_LOCAL_INDEX=1 \

                script/ferret_server -e production stop

     ;;

*)

     echo $"Usage: $0 {start, stop}"

     exit 1

     ;;

esac

cd $CURDIR

# kc - end of content for the file- Do not copy this line

[root@domU-12-31-39-00-68-93 tripladder]# chmod a+x script/ferret_init_production

[root@domU-12-31-39-00-68-93 tripladder]# svn add script/ferret_init_production

A         script/ferret_init_production

[root@domU-12-31-39-00-68-93 tripladder]# svn commit script -m "From production server - kc"

Adding         script/ferret_init_production

Transmitting file data .

Committed revision 998.

[root@domU-12-31-39-00-68-93 tripladder]# ln -s /var/www/html/tripladder/script/ferret_init_production /etc/init.d/

[root@domU-12-31-39-00-68-93 tripladder]# chmod -R a+rwx log/

[root@domU-12-31-39-00-68-93 tripladder]# chmod -R a+rwx tmp/

# -Start - section is optional - you will need to check if your dir structure already has 'index' directory required by ferret (ferret_environment.rb) .
[root@domU-12-31-39-00-68-93 tripladder]# mkdir index
[root@domU-12-31-39-00-68-93 tripladder]# chmod -R a+rwx index/

# - End - section is optional - you will need to check if your dir structure already has 'index' directory.

[root@domU-12-31-39-00-68-93 software]# service ferret_init_production start
Starting ferret DRb server.
Install the ruby-openid gem to enable OpenID support
starting ferret server...
[root@domU-12-31-39-00-68-93 software]# service ferret_init_production stop
Stopping ferret DRb server.
Install the ruby-openid gem to enable OpenID support
stopping ferret server...
process 1723 has stopped

# kc - If you have problems here you will need to check for <ROR-Home>/log/ferret.log(or ferret.out) files for errors and fix them accordingly.
# - Usually we will have problems with ownership or permission for ferret log files - 'log' directory and 'index' directory in ROR-Home.

6. Configuring Apache and getting it to serve the application

6.1 Getting Apache introduced to REE and Passenger

[root@domU-12-31-39-00-68-93 tripladder]# vi /etc/httpd/conf/httpd.conf
# kc start For Phusion and enterprise Ruby - 09Nov08
LoadModule passenger_module /opt/ruby-enterprise-1.8.6-20080810/lib/ruby/gems/1.8/gems/passenger-2.0.3/ext/apache2/mod_passenger.so

PassengerRoot /opt/ruby-enterprise-1.8.6-20080810/lib/ruby/gems/1.8/gems/passenger-2.0.3

PassengerRuby /opt/ruby-enterprise-1.8.6-20080810/bin/ruby
# kc end For Phusion and enterprise Ruby - 09Nov08

6.2 Getting Apache to recognise our application as well as some REE/Passenger config parameters

# kc start - Deploying tripladder app
RailsEnv production
PassengerMaxPoolSize 26
PassengerLogLevel 0
PassengerMaxInstancesPerApp 0
PassengerPoolIdleTime 86400
RailsSpawnMethod smart
<VirtualHost *:80>
    ServerName www.tripladder.com
    DocumentRoot /var/www/html/tripladder/public
    # kc - to redirect tripladder.com to www.tripladder.com
    RewriteEngine on
    RewriteCond %{HTTP_HOST} ^tripladder\.com
    RewriteRule ^(.*)$ http://www.tripladder.com/$1 [R=permanent,L]
    # kc - to redirect tripladder.com to www.tripladder.com
</VirtualHost>
# kc end - Deploying tripladder app

6.3 Deploying the app and checking that its working

[root@domU-12-31-39-00-68-93 tripladder]# service mysqld start
Starting MySQL:                                            [ OK ]
[root@domU-12-31-39-00-68-93 tripladder]# service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [ OK ]
[root@domU-12-31-39-00-68-93 software]# service ferret_init_production start
Starting ferret DRb server.
Install the ruby-openid gem to enable OpenID support
starting ferret server...

# - Check that the application is accessible from the URL or the IP address.

7. Setting up periodic backups, cleaning up and miscellaneous activities

7.1 Ensure that you have full version of AWS tools (Section 2.4.2)

7.2 Set a cron task to take periodic backups of mysql

# - Create a directory on EBS for your mysql dumps.
[root@domU-12-31-39-00-68-93 ~]# mkdir -p /mnt/data-store/mysql_backups/
[root@domU-12-31-39-00-68-93 ~]# chmod -R a+rw /mnt/data-store/mysql_backups/

# Check that mysqldump works manually before you put it in cron.

[root@domU-12-31-39-00-68-93 ~]# crontab -e
2       0       *       *       *        mysqldump --user=root --password=<root passwd> --single-transaction <database name> > /mnt/data-store/mysql_backups/database_name.sql

7.3 Use logrotate to clean up older mysqld dumps as well as RoR log files.

# kc - Note - I have elected to have full backups everyday and rotate them every 30 days. i.e. I can go back-data upto 30 days if I want to restore mysql.

[root@domU-12-31-39-00-68-93 ~]# vi /etc/logrotate.conf
# kc - 09Nov08 - for rotating rails logs
/var/www/html/tripladder/log/*.log {
size=10M
missingok
rotate 4
compress
delaycompress
notifempty
copytruncate
}
# kc - 09Nov08 - for rotating rails logs

# kc - 09Nov08 - for rotating mysqldumps
/mnt/data-store/mysql_backups/tripladder_prod.sql {
daily
rotate 30
compress
delaycompress
notifempty
notifempty
copytruncate
}
# kc - 09Nov08 - for rotating mysql dumps

7.4 Use EBS snapshot facility to automate backups.

[root@domU-12-31-39-02-68-78 ~]# vi software/backup_script.sh
#!/bin/bash

export EC2_PRIVATE_KEY=$(echo /root/.ec2/pk-xxxxxxxxxx.pem)
export EC2_CERT=$(echo /root/.ec2/cert-xxxxxxxxxxx.pem)
export EC2_HOME=/root/software/ec2-api-tools-1.3-26369
export PATH=$PATH:$EC2_HOME/bin
export JAVA_HOME=/root/software/jdk1.6.0_10/jre
/root/software/ec2-api-tools-1.3-26369/bin/ec2-create-snapshot vol-<myvolume>

# - test that this is working!!!
[root@domU-12-31-39-00-68-93 ~]# chmod a+x software/backup_script.sh
[root@domU-12-31-39-00-68-93 ~]# ./software/backup_script.sh
SNAPSHOT        snap-d55cbdbc   vol-8105e0e8    pending 2008-11-11T10:56:10+0000

# - kc - now we put this in the crontab.
[root@domU-12-31-39-00-68-93 ~]# crontab -e
3       0       *       *       *       /root/software/backup_script.sh
crontab: installing new crontab

# kc - start - Important -
#   We save all user uploaded data to 2 location - 1st to a directory in the RoR dir structure and 2nd to a directory on the EBS.
#   The rails app access this data from EC2 local storage.
#   If ever the instance dies, we copy the data from EBS to associated directory on directory structure. Now why don't we link the RoR pubic directory to EBS is because EBS has some
#           price attached to disk access. I could not calculate this with any conviction and hence decided not to use the disk for reads. The EBS is only used for writes for user data.
#   Having user data like this reduces your disk access costs. Please look at this tutorial to see how we achieved this in rails - http://docs.google.com/Doc?id=dckr2rnx_5ghz54qcj
#   Since Mysql can't have 2 data dirs, we host the mysql data directory on EBS.
# kc - end - Important -

7.5 Some apache optimisations

[root@domU-12-31-39-00-68-93 tripladder]# vi public/.htaccess
# kc start - enable browser caching - 24Sep08
<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 1 seconds"
ExpiresByType text/html "access plus 1 seconds"
ExpiresByType image/gif "access plus 120 minutes"
ExpiresByType image/jpeg "access plus 120 minutes"
ExpiresByType image/png "access plus 120 minutes"
ExpiresByType text/css "access plus 60 minutes"
ExpiresByType text/javascript "access plus 60 minutes"
ExpiresByType application/x-javascript "access plus 60 minutes"
ExpiresByType text/xml "access plus 60 minutes"
</IfModule>

[root@domU-12-31-39-00-68-93 tripladder]# svn commit public/.htaccess -m "From production server"
Sending        public/.htaccess
Transmitting file data .
Committed revision 1000.

# - Get Apache to read this by restarting the application:
[root@domU-12-31-39-00-68-93 tripladder]# touch tmp/restart.txt
          -- OR --
[root@domU-12-31-39-00-68-93 tripladder]# service httpd restart

7.6 We need to be able to use rake to clear caches else this will become a major headache when we update to new versions.

[root@domU-12-31-39-00-68-93 tripladder]# service httpd stop
Stopping httpd:                                            [ OK ]
[root@domU-12-31-39-00-68-93 tripladder]# rm -f public/cache/*cache
[root@domU-12-31-39-00-68-93 tripladder]# ls public/cache/
[root@domU-12-31-39-00-68-93 tripladder]#
[root@domU-12-31-39-00-68-93 tripladder]# cd public/cache/
[root@domU-12-31-39-00-68-93 cache]# cd ..
[root@domU-12-31-39-00-68-93 public]# svn del cache
D         cache
[root@domU-12-31-39-00-68-93 public]# svn commit -m "For symlinking tmp/cache to public/cache"
Deleting       public/cache

Committed revision 1041.
[root@domU-12-31-39-00-68-93 public]# ln -s /var/www/html/tripladder/tmp/cache/ /var/www/html/tripladder/public/
[root@domU-12-31-39-00-68-93 public]# chmod a+rwx /var/www/html/tripladder/tmp/cache

[root@domU-12-31-39-00-68-93 public]# service httpd start
Starting httpd:                                            [ OK ]
# - KC now to test the rake task to clear cache
[root@domU-12-31-39-00-68-93 tripladder]# ls public/cache/
application_layout_header.cache homepage_meta.cache
homepage.cache                   send_review_to_freind.cache
homepage_header.cache
[root@domU-12-31-39-00-68-93 tripladder]# ls tmp/cache/
application_layout_header.cache homepage_meta.cache
homepage.cache                   send_review_to_freind.cache
homepage_header.cache
[root@domU-12-31-39-00-68-93 tripladder]# rake tmp:clear
(in /var/www/html/tripladder)
[root@domU-12-31-39-00-68-93 tripladder]# ls tmp/cache/
[root@domU-12-31-39-00-68-93 tripladder]# ls public/cache/

# kc - Confirmed that rake works.

8. Bundling the image

8.1 First stop all running services

Now stop all services:
[root@domU-12-31-39-00-68-93 tripladder]# service httpd stop
Stopping httpd:                                            [ OK ]
[root@domU-12-31-39-00-68-93 software]# service ferret_init_production stop
Stopping ferret DRb server.
Install the ruby-openid gem to enable OpenID support
stopping ferret server...
process 1741 has stopped
[root@domU-12-31-39-00-68-93 software]# service mysqld stop
Stopping MySQL:                                            [ OK ]

8.2 Ensure that services do not start when system reboots - This is important as we have to attach EBS before we start the services.

[root@domU-12-31-39-00-68-93 software]# chkconfig --list httpd
httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@domU-12-31-39-00-68-93 software]# chkconfig --list mysqld
mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@domU-12-31-39-00-68-93 software]# chkconfig --list ferret_init_production
service ferret_init_production supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add ferret_init_production')
[root@domU-12-31-39-00-68-93 software]# chkconfig --add ferret_init_production
[root@domU-12-31-39-00-68-93 software]# chkconfig --list ferret_init_production
ferret_init_production 0:off   1:off   2:on    3:on    4:on    5:on    6:off
[root@domU-12-31-39-00-68-93 software]# chkconfig ferret_init_production off
[root@domU-12-31-39-00-68-93 software]# chkconfig --list ferret_init_production
ferret_init_production 0:off   1:off   2:off   3:off   4:off   5:off   6:off

8.3 Bundle the volume and save it to S3.

[root@domU-12-31-39-00-68-93 ~]# ec2-bundle-vol -d /mnt/ -k .ec2/pk-<xxxxxxxxx>.pem -c .ec2/cert-<xxxxxxxx>.pem -u <12 digit AWS user id without hypehns>
Please specify a value for arch [i386]:
....# Lots of activity

[root@domU-12-31-39-00-68-93 ~]# ec2-upload-bundle -b tripladder-09Nov08 -m /mnt/image.manifest.xml -a <accesskey> -s <secret>

guest@kc-laptop:~$ ec2-register tripladder-09Nov08/image.manifest.xml
IMAGE   ami-3c15f155

8.4 Cleaning up

# kc - The AWS bundle tools seem to have memory leak - The used memory never comes back to normal and seems to be using up ~ 1 GB of RAM
[root@domU-12-31-39-00-68-93 ~]# umount /mnt/data-store/
[root@domU-12-31-39-00-68-93 ~]# reboot

9. Bringing up the application after reboot.

[root@domU-12-31-39-00-68-93 ~]# mount /dev/sdh /mnt/data-store/
[root@domU-12-31-39-00-68-93 ~]# service mysqld start
Starting MySQL:                                            [ OK ]
[root@domU-12-31-39-00-68-93 ~]# service ferret_init_production start
Starting ferret DRb server.
Install the ruby-openid gem to enable OpenID support
starting ferret server...
[root@domU-12-31-39-00-68-93 ~]# service httpd start
Starting httpd:                                            [ OK ]

# kc - Site should be running perfectly.

10. Bringing up the application from AMI
If ever your instance is killed, just do the following:
1. Launch your bundled instance wih the AMI id you got when you registered your bundle,
2. Attach the EBS volume,
3. Associate the IP address
4. Start services - mysql, ferret and httpd.
5. Presto!!!

# - Important - NEVER put the mount option for EBS in fstab. When the system is brought up from AMI, the EBS volume is still not attached and hence will not be mounted. This will not allow the system to boot. You AMI will essentially be useless.
WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG !!!!!!!!!!!!!!!!
[root@domU-12-31-39-00-65-E3 ~]# vi /etc/fstab
/dev/sdh                /mnt/data-store         ext3    defaults,noatime 0 2
# - Amazon is working on attaching a volume to EC2 instance during launch - when this feature is available, we can use fstab....
WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG, WRONG !!!!!!!!!!!!!!!

Wednesday, March 3, 2010

truncate method incompatible with Ruby 1.8.7

With rails > 2.2 and ruby 1.8.7 we can find this error using rails truncate helper


undefined method `length' for  Enumerable::Enumerator:0x7f44da230548>



To solve this problem paste the below code in environment.rb (EOF)



module ActionView
module Helpers
module TextHelper
 def truncate(text, length = 30, truncate_string = "...")
   if text.nil? then return end
   l = length - truncate_string.chars.to_a.size
   (text.chars.to_a.size > length ? text.chars.to_a[0...l].join + truncate_string : text).to_s
 end
end
end
end

Tuesday, December 8, 2009

Templates with HAML

HAML(XHTML Abstraction Markup Language) is an alternative to RHTML for writing templates for views in a Rails application. Its advantages over RHTML include brevity and clarity.

Features:

Whitespace active
Well-formatted markup
DRY
Follows CSS conventions
Integrates Ruby code
Implements Rails templates with the .haml extension
it is elegant

how to install:
We can use gem or plugin

1)install gem :
sudo gem install haml
2)install plugin:
ruby script/plugin install git clone git://github.com/nex3/haml.git

Usage:
save files with extension .haml

Let's look at the HAML: index.html.haml

%h1 Hello to you all
%p This is a test page

A '%' symbol creates an HTML tag. 2 spaces are used to indent the code into sub-elements and we need not to specify end tags.so that reducing so many lines code.

I tried to post examples with brief introduction, but the blog site seems to be not supporting html tags.
There is a lot more to learn check these References:

http://haml-lang.com/docs/yardoc/file.HAML_REFERENCE.html#plugin
http://haml-lang.com/about.html
http://wiki.rubyonrails.org/howtos/templates/haml

Thursday, January 29, 2009

Migrating Stored Procedures in RoR

This similar article may be found on the internet, but i guess this article will be helpful for newbie's.

Migrations are a convenient way to alter your database in a structured and organised manner.

Here we are going to see how to create stored procedures with rake db:migrate

First create a new migration file in your db/migrate folder using

ruby script/generate migration stored_procedures. This will create the file db/migrate/001_stored_procedures.

Edit the code to tell it what to do.

The method self.up is used when migrating to a new version, self.down is used to roll back any changes if needed. The class name needs to be the same as the migration name (i.e. db/migrate/001_stored_procedures needs a class name of @StoredProcedures@).

Let us migrate a stored procedure called items with a basic sql code :

The migration file now looks like this :

class StoredProcedures < ActiveRecord::Migration
 def self.up
   execute <<-__EOI
     CREATE DEFINER=`root`@`localhost` PROCEDURE `items`(IN l_item INT,IN userid INT,OUT l_itemid INT,OUT l_item_name VARCHAR)
     BEGIN SELECT itemid,item_name,held_by,id INTO l_itemid,userid,id FROM users_items WHERE itemid=l_item AND held_by=userid;
     END
   __EOI
 end
 def self.down
   execute "DROP PROCEDURE IF EXISTS `items`"
 end
end

And now the cool part is just run rake db:migrate, Rails will create all the migration files in your database and also creates Stored procedures.

Monday, November 24, 2008

Optimising RoR application for Amazon EBS

Amazon has a great feature called EBS which enables to have data persistence in the event of an instance failure.
We have configured mysql to use the EBS for datafiles. Details for this can be found here: http://docs.google.com/Doc?id=dcn2ckbh_21gznbbjhr

Though a great feature, EBS has a couple of operational limitations.
1. It has a cryptic billing structure which bills based on (capacity + usage). Now most of us can't really predict the usage of disk and risk overshooting this.
2. EBS will be slower that locally mounted storage.

To overcome these issues, in our RoR application, we decided make some changes.

We decided to upload all user data to TWO locations - 1st location is the usual "RoR-app-home/public/" directory. This directory is in the local storage of the instance.
The 2nd location is the EBS (/dev/sdh) mounted on /mnt/data-store of the instance. Within data-store, we created a few directories for storing various types of user data.
1. During upload, the data is copied to both locations. i.e. We write to both local and EBS.
2. During read - we read it from local directory of EC2. This local reads ensures that EBS is not hit with multiple read requests and our EBS costs are low. There is alo the speed benefit of reading from local storage as opposed to reading from EBS (Amazon AWs developers can correct me on the speed issue.)

Here is the sampe code where we are uploading a video and a thumbnail associated with the video: Keep and eye out for "video.rewind".

Add in app/model/video.rb (for our application - you will have adapt for you app.)

VIDEO_UPLOAD_PATH = "public/video_player/videos/"
THUMBNAIL_UPLOAD_PATH = "public/video_player/thumbnail/"

#Manage the path depending on OS
#Hard disk usage Optimisation for AWS.
#Replicate videos, images, into AWS local hard disk.

VIDEO_UPLOAD_PATH_FOR_AWS = "/mnt/data-store/app-data/videos/"
THUMBNAIL_UPLOAD_PATH_FOR_AWS = "/mnt/data-store/app-data/thumbnails/"

# create and upload video , thumbail
def self.create_video
if valid_video?(video) && valid_thumbnail?(thumbnail)
video_filename = sanitize_attachment_name(video)
thumbnail_filename = sanitize_attachment_name(thumbnail)
@video = self.new do |video|
video.video_name = video_filename
video.thumbnail_name = thumbnail_filename
end
self.upload_video(video, video_filename, @video.id) && self.upload_thumbnail(thumbnail, thumbnail_filename, @video.id) if @video.save
end
end

def self.upload_video(video, video_filename, video_id)
video_path = VIDEO_UPLOAD_PATH + "#{ video_id}_" + video_filename
File.open(video_path, "wb") { |f| f.write(video.read) }
# code to manage video upload file to EBS
video.rewind
video_path_for_aws = VIDEO_UPLOAD_PATH_FOR_AWS + "#{ video_id}_" + video_filename
File.open( video_path_for_aws, "wb") { |f| f.write(video.read) }
end

def self.upload_thumbnail(thumbnail, thumbnail_filename, video_id)
thumbnail_path = THUMBNAIL_UPLOAD_PATH + "#{ video_id}_" + thumbnail_filename
File.open(thumbnail_path, "wb") { |f| f.write(thumbnail.read) }
# code to manage thumbnail image upload file to EBS
thumbnail.rewind
thumbnail_path_for_aws = THUMBNAIL_UPLOAD_PATH_FOR_AWS + "#{ video_id}_" + thumbnail_filename
File.open(thumbnail_path_for_aws, "wb") { |f| f.write(thumbnail.read) }
end

Friday, November 21, 2008

Tutorial on hosting RoR app on Amazon AWS with EC2, EBS, Ruby Enterprise Edition (REE) and Phusion Passenger (mod_rails)

Background:
We built a tourism portal in RoR for one of our clients. You have a look at it - - www.tripladder.com
After building it, we were requested to host and manage it for them. Initially we went with knownhost which is OK but a production RoR application needs more RAM than what we get on most VPS plans - especially if we have image processing. We did consider AWS but at that time it did not have EBS and the client did not initially expect enough traffic to justify a 'scalr' managed cluster. We were looking for a replacement to a dedicated server. Once EBS was launched, we immediately decided to move the site to AWS. The Cost-benefit analysis is compelling.

The following tutorial starts off after signing up with AWS and configuring your desktop/laptop to be able to connect to AWS and launch instances i.e. we assume that you have completed the 'Getting Started' section of AWS.

We have started with the stock Fedora image and modified it to our requirements. We could have used CentOS but Fedora-8 appeared at the top of the list and we went ahead with it.

The application hosting has the following steps.

Launching an instance.
Installing RoR, gems, plugins...We used rmagick, hence we had to install Imagemagick too.
Installing REE and Phusion (mod_rails)
Installing mysql.
Intalling the application (checkout from subversion).
Creating and attaching a EBS volume. Mysql with data on EBS
Modifying the RoR app to save user upload files to EBS.(http://docs.google.com/Doc?id=dcn2ckbh_20hk4kc4d4)
Installing and configuring a production level ferret server
Configuring Apache to serve the application, caching optimisations for performance.
Configuring permanent public IP (covered) and DNS (we have the domain parked with go daddy but this is not covered in this article)
Configuring smtp (email) support for RoR application.
Once we have the perfect server setup, save it to S3.
Periodic automated backups - Using Amazon snapshots.

The full tutorial is available here:

http://docs.google.com/Doc?id=dcn2ckbh_21gznbbjhr

Tutorial on hosting a RoR (Ruby on Rails) application on Amazon AWS with EC2 and EBS.

We built a tourism portal in RoR for one of our clients. You can have a look at it - - www.tripladder.com
After building it, we were requested to host and manage it for them. Initially we went with knownhost which is OK but a production RoR application needs more RAM than what we get on most VPS plans - especially if we have image processing. We did consider AWS but at that time it did not have EBS and the client did not initially expect enough traffic to justify a 'scalr' managed cluster. We were looking for a replacement to a dedicated server. Once EBS was launched, we immediately decided to move the site to AWS. The Cost-benefit analysis is compelling.

The following tutorial starts off after signing up with AWS and configuring your desktop/laptop to be able to connect to AWS and launch instances i.e. we assume that you have completed the 'Getting Started' section of AWS.

We have started with the stock Fedora image and modified it to our requirements. We could have used CentOS but Fedora-8 appeared at the top of the list and we went ahead with it.

The application hosting has the following steps.

Launching an instance.
Installing RoR, gems, plugins...We used rmagick, hence we had to install Imagemagick too.
Installing mysql.
Intalling the application (checkout from subversion).
Creating and attaching a EBS volume. Mysql with data on EBS
Modifying the RoR app to save user upload files to EBS.
Installing and configuring a production level ferret server
Installing and configuring mongrel cluster.
Configuring Apache to proxy to mongrel cluster, caching optimisations for performance.
Configuring permanent public IP (covered) and DNS (we have the domain parked with go daddy but this is not covered in this article)
Configuring smtp (email) support for RoR application.
Once we have the perfect server setup, save it to S3.
Periodic automated backups -This is available here: http://docs.google.com/Doc?id=dcn2ckbh_21gznbbjhr

For the full tutorial - go to this link.

http://docs.google.com/Doc?id=dcn2ckbh_20hk4kc4d4