Monday, November 24, 2008

Optimising RoR application for Amazon EBS

Amazon has a great feature called EBS which enables to have data persistence in the event of an instance failure.
We have configured mysql to use the EBS for datafiles. Details for this can be found here:

Though a great feature, EBS has a couple of operational limitations.
1. It has a cryptic billing structure which bills based on (capacity + usage). Now most of us can't really predict the usage of disk and risk overshooting this.
2. EBS will be slower that locally mounted storage.

To overcome these issues, in our RoR application, we decided make some changes.

We decided to upload all user data to TWO locations - 1st location is the usual "RoR-app-home/public/" directory. This directory is in the local storage of the instance.
The 2nd location is the EBS (/dev/sdh) mounted on /mnt/data-store of the instance. Within data-store, we created a few directories for storing various types of user data.
1. During upload, the data is copied to both locations. i.e. We write to both local and EBS.
2. During read - we read it from local directory of EC2. This local reads ensures that EBS is not hit with multiple read requests and our EBS costs are low. There is alo the speed benefit of reading from local storage as opposed to reading from EBS (Amazon AWs developers can correct me on the speed issue.)

Here is the sampe code where we are uploading a video and a thumbnail associated with the video: Keep and eye out for "video.rewind".

Add in app/model/video.rb (for our application - you will have adapt for you app.)

VIDEO_UPLOAD_PATH = "public/video_player/videos/"
THUMBNAIL_UPLOAD_PATH = "public/video_player/thumbnail/"

#Manage the path depending on OS
#Hard disk usage Optimisation for AWS.
#Replicate videos, images, into AWS local hard disk.

VIDEO_UPLOAD_PATH_FOR_AWS = "/mnt/data-store/app-data/videos/"
THUMBNAIL_UPLOAD_PATH_FOR_AWS = "/mnt/data-store/app-data/thumbnails/"

# create and upload video , thumbail
def self.create_video
if valid_video?(video) && valid_thumbnail?(thumbnail)
video_filename = sanitize_attachment_name(video)
thumbnail_filename = sanitize_attachment_name(thumbnail)
@video = do |video|
video.video_name = video_filename
video.thumbnail_name = thumbnail_filename
self.upload_video(video, video_filename, && self.upload_thumbnail(thumbnail, thumbnail_filename, if

def self.upload_video(video, video_filename, video_id)
video_path = VIDEO_UPLOAD_PATH + "#{ video_id}_" + video_filename, "wb") { |f| f.write( }
# code to manage video upload file to EBS
video_path_for_aws = VIDEO_UPLOAD_PATH_FOR_AWS + "#{ video_id}_" + video_filename video_path_for_aws, "wb") { |f| f.write( }

def self.upload_thumbnail(thumbnail, thumbnail_filename, video_id)
thumbnail_path = THUMBNAIL_UPLOAD_PATH + "#{ video_id}_" + thumbnail_filename, "wb") { |f| f.write( }
# code to manage thumbnail image upload file to EBS
thumbnail_path_for_aws = THUMBNAIL_UPLOAD_PATH_FOR_AWS + "#{ video_id}_" + thumbnail_filename, "wb") { |f| f.write( }