Engine Yard Downtime

While they might not be as big as these top services having significant outages this week, they certainly are the leaders in the Ruby on Rails world.  Our faithful and expensive, experts at Engine Yard have hit trouble not once, but twice this week.  As I am writing this adiserver has been down for over and hour and a half. This coming just a few hours after we received a credit an hour of down time from earlier this week. 

I find it funny that just yesterday we posted about moving our small applications to EC2 and the fear of single point of failure there.  We praised EY's redundant environment only to have it fail on us today. Only one of our slices seems down, yet the whole application is unresponsive, whats up with the redundancy we pay for?  Hopefully there will be an explanation in the post-mortem. 

This is all to be expected though, as the wildest super rare occurrences seem to happen whenever we move a server.  A day before our move to Engine Yard we got hit with Rackspace's really bad 36 hours.  Maybe we should warn all network admins before we move another box...

Well lets hope we come back on line soon, because as of now my head is in the clouds.

Update: We are back up, total downtime - 2 hours, 5 minutes, 39 seconds

Moving small apps from Engine Yard to EC2

When we first started at engine yard we hosted all of our applications on it, it made sense at the time, as our goal was to offload as much work from our limited resources as possible, but now that our skill levels have increased and our apps have matured, it doesn't seem as needed.

For example, our company site www.sympact.net does not get very much traffic as we don't have much of a consumer sale model.  The site is a rails 2.1 app, with a basic cms, and the only gem dependency is haml.  What justification is there for hosting it on EY?  I'm sure an engine yard representative would probably ask us why we were hosting it there in the first place and my answer would be peace of mind.  Now that we want to scale our main application out at EY, we want to best utilize out hosting budget at EY. 

As you may know, engine yard is expensive with each slice costing us $379/month (in our configuration).  Resources are very scarce, with only 3-4 mongrels recommended to be run in the memory allocation given to each slice.  Why are we giving up 1-2 of those mongrels over two slices($379) to apps that don't need it? Well that question is what lead us to EC2, and more specifically (ec2onrails + RightScale). $72 per month, per app (EY charges $50 bucks for additional apps + the slice costs).

EC2 on rails is a gem and AMI that is great for setting up rails applications that don't need very much customization.  Right Scale allows us to easily configure and design our instances... for Free. This basic setup has brought us down to a single point of failure, but we have accepted that risk for now, and still think we can scale to a redundant environment at a lower cost then EY.  Also, we now have more mongrels than before (5 in the out of the box configuration).

Just to be clear, the intention of this entry is not to bash EY, Sympact still loves and uses EY for our core application.  In fact the reason we are moving the other applications to EC2 is to get more of our EY resources for the main application.  Just be cautious about your budget with Engine Yard because your going to want more! You are going to want more slices, more memory, more cpu, and it is going to cost you.  Keep the mission critical stuff there, get the rest out.