Cloud Computing, transaction costs, and avoiding the salesperson

I was recently reading about the Ronald Coase's "Nature of the Firm", which got me thinking about Cloud Computing, EC2 and Google.

"Nature of the Firm" was written by Coase, a Nobel Prize winner in Economics. Coase analyzed the reason people form into organizations as opposed to contracting out everything. Since markets are efficient, one would think that everything would be outsourced to get the most efficient price. However, one has to look at other costs besides the price of the good or service, so Coase came up with the notion of the transaction cost. This is the cost of using the market, and includes research and bargaining costs, contract costs, etc. Although his work was published in 1937, it applies more than ever today, with outsourcing becoming commonplace.

There is an ongoing trend in technology to lower the transaction cost, and allow entrepreneurs to use the market for more goods and services. Google's self service mentality has taken low transaction costs to its limits. With Adsense, Google has created a full self service way to buy and sell advertising, without ever having to negotiate or speak to another sales person. Customers can now sign up, configure their advertising system, and be functional in a transparent manner. By taking the salesperson out of the loop, they have lowered the perceived transaction costs; people like like avoid price negotiations, and enjoy having transparency in their pricing.

Amazon has taken Google's self service mentality a huge step further with their suite of cloud products. You can provision an entire data center worth of servers using EC2 without having to make a single phone call. You can summon an unlimited amount of storage using S3. Their newest offering, CloudFront, allows you to put your content on the far reaches of the internet and avoid having to deal with expensive fixed prices and negotiations with traditional CDN providers. Everything is pay as you go.

The promise of IaaS cloud computing is that if follow rigid automation, you can achieve zero transaction costs to scale your hardware. This is the notion of autoscaling, which is automatic and free of transaction costs. Due to the dynamic nature of cloud computing, companies are forced into a different mindset, one that allows them to enjoy provisioning and configuring a whole environment in a manner of minutes, where it previously has taken days or even weeks.

We can see that Google and Amazon have done well by allowing people to use the market in a self serviceable, transparent and predictable way. People like to avoid the unknown, and perceive a higher transaction cost with it versus a transparent process. Companies can have much more success, and even create entire new markets, if they follow Google's and Amazon's footsteps by lowering customer transaction costs, and creating a well defined and transparent specification for the service they are providing.

Handling file uploads with attachment_fu, Merb 1.0, and Passenger

I have been using Merb to handle file uploads using attachment_fu and the mini_magick gem. Now that Merb 1.0 is out, I decided to upgrade my application to 1.0. I also set it up alongside a running Rails application in Passenger. Since it wasn't so clear on how to do this, I decided to do a quick write up on it.

I was previously using a hacked up attachment_fu version that I had to modify to work with Merb 0.52 (similar to attachmerb_fu). However, since then, attachment_fu has been modified to support Merb as long as you set it up correctly. The missing piece of the puzzle is a gem called merb_has_rails_plugin written by Michael Ivey that allows you to use Rails plugins in Merb. :

Here are the steps I took:

1. Install Merb
2. Generate an empty Merb app (merb-gen app file_uploads)
3. Create a "plugins" directory under the root of the merb application, and put the attachment_fu plugin there
4. Install the merb_has_rails_plugin and setup the gem in your init.rb file (add the line to your config: Merb::Plugins.config[:merb_has_rails_plugins] ).
5. Set use_orm :activerecord in your init.rb
6. You may need to install the activerecord and merb_activerecord gems.
7. You also need to install an image processing gem and ImageMagick. I am using the mini_magick gem to interface to ImageMagick.
8. At this point you should be able to create ActiveRecord models that take advantage of attachment_fu. There is plenty of documentation out there, so I won't cover how to do this.

I also decided to get this running along side a Rails app running in Passenger, so that I would not need to manage anything else except Passenger and Apache. In order to get this running, I had to do the following:

1. Add a config.ru file to my base merb directory. Here are the instructions from the Merb Wiki on how to do this.
2. Modify your apache configuration and add a RackBaseURI option. The User Manual from Passenger explains how to do this.
3. As the Passenger instructions mention, on your server, symlink the Merb public directory to a new directory on the Rails App/Public directory (same directory name as the one that your named in your RackBaseURI) .
4. I had some additional issues with permissions and ruby inline. You may or may not have those, but here is how you can fix it in case you do.

I wrote these instructions a few weeks after the upgrade, so I may have missed something, let me know if I did.

Note: Instead of installing the merb_has_rails_plugins gem, I hardcoded into the init.rb file so that I wouldn't have to install the gem. This is a hack, and it would be cleaner to use the gem, but if that is not working for you, put this code into your init.rb (taken from the merb_has_rails_plugins gem):

Merb::BootLoader.before_app_loads do
  if defined?(Merb::Plugins)
    Dir["#{Merb.root}/plugins/*"].each do |dir|
      plugin_init = dir / 'init.rb'
       plugin_lib  = dir / 'lib' 

       if File.directory?(plugin_lib) 
         if defined?(ActiveSupport) 
           Dependencies.load_paths << plugin_lib 
           Dependencies.load_once_paths << plugin_lib 
        end
        $LOAD_PATH << plugin_lib
      end
      require plugin_init if File.exist?(plugin_init)
    end
  end
end

Hosting elastic applications in the Cloud using Scalr

Cloud computing has gotten a lot of buzz lately, along with a new slew of cloud providers popping up with various offerings. Amazon has been the pioneer in the cloud space, and is a good option if you know your way around a server.

I have been looking into Amazon EC2 for my startup's hosted RoR application. We are currently using a traditional hosting provider, but the "pay as you go" model along with autoscaling appealed to us.  Since we cannot estimate the amount of traffic we will be getting, it is hard to know how many servers to provision. Another disadvantage on a traditional host is that you have to provision servers to handle peak traffic, having your servers most likely idling most of the time on your dime. Moving to a cloud hosting provider such as Amazon EC2 circumvents these issues given that your server setup is automated, and can handle a dynamic amount of servers as well as server failures.  If these requirements are met, you can fire up servers as needed, or even have new servers fire up based on cpu load or other metrics.

In order to handle dynamic scaling and disaster recovery, at the minimum the system needs to deal with:

1. Database instance going down.  You need to make sure your database is backed up periodically in case the instance gets terminated.  When the new instance is brought back up, it needs to restore from this backup.  An Amazon feature called Elastic Block Storage (EBS) helps out as it provides persistent storage, but the process still needs to be automated.
2. Need to automate monitoring and spawning of new servers, as one server goes down a replacement has to be fired up.
3. Load balancing need to be automated. When new servers come online, they need to get registered in the load balancer, while shut down servers need to be cleared out.
4. Servers need to be pre-configured for certain roles as they are spawned, so that they have all the software needed to run the site.  You have a few options on this one: either bundling and saving the image to S3 and reusing these images each time, or starting with a base image and then configuring each server on startup using scripting or an administration product like Puppet
5. Configured images, or AMIs for your server types such as a Rails app server, MySQL server, memcached, a load balancer, etc..
6. Configuring DNS entries for live instances. DNS entries have to be configured as instances go up and down.  You will typically point your DNS entry to your load balancer, but it also applies to internal resources such as databases, cache servers, etc. Amazon Elastic IPs can provide help with at an additional price by providing IP addresses that don't change even if servers are terminated and restarted.
7. For database redundancy, it is nice to have a master/slave msyql setup where the slave gets promoted to master and another slave spawned up whenever anything happens to the master. In the worst case scenario that both go down, a new slave/master are spawned and the database is restored from backup (or EBS snapshot).
8. It is also nice to have server monitoring and scaling on the fly. For example, if the load on your app servers are over 2, spawn new servers.
9. Scheduled scaling is another nice to have.  For example, you know from experience that your servers are busy on weekdays between 9-5 and barely used at night, you can schedule to have a larger number of servers at peak hours, then scale back to the minimum at off peak times, reducing your overall bill.  This would be preferable to autoscaling if your usage is predictable, because with scheduled scaling, the servers will be ready to accept traffic as opposed to autoscaling which needs some time to build up to handle the traffic.

Looking at this list you may be thinking that it isn't as easy as you originally thought. Thankfully, this is where vendors like RightScale and Scalr come in, with the infrastructure to support this sort of dynamic server configuration. While RightScale has a great product and reputation, the pricing higher than we can afford.  There is also an Amazon API coming out that will provide monitoring, scaling, and load balancing.  However, the ETA seems to be unknown, so we decided to try out Scalr, which was quite affordable at $50 a month.  So far I've been very impressed with Scalr.

Scalr solves a lot of the requirements I listed in a clean automated way:

1. Scalr has a MySQL database image that you can use with automated backups to S3 (EBS support is in the works).  When a server is started, it automatically restores database from backup.  There is also an LVM MySQL option which allow faster database backups without slowing down your database servers.  
2. You configure your server farm and specify each server role that you need, along with the number of servers required for each type.  Scalr will monitor servers and restart new servers as they go down.
3. Scalr has a load balancing role (called www), which runs Nginx and is aware of the current instances that are up.  As servers are added or removed, Nginx is updated accordingly.
4. Scalr allows you to easily save any changes to the current role and apply the change to all other servers of the same role.  As new instances start up, they will use this updated role.  You can also save the role to use later in other farms.  There is also an option to execute scripts on servers startup, allowing you to hook in any configuration or puppet callbacks as needed.  There are also hooks you can setup for URLs to be called as new instances are started or stopped.
5. Scalr provides AMIs for common server types, such as PHP, Ruby on Rails, MySQL, Nginx, and memcached.  If you need more, you can start with their base image and configure the server yourself.  Currently, servers are running a debian based OS.
6. Scalr runs a set of DNS servers that are also aware of current instances.  There are also some preconfigured DNS entries you can use within EC2 to refer t your master database, slave database, etc.   There are also entries for load balanced mysql if you don't care which.  Scalr can also use DNS round robin for your load balancers, providing additional fault tolerance in case your load balancer goes down.
7. Scalr supports Master/Slave MySQL setup.  If you configure more than 1 database instance on your farm, it will automatically set one as the master, and others as slaves.  If the Master goes down, the slave gets promoted to Master, and another Slave is spawned automatically.  DNS entries are automatically configured.
8. Scalr will monitor your server instances, and you can specify the cpu load values where you want a new instance type to be spawned.  It will then start more instances as those load values are reached.

As you can see, Scalr seems to do a pretty decent job with my original requirements.  Some of the things currently missing are the MySQL EBS setup, which is in the works (which allows you better backup and recovery in case of database crashes), as well as scheduled scaling, which I haven't seen any plans for.  However, Scalr is open source, so theoretically you can write this yourself.   

That's right, Scalr is open-source, so you can run the Scalr instance on your on servers, or on the $50 a month hosted plan.  (If you run it on your own server, you would also need to setup some DNS servers as well.  Also, given that a small amazon instance would run $70, it might be more convenient to run on the hosted plan and take advantage of automatic updates and better uptime).  So far I have been pretty impressed with it, and the features it provides compared to other offerings such as RightScale which run at $500 a month.  Not to mention that setting up this infrastructure on my own would have taken me a lot of work, so I am happy someone else has provided it, and open sourced it for anyone to use.  I will keep this post updated as we roll into production.  My next post will go over how to set up a Rails application on Scalr in more details.

Custom application level observers in Rails

Have you ever wanted to have an observer for your own custom application level events instead of the default ActiveRecord callbacks? I needed this type of thing, and here is a quick way I ended up doing it..

But before we get into the custom observers, let's get some background. The ActiveRecord Observer is a great way to monitor your model objects in a clean, encapsulated way. They allow you to hook into ActiveRecord events such as object creation, update, and deletion. I have found them very useful, and my code has been cleaner and easier to maintain.

For example, suppose you have an application where changes to certain models require you to send out notification emails to a user whenever a user or order has been updated. Instead of scattering this code throughout the application, you can have one observer class that monitors the required models and deals with the email requirements in one place. Here is a simple observer that will monitor the User and UserOrder models and send out emails when they are saved:

class EmailUpdateObserver < ActiveRecord::Observer
  observer User, UserOrder

  #send out email when user or order is updated
  def after_save(model)
    MailManager.send_update_email(model)
  end

  #send out email when they are removed from system
  def after_destroy(model)
    MailManager.send_delete_email(model)
  end
end

This results in separation of concerns, the email sending code is separated out from the normal model code and is easier to maintain.

However, what if had a requirement that you only want to send out an email after an order has been approved or a user password has been changed? We are no longer sending out emails after every update - we need to figure out what is being updated in the model and in what context.

One possible way to do this is to to use new dirty checking in Rails 2 to see which fields have changed, and only send the email if the password or order status is changing. In this case it may be simple, but there may be situations where there are multiple fields that the state depends on that you need to look at, further complicating the matter. Another possibility is to set flags on your models depending the application event occurring, however, this just doesn't seem clean. It would be nice to define these application level events just once, and be able to observe them the same way you can observe ActiveRecord events, so you don't have to figure out what is happening, just deal with the result. This is what I needed for my requirements..

The first thing I looked at is the Observable module that Rails define that can be mixed into your class. It allows you to publish and subscribe to events from a particular model. However, it didn't quite fit what I needed. It allows you to observe a class instance, and receive "update" method calls from the model. The catch was, I didn't want to specify which instance to hook into, I wanted to do this on the class level. I also wanted to have custom methods for each event types, just like the ActiveRecord Observer. I also looked at acts_as_state_machine, but I didn't want to deal with different states, as in my case they were one time events that occurred. I decided to tweak the observer.rb class and create my own observer type. Here is how I can use this module to receive my own application level events:

class EmailUpdateObserver < ActiveRecord::Observer
  observer User, UserOrder

  def initialize        
    #custom observers
    User.add_callback_observer(self)
    UserOrder.add_callback_observer(self)
    super    
  end

  def after_password_changed(user)
    MailManager.send_password_email(user)
  end

  def after_order_approved(order, user)
    MailManager.send_order_approved_email(order, user)
  end

  #send out email when they are removed from system
  def after_destroy(model)
    MailManager.send_delete_email(model)
  end
end

In this example, I am registering to receive custom callbacks from the User and UserOrder classes, and defining the listener methods on the same ActiveRecord observer I have been using. I can now start defining meaningful application level events, and hook into them in a nice clean way. Here is how I notify the observers when something happens:

class User < ActiveRecord::Base
  include CallbackObservable

  def change_password(new_password)
    self.update_attributes! :password=>new_password
    notify_observers(:after_password_changed, self)      
  end
end

And finally, here is my CallbackObservable module:

#modified from Rails observer.rb.  Changed to create custom callback methods
module CallbackObservable
  def self.included(base) 
    base.extend ClassMethods
    base.class_eval do
      cattr_accessor :callback_observer_peers
    end
  end
  
  module ClassMethods
    def add_callback_observer(observer)
      self.callback_observer_peers = [] unless defined?(self.callback_observer_peers) && !self.callback_observer_peers.blank?
      self.callback_observer_peers.push observer unless self.callback_observer_peers.include?(observer)
    end        

    def delete_callback_observer(observer)
      self.callback_observer_peers.delete observer if defined?(self.callback_observer_peers) && !self.callback_observer_peers.blank?
    end

  def delete_callback_observers
    self.callback_observer_peers.clear if defined?(self.callback_observer_peers) && !self.callback_observer_peers.blank?
  end

  def notify_observers(*arg)
    if defined?(self.callback_observer_peers) && !self.callback_observer_peers.blank?
      for i in self.callback_observer_peers.dup       
        i.send(arg[0],*arg[1,arg.length-1]) if i.respond_to?(arg[0])
      end
    end
  end
end

This code can be refactored, as this was just a quick implementation on my part, and I I kept it simple - I did not add any acts_as or metaprogramming tricks for syntactic sugar.. However, anyone is welcome to do so.

Some amazing photos

Just ran across this Flickr account for hkvam and there are some amazing photos from iceland. The slideshow took 30 minutes to go through and almost every photo is incredible..

SoCon 08

I attended SoCon08 this weekend at the Kennesaw State University campus. It was an interesting event, bringing together bloggers, entrepreneurs, online media folks, and business people. It was a great mix of people and there were a lot of great discussions that took place.

Jeff Haynie kicked off the event and spoke about the five things that changed our lives since SoCon07. These included:

• Facebook
• The iPhone
• Radiohead
• Digital Content Distribution
• Guitar Hero
• MySQL

The floor was then opened to the crowd to talk about the changes that most affected their lives within the last year. It was great to hear people talk about the ventures or businesses they were trying to create and how many things can change in a year. For me personally, I would include Meetup, LinkedIn, and Flickr in that list as well. Meetup and LinkedIn have been a great way for me to make a connection on the online world, and translate those into real connections on the offline world with people that share the same interests as I do. Also, I have been getting into photography lately, and Flickr is becoming pretty addictive.

It was fun to see keep track of Twitter and Flickr as the event was going on. There were some great pictures taken and posted on Flickr (Tagged with socon08), and it was amazing to see 3 pages of photos already posted by lunchtime.

I attended the session on "What It Takes to Be a Winning Technology Entrepreneur". It was an interactive discussion on several topics, including: what it takes to open a successful venture, how to translate your technical skills into a business success, obtaining funding versus bootstrapping, how to network and find a business partner, and also some discussion about the capital situation in Atlanta compared to Silicon Valley. Overall, I took some good points away from this discussion, and also got a new reading recommendation, the eMyth. I also realized that I need to improve my networking skills, especially when dealing with business people. Being from the technical community, I can network and communicate pretty well when it comes to people in my field. However, when dealing with non-technical business people, I need to be better about focusing on high level strategy and business benefits as opposed to the technical aspect of things.

One point that was mentioned was that Atlanta did not have as many successful companies with big exits compared to the West Coast (JBoss and ISS were mentioned). Something that struck me as odd when I was having lunch and talking to various people, was that many of them have never even heard of JBoss. It seems people here are not as plugged in to the local community as people are in the west coast. JBoss was huge for the Atlanta area, as well as the open source world, and I would have expected other entrepreneurs to have heard about them.

Overall, I am very happy to see things in Atlanta starting to pick up with events like SoCon, PodCamp, and Startup weekend, and I am going to try to attend more of these.

Atlanta Web Entrepreneurs and Facebook

I attended the Atlanta Web Entrepreneurs meeting tonight, where we discussed the Facebook platform, and how to leverage it for web businesses. The format was a round table type discussion with no preset agenda. The discussion started out somewhat slow, as not everyone was familiar with Facebook and how to use it. However, things picked up, and we ended up having quite a good discussion. Topics covered included:

• Basic Facebook introduction
• Advertising your brand on Facebook
• Using basic profile pages and groups to advertise
• Facebook ads, and targeting more specific demographics
• Facebook applications and how they work
• Privacy for Facebook apps and advertising
• Subtlety in advertising - need to add something of value as opposed to direct selling
• What makes a successful Facebook application
• The tipping point and long tail of Facebook applications

There were several other people that had either advertised through Facebook, or had written a Facebook application. Ali Suleman, who developed the Entourage and Superlatives applications (both of which are successful applications on Facebook) was also present, so it definitely was good to hear some of his insight about Facebook apps. Seems the barrier to entry to these applications are pretty low, however, first to market seems to be pretty important. It all makes me when we will be hearing about the first "Superpoke" Millionaire... Overall, it was a great meeting and I learned a good amount. Also, it was good to be in touch with the community and to see that there are interesting things going on in Atlanta as well. Check out the Meetup page. There were also some articles of interests tagged by the group members. I definitely plan to attend more meetings in the time to come.

New Toy

I just got a new toy to play with - the Pentax K100D Super Digital SLR camera. Coming from a point and shoot camera, shooting with this new toy is tons of fun. For one, you can snap a lot more pictures without having to wait for the camera to catch up. The picture quality is noticeably better from my previous pictures I've shot with normal P&S cameras. Also, you can control shutter speed, aperture, and other settings to give you photos different looks. So far I am very happy with this choice - and it is currently on rebate right now, making it only $400! Here are the pros of the camera from a beginner's point of view:

• Great price
• Great build quality - doesn't feel cheap
• Great lens kit
• Shake reduction

Cons:

• 6.1MP as opposed to some other ones that are 10MP
• Slower in continuous mode, may not be the best for sports photos

Here is a more complete review from Steve's Digicam Here are a few pics I like that I shot. These are from piedmont park (for more pics, checkout my Flickr page)