Yet Another Programming Blog

For a recent project I had a need to receive emails (actually MMSs, but that’ll be the subject of a future post) in a Rails application. My requirements for the solution were:

1. Shouldn’t require root access: I’m deploying this app to a shared hosting service on which I don’t have root.
2. Shouldn’t require firing up the Rails stack for every incoming email: A number of the solutions I came across use this method. Seems awfully inefficient, especially if you expect to receive lots of messages. Since I’ve already got a running rails application, why not just create a new service/controller in it?
3. Can be scheduled to run automatically: Pretty obvious. I want to be able to retrieve messages periodically.

There is already a lot of information out there on how to achieve parts of these requirements, including the Rails wiki, Rails Tips and Craig’s writeup. My solution is really just piecing together bits of each. Here’s a high level overview of the components in my solution :

High level overview of components and desired interaction

Mail Poller:

This is the component responsible for :

• Connecting to the email provider (Gmail)
• Downloading new emails
• Posting contents of new emails to the Rails app.

I started by creating a new ruby script in RAILS_ROOT/script/ called mail_poller. This is based heavily on the script by John at RailsTips

#!/usr/bin/env ruby
require 'net/imap'
require 'net/http'
require 'rubygems'
require 'logger'

config = YAML.load(File.read(File.join(File.dirname(__FILE__), '..', 'config', 'mail.yml')))
log = Logger.new(STDOUT)

begin
  imap = Net::IMAP.new(config['host'], config['port'], true)
  imap.login(config['username'], config['password'])

  # select inbox as our mailbox to process
  imap.select('Inbox')
  # get all emails that are in inbox that have not been deleted
  imap.uid_search(["NOT", "DELETED"]).each do |uid|
    # fetches the straight up source of the email
    source   = imap.uid_fetch(uid, 'RFC822').first.attr['RFC822']
    # Post the email to the Rails app's restful service
    res = Net::HTTP.post_form(URI.parse(config['service_url']), {'email'=> source})
    case res
      when Net::HTTPSuccess, Net::HTTPRedirection
        # OK
      else
        res.error!
    end
    # Delete the email
    imap.uid_store(uid, "+FLAGS", [:Deleted])
  end

  # expunge removes the deleted emails
  imap.expunge
  imap.logout

  # NoResponseError and ByResponseError happen often when imap'ing
  rescue Net::IMAP::NoResponseError => e
    # Log if you'd like
  rescue Net::IMAP::ByeResponseError => e
    # Log if you'd like
  rescue => e
    log.warn e
end

This script relies on a RAILS_ROOT/config/mail.yml script for configuration of mail server, username, password etc. Here’s a sample of that config file. It is pretty ugly, and needs to be refactored to be environment aware. Perhaps I’ll update this post once I’ve refactored it. The username & password in the service_url are explained in the “Mail Controller” section below.

host: 'imap.gmail.com'
port: '993'
username: 'your.address@gmail.com'
password: 'your password'
service_url: 'http://admin:pass@localhost/admin/incoming'

I like this solution over a postfix/sendmail/qmail/et all solution because it keeps the logic of fetching mail ‘close’ to my application. What does that mean? If I needed to deploy my application to a new host, I wouldn’t have to reconfigure an MTA on that host. It also has the added benefit of not requiring sudo/root privs.

Mail Poller Scheduler:

Since this is running on *nix systems, cron is a well vetted, ubiquitous solution. Creating a cron job to run our mail poller every minute is easy. Add the following to your crontab entry (crontab -e). Replace the full path to ruby with what is correct for your system (a ‘which ruby’ command should give you the  path you need). Replace RAILS_ROOT with the full path to your rails application.

* * * * * /opt/local/bin/ruby RAILS_ROOT/script/mail_poller >> RAILS_ROOT/log/mail_poller.log

This will ask cron to invoke the mail_poller script every minute, and redirect STDOUT to the ‘mail_poller.log’ log file.

Mail Controller:

This is the last piece in the puzzle. It is invoked via a POST when the poller retrieves a new email. In my case, the controller is namespaced to an /admin/ area which is password protected, thus the need for the username & password in the service_url above.

Lets start of with my routes.rb

  ...
  map.namespace :admin do |admin|
    admin.resource :incoming,   nly => [:create]
  end
  ...

This creates the folowing route :

admin_incoming POST   /admin/incoming                  {:controller=>"admin/incomings", :action=>"create"}

My controller looks like this :

class Admin::IncomingsController < ApplicationController
  before_filter :admin_required
  skip_before_filter :verify_authenticity_token  

  # Being invoked as a POST from the mail_poller
  def create
    mail = TMail::Mail.parse(params[:email])
    # Do stuff with the mail object
    ...
  end
end

I’m using TMail to parse the email (which is also used by ActionMailer’s receive method). Once you’ve got the mail object parsed, you can process away at will. There are plenty of examples out there on how to walk through the email looking for what you want.

Acknowledgments:

As mentioned above, this solution is heavily based on the following resources. Many thanks to the folks behind that information

• Rails wiki
• Rails Tips
• Craig’s writeup

Summary:

I really like the simplicity & separation of concerns this solution provides. The poller is pretty ‘dumb’, i.e. it doesn’t have any application logic (does nothing with the contents of the email), and it doesn’t have any timer/sleep logic. The application logic is relegated to the already running rails application. The timer logic is relegated to the old stalwart, cron.

I’d love to hear your thoughts on this approach. Post a comment below and let me know what you think.