Emacs in a terminal on OSX

My development workflow is based around a full-screen terminal running GNU screen (actually, Byobu, with the escape key mapped to j so that ctrl-a will still work…nice since is under the index finger, so very fast and gives the pinky a break), with one screen devoted to Emacs. I recently started working for a great company, and they supplied me with a new MacBook, so I needed to figure out how to get my Ubuntu workflow working on OSX.

Here is the short list of how I did that:

• Install iTerm for true full-screen terminal loveliness
• Disable OSX option-key madness so option keys work in the terminal:
1. Bookmarks > Manage Profiles
2. expand Keyboard Profiles
3. select your keyboard profile and change “Option Key as” to “Meta”
• Install a modern version of Emacs (the one that comes on OSX 10.6 is way out of date). Unfortunately, $ brew install emacs failed, but a blog post pointed out how to install Emacs 24 from Homebrew head:

  $ brew install emacs --use-git-head --HEAD
  

  (note that I removed the –cocoa option from the line used in the blog post since the goal is Emacs in a terminal)

• Set up your Emacs configuration (or use the Emacs Starter Kit–here are my custom configurations on top of the starter kit)

The big problem remaining after taking these steps was that the kill-ring could not access the clipboard. On Ubuntu, xclip.el enabled a terminal based Emacs to access the Gnome clipboard via the xclip utility. On Mac OSX, there are a pair of utilities called pbcopy and pbpaste that do the same thing, so I modified xclip.el (written by Leo Shidai Liu) to work with those.

Here it is:
https://gist.github.com/1023272

Activate via:
(require ‘pbcopy)
(turn-on-pbcopy)

More Internet Explorer + respond_to trouble

In a previous post, I covered one situation in which Internet Explorer breaks the Rails respond_to method. But other situations, specifically respond_to blocks that differentiated between HTML and PDF, seemed safe. Now that appears to be broken, too.

A client of mine has a Ruby on Rails application (Rails 2.3.8) with just over 4,000 users. Up until last week, the following respond_to block worked fine:

    respond_to do |format|
      format.html { render(:show) }
      format.pdf do
        render(:pdf => @document.title,
                  :template => 'documents/show.html.haml',
                  :stylesheets => ['main', 'print', 'prince'])
      end
    end

On 21 October 2010, a new user joined the system and started receiving the PDF version instead of the HTML version even though the URL omitted an explicit format. That is, the user clicked a link to http://example.com/documents/1234, but was still delivered a PDF. The user was browsing with Internet Explorer. I can’t reproduce the problem, and no other IE users of our system have reported this problem, but I presume the user’s version of IE is explicitly listing “application/pdf” in its HTTP_ACCEPT header, which gives priority to PDF over HTML, which is not explicitly listed (“*/*” at the end of the list catches HTML).

If it takes 4,000+ users to discover that respond_to doesn’t work in a particular situation, it makes it very difficult to know when one can trust respond_to to work and when not. Will it start breaking for JSON next because somebody installs some clever plugin on IE that makes IE prefer JSON over HTML? I hope not.

In order to solve this for PDFs, it looks like I’m going to break out .pdf into its own case just as I did for .doc and .xls.

Nonsecure warning in Internet Explorer

It documented that url() references in CSS served over HTTPS need to explicitly list the full URL in order to avoid the ‘nonsecure’ warning on IE7 & IE6, but I just ran into a variation on this that I found non-obvious:

I’m making use of http://fancybox.net/ to display contextually relevant tutorial videos. In jquery.fancybox-1.2.6.css, there was an embedded, blank image:

background-image: url("data:image/gif;base64,AAAA");

Because that URL is not explicitly “https://” it caused IE7 to throw the “this page contains both secure and nonsecure items” even though the data was embedded in the CSS file. Changing that url to the explicitly https url of an actual gif solved the problem.

Resque and Redis on Ubuntu

If you are getting a message such as

rake aborted!
You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.map

when you try to run Resque workers (for example by QUEUE=* rake environment resque:work), then here is the solution: http://github.com/defunkt/resque/issues/110

I’m simply posting here because Google did not turn up that Github issue in all my variations on the above search terms. I’m also including the actual solution, posted by Github user djanowski:

djanowski 2 days ago | link

Hello,
The issue is that Redis changed this behavior: before, calling commands operating on non-existent keys would return nil. Now it returns the same result as a key containing an empty structure.
On the other hand, the Ruby client had a bug that was treating a nil response from Redis as an empty array. This was fixed in 2.0.1. So now the client correctly reflects Redis’ behavior.
Thus, your options are:

* Use redis-rb 2.0.0.
* Upgrade Redis to a more recent version (something later than antirez/redis@4e27f26).
* Patch Resque to handle this bug (less optimal in my view). Basically: redis.smembers(:workers) || [].

PostgreSQL tables owned by different user than the containing database

After restoring a PostgreSQL database, I recently ran into “PGError: ERROR: permission denied”:

$ rake db:create
$ sudo su postgresql # actually, I was restoring a number of development databases after re-installing Ubuntu recently, so I was already logged in as user postgresql in one of my screens, and I just used that to restore several databases at once
postgresql$ psql db_name < backup.sql


But then, when I attempted to apply further migrations:

$ rake db:create
rake aborted!
PGError: ERROR: permission denied for relation schema_migrations
: SELECT version FROM schema_migrations


The problem is that, although the database was owned by the correct PostgreSQL user (handled by rake db:create and the database.yml file), when I restored the database contents, I was logged in as the postgresql super user, so the tables within the database were owned by postgresql, not the user who owned the database. The solution was to restore databases using the user who owns the database:

$ rake db:drop
$ rake db:create
$ psql -d db_name -U db_owner_username -W < backup.sql


Testing Ajax with RSpec

Overview

Every now and again I think I should maybe be integrating Javascript tests into my test suite. Then I take a look at the setup (Celerity, running in Java, plus Culerity, on top of my existing setup of RSpec + Cucumber + ROTS), and I think, “ick”.

Culerity & Co. probably makes sense for applications that are heavy on front-end modification using Javascript. But for my purposes, insofar as it is possible, I build my applications to work without Javascript, and then Ajaxify forms using jQuery. The JSON response includes the output of a rendered partial that replaces existing content already on the screen. Those partials are thoroughly tested via the standard, non-Ajaxian pathways.

The only part that has bitten me (and that not in a full-fledged production app) is when the underlying partial changes in some way that makes it incompatible with the way the partial is called when responding to JSON (for example, by making the partial require a new local variable).

Details

To ensure that all rendering pathways are tested, I added an RSpec example for every Ajax call. All it needs to test is whether the call is successful because all other aspects of the call are tested via the non-Ajaxian pathway. That gives the following:

  describe 'POST create via AJAX' do
    integrate_views
    it 'should be successful' do
      activate_authlogic
      UserSession.create(@contributor)
      post(:create,
           :node_id => @node.id,
           :score => 15,
           :format => 'json')
      response.should be_success
    end
  end

The “integrate_views” line is important. Usually, controller specs don’t render views, but in this case, the ability to render the view is the main thing we’re testing. It’s great that RSpec makes this option local to a describe block.

Windows + Word + IE + respond_to = trouble

Backstory
So, I had a respond_to block handling PDF and HTML. Without an explicit format, the RESTful route returned HTML. With an explicit PDF format, it returned PDF (using Prince XML and Princely to keep DRY). Straightforward stuff.

Then the client requested that the documents also be downloadable in MS Word format. So, after building a failing Cucumber spec, I added a Doc format block and returned some paired down HTML with inline CSS as an application/msword attachment. This gave the following respond_to block:

  def show
    respond_to do |format|
      format.html { render(:show) }
      format.pdf do
        render(:pdf => @document.title,
                  :template => 'documents/show.html.haml',
                  :stylesheets => ['main', 'print', 'prince'])
      end
      format.doc do
        @format = :doc
        response.headers['Content-Disposition'] = "attachment; filename=\"#{@document.title}.doc\""
        render(:layout => 'document.doc.haml', :template => 'documents/show.doc.haml')
      end
    end
  end

Tests went green. The Doc format worked on Ubuntu with OpenOffice and on OSX with MS Office 2004 for Mac. It worked with Windows and Word, too…only problem is, it worked “too well”.

After pushing the “download as Doc” update into production, I received a panicked call from the client saying he couldn’t reach the documents…every time he tried to visit a document from the index, it just downloaded a Word document. Say what!? I tried it out in IE7 on WinXP and IE8 on Vista. No problems on my end. Visiting documents from the index worked as expected, resulting in the HTML version, from which I could access the Doc formatted link and download the Word version.

Solution
It turns out that if MS Word is installed, then Internet Explorer prefers Word format over HTML. So the .doc format in the respond_to block is the one that is executed even when no explicit format is given and the .html block precedes the .doc block. That is, given the above #show method, http://example.com/document/1234 —when called by IE on Windows with Word installed—caused a Word document to be downloaded every time.

The first post in this thread explains this situation (it has to do with the HTTP_ACCEPT header sent by IE). I don’t have MS Word or MS Office installed on any of my Windows machines. I prefer OpenOffice, and in any case, I only use Windows for testing, or when I need to use an app for which I only have a Windows license, such as Photoshop or Illustrator.

The solution I’ve settled on is to use an outer case statement on the explicit :format param to isolate explicit Microsoft formats from the other, saner formats, and then to execute the respond_to block within each of the cases, giving:

  def show
    case params[:format]
    when 'doc'
      respond_to do |format|
        format.doc {
          response.headers['Content-Disposition'] = "attachment; filename=\"#{@document.title}.doc\""
          render(:layout => 'documentdoc.haml', :template => 'documents/show.doc.haml')
        }
      end
    else
      respond_to do |format|
        format.html { render(:show) }
        format.pdf {
          render(:pdf => @document.title,
                    :template => 'documents/show.html.haml',
                    :stylesheets => ['main', 'print', 'prince'])
        }
      end
    end
  end

OpenID Integration Testing in Rails

Overview

Providing the option of OpenID for account creation and sign-in helps keeps things easy on your users. It may even lower the barrier to people signing up for (and continuing to use) your service.

Authlogic implements user authentication beautifully, and it is readily extensible. By way of illustrating how to extend Authlogic, its author, Ben Johnson, released Authlogic OpenID. That makes supporting OpenID authentication as clean, easy, and flexible as Authlogic makes user authentication.

The difficulty, up to this point, has been testing OpenID. Some time ago, I was looking for ways to test OpenID, and I happened upon the Ruby OpenID Test Server (ROTS), by Román González. Googling various combination of ROTS and OpenID or Rails, one would think the experiment had failed, because of the lack of mention and its author’s only two blog posts on the topic [1] [2]. But I contacted Román via Git, and he offered very helpful assistance and pointed me to the rots_consumer, which is an example Rails (and Merb) application that uses ROTS to test OpenID. Today, I followed Román’s instructions and now have operational OpenID scenarios in my Cucumber suite (I was inspired to do so today because I wanted to add another OpenID related feature, and felt dirty not being able to do BDD).

This post covers how to get OpenID working in Cucumber (I’m running Cucumber 0.6.1, Cucumber-Rails 0.2.3, and Rails 2.3.5).

Details

To quote its ReadMe, “ROTS is a minimal implementation of an OpenID server.” When running OpenID tests against ROTS, the ROTS server must be running locally. It seems pretty lightweight, so isn’t a big deal, but if you don’t want to run against it all the time, tag your Cucumber scenarios that involve OpenID (or an entire file) with @openid and include “–tags ~@openid” as one of the std_opts in config/cucumber.yml. Then, when you want to test OpenID, run
$ cucumber --tags @openid features

Setting up your application to work with ROTS is pretty easy. First get ROTS. Román’s version is here. I ran into a problem in his current version with an application that requests dob from the OpenID server, so I wrote a patch. If you request dob as part of your OpenID request, check to see whether Román’s repository has fixed that issue, or use my fork. Either way, clone ROTS to your local system.

Once you have ROTS locally, build, install, and run the gem:
$ cd rots
$ gem build rots.gemspec
$ sudo gem install rots-0.2.2.gem
$ rots

I changed my default Screen settings to include a rots window and automatically launch rots. That way, I always have rots running when I open my work environment:
screen -t rots /bin/bash -c 'rots'

As an aside, I’m running Ubuntu 9.10, which comes with an elegant version of Screen called Byobu. Byobu makes it easy to set all the Screen preferences, such as changing ctrl-a to ctrl-j (not only is this better for Emacs, but J is the rest position for the right index finger, so is very fast).

Now for your Rails app.

Create features/support/openid.rb with the following contents:

require "rots"

World(Rots::TestHelper)

And features/step_definitions/openid_steps.rb with the following:

Then /^I (?:should )?receive a response from the OpenID Server$/ do
  server_request_uri = response.headers['Location']
  hash = openid_request(server_request_uri)
  visit(hash[:url], :get, hash[:query_params])
end

At this point, you can write your Cucumber scenarios. The important step to remember is, after any successful call to the OpenID server, you must have the line

    And I receive a response from the OpenID Server

or

    Then I should receive a response from the OpenID Server

Here are the OpenID steps I implemented for one application:
http://gist.github.com/334789

Migrating MySQL to PostgreSQL in Rails

The AR-DBcopy gem, by Michael Siebert, made this very easy. The ReadMe doesn’t cover use of the gem, so I’ll cover that here.

Overview:

AR-DBcopy uses ActiveRecord to achieve database agnosticism. Both the source and target databases are accessible a single instance of Rails (in my case, both databases were on the same machine, but if you configure database.yml to connect to a different host, then they need not be). The source contains the data; the target starts out empty, but with the correct schema (generated by running rake db:migrate).

Steps: (Be sure that nobody will be interacting with the web application during this process.)

1. Set up everything you need to in order for your Rails app to work with Postgres (install Postgres server, pg gem, create a Postgres user, change your database.yml file (keep the MySQL settings handy), and run rake db:create and rake db:migrate). That leaves you with a MySQL database with your data and an empty Postgres database with the correct schema.
2. Modify database.yml so that it matches the example. This is easy, simply copy your old MySQL settings to “source:” and your new Postgres settings to “target:”

   Original database.yml:

   production:
     adapter: mysql
     encoding: utf8
     database: old_dbname
     username: old_dbuser
     password: old_pass
     socket: /var/run/mysqld/mysqld.sock
   

   Modified database.yml:

   source:
     adapter: mysql
     encoding: utf8
     database: old_dbname
     username: old_dbuser
     password: old_pass
     socket: /var/run/mysqld/mysqld.sock
   
   target:
     adapter: postgresql
     encoding: unicode
     database: new_dbname
     username: new_dbuser
     password: new_pass
   
   production:
     adapter: postgresql
     encoding: unicode
     database: new_dbname
     username: new_dbuser
     password: new_pass
   

3. Install the ar-dbcopy gem:
   $ sudo gem install ar_dbcopy
4. From the Rails root directory, run the gem:
   $ ar_dbcopy config/database.yml
5. After this finishes (for me, it took on the order of 5 minutes for 100K records), you will need to reset the PostgreSQL id sequences by performing
   select setval(‘users_id_seq’, (select max(id) + 1 from users));
   for every table in the database that has a corresponding _id_seq

PGError: relation “model_name” does not exist

Was getting this error in a Postgres backed Rails app I was building. After much googling, I found someone mention something about having messed up their migration. My migrations were fine, but I dropped, created, and migrated the test database anyway, and the problem disappeared.

Here’s the error I was getting, in case it helps Google direct someone here:

ActiveRecord::StatementInvalid in 'CardsController GET index with user logged in assigns all cards as @cards'
PGError: ERROR:  relation "cards" does not exist
LINE 4:              WHERE a.attrelid = '"cards"'::regclass
                                        ^
:             SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
              FROM pg_attribute a LEFT JOIN pg_attrdef d
                ON a.attrelid = d.adrelid AND a.attnum = d.adnum
             WHERE a.attrelid = '"cards"'::regclass
               AND a.attnum > 0 AND NOT a.attisdropped
             ORDER BY a.attnum