Better XML needed in Python

I must say that I have just about lost patience with the heroic maintainers and developers of python xml libraries. I'm grateful to have those tools at my disposal but am frustrated that so many key ones seem to languish or are somewhat incomplete. For example, I love elementtree but there still is incomplete XPath support. I really need XPath and have to use 4Suite to get it. I also use 4Suite for applying style sheets. 4Suite is good but is just awkward enough to be not joyful to use. It went for a while without any updates and sat at beta status seemingly forever. Then there's ZSI for SOAP. Well, SOAP is pretty unhappy stuff anyway so you just kind of have to make the best of a bad situation.

Believe it or not, I have to use elementtree, 4Suite, and ZSI on a single project. Oh well, at least I was able to ditch PyXML on that project.

Thanks for the rant space.

mod_rails is smokin' fast

I recently switched a heavily used production Ruby on Rails server from using a cluster of mongrels proxied by Apache's load balancer to using only mod_rails in Apache. No more mongrels.

Right away, I could tell that the application was noticeably more responsive. I plan to let this run for a week and then I may put up some pre and post benchmark data.

My first impressions:

• mod_rails was easy to install using the documentation provided. I needed to install two pre-requisite packages on my CentOS 5.1 server: gcc-c++ and httpd_devel along with their own dependencies. The Yum package manager made quick work of that.
• mod_rails has a friendly installer which interpreted into plain English what otherwise would have been hairy looking build errors.
• mod_rails was easy to configure. I already had a virtual host set up for the Apache proxied mongrels. I only had to load the mod_rails module, remove the rewrite engine rules (as the docs instructed), restart Apache, and that was it.

My takeaways:

• Deploying with mod_rails is much simpler than deploying with proxied mongrels.
• It appears that mod_rails is much quicker than mongrel.
• There also appears to be a substantial memory savings in not have to run so many mongrels.

Case sensitivity in databases

Recently, I had to clean up some SQL to be case-insensitive. It was originally written to be run in MySQL which defaults to case-insensitive searches but can be set otherwise. Now, however, the SQL run is being run in PostgreSQL which always performs case-sensitive searches.

Here's an example using Ruby on Rails's Active Record:

Suppose that a name is stored in the providers table as "SMITH".

Below, the set collection will be empty because Smith is not like SMITH.
@search_field = "Smith"
set = Provider.find(:all, :conditions => "name LIKE '%#{@search_field}%'")


But this yields the expected results:
@search_field = "Smith"
set = Provider.find(:all, :conditions => ["LOWER(name) LIKE LOWER(?)", "%#{@search_field}%"])

The reason the second example works is because it uses the database's LOWER() string function to change Smith to smith and SMITH to smith.

And, as it so happens, the UPPER() and LOWER() string functions both exist and work identically in MySQL and PostgreSQL so the code above is portable.

Rails and OS X

I spent hours today getting a Ruby on Rails application running on an OS X 10.5.2 server.

The problems were numerous but, honestly, it took so long mainly because I never use OS X and didn't have familiarity with some things that I thoroughly understand in Linux and Windows.

Before I could even begin the process, I had to export the tagged version from my company's CVS repository. CVS cannot export empty directories. From the CVS manual:

(Note that cvs export always removes empty directories.) Probably the best way to do this is to always specify `-P'; if you want an empty directory then put a dummy file (for example `.keepme') in it to prevent `-P' from removing it.

The person who initially applied the version tag didn't know to take care with the empty directories. As a result, I had to add an empty '.cvsignore' file into each directory, commit those dozens of .cvsignore files, and then tag them. That wasn't a big deal to do with "find" once I knew what was needed. This was a shotgun approach but was benign.

First, I could not get mysql to listen to the TCP/IP interface despite what was in the /etc/my.cnf file. Turns out that one has to fiddle with the setting with:
serveradmin settings mysql:allowNetwork = yes
(Is that going to persist after a reboot?)

Second, I couldn't get the C-based mysql ruby gem installed. I simply didn't have the header and library files against which it could build. This should not have been a big deal except that there wasn't a clear source package to download from the mysql site. I had to rely on a tech. support person to do that.

Finally, I had to make sure that the various mongrel instances would start automatically. That took some doing and I finally got them configured using launchd and launchctl. Launchd functions as a complete replacement for inetd, init.d, cron, and a few other things. Basically, one has to configure some XML files. It took some time to get my xml files just right.

One weird thing that I encountered is that if I told the mongrel service to bind to the "localhost" interface it would only do so using IP v.6. I had to specifically tell it to use "127.0.0.1" to coerce it to use IP v.4.

The Good:
Once I got into launchd, I began to appreciate how it could simplify a lot of the traditional unix utilities and configurations.

The Bad:
OS X and running Rails on OS X is a niche market for my company. I can't help but to think what a waste of time this was because I won't be able to leverage this experience over many installations. Even if I could, I wouldn't want to because I have zero interest in OS X.

The Ugly:
In reading the launchd documentation, the Apple folks come off as unjustifiably arrogant. Check out this quote from http://developer.apple.com/macosx/launchd.html:

If this is not sufficient for your job, you need to modify your program or file an enhancement request against launchd with Apple.

Shibboleth and Rails

My company wrote a Rails application for a client at a major university that was intended to authenticate via Shibboleth to the campus's single-sign-on site. The application is fronted by a cluster of mongrel servers which are proxied by Apache 2.2.

The idea is to authenticate against Shibboleth which will set the REMOTE_USER variable in Apache to the authenticated user's username. The Rails application would then authorize against that REMOTE_USER variable.

The Shibboleth setup went okay and I was able to secure the site. The problem, however, was that the REMOTE_USER variable was not showing up in the request.env hash in Rails as it should.

Ultimately, the fix was to include this chunk of code in my Apache virtual host config file.

RewriteEngine On
RewriteCond %{LA-U:REMOTE_USER} (^(.+?)@.+$)
RewriteRule . - [E=RU:%2]
RequestHeader add X-Forwarded-User %{RU}e

What this did was to take the remote user variable that gets set by Shibboleth and write it to a new header. That header, in turn, gets renamed by mongrel as HTTP_X_FORWARDED_USER. In the Rails app, I could then read the value of request.env['HTTP_X_FORWARDED_USER'] in my authorization routine.

Oh, and that regex in the RewriteCond line was to strip of the "@foo.edu" that gets set even though Shibboleth was configured to strip that off.

Credit for this solution goes to the author of the post at http://www.ruby-forum.com/topic/83067#151189.

ActiveRecord and SQL Server

I was having a really frustrating problem in that long-running queries to a (MS) SQL Server database were resulting in a TimeoutError being raised. After much research I determined that the issue was with the ADO driver's CommandTimeout property. It needed to be set to be much higher. From circumstantial evidence, it seemed to timeout after 30 seconds.

Setting that property via ActiveRecord was tricky. The property can be set directly against the underlying DBI database handle object. That object can be gotten at through the ActiveRecord::Base.connection object.

The name of my ActiveRecord model is IDX. So, I set the timeout with:
IDX.connection.instance_variable_get(:@connection).handle.instance_variable_get(:@handle).setproperty('CommandTimeout', 7200)
That works but, are you kidding me?!

Credit goes to the author of the blog at http://www.enterpriseetc.com/post/Ruby-in-Practice.aspx. His update at the very end ultimately pointed me in the right direction.

Moving to PostgreSQL

We had a mini-throwdown on database servers yesterday. After some research, we determined that we could not use MySQL in one of our commercial products without purchasing the commercial license for MySQL. That license costs $600/year/installation. That kind of money adds up over 40 installations.

Given that, we have decided to try using PostgreSQL instead. It is a robust database with plenty of good software APIs written around it. We need to be able to connect with it via PHP, Phython, Ruby, and ODBC and as far as I can tell, all of those will work just fine. PostgreSQL, besides being robust, is absolutely free for us to use in our commercial application without a requirement to open source the code. We have even decided to try using it in our custom applications too.

I spent some time in postgres yesterday and got up to speed quickly. It has a wealth of enterprise-worthy features but I'm sure that we won't be using those. If I'm wrong, then, hey, at least they are already in the server and we can scale up as needed. One explicit technology decision that my company has made is to use the back-end database simply as a data store and not to use its advanced features such as views, triggers, and stored procedures, unless it becomes necessary on a case-by-case basis. We are pushing all of that business logic into the application layer so that it will all be in one place and will be easily maintainable by multiple developers. The goal is for our code to be database agnostic. The trade-off is a sligtly slower application.

Now my issue is selecting getting an ORM for Python to work with both MySQL and PostgreSQL. More to come...