05 August 2010

Changing jobs

Well, tomorrow is my last day at SciMed Solutions. I have been there (and with sister company Net Friends) for almost 7 years. That's the longest I have ever been with a single employer. In that time I have learned LOTS.

On Monday, I start over with Epiphany Cardiography Products. They are small but successful and do a lot of things well. I like the team that I'll be on too. My role there will be as a system administrator and software developer. That's pretty much the role that I have always played.

I'll miss the folks back at "SciFriends". Both companies, SciMed and Net Friends, are really top-notch. I feel good about the move to Epiphany though and am just so excited about my future there.

18 February 2010

Efail

Lately, I have been really dissatisfied with communicating through email. It just seemed to be such a low rate-of-return on my time spent communicating. I found a few links from people that think this way too that spell it out pretty well.

Is Email = Efail?

Email: The Variable Reinforcement Machine

Email Is For Old People?

15 October 2009

Shibboleth2 on Ubuntu 9.04

Here are my notes on how I got Shibboleth2 compiled from source on Ubuntu 9.04. YMMV.

Adapted from https://spaces.internet2.edu/display/SHIB2/NativeSPLinuxSourceBuild
  1. apt-get install wget
  2. apt-get install build-essential
  3. apt-get install apache2
  4. apt-get install libxerces-c28 libxerces-c2-dev
  5. apt-get install libxml-security-c14 libxml-security-c-dev
  6. apt-get install libcurl4-openssl-dev
  7. apt-get install libxmltooling1 libxmltooling-dev
  8. apt-get install libsaml2 libsaml2-dev
  9. download log4shib source from http://shibboleth.internet2.edu/downloads/log4shib/latest/
  10. ./configure --disable-static --disable-doxygen --prefix=/opt/shibboleth-sp
  11. make
  12. make install
  13. download XMLTooling-C source from http://shibboleth.internet2.edu/downloads/opensaml/cpp/latest/
  14. ./configure --with-log4shib=/opt/shibboleth-sp --prefix=/opt/shibboleth-sp -C
  15. make
  16. make install
  17. download OpenSAML-C source from http://shibboleth.internet2.edu/downloads/opensaml/cpp/latest/
  18. ./configure --with-log4shib=/opt/shibboleth-sp --prefix=/opt/shibboleth-sp -C
  19. make
  20. make install
  21. download shibboleth2 source from http://shibboleth.internet2.edu/downloads/shibboleth/cppsp/latest/
  22. ./configure --with-log4shib=/opt/shibboleth-sp --prefix=/opt/shibboleth-sp
  23. make
  24. make install

04 August 2009

Installing ESX 4 from USB Flash

The target server had a CD drive and I only had a DVD .iso file that was about 835Mb. I discovered that the server's BIOS would allow me to boot from a USB flash drive. I used unetbootin to burn the .iso file to my USB drive. Next, and this is the important part, I booted from the USB drive and pressed the Tab key when presented with the boot options for ESX. That then brings up the standard line of options to pass into the kernel. I appended askmedia to that line. The askmedia option allows one to specify that the install media is hosted and available elsewhere, for example, HTTP, FTP, NFS, or USB. Of course, I selected USB when prompted and the rest of the install proceeded automatically with no problems.

03 August 2009

stackoverflow.com is the best thing since sliced bread

I love stackoverflow.com. If you do any programming at all you must check it out. I had a really tough programming bug last Friday and posted a question on stackoverflow with the appropriate tags to help with classifying the questions. Within an hour I had two excellent responses one of which was from Python luminary, Alex Martelli. (His answer turned out to be correct -- no surpise there.)

A feature that I particularly like about stackoverflow is that I can use my OpenID credentials to authenticate. Beyond that, I can link my accounts to their sister site at serverfault.com.

These sites are always the ones that I look at first now for answers even before google! Well done!

13 July 2009

IPTables protection against brute SSH attacks

One annoying thing I see in my servers' logs for which their respective ssh port is not restricted is that there are always brute force attacks every day. For various reasons some of the servers that I administer have to have wide open ssh ports. I found two sites today that show how to use IPTable's "recent" module to slow down those brute force attacks. It works great!

11 February 2009

Python multiprocessing vs. threading performance

I recently wrote an application using the threading module in the Python standard library. The application itself was basically attempting to discover Open Reading Frames (ORFs) in a DNA sequence. The application appeared to be mostly CPU bound.

Running the application in a single thread took about 6 seconds for my test data. Running it continuously over 3 threads took about 30 seconds per run! The more threads added, the slower it ran on average. That is actually what I expected because of Python's Global Interpreter Lock (GIL). I decided to look at the multiprocessing module to see if I could get the average run time back down to 6 seconds.

Here's the result running the same data over 3 threads using the threading module versus the same setup but with 3 processes using the multiprocessing module.

Threading Data

Thread-3 took 22.3910000324 seconds
Thread-1 took 23.2190001011 seconds
Thread-2 took 38.8129999638 seconds
Thread-3 took 24.7969999313 seconds
Thread-1 took 26.375 seconds
Thread-2 took 35.2030000687 seconds
Thread-3 took 30.1089999676 seconds
Thread-1 took 29.375 seconds
Thread-1 took 24.109000206 seconds
Thread-3 took 26.5 seconds
Thread-2 took 36.0160000324 seconds
Thread-1 took 29.390999794 seconds
Thread-3 took 30.6720001698 seconds
Thread-2 took 32.5779998302 seconds
Thread-1 took 31.25 seconds
Thread-3 took 30.8439998627 seconds
Thread-2 took 32.0150001049 seconds
Thread-1 took 30.9220001698 seconds
Thread-3 took 30.6089999676 seconds
Thread-2 took 23.125 seconds

AVERAGE = 29.4 seconds


Multiprocessing Data
OrfDetection-2 took 6.65599989891 seconds
OrfDetection-1 took 12.4379999638 seconds
OrfDetection-3 took 12.4530000687 seconds
OrfDetection-2 took 6.43799996376 seconds
OrfDetection-2 took 6.375 seconds
OrfDetection-1 took 12.3589999676 seconds
OrfDetection-3 took 12.4070000648 seconds
OrfDetection-2 took 6.39099979401 seconds
OrfDetection-2 took 6.35900020599 seconds
OrfDetection-1 took 12.3280000687 seconds
OrfDetection-3 took 12.4059998989 seconds
OrfDetection-2 took 6.45399999619 seconds
OrfDetection-2 took 6.3900001049 seconds
OrfDetection-1 took 12.25 seconds
OrfDetection-3 took 12.2660000324 seconds
OrfDetection-2 took 6.43799996376 seconds
OrfDetection-2 took 6.42199993134 seconds
OrfDetection-1 took 12.3439998627 seconds
OrfDetection-3 took 12.2650001049 seconds
OrfDetection-2 took 6.15600013733 seconds
AVERAGE = 9.4 seconds

Besides the faster average run times, one other difference between the two implementations was that the application using threading tended to run at about 40% of CPU whereas the one using multiprocessing ran at 100% of CPU (each python process took about 33%).

I just now noticed that in the multiprocessing implementation, OrfDetection-2 always took around 6 seconds whereas OrfDetection-1 and OrfDetection-3 always took around 12 seconds or twice as long. Hmmmm. Wonder what that means. I'll have to investigate that further. I expected each to run in around 6 seconds.

09 January 2009

Database Trigger vs. ActiveRecord Callback

Out of curiosity, I wanted to compare the performance of an ActiveRecord "after_save" callback versus a PostgreSQL "AFTER INSERT" trigger.

I prototyped the functionality in Rails. I got the logic clean and simple. Next, I ported that logic to PL/pgSQL. I wrote a few time-related functions to keep the code clean but it was identical in flow and logic to the Rails code.

Lastly, I ran the the callback and trigger forms of the business logic.
The measurement here is the time it took for the POST action to complete. I know that it is imperfect but, in this case, its a good proxy because, ultimately, the point of this is to enhance responsiveness to the end-user. All other things are equal except for how this one chunk of business logic is implemented.
  • ActiveRecord callback- 181750ms
  • Database Trigger - 93729ms
So, that works out to a 93% decrease in execution time of the Ruby on Rails action when implemented with a database trigger. Honestly, for no particular reason I expected the trigger form of the logic to blow away the ActiveRecord form. 93% is whopping but I was unrealistically expecting something much faster (illogical, I know).

19 December 2008

Native Postgresql Data Types for ActiveRecord

In researching ActiveRecord migrations, I came across this bit of new information in their documentation:

Instantiates a new column for the table. The type parameter is normally one of the migrations native types, which is one of the following: :primary_key, :string, :text, :integer, :float, :decimal, :datetime, :timestamp, :time, :date, :binary, :boolean.

You may use a type not in this list as long as it is supported by your database (for example, "polygon" in MySQL), but this will not be database agnostic and should usually be avoided.

So, what? Well, this came in handy immediately because I was creating a table that used Postgresql's native inet data type. I plan to use some of the related inet functionality at a later date so this is a big win.

Here's an abbreviated version of my migration that uses the inet datatype. I am running Ruby on Rails 2.2.


class CreateDevices < ActiveRecord::Migration
def self.up
create_table :devices do |t|
t.column("ip_address", :inet) # this works beautifully
# t.inet :ip_address # this won't work

t.timestamps
end
end

def self.down
drop_table :devices
end
end

Notice that one has to be explicit and use the column method and not the shortcut.

18 December 2008

Ruby's native postgres gem on Windows

The ruby-postgres gem is a compiled and fast adapter. It has some dependencies for Windows, though, so follow these steps to get it running properly:

  1. go to the http://www.postgresql.org/ftp/binary/ and navigate to the release that matches the Postgres server that you installed.
  2. look for and download a file that follows the pattern postgresql-8.X-binaries-no-installer.zip where the X corresponds to the database release
  3. unzip the file and navigate to its bin subdirectory
  4. copy the following files into your ruby installation's bin directory
    • comerr32.dll
    • krb5_32.dll
    • k5sprt32.dll
    • libeay32.dll
    • libpq.dll
    • libiconv2.dll
    • libintl3.dll
    • ssleay32.dll
    • gssapi32.dll
  5. after you have done all that, you should be able to open an interactive ruby session (irb) and type require 'postgres' and get 'true' as a result, i.e., a non-error

17 December 2008

Pylons and SQLAlchemy

I recently gave Pylons and SQLAlchemy a spin. I mainly wanted to check out SQLAlchemy so that I could evaluate it for upcoming projects. Well, my opinion is that it is wildly flexible and, therefore, came with a steep learning curve. Pylons wasn't too hard to grasp although it seemed sort of arcane at times. I genuinely liked Pylons though.

This has been a disappointing experience for me. I wanted Pylons and SQLAlchemy to be fantastic so that I'd have a compelling reason to come back to Python full-time from Ruby. Not the case, I'm afraid. Looks like I'm going to be spending more time in the Ruby and Ruby on Rails camps in the future.

Ubuntu destop is so broken...

I've had it with Linux on the desktop.

Things that didn't work...
  • my office's Java based SSL VPN
  • flickr uploader
  • all sound and, therefore, dependent applications like Skype
  • and more...
To be fair, the issue is mostly with my computer's particular hardware. For example, my computer's NVidia card driver constantly pegged the processor at 25%. Not acceptable.

I am sad that Ubuntu on the desktop didn't work out for me. Oh well, back to XP for at least another year.

13 December 2008

So far, so good

The switch from XP to Ubuntu has gone just fine so far. There are a couple of things that I'd like to work out. One is that I can't seem to connect to my office's SSL based VPN which uses Java. I have the 1.6 JRE from Sun installed and configured but, when I connect to the office VPN, it just freezes up Firefox.

I also need to try the proprietary NVidia Geoforce graphics card driver. The screen transitions and scrolling with Firefox seem pretty jerky without it.

12 December 2008

Ubuntu - here we go again

So, I got a new laptop at work. It's a Dell Vostro 2510 with a 250gb 7200rpm SATA HD and 4 Gb RAM. It came downgraded to XP. I had installed Ubuntu to dual-boot with XP. That worked okay but for some reasons that I can't remember I wasn't quite satisfied and decided to remove Ubuntu. That happened on Monday.

After having settled for XP I started to set up my many applications. After a day or so I noticed that many fundamental programs were not working for me. That led to me conclude that I probably needed to wipe XP and reinstall from the source media that Dell provided.

Well, as long as I'm going to reinstall, why not put Ubuntu back on and force myself to use it for a week? Well, that's where I am now. I'm going to go Ubuntu whole hog for at least a week. The only thing that I know doesn't work is my soundcard. The graphics card will require a proprietary driver to do compiz. Even then, I think that the compiz didn't work quite right so I'll probably not get to use that. Small disappointment.

I hate that I have burned so much time on this laptop when I have more interesting things to be doing at work.

19 November 2008

PHP+IIS="No input file specified"

I was having frustrating problems with PHP on an IIS site complaining that "No input file specified." The frustration came from the fact that I am facile with Apache but know next to nothing about IIS. Ultimately, I came across this link which completely solved my problem.

05 November 2008

Narrative tests? Maybe but probably not.

There is a lot of buzz (probably always has been) for moving to narrative test frameworks and away from classic Unit Test frameworks such as Python's unittest and Ruby's Test::Unit. Having never tried a narrative test framework I'm partial to the classic frameworks for both Python and Ruby if for no other reason then they function identically so the framework is the same for both languages although the grammar is different.

Here's a few blog posts against narrative test frameworks. The first two give good reasons not to prefer them.

09 October 2008

Apache https proxying Mongrel with file upload

We had a Ruby on Rails form that did a POST of some form data including a file upload. The setup was Apache 2.2 proxying a bunch of mongrels on a Windows server. Port 80 was set to redirect to 443 via a rewrite rule. The proxy modules that we were using included proxy_http_module, proxy_balancer_module, and proxy_module. With that setup the file upload was failing. It appeared to timeout and not even hit the controller method that handled the POST.

We made one change to make the file upload work over https to proxied mongrels and that was to enable the proxy_connect_module.

17 September 2008

Python popen deadlocks

I have a python script that calls out to an external program using os.popen3(). That function returns file handle like objects for stdin, stdout, and stderr. I need to examine both stdout and stderr separately.

After running this script for a while I started to see it hang at times. This was especially true after I added in a parameter which increased the amount stderr. It turns out that there is a well known deadlock issue with the popen family of commands. Or, rather, it was not known to me until yesterday.

I started to tinker in my script using select.select() as a solution. That seemed a little foreign to my way of thinking so I changed directions and used separate threads for reading from the stderr and stdout file handles that os.popen2() returns. That seemed to do the trick so I'm pretty happy about that.

11 September 2008

Backups Reorganization pt. 10: Verification

I set up automated verification on the 4 backups servers on every backup job that they run. That simply involved the correct usage of the --verify option to rdiff-backup. This option calculates SHA1 checksums on files in the backups and compares them to the backups metadata. I don't put much stock in this process but feel that it is necessary to actually perform. Besides, the overhead is pretty low since the processing stays entirely on the backups server.

Rdiff-backup also sports a --compare-hash option in addition to --verify. The --compare-hash option actually calculates SHA1 checksums on the source server to compare to what is in the backups metadata. That seems nice but is probably going to be CPU intensive on the source server which I don't want. Still, I might just set it up as a weekly process to run in off hours. We'll see.

One thing that I learned is that Python < 2.4 doesn't support the decorator syntax in 2.4 and up. Rather than monkey with the small syntax differences and determining at runtime which Python version was executing the script, I just decided not to use decorators in one place where it would have been nice (but not necessary) to do so.

The next piece of this project will be to automate the test restores. That'll be slightly more tricky but ought to be satisfying to actually program up.

03 September 2008

Backups Reorganization pt. 9: Retrospect

I finally got around to addressing the backup of the one server that resides behind the firewall. I can't use the approach I had been using, that is, having the backups server initiate the backup because the backups server is itself outside of the firewall. The simplest thing to do is to backup the target server from within the firewall. As it turns out, my company has a couple of dozen terabytes of backup space inside the firewall all controlled by the Retrospect backup software. Setting up the job on the Retrospect server was easy and the initial backup of the target server is running now.