Strange Crash in OS X: securityd

A few days ago my Mac started having problems. I would be in the middle of some task when it would suddenly refuse to launch any new applications. Whenever I tried to launch any app, it would bounce a few times in the dock then exit.

As far as I could tell, any apps that were running when I got into this problem state would continue working fine. The OS would never completely freeze but I noticed that my CPU started being monopolized by CrashReporter. I tried killing that process but it would just immediately relaunch and peg the CPU again. I looked inside /Library/Logs/CrashReporter/ and saw that a new crash log was being created about every three seconds. The crash logs were for many different applications but none of the stack traces was useful. I had trouble spotting a pattern to what might trigger the problem.

Once my box was in the bad state I tried to ssh in to see if I could gather any useful information. SSH would prompt me for a password but it always denied access saying that I had entered an invalid password.

The only way out of this state was to restart the machine. When I tried to reboot, OS X would successfully log out but then get stuck on a blue screen. I would see the indeterminate NSProgressIndicator for a few seconds then it would disappear for a few seconds then come back again. I was forced to power cycle the machine.

I finally noticed that when this problem occurred, the first crash log was always for securityd. /Library/Logs/CrashReporter/securityd_2009-03-23-204700_macpro.crash:

Process:         securityd [22]
Path:            /usr/sbin/securityd
Identifier:      securityd
Version:         ??? (???)
Code Type:       X86 (Native)
Parent Process:  launchd [1]
Date/Time:       2009-03-23 20:47:00.211 -0600
OS Version:      Mac OS X 10.5.6 (9G55)
Report Version:  6
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000001000000
Crashed Thread:  0

This information finally led me to the solution.

The problem was that securityd would crash then any app that needed to authenticate was unable to do so. One newsgroup noted that the problem could be temporarily solved by relaunching the process:

$ launchctl load /System/Library/LaunchDaemons/

After a bit more searching I found a permanent answer in a mailing list archive: Keychain access crashing on SecKeychainFindGenericPassword. The solution was incredibly simple (and completely unintuitive). I had to remove the file /var/db/CodeEquivalenceDatabase and reboot. That’s it!

The thread offers more details but basically, “that file [/var/db/CodeEquivalenceDatabase] has gotten corrupted and runs securityd into an endless memory-eating loop that (usually) ends up running your system out of memory and into the ground.”

Testing Routes in Rails

Routes in rails are really cool but they can be confusing to newbies and, as they become more complex, it can be difficult to make sure that all of your paths are still working like you expect. My intention here is not to provide a primer on routes but rather to show how easy it is to test and verify routes.

As you define new routes it is important to write tests to validate url formats. You need tests to verify that urls are mapped to the right controller and action:

                 :controller => "dashboard",
                 :action => "directory_stations")

And you need tests to verify that controller/action combinations can be correctly mapped back to a url:

assert_recognizes({ :controller => "dashboard",
                    :action => "directory_stations" },

The console is your friend when you are trying to test or debug a tricky mapping. Launch the console and, to save some typing, assign a local variable:

$ script/console
>> r = ActionController::Routing::Routes

If you want to see currently defined routes:

>> puts r.routes
ANY /:controller/:action/:id/ {}
ANY /:controller/:action/:id.:format/ {}
=> nil

The first part of the output tells you what HTTP verb the routing applies to, ANY in this case. Other possible values are GET, POST, HEAD, etc. The next element in the output is the pattern that matches the route. The empty braces show that we don’t have any of the optional parameters defined that might modify the behavior of the route.

To verify that urls are mapped to the right controller and action use recognize_path:

>> r.recognize_path "/station/index/42.html"
=> {:controller=>"station", :action=>"index", :format=>"html", :id=>"42"}
>> r.recognize_path "/station/index/42"
=> {:controller=>"station", :action=>"index", :id=>"42"}
>> r.recognize_path "/station/index"
=> {:controller=>"station", :action=>"index"}
>> r.recognize_path "/station"
=> {:controller=>"station", :action=>"index"}

To verify that controller/action/id/etc. combinations correctly map back to a url:

>> r.generate :controller => :station
=> nil
>> r.generate :controller => :station, :action=> :index
=> nil
>> r.generate :controller => :station, :action=> :index, :id=>42
=> nil
>> r.generate :controller => :station, :action=> :index, :id=>42, :param => "xyzzy"
=> nil