Strange Crash in OS X: securityd

A few days ago my Mac started having problems. I would be in the middle of some task when it would suddenly refuse to launch any new applications. Whenever I tried to launch any app, it would bounce a few times in the dock then exit.

As far as I could tell, any apps that were running when I got into this problem state would continue working fine. The OS would never completely freeze but I noticed that my CPU started being monopolized by CrashReporter. I tried killing that process but it would just immediately relaunch and peg the CPU again. I looked inside /Library/Logs/CrashReporter/ and saw that a new crash log was being created about every three seconds. The crash logs were for many different applications but none of the stack traces was useful. I had trouble spotting a pattern to what might trigger the problem.

Once my box was in the bad state I tried to ssh in to see if I could gather any useful information. SSH would prompt me for a password but it always denied access saying that I had entered an invalid password.

The only way out of this state was to restart the machine. When I tried to reboot, OS X would successfully log out but then get stuck on a blue screen. I would see the indeterminate NSProgressIndicator for a few seconds then it would disappear for a few seconds then come back again. I was forced to power cycle the machine.

I finally noticed that when this problem occurred, the first crash log was always for securityd. /Library/Logs/CrashReporter/securityd_2009-03-23-204700_macpro.crash:

Process:         securityd [22]
Path:            /usr/sbin/securityd
Identifier:      securityd
Version:         ??? (???)
Code Type:       X86 (Native)
Parent Process:  launchd [1]
Date/Time:       2009-03-23 20:47:00.211 -0600
OS Version:      Mac OS X 10.5.6 (9G55)
Report Version:  6
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000001000000
Crashed Thread:  0

This information finally led me to the solution.

The problem was that securityd would crash then any app that needed to authenticate was unable to do so. One newsgroup noted that the problem could be temporarily solved by relaunching the process:

$ launchctl load /System/Library/LaunchDaemons/

After a bit more searching I found a permanent answer in a mailing list archive: Keychain access crashing on SecKeychainFindGenericPassword. The solution was incredibly simple (and completely unintuitive). I had to remove the file /var/db/CodeEquivalenceDatabase and reboot. That’s it!

The thread offers more details but basically, “that file [/var/db/CodeEquivalenceDatabase] has gotten corrupted and runs securityd into an endless memory-eating loop that (usually) ends up running your system out of memory and into the ground.”