Strange Crash in OS X: securityd

A few days ago my Mac started having problems. I would be in the middle of some task when it would suddenly refuse to launch any new applications. Whenever I tried to launch any app, it would bounce a few times in the dock then exit.

As far as I could tell, any apps that were running when I got into this problem state would continue working fine. The OS would never completely freeze but I noticed that my CPU started being monopolized by CrashReporter. I tried killing that process but it would just immediately relaunch and peg the CPU again. I looked inside /Library/Logs/CrashReporter/ and saw that a new crash log was being created about every three seconds. The crash logs were for many different applications but none of the stack traces was useful. I had trouble spotting a pattern to what might trigger the problem.

Once my box was in the bad state I tried to ssh in to see if I could gather any useful information. SSH would prompt me for a password but it always denied access saying that I had entered an invalid password.

The only way out of this state was to restart the machine. When I tried to reboot, OS X would successfully log out but then get stuck on a blue screen. I would see the indeterminate NSProgressIndicator for a few seconds then it would disappear for a few seconds then come back again. I was forced to power cycle the machine.

I finally noticed that when this problem occurred, the first crash log was always for securityd. /Library/Logs/CrashReporter/securityd_2009-03-23-204700_macpro.crash:

Process:         securityd [22]
Path:            /usr/sbin/securityd
Identifier:      securityd
Version:         ??? (???)
Code Type:       X86 (Native)
Parent Process:  launchd [1]
Date/Time:       2009-03-23 20:47:00.211 -0600
OS Version:      Mac OS X 10.5.6 (9G55)
Report Version:  6
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000001000000
Crashed Thread:  0

This information finally led me to the solution.

The problem was that securityd would crash then any app that needed to authenticate was unable to do so. One newsgroup noted that the problem could be temporarily solved by relaunching the process:

$ launchctl load /System/Library/LaunchDaemons/

After a bit more searching I found a permanent answer in a mailing list archive: Keychain access crashing on SecKeychainFindGenericPassword. The solution was incredibly simple (and completely unintuitive). I had to remove the file /var/db/CodeEquivalenceDatabase and reboot. That’s it!

The thread offers more details but basically, “that file [/var/db/CodeEquivalenceDatabase] has gotten corrupted and runs securityd into an endless memory-eating loop that (usually) ends up running your system out of memory and into the ground.”

Thoughts on Doing Contract Work as a Software Developer

As I was recently looking for new employment I spent quite a bit of time deciding wether I might enjoy doing contract work full-time. I enjoy working on different projects and learning new things but there is one major roadblock to becoming a full-time contract developer. My personality doesn’t let me write software that is anything less than my best.

WARNING: Gross generalizations and simplifications below. I’m not trying to offend anyone, just describe my experiences.

Most of the contract work I’ve done has been for people who are not technically savvy. They come to me with a very vague idea of the software they want. They expect me to tell them how much it will cost before we’ve discussed specific requirements. When the requirements are incomplete or incorrect they expect that I’ll just fix it without additional cost to them.

Not all of my contract experience has been negative. In fact, most of it has worked out quite well. Usually both my client and myself are pleased with the software and the cost of building it. But I’ve had enough negative experiences to be careful when considering a new job.

Part of the problem is that it is nearly impossible for anyone to completely define the scope of a project. There is always some miscommunication or misunderstanding, there is always some unforseen problem.

“You want me to setup a blog for your company? No problem, I can get WordPress setup for you in an hour.”

“Wait, I didn’t realize that by ‘blog’ you meant store front application that can accept payments, handle accounts payable, accounts receivable and inventory tracking. That will take slightly more than an hour.”

That kind of situation actually isn’t bothersome to me. As a contractor it is part of my job to understand what you want before making a bid. If a potential client obviously doesn’t know what they want, I can either decline to bid on that job or I can adjust my bid to account for a large amount of unknown. I don’t love it, but that type of risk is manageable.

The part of contract work that I dislike is being forced to compromise quality. When I’m working on a fixed cost contract, it is in my best interest to deliver exactly what is specified, as quickly as possible. As long as my client is reasonably happy with the deliverable, I am going to get paid $20k regardless of whether it took two days, two weeks or two months to create. I don’t get additional money for clean code. I don’t get extra for having good test coverage.

When I complete a project more quickly than I had anticipated there is no problem. I can spend time verifying that the code is tight and that everything is working as expected. But if I am running behind schedule, it becomes more difficult to care about testing the code or fixing “little” bugs.

There may be a bug in the code where order totals aren’t calculated correctly, but what are the odds that my client will notice the bug before he signs off on the project? If he does find the problem and I correct it, will he think to test for that same bug in every release?

This is the dilemma that makes contract work difficult for me. If I see a bug in my code, I’m going to fix it. If I’m writing a tricky or important calculation (like calculating totals), I’m going to write a test. I need to have confidence that my code is doing what I expect. I’ve never shipped any software that didn’t have a list of known bugs but I have also never shipped any software in which I didn’t have a high level of confidence that it was working correctly.

For me, doing the bare minimum isn’t an option for two reasons:

  1. Quality is extremely important to me. I can’t just hack something together that meets the contract requirements. When I write software, I want to deliver my personal best.
  2. Most of the time, the fast/crappy way of implementing something simply doesn’t occur to me.

I understand a company’s need to understand cost before approving custom softare. But if you want me to do contract work, pay me on an hourly basis. I’ll give you a projected timeline for project completion.

With an hourly rate, you only pay me for the time I actually spend working. With an hourly rate I know that I won’t lose money just because I insist on high-quality code. We’ll both be happier in the long run.