The battle of the iOS crash reporters

When I released my first iOS app, I noticed the crash reports section in iTunes Connect. I naively assumed I would get any crashes reported to me through this interface. When my app never had any, I thought it was never crashing!

Boy, was I wrong.

It turns out that crash reports are only submitted by users who agree to do so. I honestly don’t remember ever being asked this, so I don’t even know if I opted in. This makes me think that most users probably don’t. Next, the user has to sync their device to a computer in order to report the crash. Yeah, most iOS users don’t bother with this. Even worse, only crashes that occur a certain number of times will actually be reported to you through ITC.

The moral of the story? ITC crash reports are completely, utterly useless. Odds are, you won’t even get any.

Clearly this isn’t the way. I’m a firm believer in thorough analytics, and my highest priority is rich crash and exception reporting.

I probably don’t need to tell you that your app does crash, and you won’t see most of the crashes during testing. The reasons for this vary from device and iOS differences, to network conditions, to arcane multithreading issues, to memory management errors. You need a way to find those crashes (and more importantly, be able to symbolicate them) in the wild.

OK, hopefully I’ve convinced you. So what to do? You can and should roll your own crash reporting in your apps. I tested out three possibilities. Let’s dig in.

Want the short version? Use HockeyApp.

Hoptoad

http://www.hoptoadapp.com
Pricing: Free for one app, $5-45/month beyond that
Features: Crash reporting (raw stack trace only)
Pros: Cheap, Github and Lighthouse integration, RSS support
Cons: No symbolication, clumsy app version support, company is rooted in Ruby and iOS is a secondary thought

Hoptoad was originally intended for Ruby applications, and you can tell. It has some clever features like Github tracking (where you say in your commit message that you fixed a crash, and Hoptoad marks it fixed automatically), and Lighthouse integration (I don’t use Lighthouse, but integrating your crash reporter with an issue tracking tool is a great idea). Unfortunately, the focus on Ruby shows when it comes to using this for iOS.

My current shipping version of War of Words integrates Hoptoad. As with all three of these examples, integration just takes adding the library to your project, plus one line of code. Hoptoad was a step in the right direction, but it has several fatal flaws:

• The biggest problem was version support. All crashes reported by all versions of my app get lumped into one big pot. So although I may have fixed a crash in a later version, I’ll still get hundreds of them populating my crash list. (Note: Hoptoad has supposedly solved this issue by allowing you to accept crashes from just one version of your app. I cancelled my account before I could test this out.)
• The list of crashes can’t be sorted
• Crashes are poorly grouped. This might be a problem with other services too, but I was getting one single crash type reported under a bunch of separate groups.
• I couldn’t get symbolication to work. You have to run atos manually for every line you wish to symbolicate, and I could never get it to work. I’m not an atos expert, but I think I was doing it right. Once I tried HockeyApp (below), I realized just how well this could work, but doesn’t.
• I engaged with tech support several times, and although they responded very quickly, it quickly became apparent that their iOS support was playing second fiddle to their Ruby support. It took them a few months to add app version filtering, and when they did, it was a half-assed implementation (and the dev who implemented it was a rude guy who didn’t seem to understand the importance of the feature).

QuincyKit

http://quincykit.net/
Pricing: Self hosted on your own PHP/MySQL server
Features: Crash reporting (can be auto-symbolicated if you set it up properly)
Pros: Cheap
Cons: Very bare-bones user interface, completely self-hosted means more work on your end (I didn’t test version support)

QuincyKit is an open source crash reporting tool that uses PLCrashReporter, also open source. The whole kit and kaboodle is free, assuming you have access to a PHP/MySQL server and know how to use it. Free, though, means more work for you. This sort of thing might appeal to the open source junkies out there, but I’m a solo developer and a firm believer in doing the minimum amount of work possible for a high quality result. This means I’m not going to waste any time on tools if I don’t have to.

The UI on the server side is pretty bad, which may not matter to you. Just thought I’d mention it. You could certainly do your own UI if you wanted.

QuincyKit is cool because it will supposedly symbolicate your crash reports for you, if you set it up. Setting it up involves a fair bit of work, though, and it wasn’t worth the effort for me.

I won’t spend a great deal of time focusing on QuincyKit, because it’s the same library that’s at the heart of the next tool: HockeyApp.

HockeyApp

http://www.hockeyapp.net/
Pricing: $20-100/month (free 30 day trial)
Features: Beta testing, crash reporting, seamless auto symbolicating, status feedback to users
Pros: Rich version support, auto symbolication that works, great UI
Cons: Expensive

Now that is a beautiful stack trace.

They say you get what you pay for in life, and nowhere is this more true than with HockeyApp. It’s definitely the most expensive option listed here, starting at $20 per month after the free 30-day trial. But it does everything right, and it does a whole lot more than just crash reporting.

(Just to be clear: HockeyApp is created by the same person who made QuincyKit. In fact, the QuincyKit library is required in your app in order to use HockeyApp. HockeyApp is basically the icing on the cake.)

HockeyApp’s big claim to fame is it’s beta testing ecosystem, offering a similar experience to the much-lauded TestFlight. My beta tests have been small scale so far, and I haven’t had an opportunity to really test out TestFlight or HockeyApp when it comes to this sort of thing. The focus of this post is on crash reporting. Just be aware that the reason HockeyApp charges so much more than the others is likely due to this feature.

OK, so how good is it? The short answer is very good. The single best killer feature is auto symbolication. You specify a version and build number (which lets you differentiate between different builds of the same version), and upload your dSYM file through the web interface. That’s it — your crash logs will be symbolicated when you see them. Not only that, but they’re super readable and a joy to look at. It’s a huge improvement from Ye Olde Xcode crash logs. This auto symbolication thing is worth the $20 to me alone.

It doesn’t stop there. Once you’ve fixed a crash, you can mark it as fixed on the web site. Users of your app who report that crash down the road will be told that the crash has been fixed, and that it’s either submitted to Apple or already available in the next update. This stuff is worth it’s weight in gold. You can also elect to stop receiving crash reports from any version of your app on a case-by-case basis, far outstripping the capabilities of Hoptoad’s version support.

Oh, and the UI is great.

Once you experience the HockeyApp brand of crash reporting, you’ll realize just how spoiled you’ve become. You’ll never want to go back to the dark ages of Hoptoad and it’s ilk. Welcome to the future.

Other alternatives

As people have pointed out in the comments below, there are other alternatives for crash reporting (and beta testing). I haven’t had any real-world experience with these, so I didn’t want to write them up. There are probably many more out there; this isn’t an exhaustive list.

Services

• Crittercism
• Apphance
• Exceptional (with the iOS API wrapper)
• BugSense

Libraries
You’ll have to roll your own web service and user interface to handle crashes coming from these libraries.

• PLCrashReporter (remember QuincyKit and HockeyApp already use this)
• CrashKit
• CrashReporter