HockeyApp: still the best thing since sliced pizza
Sliced bread is good, but pizza is where it’s at. It’s the same with HockeyApp: once you try it, you’ll be hooked for life.
I previously wrote about HockeyApp as part of a crash reporter mini-showdown. If you haven’t read that yet, you probably should.
Since then, I’ve been using HockeyApp extensively for alpha and beta testing, and it seriously kicked my testing process up to a whole new level.
What in the heck is a Hockey App?
Yes, the name is a bit weird. Apparently it’s a play on words based on the concept of an “ad hoc” build. You can read some info about the company’s team here.
HockeyApp is two things: a service to enable beta testing of apps on iOS/Android/MAS, and a service to acquire crash reports from clients.
You get both of those services at once by signing up for an account. There isn’t any free account, but the $10 Indie account is pretty generous.
Crash report extravaganza
Your clients crash far more frequently than you think:
Yes, really.
Your job is to figure out why. HockeyApp uses the super powerful PLCrashReporter library to make this happen, and it uploads the crashes to the HockeyApp server.
The real magic happens after the upload—HockeyApp instantly symbolicates your crash and gives you a stack trace. You see all your crashes grouped by type. Your client can submit some user information with each crash. For example, I send my user’s Game Center ID and nickname, as well as whether the user has advertising on or off in my app.
So many crashes, so little time…
This is a lovely UIKit crash that I’m not aware of any fix for. Sweet!
You can mark crashes as Resolved or Ignored, or you can send them to a third-party issue tracking tool. HockeyApp integrates with a ton of them, including Jira, Assembla, Bugshelf, Codebase, Fogbugz, Github, Lighthouse, Mantis, Redmine, Trac, Unfuddle, BaseCamp, and Pivotal Tracker.
If you’ve ever tried to manually symbolicate crash reports (or use Xcode to do it), you’ll instantly appreciate just how easy HockeyApp makes it.
I use HockeyApp crash reporting in all my test and App Store builds. It can be a bit disconcerting seeing the immense variety of strange and arcane ways your app crashes in the real world, but I guess that’s better than being blissfully ignorant!
Your own personal App Store
Let’s talk about beta testing. The old fashioned way was to create an ad-hoc build with a special entitlements file, manually zip up your app bundle with a provisioning profile and a specific folder layout, rename it to .ipa, email it to your testers, and give them detailed instructions on how to drag the IPA file to iTunes and sync it to their device. Said instructions had to be replete with corner cases and instructions for if things went wrong (which they usually did). It was, to say the least, a total pain in the ass.
(Yes, Xcode was supposed to help with some of those steps in the early days, but in reality I could never get it to do anything remotely helpful.)
Luckily, with iOS 4.0, Apple introduced over-the-air distribution of ad-hoc builds. The technique wasn’t in and of itself very easy to use, but it provided the key building blocks to enable a whole new class of services, including HockeyApp.
Beta testing with HockeyApp is easy. First, integrate the SDK (instructions here). Then, in Xcode 4, create a new Scheme for Distribution, and point it at an ad-hoc provisioning profile that you created using Apple’s developer site (this provisioning profile needs to contain all of your tester’s devices). Archive your app using Xcode, then upload the .ipa and .dsym files to HockeyApp. (There’s an uploader app that will do this last step for you, which I’ll discuss in the next section.)
Your app, when enabled for beta testing, now gets a public download page.
The download page on a desktop computer.
The download page on an iPhone.
I like to use a URL shortener like bit.ly to create a more memorable URL to my download page, then hand that to my testers. Tapping the install button gives you a prompt like this:
Click Install again, and it installs just like any app from the App Store:
The only thing that can go wrong at this point is if you did your provisioning profile badly (not including this user’s device, or accidentally using an App Store profile instead of an Ad Hoc profile).
Cool! The app is installed and ready for testing!
No, really, this is better than pizza
Testing is going great, and now it’s time for you to distribute a new build to your testers. It gets even easier now.
You build and upload the app as before. But your testers immediately get a prompt right inside your app that a new build is available:
Tapping Show lets you see the changelog for this build. You can then install the update right from within the app. It’s really slick.
You can even mark updates as mandatory, forcing 100% adoption of updates throughout your test cycle. Awesome!
You also get stats on the testing. How many downloads did you get? How many hours of testing have you accumulated per build? HockeyApp will tell you:
I’ve only got one build in testing right now, so this graph is a bit lopsided.
And of course, you’re also collecting crashes from these test builds.
There’s a lot more you can do with beta testing, including:
- Public recruitment page (users can sign up to test your app, and you can accept/reject them as needed).
- Users can create accounts to be notified by email when a new build is ready for testing.
- You can create “managers”, which are users who can administrate the nuts and bolts of your HockeyApp apps.
- You can invite users to test your app.
- There’s an API that you can leverage to do probably everything you’ve seen so far, programmatically.
The kitchen sink
As if this weren’t cool enough already, there’s one more thing that makes the HockeyApp ecosystem a breeze: the HockeyMac uploader application. You can add an item to your Xcode Scheme that launches HockeyMac, ready to automatically upload your .dsym and .ipa files. You can throw in your changelog right there, hit a button, and away you go. There is no need to visit the HockeyApp web site at all.
Conclusion
Here’s the deal: you should be using HockeyApp. There are a lot of competitors out there, most famously TestFlight. I haven’t personally used TestFlight, but I did try a few other crash reporters. HockeyApp gets crash reporting really, really right. The beta testing workflow is literally everything I could ever ask for, so I can’t recommend anything else.
Recently I’ve been seeing more and more iOS developers on Twitter who are switching away from whatever crash/beta solutions they were using, and changing over to HockeyApp. I’m not sure what’s driving this exodus, but comments so far indicate that HockeyApp is a better solution for the majority of developers.
The guys at Hockey are really responsive to support issues. (It seems like they never sleep, honestly!) If you have a suggestion, they’re really open to hearing it. They’ve implemented every suggestion I’ve ever thrown their way so far, and I imagine that’s the case for just about everyone.
And the best part: they aren’t some new highly funded VC flavor-of-the-month. It’s just a small team of guys who love what they do and want to provide the best service possible for developers.
And they nailed it.
(Note: I am not a spokesperson for HockeyApp or Bit Stadium, nor was I in any way compensated for writing this post. One of the HockeyApp developers did reach out to me to suggest a follow-up article, but that’s it. I’m just a big fan!)
Monitoring your app using a dashboard
Last year I posted a thumbnail image of part of my in-house dashboard, which I use to monitor my games and apps. Someone asked for more details, and I’m finally getting around to writing something up.
Before I dig in, let’s take a look at a full-size, unaltered screenshot of my War of Words Apocalypse dashboard, as it stands today. This is how it looks at the conclusion of my beta test, so obviously most of the numbers are pretty low. (Click on the thumbnail for the full-size version. It’s a very tall image, but not a large download.)
Why have a dashboard?
All of my apps include some form of server-side component. In the case of the War of Words series, this is a fairly involved infrastructure (albeit quite standard as these things go). I wanted a measure of control over all my metrics, and I also wanted persistent storage so I could go back and look at trends as needed. Therefore, instead of relying on a third party like Flurry or Amazon CloudWatch, I rolled my own proprietary platform.
The disadvantages: more time spent working on tracking and writing dashboards, database storage and CPU usage dealing with the metrics, and a relatively slower dashboard refresh time. The advantages: no vendor lock-in (to prove it: I recently migrated my entire platform from Rackspace Cloud Sites to Amazon Web Services), ultimate control over exactly how and what I track, and totally custom dashboards.
I’ve done a lot of reporting and charting over the years, so creating dashboards comes naturally to me.
The ultimate purpose of having this dashboard is fourfold:
- Monitor server performance to notice problems immediately, and to track exceptions as they happen.
- Watch the app’s sales and utilization in a much more detailed way than just looking at sales numbers. You can also identify trends such as what time of day there is the most activity.
- When making changes on the server, monitor before-and-after results in terms of performance.
- The dashboard includes a full suite of support tools that let me fix issues with user and game data.
Where does the data come from?
The data driving the dashboard comes from four sources.
Application server logs stored in SimpleDB
The vast bulk of information comes from a proprietary system I developed within each of my application servers. Every single request to the server gets logged into memory, including the type of request, how long the request took, whether the request was successful, etc. I also log things like News article hits, number of house ad clicks, number of cache hits and misses (at the logical layer instead of at the Memcached layer), etc. Every 10 minutes, a background thread aggregates all this information up and spits out a single very wide row to a SimpleDB domain dedicated to the task.
On the server, I’ve done everything I can to minimize the overhead from this logging. Everything uses hash tables (Dictionaries in C#), and the aggregating and storage of the data is handled by a background thread. Client requests are never blocked for logging purposes.
The dashboard’s primary job is to pull this data down from the activity domain. I store all the raw data in a local SQL Server database for later use. The 10 minute granularity helps keep load and storage size down while still providing enough granularity for me to get a clear picture of what’s happening.
If an exception occurs on the server, I immediately log it to another SimpleDB domain. I include a stack trace and everything else about the exception. When I have any exceptions in the domain, a nice big red number appears at the top of the dashboard, instantly drawing my attention. Once I’ve resolved the exceptions, I can hit a button to clear them out of the domain.
Similarly, push notification failures are also logged to their own domain. In practice, I only see failures logged when a push SSL certificate has expired (this happens on my debug certificates, but people are almost never running debug builds so it’s not an issue).
Amazon CloudWatch
I monitor CPU usage, memory usage, cache performance, and load balancer latency via CloudWatch. The Amazon Web Services .NET SDK makes it very easy to do this, and I don’t bother to store this data for posterity. (I do store the record high or low for each metric—this let’s me keep track of the highest or lowest value I’ve ever seen.)
Local SQL Server database
If I click the “Full Update” button at the top, the dashboard goes into a 5-minute-long update cycle where it pulls a great deal of information down from all ~200 of my SimpleDB shards and stores the information in a local SQL Server database. This allows me to run more complicated queries against the data, including the distribution of device types, operating system versions, etc. Because this takes so long, I only do this every few days (more frequently during a launch month). Without the Full Update mode, the dashboard takes about 25 seconds to pull on average.
Other third parties
Greystripe, one of my advertisers, has an API which developers can use to pull information on their performance. In the screenshot above, the War of Words Apocalypse app hasn’t been approved by Greystripe yet, so you can’t see the metrics. There are a set of charts that would normally go there, exposing my impressions, daily average CPM, clicks, and revenue.
How is the dashboard created?
The look-and-feel of the dashboard is primarily provided by Telerik’s ASP.NET AJAX component library. This includes the charts. All the numbers and text are just hundreds of tables and labels. It ends up being thousands of lines of somewhat messy code, but it serves it’s purpose.
I actually host the dashboard on one of my computers at home. This lets me skip the messy deploy process to an external server, and just seems easier at the moment since I’m a one-person operation. If I ever grow, I’m sure I’ll end up putting it out on my front-end web server.
Grabbing the data is simply a case of using the Amazon Web Services .NET SDK. I already use this SDK for all of the server-side operations, so it’s a piece of cake. Aggregating the values up properly for each chart can get somewhat tricky, but it’s just an algorithmic exercise any programmer can manage.
What else can it do?
I’ve also used this dashboard as a jumping-off point for my tech support tools. (My customer-facing support tools are through Zendesk.) This lets me do the following:
- Examine a specific player’s data, including their Game Center ID, their list of games, all their statistics, etc.
- Modify a player’s statistics.
- In the case of Apocalypse, I can manually grant or remove anything from the Store for a player (including the advertising disable). I can also change a player’s level.
- Look at any game in the system. This includes all tiles in play, all tiles that have been played in the past, the supply of tiles and players’ racks, etc.
- View the chat in any game.
- In Apocalypse, view the complete Combat Log for any game.
- Delete games completely from the system.
- Perform additional “consistency” operations on a game. This helps ensure that the various database shards are in agreement about a game’s state.
- Create a new News topic, which gets handed down to the clients within about an hour. The News articles include a button which can either point to a web site, or initiate any in-app purchase. The Apocalypse client is smart enough to notice future IAPs or IAPs for expansion packs that don’t exist yet, and prompt the user to update their game from the App Store first.
- Modify my advertising waterfall approach, including enabling/disabling specific advertisers. This can take a few days to get out to the clients because it requires a restart of the client.
- Modify my house ads, including the creatives and links, as well as how often they appear.
What can’t the support tools do?
- I can’t grant or remove achievements from players.
- I can’t modify leaderboards or manually report scores to leaderboards.
- I can’t yet ban players or let one player blacklist another player. This is recently becoming more of an issue as certain abusive players enter the system, and I’ll be working on this.
Dashboards aren’t the entire solution
As great as charts are, I’m not looking at my dashboards 100% of the time. That’s where Amazon CloudWatch alarms come in handy. This is one of the single greatest reasons to switch off of Rackspace over to AWS. For example, here are some of the alarms I have setup:
- If any one server in my application cluster goes down (fails it’s health check) for 5 minutes, I get an email and a text message.
- If the load balancer maintains a high latency for 3 minutes or more, I get an email and SMS.
- If one of my application servers experiences 90% or higher CPU activity for 10 minutes solid, I get an email and SMS. This has proved extremely helpful during development, since it emailed me when I accidentally introduced a runaway thread that caused the worker process to spin into infinity and block the entire server.
- After this post was originally written, I also added the ability to reboot any server or the cache cluster remotely via the dashboard. This really comes in handy when you’re out to dinner and you get a text that your server is down. Reboot!
In Rackspace land, I didn’t know my servers were down until somebody told me (usually on Facebook). At that point, it’s a PR disaster in the making. This way, I get an email proactively telling me about the problem (assuming I’m awake). Alarms are your best friend. Use them well.
Conclusion
Hopefully this post helped you get an idea of what types of metric you could consider tracking for your products. It also illuminated my methodology for tracking metrics, as well as the reasons behind them.
When I’m not actually playing my games, I tend to live in the dashboard. It’s great to be able monitor every aspect of the operation, and it’s critical in order to manage scaling as the game grows. It’s also helpful when the server throws exceptions, allowing me to quickly notice the problem and get it solved.
It was certainly a lot of work to create, but it has grown organically over the years as I became interested in tracking different things. The technology on the server is very reusable—the same code is being used by all my products.
IAP receipt verification on your server using C#
After an in-app purchase is made in iOS, you need to call your web service to validate the receipt data. This extra step, while cumbersome, is necessary to prevent hackers and cheaters from faking the purchase.
I know that most iOS developers probably don’t use C# on their backend, but there must be some out there. Today I was searching for some code that would validate an in-app purchase receipt with Apple using C#. I came up empty, so I rolled my own, and thought I’d share it in case it’s helpful to others. I’ll show you part of the Objective-C client-side, and the C# server-side. I’m not going to show you how to do the actual IAP transaction. For that, I recommend checking out this tutorial.
Before we start, let me make sure you’ve seen Apple’s official documentation on the subject.
Prerequisites
My server-side code uses JSON.NET, which can be found here: http://json.codeplex.com/
Client Side
After we’ve completed the transaction, your code has an SKPaymentTransaction object. You need to call your web service with the item ID that was purchased, as well as the Base64 encoded receipt data. I recommend doing the encoding on the client. Here are some NSString categories that will do the job.
[sourcecode language="objc"]@implementation NSString (WolfAdditions)
static const char _base64EncodingTable[64] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+ (NSString *)encodeBase64WithString:(NSString *)strData
{
return [NSString encodeBase64WithData:[strData dataUsingEncoding:NSUTF8StringEncoding]];
}
+ (NSString *)encodeBase64WithData:(NSData *)objData
{
const unsigned char * objRawData = [objData bytes];
char * objPointer;
char * strResult;
// Get the Raw Data length and ensure we actually have data
int intLength = [objData length];
if (intLength == 0) return nil;
// Setup the String-based Result placeholder and pointer within that placeholder
strResult = (char *)calloc(((intLength + 2) / 3) * 4 + 1, sizeof(char));
objPointer = strResult;
// Iterate through everything
while (intLength > 2) { // keep going until we have less than 24 bits
*objPointer++ = _base64EncodingTable[objRawData[0] >> 2];
*objPointer++ = _base64EncodingTable[((objRawData[0] & 0×03) << 4) + (objRawData[1] >> 4)];
*objPointer++ = _base64EncodingTable[((objRawData[1] & 0x0f) << 2) + (objRawData[2] >> 6)];
*objPointer++ = _base64EncodingTable[objRawData[2] & 0x3f];
// we just handled 3 octets (24 bits) of data
objRawData += 3;
intLength -= 3;
}
// now deal with the tail end of things
if (intLength != 0) {
*objPointer++ = _base64EncodingTable[objRawData[0] >> 2];
if (intLength > 1) {
*objPointer++ = _base64EncodingTable[((objRawData[0] & 0×03) << 4) + (objRawData[1] >> 4)];
*objPointer++ = _base64EncodingTable[(objRawData[1] & 0x0f) << 2];
*objPointer++ = ‘=’;
} else {
*objPointer++ = _base64EncodingTable[(objRawData[0] & 0×03) << 4]; *objPointer++ = ‘=’; *objPointer++ = ‘=’; } } // Terminate the string-based result *objPointer = ‘\0’; // Return the results as an NSString object NSString *s = [NSString stringWithCString:strResult encoding:NSASCIIStringEncoding]; free(strResult); return s; } + (NSData *)decodeBase64WithString:(NSString *)strBase64 { const char * objPointer = [strBase64 cStringUsingEncoding:NSASCIIStringEncoding]; int intLength = strlen(objPointer); int intCurrent; int i = 0, j = 0, k; unsigned char * objResult; objResult = calloc(intLength, sizeof(char)); // Run through the whole string, converting as we go while ( ((intCurrent = *objPointer++) != ‘\0’) && (intLength– > 0) ) {
if (intCurrent == ‘=’) {
if (*objPointer != ‘=’ && ((i % 4) == 1)) {// || (intLength > 0)) {
// the padding character is invalid at this point — so this entire string is invalid
free(objResult);
return nil;
}
continue;
}
intCurrent = _base64DecodingTable[intCurrent];
if (intCurrent == -1) {
// we’re at a whitespace — simply skip over
continue;
} else if (intCurrent == -2) {
// we’re at an invalid character
free(objResult);
return nil;
}
switch (i % 4) {
case 0:
objResult[j] = intCurrent << 2; break; case 1: objResult[j++] |= intCurrent >> 4;
objResult[j] = (intCurrent & 0x0f) << 4; break; case 2: objResult[j++] |= intCurrent >>2;
objResult[j] = (intCurrent & 0×03) << 6;
break;
case 3:
objResult[j++] |= intCurrent;
break;
}
i++;
}
// mop things up if we ended on a boundary
k = j;
if (intCurrent == ‘=’) {
switch (i % 4) {
case 1:
// Invalid state
free(objResult);
return nil;
case 2:
k++;
// flow through
case 3:
objResult[k] = 0;
}
}
// Cleanup and setup the return NSData
NSData * objData = [[NSData alloc] initWithBytes:objResult length:j];
free(objResult);
return objData;
}
@end[/sourcecode]
Armed with those categories, it’s easy to encode the receipt data:
[sourcecode language="objc"]SKPaymentTransaction *t; // Assuming you’ve obtained your completed transaction object
NSString *receiptStr = [NSString encodeBase64WithData:t.transactionReceipt];[/sourcecode]
Use that to call your web service. I’m not going to show you how to do that. I use a modified version of Sudz-C to communicate with my WCF service via SOAP.
Server Side
On the server, let’s assume your web service knows the item ID you purchased via StoreKit, as well as the Base64 encoded receipt data. Below is most of the code I use to verify the receipt with Apple. You can do this a few different ways, but I chose to check that a) the receipt is valid, b) the product ID matches what I expected, c) the application who sold the item matches what I expected, and d) the transaction occurred within 24 hours +/- of now. I give it that much leeway because I don’t care that much, and I want to avoid any possible time zone glitches.
Why the above checks? Because one way IAP cheaters operate is by capturing a valid IAP receipt, and re-using that receipt over and over. We could be getting a valid receipt from some other app, or a receipt for some other product in our app, or a receipt that happened months ago. We can avoid all those things with the additional checks.
OK, here’s the code. It’s not commented well, but basically it’s doing a JSON POST, then getting the HTTP response and parsing that using JSON.NET.
There are all kinds of little “gotchas” that hit me as I worked through this. I’m sorry for not walking you through all of them; I’ll leave it up to you to work through what’s going on here.
Hope it helps!
[sourcecode language="csharp"]
public static string[][] PurchaseItem(string itemID, string receiptData)
{
try
{
// Verify the receipt with Apple
string postString = String.Format("{{ \"receipt-data\" : \"{0}\" }}", receiptData);
ASCIIEncoding ascii = new ASCIIEncoding();
byte[] postBytes = ascii.GetBytes(postString);
HttpWebRequest request;
request = WebRequest.Create("https://buy.itunes.apple.com/verifyReceipt") as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.ContentLength = postBytes.Length;
Stream postStream = request.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Close();
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
StringBuilder sb = new StringBuilder();
byte[] buf = new byte[8192];
Stream resStream = response.GetResponseStream();
string tempString = null;
int count = 0;
do
{
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
tempString = Encoding.ASCII.GetString(buf, 0, count);
sb.Append(tempString);
}
}
while (count > 0);
var fd = JObject.Parse(sb.ToString());
// Receipt not valid
if (fd["status"].ToString() != "0")
{
// Error out
}
// Product ID does not match what we expected
var receipt = fd["receipt"];
if (String.Compare(receipt["product_id"].ToString().Replace("\"", "").Trim(), itemID.Trim(), true) != 0)
{
// Error out
}
// This product was not sold by the right app
if (String.Compare(receipt["bid"].ToString().Replace("\"", "").Trim(), "com.wolf-studios.warofwords-pro", true) != 0)
{
// Error out
}
// This transaction didn’t occur within 24 hours in either direction; somebody is reusing a receipt
DateTime transDate = DateTime.SpecifyKind(DateTime.Parse(receipt["purchase_date"].ToString().Replace("\"", "").Replace("Etc/GMT", "")), DateTimeKind.Utc);
TimeSpan delay = DateTime.UtcNow – transDate;
if (delay.TotalHours > 24 || delay.TotalHours < -24)
{
// Error out
}
// Perform the purchase — all my purchases are server-side only, which is a very secure way of doing things
// Success!
}
catch (Exception ex)
{
// We crashed and burned — do something intelligent
}
}
[/sourcecode]
Drag to create IBOutlet/IBAction in Xcode 4
Just in case some of you missed this extremely cool feature:
In Xcode 4, you can now very quickly create IBOutlets/IBActions and their corresponding properties. Just right-click-drag from anything in Interface Builder to your view controller’s header file. It will ask you for a name and a couple other options, and then automatically create the @property and @synthesize statements, and wire up the outlet in the NIB. Supposedly it will also create the ivar under certain situations, but it didn’t do that for me. (Never fear, explicit ivars are no longer needed by the compiler.) Code is added to viewDidUnload to nil out the property, and if ARC is turned off, it will also create the dealloc statement for you.
Check out this excellent walk-through by Ole Begemann.
My workflow for certain screens, particularly modals, is to build the comp in Photoshop, approve it, then build out the interface in IB. Here’s the one I am working on today. Every element in this screen is going to animate on separately, so I need a ton of outlets. Boy was I glad to have this new feature!
Quiet lately
I’ve been quiet for some time now on this blog, and I’ve been staying off Twitter lately too. I’ve had a Super Secret™ project under heavy production mode for the past month or so, and I’m devoting all my attention to it. I’m hoping to launch an early gameplay alpha in a few weeks.
Since I last posted here, a few things have happened:
- I produced a new trailer for War of Words. This was a fun diversion that required an extreme crash course in After Effects.
- I released War of Words v3, which introduced the first new Action Tile: the Shield Tile. It was also a migration to ARC/iOS 5. Several bug fix updates followed.
- I pounded the pavement doing as much marketing as I could for the game. It got written up on several sites, but not the holy grail (Touch Arcade). Still, user gain and engagement is considerably higher than it was before the effort.
- I returned to World of Warcraft and created one of my most popular new add-ons yet: Rarity.
- War of Words was nominated for Best Word Game in the Best App Ever Awards. Sadly, I’m up against the likes of Words With Friends, Wooords, SpellTower, and W.E.L.D.E.R., so I don’t hold out much hope of winning.
- I started my next Super Secret™ project, which is shaping up nicely. It probably won’t make me much money, but I needed to get the ideas out of my system and the engine was already written.
- Bourbon Enthusiast has been doing unexpectedly well lately. Maybe people drink more bourbon during the winter?
- And I just ordered a shiny new 27″ iMac for my office! This’ll really help speed up my development time—my MacBook Pro is starting to really slow down with these huge iOS projects.
Anyway, just thought I’d post an update here for posterity, if anything just to prove I haven’t been idle!
The battle of the iOS crash reporters
When I released my first iOS app, I noticed the crash reports section in iTunes Connect. I naively assumed I would get any crashes reported to me through this interface. When my app never had any, I thought it was never crashing!
Boy, was I wrong.
It turns out that crash reports are only submitted by users who agree to do so. I honestly don’t remember ever being asked this, so I don’t even know if I opted in. This makes me think that most users probably don’t. Next, the user has to sync their device to a computer in order to report the crash. Yeah, most iOS users don’t bother with this. Even worse, only crashes that occur a certain number of times will actually be reported to you through ITC.
The moral of the story? ITC crash reports are completely, utterly useless. Odds are, you won’t even get any.
Clearly this isn’t the way. I’m a firm believer in thorough analytics, and my highest priority is rich crash and exception reporting.
I probably don’t need to tell you that your app does crash, and you won’t see most of the crashes during testing. The reasons for this vary from device and iOS differences, to network conditions, to arcane multithreading issues, to memory management errors. You need a way to find those crashes (and more importantly, be able to symbolicate them) in the wild.
OK, hopefully I’ve convinced you. So what to do? You can and should roll your own crash reporting in your apps. I tested out three possibilities. Let’s dig in.
Want the short version? Use HockeyApp.
Hoptoad
http://www.hoptoadapp.com
Pricing: Free for one app, $5-45/month beyond that
Features: Crash reporting (raw stack trace only)
Pros: Cheap, Github and Lighthouse integration, RSS support
Cons: No symbolication, clumsy app version support, company is rooted in Ruby and iOS is a secondary thought
Hoptoad was originally intended for Ruby applications, and you can tell. It has some clever features like Github tracking (where you say in your commit message that you fixed a crash, and Hoptoad marks it fixed automatically), and Lighthouse integration (I don’t use Lighthouse, but integrating your crash reporter with an issue tracking tool is a great idea). Unfortunately, the focus on Ruby shows when it comes to using this for iOS.
My current shipping version of War of Words integrates Hoptoad. As with all three of these examples, integration just takes adding the library to your project, plus one line of code. Hoptoad was a step in the right direction, but it has several fatal flaws:
- The biggest problem was version support. All crashes reported by all versions of my app get lumped into one big pot. So although I may have fixed a crash in a later version, I’ll still get hundreds of them populating my crash list. (Note: Hoptoad has supposedly solved this issue by allowing you to accept crashes from just one version of your app. I cancelled my account before I could test this out.)
- The list of crashes can’t be sorted
- Crashes are poorly grouped. This might be a problem with other services too, but I was getting one single crash type reported under a bunch of separate groups.
- I couldn’t get symbolication to work. You have to run atos manually for every line you wish to symbolicate, and I could never get it to work. I’m not an atos expert, but I think I was doing it right. Once I tried HockeyApp (below), I realized just how well this could work, but doesn’t.
- I engaged with tech support several times, and although they responded very quickly, it quickly became apparent that their iOS support was playing second fiddle to their Ruby support. It took them a few months to add app version filtering, and when they did, it was a half-assed implementation (and the dev who implemented it was a rude guy who didn’t seem to understand the importance of the feature).
QuincyKit
http://quincykit.net/
Pricing: Self hosted on your own PHP/MySQL server
Features: Crash reporting (can be auto-symbolicated if you set it up properly)
Pros: Cheap
Cons: Very bare-bones user interface, completely self-hosted means more work on your end (I didn’t test version support)
QuincyKit is an open source crash reporting tool that uses PLCrashReporter, also open source. The whole kit and kaboodle is free, assuming you have access to a PHP/MySQL server and know how to use it. Free, though, means more work for you. This sort of thing might appeal to the open source junkies out there, but I’m a solo developer and a firm believer in doing the minimum amount of work possible for a high quality result. This means I’m not going to waste any time on tools if I don’t have to.
The UI on the server side is pretty bad, which may not matter to you. Just thought I’d mention it. You could certainly do your own UI if you wanted.
QuincyKit is cool because it will supposedly symbolicate your crash reports for you, if you set it up. Setting it up involves a fair bit of work, though, and it wasn’t worth the effort for me.
I won’t spend a great deal of time focusing on QuincyKit, because it’s the same library that’s at the heart of the next tool: HockeyApp.
HockeyApp
http://www.hockeyapp.net/
Pricing: $20-100/month (free 30 day trial)
Features: Beta testing, crash reporting, seamless auto symbolicating, status feedback to users
Pros: Rich version support, auto symbolication that works, great UI
Cons: Expensive
Now that is a beautiful stack trace.
They say you get what you pay for in life, and nowhere is this more true than with HockeyApp. It’s definitely the most expensive option listed here, starting at $20 per month after the free 30-day trial. But it does everything right, and it does a whole lot more than just crash reporting.
(Just to be clear: HockeyApp is created by the same person who made QuincyKit. In fact, the QuincyKit library is required in your app in order to use HockeyApp. HockeyApp is basically the icing on the cake.)
HockeyApp’s big claim to fame is it’s beta testing ecosystem, offering a similar experience to the much-lauded TestFlight. My beta tests have been small scale so far, and I haven’t had an opportunity to really test out TestFlight or HockeyApp when it comes to this sort of thing. The focus of this post is on crash reporting. Just be aware that the reason HockeyApp charges so much more than the others is likely due to this feature.
OK, so how good is it? The short answer is very good. The single best killer feature is auto symbolication. You specify a version and build number (which lets you differentiate between different builds of the same version), and upload your dSYM file through the web interface. That’s it — your crash logs will be symbolicated when you see them. Not only that, but they’re super readable and a joy to look at. It’s a huge improvement from Ye Olde Xcode crash logs. This auto symbolication thing is worth the $20 to me alone.
It doesn’t stop there. Once you’ve fixed a crash, you can mark it as fixed on the web site. Users of your app who report that crash down the road will be told that the crash has been fixed, and that it’s either submitted to Apple or already available in the next update. This stuff is worth it’s weight in gold. You can also elect to stop receiving crash reports from any version of your app on a case-by-case basis, far outstripping the capabilities of Hoptoad’s version support.
Oh, and the UI is great.
Once you experience the HockeyApp brand of crash reporting, you’ll realize just how spoiled you’ve become. You’ll never want to go back to the dark ages of Hoptoad and it’s ilk. Welcome to the future.
Other alternatives
As people have pointed out in the comments below, there are other alternatives for crash reporting (and beta testing). I haven’t had any real-world experience with these, so I didn’t want to write them up. There are probably many more out there; this isn’t an exhaustive list.
Services
- Crittercism
- Apphance
- Exceptional (with the iOS API wrapper)
- BugSense
Libraries
You’ll have to roll your own web service and user interface to handle crashes coming from these libraries.
- PLCrashReporter (remember QuincyKit and HockeyApp already use this)
- CrashKit
- CrashReporter
Facing the music
After studying music composition in college, I got married and was very poor. To help bootstrap married life, I had to sell all my instruments (keyboard, drum set). Since then, my income grew greatly, but we just never prioritized getting new instruments.
I am really excited that today, 8 years later, I’m taking delivery of a Roland RD-700NX. This is a professional stage keyboard that really mimics the feel and sound of a real piano. I plan to pair it with Reason, Record, and Sibelius, which will enable sequencing, scoring, orchestrating, and recording using the keyboard.
My workstation is all setup and ready, including a great set of Klipsch studio monitors, a Sony receiver, and a dedicated 23″ monitor for scoring/sequencing work (separate from my 27″ Apple Cinema Display).
Back in college and before, I did lots of composing, orchestrating, arranging, and live performances. It’s going to be great to not only have music back in my life again after all this time, but to finally close my “talent loop”. I’ll be able to compose and produce my own music for iOS projects. It will also give me an additional creative outlet that will hopefully get my energy going.
It’s an exciting new chapter. Who knows what this will bring! I expect great things.
Update: Keyboard is now installed in the studio. Man, keyboard technology has vastly improved since I bought my first keyboard in 1998. This thing looks, feels, and sounds exactly like a piano. Best of all, I can tune every aspect of the tone. The only downside? I have to relearn how to play, since it’s been at least 8 years since I’ve touched a piano. Luckily, it’s already coming back fast. I can already almost play through one of my better original songs from my high school days.
Xcode 4: monkey business with rootViewController
I recently converted War of Words to Xcode 4. This had gone quite swimmingly for Bourbon Enthusiast. Both apps are universal, and both use a UISplitViewController as the root view controller.
But War of Words broke terribly on iPad after the conversion.
It really was just the conversion to Xcode 4 that caused the problem, and nothing else. After casting about the web for possible solutions, and heaping a huge share of hate toward Xcode 4, I was able to (mostly) figure out the problem.
There is a bug in Xcode 4, verified by several people on the Apple Developer Forums, where sometimes the Interface Builder NIBs get inexplicably bugged in some obscure way. The effect is that your app’s rootViewController gets loaded twice on top of itself. This screws up your app in all kinds of crazy ways. It rendered War of Words completely broken in every way.
I solved it by:
- Removing the rootViewController connection in Interface Builder (this causes the app to load with
window.rootViewController = nil)
- In viewDidLoad for the main controller (the one that was being loaded twice), I then manually assign
appDelegate.window.rootViewController = self
So far, this seems to have brought things back to normal. It’s crap like this that made me wait this long to migrate to Xcode 4. Now I’m used to the interface and generally like it better, so I hope there aren’t any more surprises like this waiting for me.
Here’s my answer on Stack Overflow.
Lessons learned in iOS development
Everybody's doing it!
It’s been eight months since I took up what seems to be every programmer’s favorite hobby: iOS development. In that time, I’ve shipped one lifestyle app and one word game. The game (War of Words) was an ambitious project for a solo developer—but I did manage to ship on schedule and with a fairly stable product right out the gate.
War of Words has been out for a little over three months now, but I’m not ready for a numbers post just yet. Instead, I’d like to share with you some of the lessons I’ve learned from the whole experience.
I’d figured out some of these lessons in advance—but most of them had to be learned the hard way. I write this with the hope that my experiences can help someone else out there who is embarking on their own ambitious project.
I should warn you—this is a long one.
Zenburn theme for Xcode 4
I finally upgraded to the GM seed of Xcode 4 today. It’s a bit of an adjustment process as usual, but so far I really like what I see.
One of the biggest annoyances is that Xcode 4 doesn’t automatically take many of your Xcode 3 settings during the update. Most egregious is the lack of theme compatibility. My favorite theme is the Zenburn theme (I use a similar theme in Visual Studio).
I’ve taken the liberty of converting the theme to Xcode 4 format and posting it for download here. Enjoy!
Update: I have since downgraded back to Xcode 3. Xcode 4 just wasn’t ready—it even introduced compilation issues that caused a previously working build to start crashing on the device.
