Back in Spring of 2015 I started doing some work for a new release of Soundproof, my iOS app for music practice. We’d just been through launch in Autumn 2014 having gone through a rapid migration to Xcode 6 and iOS 8. The plan was to add a few little feature enhancements and push out a release.

That Xcode 6 migration had been rather painful. We used CocoaPods for our dependencies and at that point there were problems with building for Xcode 6 and its new dynamic frameworks, BitCode, the iPhone 6 and 6 Plus with new screen sizes, assets and layout challenges. For long periods it had been difficult to build the code, so later in the Summer of 2015 I decided to rework all our dependencies to work with Carthage, including a few little open source frameworks we pulled in.

Throw in some experimentation with Fastlane and Xcode Bots, and you have a recipe for months of delays while the code was out in the weeds.

By this time I had taken contract work at Upthere, Inc. because, and this will be no surprise to indies, it is very hard to make good money from iOS apps. Nevertheless I tinkered on Soundproof every now and again to try to keep it moving forward.

However I hit a rather painful roadblock. I would run the app in Xcode and it would start… but the debugger would disconnect from the app. I had no idea why this was. It meant that I could not debug any of my code. I couldn’t finish the features I was working on. I couldn’t even get print statements or log output. OK in hindsight I could have set up a persistent file logger but… seriously, finding a fix for the debugger problem couldn’t take that long could it?

Time passed. I investigated Xcode issues. I looked at logs. I asked people for help. Caught between work and the utterly frustrating situation I stopped bothering and got on with the day job.

I tried again in January 2016, and asked the fine people in #code on the Core Intuition Slack. I made no progress and dropped it again. There were less horrific things to deal with after all.

Which brings us to the last few days, where I decided to try again so I could do some Soundproof feature releases of the cool stuff I already had nearly finished for two years. Also, I didn’t want to be defeated by this stupid problem.

The good news is that I have today solved the mystery, with the help of the fine folks on Core Intuition Slack again. Here’s a little walk through of how we got there. Three Xcode releases and three iOS releases after the original problem surfaced.

It is worth noting that in all this time, Soundproof as released to the App Store in 2014 still runs well on iOS through to 11 and on the new devices launched since. That is by design, because we stringently avoid applying weird hacks, workarounds and bending the UI frameworks to our will. We build to last.

That is except for a couple of things which, as luck would have it came back to bite us. (Literally only us as this is just a development problem).

We need to root cause this mofo

So Xcode would run the app, start attaching the debugger, and then as the app completed startup — before it executed any code in main(…) — the debugger would disconnect.

The logs from Xcode “Devices” or Console.app for the Simulator weren’t much help. They are incredibly noisy and have a lot of confusing internal terminology about creating “assertions” (not like assert() you would use in code).

On a real device, you’d get logging like this around the point the debugserver process (as I now know it to be called) was seemingly crashing:

Note the part Formulating report for corpse[387] debugserver. It does seems like debugserver is crashing. If I ran it on the Simulator sometimes I would get a crash in debugserver on the Mac, which would give an unsymbolicated crash trace:

The observant hacker-guru types among you may notice there is a clue in there that I did not know to pick up on until resolving the problem. Anyway, the folks on Core Intuition Slack were great and the recommendation was to strip out code from the app until debugserver stopped crashing.

This classic programming trick is actually a really difficult task in a non-trivial app like Soundproof, especially as the problem occurs before main() is called, so if anything it is related to statically initialised values.

I decided to go the other way: I created a new Xcode Project for a single-view app and added all the Carthage dependencies the main app uses. I thought I should first find out if it is my app code or something I’m pulling in as a dependency.

As luck would have it, the test project crashed debugserver. This was great news. Quickly removing one dependency at a time and running the build after each removal (slightly more fun as there are interdependencies so I had to start with leaf nodes first), I quickly came to the dependency to blame: “GBDeviceInfo”, an open source framework for accessing data about the device the app is running on.

This was fantastic if slightly depressing news, I just had to find out what there was in this framework that was crashing the debugserver process.

The sequence of unfortunate events…

You might wonder why I needed to check for the characteristics of the current iOS device the app was running on. This is generally a bad idea, and I was never happy with it. However, in the iOS 8 era when this code was written, things were a bit rough to say the least.

We just had all the iOS 7/8 blurring trend land on us, but without mature visual effects view support and so we had some manual code to perform a static blur on a large background. This couldn’t run on some of the devices like iPhone 4, 4S and 5 that were still very popular at the time. So I added some code to test the device type to see if we would just apply some alpha instead of a blur in those cases.

In addition to this, there was a fun bug in the middle of the iOS 8 point release cycle, circa iOS 8.0.2. We use AVSpeechSynthesizer to speak the name of the next track for the user, and someone at Apple managed to change the meaning of playback rate 1.0 for the utterances. This resulted in speech that was way too fast, but only on devices running certain point releases of iOS 8. So again we had used the device info dependency as a quick and easy way to test for iOS 8 and apply a different playback rate.

These challenges with iOS 8 meant we needed a quick and easy way to tell which device and iOS release we’re running on, and we pulled in GBDeviceInfo, a handy little framework for this.

We shipped with this and all was fine. Debugging was fine. However due to the CocoaPods debacles with Xcode 6, I had to fork GBDeviceInfo to experimentally hack in my own Carthage support — essentially just adding a new shared Xcode Scheme.

I did this fork in mid-2015 from a newer version of the library than the released App Store version, and it all seemed OK enough. Although that was when I started noticing something funny with the debugging not being reliable in Xcode.

Today, trawling through the commit history of GBDeviceInfo to find something that might cause the problem I was having with debugserver I came across a commit in October 2015 entitled:

“Removed debugger attachment prevention”

Commit 6cd971aaf75b385dd37ca1e30a29416d0468b6b4

Yep. All the while I suspected some obscure debug symbol problem causing the crash but it turns out that the library was working as intended and had previously contained code to prevent debuggers attaching:

(here’s where the observant among you will see that this denies ptrace and ptrace was on the stacktrace from the Simulator debugserver crashes.)

As I had taken my fork of the library before that code to remove the debug prevention landed, and not suspecting I needed anything new from the upstream repository, and with “in progress” Carthage changes in place… I had not pulled those newer changes.

I am not going to be hard on myself about this though. Who on earth would have thought someone would put this code into an open source library?

Ultimately, I should not have incorporated a dependency for which I did not review all the code. I did review it originally when deciding which library to use. However the change came in after we originally released so I would also have had to review all commits in between. I suppose this is the fallacy of using external dependencies. If you factor in all the time to review all the code and all the updates, the gains are not nearly so great. A lot of the benefit comes from trust or wishful thinking frankly.

I have definitely learned a painful lesson. Having GBDeviceInfo available as open source got the app released quicker than if I had to write that stuff myself, and while it is unfortunate for me the author added this pretty bonkers bit of code for a short period of time, it is more an unfortunate sequence of events.

I am pleased however that I can release a new Soundproof build soon… ish.