Guide to symbolicating iPhone app crash logs with Xcode 4.2

I was recently investigating one very strange crash in an app I’m working on. The app is distributed ad-hoc (through wonderful TestFlightapp.com) and each release is archived in Xcode 4.2. I got the .crash file from beta tester and dragged it into Xcode’s Organizer, which did symbolicate everything in the stack except two lines from my code.

Incident Identifier: 8B369A29-8421-4686-B1F6-9D66524937B5
CrashReporter Key:   746834fde1cb2033d4103df311e8f3b6ee9e8817
Hardware Model:      iPhone3,1
Process:         myApp [74]
Path:            /var/mobile/Applications/78EC36CC-DAF5-4910-9D60-A03DDC8E24BC/myApp.app/myApp
Identifier:      myApp
Version:         ??? (???)
Code Type:       ARM (Native)
Parent Process:  launchd [1]

Date/Time:       2011-10-27 11:37:55.841 +0100
OS Version:      iPhone OS 4.3.3 (8J2)
Report Version:  104

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x00000000, 0x00000000
Crashed Thread:  0

Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x3657aa1c 0x36569000 + 72220
1   libsystem_c.dylib             	0x366513b4 pthread_kill + 52
2   libsystem_c.dylib             	0x36649bf8 abort + 72
3   libstdc++.6.dylib             	0x36613a64 __gnu_cxx::__verbose_terminate_handler() + 376
4   libobjc.A.dylib               	0x3548806c _objc_terminate + 104
5   libstdc++.6.dylib             	0x36611e36 __cxxabiv1::__terminate(void (*)()) + 46
6   libstdc++.6.dylib             	0x36611e8a std::terminate() + 10
7   libstdc++.6.dylib             	0x36611f5a __cxa_throw + 78
8   libobjc.A.dylib               	0x35486c84 objc_exception_throw + 64
9   Foundation                    	0x3522d924 __NSThreadPerformPerform + 648
10  CoreFoundation                	0x31942a72 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 6
11  CoreFoundation                	0x31944758 __CFRunLoopDoSources0 + 376
12  CoreFoundation                	0x319454e4 __CFRunLoopRun + 224
13  CoreFoundation                	0x318d5ebc CFRunLoopRunSpecific + 224
14  CoreFoundation                	0x318d5dc4 CFRunLoopRunInMode + 52
15  GraphicsServices              	0x31254418 GSEventRunModal + 108
16  GraphicsServices              	0x312544c4 GSEventRun + 56
17  UIKit                         	0x319fbd62 -[UIApplication _run] + 398
18  UIKit                         	0x319f9800 UIApplicationMain + 664
19  myApp                      	0x00003a4c main (main.m:14)
20  myApp                      	0x00003a04 0x1000 + 10756

...

Binary Images:
    0x1000 -    0x67fff +myApp armv7  <a0b39fab741e34eb831048cc752d8e0c> /var/mobile/Applications/78EC36CC-DAF5-4910-9D60-A03DDC8E24BC/myApp.app/myApp

Great. The usual reason for this is that Xcode did not find .dSYM file. This is actually really strange since Xcode has the .xcarchive, saved in its default location. But no matter what I tried, it did not symbolicate properly. One possible reason that I’ve seen people mention is that Spotlight has not indexed the archives and you can force this with this Terminal command:

mdimport ~/Library/Developer/Xcode/Archives/

This did not help either. I tested Spotlight and strangely it could not find the app signature (from the bottom of the crash log excerpt above):

mdfind "com_apple_xcode_dsym_uuids == A06EC84E-53BF-3209-8B63-C0CAEBDB45B6"

At this point I tried to do it manually and in series of attemps encountered all kind of possible issues.

First, I had a custom symbolicatecrash script in /usr/local/bin/ from several months ago. This script often changes as Apple devs fix bugs in it. While they do that and ship the fixed version with next Xcode update, good souls on the internet have already resolved the given issue. Thus at some point I downloaded one of those custom versions and placed it there. Hence when I tried to run symbolicatecrash manually, it used that version. You can check this in Terminal, by typing which symbolicatecrash – if it finds the script anywhere in your default paths, it will show it. If not, then you are ok.

Second, in Xcode 4.2, the current version of this script is at this location: /Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources/symbolicatecrash. Files in this private framework are not accessible from any other folder, so in order to run it manually you need to either use this full path or copy it to the folder where you acquired .crash log files are and run it from there.

Third, I extracted the .app and .dSYM file from the archive and placed in the same folder. I then tried to run the script from there, with all of these line (one by one):

./symbolicatecrash -v *.crash . > s.crash

./symbolicatecrash -A -v myApp_2011-10-27-113755_Alex-iPhone.crash myApp.app.dSYM > s.crash

None did anything differently. Things began to look really crazy at this point and I started doubting is this a crash report really from this version. Double-checked with tester, also double checked the .app.dSYM file as described here on Stack Overflow. All was fine, this was the binary used.

Forth, I tried to directly look for the given hex address in the crash log:

dwarfdump --lookup 0x00003a04 --arch armv6 myApp.app.dSYM

And this finally gave me what I needed:

----------------------------------------------------------------------
 File: myApp.app.dSYM/Contents/Resources/DWARF/myApp (armv6)
----------------------------------------------------------------------
Looking up address: 0x0000000000003a04 in .debug_info... found!

0x0000012c: Compile Unit: length = 0x00005a18  version = 0x0002  abbr_offset = 0x00000000  addr_size = 0x04  (next CU at 0x00005b48)

0x00000137: TAG_compile_unit [1] *
             AT_producer( "Apple clang version 3.0 (tags/Apple/clang-211.9) (based on LLVM 3.0svn)" )
             AT_language( DW_LANG_ObjC )
             AT_name( "/Users/aleck/dev/consulting/myApp/trunk/Classes/AppDelegate.m" )
             AT_entry_pc( 0x0000323c )
             AT_stmt_list( 0x00000112 )
             AT_comp_dir( "/Users/aleck/dev/consulting/myApp/trunk" )
             AT_APPLE_optimized( 0x01 )
             AT_APPLE_major_runtime_vers( 0x02 )

0x00000dc3:     TAG_subprogram [15] *
                 AT_sibling( {0x00000dfb} )
                 AT_name( "-[AppDelegate dealloc]" )
                 AT_decl_file( "/Users/aleck/dev/consulting/myApp/trunk/Classes/AppDelegate.m" )
                 AT_decl_line( 250 )
                 AT_prototyped( 0x01 )
                 AT_APPLE_isa( 0x01 )
                 AT_low_pc( 0x000039d4 )
                 AT_high_pc( 0x00003a6c )
                 AT_frame_base( r7 )
Line table dir : '/Users/aleck/dev/consulting/myApp/trunk/Classes'
Line table file: 'AppDelegate.m' line 254, column 5 with start address 0x0000000000003a02

Looking up address: 0x0000000000003a04 in .debug_frame... found!

0x00000100: FDE
        length: 0x0000000c
   CIE_pointer: 0x00000000
    start_addr: 0x000039d4 -[AppDelegate dealloc]
    range_size: 0x00000098 (end_addr = 0x00003a6c)
  Instructions: 0x000039d4: CFA=4294967295+4294967295

Or so I thought. When I looked into my code at this line, this was in dealloc method, where NSPersistentStoreCoordinator was released. And that was really nutty place to have a bug. At this point I was completely stumped and had no idea what else to try.

Luckily in this case, tester was able to give login credentials so I could repeat the scenario using his data, while debugging the app on the device. And found that I had a badly formatted NSPredicate which worked in 99.9% of cases and this tester stumbled on 0.1% where it did not. Great find, it would be a serious head-scratcher if app went live with this.

But I still don’t understand why Xcode 4.2 won’t see the .dSYM file and while the script itself won’t symbolicate this .crash log. Honestly, I can’t help but think this is seriously more complicated than it needs be. But it is as it is. Here some other useful stuff I encountered while working on this.

Place where Xcode puts .crash logs picked from your testing devices: ~/Library/Logs/CrashReporter/MobileDevice/<your device name>.symbolicated

Various possibly helpful links:
· http://www.goosoftware.co.uk/blog/the-symbolicator-helps-those-who-help-themselves/
· https://devforums.apple.com/message/404524
· http://stackoverflow.com/questions/2697067/symbolicate-adhoc-iphone-app-crashes

If you have any ideas that could help, please write in comments.

While listening my favorite podcasts these days, quite a bit of them are discussing this quote from the Steve Jobs biography:

“I’d like to create an integrated television set that is completely easy to use. It would be seamlessly synced with all of your devices and with iCloud. It will have the simplest user interface you could imagine. I finally cracked it.”

Mouth-watering, isn’t it? Given the fact what was done for phone and tablets, this is beyond interesting. John Gruber, Marco Arment and John Siracusa all shared how they see this working. As usual, I agree with bits from all of the stuff they spoke, but not all of it (and them do not agree on all counts). There’s also an oft-linked NY Times article on Siri as the main interface for this new TV.

Here’s what I believe will be main features of the future Apple TV business.

2 Comments

Feel free to chime in, looking forward to it. Leave a Comment

Brian Gorby says:

Thu, Jan 5th, 2012 at 1:52 ·

I’ve been battling this today. Thanks for confirming I’m not the only one going crazy.

I found that you could force the indexing of the Spotlight attribute (com_apple_xcode_dsym_uuids) with the mdimport command-line tool. The argument to this should be your .xcarchive bundle . You can then run mdfind “com_apple_xcode_dsym_uuids == ‘*’”, and the .xcarchive bundle should appear, in theory making the symbolicatecrash script work again.

No clue why XCode randomly decides to not do this step during the archiving of builds.
Simon Black says:

Sat, Feb 11th, 2012 at 1:35 ·

You are not alone. Been dealing with this kind of crap through two apps over the last year or more. It seems to break with every iOS and/or Xcode upgrade. Basically, Apple made this too “magical” for its own good and it breaks all the time.

Currently, with Xcode 4.2.1 and iOS 5.01 I had to deal with these issues:

1. Standard Apple libraries were not symbolicating on iOS 5.01. Solution: Upgrade Xcode to 4.2.1, delete /Library/Developer/Xcode/iOS DeviceSupport/5.0.1 (9A405). Reattached your iOS device. See:

http://stackoverflow.com/questions/8176393/symbolicatecrash-with-xcode-4–2-not-symbolicating-system-symbols

2. My app would not symbolicate either. I forgot that you seemingly must have BOTH the .app and the .app.dsym directories on the Mac running Xcode in the same subdirectory. Apple basically assumes you use your Mac running Xcode to develop an app and so Xcode depends on this standard situation where after a build, those files are on your Mac in the same directory.

You have pretty much covered all the other possible issues, most of which were things that came up in the transition to Xcode 4.2, in your post.

Anyway, this sucks and I wish it was a lot easier.

The Blog