Guide to symbolicating iPhone app crash logs with Xcode 4.2

I was recently investigating one very strange crash in an app I’m working on. The app is distributed ad-hoc (through wonderful TestFlightapp.com) and each release is archived in Xcode 4.2. I got the .crash file from beta tester and dragged it into Xcode’s Organizer, which did symbolicate everything in the stack except two lines from my code.

Incident Identifier: 8B369A29-8421-4686-B1F6-9D66524937B5
CrashReporter Key:   746834fde1cb2033d4103df311e8f3b6ee9e8817
Hardware Model:      iPhone3,1
Process:         myApp [74]
Path:            /var/mobile/Applications/78EC36CC-DAF5-4910-9D60-A03DDC8E24BC/myApp.app/myApp
Identifier:      myApp
Version:         ??? (???)
Code Type:       ARM (Native)
Parent Process:  launchd [1]

Date/Time:       2011-10-27 11:37:55.841 +0100
OS Version:      iPhone OS 4.3.3 (8J2)
Report Version:  104

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x00000000, 0x00000000
Crashed Thread:  0

Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib            0x3657aa1c 0x36569000 + 72220
1   libsystem_c.dylib                 0x366513b4 pthread_kill + 52
2   libsystem_c.dylib                 0x36649bf8 abort + 72
3   libstdc++.6.dylib                 0x36613a64 __gnu_cxx::__verbose_terminate_handler() + 376
4   libobjc.A.dylib                   0x3548806c _objc_terminate + 104
5   libstdc++.6.dylib                 0x36611e36 __cxxabiv1::__terminate(void (*)()) + 46
6   libstdc++.6.dylib                 0x36611e8a std::terminate() + 10
7   libstdc++.6.dylib                 0x36611f5a __cxa_throw + 78
8   libobjc.A.dylib                   0x35486c84 objc_exception_throw + 64
9   Foundation                        0x3522d924 __NSThreadPerformPerform + 648
10  CoreFoundation                    0x31942a72 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 6
11  CoreFoundation                    0x31944758 __CFRunLoopDoSources0 + 376
12  CoreFoundation                    0x319454e4 __CFRunLoopRun + 224
13  CoreFoundation                    0x318d5ebc CFRunLoopRunSpecific + 224
14  CoreFoundation                    0x318d5dc4 CFRunLoopRunInMode + 52
15  GraphicsServices                  0x31254418 GSEventRunModal + 108
16  GraphicsServices                  0x312544c4 GSEventRun + 56
17  UIKit                             0x319fbd62 -[UIApplication _run] + 398
18  UIKit                             0x319f9800 UIApplicationMain + 664
19  myApp                         0x00003a4c main (main.m:14)
20  myApp                         0x00003a04 0x1000 + 10756

...

Binary Images:
    0x1000 -    0x67fff +myApp armv7  <a0b39fab741e34eb831048cc752d8e0c> /var/mobile/Applications/78EC36CC-DAF5-4910-9D60-A03DDC8E24BC/myApp.app/myApp

Great. The usual reason for this is that Xcode did not find .dSYM file. This is actually really strange since Xcode has the .xcarchive, saved in its default location. But no matter what I tried, it did not symbolicate properly. One possible reason that I’ve seen people mention is that Spotlight has not indexed the archives and you can force this with this Terminal command:

mdimport ~/Library/Developer/Xcode/Archives/

This did not help either. I tested Spotlight and strangely it could not find the app signature (from the bottom of the crash log excerpt above):

mdfind "com_apple_xcode_dsym_uuids == A06EC84E-53BF-3209-8B63-C0CAEBDB45B6"

At this point I tried to do it manually and in series of attemps encountered all kind of possible issues.

First, I had a custom symbolicatecrash script in /usr/local/bin/ from several months ago. This script often changes as Apple devs fix bugs in it. While they do that and ship the fixed version with next Xcode update, good souls on the internet have already resolved the given issue. Thus at some point I downloaded one of those custom versions and placed it there. Hence when I tried to run symbolicatecrash manually, it used that version. You can check this in Terminal, by typing which symbolicatecrash - if it finds the script anywhere in your default paths, it will show it. If not, then you are ok.

Second, in Xcode 4.2, the current version of this script is at this location: /Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources/symbolicatecrash. Files in this private framework are not accessible from any other folder, so in order to run it manually you need to either use this full path or copy it to the folder where you acquired .crash log files are and run it from there.

Third, I extracted the .app and .dSYM file from the archive and placed in the same folder. I then tried to run the script from there, with all of these line (one by one):

./symbolicatecrash -v *.crash . > s.crash
./symbolicatecrash -A -v myApp_2011-10-27-113755_Alex-iPhone.crash myApp.app.dSYM > s.crash

None did anything differently. Things began to look really crazy at this point and I started doubting is this a crash report really from this version. Double-checked with tester, also double checked the .app.dSYM file as described here on Stack Overflow. All was fine, this was the binary used.

Forth, I tried to directly look for the given hex address in the crash log:

dwarfdump --lookup 0x00003a04 --arch armv6 myApp.app.dSYM

And this finally gave me what I needed:

----------------------------------------------------------------------
 File: myApp.app.dSYM/Contents/Resources/DWARF/myApp (armv6)
----------------------------------------------------------------------
Looking up address: 0x0000000000003a04 in .debug_info... found!
0x0000012c: Compile Unit: length = 0x00005a18  version = 0x0002  abbr_offset = 0x00000000  addr_size = 0x04  (next CU at 0x00005b48)
0x00000137: TAG_compile_unit [1] *
             AT_producer( "Apple clang version 3.0 (tags/Apple/clang-211.9) (based on LLVM 3.0svn)" )
             AT_language( DW_LANG_ObjC )
             AT_name( "/Users/aleck/dev/consulting/myApp/trunk/Classes/AppDelegate.m" )
             AT_entry_pc( 0x0000323c )
             AT_stmt_list( 0x00000112 )
             AT_comp_dir( "/Users/aleck/dev/consulting/myApp/trunk" )
             AT_APPLE_optimized( 0x01 )
             AT_APPLE_major_runtime_vers( 0x02 )

0x00000dc3:     TAG_subprogram [15] *
                 AT_sibling( {0x00000dfb} )
                 AT_name( "-[AppDelegate dealloc]" )
                 AT_decl_file( "/Users/aleck/dev/consulting/myApp/trunk/Classes/AppDelegate.m" )
                 AT_decl_line( 250 )
                 AT_prototyped( 0x01 )
                 AT_APPLE_isa( 0x01 )
                 AT_low_pc( 0x000039d4 )
                 AT_high_pc( 0x00003a6c )
                 AT_frame_base( r7 )

Line table dir : '/Users/aleck/dev/consulting/myApp/trunk/Classes'
Line table file: 'AppDelegate.m' line 254, column 5 with start address 0x0000000000003a02

Looking up address: 0x0000000000003a04 in .debug_frame... found!

0x00000100: FDE
        length: 0x0000000``` objective-c
   CIE_pointer: 0x00000000
    start_addr: 0x000039d4 -[AppDelegate dealloc]
    range_size: 0x00000098 (end_addr = 0x00003a6c)
  Instructions: 0x000039d4: CFA=4294967295+4294967295

Or so I thought. When I looked into my code at this line, this was in dealloc method, where NSPersistentStoreCoordinator was released. And that was really nutty place to have a bug. At this point I was completely stumped and had no idea what else to try.

Luckily in this case, tester was able to give login credentials so I could repeat the scenario using his data, while debugging the app on the device. And found that I had a badly formatted NSPredicate which worked in 99.9% of cases and this tester stumbled on 0.1% where it did not. Great find, it would be a serious head-scratcher if app went live with this.

But I still don’t understand why Xcode 4.2 won’t see the .dSYM file and while the script itself won’t symbolicate this .crash log. Honestly, I can’t help but think this is seriously more complicated than it needs be. But it is as it is. Here some other useful stuff I encountered while working on this.

Place where Xcode puts .crash logs picked from your testing devices: ~/Library/Logs/CrashReporter/MobileDevice/<your device name>.symbolicated

Various possibly helpful links:

· http://www.goosoftware.co.uk/blog/the-symbolicator-helps-those-who-help-themselves/

· https://devforums.apple.com/message/404524

· http://stackoverflow.com/questions/2697067/symbolicate-adhoc-iphone-app-crashes

If you have any ideas that could help, please write in comments.

Aleksandar • Vacić

iOS bits and pieces

Guide to symbolicating iPhone app crash logs with Xcode 4.2