A 3-post collection

Post mortem analysis of Swift server code

Just a very quick idea of how you could handle server side crashes of a Swift binary. Swift itself has no Stack unwinding functions that you could use for debugging purposes but lldb has.

So what if the currently crashing program would attach lldb to itself and create stack traces before vanishing into nirvana?

import Darwin

private func signalHandler(signal: Int32) {
    // need my pid for telling lldb to attach to parent
    let pid = getpid()
    // create command file
    let filename = "/tmp/backtrace.\(pid)"
    var fp = fopen(filename, "w")
    if fp == nil {
        print("Could not open command file")
    // attach to pid
    var cmd = "process attach --pid \(pid)\n"
    fputs(cmd, fp)
    // backtrace for all threads
    cmd = "bt all\n"
    fputs(cmd, fp)

//    // save core dump
//    cmd = "process save-core coredump\n"
//    fputs(cmd, fp)

    // kill the process
    cmd = "process kill\n"
    fputs(cmd, fp)
    // delete the command file
    cmd = "script import os\nscript os.unlink(\"\(filename)\")\n"
    fputs(cmd, fp)
    // quit lldb
    cmd = "quit\n"
    fputs(cmd, fp)

    // add signal type to backtrace.log header
    fp = fopen("backtrace.log", "w")
    if fp == nil {
        print("Could not open log file")
    fputs("Signal \(signal) caught, executing lldb for backtrace\n", fp)
    // run lldb
    let command = "/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin/lldb --file \"\(Process.arguments[0])\" --source \"\(filename)\" >>backtrace.log"

// Install signal handler
signal(SIGILL, signalHandler)
signal(SIGTRAP, signalHandler)
signal(SIGABRT, signalHandler)
signal(SIGSEGV, signalHandler)
signal(SIGBUS, signalHandler)
signal(SIGFPE, signalHandler)

// Now crash
print("Hello, World!")
var forcedUnwrap: Int! = nil


This code traps all fatal error signals and calls lldb with a small command file that looks like this and is generated by the crashing program:

process attach --pid <my_pid>
bt all
process kill
script import os
script os.unlink("<command_file>")

So it attaches lldb to a PID, fetches a backtrace for all threads, kills the parent process and deletes the command file. Log output is diverted to backtrace.log and contains all lldb output.

Additionally you could include process save-core coredump to write a core dump to the current directory which can be loaded for further inspection, but beware a core dump for our simple program up there will be around 500MB and will take about 30 seconds to write to disk (SSD).

You can load the core dump like this:

lldb -f <binary> -c coredump

Now you can inspect memory and variables like the program crashed while the debugger was attached.

The cool thing is, this code will not disable the Xcode internal debugger, so you get the usual EXC_BAD_INSTRUCTION when running the code in Xcode.

Useful Xcode breakpoints

Here I will document useful breakpoints when you're developing for OSX or iOS with Xcode. This is primarily for me to remember what is useful as I am googling some of these all the time.

It's sad that there are no "breakpoint-templates" that will automatically apply to all Xcode projects that you'll ever create. But enough of the introductory words, here comes the list:

Objective C exception breakpoint

Obviously the most important breakpoint there is, this one has a "template" of some sort as it has its very own menu entry:

Objective C Exception Breakpoint

So far so good, but the annoying thing is that the exception message will not be printed when the breakpoint is reached but only if you continue (and crash your program for real, and of course void your stack backtrace)

So add this to the default:

i386/iOS simulator (32Bit)

Objective C Exception Breakpoint modifications

Just add the following actions:

  • po *(id*)($esp+4)

which will print for example:

-[__NSCFConstantString characterAtIndex:]: Range or index out of bounds

x86_64/iOS simulator (64Bit)

Use the following actions:

  • po $rdx
  • po $rcx
  • po $r8

iOS device (ARM/ARM64)

Use these:

  • po $r2
  • po $r3
  • po *(id*)($sp)

If all this is to verbose to you look at the prebuild lldb script from here: http://qwan.org/2013/06/18/how-to-snatch-the-error-code-from-the-trap-frame-in-xcode/

Memory errors

Attention: This may be superseeded by the new Address sanitizer in Xcode 7

Sometimes it happens that you find a nasty memory error that is not at all obvious where it came from. Mostly it appears somewhere far away from the original error because the stack was smashed or a buffer overflow bleed into neighboring variables and overwrote something.

To find those errors you'll usually first enable all the memory protection error loggers that are available:

Memory debugging

And now you'll add a symbolic breakpoint at malloc_error_break to catch the offender that is smashing your stack right away (or at least very near to the cause)

Core Graphics Errors

All of Core Graphics error logging will go through the function CGPostError so it is sensible to add a breakpoint at exactly that location. It will break at anything Core Graphics will spit out. You know those 'XXX tried to draw to a nil context` errors may be hard to find if a lot is going on in your application at the same time.

Slow loading views

Perhaps because you used a wrong font-name? Who knows! Set a symbolic breakpoint to CTFontLogSuboptimalRequest to be notified when Core Text does not find the font directly but has to resort to a detailed search (that is very slow, on an iPad 4 it takes about 3 seconds to complete)

Auto layout

This one is obvious when it happens. But it's better to catch it directly instead of waiting for one to happen so for completeness, set a symbolic breakpoint to UIViewAlertForUnsatisfiableConstraints

Improve the debugger

Sometimes lldb can be quite stubborn:

(lldb) p self.window.bounds
error: property 'bounds' not found on object of type 'UIWindow *'
error: 1 errors parsing expression

But this can be fixed:

(lldb) expr @import UIKit
(lldb) p self.window.bounds
(CGRect) $4 = (origin = (x = 0, y = 0), size = (width = 375, height = 667))

Wow, much better, but what has this to do with breakpoints you may ask? Just try the following: Set a new breakpoint in your app delegate that is hit immediately after starting your application, set it to auto-continue and add an action expr @import UIKit.

The next run of your app will stop briefly at that breakpoint, execute that command and continue running. If you hit any other breakpoint (or press pause) the command already has been executed and lldb knows everything about UIKit already! Great!

(You may want to @import Foundation and @import CoreGraphics too)

Xcode build phases

This is just a short brain-dump what to put in each and every project's build phases:

Show all TODO: and FIXME: comments as warnings

Based on another blog article of Jeffrey Sambells, put this as a run script build phase into your build process:

echo "searching ${SRCROOT} for ${TAGS}"
find "${SRCROOT}/Source" \( -name "*.h" -or -name "*.m" \) -print0 | xargs -0 egrep --with-filename --line-number --only-matching "($TAGS).*\$" | sed -e 's/\([^ ]*[0-9][0-9]*:\)\(.*\)/\1 warning: \2/'

My changes include the doxygen @ tags and replace the invocation of perl with a better working version in sed script.

Automatically build documentation

Install the awesome appledoc to build better documentation than doxygen might do (all with the same comment markup ;) )

Best install not via homebrew but via the install-script in the git repo (as you will get no templates in a homebrew install)

Add a new custom target, name it documentation and add this script to the build phases:

cd "${SRCROOT}"
if [ -x /usr/local/bin/appledoc ] ; then
    /usr/local/bin/appledoc . 2>&1 |sed -e '/warning: Ignoring/d' >&2
    echo "error: appledoc not installed" >&2
    exit 1

You'll need a config-file named AppledocSettings.plist in your $(SRCROOT):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">