swift

A 1-post collection

Post mortem analysis of Swift server code

Just a very quick idea of how you could handle server side crashes of a Swift binary. Swift itself has no Stack unwinding functions that you could use for debugging purposes but lldb has.

So what if the currently crashing program would attach lldb to itself and create stack traces before vanishing into nirvana?

import Darwin

private func signalHandler(signal: Int32) {
    // need my pid for telling lldb to attach to parent
    let pid = getpid()
    
    // create command file
    let filename = "/tmp/backtrace.\(pid)"
    var fp = fopen(filename, "w")
    if fp == nil {
        print("Could not open command file")
        exit(1)
    }
    
    // attach to pid
    var cmd = "process attach --pid \(pid)\n"
    fputs(cmd, fp)
    
    // backtrace for all threads
    cmd = "bt all\n"
    fputs(cmd, fp)

//    // save core dump
//    cmd = "process save-core coredump\n"
//    fputs(cmd, fp)

    // kill the process
    cmd = "process kill\n"
    fputs(cmd, fp)
    
    // delete the command file
    cmd = "script import os\nscript os.unlink(\"\(filename)\")\n"
    fputs(cmd, fp)
    
    // quit lldb
    cmd = "quit\n"
    fputs(cmd, fp)
    fclose(fp)

    // add signal type to backtrace.log header
    fp = fopen("backtrace.log", "w")
    if fp == nil {
        print("Could not open log file")
        exit(1)
    }
    fputs("Signal \(signal) caught, executing lldb for backtrace\n", fp)
    fclose(fp)
    
    // run lldb
    let command = "/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin/lldb --file \"\(Process.arguments[0])\" --source \"\(filename)\" >>backtrace.log"
    system(command)
    exit(1)
}

// Install signal handler
signal(SIGILL, signalHandler)
signal(SIGTRAP, signalHandler)
signal(SIGABRT, signalHandler)
signal(SIGSEGV, signalHandler)
signal(SIGBUS, signalHandler)
signal(SIGFPE, signalHandler)

// Now crash
print("Hello, World!")
var forcedUnwrap: Int! = nil

print(forcedUnwrap)

This code traps all fatal error signals and calls lldb with a small command file that looks like this and is generated by the crashing program:

process attach --pid <my_pid>
bt all
process kill
script import os
script os.unlink("<command_file>")
quit

So it attaches lldb to a PID, fetches a backtrace for all threads, kills the parent process and deletes the command file. Log output is diverted to backtrace.log and contains all lldb output.

Additionally you could include process save-core coredump to write a core dump to the current directory which can be loaded for further inspection, but beware a core dump for our simple program up there will be around 500MB and will take about 30 seconds to write to disk (SSD).

You can load the core dump like this:

lldb -f <binary> -c coredump

Now you can inspect memory and variables like the program crashed while the debugger was attached.

The cool thing is, this code will not disable the Xcode internal debugger, so you get the usual EXC_BAD_INSTRUCTION when running the code in Xcode.