# Graphical debugging on the Librem 5 using QtCreator

**This blog post was paid for by Purism, as part of the work I do on the Librem 5 mobile phone.**

I've always had a weak spot for debugging with graphical, user-friendly tools. Whenever I run bare GDB, I'm limited to what I can see at once, and learning about the choices available to me is a chore. Navigating the code is not integrated either. Clearly, I need some extra tooling to make GDB worthwhile.

QtCreator is such a tool. It checks the above boxes, and also some more. One feature in particular was taunting me for a while.

## Remote debugging

Wouldn't it be great to compile stuff on my powerful desktop and debug it from my laptop on the sofa? Or, for that matter, to debug programs that access the camera on the Librem 5?

It isn't as easy as it could be. Sure, you could always attach to a GDB server ad-hoc, but then you have to deploy manually, start the GDB server, and you don't get to see the code you're debugging. Kinda useless.

There's also the mode where the debugger integrates with your project, taking care of deployment and startup automatically using SSH. Just like local debugging! Unfortunately, it quickly became very clear that it's meant for embedded devices. There's a strong focus on setting up a cross-compiling toolchain, compiling locally, and only then doing the remote debugging. Sorry, ain't nobody got time for that.

But QtCreator is flexible enough to allow for custom deployment, and I finally cracked it! This is how you get

## Remote debugging on the Librem 5

Instructions for QtCreator 4.13.2.

### Devices and Kits

Let's make QtCreator know that you have a device you want to take care of. Go to Go to Tools → Options → Devices → Devices tab → Add. You'll be presented with a choice, select "Generic Linux device". Fill in choices in "Connection". I chose the name "evergreen". The next screen will let you choose the SSH key to use. If you're confused here, you should read up about SSH key-based authentication. QtCreator will open SSH connections to the Librem 5, so you don't want to get password prompts all the time (once you have the key, make sure it's in your keychain. `ssh-add ~/.ssh/my_key.pub` works for me).

QtCreator has a notion of "kits", which define the set of libraries used for building a project. We'll need to define one on the new device for the Librem 5. Go to Tools → Options → Kits, and add a new one. Duplicating the default one also works, I think most fields don't matter. Choose the name for your kit (I chose "evergreen" again), change device type to "Generic Linux Device", and select the device you just added. Make sure to select some compiler, otherwise you'll get complaints from QtCreator later.

### Building and deployment

You need to import a project if you want to compile and then debug things. I'm going to skip this – you can easily look up some tutorials online – and go straight to the details regarding remote deployment.

Deployment is configured in the "projects" section on the left panel. I select the project I'm interested in (simplecam), and notice that, in addition to the default "Desktop" kit, there's also an "evergreen".

There are two positions here: "Build" and "Run".

#### Build

Let's rework "Build" first. Make sure that there are no build or clean steps. (If you intend to build the project manually, you can now skip to the "Run" subsection.)

We're going to use this step to synchronize git sources between the local checkout and the remote one. That means we need to create a checkout on the remote host. Adjust the following command to match your preference, and issue it:

```
ssh purism@10.42.0.185 git init ~/simple-cam
```

Now, add the new repository as your remote in the local git repository (adjust to match the remote host):

```
git remote add evergreen purism@10.42.0.185:~/simple-cam
git push evergreen master:foo
```

Now come back to the remote host and initialize the build directory.

```
# this is on purism@10.42.0.185
mkdir simbuild
cd simbuild
meson ~/simple-cam
```

Now, save the following script somewhere on your local computer. It'll be responsible for deploying each version. **Make sure to adjust the paths if yours are different!**

```
#!/bin/sh
set -e

REMOTE=purism@10.42.0.185
DEST="~/simple-cam"

ssh $REMOTE "git -C $DEST checkout -f foo"
git push evergreen -f HEAD:master
ssh $REMOTE "git -C $DEST checkout master"
git push evergreen -f HEAD:foo
ssh $REMOTE "cd ~/simbuild && meson $DEST && ninja && ninja install"
```

Finally, go back to QtCreator's "Project" section, and choose the "Build" configuration for Evergreen. Add a process step, and point it to the file you just created. Make sure it executes in the git checkout by setting "working directory" to "%{sourceDir}".

I used "sh" as "Command", and the path to the script (redacted) as Arguments.

### Run

Here's the meat of the operation. You're given one method: "Deploy to Remote Linux Host". We don't actually deploy anything here, because the executables are built on the remote host. Remove all deployment steps.

Add a new run configuration on the remote host. For me, it's called "Custom Executable (on Evergreen)". Add the executable path on the remote host. I use: `/home/purism/simbuild/simple-cam`. "Local executable" can stay empty. Environment changes take effect as expected.

## Debugger

When I tried debugging at first, I was dropped into disassembly mode. That is clearly not optimal. A lot of the appeal of a debugger is seeing the source code being debugged.

Fixing this involves some extra steps. First of all, this affects only some binaries. For me, what was not picked up was what I installed manually: the simple-cam binary, and libcamera. To fix this, you need to find the correct mapping between the files on the local computer and the paths embedded in the remote executable.

This method worked for me: I added a new breakpoint at "main" **by function name**, and started debugging. Make sure to select the correct configuration: it's on the left panel, above the "Run" button. Select your project, the kit you just made, and the remote executable.

Then, I opened the debugger log via View → Views → Debugger Log. Put the cursor in the "Command" prompt while the program is paused on disassembly, and type "bt". In the right pane you'll get something like:

```
1918bt
>&"bt\n"
>~"#0  main () at ../simple-cam/simple-cam.cpp:139\n"
>1918^done
```

This shows the path embedded in the binary that QtCreator (or GDB?) failed to find. Now you need to map it to a path on your local file system. Open Tools → Options → Debugger → General, and add the paths to the "Source Path Mapping" table. Here, I enter "../simple-cam" as "source path", and my local path to "simple-cam" as "target path".

Keep in mind that I redacted part of the local paths.

You can stop the debugging now, and start it again. Now you should see the sources for the paths you configured. You should see something like this in the "Application Output" tab on the bottom:

```
18:09:20: Checking available ports...
18:09:21: Found 101 free ports.
18:09:21: Starting gdbserver --multi :10000...
18:09:21: Debugging starts
Listening on port 10000
Remote debugging from host ::ffff:10.42.0.1, port 44146
Process /home/purism/simbuild/simple-cam created; pid = 22866
File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
```

Congratulations, you are now debugging your Librem 5 from your workstation!

### Debugging symbols

If you need to debug a call into a library that isn't part of your application, you may see that this setup balks. It either ignores the call, drops down to assembly, or displays "??" in the stack trace.

This is a telltale symbol of missing debugging information for the library that the remote host is executing. But even if you install the debugging symbols and sources (in Fedora it's `dnf debuginfo-install [name]`, in Debian it's hell), on the next run you get… no change.

It turns out that the debug symbols and sources must be available on the host side of the debugging connection. Let's get the symbols over:

```
scp -r purism@10.42.0.185:/usr/lib/debug /somewhere/l5_debug/
```

and hook them up to GDB by adding a new line to Tools → Options → Debugger → GDB → Additional Startup Commands:

```
set debug-file-directory /somewhere/l5_debug/debug
```

Now you'll be able to see symbol names, but you still can't inspect the sources of the library. Sadly, you have to find the sources of the package on the remote end, and hook them up yourself in the "Source Path Mapping" table, like we did a couple paragraphs earlier. Copy if needed, no automation here.

## Limitations

This relies on git to push your changes to the remote host, so **you must commit your changes if you want to run them**. Otherwise the code shown may not match the code executed.

It could be adjusted for plain files, but this is what I have now.

When you try to debug libraries provided by the OS, make real sure that you're using the correct sources. Debugging something when the cursor is randomly one line off is hard to notice, and it makes for a frustrating debugging session.