Peek and Poke, Vol 4

Published Friday October 15th, 2010 | by

Qt Creator 2.1 is looming over the horizon, and therefore, the time has come to add another chapter to the Tale of Debugging.

To keep it short, I’ll try to focus on two issues: architectural changes in the 2.1 cycle and some kind of update, long overdue, on extending the debugger to display your own data types.

Architectural Changes

There have been quite a few fundamental changes to Qt Creator’s debugger plugin during the recent development cycle. They were mainly driven by the need to extend the debugger scope beyond the original C++ debugger front-end to allow mixed C++/JavaScript/QML debugging.

Incidentally, the results are beneficial also for the more traditional use cases. Previously, we could take static snapshots of a process and jump between them to compare variable contents, memory, and so on. However, the first such jump had to kill the live process, because most of the debugger plugin was a singleton (nobody wants to debug more than one process at a time, right? ;-}), and before switching the data models to one of the snapshots, the live process had to make way. Not anymore. We can now have as many engines running at the same time as we want, so the live process engine can just continue while examining a snapshot. Not to mention that switching between snapshots is faster now…

Talking about performance: While the CDB integration (that is used to access binaries compiled with the Microsoft compiler on Windows) is still far from snappy, startup times have been reduced by 15 seconds. Each, time, the, debugger, is, started. That makes it bearable at least.

In the Free gdb world (i.e. Linux, Symbian, Maemo, Windows/MinGW) we now can take advantage of gdb’s shiny new “Dwarf Index” generation. This significantly reduces warm start times. For the Qt Creator project itself (which consists of more than a half a million lines in .cpp by now, not to mention headers or generated code, pulling in more than a hundred shared objects) I now get a warm start inside the debugger in 13 seconds (and a cold start in around 40). By leaving out non-vital plugins this can go down (depending on the definition of vital) to four(!) seconds. For smaller projects, such as the Qt examples, this is barely noticeable anymore. So you now really have no reason not to use Qt Creator’s gdb integration out of fear of falling asleep while waiting for application startup.

And getting started with debugging is still as simple as it used to be: Load or create a project. Press F5.

That’s it. No magic dancing needed on your side, not launch configuration to setup, no black roosters harmed in the process. For having a quick look at a core file or a running process, you can even skip the project bit and directly start Qt Creator with the -debug <name_of_core_file> or -debug <process_id_of_running_process> command line arguments.


And now for something completely different:

(Re-)Introduction to Data Dumpers for the Qt Creator gdb front-end

The topic has been covered in the first and second post, but by now, so many details have changed that following the process outlined there will not directly lead to success.

So let’s start over.

The beast has appeared under different names in the past: Data Dumpers, Pretty Printers, Debugging Helpers. I guess none of them is really to-the-point anymore, but in my brain it’s still “Data Dumpers”. It’s all about examining data structures at a higher level. No funny d-Pointers, vector data hidden behind void *, image data as hex string (or not visible at all), and so on. I really don’t want to see any of that when debugging code using containers. Or a QObject. All I want to do then is to see the actual contents of the container or the properties of the QObject I am working with, and maybe have a way to navigate to the objects that are connected to it.

That need led to the original incarnation of the Dumpers, as introduced in the first blog. They worked, but a recurring request since then – starting with the comments to the blog post – was to make that easily extensible by users of Qt Creator.

By now, and a change of technology later, I think we are there. Gdb nowadays supports Python as a scripting language which is way more complete and especially faster than the old canned sequences of commands. It even has built-in pretty printing but, while this is nice and usable on the command line, it has some severe limitations that makes it unusable for our purpose. In a display hierarchy, it creates only one level at a time, which makes virtual data, such as the “signals” group subnode in a QObject, difficult to display, and the MI “var object” machinery impossible to use. Not to mention robustness issues in case of uninitialized objects which are rather common in C++. Luckily, all this does not matter: Having a proper scripting language is all that’s needed.

So… how to get started?

Suppose we have the following code:

QUrl url(QString("http://www.nokia.com"));

Traditionally, the debugger would show you a tree, as follows:

I think “Eeeks!” is about the most common reaction to such a display of “information”.

How did I find out the data was visible in encodedOriginal?

By trial and error. And note that I was lucky that it was a QByteArray which happens to have a char * member for the data which happens to be handled natively by gdb. If it had been a QString, it would have been a ushort * and all I would have seen would have been the numerical value ’104′.

I’d pretty much prefer to see something like:

at the top-level, instead, with the option (but not the need) to look at the details — even if they could look a bit nicer:

So what do I do? Knowing that Qt Creator will set up its Python-based data dumpers anyway, I just need to have: (a) a dumper for QUrl, and (b) inject it into the process.

(a) is easy: We need to define a python function  qdump__QUrl which will be automatically called when an object of the type QUrl (or myns::QUrl when Qt was built in the namespace myns) is encountered:

def qdump__QUrl(d, item):
    d.putStringValue(item.value["d"].dereference()["encodedOriginal"])
    d.putNumChild(0)

The function basically writes a put QString item taken from url.d->encodedOriginal into the value column of the Locals&Watchers view and disallows further expansion.

If expansion is wanted, a full implementation of the dumper would look something like the following:

def qdump__QUrl(d, item):
    d.putStringValue(item.value["d"].dereference()["encodedOriginal"])
    d.putNumChild(1)    # Any non-0 value will do. A wart...
    if d.isExpanded(item):
        with Children(d):
            d.putFields(Item(data, item.iname))

 

(b) is easy, too: There is a Gdb Startup Script setting in the gdb options page. This points to a script that is parsed by gdb after the Python machinery is set up. Therefore, we can just write the function above into a mydumpers.py file and point the setting there. Unfortunately (or fortunately, as it speeds up things), the list of known dumpers is cached, so the cache needs to be flushed. There are several ways to do that. The simplest, but least obvious, is to add a line containing only bbsetup() at the end of mydumpers.py. (Smells like a bug, doesn’t it? I guess I’d better fix that in 2.2…)

It is also possible to implement several ways to display data of a given type, and switch between them on-the-fly, globally or on a per-object base. Qt Creator uses that by itself to allow the selection of different encodings for string-like data, such as char * or wchar_t *, or to allow the viewing of data in separate windows.

I’ll spare you the details for now and rather finish with the list of latest additions to the Dumper Zoo: New in 2.1 are QDate, QTime, QHostAddress, QRegion, boost::optional, QSharedData, QSharedDataPointer and QTextCursor. Some others like QObject, QList, C-style arrays, QSize, QSizeF, QImage/QPixmap, std::vector, QVariant(QString), qulonglong, std::size_t, std::ptrdiff_t, QScopedPointer, QStandardItem have seen some bug fixes or improvements. The most notable change here is that QObject now displays dynamic user defined properties.

Did you like this? Share it:

Posted in C++, Qt, QtCreator

31 comments to Peek and Poke, Vol 4

Jérémie says:

Great news!

The debugger improvements and the recent llvm 2.8 release with clang compiling C++ correctly makes me wondering if you have any plan to support clang and lldb (when it will be usable!) into Qt Creator?

Don says:

I’d like to vote for Jérémie’s point, and raise it, by suggesting you make llvm (well clang) the preferred compiler that runs on all platforms and compiles for those platforms as well as Symbian, a single awesome compiler with deep integration in Qt Creator, without needing the mingw or old gcc’s or proprietary ARM compilers or worrying about which gdb is new enough or has python support or is fast enough :)

By the way these blogs are extremely interesting, and it’s awesome to be able to “git pull” the Qt repository and see Nokia improve and fix things almost in real time. I don’t know how many there are of you, but I’d buy you all a beer :) Keep up the great work. And get this 4.7 for S^3 out of the door :)

Stephen Chu says:

None of the improvements here apply to OS X version, right? :(

André says:

@Jérémie, Don: The interest clearly exists, but I don’t think that lldb is already “there” right now. It seems to work (ok-ish) on Mac, but it {b|r}arely compiles on Linux and I haven’t even dared to look at it under Windows. So at the moment it’s far less cross-platform than gdb, but I certainly see a huge potential there.

Mad Fish says:

+1 for clang. This would allow perfect code completion, features like “go to definition”, refactoring, etc, and all of those with FULL knowledge of the code. Really, clang is the future, move to it.

André says:

@Stephen: The “multiple engines” bit is cross-platform. The additional python dumpers could in theory be ported to C++ to be used in the compiled version (and would also benefit Windows/MSVC then), and so could the “per-type display switching”, but it’s a bit of a hassle. If you need one or two of them specifically, create a feature request on bugreport.qt.nokia.com and I’ll have a look, but I won’t subscribe to full feature parity for Apple’s gdb ;-}

Another option is of course to try to get FSF gdb up and running on Mac. The first big obstacle on that route is known, though: http://sourceware.org/bugzilla/show_bug.cgi?id=11488

TemporalBeing says:

Any chance these data dumpers could be published so we can use them with standard GDB – even when not using Qt Creator?
It would be very helpful for those of us directly using GDB 7.x.

André says:

@Mad Fish: The sole fact that something is the future does not necessarily make it a feasible option for the day. As I mentioned, lldb based debugging will you buy exactly _nothing_ on non-Mac right now, and I actually don’t think this will change before Christmas.

People tend to equate the availability of a perfect compiler with the availability of something that’s perfectly suited for “editing support”. Unfortunately, that’s not the case. A compiler is only required to work on well-formed code. During editing you almost always have “broken” code, and you also have real time requirements that are typically not met by a “let’s throw the compiler at the full translation unit on each keystroke”. Also, for editing it’s worth spending a thought or two on what should happen to #ifdef FOOBAR’ed code during refactoring. Should it really be handled the “compiler way” (i.e. ignored if the condition is not met) and break your colleague’s build?

Nevertheless: I already admitted that I see a huge potential in this direction, so rest assured clang and lldb will not fall through the cracks over here ;-)

André says:

@TemporalBeing: These data dumpers are already published: http://qt.gitorious.org/qt-creator/qt-creator/blobs/2.1/share/qtcreator/gdbmacros/gdbmacros.py (and dumpers.py for the “framework”). It’s all LGPL, too. Using them directly on the gdb command line might be a bit tricky, though, as it creates responses in the gdb/MI format which is geared towards machine-readability and not meant for human consumptions.

Don says:

André: “Nevertheless: I already admitted that I see a huge potential in this direction, so rest assured clang and lldb will not fall through the cracks over here”

That’s all you needed to say :)

Anshul Mehta says:

Hi Andre
Suppose in my code I do :
typedef std::vector pdkvVecS

Now I create objects of type pdkvVecS. But pre compiled debugging helpers don’t work on them.I think the debugging helpers are capable of viewing contents of std::string but due to this typdef it can’t handle this situation. What is the way out ? Is there any need to change the debugging helpers code for this situation ? How will python based debugging helpers react to this situation ?

On a different context I’m still not using python based debugging helper since compiling a python enabled gdb is a bit pain. I managed to do it once but still wasn’t able to see QString with this gdb. Is there some specific python version I require? I compiled gdb 7.1 from sources

Anshul Mehta says:

In my previous post I meant:

typedef std::vector pdkvVecS

NOT

typedef std::vector pdkvVecS

Andre' says:

@TemporalBeing: On second thoughts, it might be pretty straightforward: locate ‘putField’ and similar functions in dumper.py and replace the ‘%s=”%s” % ( …, …) stuff by something less noisy, use a newline instead of a comma to separate fields, introduce a depth counter for indentation in the Children class and get rid of most of the string encoding and any data fields that are not ‘name’, ‘value’, or ‘type’.

Andre' says:

@Anshul Meshta: The inability of resolving typedefs is a known limitation of the compiled dumpers. It is possible in theory but would involve a prohibitive amount of extra roundtrips through gdb. Trust me, you would not like the result ;-)

Compiling gdb with python is (except for Window) just installing some suitable python development package, ./configure && make. Minimum required version is (I believe…) 2.5. Python 2.6 is a safe choice. Making it work on Windows is admittedly a pain, but you could just download the Qt SDK and use the prebuild gdb that’s in there. Actually, self-compiling on Linux should not be necessary either, the distributions seem to be reasonably up-to-date (i.e. have a python-enabled gdb 7.0.1 or newer) nowadays.

Mad Fish says:

One more thing: Why there is no GDB console in Qt Creator? GUI frontend is not always the best choice, especially when it’s quite limited (at least as it is in current, 2.0, version). Interactive GDB prompt will help a lot.

Andre' says:

@Mad Fish: The real reason? Because the input pane in the log view is good enough for me personally to enter the occasional direct gdb command, and so far nobody filed a merge request implementing the feature. I definitely feel not tempted to play around with real ttys, cross-platform, just to “have it.”

The topic “Why no console?” comes up every now and then, but nobody actually listed real use cases that are only solvable (or at least much easier) with a “real” gdb console. [And no, "I am used to type 'n' and then on gdb cli and can't be bothered to press F10 in Creator instead" does not count ;-) ]

Anshul Mehta says:

@Andre
Ok..I’ll somehow manage to get a python enabled gdb . But will this python enabled gdb be able to handle typedef’s ??

Mad Fish says:

@Andre’
“the input pane in the log view”. I don’t get it, where it is? There is Window>Views>Debugger, but it doesn’t allow me to enter commands.

dt says:

It’s Ctrl+Enter to send the commands, that’s probably not obvious.

Mad Fish says:

Also, there is currently no memory view. How am I supposed to look into memory buffer with raw binary data with GUI?

Btw, here are some real-life use cases for GDB console:
1. Print the variable, so you can copy-paste it for later reference.
2. Call arbitrary function, calculate expressions (immediate mode).
3. Format data in different ways, depending on context. For example, printing 32 words from raw memory buffer in hex: “x/32wx data”. Can you provide THAT flexibility with a GUI? I don’t think so.

Andre' says:

Select “Window->Views->Memory…” to get a memory view, or “Open Memory Editor at Object’s address” in the Locals and Watchers context menu, or “Open Memory Editor at [current stack frame]” in the Stack view’s context menu.

Regarding the other use cases you gave for a console, I think they are already covered rather well, to a degree that I’d even argue that it is more flexible than the console output:

1. Locals and Watchers, context menu, “Show View Contents in Editor”. This also allows to select the depth you need to look at children on a case-by-case base.
— or –
Create a snapshot of the current process state and change back to it when you want to see the “old” data again. This even allows you to expand items further after switching back and forth.

2. Locals and Watchers, context menu, “Insert New Watch Item” (or double click in an ampty part of Locals and Watchers). Put the expression there. This also allows to expand the children properly or to filter them (see Vol 3).

3. Locals and Watchers, context menu, “Insert New Watch Item” (or double click in an ampty part of Locals and Watchers). Put {int[32]}data there. Context menu, “Change Format for Type ‘int’”, Select “hexadecimal”.

Mad Fish says:

Well, in 2.0 there is no “Window->Views->Memory”. :) But “Open Memory Editor” is available, I’ve missed that.
“Show View Contents in Editor” isn’t there in 2.0. Well, so I’m looking forward for 2.1.
For 2nd use case, what if I want to call a function that has side effects? Putting it into a watch doesn’t seem to be a good idea.

Also, why not just make interface for running GDB commands (which is already there) more friendly? Like a single input box, where you can type a command, and a text view that allows you to view the result. That will not require dealing with real ttys. I think that will be enough (but will miss GDB’s tab completion, however).

Anyway, thanks for your response.

Anshul Mehta says:

Hi Andre
I think you missed my previous comment.Will this python enabled gdb be able to handle typedef’s ??

Andre' says:

Hi Anshul. With a python enabled gdb we can handle typedefs properly. In this case, gdb is “only” used to retrieve information like “is it a typedef?”, “what members does the object have?” etc, and any formatting is done in a python script that’s loaded into gdb on startup. We have full control over that script.

Anshul Mehta says:

Great..Thanks ! I’ll looking forward to work with a python enabled gdb when I get time.

silvansky says:

I think that it would be a great improvement if you add a possibility to debug dlls/plugins on a symbian devices. For now only exes are available to debug.

Love the blog, thank you for the excellent work, and of course letting us get a peek into your development process. Great stuff!

That said, I did have a quick comment/question on the subjects of debugging:

In Mac OS X 10.6 & Qt Creator 2.0.1, I can create simple SIMD code like:

float fa1[] = {1, 2, 3, 4};
float fa2[4];

__m128 *v4 = (__m128*)fa1;
__m128 *v5 = (__m128*)fa2;

*v5 = _mm_mul_ps(*v4, *v4);

And the __m128′s are fully visible to me in terms of the value’s they currently contain. That is, in the debugger I have an arrow that when clicked, drops down and shows me the 4 floats contained in the vectors.

However, in Ubuntu 10.10, both 32- and 64-bit, only the float[] items show extended value, not the __m128′s. The __m128s show the arrow, but when clicked, nothing else shows.

This isn’t a huge deal, as I use the pointer/float[] method for most code anyway, but it would be nice if worked on both platforms. This is especially so because as it works for one but not the other it feels as if a solution has to be close at hand.

Thoughts or ideas? I love the idea of being able to extend the debugger, perhaps this would be a good line of attack.

PS. Upon further reflection I’m pretty sure it works on OSX, but not 100%. Doesn’t change the overall question though : )

W. Dobbe says:

Thanks for the continuing story.

In our company we couldn’t get the python helpers to work for some time. I found out this is because we have set PYTHONPATH in our environment (we use some python stuff in our build system).
In order to get the python helpers to work I have to set PYTHONPATH to C:Qtqtcreator-2.0.92pythongdbpython2.5;C:Qtqtcreator-2.0.92shareqtcreatorgdbmacros
Haven’t checked yet if this also the case on linux.

It would be nicer if QtCreator wasn’t influenced by the PYTHONPATH variable in the user’s environment….

regards ,
Winfried

ShD says:

André, you really should take a closer look at clang. clang is not meant to be only a compiler. it was designed to be used in an ide for all parsing needs. clang can produce full unoptimized asts, unlike gcc and most other compilers, that includes everything up to the c preprocessor stuff. so it won’t break your preprocessor code while refactoring. it currently has much better error recovering than gcc, probably the best of any c compiler (not sure about c++, as c++ support of clang is still quite new). it goes to quite a few lengths to try and interpret illegal code, and produces very sensible error (and warning) messages along the way to tell you what would be needed to turn this mess into something it could compile to a real binary ;) .

it is also very fast, and, as far is i know, has at least partial support for recompiling only parts of a changed file.

clang IS the future of any c/c++ parsing needs. it is not only a compiler.
BUT there is still much to do. it currently does not have a stable api for most of its features – especially for ide integration. though this means now is the time to look at the project. the clang developers are most certainly interested in what would be needed to replace your custom c++ parser as well as integrate into your code completion and refactoring back ends. real world use cases will make clang ready for ide integration much faster than just waiting and hoping they will get everything right on their own.

Andre' says:

@ShD: How is clang related to the topic of the blog post?

Andre' says:

Hello Winfried.

It was a conscious decision that a PYTHONPATH set by the user is not overwritten by Qt Creator by default but honored as the user’s explicit wish. It still can be replaced in the projects run settings, so there is no loss of functionality.

Andre’

Commenting closed.