Improving Debug Code Performance in EFL

I work on EFL, a cross-platform graphical toolkit written in C. I recently decided to improve one aspect of the experience for developers using the API (otherwise known as users) by making EFL provide more information and stricter sanity checks when developing and debugging applications. A key requirement was ease of use. With these requirements, the solution was obvious, unfortunately obvious solutions don’t always work as well as you expect.

The Obvious Solution: an Environment Variable

Using an environment variable sounds like a good idea, but it comes with one major, unacceptable flaw: a significant performance impact. Unfortunately, one of the places we wanted to collect debug information was Eo, a hot-path in EFL. Believe it or not, adding a simple check to see if debug-mode is enabled was enough to degrade performance. To illustrate, consider the following code:

Note: thanks to Krister Walfridsson for pointing out a mistake in the following example.

#include <stdio.h>
#include <stdlib.h>

#define N 100000000 // Adjust for your machine

int main(int argc, char *argv[])
{
   int debug_on = !!getenv("DEBUG");
   unsigned long sum = 0;
   unsigned int i;

   for (i = 0 ; i < N ; i++)
     {
        unsigned int j;
        for (j = 0 ; j < N ; j++)
          {
#ifdef WITH_DEBUG
             if (debug_on)
                printf("Debug print.\n");
#endif

             sum += i;
          }
     }
   printf("Sum: %lu\n", sum);

   return 0;
}

Which when executed would result in:

$ # Debug disabled
$ gcc -O3 sum.c -o sum
$ time ./sum
Sum: 996882102603448320
./sum  0.11s user 0.00s system 95% cpu 0.118 total

$ # Debug enabled
$ gcc -O3 -DWITH_DEBUG=1 sum.c -o sum
$ time ./sum
Sum: 996882102603448320
./sum 0.17s user 0.00s system 98% cpu 0.176 total

This creates a 50% performance penalty just to have it compiled in;, this doesn’t even include enabling it! While this is an extreme example, code-paths with a similar impact do exist in our codebase.

This performance penalty was unacceptable, so I went back to the drawing board.

An Alternative Solution: a Compilation Option when Compiling the Library

The next idea was to add an option to compile the library in debug mode, so it will always collect the data whenever it’s used. In EFL’s source code we would have something like the following code:

EAPI Eo *efl_ref(Eo *obj)
{
   // snip ...
#ifdef EFL_DEBUG
   collect_debug_info();
#endif
   // snip ...
}

API users will have to recompile the library with debugging on and use it instead of the version included with the distro. This introduces a few issues, some of which are solvable, some are not. The primary issue was that users had to download and compile the library: a big enough hurdle to render this solution almost useless.

This had been the implementation for quite some time because it was good enough for the most of us, but I recently decided to  implement a better, user-friendly solution I had in mind.

A Better Solution: Automatically Provide Both Versions

Eo is small, so compiling it twice wouldn’t have a significant impact on either the build time or the deliverable size. Therefore, I modified our build-system to build Eo twice, once as libeo.so and once as libeo_dbg.so: the first version compiled as normal, and the second with EFL_DEBUG set.

This was already a better solution because it meant users of the API could simply link against the new library when compiling their applications to use the debug version, and when they are ready to release, link against the non-debug one. This, while much better, still requires some advanced know-how from the users that I wanted to eliminate.

An Even Better Solution: Use DLL Injection to Simplify Usage

I wanted to create an easy way for users to toggle between the two at runtime, without having to relink their applications. I realized I could use LD_PRELOAD (more info) in order to force the debug version of the library to be loaded first; problem solved.

I wrote a small script to ease usage:

#!/bin/sh
# The variables surrounded by @ are replaced by autotools
prefix="@prefix@"
exec_prefix="@exec_prefix@"
if [ $# -lt 1 ]
then
   echo "Usage: $0 <executable> [executable parameters]"
else
   LD_PRELOAD="@libdir@/libeo_dbg.so" "$@"
fi

Using this is now as simple as wrapping execution with the previous script:

$ # Debug off
$ ./myapp param1 param2

$ # Debug on
$ eo_debug ./myapp param1 param2

Mission accomplished!

Note: The script and libeo_dbg.so are automatically built and installed with the rest of EFL. This means they will be available for everyone who installs the efl package, or in some distributions the efl-dev package.

Conclusion

We now have an easy-to-use tool for API users to debug and check their applications. As a side effect, we also earned a useful tool for end users (users who use applications written with EFL) for providing better information when reporting bugs. All of this without any impact on normal execution.

Please let me know if you spotted any mistakes or have any suggestions for improvement.

This article was originally posted on Tom’s personal blog, and has been approved to be posted here.

Author: Tom Hacohen

Tom has been a core developer and part of the leading team at SHR (Openmoko), he is currently a core developer for EFL.