Everybody hates bugs
[citation needed]. Why spend time hunting down bugs when you could be shipping code and earning money? This is especially true of low-level memory management bugs in C/C++. If you’re a newbie, words like memory leak, use-after-free or double-delete may not have an effect on you but any experienced developer will recoil at these words in horror or start chanting “asan, asan, asan, …” So, OK,
we don’t want these bugs in our code, let them go somewhere else. This is especially true in my field of crypto engineering where bugs in crypto code or protocols can lead to complete failure of the projects. And
bugs in
ordinary programs can cause security problems just as well.
The weapons I choose for this battle are Modern C++, automated testing, static analysis from `the compiler, dynamic analysis tools, fuzzers and patience.
Some people don’t like C++ or code C++ like it was plain old C. Sure, those Linux kernel programmers have a quantum computer in their head that lets them simulate all possible program paths at the same time to see where a lock or memory are not released. But I’m just a human and can barely keep track of the socks in my drawer so I have to find another way, one that is more fool-proof and doesn’t require so many socks.
So here are my top
n tools and techniques to make my C++ less buggy, crashing less often and more reliable. It’s totally not a complete list but rather an introduction, a base level of tooling that everybody should know about but somehow not always does.
1 Deleting objects
C doesn’t have a garbage collector so we have to clean up our garbage manually. If we don’t, the program will eventually drown in garbage and run out of memory (but not before trashing the hard drive by swapping). The old and deprecated option is to do a
delete obj; manually. This is completely unreliable (because it’s manual). You may forget to do this when having multiple exits from a method. Or when a method throws an exception. Or when another method you call throws an exception. Not to mention returning objects that the caller then has to free.
The
modern approach is to use RAII which is a weird name for a simple concept: use the destructor to do all cleanup at the end of scope. If we have a scope with an object on the stack like this:
{
VictorTheCleaner v;
...
}
then the destructor of
v will be called when the scope is exited and it can take care of any deleting, releasing and cleaning that’s necessary. A scope can be exited in several ways:
- normal program flow reaches the
} brace
- a statement such as
return, breakor continue causes program flow to jump out
- an exception is thrown
These are cases where you would have to manually do a delete and where the destructor will do it for you. The most common usage is a smart pointer that “owns” a non-smart (stupid) pointer and is responsible for deleting it. Another usage can be making sure you close a database connection. Or close a file. Or a network socket. Or change the process current directory to what it was before. This is equivalent to the
using clause in languages such as C# and Python. C++ doesn’t have a dedicated keyword, instead we use the destructor.
1 2 unique_ptr
The standard library template class
std::unique_ptr<T> is designed to take care of the most common case - when you need to automatically delete an object. Use
std::unique_ptr<T>. Use it for function local variables, use it as object members (to be safe if your constructor throws). Use it
especially for C land objects that are created with functions like
EC_POINT_new() and must be deallocated with
EC_POINT_free(). This is how you can set a user defined function as the deleter:
template<typename T, void >
class function_deleter ;
};
template<typename T, void >
class unique_ptr_ex ;
The first class defines a functor (something that has
operator()). This is necessary because the second template parameter of
unique_ptr is a type. The purpose of the second class is to act as a
typedef with a parameter. It’s a workaround because not all compilers I’m using support the new
template typedef in C++11. These two classes simplify creating
unique_ptr with different cleanup functions:
unique_ptr_ex<BIGNUM, BN_free>::type m_privkey;
unique_ptr_ex<EC_POINT, EC_POINT_free>::type m_ec;
Then just initialize
m_privkey with a new BIGNUM that belongs to you and deallocation using the openssl-provided function
BN_free is taken care of automagically. Same for
m_ec!
For details how to use
unique_ptr,
see the reference. I’ll try to give a few basic guides here.
- Use it for local variables that you created in the function and need to clean in the same place.
- Use it for member variables of a class that “owns” these variables. Owning them means that the class is responsible for cleaning them which typically happens when the class itself is deleted.
- You can also use them as return types for functions that create an object and pass its ownership (responsibility for deleting) to the calling function. No more uncertainty on who should do the cleaning,
unique_ptr tells you you are the responsible owner and will do it automatically for you too.
- Do not use it for object pointers that do not transfer ownership. If I call a function that uses an existing object, I use a naked pointer or a reference (if it can’t be null).
- Do not use it for member variables of a class if the class is not owning that object.
The last case is a little tricky. In some cases, more than one class are responsible for a cleanup job and that’s where a smarter smart pointer such as
shared_ptr would come in. I try to keep things simple and have only one owner for each object so that I can use
unique_ptr.
1 3 Other cleanup
Sometimes you need to do more things besides deleting an object upon scope exit. Maybe you need to close a database connection, restore the original value or roll back an object to a previous state.
You could create a new class with the appropriate code in destructor for each of these cases (for example
PostgreCloser,
PwdRestorer, …) but that is a little inconvenient. That’s why there is
ScopeGuard, a class where you can redefine the cleanup code in-place.
2 Program invariants
A short intermission is necessary to explain the term
invariant. The term literally means “something that doesn’t change” and in programming that will be a condition (a logical statement) that mustn’t change and be always true while the program is running because the code relies on it and things would break otherwise. There can be invariants on particular lines in the code (at the start of a function, in a loop) or they are related to a data structure or an OOP class. Invariants are usually not expressed in the programming language itself, it’s just something that we keep in mind and use when thinking about how the code will behave. Or at least should keep in mind. Ideally.
For example a data structure invariant for binary search tree is that each node has at most 2 children. A more interesting invariant requires that nodes under the left child are all less than the current node and all under the right child are more than the current node. If this condition doesn’t hold then efficient search in binary search tree will be broken.
A string splitting function will have an invariant at the end of the function (called post-condition) that says that the two returned parts, actually form the original string if reassembled. A splitting function would not otherwise be very useful. A memory allocator will have an invariant stating that all active allocations are kept track of so that a further allocation cannot occur in a piece of memory already given to someone else.
Invariants help us describe program behaviour and requirements using logic. At this time, mainstream programming languages don’t have any support for working with them. But there are languages that focus on correctness in the academic research community and they let you write those invariants in the code for formal checking.
3 Error handling
I’m coding a
Modern C++ interface around
some library from C land and using exceptions to signify any errors because I don’t want to deal with manual propagation of error codes up the stack and I always want to know when an error happens[ (see here)]](
https://www.securecoding.cert.org/confluence/display/cplusplus/ERR02-CPP.+Avoid+in-band+error+indicators). Manual error code propagation clutters the code, makes it harder to grasp and is prone to human errors. Modern C++ / OOP programming encourages proper object initialization in the constructor (as opposed to an additional
Init() method) and exceptions are the only way to report errors from there (
also note this Boost article).
Now some people don’t
like exceptions in C++ but I found that most (not all) of their arguments are based on limitations in the old versions of the language or simply ignorance of how exactly the language works. Be sure to get familiar with recent development in the C++ language, most significantly the RAII pattern. Of course exceptions have some drawbacks too and some properties that need to be kept in mind (or your quantum computer brain):
- Throwing and catching exceptions is s……l..o…w. If you expect this to happen often, for example in parsing code, you need error codes (or
Expected, see below) see comparison.
- Throwing an exception in a destructor is very destructive. It will probably crash your program (see here for an example).
- Throwing an exception in the constructor means the object construction was cancelled and destructor won’t be called. That makes sense but also could mean that a
delete m_obj; in the destructor may never be invoked (again, example here) even though you have already new’ed it in constructor. That is one more reason to use unique_ptr for member variables since these variables will be protected against a sudden death by exception from constructor.
- Only one exception can be in flight for a thread at a moment. This means that the exception model cannot support async callback based programming aka. Node.js or Python Twisted and you need to store exceptions manually (mentioned in this talk).
- Exceptions can appear anywhere and there’s nothing in the code that will warn you about them.
If exceptions are not an option, you’ll be returning error codes. I’d suggest a slightly more sophisticated approach. The
Expected type can be returned by functions that are supposed to produce a result but could also fail. It has type parameters for the result value as well as for the error. Then (for example) a parsing library could use the following interface:
struct ParseError {
int line, col;
std::string expected;
};
Expected<int, ParseError> parseInt(std::string);
Expected<int, ParseError> parseHex(std::string);
Expected<Url, ParseError> parseUrl(std::string);
The
Expected class also seems to support the error monad programming style but that would be for another day :)
3 1 Error mis-handling
Beginners tend to ignore errors, I still remember I was doing it. I could barely manage to write the code to do what I wanted in the first place. But if we’re talking about reliable software, ignoring errors is unacceptable. Quite the opposite, we need to know each and every error that happened, is happening or may happen.
Errors that have happened need to be logged using an appropriate logging framework and in some cases may even be stored in a database for further analysis or sent to a remote monitoring server. See more here
Exception Driven Development
Errors that are happening right now need to be detected. If calling an external library that doesn’t throw exceptions but returns an error code,
check the returned code and throw, log or handle (retry?) the problem! From this point of view, exceptions (as opposed to returned error codes) really help because they are not ignored by default so the risk of forgetting to report an error is lower. But then again some newbies will very carefully put a
catch block around each function so that they can ignore the valuable exception object that bears details about the problem.
How about errors that may happen in the future? Try to anticipate possible problems in the future but don’t try to recover, auto-repair or anything like that. Instead, check invariants and assumptions about your data structures consistency using assertions that report problems immediately. For example, if you have a class that is not thread-safe and you designated it to be only accessed from a single thread, assert it (this is a trick I found in Chrome source code):
void Gui::UpdateBlinkies() {
assert(GetCurrentThread() == MainThread);
m_blinkie ++;
}
The point is, again, to discover any problems, inconsistencies and unexpected situations as soon as possible because you are better able to debug and fix the problem. If the program crashes 10 minutes after the problem started, how can you trace the crash 10 minutes back to its original cause?
Having some hard-to-debug problem that a client reports but you can never see on your machines? Then you should have a log file that provides additional information. If even that doesn’t help, rather than spending a week trying to reproduce the problem on your machine, you can ship a debug build to the customer that collects information that you need or one that has enabled
asserts. If you have used them well, that alone may be able to pinpoint the bug.
3 2 Exception safety
But no matter if you choose exceptions or error codes to handle those unusual unhappy cases, you still need to be careful about exception (or error safety). This means that you need to
1) release any resources that were acquired before an error
2) return the application into a consistent state (invariant safety)
Everybody should be pretty familiar with point #1 where doing some
new BigObject() must be always followed by a
delete even if you get an exception in between.
Point #2 is similar except that it is specific to your application invariants.
For example, if you are keeping some data in two structures and always need to update both of them on inserting, you need to make sure both things happen (or get rolled back) even if an exception is thrown:
void insert_both(string a, string b) {
m_by_name.insert(a, b);
DoSomethingElse();
m_by_addr.insert(b, a);
}
Handling this case could be still easy, just add a catch, remove the item and exit:
void insert_both(string a, string b) {
m_by_name.insert(a, b);
try {
DoSomethingElse();
m_by_addr.insert(b, a);
} catch (...) {
m_by_name.remove(a, b);
throw;
}
}
But if you need to do something like this twice in a function, it starts to get complicated. Fortunately, C++ provides an elegant and convenient way to take care of both requirements. Memory safety has already been described (remember
unique_ptr). Invariant safety can be done with a
ScopeGuard which is a more flexible alternative to
unique_ptr. There’s an implementation in the
Facebook’s folly library. For the above example with two maps, you could use it in the following way:
void insert_both(string a, string b) {
m_by_name.insert(a, b);
ScopeGuard insert_guard = makeGuard([&] { m_by_name.remove(a, b); });
DoSomethingElse();
m_by_addr.insert(b, a);
insert_guard.dismiss();
If any exception occurs in the code, the
insert_guard will execute the remove operation to restore the original state. If everything goes smoothly to the end of the function, the scope guard will be cancelled by the
dismiss() call.
This way you can have nice linear code which is easy to understand even if there are more than 1 rollbacks. Just imagine the scope guard as “do this cleanup if anything goes wrong down there” as opposed to having nested
try-catch clauses with many possible combinations of control flow.
3 2 1 Testing exceptions
When we test our code, we usually focus mostly on the ‘happy path’ where everything goes as planned and the edge cases receive less attention. But if we want to have truly reliable code, even those error or edge cases deserve some attention. If you use code coverage tools, they will keep flashing their red warnings in the exception handlers at you until you add them to ignore list (err, I mean, fix them).
Proper unit testing (covered later) of course requires also testing the error and edge cases. You should try to come up with possible incorrect inputs (or problematic program state) and know, for each of them, how the program should handle it. And write this down in an unit test. In this way both the “happy” and failure behaviours of the function are well documented and verified to be correct.
This approach for unit-level testing is well established. On the more coarse scale, there is another technique where we artificially throw exceptions and check that they are handled appropriately. We can throw exceptions at various places in the program and check general properties such as whether it causes memory leaks, memory corruption, or crashes. This is only relevant in languages such as C++ which have those memory issues by default.
To automate this, you would put
instrumentation points at interesting places in your program. Then you run your program or test suite over and over, triggering these instrumentation points in sequence. If each run of your program is deterministic (it takes the same path each time), you will have triggered each of the N points in the end, after running the test suite N times.
Since this is sorts of mass exception injection approach, we cannot test for specific behaviour of specific cases, only for overall response to exceptions. Memory correctness will be the most typical case. Another one could be ensuring that all those exceptions are properly logged. This method is very useful if you’re creating a binding to another programming language such as Java or Python or even plain old C. Typically you need to catch exceptions in the C++ world and translate them somehow into exceptions or at least error codes in the target language without messing up the memory or exception safety.
You also need to run this under a memory checker such as Valgrind, Asan or PageHeap which will inform you if any memory leak or access violation occurred. If all goes smoothly, you’ll know that exceptions can’t mess with you. It also probably means that you used RAII and
unique_ptr correctly because without them it’s hard to make memory management right in the face of exceptions.
This approach has also been described in
Exception-Safety in Generic Components.
This is how you may implement it:
#define INSTRUMENTATION_POINT { g_instrument->RunPoint(); }
class ExceptionInstrument {
public:
ExplosiveInstrumentator();
static ExplosiveInstrumentator &instance();
void dispose();
bool should_throw();
void maybe_throw(const std::string &file, int line);
static void instrument(const std::string &file, int line);
void next_run();
void set_run(int no) { m_run_id = no; }
private:
static ExplosiveInstrumentator *g_instance;
std::string get_filename_base(const std::string &path) const;
int m_run_id;
int m_counter;
int m_threw_cnt;
};
void ExceptionInstrument::dispose() {
if (g_instance != NULL) {
delete g_instance;
g_instance = NULL;
}
}
void ExceptionInstrument::maybe_throw(const std::string &file, int line) {
string basename = get_filename_base(file);
stringstream ss;
ss << "Instrumentation exception F " << basename << " L " << line;
if (should_throw()) throw std::runtime_error(ss.str());
}
void ExceptionInstrument::instrument(const std::string &file, int line) {
instance().maybe_throw(file, line);
}
ExceptionInstrument &ExceptionInstrument::instance() {
if (g_instance == NULL) {
g_instance = new ExplosiveInstrumentator();
}
return *g_instance;
}
std::string ExceptionInstrument::get_filename_base(const std::string &path) const {
string::size_type bk_pos = path.rfind('\\');
string::size_type fw_pos = path.rfind('/');
string::size_type pos;
if ((bk_pos != string::npos) && (fw_pos != string::npos)) {
pos = std::max(bk_pos, fw_pos);
} else if (bk_pos != string::npos) {
pos = bk_pos;
} else if (fw_pos != string::npos) {
pos = fw_pos;
} else {
return path;
}
return path.substr(pos);
}
bool ExceptionInstrument::should_throw() {
bool res = false;
if (m_counter == m_run_id) {
res = true;
m_threw_cnt += 1;
}
m_counter += 1;
return res;
}
void ExceptionInstrument::next_run() {
#ifdef ENABLE_INSTRUMENTATION_THROW
if (m_threw_cnt == 0) {
cerr << "Instrumentation run " << m_run_id << " threw no exceptions." << endl;
}
#endif
m_counter = 0;
m_threw_cnt = 0;
m_run_id += 1;
}
Another technique is using what I call
explosive mocks. They work as ordinary mocks but throw a random exception when called. They may help you test correct handling of exceptions that you didn’t expect when first writing the code. For example in connecting to a network API, all kinds of things can go wrong, from network, DNS problems to authentication, API changes, invalid parameters, …). It’s not a very systematic method but can be useful as
exploratory testing to find bugs.
QUESTION: how to handle exceptions in message loops / GUI / … in a way that is debuggable, readable, testable?
4 Automated testing
I use automated unit tests and (more or less) automated integration tests where it makes sense. This is a big and complicated topic and I have an article with a few of my observations in progress. Make sure to keep your interpipes to this blog clean so you don’t miss it ;)
In no way complete or sufficient but this is what I use.
5 1 Address Sanitizer (+ more)
If you write in
C++ (or god-forbid,
C), you’re going to have memory bugs. Well, unless you are using Address Sanitizer or Asan. This kind of bugs can cause your program to crash, which is a little annoying to see. But in some cases a crash can lead to a
Remote Code Execution exploit. RCE is basically when a snake whisperer (hacker) convinces your program to start executing some code the
hacker offered.. So yeah, that’s a little less convenient, especially when they use it to steal your money or data.
Asan helps you catch these bugs. Also memory leaks. It can’t detect every problem with your code, it only detects problems in code that you execute. So you still need a good test suite. It catches every access violation (segfault for unix folks) and prints beautiful coloured output in the console. And since everybody loves coloured console output (and not having segfaults), I hereby endorse using Asan for everything.
Originally developed for the clang compiler, it’s now available for gcc as well (sorry for people stuck with gcc 2.x)(I wonder if that gcc version is written on papyrus?).
Detailed usage is here but in short, you will need to create a new build variant for your project that will generate an Asan-instrumented build with the
-fsanitize=address flag. This build will be around 2x slower and will abort immediately when an access violation is detected. It does not report false positives, that abort will be something you’ll have to fix.
You may have used Valgrind. For memory access violations, Asan is similar but works better. It does require you to recompile the code with Asan enabled but then it’s much more accurate.
 |
| Asan output |
5 2 afl-fuzz
So even though awesome, Asan won’t catch problems in obscure code branches that don’t get executed. One thing you can be sure: hackers will try to find them and run them so that they can pwn your machine and steal your candy. Here come fuzzers, tools that are designed specifically to execute code paths that normally never see the light of day. They do it by running your code, like, a million times, each time with a slightly different input and observing whether it caused some different behaviour in your code.
This is mostly suitable for programs that read and parse some input files such as images, videos, PDFs or even
antivirus software reading
.exe files.
 |
| See, it has colours |
afl-fuzz is one such program, free, open source and pretty good. It’s not complete magic though. You need to adjust your project to do nothing but read the input data and sometimes you need to help the fuzzing process a bit with a hint.
afl-fuzz works by inserting instrumentation during compilation that informs the fuzzer where the control flow is going. Based on this instrumentation, it tries to alter the input data to find new control flow paths. And when it finds a control flow branch that crashes your code, it’ll happily returns the bad input. Programmers will take that sample input to go and fix the bug, hackers will take that sample to develop an exploit.
5 3 Catch
As far as automated testing frameworks for C++ are concerned, there’s quite a choice. You’ve got the one from
Google,
Boost, even
Visual Studio comes with one. I can’t compare them but I tried
Catch and enjoyed it very much. So besides the #1 requirement of coloured console input, it has the benefit of being very light and easy to include in the project (just one header file!) and very easy to use.
To assert, you would simply write
REQUIRE( factorial(2) == 2 )
and it will automatically deconstruct it into two sides of the
== operator, showing expected and actual if it doesn’t match. Testing this way is much more natural than the classic
Assert.AreEqual(factorial(2), 2) or even
Assert.That(factorial(2)).Equals(2) or whatever the latest fad in fluent interfaces is.
The
BDD style for organizing tests has been the most convenient from what I’ve seen so far. You can have a hierarchy of test conditions, delimited using
SCENARIO,
GIVEN,
WHEN,
THEN and at each step of the hierarchy, you can set up some objects that will be used in levels below. The test framework will then take this tree and run
each path independently. Let me give an example with a completely imaginary API:
SCENARIO("web service test", "[web][http]") {
WebServiceFake fake;
RestClient client(fake.url());
GIVEN("authenticated client") {
client.user("pete");
client.password("abcd");
WHEN("makes request about self") {
Request r(client.new_request("/user/pete");
r.get();
THEN("gets its data") {
REQUIRE(r.json().get("salary") == 123);
}
}
}
GIVEN("guest client") {
WHEN("makes request about pete") {
Request r(client.new_request("/user/pete");
r.get();
THEN("gets nothing") {
REQUIRE(r.json().count() == 0);
}
THEN("error is reported") {
REQUIRE(r.status() == 403);
}
}
}
}
In this example, both “authenticated client” and “guest client” will be run separately, with a fresh instance of
client object each time. In all but the most basic unit tests, we have to deal with setting stuff up and this layered structure is really helpful because it helps avoid duplication while putting the code where it’s easy to see.
5 4 PageHeap
While I believe there are plans to make Address Sanitizer available for Windows, at the time of writing that port was not yet ready.
PageHeap is a debugging tool built into Windows that can be used to detect buffer overflow errors. It’s not as versatile as Asan but it also helped save my code’s neck a few times (was particularly useful to catch a bug at the boundary of C# and C++ code). It doesn’t require you to recompile the code, you just enable it for a particular program using
gflags.exe available with the Windows SDK. It works by putting each allocation at the end of a virtual memory page which allows the OS to catch any access over the page boundary.
- WinDbg is a very powerful debugger with bunch of scripts and extensions available. For source code based debugging, Visual C++ is pretty sufficient because you can see everything. But WinDbg sure comes handy when you don’t have the code and need to debug issues outside your own code or have problems calling closed source or system libraries. On a second thought, you probably don’t want to end up digging there unless you enjoy this kind of self-punishment.
- radare2 looks pretty rad for digging in assembly. Sadly I didn’t have much time to play around with it. Yes I seem to enjoy this kind of self-inflicted pain.
- rr is a project from Mozilla that lets you record a program run and then debug the bug out of it by running it over and over and over until you find it.
- F* is the absolute heavy-weight here. It lets you write code, prove that it’s absolutely correct and then transalte it to C/C++. Except for the part where you have to be a genius to prove the correctness of any larger program.
6 That’s it?
I’ve tried to compile my my approach to not shooting yourself in the foot while coding in C++. Note that while it’s not exactly short, it still doesn’t cover everything, for example how not to shoot yourself in your hand, knee or the back of your neck.
The story is not over though, perhaps you, dear readers, can reveal some tricks you have up your sleeve? Discuss!
