C++ Exception Handling and It’s Effect on Real World Performance, Part III

For these tests, I've changed the source code that we've been using, somewhat. There are now three very basic classes: BaseClass, which provides both virtual and non-virtual functions, one each of which throws an exception or returns an integer result; DerivedClass, which provides overriding implementations of the virtual functions in BaseClass; and DummyClass, which is just a copy of DerivedClass, to ensure that the compiler will not be able to assume that the pointer is actually to an instance of DerivedClass.

The member functions of BaseClass, DerivedClass, and DummyClass essentially do the same thing as the SimpleFunc() functions in the previous posts.

For these tests, since there are multiple compilation units (.cpp files), we're going to use the -flto optimization argument of g++. That will help the compiler to perform better optimizations across the compilation units (such as inlining). This wasn't necessary in the previous posts because each test case consisted of a single compilation unit.

Here is the implementation of main() (click the source code box to expand it). The interesting lines are highlighted. The entire source file, containing the main() function can be downloaded here (except-06.cpp).

#include <iostream>
#include <stdlib.h>
#include <time.h>
#include "except-06a.h"

using namespace std;

int main ( int argc, char** argv )
{
	//	The iterations at which to return a failure from SimpleFunc(), the lower this is, the more
	//		frequently an error is reported.
	unsigned long nFailIter = 1;
	if ( argc >= 2 )
		nFailIter = strtoul ( argv[1], NULL, 10 );

	//	The total number of iterations to perform.
	unsigned long nIters = 10000000000;
	if ( argc >= 3 )
		nIters = strtoul ( argv[2], NULL, 10 );

	//	Generate psuedo random values to use as input to the function, so the compiler
	//		doesn't optimize them away.
	int nArg1 = rand ( ) % 5;
	int nArg2 = rand ( ) % 7;
	long nArg3 = rand ( ) % 13;

	//	Prevent the compiler from being able to determine the actual type of pTestObject at compile
	//		time.
	int nArg4 = 0;
	if ( argc >= 4 )
		nArg4 = strtol ( argv[3], NULL, 10 );

	//	Create an instance of the derived class.
	BaseClass* pTestObject = DerivedClass::CreateTestObject ( nArg4 );

	//	Invoke the non-virtual function that doesn't throw an exception.  This will call the 
	//		implementation in BaseClass.
	{
		unsigned long nSum = 0;
		unsigned long nNumFailures = 0;
		time_t tmStart = time ( NULL );
		clock_t nStart = clock ( );
		for ( unsigned long i = 0; i < nIters; ++i ) {
			unsigned long nFuncResult = pTestObject->NonVirtFuncNoExcept ( nArg1, nArg2, nArg3, nFailIter );
			if ( nFuncResult == 0 ) {
				++nNumFailures;
				continue;
			}
			nSum += nFuncResult;
		}
		clock_t nEnd = clock ( );
		time_t tmEnd = time ( NULL );

		wcout << L"NonVirtFuncNoExcept() **************************************************" << endl <<
				 L"    Total failures:         " << nNumFailures << endl <<
				 L"    Total processor time:   " << nEnd - nStart << L" ticks." << endl <<
				 L"    Total wall clock time:  " << tmEnd - tmStart << L" seconds." << endl;
	}


	//	Invoke the virtual function that doesn't throw an exception.  This will call the 
	//		implementation in DerivedClass.
	{
		unsigned long nSum = 0;
		unsigned long nNumFailures = 0;
		time_t tmStart = time ( NULL );
		clock_t nStart = clock ( );
		for ( unsigned long i = 0; i < nIters; ++i ) {
			unsigned long nFuncResult = pTestObject->VirtFuncNoExcept ( nArg1, nArg2, nArg3, nFailIter );
			if ( nFuncResult == 0 ) {
				++nNumFailures;
				continue;
			}
			nSum += nFuncResult;
		}
		clock_t nEnd = clock ( );
		time_t tmEnd = time ( NULL );

		wcout << L"VirtFuncNoExcept() *****************************************************" << endl <<
				 L"    Total failures:         " << nNumFailures << endl <<
				 L"    Total processor time:   " << nEnd - nStart << L" ticks." << endl <<
				 L"    Total wall clock time:  " << tmEnd - tmStart << L" seconds." << endl;
	}

	//	Invoke the non-virtual function that throws an exception.  This will call the 
	//		implementation in BaseClass.
	{
		unsigned long nSum = 0;
		unsigned long nNumFailures = 0;
		time_t tmStart = time ( NULL );
		clock_t nStart = clock ( );
		for ( unsigned long i = 0; i < nIters; ++i ) {
			unsigned long nFuncResult;
			try {
				nFuncResult = pTestObject->NonVirtFuncExcept ( nArg1, nArg2, nArg3, nFailIter );
			} catch ( unsigned long& exc ) {
				++nNumFailures;
				continue;
			}
			nSum += nFuncResult;
		}
		clock_t nEnd = clock ( );
		time_t tmEnd = time ( NULL );

		wcout << L"NonVirtFuncExcept() ****************************************************" << endl <<
				 L"    Total failures:         " << nNumFailures << endl <<
				 L"    Total processor time:   " << nEnd - nStart << L" ticks." << endl <<
				 L"    Total wall clock time:  " << tmEnd - tmStart << L" seconds." << endl;
	}


	//	Invoke the virtual function that throws an exception.  This will call the implementation 
	//		in BaseClass.
	{
		unsigned long nSum = 0;
		unsigned long nNumFailures = 0;
		time_t tmStart = time ( NULL );
		clock_t nStart = clock ( );
		for ( unsigned long i = 0; i < nIters; ++i ) {
			unsigned long nFuncResult;
			try {
				nFuncResult = pTestObject->VirtFuncExcept ( nArg1, nArg2, nArg3, nFailIter );
			} catch ( unsigned long& exc ) {
				++nNumFailures;
				continue;
			}
			nSum += nFuncResult;
		}
		clock_t nEnd = clock ( );
		time_t tmEnd = time ( NULL );

		wcout << L"VirtFuncExcept() *******************************************************" << endl <<
				 L"    Total failures:         " << nNumFailures << endl <<
				 L"    Total processor time:   " << nEnd - nStart << L" ticks." << endl <<
				 L"    Total wall clock time:  " << tmEnd - tmStart << L" seconds." << endl;
	}

	return 0;
}

In the above code, on line 34, we create an instance of a BaseClass derived object. The reason we're using a factory function which takes an argument based on a command line parameter is to ensure the compiler cannot determine, at compile time, the actual type of the object being pointed to. If the compiler were able to determine the type then it may perform optimizations which could affect the test results. Since the goal of these tests is to determine the performance cost of throwing exceptions from virtual functions then we want to ensure the vtable is actually used to invoke the appropriate implementation at runtime since, in most real world scenarios, that will be the case.

For the tests results presented here, we used an nIters of 10 billion.

The definitions of BaseClass and DerivedClass are as follows. The entire source file, containing the definitions of all three classes, can be downloaded from here (except-06a.h).

class BaseClass
{
public:
	unsigned long NonVirtFuncNoExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter );
	virtual unsigned long VirtFuncNoExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter );

	unsigned long NonVirtFuncExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter );
	virtual unsigned long VirtFuncExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter );
};

class DerivedClass : public BaseClass
{
public:
	virtual unsigned long VirtFuncNoExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter );
	virtual unsigned long VirtFuncExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter );

	static BaseClass* CreateTestObject ( int nArg );
};

Pretty straightforward. Not much to explain there. We won't bother with the definition and implementation of DummyClass because it's essentially the same as DerivedClass, and we're not really using it anyway - it's just to ensure the compiler cannot optimize away the virtual function tables. If you're interested in reviewing it anyway, it is contained in the source files.

The implementations of the member functions are as follows. There are really only two versions of the function bodies: the ones that don't throw an exception; and the ones that do throw an exception. For the sake of brevity, I'm just posting one implementation of each below. The entire implementation is contained in the source file which you can grab from here (except-06a.cpp).

unsigned long BaseClass::NonVirtFuncNoExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter )
{
	static unsigned int nCurIter = 1;

	int nReturn = 0;
	//	If an interval of 0 was specified then never fail.
	if ( nFailIter ) {
		//	Otherwise, if this is an iteration we should fail on...
		if ( nCurIter == nFailIter ) {
			nCurIter = 1;
			return nReturn;
		}
		else {
			++nCurIter;
		}
	}

	nReturn += nArg1 + nArg2 + nArg3;

	return nReturn;
}

unsigned long BaseClass::NonVirtFuncExcept ( int nArg1, int nArg2, long nArg3, unsigned long nFailIter )
{
	static unsigned int nCurIter = 1;

	int nReturn = 0;
	//	If an interval of 0 was specified then never fail.
	if ( nFailIter ) {
		//	Otherwise, if this is an iteration we should fail on...
		if ( nCurIter == nFailIter ) {
			nCurIter = 1;
			throw (unsigned long) 0;
		}
		else {
			++nCurIter;
		}
	}

	nReturn += nArg1 + nArg2 + nArg3;

	return nReturn;
}

So, essentially, what we have is a base class which contains:

• A non-virtual function which returns an integer status and doesn't throw an exception;
• A virtual function which returns an integer status and doesn't throw an exception;
• A non-virtual function which throws an exception; and
• A virtual function which throws an exception.

And, we have two derived classes which provide overriding implementations of the two virtual functions from the base class.

First, we're going to consider the runtime differences without the use of any compiler optimizations. This will give us an idea of the overhead differences between calling a non-virtual function as opposed to a virtual function. We should expect that a virtual function should have slightly more overhead simply because the layer of indirection caused by having to use the virtual function table. And based on the results we've seen from the previous posts/tests, we should expect that the differences between returning and integer and throwing an exception should be negligible until the number of exceptions thrown gets above 10 million.

As expected, what we see from the above table, is that the virtual functions consistently perform below the non-virtual implementations. Specifically, 9.0% slower, in the no exception case; and, 7.2% slower in the exception throwing case. But, what is interesting is that the exception throwing functions performed better than the non-exception functions until the number of exceptions actually thrown is over 1 million.

Now, let's consider what happens when we compile with level 2 optimizations.

Based on what we've seen from previous tests with level 2 optimizations, we should expect that a function which returns an integer to indicate result status should execute in constant time, regardless of the number of failures that occur. And, moreover, functions which throw an exception should, typically, take about 2.65 times longer until the number of exceptions thrown exceeds 1 million.

Now we see that, in the case of virtual functions which don't throw exceptions, the runtime also remains constant regardless of the number of failures. That's reasonable. However, the virtual function which doesn't use exceptions consistently takes about 2.24 times longer than the non-virtual function. We also find that a virtual function without exceptions performs slower than a non-virtual function with exceptions - as long as the number of exceptions actually thrown is less than 10 million.

In the case of the functions which throw exceptions, we see that the virtual implementation generally takes about 10% longer than the non-virtual implementation.

Now, for the really interesting comparison: virtual functions, with and without exceptions. The virtual function that throws an exception performs better than the virtual function that doesn't throw an exception, as long as the number of exceptions thrown is below 1 million!

Before we draw our conclusions and make any life-altering decisions about exception handling, let's consider the results when using level 3 optimizations.

As with our previous level 3 optimization tests, we see that the non-virtual, no exception handling implementation performs consistently until the number of failures reaches 100 million. At that point, we see a slight drop in performance. Though, nothing significant enough to worry about.

The virtual implementation that doesn't use exceptions remains constant, though significantly slower than the non-virtual, non-exception implementation - typically around 4 times as long as the non-virtual implementation. Rather disconcerting results, if I do say so.

There aren't any surprises amongst the exception throwing implementations. The virtual implementation that throws an exception performed, on average, about 20% worse than the non-virtual implementation.

As with the level 2 optimizations, we find that the non-virtual implementation with exceptions, actually performed better than the virtual implementation without exceptions! This suggests that the use of virtual functions, in typical scenarios, actually harms performance more than the use of exception handling.

As with the test results from the previous post on this topic, we really need to consider the applicability of these test scenarios to our real world development. For example, these tests, as with the ones in the previous post, do nothing other than either return an integer result value or throw an exception. There is no collateral code to water down the effects of throwing the exception.

So, can we really say that these test results will have any bearing on code we'd implement on a real project? The short answer: No, absolutely not! No serious, professional software developer with any self-respect, and who has any interest in being good at what he does, should make design and/or implementation decisions based on tests like these. They are fundamentally flawed in that they focus only on one very specific, isolated point, and ignore all of the complexities and realities of real world development. As I'd pointed out in the previous post: unless the code you're writing is a very tight loop which executes a huge number of iterations then these test results will not be representative of the results you would see in real world scenarios. You must consider them academic, at best.

The use of virtual functions comes with runtime overhead. It is an inescapable reality of object oriented programming.

Since we're (and by "we're", I mean pretty much all of the civilized world) using object oriented design and programming, we can assume that much of our code is going to be implemented as virtual functions. Therefore, when considering the performance implications of using exception handling it is unrealistic and naive to only compare the performance when using global, non-virtual functions, as we've done in the previous posts. Most of the comparisons you will find on the Internet, in books, and in school rely on such naive comparisons.

But, as we see from the above test results, throwing exceptions from virtual functions does not actually adversely affect the runtime performance unless the number of exceptions actually being thrown is ridiculously high. Moreover, in the case of using level 2 compiler optimizations (which is the typical scenario), using exception handling may even perform better than using a return value to report errors.

Based on everything we've seen so far, there is no question that the presence of throw expressions within non-virtual functions interferes with the compiler's ability to perform certain optimizations and may result in degraded runtime performance of such functions. And, the throwing of exceptions from such, non-virtual functions will adversely affect performance when exceptions are actually thrown - but as the term "exception" implies: that should be the exceptional case, not the norm. In the case of virtual functions, however, there is no evidence to support the belief that exception handling will have any adverse effects on runtime performance in real world scenarios.