Other Articles in the Series

Welcome back to the fifth and last installment of our series of xdebug articles on Zend DevZone. xdebug is the swiss army knife extension for PHP developers. In addition to some nice features like beautifying var_dump() output, adding stack traces to error messages, xdebug supports tracing, profiling, and debugging, which were covered in the previous articles.

Today, we will have a look at another great feature of xdebug - creating code coverage statistics. Code coverage statistics show how many times each line of the code has been executed. Conversely, they also show which lines of code have not been executed, which is in fact much more interesting.

As we know, each program has different execution paths (unless a program has no branches at all). A program‘s execution path depends on the input data, which includes the parameters passed to the program, but any data in the database or from other data sources.

As an example imagine a website that allows you to create a user account. This account must be verified by visiting a link that is emailed to you. If you log in to a website, you might see a different response depending on wether you have already confirmed your account or not, though the GET or POST data passed to the script is the same. In this case, the program‘s execution path depends on the confirmed flag of the user record in the database.

Code coverage statistics of one single program run are not very useful. But when we collect and aggregate code coverage statistics of many program runs (with different execution paths), the coverage statistics show us which program parts have not been executed. These parts are either obsolete or should be tested.

The preferred way of creating code coverage statistics is to run unit tests while collecting coverage statistics of the tested code. Note that coverage statistics are not created for the unit tests themselves, but of the code that we are testing. A unit test is a test of a class, module, or any other part of an application. Unit tests make it much easier to feed different input data to the code to test than if we would be testing an application in whole.

Technically, creating code coverage statistics is a rather complex issue that involves various tools. Coverage statistics in xdebug are basically an array for each file where the line number is the array key and the array value is the count how often this line has been executed. We‘ll need some other tools to visualize these coverage statistics.

To run automated tests, and collect the coverage statistics that xdebug provides, we will use PHPUnit. To make testing with PHPUnit easier, we will use the build automation tool phing, which allows us to define a file set of all tests, run PHPUnit tests on each of these files, and gather and aggregate the collected coverage statistics. phing also converts the coverage statistics to a nicely formatted HTML report using XSL.

Assuming that you (still) have xdebug installed on your system, we‘ll need to install the other tools before we can start.

Installing phing and PHPUnit

phing is based on the build tool Apache Ant, which is written in Java. phing was created so that no Java Runtime Environment would be required to automate build processes on PHP development systems. The main advantage of phing over Ant is that you can extend phing by writing your own PHP classes. To extend Ant, you need to write Java code.

To install phing, you need PHP 5 with XML and XSLT support. On Unix, you must compile PHP with the configuration switch --with-libxml. On Windows, XML support should be enabled by default. To make sure you have XML support enabled, check the output of phpinfo() for a libxml section.

To activate XSL support in PHP on Windows, you must load an extension by adding

extension=php_xsl.dll

to php.ini. Remember to restart your web server to make the change effective. On Unix, the XSL extension is named php_xsl.so and will only be present, if you have configured PHP using the switch --with-xsl. When the XSL extension has been loaded in PHP, a section XSL will appear in the phpinfo() output.

phing is deployed as as PEAR package. To install phing, you must have a PEAR environment on your system. Run pear version at the command line if you are not sure wether PEAR is installed. As I write this article, the current PEAR version is 1.6.2. If you have an older version, you should upgrade PEAR first:

pear upgrade-all PEAR
pear channel-update PEAR

Normally, when we install phing with the switch --alldeps, all dependencies like PHPUnit should be installed automatically. PHPUnit, however, has moved from PEAR to its own channel server, and the classes were renamed from PHPUnit2 to PHPUnit (instead of PHPUnit3, as PEAR would have dictated). Therefore, PEAR currently fails to install PHPUnit 3.x from the channel pear.phpunit.de. The best idea is to install the latest version of PHPUnit3 manually before installing phing:

pear channel-discover pear.phpunit.de
pear install pear.phpunit.de/phpunit

Now we can install phing. As I write this article, the current version of phing is 2.3beta1. The PEAR installer will automatically install the latest version of phing, but since this version is in beta state, you should add the command line swich --force, otherwise an older version of phing might be installed instead.

pear channel-discover pear.phing.info
pear install --force --alldeps pear.phing.info/phing

phing is based on XML files that define the various targets ("jobs") of a build process, like build, test, or - for web sites - deploy which might copy the web site right to the live server. The XML format makes phing builds portable across different platforms, unless, of course, you rely on

Creating unit tests with PHPUnit

Now that we have installed all necessary tools, we must create some unit tests. I would recommend keeping the tests in the same directory tree where your production code is kept. When deploying your application, you can always leave out the test files, unless you want to create version that allows your end users to run the unit tests as well.

<?php
require_once ‘PHPUnit/Framework.php‘;
require_once ‘a(chǎn)utoload.php‘;
class FooTest extends PHPUnit_Framework_TestCase
{
public function testFoo()
{
$foo = new Foo;
$this->assertEquals(0, $foo->getNumberOfBars(), ‘Foo is not empty‘);
$bar = new Bar;
$foo->addBar($bar);
$this->assertEquals(1, $foo->getNumberOfBars(), ‘Foo has no bar‘);
}
}
?>

This example shows a rather simple unit test that makes not much sense by itself in the real world, but serves to illustrate the purpose. A real-world test case would probably contain more test methods and does not even use a test fixture. We assume that autoloading is used, so that the Foo and Bar classes are loaded when we are trying to instantiate the objects.

Creating a phing build file

For the purpose of this article, our build file will have only one task that creates code coverage statistics. By default, phing build files should be named build.xml. The following listing shows the phing build file that creates code coverage statistics:

<?xml version="1.0"?>
<project name="my_project" default="code_coverage" basedir=".">
<fileset dir="src" id="php">
<include name="*.php"/>
<include name="**/*.php/"/>
<exclude name="*Test.php"/>
<exclude name="**/*Test.php"/>
</fileset>
<fileset dir="src" id="tests">
<include name="*Test.php"/>
<include name="**/*Test.php"/>
</fileset>
<target name="code_coverage">
<mkdir dir="coverage_db"/>
<mkdir dir="coverage_result"/>
<coverage-setup database="coverage_db/coverage.db">
<fileset refid="php"/>
</coverage-setup>
<phpunit2 codecoverage="true">
<batchtest>
<fileset refid="tests"/>
</batchtest>
</phpunit2>
<coverage-report outfile="coverage_db/coverage.xml">
<report todir="coverage_result"/>
</coverage-report>
</target>
</project>

Like in PHP itself, always use forward slashes as directory separator, even on Windows.

We are using two filesets, php and tests. The first fileset contains all PHP files of our application. These are the files we want to create the code coverage statistics for. The second fileset, tests, contains all unit tests of our application. Please note that all unit test files are explicitly excluded from the first fileset.

Each fileset has a base directory, src that is specified in an attribute of the fileset tag. You can include or exclude files using wildcards, with * matching files in the given directory, and ** matching the subdirectories.

In the code_coverage build target, the coverage-setup task sets up a database with code coverage data provided by xdebug via PHPUnit. This database is in fact just a flat file containing the serialized array of coverage data that xdebug provides. The php fileset given inside the coverage-setup specifies the PHP files to create collect coverage information for.

The phpunit2 (the 2 being a reminisence of PHPUnit2, but it also works for PHPUnit3) tag with the codecoverage="true" runs a batch of unit tests that are given by the fileset tests. For each file in this fileset, all tests will be run (and coverage statistics be collected, of course).

Now the coverage-report converts the coverage database that was previously set up to the given outfile to XML format, in our example overage_db/coverage.xml. This XML file is transformed to a HTML report, triggered by the report tag.

In many cases, you want phing to stop the build when unit tests fail. You‘ll probably not want to create a release when you know that the code did not pass all tests. To stop a build when a test fails, you can use the haltonfailure attribute of the phpunit2 task. To stop a build when an error occurs, use the haltonerror attribute.

It is important to distinguish between an error and a failure. A failure is a failing assertion in a test. For example, if you expect a function to return true and the return value is false, NULL, or something else, your test fails. An error would be a PHP error that occured because there is a syntax error in one of the tested files.

When creating code coverage statistics, however, we want to execute all unit tests, regardless of how many of them fail or have errors. It would not make much sense to stop calculating the coverage statistics when the first test fails. When you create a beta release of your application, you might accept failing unit tests, but probably no errors, as you wouldn‘t want your application to crash due to a PHP error at runtime.

Gathering coverage statistics with xdebug

There are no php.ini settings that control the creation of code coverage statistics. Gathering of the statistics is turned on and off by PHP function at runtime, allowing for a fine-grained level of control over which pieces of code to create coverage statistics for.

To start gathering code coverage statistics, use xdebug_start_code_coverage(). To stop gathering statistics, use xdebug_stop_code_coverage(). xdebug keeps the gathered statistics in memory while the script is executed, so you can turn on and off code coverage within a script as often as you like. To retrieve the collected statistics as an array, use xdebug_get_code_coverage().

When using PHPUnit together with phing to run the unit tests and gather code coverage statistics, you will never have to worry about using these functions. PHPUnit will use them for you to collect coverage information about the tested code.

Putting it all together

When working with PHPUnit, you need to define a test suite as a collection of tests to run multiple tests with one command. Using phing frees you from having to deal with test suites, since all test cases in the given file set are executed. You just create as many individual test case classes as you like, define a fileset that includes all test cases, and have phing run all tests. It couldn‘t be easier.

Like PEAR and PHPUnit, phing has a command line interface. To execute a build target, run phing with the target to run as parameter. To create the code coverage statistics, run

phing code-coverage

This makes phing execute the target, which includes running all unit tests, collecting the coverage statistics, and then creating the HTML report of the code coverage statistics, even if one or more unit tests fail.

The following screen shot shows the HTML report generated by phing. This report is located in the coverage_result subdirectory of your build directory, with your build directory being the directory where build.xml is located.

Code coverage statistics can help you find parts of an application that are never executed, respectively tested. Do not over-estimate the value of code coverage statistics. Even 100% code coverage does not mean that your code is thoroughly tested, as the following example shows:

<?php
...
if ($a)
{
print ‘1‘;
} else {
print ‘2‘;
}
if ($b)
{
print ‘1‘;
} else {
print ‘2‘;
}
...
?>

This small code snippet (which, admittedly, does not make much sense in the real world) outputs numbers based on the values of $a and $b. The following table shows the possible combinations of input values and the resulting output:

You could achieve 100% code coverage just by testing the combinations (true, true) and (false, false). In real code, there are often just too many execution paths to test all combinations. What is more, defensive code often has handlers for error conditions "that should never occur". It does not make much sense to write complex test code just to test such a rare condition.

Code coverage statistics are a nice way of getting a feel for the test coverage of an application. But please do not overestimate the code coverage. Even 100% code coverage does not mean that your code is bug-free or even thouroughly tested. Still, a high code coverage certainly helps building trust in the code quality of your application.

More important than achieving a high percentage of code coverage is the fact that the statistics show you potentially critical, and untested parts of the code. These are the parts you should focus on, either by creating tests for them, or by removing them because they are obsolete.

Conclusion

These five articles should have given you some good reasons to start using xdebug, which really is a very good tool. I could not do without xdebug on my development system, even if it was only for the var_dump() output and the stack traces in error messages alone.

This article ends our series of xdebug articles, so I would like to thank you for joining in. I hope you have enjoyed reading the articles as much as I enjoyed writing them. Very special thanks to Derick Rethans for creating and maintaining xdebug. And to you: Happy coding - and since you have become an xdebug user now, please remember to send a postcard to Derick.