========================================================================== ECE 2400 Debug Process ========================================================================== Step 1: Use 'make check' to run all of the tests to get a high-level view of what test cases are passing and what test cases are failing. Step 2: Pick one failing test program to focus on. Pick the most basic test program that is failing. Run just that test program in isolation, maybe with the --verbose flag to get a list of the test cases. Step 3: Pick one failing test case to focus on, and run just that test case in isolation using --filter. You can use --list to get a list of test cases. Pick the most basic test case that is failing. Step 4: Look at the error message. Determine what is the observable error. The error might be a failed Criterion assertion or a crash. Step 5: Look at the actual test case source code. Make absolute sure you know what the test case is testing and that the test case is valid. You have no hope of debugging your code if you do not understand what correct execution you expect to happen! Step 6: Your goal in this step is to narrow the focus of your bug hunt. You want to look at each C/C++ statement in the test case. You will be checking for one of three things: (1) are the statement's inputs incorrect and the outputs incorrect? if so, then the bug is earlier in the code and you need to continue working backwards to an earlier statement; (2) are the statement's inputs correct and the outputs incorrect? if so then you have narrowed the bug to be at this statement; if the statement is a function call then the bug is in the called function; or (3) are the statement's inputs correct and the outputs correct for this statement? if so, then the bug is later in the code and you need to continue working forwards to a later statement. For conditional statements, the "inputs" are the variables used in the condtional statement and the "output" is which direction the "execution arrow" goes (i.e., did the execution arrow go to the then or else block?). For iteration statements, the "inputs" are the variables used in the loop and the "output" is whether or not the loop body is executed or the loop exits. You can use printf debugging, gdb single-stepping, or gdb back traces to help examine the inputs and outputs of the statements in your test case. Step 6a: For printf debugging, insert printfs to determine the path through the control flow and the values of certain variables. Start with a printf right at the beginning of the test case. This should display correctly. You can gradually continue adding printf statements working _forwards_ until you find a statement which has correct inputs and incorrect outputs. At the same time you can also add printf statements starting from the observable error working _backwards_ until you find a statement which has correct inputs and incorrect outputs. So you might add a printf right before the Criterion assertion which is failing to display the values of the variables being used in the assertion and then work backwards from there. Step 6b: For gdb debugging, use cr-gdb to drop into the test case you are focusing on. Single step forward through the code. Observe the control flow and use 'print' commands to display the values of variables. You are moving _forwards_ until you find a statement with correct inputs and incorrect outputs. Step 6c: For crashes, you can also use either Step 6a or Step 6b to determine where the crash is happening. An alternative is to use gdb to give you a stack trace (i.e., the sequence of nested function calls) that led to the crash. this is also called a back trace. Use cr-gdb and then use the 'continue' command to run your test until it crashes. Then use the 'backtrace' command to get the sequence of function calls including line numbers that lead to the crash. Then use either Step 6a or 6b to narrow the focus of your bug hunt. Step 7: Once you find a statement which has correct inputs but incorrect outputs, make a hypothesis about what should happen if you fix the bug. Your hypothesis should not just be "fixing the bug will make the test pass." It should instead be something like "fixing this bug should make this specific variable be 1 instead of 0" or "fixing this bug should make this specific if statement execute the else block". Fix the bug and see what happens by looking at the output of your printf statements or by using gdb. Do not just see if it passes the test -- literally check the specific statement you identified in Step 6 using printf or gdb. One of four things will happen: (1) the test will pass and the printf/gdb behavior will match your hypothesis -- bug fixed! (2) the test will fail and the printf/gdb behavior will not match your hypothesis -- you need to keep working -- your bug fix did not do what it was supposed to, and it did not fix the error -- undo the bug fix and go back to step 6. (3) the test will fail but the printf/gdb behavior _will_ match your hypothesis -- this means your bug fix did what you expected but there might be another bug still causing trouble -- you need to keep working -- go back to step 6. (4) the test will pass and the printf/gdb behavior _will not_ match your hypothesis -- you need to keep working -- your bug fix did not do what you thought it would even though it cause the test to pass -- there might be something subtle going on -- go back to step 6 to figure out why the bug fix did not do what you thought it would. If you are passing all of your tests but failing the evaluation you need to craft a minimum test case which causes the same bug. Then you can start this seven step process. If at all possible you want to avoid debugging your code using the evaluation. Worst case, if you must debug your code using the evaluation you really must add a new test case once you figure out the bug. Note a couple things about this systematic seven step process. First, it is a systematic process - it does not involve randomly trying things. It does not involve randomly commenting out code. Second, the process uses all tools at your disposable: output from Criterion, printfs, gdb single-stepping, and gdb back traces. You really need to use all of these tools. If you use printf debugging but never use GDB or you use GDB and never use printf debugging then you are putting yourself at a disadvantage. Third, the process requires you to think critically and make a hypothesis about what should change -- do not just change something, pass the test, and move on -- change something and see if the printf/gdb behavior change in the way you expect. Otherwise you can actually introduce more bugs even though you think are fixing things.