Problem 2: Stepping Through Code and Looking at Data

Now we'll look at problem2.c:

#include <stdio.h>
#include <stdlib.h>

int puzzle1 = 0x40490fdb;
int puzzle2[2] = {0x8b145769, 0x4005bf0a};
char puzzle3[] = {87, 97, 115, 32, 116, 104, 97, 116, 32, 114, 101,
    97, 108, 108, 121, 32, 115, 111, 32, 104, 97, 114, 100, 63, 0};

int hmc_pomona_fix(int x)
{
    if (x == 42)
        return 47;
    else if (x == 47)
        return 42;
    else
        return x;
}

void fix_array(int *a, int a_size)
{
    int i;

    for (i = 0;  i < a_size;  i++, a++) {
        *a = hmc_pomona_fix(*a);
    }
}

int main(int argc, char *argv[])
{
    int *array;
    int i;

    array = (int *)malloc(argc * sizeof(int));
    /* argv[0] is the program name so we skip it */
    for (i = 1;  i < argc;  i++) {
        array[i] = atoi(argv[i]);
    }
    fix_array(array, argc - 1);
    for (i = 1;  i < argc;  i++) {
        printf ("%d ", array[i]);
    }
    printf ("\n");
    return 0;
}

This file contains three static constants and three functions. Read the functions and figure out what they do. (If you're new to C, you may need to consult your C book (e.g., Kernighan & Ritchie) or some online references.)

Here are some hints:

• argv is an array containing the strings that were passed to the program on the command line (or from gdb's run command);
• argc is the number of arguments that were passed.

By convention, argv[0] is the name of the program, so argc is always at least 1. The malloc line allocates a variable-sized array that's big enough to hold argc integers (which is slightly wasteful, since we only store argc-1 integers there, but what the heck).

By now we hope you've learned that optimization is bad for debugging. So compile the program with -Og -g,

gcc -Og -g -o problem2 problem2.c

and bring up the debugger on it with

gdb problem2

Steps and Questions

1. Gdb provides you with lots of ways to look at memory. For example, type print puzzle1 (something you should already be familiar with).

   What is printed?

2. Gee, that wasn't very useful.

   Sometimes it's worth trying different ways of exploring things. How about p/x puzzle1?

   What does that print? Is it more edifying?

3. You've just looked at puzzle1 in decimal and hex. There's also a way to look at it as a string, although the notation is a bit inconvenient. The x (examine) command lets you look at arbitrary memory in a variety of formats and notations. For example, x/bx examines bytes in hexadecimal.

   Let's give that a try. Type x/4bx &puzzle1 (the & symbol means “address of”; it's necessary because the x command requires addresses rather than variable names).

   How does the output you see relate to the result of p/x puzzle1? (Incidentally, you can look at any arbitrary memory location with x, as in x/wx 0x404078.)

4. OK, that was interesting and a bit weird. But we still don't know what's in puzzle1. We need help! And, fortunately, gdb has help built in. So type help x. Then experiment on puzzle1 with various forms of the x command. For example, you might try "x/16i &puzzle1". (x/16i is one of our favorite gdb commands---but since here we suspect that puzzle1 is data, not instructions, the results might be interesting but probably not correct.) Keep experimenting until you find a sensible value for puzzle1. What is the human-friendly value of puzzle1?

   • Don't accept an answer that is partially garbage!
   • Although puzzle1 is declared as an int, it's not actually an integer. But on a 32-bit machine an int is 4 bytes, 2 halfwords, or one (in gdb terms) word.
   • There are 44 possible combinations of sizes and formats. But you know the size, right? And the c, i, and s formats don't make sense with a size, so you have a manageable number of choices. Try them all! Be systematic.

5. Having solved puzzle1, look at the value carefully.

   Is it correct? (You might want to check it online.)

   If it's wrong, why is it wrong?

6. Now we can move on to puzzle2. It pretends to be an array of ints, but you might suspect that it isn't. Using your newfound skills, figure out what it is. (Hint: Because there are two ints, the entire value occupies 8 bytes. So you'll need to use some of the size options to the x command.)

   What is the human-friendly value? (Hint: It's not “105”. Nor is there garbage in it.)

7. Are you surprised?

8. Is it correct?

9. We have one puzzle left. By this point you may have already stumbled across its value. If not, figure it out; it's often the case that in a debugger you need to make sense of apparently random data. What is stored in puzzle3?

10. So far, we've done all this exploration without actually running the program! But now it's time to execute!

    Set a breakpoint in fix_array. Run the program with the arguments 1 1 2 3 5 8 13 21 44 65. When it stops, print a_size and verify that it is 10.

    Did you really need to use a print command to find the value of a_size? (Hint: Look carefully at the output gdb produces.)

11. Existing breakpoint at fix_array; stopped at that breakpoint.

    What is the value of a?

12. Existing breakpoint at fix_array; stopped at that breakpoint.

    Type display a to tell gdb that it should display a every time you step (although gdb will only obey part of the time). Step five times.

    Which line last displayed a?

13. Existing breakpoint at fix_array; after hitting that breakpoint and then stepping five times.

    step twice more (a sixth and seventh time).

    What is the value of a now?

    What is i?

14. Existing breakpoint at fix_array; after hitting that breakpoint and then stepping seven times.

    At this point you should (again) be at the call to hmc_pomona_fix. You already know what that function does, and stepping through it is a bit of a pain.

    The authors of debuggers feel your pain, and they always provide two ways to step line-by-line through a program. The one we've been using (step) is traditionally referred to as “step into”—if you are at the point of a function call, you move stepwise into the function being called.

    The alternative is “step over”—if you are at a normal line it operates just like step, but if you are at a function call it treats the whole function as if it were a single line. Let's try that now.

    In gdb, the step-over command is next (or n).

    Type n twice (i.e., type n and Enter, and then repeat or just press Enter).

    What line do we wind up at?

    What is the value of i now? (Recall that in gdb as in most debuggers, the line shown is the next line to be executed.)

15. Existing breakpoint at fix_array; after hitting that breakpoint, stepping seven times, and typing next twice.

    It's often useful to be able to follow pointers. GDB is unusually smart in this respect: you can type complicated expressions such as p *a.b->c[i].d->e.

    Here, we have kind of lost track of a, and we just want to know what it's pointing at.

    Type p *a.

    What do you get?

16. Existing breakpoint at fix_array; after hitting that breakpoint, stepping seven times, and typing next twice.

    Often when debugging, you know that you don't care about what happens in the next three or six lines. You could type s or n that many times, but we're computer scientists, and CS types sneer at doing work that computers could do for them—especially mentally taxing tasks such as counting to twelve. So, on a guess, type next 12.

    What is the value of *a now?

17. Existing breakpoint at fix_array; after hitting that breakpoint, stepping seven times, and (in effect) typing next 14 times (whew!).

    Let's use n to verify that it works just like s when you're not at a function call. Type n until you see a line from main. Then type n one more time.

    Which two lines of main were displayed?

(When logged in, completion status appears here.)