If you look at the winning scrambler car at nationals last year, they used balsa wood. (I think it was solons, there's a pic on the image gallery). Balsa may break easily, but if you provide the proper padding, and glue the screws down, I'm pretty sure it will last. Anyway, we can say a well built balsa buggy will probably win over a well built metal/plastic/wood buggy, simply because of newton's second law.
And think, if a scrambler run goes wrong, your car is going to crash onto a wall. Buggy, not so much.