The risk is inherent, you are right. Exciting =/= dangerous in every situation. Riding over sand is not exciting to me.
I think riding over dirt is fascinating. From what I have seen, the 'dirt' track they used was fine for riding, even on a TT bike. The pros go over much worse dirt roads in races like the Strade Bianche.