How can I recursively delete all files of a specific extension in the current directory?

You don't even need to use rm in this case if you are afraid. Use find:

find . -name "*.bak" -type f -delete
But use it with precaution. Run first:

find . -name "*.bak" -type f
to see exactly which files you will remove.