Here's one that bugs me, although I'm sure I used to say it. I said it until I realized how terribly arrogant and insulting it is.
"I'm too busy to ____________" (fill in the blank with whatever activity your conversation partner has just mentioned that they do.)
Here's the truth. This has nothing to do with busy-ness. Everyone has the same amount of time and we all choose how we spend it. If I don't do the same activity as another person, it's because I choose to spend my time in a different way, not because I am too busy to do it. When I say I am too busy to _______(scrapbook, read, exercise, play video games, be on Facebook, etc), I am really saying that I don't find that activity worthy of my time and it passes a bit of judgement on the person who does spend their time that way.
Instead try: "I'm glad you enjoy _______ing. I don't have a pressing desire to do that right now."