In a well-balanced report, Tim Oren at the Due Diligence blog says that the privacy concerns about RFID shouldn't be focused on the RFID technology itself, but on databases of information about individuals. And those databases of information are already out there, and for the most part unrelated to RFID:

"You want a policy issue? Go after the ownership of that data.... And, by the way, there's already quite a lot of this going on - what do you think merchant 'loyalty cards' facilitate? You want to hyperventilate? Learn something about database and transactional systems architecture, and take on a much larger and nastier privacy and information ownership policy issue. "

He also points out that the biggest limitation on RFID being used to mine consumer data is simply the lack of profit potential.