Ever been conned by a beggar?

Yesterday I bought some pizza slices on my way home from work. I had 3, for $5. Just enough to fill the hole before I get home. As I was just about to attack slice 3, a beggar came up to me in the street, and asked for money. I awkwardly hid my body language of a person about to take a bite of dinner, and smoothly transitioned it into an offer, of my last piece of pizza. Immediately the small silver stubbled man spoke from under his cap that he needed money for a place to stay tonight. Not pizza. Now I didn't want to give money today. My daily account was a bit short, with only $20 left. Payday was 2 days away. I said no. He replied in disgust "I don't know how you people can sleep at night... you can give money. you know you can. But you won't. You just don't give a shit". I held back the temptation to inform the man i worked for an org that supported homeless people and instead awkwardly removed my self from the conversation, and continued home munching my pizza. it had gotten cold.

Today, after getting off the tram on my way to work, 15 minutes early, I stopped by the ATM to pull out my $20. Payday was tomorrow, and I was looking forward to my first coffee of the day. On the walk to the cafe, a new beggar, exclaimed a huge hello from half way down the street. He came happily over to me, and in an extremely needy tone, asked for some loose change. I replied that I could give him the coins I would get once I broke my note when I bought my coffee. He said great! and we proceeded to the cafe. A moment later he asked me, could he have $5 seeing as I was breaking a $20? I was caught off guard, and ummed and arred, he pounced on my hesitancy and the next thing I knew I had agreed.
I bought my coffee, recieved $16.50 back, and handed him the $5. He whipped out another $5, all crumpled and dirty. He said, can i give you back these two $5's and get your $10? This $5 looks so dirty and It makes me look bad.
confused by the pace of his comments, and seeing the way he might be looked down upon with that $5, I passed him the $10 as he reached out to give me his two $5's. But my hand returned with only one $5 in it... the dirty one. I looked up and he was out the door, with a cheeky grin of success on his face. I exclaimed after him, but there was no catching him... my last words were... "You're good mate..." he actually thanked me, aware, that he was in fact, quite good.
And I was left standing there. In front of my regular barista, not just $10 poorer, but poorer in my sense of control. He had outwitted me. My pride a bit beat, I proceeded on my day with my last $6.50, off to my free lunch at Credo, most likely followed by another coffee or two, and a pay check at midnight. The thing is I can afford to get ripped off.
Why are we so afraid to be ripped off?
Is it really just our pride?