I almost never give out money either. I have to dodge panhandlers almost daily at the local Safeway, many of them in reasonably good clothes. Sorry, I just don't believe it.
I'll give out money for a really imaginative line ("buddy, can you spare a dollar so I can get out of this hellhole town?") though the classic "want money for booze and pot" joke has gotten old enough that I don't care anymore.
A lot of the people I see asking for money don't mean it. It's just become a line to say. There's no hope there . . . but there's no despair either. It's just a line, it's just a job. Someone asking if I have a lighter has more hope than most panhandlers, because my response means something. If I said yes (if I ever had a lighter) they'd thank me and maybe talk to me. When I gave out money frequently, they wouldn't even acknowledge my presence. It was "okay, this guy's given me money, time to look for the next one."
So I stopped acknowledging them.
A few months ago, when I was in the middle of getting dumped by the most recent longtermish girlfriend, I was sitting down on Broadway despairing and slowly making my way through a Dick's Deluxe. (It's comfort food to me - it doesn't help much, but it's better than nothing.) It was approaching midnight, as I remember. For those who know what SCCC looks like, I was sitting in one of the alcoves in the front with windows in them - the ones that are fantastically sheltered from rain, but sloped so that you couldn't sleep in there without serious problems.
A few people were walking by - your basic cluster of four or five humans, one female. She turned towards me - probably seeing me as no more than just a shadow - and asked if I had any spare change. I mumbled something pretty unintelligable . . . I wasn't really in a shape to talk. I suppose in a slightly different world she probably would have assumed I was drunk, but as it was, maybe one of my tears caught the light. She changed expression, stepped closer, and asked "You okay?"
She came over and talked for a minute or two, gesturing for the rest of the cluster to move on (which they did). Nothing special, just recognizing how lonely and hurting I was. I don't remember what either of us said. When she left, I stopped her and gave her a handful of change.
When someone really needs it, you do what you can.
I haven't seen anyone really hurting like that on my own, though I've seen a few that I watched closely to see if they were crying or not. If I do . . . well, I owe the world one. Probably more - by all rights I really shouldn't be alive now, and yet I am. In the meantime, I give out change when they actually need it. Someone looking for phone change or bus fare - hey, I'm easy, they're probably lying, but maybe not.
People who seem like they mean it - like it's not just a series of words to them. I guess that's the difference.