Referee Mark Clattenburgh waved away appeals after Mark Delaney tumbled under Erik Edman's challenge in the fourth minute.
A minute later and we were 1-0 up though Fredi Kanoute and soon 3-0 up as Kanoute and Ledley King put us in charge.
However, Villa were awarded a more debatable penalty for handball just before half-time that Gareth Barry drilled home for 3-1 and were the better side at the start of the second half until Andy Reid's crashed home a game-clinching fourth after 68 minutes. Stephen Kelly completed the scoring in the last minute.
O'Leary said: "It was a definite penalty and a double kick in the teeth for us.
"Whether we would have scored the penalty is a different matter but I can tell you that if we had scored, Spurs would not have gone straight down the other end and got their goal.
"But for me, the fourth goal knocked the stuffing out of us - it was the killer.
"The best team won the game. We let ourselves down today but I have got a small group of players and for the second year running they have done me proud.
"They've given me everything but let themselves down today because we looked very vulnerable at the back.
"We defended poorly from set-plays. But I am not ruling ourselves out of a European place with six points left to play for."
Villa boss David O'Leary felt his side should have been awarded a penalty before we opened the scoring but had few complaints about our 5-1 win on Sunday.