Paul Ince thought the sending off of Martin Olsson cost his Blackburn side on Sunday.
The left-back was run ragged by Aaron Lennon in the first half - the winger got the better of him to create what turned out to be the winning goal for Roman Pavlyuchenko in the ninth minute and was sent off after being shown a second yellow card - both for fouls on Lennon - with 39 minutes on the clock.
The scoreline stayed at 1-0 with the 10 men defending stoutly.
"I thought we were unlucky," said Rovers boss Ince. "I fancied us, but the chance was taken away from us with the sending off.
"I didn't think the first yellow was a booking and the second one was perhaps naivity. The lad was devastated.
"It was a good performance against a good Tottenham side and apart from a few skirmishes late on, we weren't really troubled in the second half.
"We tried to stay compact and nick something and we nearly did in the last few minutes."
Meanwhile, Ince had praise for Lennon. "You'll struggle against anyone who has that pace and he's a fantastic player who will cause most teams problems, although I felt Simmo (Danny Simpson) did well in the second half."