Stoke boss Tony Pulis conceded his side lost Tuesday night's game in the first half-hour at the Lane.
The Potters were 3-0 down in that time thanks to goals from Aaron Lennon, Jermain Defoe and Michael Dawson.
They rallied after the break and pulled a goal back through James Beattie but didn't really threaten again until late on when Richard Cresswell headed wide.
"We had a very poor first half-hour, we didn't get close to Tottenham and allowed Modric to run the game," said Pulis.
"There haven't been many games where our players haven't had a go but we lost the game in the first half-hour.
"That was as poor as we've performed and everything after that was a bonus.
"Richard Cresswell had a chance with eight minutes left which would have made Harry twitch a bit but Tottenham deserved to win the game, they were better than us."