Frank Lampard's spot kick divided the teams for most of the match before the midfielder pounced again to seal it in the final minutes.
Graham Poll had little hesitation in awarding Chelsea what turned out to be a match-defining penalty decision after 39 minutes.
Alexei Smertin certainly seemed to make a meal of it as Ledley King tried to poke the ball away from him in the box, replays proved inconclusive but Lampard cared little and buried from 12 yards.
Lampard's late second goal was a bit harsh after all our hard work but, like most teams, we struggled to break down Chelsea's defence.
Petr Cech was forced into serious action once to palm away Jermain Defoe's deflected rasper in the first half but when he was helpless, luck wasn't on our side as Defoe's corner was sliced agonisingly inches wide by Smertin.
The first half started at a relentless pace with Erik Edman, Robbie Keane and Duff all firing in efforts inside the first six minutes, Edman's rising drive the closest.
Drogba went close after 17 minutes and then a fine save at either end - Cech to parry out Defoe's deflected shot and Robinson at full length to keep out Duff.
Chelsea had appeals for a penalty waved away just after the half-hour as Robben went over after tussle with Pamarot - in all honest, that looked a better case than the one awarded.
Both sides continued to zip the ball around with Mendes lobbing wide after King laid back Carrick's pinpoint crossfield pass.
Then the big penalty call. Duff did superbly to keep the ball in on the right touchline and rolled into the supporting Smertin, King met him and tried to prod the ball away - Smertin went flying, it looked 50-50 on replays but Poll had no hesitation and Lampard stuck the spot kick away.
The second half followed a similar pattern - both sides pushing the ball around with little success - until Martin Jol sacrificed Reto Ziegler to throw Ledley King up front with 25 minutes left.
The skipper certainly made his presence felt but, again, chances were few and far between - Mendes lashed a shot wide, Keane blasted over, Defoe dragged wide and Cech comfortably picked Carrick's free-kick out.
Chelsea has rarely threatened on the counter attack but managed to bag a second in injury time as Gudjohnsen raced into the area and cut the ball back from the byline for Lampard to end the contest from 12 yards.
GUIDE TO THE ACTION
Press refresh for updates...
2mins: Decent start, good possession, Keane played ball wide to supporting Edman, cut inside Johnson, rising right-foot shot just wide.
4mins: Keane headed Pamarot's cross just wide but penalised for push.
6mins: Clever pass from Makelele to Johnson, we claimed offside, Johnson cut ball back for Duff, King there to block.
17mins: CLOSE. Drogba gave chase to Ferreira's long pass, beat Robinson to it on edge of box but poked a foot wide.
19mins: SAVE. Great move, Carrick superb pass to Defoe on left, cut inside three players and let fly, appeared deflected, Cech made decent save.
21mins: Robben pass between Edman and King intended for Drogba, superb tackle on edge of box by King to avert danger.
22mins: Carrick piledriver blocked by Terry.
29mins: SAVE. Duff shot deflected out to Johnson, he played it straight back to Duff, now central, 18 yards out, Robinson full length save to his left.
31mins: PENALTY? Chelsea appeals as Robben went over after coming together with Pamarot, appeals waved away. Bit of a scuffle afterwards, nothing doing though.
34mins: YELLOW. Duff pulled back Keane but booked for preventing free-kick being taken.
35mins: Carrick switched play from left to right, King laid ball back to Mendes whose lob floated wide.
39mins: GOAL. Penalty for foul on Smertin by King, looked 50-50, Lampard stuck it away to Robinson's left.
Spurs 0, Chelsea 1
41mins: YELLOW. Johnson for foul on Ziegler.
45mins: MISS. Great control by Drogba, fed Robben, made complete hash of it from 15 yards.
Half-time: Spurs 0, Chelsea 1
49mins: PENALTY? Appeals waved away after Keane went over as Gallas cleared.
56mins: Drogba threatened to break through but Naybet back to make fine sliding tackle.
59mins: CLOSE. Defoe's driven near-post corner sliced inches wide of the far post by Smertin.
63mins: Good link play by Defoe, controlled Brown's pass on chest and rolled into Ziegler who fired wide from 25 yards.
65mins: Drogba met Ferreira's cross with powerful header at near post, ball always rising over.
66mins: SUB. Gardner for Ziegler - King moved up front.
69mins: SUB. Jarosik for Smertin.
70mins: Edman deep cross nodded back into danger zone by King, Defoe laid ball back to Mendes who lashed shot wide.
71mins: SUB. Marney for Pamarot.
71mins: Marney's first touch a superb cross that King missed by inches, ball went through to Keane, cut inside Duff but blazed over.
75mins: SUB. Gudjohnsen for Drogba.
77mins: SUB. Yeates for Mendes.
79mins: SUB. Cole for Duff.
83mins: Brown 50-yard crossfield pass to Yeates, he fed Defoe who shot tamely wide.
84mins: YELLOW. Terry for delaying free-kick being taken.
85mins: YELLOWS. Brown and Makelele for bust-up as Edman tried to take free-kick.
85mins: Carrick eventually took free-kick, curler from 20 yards easily taken by Cech.
Injury time: GOAL. Gudjohnsen dribbled across byline and cut back to Lampard who drilled into corner.
Spurs 0, Chelsea 2
Injury time: Gudjohnsen beat offside trap, curled around Robinson and just wide from tight angle.
Full-time: Spurs 0, Chelsea 2
Spurs (4-4-2): Robinson; Pamarot (Marney 71), Edman, King, Naybet; Mendes (Yeates, 77), Brown, Carrick, Ziegler (Gardner, 66); Keane, Defoe. Subs: Fulop, Kelly.
Chelsea: Cech, Johnson, Ferreira, Makelele, Terry, Gallas, Smertin (Jarosik, 69), Lampard, Drogba (Gudjohnsen, 75), Duff (Cole, 79), Robben. Subs: Bridge, Cudicini.
A controversial penalty and injury time goal gave us the blues against Chelsea once more in the Barclays Premiership on Saturday.