The striker made no mistake one-on-one in the 29th minute, slotting low past Robins' goalkeeper Dale Reaney as we safely negotiated a real test at Whaddon Road.
Coulthirst's goal sparked a purple patch in the first half that could have put the tie beyond doubt, but the home team stuck to their task and pushed us all the way, forcing Luke McGee into a couple of fine saves to preserve the lead in the second half.
We now face an away trip to the Hammers in the fourth round, to be played by January 19.
The first chance fell to Josh Onomah in the ninth minute and he must have wondered how Reaney kept out his effort from six yards after Kenny McEvoy's cross. Elliott Keightley's block from Roman Michael-Percil's follow up was just as good.
Bobbie Dale curled Cheltenham's first sight of goal over from 20 yards but we then clicked into gear.
McEvoy curled inches wide from the edge of the box on 28 minutes and the opener arrived seconds later. Michael-Percil won possession on the right and released Coulthirst through the middle and the striker did the rest, slotting home under the advancing Reaney.
Milos Veljkovic's attempted cross from the left clipped the top of the crossbar and we had the ball in the net again on 33 minutes but McEvoy was denied by the linesman's flag, following in after Coulthirst's shot was palmed out by Reaney.
Alex McQueen tried his luck from 25 yards and then fantastic skill from Michael-Percil lost Spencer Hamilton but Reaney was out in a flash to block from close range.
Cheltenham continued to push forward and had two chances just before the break, both falling to Zack Kotwica. He struck the first a yard wide from the edge of the box after the home team kept the ball in the danger area and he was there again when Dominic Ball misjudged a clearance, forcing Luke McGee into his first action, a comfortable save down to his right.
The start of the second half was a little scrappy. The home team got into some potentially dangerous positions without creating a chance while a couple of crosses from Michael-Percil came to nothing.
McGee had to be alert to tip over Kotwica's inswinging cross on 62 minutes and did even better when Cheltenham's best chance fell to the same player four minutes later. Substitute Ed Williams released Kotwica through the middle but McGee was out quickly and stood firm to block with his left hand.
Again, the next passage of play was relatively quiet until Onomah showed strength to get onto a long clearance before dragging his effort wide.
Coulthirst had our best chance of the second half on 81 minutes as Grant Ward crossed deep, McEvoy nodded back across goal and Coulthirst headed into the ground only for the ball to pop up and over the crossbar.
McGee flung himself to his right to claim Joe Hanks' 25-yarder three minutes from time and when the goalkeeper easily gathered Dale's header in added time, it was all over.
Cheltenham (4-1-4-1): Reaney; Thomas, Gonzalo, Hamilton, Keightley; H Williams; Donaghue (E Williams, 64), Hanks, Powell (Marshall, 88), Kotwica; Dale. Unused subs: Rivers, Kirkpatrick-Jones, Morana.
Spurs (4-3-3): McGee; McQueen, Ball, Ogilvie, Ward; Dombaxe, Veljkovic, Onomah; Michael-Percil (Miller, 64), Coulthirst, McEvoy. Unused subs: Smith, Priestley, Lameiras, Zapata-Caicedo.
Goal: Spurs - Coulthirst (29).
Referee: Mr M Rushton.