The ladies trailed twice in the second round tie at the Withdean Stadium but fought back on both occasions, only for the Seagulls to grab the winner 15 minutes from time.
Brighton took the lead through Tammy Waine on six minutes but the team responded well and Jess Cheeseman forced a fine save from Toni-Anne Wayne in the home goal.
It took just four minutes for Spurs to level in the second half as Jenna Schillaci picked up a loose ball after a corner and curled back into the top corner.
Bianca Baptiste struck the crossbar from distance but within seconds, ex-Spur Rosie Kmita was sent clean through on goal and slotted past Chloe Morgan.
The lead only last five minutes though as Fran Gamble floated in a free-kick from half-way and Wendy Martin planted her header home for 2-2.
Brighton grabbed what turned out to be the winner when Kirsten Barton curled home a free-kick from 20 yards.
The team piled forward in search of a third equaliser but Jenna was denied by Wayne, substitute Katie O'Leary headed just over and Wendy's goalbound effort was blocked.
"We're all disappointed not to go through," said manager, Karen Hills.
"We have to have a look at how we conceded the goals, because I felt we gifted them a couple, but if we put it right then I'm positive we can put together another good performance when we travel back to Brighton this weekend."
The ladies are back at the Withdean on Sunday, this time in FA Women's Premier League action.