The USA international, who has been a model of consistency in his performances this season, had, what could be described, as a bad day at the office when the team crashed out of the Cup by a 4-0 scoreline on the South Coast.
It would have been harsh to single out one particular player for criticism on such an evening, but Kasey was quick to hold up his hands for his part in a performance that attracted much outside ridicule.
Kasey and the team got their campaign back on track with the thrilling 4-3 victory over Everton last Sunday and, while happy to be on the winning side, goalkeeping pride takes a little knock when three are added to the goals against column.
"It was really my worst day in a Tottenham shirt last week," reflected Kasey with a sigh. "You always want to come back and do as well as you can and, obviously, conceding three goals against Everton isn't a lot of fun.
"You always have to look at each goal the way it happens. I also look at the three points that puts us back up into contention again for the European spot that we're fighting so hard for."
Many commentators pointed to Southampton's opening goal as a goalkeeping error, where the ball appeared to ping off Kasey's chest from a James Beattie free-kick. Kasey counters that it was not his chest the ball bounced off.
"Actually, it didn't come off my chest - it came off my face and that was probably the hard part about it," he added, before confirming that the hardest part of his trade is that, unlike outfield players, keepers cannot compensate for mishaps and have to keep a clear head and get on with the game.
"Everybody knows that people are going to miss easy chances. Defenders are going to slip, goalkeepers are going to give up goals that they wouldn't normally give up. It's part of it.
"It would be kind of a boring game if everybody did just what they were supposed to do every time on the field.
"The key is to come back, work hard and try to be yourself. In the next game you have to try and just control the controlables. It is so difficult at times when you have had a bad game and you try and chase it.
"That is one thing you can't do in goal, it is very hard to be proactive. Once you start chasing things you start getting into positions that you're not supposed to be in. You have to react to what happens and go from there.
"If you are a defender you could go up on a corner and maybe make a difference. If you are a striker you could come back and tackle somebody.
"For us, not being able to do anything extra can be a bit frustrating."
In what appears to be a campaign of goalkeeping calamities, Kasey has done pretty well to avoid making any contenders for a end-of-season blooper video, but accepts he was never going to get through the whole term unscathed.
"You have to try and be as consistent as possible and that's what I strive for. It's impossible to play 30-40 games on 'interesting' pitches and five games in 12 games thinking you're going to get through that without making a mistake and getting punished for it.
"You must keep going and keep learning. That's why goalkeepers continue to play until their late thirties because it's always 'what can I do different next time?'. After 400-odd games you start to figure it out."
Kasey Keller concedes that he endured his worst day in a Spurs shirt at Southampton in the FA Cup but explains that, as a goalkeeper, it is about how you carry on and react to such adversity.