Sven Goran's Eriksson's men bowed out against the favourites after failing to take advantage of a Ronaldhino's sending off with 33 minutes left.
England failed to create another chance - Teddy Sheringham was helpless after coming on for the last 10 minutes - and went out to a 2-1 deficit.
Spurs legend Gary reflected: "Before the World Cup started the first thing was could we get out of a tough group? There were doubts if we could do that, but I was confident we would.
"If the tournament had gone to plan we would have played France then Brazil.
"I suppose the disappointment is there because we got to the quarter-finals at such a canter.
"We were also playing a Brazil side who have great individuals but aren't a strong unit and if we could have beaten them, the door looked wide open.
"The first half went well, Michael Owen cashed in on a defensive error and it looked ours for the taking. But obviously they hit back with a great goal and then took the lead.
"Ronaldhino was then sent off and normally against 10 men you'd really push on and throw balls into the box, but I felt we let them off the hook a little.
"The players will be devastated and I'm sure they'll feel they could have done more in that last period. If we'd have gone out throwing everything at them, okay, but I think we could have done more. That's the most frustrating thing, we've gone out with a whimper rather than a bark."
Gary Mabbutt believed pre-World Cup that reaching the quarter-finals would be deemed a success for England - but he can't help feeling an opportunity was missed against Brazil.