The Iceland midfielder entered the action just after Clint Dempsey fired us ahead in the 76th minute at Goodison Park.
He was so close to making it 2-0 in the 85th minute. Jan Vertonghen broke down the left and fired in a pass that Gylfi controlled and then shaped to curl into the top corner from 25 yards.
Everything looked right but with Everton goalkeeper Tim Howard rooted to the spot, the ball hit the top of the crossbar.
We looked in control in those last few minutes 1-0 up but it all changed in injury time. Steven Pienaar headed home the equaliser in the 91st minute and 90 seconds later, Nikica Jelavic pounced to slot home the winner from close range.
“I think if my shot had gone in it would have killed the game off with six, seven minutes left,” said Gylfi.
“If we’d have gone 2-0 up we’d have been more comfortable but Everton are always dangerous and towards the end, they pushed players up front and eventually punished us.
“The lads battled really hard in the first half, we got the goal in the second half and then I thought we did well, we kept the ball but they put more pressure on us in the last few minutes and got the goals in the end.
“To go home with nothing after a performance like that was very disappointing.”