Michael Brown had the best chance and forced Thomas Sorensen into the best save of a first half where we more than held our own.
We had our best spell of the game at the start of the second half with Sorensen keeping out Defoe before the striker fired just wide.
But with us looking in control, Villa struck what turned out to be the winner when Nolberto Solano swivelled and volleyed home.
Now both sides created chances with Paul Robinson saving superbly from Solano, Fredi Kanoute heading inches wide and Defoe having legitimate claims for a penalty waved away.
"I won't say we dominated the first half but we didn't give away anything and we could have scored through Michael Brown," reflected Martin at Villa Park.
"In the second half we started even better. It was our best spell but then their corner came and we gave away a sloppy goal because we had a lot of height so we didn't expect them to score from a set play.
"After that I thought we had a couple of moments where we could have had a draw and we should have had a draw."
Nevertheless, Martin remained positive. "Before the game we were a bit worried because we had given away eight goals in three matches but I thought we had a good performance.
"Coming to Villa Park isn't easy and Arsenal at home and Villa away are not the games you want at this stage but we did well and we can build on this performance."
Martin Jol thought we deserved a draw from last night's Premiership encounter against Aston Villa - but insisted we can build on the performance after the 1-0 defeat.