On a night of landmark goals, Yossi Benayoun upstaged Robin van Persie and Marc Albrighton to head home three minutes from time to haul Arsenal level with Chelsea and pile more misery on Alex McLeish and Aston Villa. Benayoun came off the substitutes' bench late in the second half to strike a blow that felt cruel on a Villa side who had produced an impressive performance that deserved so much more than a fourth home defeat in five matches.

Arsenal's winner came after Albrighton had etched his name into the record books with the 20,000th Premier League goal, the winger capitalising on a defensive mix-up between Thomas Vermaelen and Per Mertesacker. At that point Villa looked the more likely to go on and win the game, as an Arsenal team who had taken the lead through Van Persie's 34th Premier League goal of the calendar year – he has now equalled the club record Thierry Henry set in 2004 and is only two behind Alan Shearer – were pinned back for long periods.

Yet Villa's inability to defend corners resurfaced when Van Persie swung in a kick that Benayoun, one of the smallest players on the pitch, met with a free header to beat Brad Guzan. It was the ninth goal that Villa have conceded from a corner this season – both Liverpool's came via that source on Sunday – and allowed Arsenal to celebrate a win that even Arsène Wenger conceded was fortunate. "I can only congratulate Villa, they played well and gave everything," the Arsenal manager said.

"We got a very late three points that are vital for us. We were a bit unlucky on Sunday [against Manchester City] and today maybe a bit lucky. You could see that some legs were jaded from Sunday and some minds as well in the speed of the decision-making. But we kept going and there is an exceptional attitude and spirit in the side. And tonight, very unusually for us, we scored from two set-pieces. That is basically historical for us."

To rub salt into McLeish's wounds, Villa had Alan Hutton sent off in the closing seconds after the Scotland international picked up two yellow cards in the space of little more than a minute in stoppage time. Hutton became embroiled in a flare-up with Van Persie, for which he was cautioned, and then collected a second yellow followed by a red when he recklessly brought down Vermaelen. McLeish was particularly annoyed with the first incident. "Alan has just got to walk away from that," he said.

The Villa manager could, however, take great encouragement from a team display that galvanised their supporters. Villa played at a high tempo, pressing Arsenal when they were without the ball and attacking with purpose whenever they broke forward. Albrighton and Charles N'Zogbia in particular looked particularly dangerous on the flanks against a makeshift Arsenal back four that saw Francis Coquelin, a 20-year-old Frenchman who is a holding midfielder by trade, deployed at right‑back.

Villa had made the better start, as N'Zogbia created two chances for Gabriel Agbonlahor inside the opening 11 minutes, yet it was Arsenal that drew first blood. Theo Walcott turned Ciaran Clark with a neat piece of skill and the Villa midfielder, who was one of four changes from the side that lost so meekly to Liverpool, tugged at the shirt of the Arsenal winger, leaving Jonathan Moss, the referee, with no option but to point to the spot. Van Persie dispatched his penalty high into the roof of the net.

Although Walcott and Ramsey both had chances to double Arsenal's advantage, Villa finished the first half much the stronger and it was no surprise when they equalised shortly after the restart. It was a poor goal for Arsenal to concede, Vermaelen's header from Guzan's long punt upfield dropping a little short of Mertesacker, who was caught on his heels. Albrighton nipped in front of the central defender and beat Wojciech Szczesny with an assured finish.

Within the blink of an eye Mertesacker was denied at the opposite end and Van Persie was later booked for diving, when Moss adjudged the striker was falling to the ground before Stephen Warnock made contact with him in the area. That was a measure of Arsenal's desperation at a time when Villa were largely in control, although Benayoun's introduction nine minutes from time provided the late twist that left McLeish cursing what might have been. "If we play like that every game, I'll be quite confident," the Villa manager said.