Not since Britpop swaggered and staggered across the country, high on its own importance and plenty else, have Tottenham Hotspur fans been able to brag about finishing above Arsenal in the league. That was in 1995: the year of Braveheart and Barings. Nearly a generation ago. At the time Gareth Bale was a five-year-old schoolboy.

Times have changed in other ways. Spurs' original Welsh left-wing wonder, Cliff Jones, one of the stars of the 1961 Double-winning side, recalls that before his first north London derby his manager, Bill Nicholson, told him: "Son, this is the biggest club game you will ever play in. It is bigger than Rangers v Celtic, Manchester United v Manchester City or Everton v Liverpool. The two most important results for Tottenham are always Arsenal at home and Arsenal away. And if you win those two the fans will be happy."

Nicholson's maxim no longer holds. Spurs have won their share of recent skirmishes – since 2008 both sides have four victories, with four draws, in north London derbies – but parity on the pitch has not been reflected in the final league table. Tottenham fans long to look down on their neighbours when the season packs up in May, not just when spring is warming the loam. That they have come so close, so often – Spurs have finished a place behind Arsenal in five of the past seven seasons – makes it hurt even more.

Jones, who still works for Tottenham on match days, is optimistic that his old club can scratch their 18-year itch – and says that he sees parallels between Nicholson, the greatest manager in Spurs' history, and the incumbent, André Villas‑Boas.

"Bill was very forward-thinking," he says. "He was one of the first to introduce weight training and our fitness took off. At the start of pre-season there was always a posse of overseas coaches noting down Bill's schemes of work, training and coaching because it was so advanced. From what people tell me, Villas-Boas has a very similar mindset and approach.

"They are both fascinated by detail but it's more than that. I can see a bond and spirit there, similar to what we had in the late 50s and 60s, and a lot of that is down to the manager. He is doing a fantastic job."

Martin Chivers, whose goals helped lead Spurs to victory in the 1972 Uefa Cup final, concurs. "Bill was a magnificent manager and our team was like a family," he says. "AVB is building a family too. The players are very happy, and that's a big thing. They are starting to blend. The club had a taste of the Champions League a couple of years ago and now we want it again."

The formbook offers plenty of encouragement too: Spurs are unbeaten in the league since losing at Everton on 9 December, and have lost one in their past six at home to Arsenal. They also have the x-factor in Bale, whose goals have been rewarded with a sustained media love-in and the player of the month award for February. Recent history, however, holds up a huge warning flag.

This time last year Spurs were 10 points clear of Arsenal going into the derby. The narrative was as stale and recycled then as it is now – talk of Arsenal's flimsy defence and Arsène Wenger's fading powers was everywhere. And then things veered wildly off-script, with Arsenal gloriously coming from 2-0 down to win 5-2 and finishing the season with a sprint to beat Spurs in the photo-finish for third.

Could it happen again? Bob Wilson, Arsenal's goalkeeper in their 1971 double-winning side, does not sound convinced. "A majority of Spurs fans I meet still think that Arsenal will finish ahead of them. They tell me: 'We'll blow it, we'll blow it'. But that will be hard to do it now and they could easily finish third."

Ossie Ardiles agrees. "In the Wenger era they have always been favourites," he says with a glint in his eye. "But for this game on Sunday I think we are. But it's a derby. Who knows?"

The murk surrounding the battle for Champions League places may become clearer in the next fortnight. Spurs face Arsenal, Internazionale twice in the Europa League, and Liverpool at Anfield. Arsenal have Everton and visits to Bayern Munich and Swansea City on the horizon. But for now all focus is on Sunday.

"The players will already be in a kind of haze because there is so much pressure involved in the game," says the former Arsenal striker Ian Wright. "When I played with Tony Adams, Paul Merson, and all those guys that grew up in the Arsenal system, the Tottenham match meant the world to them. We talked about it for a month beforehand and there was always a very high energy in the dressing room. I just hope it's the same on Sunday."

Wright was part of the last Arsenal team to finish below Spurs in 1995. As was Alan Smith, who is honest about the uncontrollable slip-slide in the club's fortunes in the latter stages of George Graham reign and then, briefly, under Stewart Houston. "We were a cup team by then," the former forward admits. "We'd won the league and cup double in 1993 and the Cup Winners' Cup in 1994 but we weren't challenging for the league. We could keep a clean sheet with our back four but we weren't good enough for a title tilt."

Arsenal – the Arsenal of Seaman, Adams, Winterburn, Dixon and Keown, remember – finished 12th in 1995. Spurs were seventh. But any gloating and glee did not linger. That summer the Famous Five, Tottenham's pentagon-shaped attack, was dismantled. The departure of the Golden Bomber, Jürgen Klinsmann, coincided with arrival in N5 of the non-flying Dutchman, Dennis Bergkamp. A year later Wenger joined and a holding pattern was established. From 2000 to 2007 Arsenal won 12 and drew seven of their 19 games against Spurs. It was not so much a derby as a mismatch.

The situation has turned since then and it could yet flip completely. Wilson hopes that Arsenal fans realise how lucky they have been. "I'll be there on Sunday and I'm almost wishing I wasn't going," he says. "But this is the time when if you are a true fan, you go there and you stand up and be counted. And if they beat us, you have to take it on the chin. Because for 18 years we've been living in cuckoo land.

"The young ones, those who don't remember all the lows we have had, are living in a false world," he insists. "When Arsène arrived as an unknown manager we would have bit his hand off to win three titles, four FA Cups and finish in the top four every year. And remember we went 17 years without a trophy before we won the Fairs Cup in 1970."

You suspect context and reason will soon dive out of the window come 4pm . And probably defending too, given there have been 53 goals in the last 11 derbies. Unsurprisingly Spurs are strong favourites given the fitful way Arsenal have been playing recently; then again, football's animal spirits may yet befuddle us once more.

Sean Ingle was at an event for the Willow Foundation. Founded by Bob Wilson and his wife, Megs, the charity provides special days for seriously ill 16- to 40-year-olds. Visit www.willowfoundation.org.uk