Eight years ago, I was in my usual seat in the lower tier of the Bobby Moore Stand with a few friends for a nondescript Championship match between West Ham and Plymouth Argyle. Indeed the match was thoroughly forgettable, West Ham cruising to a 5-0 win, and yet the events of that day will never leave me. It had nothing to do with the football and everything to do with the vicious antisemitic abuse and intimidation a friend and I experienced from our fellow supporters.

Midway through the first half the usual anti-Tottenham songs started. They were nothing to get offended about; just typical football songs. West Ham fans tend to sing about Chelsea and Spurs every week. Just banter. Nothing to worry about – until someone tried to get a chant of "Hitler was a cockney" going. The implications were clear: Hitler supported West Ham because he killed Jews and Jews support Tottenham. The fact that the song came from one person did not make it any less shocking and the fact that there were two Jewish West Ham fans sitting in front of him, not that he was to know, did not bother the perpetrator. Nor did the fact that the Nazis flattened the East End during World War Two. Some cockney.

It can be preferable to bite your lip – I was only 17 but my friend, who was older, turned around and told the idiot to "fuck off". Unfortunately he deigned not to and the abuse threatened to turn physical from him and his mates as they tried to snatch my friend's season ticket away and told us to "fuck off to Tottenham". Luckily our group was large enough to stop anything truly ugly developing and the stewards also weighed in. The way our friends defended us showed the inherent decency of some West Ham fans, as did a number of people we did not know offering their support at the next home match. Even so, when I arrive at Upton Park on Saturday for the Chelsea game I will probably spot some of the same people from that February afternoon and the memories will come flooding back.

Maybe they were the same people who brought shame to West Ham with the unforgivable antisemitic chants that blighted Sunday's match at White Hart Lane, not to mention the glorifying of attacks on Spurs supporters by Italian supporters in Rome last week which left one supporter fighting for his life in hospital. Nazi salutes, hissing to mimic the gas chambers and chants of "Adolf Hitler, he's coming for you", "You're getting gassed in the morning" and "Viva Lazio" – anyone who wishes to defend any of this has no place in a football ground in the 21st century. Believe it or not, and it is astounding that this still needs saying, the Holocaust is not funny.

A look at West Ham messageboards on Monday reveals the usual mealy-mouthed apologists spouting stone-age drivel about the sanitisation of football and political correctness gone mad. Yet if we are to accept that it was only a minority who disgraced themselves, it is also true that this minority are the ones who shout loudest – and perhaps punch hardest when challenged.

Antisemitism and racism have existed at West Ham for years. Before a play-off semi-final at Ipswich in 2004, I heard a chant of "Spurs are on their way to Auschwitz, Hitler's gonna gas them again". No one did anything. There is a chant mocking Spurs fans for having no foreskins that ends with a cry of "Fucking Jew".

People call Carlton Cole a black bastard. When Jermain Defoe missed a last-minute chance during a draw with Burnley in 2003, the person in front of me lost the plot, kicking the chair in front of him and screaming racial abuse. During a match against Everton in 2010, Cole missed a late sitter, prompting one fan to bellow that he was a "fucking nigger". He's still there every week.

Football is a working-class sport and its blokiness means that people do not want to be seen as a grass or a snitch. You could tell a steward but chances are people will know and that creates its own problems. Ultimately is it worth the hassle? How do you reason with someone who thinks that the industrialised murder of six million Jews is an acceptable way to score points against Tottenham fans?

Yet it needs people to be brave enough to stand up to this. It is no good doing nothing during the match and then taking to the internet to write it off as a minority action afterwards, because the whole club is tarred by association. It is no good to claim these people are not true West Ham fans, because that is merely a semantic debate which moves the goalposts. They were at the game supporting West Ham. Sounds like a West Ham fan to me. That is not to suggest that the majority are not sickened by what occurred but they must show it and not retreat into the default position all fans assume when their club is in the dock.

Of course, supporters can only do so much and stewards and police actively need to be on the look-out for this sort of behaviour. Those identified should be thrown out and banned for life. To label West Ham a racist or antisemitic club would be wrong yet they might have to accept any punishment that comes their way from the Football Association, even if they cannot truly be held responsible for the actions of morons who bought a ticket.

The irony is that West Ham is a club withhave a Jewish heritage, albeit not as pronounced as Tottenham's, whose fans call themselves the Yid Army to counteract the jibes. The co-chairman, David Gold, is Jewish, they have had a Jewish manager, Avram Grant, they have a Jewish player, Yossi Benayoun, and there are many Jewish supporters. Spurs are seen as The Jewish Club, but not every Spurs fan is a Jew and not every Jew is a Spurs fan. Some will say that Spurs fans are asking for it by calling themselves Yids; again this simply feels like a roundabout way to justify the abuse.

For a West Ham fan, there is nothing bigger than a match against Tottenham. Yet on Sunday, as the Spurs goals flew in, I found myself not caring very much about the outcome. It is a crisis of identity; part of the thrill of supporting a side is the sense of belonging, the rush of being part of a crowd, a community, but that does not feel particularly appealing now. What would make me happy – West Ham winning – would also make these people happy and how can that make me happy?