When Ian Holloway left Crystal Palace on 23 October he was a broken man. The effervescent character and cheery demeanour were replaced by a haggard, withdrawn look. But when Millwall, who teeter a point above the Championship relegation zone, came knocking last weekend, his excitement returned. It was nice to feel wanted again, even if he had not envisaged such a swift return.

While Palace have steadied under their new manager, Tony Pulis, and sit a point off Premier League safety, Holloway still believes the team had lost their fight under his guidance and, suffering from exhaustion, he decided it was best for all concerned to step aside. He even suggested Pulis as a replacement to the Palace co-chairman, Steve Parish, on the night of his final game, a 4-1 defeat by Fulham.

"It was my philosophy," he said of his reason for vacating Selhurst Park. "What Tony Pulis did at Stoke was outstanding and he wasn't appreciated. He has never been relegated and after the Fulham game I felt as flat as a pancake. We were excellent for 43 minutes and they bang two goals in and I felt the heart was out of the team. The chairman [Parish] spoke to me afterwards and I said you are better off with Tony.

"He has more experience at locking up defences and coaching that way, so I believe they have a better chance. I helped them get all that money [£15m from the sale of Wilfried Zaha to Manchester United] and found them someone else but when you work that hard it drains you. Anyone who goes up, especially in the last [play-off] position, is immediately playing catch-up. All of a sudden all the new players came in [Palace made 16 signings] and I was exhausted because I had no chief scout and was blind, and you don't want to get promoted and be blind."

The manner of his departure led to inevitable questions about his credibility as a manager in the top tier, yet Holloway rejects the claim that, despite success with Palace and Blackpool in the second tier, he might be better suited to the Championship. "If I had a fair crack as a Premier League manager, with a budget as good as some, I believe I can do it. But I'm going to have to get there – I'm not one of these foreign fellas who comes in because he has known José Mourinho for 10 minutes. I'm not being funny but that normally gets you in the door, doesn't it?"

He sees no reason why Millwall cannot follow in the steps of Palace and Blackpool, even if the immediate objective is survival. It was a phone call from their former manager, Kenny Jackett, that sparked the idea of succeeding Steve Lomas, who was sacked on Boxing Day, and once he met the club's US-based owner, John Berylson, he was "blown away" and had no hesitation in committing to a two-and-a-half-year deal.

"It's a privilege to be here. End of story," he said. Yet talk of success is tempered by the starkness of the club's current position. They have conceded a league-high 49 goals in 24 games and have managed two wins in 15, capped by a chastening 4-1 defeat by League Two Southend in the FA Cup last Saturday. There are also suggestions that having so many senior personalities in the changing room is detrimental.

"I have to unite us, join everybody together. At the minute there are too many chiefs here, not enough Indians. There are a lot of senior pros just added, I have to clear their minds. I have already told them I can't pick them all in the one team so stop moaning, sit on the side and be happy to help when you come on.

"They all know I will tell them the truth and give them what they want. That senior group is something I can't wait to work with. I might have to prune it, I don't know yet until I see how they respond to each other. There needs to be a balance between senior players and youth players, experience and inexperience. It isn't going to be a quick fix. I have to dig in and I have guaranteed that, if anybody in the world came in for me right now, I would not go."