Ian Austin has become the ninth MP to quit Labour this week, blaming leader Jeremy Corbyn for “creating a culture of extremism and intolerance”.
He told the BBC the leadership had failed to tackle anti-Semitism and had turned the party into a “narrow sect”.
But the MP for Dudley North said he had no plans to join the new Independent Group of former Labour and Tory MPs.
A Labour spokesman expressed “regret” over Mr Austin’s decision but called on him to face a by-election.
“He was elected as a Labour MP and so the democratic thing is to resign his seat and let the people of Dudley decide who should represent them,” the spokesman added.
However, Mr Austin told BBC West Midlands his “work for the people in this community is going to carry on as it always has”.
An MP since 2005, Mr Austin said: “This is the most difficult decision I’ve ever had to take but I’ve taken it because I’ve become ashamed of the Labour Party.
“I grew up listening to my dad, who was a refugee from the Holocaust, teaching me about the evils of hatred and prejudice.
“One of the main reasons I joined the Labour Party as a teenager here in Dudley more than 35 years ago was to fight racism and I could never have believed I would be leaving the Labour party because of racism too.”
Khalid Mahmood, Labour MP for Birmingham Perry Barr, called Mr Austin’s decision to quit the party “very sad”.
But he said Labour under Mr Corbyn was “doing its best” on anti-Semitism, and was operating a “zero-tolerance” policy on the issue.
“On that basis, we should move forward, together,” he said.
Mr Austin was a minister for regional affairs under Gordon Brown and part of Ed Miliband’s front-bench team in opposition.
He said he “agreed” with the eight MPs who left Labour earlier this week that things “have got to change”.
However, he wants a Brexit deal concluded, rather than a further referendum on EU membership.
One of the defectors Luciana Berger, who has described the party as “institutionally anti-Semitic”, tweeted that she fully understood why he had come to “this difficult and painful decision”.
Another, Chuka Umunna, tweeted his “massive respect” to Mr Austin, adding: “It’s painful and hard but he has stayed true to his values and what he believes to be the national interest.”