Voters angered by David Emerson's jump to the Conservatives held another "de-elect" protest outside his Vancouver riding office on Sunday, while Emerson said he will not step down.

"The issue here is representing the will and the wishes of the electorate," said protester Kevin Chalmers.

"You don't run on one party's platform and policy, and then the next day decide to go to another party. That's not how democracy works."

Chalmers is organizing a "De-Elect Emerson" campaign, and a dozen of his signs have already been sighted on front lawns throughout the riding.

"I think (Emerson) should come back and have a byelection and do things properly," said David McMillan, another frustrated constituent.

The Vancouver-Kingsway riding where Emerson was elected is a traditional Liberal stronghold. The last time a Conservative won the seat was in 1958. In the most recent election, the Conservative candidate won just 18 per cent of the vote.

But Emerson is not backing down from his decision to cross the floor of the House of Commons.

"It may make some people angry and it may be offensive to some, but I'll stand by my decision," he told CTV's Question Period on Sunday.

Although, he added that he will send a letter to his constituents that contains an apology.

Emerson said he had no idea his decision would cause so much outrage. "When I came into politics I came in as a 'Paul Martin Liberal,'" he said.

"I was not a political person. I am still not much of a political person, as perhaps you can now tell ... But I got into public life because I wanted to serve my community, and serve my riding, and serve my province and serve my country."

During the campaign, Emerson harshly criticized the Conservatives, saying Harper's vision for Canada would be one where the rich got richer and the poor would get poorer.

But Emerson said his opinion of Harper has evolved since the election campaign.

The former CEO of lumber giant Canfor, who will serve as Minister for International Trade and the Vancouver Olympics, said he was not worried that the controversy would hurt his credibility.

"I don't think that a lot of the people who are involved in softwood lumber are going to look at me as any less effective today than I was 10 days ago," he said.

Harper remains quiet

Meanwhile, Prime Minister Stephen Harper is avoiding questions about Emerson's jump and appointment as minister for international trade.

Former Newfoundland premier Brian Tobin said Harper has much to answer for, since he invited Emerson to cross the floor.

"I think he should go out there," Tobin said on Question Period. "He made the decision. It's his. It's wasn't David Emerson's decision to cross the floor, it was Stephen Harper's decision to invite him to."

Harper's supporters say the new prime minister is simply too busy to answer the media's questions.

"I think the public, if I may say so, are prepared to cut the government a bit more slack as they get organized than the media are," said Senator Hugh Segal.

With a report by CTV's Roger Smith in Ottawa