NYSE Euronext and Deutsche Börse of Germany said on Wednesday that they are in talks to call off their planned merger, after being notified that European antitrust regulators formally oppose the deal.

Both exchanges said that they fundamentally disagreed with concessions that the European Commission had requested, notably the divestiture of major parts of the combined company’s business.

“This is a black day for Europe and for its future competitiveness on global financial markets,” Deutsche Börse’s board said in a statement. “The E.U. Commission’s decision is based on an unrealistically narrow definition of the market that does no justice to the global nature of competition in the market for derivatives.”

The E.U. competition commissioner, Joaquin Almunia, said the proposed deal would have resulted in a quasi-monopoly in the area of European financial derivatives traded globally on exchanges and that the companies would have controlled more than 90 percent of global trade in these products.

“These markets are at the heart of the financial system and it is crucial for the whole European economy that they remain competitive,” Mr. Almunia said on Wednesday. “We tried to find a solution, but the remedies offered fell far short of resolving the concerns.”

NYSE Euronext’s board learned of the European Union’s opposition at a board meeting in London on Wednesday morning.

The decision leaves NYSE Euronext, the operator of the New York Stock Exchange, to figure out a new course after having wagered on creating the world’s biggest market operator. The company is likely to look to smaller acquisitions and cost-cutting as a way to grow, and said on Wednesday that it plans to resume a $550 million share buyback program.

It also marks the latest collapse of a merger of exchanges over the past 12 months. Many ran aground because of a failure to win regulatory approval, including the proposed union of the Australian and Singaporean exchanges.

Executives at NYSE Euronext and Deutsche Börse have acknowledged in recent weeks that their merger appeared increasingly doomed. Duncan Niederauer, NYSE Euronext’s chief executive, told The Financial Times last week that he held only “a glimmer of hope” that the deal would succeed, and conceded that he had “misjudged” the review process.

“While we are disappointed and strongly disagree with the E.U. decision, which is based on a fundamentally different understanding of the derivatives market, it is now time to move on,” NYSE Euronext’s chairman, Jan-Michiel Hessels, said in a statement on Wednesday.

The deal’s collapse will come as little surprise to analysts and investors, many of whom were cool to the idea. Shares in NYSE Euronext have fallen nearly 17 percent over the past year, while those in Deutsche Börse have tumbled 19 percent.

Announced last February, the merger of NYSE Euronext and Deutsche Börse would have formed a powerhouse new exchange, one with a formidable platform for the trading of stocks and derivatives.

The two had surmounted several challenges, including a rival bid for NYSE Euronext by Nasdaq OMX and the IntercontinentalExchange, secured shareholder approval for the tie-up.

But the management teams of both companies had underestimated the potential for opposition from European antitrust officials. Led by the European Union‘s competition commissioner, regulators argued that the merger would wield too much power in the market for derivatives traded on exchanges.

“It took American observers by surprise, given how accommodating U.S. regulators were to the roll-ups by the CME Group,” Ed Ditmire, an analyst at Macquarie Securities, said. “I can’t help but think that governments and politicians are treating the financial industry much more skeptically than they were before the crisis.”

NYSE Euronext and Deutsche Börse officials countered that their combined company would control only a small portion of the entire market, which includes the trading of derivatives over the counter. They have also said that the regulators’ analysis was too focused on the European market.

Among the concessions requested by European officials was the sale of either NYSE Euronext’s Liffe platform or Deutsche Borse’s Eurex unit. Both companies have strenuously opposed such a move.

“As we made clear throughout this process, we would not agree to any concessions that would compromise or undermine the industrial and economic logic of the proposed combination,” Jan-Michiel Hessels, NYSE Euronext’s chairman, said in a statement.

Under the terms of the merger, neither NYSE Euronext nor Deutsche Börse will be responsible for paying a break-up fee of 250 million euros, since the deal was scuttled because of regulator opposition. But either company would need to pay that fee if it strikes a merger with a different partner within the next nine months.

Instead, both companies have said that they will focus on smaller acquisitions that will bolster existing businesses.

Mr. Ditmire suggested that other kinds of mergers might still pass regulatory muster. NYSE Euronext could look to bolster its commodities or futures trading operations, potentially with an American partner like the IntercontinentalExchange.

The company could also look to find a merger partner in Asia, but would be wary of running into the kind of nationalist opposition that scuttled the Australian-Singaporean exchange deal.

In the meantime, NYSE Euronext will likely turn on other alternatives to improve its stock price. One is more cost-cutting, something that had slowed down last year as executives focused on the merger. Mr. Ditmire said that NYSE Euronext still has low operating margins compared with its peers, running at about 35 percent to some rivals’ margins that run higher than 60 percent.

“We expect NYX to return to is multi-year cost-reduction program that similar (if not better) cost efficiencies can be attained in the coming years,” Richard Repetto, an analyst with Sandler O’Neill & Partners, wrote in a research note earlier this month.

NYSE Euronext could also turn to repurchasing stock. Mr. Repetto argued that the company’s strong cash flow, sound credit ratings and low debt levels could let it borrow money to augment an existing buyback program.

James Kanter contributed reporting from Brussels