It could cost them the House in the midterms. The better tactic in my opinion for the Republicans to use would be to just use the "scandals" to fire up their base and get them to turn out at the polls. An attempt to impeach Obama would have a severe backlash.
Long term Republicans are not real fond of the Tea Party for good reason but I believe they are caught between a rock and a hard place. In reality they probably were not overly concerned that the ultra right wing of their party was undergoing scrutiny by the IRS. What they may fear right now is the IRS scandal will revive the Tea Party and further threaten their control of their party.
The older leadership of the Republican Party probably wishes the Tea Party would simply disappear and become a little known footnote in political history. The currently elected Tea Party Republicans make it difficult for the more moderate Republicans to compromise and consequently the entire Republican Party is becoming known as the party of No. The established Republicans are more than likely scared that an ultra conservative Tea Party candidate will win the Republican nomination and run in the next Presidential election. That will once again doom their chances of controlling the White House.
To me it is possible that the more important issue will prove to be the investigations launched against the press. While the press has not been as favorable to Obama as Fox News insists, it is not wise to really piss them off which may happen if more investigations are uncovered. The press may not totally support the Second Amendment but they really value the First. It seems that every President develops a dislike of the press during his time in office. Still the press has the responsibility of being a watch dog in our political system even when they limit what a President can accomplish. I fear the day when the freedom of the press is limited as every citizen should. I remember "Tricky Dick" Nixon all too well.
Another major factor in the midterms and the next Presidential election will be the implementation of Obama Care. If all goes well, Democrats will fare well in both elections but if it is indeed a "train wreck" as even some Democrats have suggested the elections may go poorly. Of course the Republicans and the Tea Party will be largely responsible for the difficulty but the voters may not realize this.
Obama has great visions for our nation and he is well liked. Still I would have liked him to show more leadership in the early stages of the creation of our new health care system. With all his oratory skills I feel he could have forced a much better plan through Congress. Still while it may not be all I hoped for, the new health care system is a vast improvement over what we had.