Do they have recourse? Well, they probably won't spend a lot of time and money trying to get it back but they will have to get the money back into the plan, either from you or by paying up themselves. They must make the plan whole. They may try to work with your IRA provider but I think they'd start with you first. Ultimately when distributions are overpaid they are sometimes not recovered.

The bigger concern for you might be in not returning the funds. If the money is not yours and you agree with that I would work with them to get it corrected. There is a risk of your entire rollover being declared ineligible by the IRS down the road since you cannot rollover funds that were not yours. Think of it as potentially tainted money. The worst case scenario for you would be to lose tax favored status on the funds and owe taxes and penalties on it.

I believe the best thing for you is to make the correction and get some assurance from them and from your new provider that the issue was corrected properly and that your remaining rollover is valid. If you have lost money on your rollover since you deposited it you should not have to pay that (you should only have to return today's value of the original excess deposit). I think you will sleep best at night knowing your account is as it should be and that it would make the IRS happy.