There are several laws that come into play in your situation. The laws would depend on which state you legally reside in. For a situation as complex as yours, you may be best off contacting a liscensed attorney for your state. In most cases they will charge a consultation fee but at least you will know what to do from word of mouth ie Licensed attorney. Good luck

The first question that comes to play here is the state of laws that applies to where you reside.

But one thing is however clear here, in the absence of any proof that he shared the account with you, he has no right to one dollar in that account. Even if he made regular deposits into that account is not enough proof that he has any right in the money in the account.

Be that as it may, the document you guys made might contain something that gave him right to that fund in your account.

In all, seeking the advice of an attorney will help you take a better decision, this does not mean you are getting involved in any legal action. As it appears, i don't think he has right to your money unless you are given it to him purely on compassionate ground since the account is not joint. And make sure you don't say any implicating thing as far as the issue is concerned. See an attorney, it could even be for free

Also if the home is built by the both of you then you both have rights to the house. See a lawyer