EX Wife's grandaughter....

I have recently married an older man, he was in a very bad relationship for 15 years with a woman who had 2 children. He had help raise them and the daughter, had a baby girl he support as well. The mother and the daughter dumped the baby on him quite often so they established a close relationship. They divorced about 4 years ago. We've been married for 2 years. He says in contact with the grandaughters father (who is a jerk to me)and divorced now for quite sometime from the ex-daughter. My husband insists we drop everything if the ex-son in law calls and wants to bring over the ex-grandaughter. I've had about enough. I do not mind the little girl, but the ex-son-in-law, his new wife and now they have a new baby are more than I can tolorate. It would be so different for me if my husband treated his own blood grandkids (he has 4) the same, but he does not. NOT at all. I expressed my feelings to him and told him if he wants to spend this time with these people who are not even relatives anymore, he can do it on his own. It still hurts my feelings that he cares for them so much and not his own family, and these people flat out told me when he was not around that they were just using him. I think they think he will put them in his will...and he may. Am I doing the right thing? or should I be more supportive? I just cannot stand these people knowing they really do not care for him, but his generocity and money....they know how to get to him...it is thru the grandaughter he was once very close with. it is sad to watch.