I agree with Rodent. Have you talked to him about how you are feeling? Would this still be a problem if you had just as many "friends"?
Also, you say you can't do physical activities and you have a daughter that needs extra attention, but that doesn't mean you can't have a hobby. There are many things you can do at home or even just an hour or so a week outside of the home, it doesn't have to be a physical activity. Could be knitting or adult coloring books or crossword puzzles or even origami. Just find something that you can enjoy that might take some of the resentment away from you for a little while.
This next part might hit a nerve and if it does, I'm sorry and I truly mean no disrespect or fault you in any way, as I also have a child with health (physical and mental) issues. Do you ever feel resentment toward your daughter because you have to take care of her so much? Now, don't get me wrong, I am in no way saying you don't love her or that you would trade her for anything, but when you add that, onto your own issues, and the issues with your partner....well, I can understand it and I have been there before. You have to do everything, it's like there's no help for you, no one to help carry the strain and exhaustion it causes.
I know you said you have a shared arrangement with the father, but how often does he have her? That could be another part of the problem. Is there any type of group or camp or activities she could be in? A support group or something like that....something will would enable her to get out there and learn more about what happening, give her more freedom, more understanding to take her own role in her healthcare. And on top of that, give you the opportunity to meet people going through the same thing.
But yeah, you definitely need to talk to your partner. Don't be accusatory, don't make it into a big argument, talk to him calmly, help him understand how you feel, find some solutions. What about date night? Or set aside some time each day to just talk to each other, even if it's only 30 minutes or an hour. If you don't eat dinner together or it's a big rushed event, start eating at the table and share your days with each other. But you really do need to stop trying to control his friend counts.