This may take more than one edit to finish. Please be patient.
There is no reason to give up. There are many highly ranked people in the CS community who are pretty far out on the autism spectrum. But they have learned to act productively. With effort, you can too. But it takes a lot of effort, as you know.
My first recommendation is to develop a skills assessment for yourself. What do you do well and what do you find hard. Your assets and liabilities. A trusted advisor can help you with this as he/she can see things that you miss. Then you want to apply your strengths to help overcome your weaknesses.
The most important thing, in my view, is not that you achieve any particular level of competence, but that you develop a satisfying life for yourself, whatever that is, and whatever that takes. Your post seems to suggest that it will include pushing yourself as hard as you can.
Since you believe yourself to have both poor math and CS skills, you may need to work on both of those. But you can do it together, and a bit at a time. But if you have a typical case, one of your strengths is that you can focus intently. That is helpful in both math and CS. Work on some CS idea/problem. If the math gets sticky, then go to that for a bit to work on the basics. Your progress may be slow at first, but that is true of nearly everyone.
Don't depend on memory alone for learning. Take lots of notes - and I recommend paper notes for it, not computer notes. Writing out ideas is a form of practice that will solidify the concepts in your mind. When you are out wandering about, make sure you have something to write on - such as a few index cards, on which you can capture ideas.
If social interaction is a big difficulty for you, you should also find ways to work on that. One way to interact with people without losing yourself is to learn to role-play the way actors do. You try to "become" someone that can interact freely, just by pretending you are someone else who has that skill. An important CS researcher/speaker/writer developed these skills by joining an acting group in which the roles were formalized. He is an excellent public speaker now. It will be psychologically very difficult at first, but this idea has been used successfully. I don't have quite the same difficulty, but improved my social interaction just by forcing myself to speak when I'd rather just "hide."
One thing you may need is an academic guide. The classroom may not be your most comfortable environment, but the stated curriculum of a college can give you a sense of what is important to learn so that you work toward meaningful goals.
(I'll try to improve this as I think of things)