If I take low level math in college will I be behind everyone else?

Overall, about half the people going to college need to take a remedial math cl. More at community college, fewer at universities, and while it will set you back a bit you won't be the only one there.