What can I do? Please i need advice, before its too late!!?

So this guy i like, we are friends and get to hang out alot. we also get alot of alone time one on one, and its really easy to be myself around him. We laff and smile and i catch his eye from across the room all the time. The other day we were stting next to eachother and he put his arm on my shoulder as i leaned my head against his arm and let me wear his shorts becasue I forgot mine for basketball. He gets nervous sometimes, and i can tell he likes being around me. But the thing is he is an amazing swimmer, and every single girl is all over him when we are in public flirting with him and competing for his attention. I back off bcuz it all frustrates me, and im not sure what to do, because i like him alot but i am not sure how he really feels about me, and then all the flirty girls..... please help me