No, it's not just you, he may be biracial but with shaved hair he could easily pass for an Arab on the street, he's very light-skinned and has some caucasian features. http://www.all-pictures.org/celebrities/v/athletes/male/James+Blake/James+Blake+e3.jpg.html
And before the one-drop-rule police bash me for saying that let me just say that it all depends on where you are, if you see him in the US you'll probably think he's black but if you see him on the street in Sweden for example you'd think he's Arab.
I hear you guys. But if he could pass for an Arab the reason is because Arabs themselves are physically some sort of intermediary spot on the color line between a dark black man and a pale white man. For me to mistake him for an Arab I would have to see more nose (not intending to stereotype, but generally speaking). To me he just looks like a mixed kid who got dad's color and some of mom's features.
Not the one-drop rule police, but we had a LONG go-round on this topic a couple of weeks ago. And I do mean LONG, lol.