It really varies tremendously depending on where you are. To start, I actually did meet a Japanese guy who identifies as a xenophobe. Nice guy. He was just kind of tense and uncomfortable around me, but that's all. When you're in the larger cities like Kyoto and Tokyo, no one takes too much notice. Tokyo is Japan's largest city and is a bustling business area, so foreigners are nothing too out of the ordinary. Kyoto is among the top tourist locations in Japan (the whole city is a tourist spot - it's amazing), so they don't find it too strange when foreigners visit either.
When I was there, I was in Nagoya. It's a large city, but nothing like Tokyo (not many places are). It has the very urban-feeling downtown area, but as you get further out, it becomes more residential with more farmland. Pretty standard situation there. When I would ride the subway, I was hardly ever noticed (at least not blatantly so). When I would switch to the bus, that's when things got iffy. Things were usually just fine, but the only negative experiences I can think of right now occurred on the bus when I was getting further and further away from downtown. I had a young guy, maybe about middle or high school age, get on the bus and just give me this random death glare. He was wearing a mask (not sure what the specific term is, but it's the one they use when they're sick or trying to prevent becoming sick) which only made it creepier cause I could only see his eyes. I was right next to the guy and every time I looked in his general direction, he was glaring at me.
A similar thing happened but with a much older guy on the bus. I was standing in the middle and he was toward the front. It was a pretty empty bus. Like the younger guy, he gave me the death glare for the longest time for no reason.
I was there for a full nine months or so. In all that time, those are really the only experiences that really stick out as being negative. On the whole, I wouldn't go as far as saying they're xenophobic. They're definitely comfortable with their current situation (mostly just Japanese in the country), but that's not surprising.
There's some friction between the Japanese and what are referred to as "zainichi kankokujin," or Koreans residing in Japan. These are Koreans who were brought to Japan during Japan's annexation of Korea and some of them have stayed and raised families. At this point, though, there are third- and possibly even fourth-generation zainichi kankokujin living in Japan who have never even been to Korea and don't know the language. Yet they are still treated as foreigners because they come from a Korean family.
There are complications with the Chinese, too, but I don't want to bore everyone anymore than I probably already have

Suffice it to say that Asians are placed under particular scrutiny, and on the whole the Japanese are
not what can be called xenophobic.
That quickly exploded into much more than I intended to write >_> Take from it what you will. Should you or anyone else want more explanation, just say the word.