That's exactly what I was going to say, except without the personal experiences. It's probably much the same way many babies use the word "mommy" to mean, "I want something". Because mommy is diaper change, food, hugs, etc. The switch of you/me is well known for being difficult for kids. Both because they don't realize at first that those words are referants that vary and thus need to be changed, and because perspective takes time to build up.

One of the classic experiments to see how well children understand someone else's perspective is to ask them to show a picture to someone. When they are very little, they generally hold it so that they can see it. As they get older, they take more care to make sure the other person can see it.

Even my nephews, in kindergarten and Elementary School, seem to have gotten confused when I asked them to give something to my father, rather than asking them to give it to their grandfather.

Basically, I think it's just actually difficult to catch on to these sorts of things. Fortunately, kids do it anyway.