Black authority figures on the screen.

I'm catching up on old TV, and I'm noticing something... disturbing. TVTropes actually has a discussion on the subject, but their discussion is limited to "Majority white country, black authority figure on the show", and I'm thinking it's more complicated than that.

Often, it seems, you have a white or mostly white cast, and their boss is black, and said boss doesn't get very many lines. Which makes sense, because we're interested in our main character, not their manager... but after watching now a few shows with this dynamic it's starting to feel like they're deliberately casting their token black guy into the role that justifiably will have few lines - and he's in charge, so it's totally not racist, right?

Am I wrong about this? Is my TV-watching very skewed by the fact that I don't get out all that much or something?