(This may be a little long, I apologize.)
Two things to keep in mind before we begin.
1. It’s difficult to “date” this problem and I think our analysis of its impacts would be limiting if we did. Rather than attempting to date this, it should be apprehended as a gradual historical progression.
2. Based on those few lines, I suspect you are American, or at least live in the United Sates? I say this because it is very important not to project understandings of race and color from one country to another. This is something repeatedly done by African-Americans and other Americans. While it could be tempting because it does offer a practical theoretically framework, I think, at best, it should be avoided, as it fails to grasp the complexities of race and color of the “receiving” country. These are historically bounded social constructs and cannot simply be imported from one country to the other without loosing some of their meaning. Every country, at different times, has found varying ways of dealing with these questions. There may be some similarities, but we should be careful not to generalize and assume historical developments were the same everywhere.
Now, to answer the first question, as to where this color question “originates from,” I think we need to look back at the colonial era.