I grant that I am old, but I have never thought of this common expression as in any way a racial slur. If one is to interpret it that way, one must consider that a lot of white share croppers also picked cotton.
https://www.freep.com/story/news/local/farmington/2022/02/10/farmington-schools-remove-substitute-teacher-following-racist-language/6740617001/