Any community, no matter how small, would be wise to distance itself from harmful personalities who think it's fun to be racist, sexist, homo- or transphobic. Even if you temporarily decrease the community by banning individuals who use racist slurs publically or harass others, it will only benefit the community in the long run by making a place more accepting for everyone except idiots.
Very few people actually think racist or sexist jokes are funny, some just use them as attention grabs. Don't enable that behavior.
Leagues, websites, forums, etc. banning people for publicly spouting insulting slurs directed towards any minority would be a good way to start. In time people will learn that you're not welcome here if you use the n-word or harass others or whatever. Maybe start with short term bans going up for repeated offenses, but act on it. And tbh it's not really wasted talent even if they are prem players, others will take their place, hopefully someone who doesn't think it's fun to be mean to others. Otherwise you're only driving the good examples out of the community because they are fighting against windmills.
And I don't mean trash talking, you can do that without using racial slurs. And it's not really like this would be difficult to enforce, looking at a case-by-case basis it's very easy to draw the line between trash talk and being a racist.