China is considering a new "social credit" system, designed to rate everyone's trustworthiness. Many fear that it will become a tool of social control -- but in reality it has a lot in common with the algorithms and systems that score and classify us all every day.
Human judgment is being replaced by automatic algorithms, and that brings with it both enormous benefits and risks. The technology is enabling a new form of social control, sometimes deliberately and sometimes as a side effect. And as the Internet of Things ushers in an era of more sensors and more data -- and more algorithms -- we need to ensure that we reap the benefits while avoiding the harms.
Right now, the Chinese government is watching how companies use "social credit" scores in state-approved pilot projects. The most prominent one is Sesame Credit, and it's much more than a financial scoring system.
Citizens are judged not only by conventional financial criteria, but by their actions and associations. Rumors abound about how this system works...
...
This is what social control looks like in the Internet age. The Cold-War-era methods of undercover agents, informants living in your neighborhood, and agents provocateur is too labor-intensive and inefficient. These automatic algorithms make possible a wholly new way to enforce conformity. And by accepting algorithmic classification into our lives, we're paving the way for the same sort of thing China plans to put into place.
It doesn't have to be this way. We can get the benefits of automatic algorithmic systems while avoiding the dangers. It's not even hard.
The first step is to make these algorithms public. Companies and governments both balk at this, fearing that people will deliberately try to game them, but the alternative is much worse.
The second step is for these systems to be subject to oversight and accountability. It's already illegal for these algorithms to have discriminatory outcomes, even if they're not deliberately designed in. This concept needs to be expanded. We as a society need to understand what we expect out of the algorithms that automatically judge us and ensure that those expectations are met.
