Who Has
Power
Over AI?

Another way power is reflected is at the very heart of AI research and development.

The cost of training machine learning systems has decreased, and the availability of data is greater. But even as more of the world delves into AI, a major imbalance is reflected in the landscape of AI research papers.

In thousands of papers, it is the same datasets from just a few countries that are used most often to evaluate the performance of machine learning models everywhere.

This doesn’t mean that datasets or machine learning models aren’t being developed in the rest of the world. They are!

But the discourse about how AI should be used — and who should benefit from it — is currently heavily weighted toward people and institutions who already wield tremendous power over the internet (and the world).

In fact, more than half of the datasets used for AI performance benchmarking across more than 26,000 research papers were from just 12 elite institutions and tech companies in the United States, Germany, and Hong Kong (China).

A large and frequently reused dataset does not guarantee better machine learning than a smaller one designed for a specific purpose.

On the contrary, many of the most popular datasets are made up of content scraped from the internet, which overwhelmingly reflects words and images that skew English, American, white, and for the male gaze.

For instance, the machine learning models most credited for advancing the field of automated language generation, like GPT-3, frequently reproduce racist and sexist stereotypes, in large part due to their training data.

Why not use other datasets? Machine learning models and datasets reflect both the biases of their makers and power dynamics that are deeply rooted in societies and online, but this is not widely acknowledged. More datasets should be created specifically to diversify machine learning methods for equity.

Each of these companies make money in different ways, but AI is core to the business operations of all of them.

Let’s consider Amazon, the highest earning tech company in 2021. AI is key to every major revenue category reported by Amazon last year.

Big tech companies play an outsized role in shaping our experience of the internet, and life itself. What we see, what we buy, even what we believe is nudged along by them daily.

Yet, according to Ranking Digital Rights, there is little to no transparency into how companies test and deploy algorithmic systems and use our personal data, including when it comes to curation, recommendation, and ranking.

Recommender systems determine what people see in social media feeds and what content is rewarded with ad revenue. False and harmful content that drives up engagement is frequently recommended by platforms, even when it may later be found to violate platform rules.

In a crowdsourced study of YouTube by Mozilla, 71% of the videos people said they “regretted” watching were algorithmically recommended to them.

Because there is so little transparency into how systems work, researchers often need to recruit users for data donations to help study platforms and seek answers.

The boom in AI is accelerated by astronomical levels of data collection, by big tech and others. Almost everything we do is tracked and analyzed.

It’s hard to fathom just how much data is collected, kept, and sold.

Why does this matter?

Because AI offers companies with intimate knowledge about us new ways to predict what we’ll do — and new ways to influence how we behave.

It can be as simple as offering two people a different price for the exact same service without telling you.

Digital ads and social media networks are also weaponized to spread disinformation. In the absence of greater transparency and collaboration with researchers, a global industry of companies and organizations engaged in covert messaging is thriving.

Automated systems are frequently trained on data that is manually annotated by humans. For example, it can take hundreds of hours of labeling to prepare just one hour of video for the computer vision of a self-driving car.

This labor that is invisible to end-users is often exported to countries with low wages or completed via global crowd work platforms like Amazon Mechanical Turk.

Often, when the algorithmic management of labor is central to a business, fairness takes a backseat to maximizing productivity. This is true of hundreds of “gig work” platforms around the world. An estimated 40% of gig workers earn below the minimum hourly wage in their own countries.

This imbalance is amplified by the lack of data privacy regulations in many countries. And even in countries that do have them, AI is increasingly used by authorities to expand surveillance and control with facial recognition, license plate readers, and more.

There are an estimated upto 1 billion surveillance cameras worldwide. This means one camera for every eight people on the planet.

The appetite for applying AI to policing and carceral systems worldwide is huge. For instance, via predictive policing and pre-trial risk assessments.

However, AI is sending people to jail and getting it wrong.

In real life, over and over, the harms of AI disproportionately affect people who are not advantaged by global systems of power.

News headlines about algorithmic biases are glaring signals of how technology can be used to oppress rather than uplift people.

By some measures, ethical considerations in the research field are on the rise, but more interdisciplinary understanding of risk and harms is still needed.

The vast majority of leading research papers are centered on technical performance, not social needs or risks. And few of the most cited papers discuss ethical principles or user rights.

The barriers and costs for building AI are lower thanks in part to a new generation of open source tools and independent and grassroots AI developer communities worldwide.

But many of the same harms of big tech and big data AI development will be repeated if more trustworthy research, data, and development practices are not adopted.

Communities guided by values of fairness and human rights are challenging us to rethink not just how AI is built but for whom. Such questions are not being asked or answered by big tech.

Regulation can help set guardrails for innovation that diminish harm and enforce data privacy, user rights, and accountability. Many laws already apply to AI, but policies that are specific to AI (or sometimes bans) are also surfacing in different regions, countries and cities. In the public sector, there are many novel approaches to AI accountability.