What if your future—a job offer, a medical diagnosis, even your freedom—depended on an algorithm that doesn’t even know you exist?

Imagine being arrested for a crime you didn’t commit because a facial recognition algorithm got it wrong. Or being rejected for your dream job because an AI system decided your resume wasn’t “the right fit.” It might sound far-fetched, but these scenarios are happening today.

AI systems are making decisions that impact our lives in ways most of us don’t even realize. While these systems promise efficiency and fairness, they often amplify the biases present in their training data. The consequences aren’t just technical—they’re personal, social, and deeply ethical.

This post explores how AI systems inherit and magnify bias, the real-world impacts of these failures, and what it will take to build systems that serve everyone fairly. Along the way, we’ll draw on insights from Amy Ko, a researcher specializing in software ethics, inclusivity, and design. Ko’s work, such as Cooperative Software Development, provides a framework for tackling the ethical challenges posed by modern technology.

 

Why AI Gets It Wrong: Data That Mirrors Real-World Biases

AI systems aren’t inherently biased—they’re trained on data that reflects the real world, which is often far from fair. For example, in 2020, Robert Williams, a Black man in Detroit, was wrongfully arrested because a facial recognition system flagged him as a suspect. Despite having no connection to the crime, Williams was detained for hours. His story, covered by The New York Times, highlights a major flaw in AI: it’s far less accurate when identifying darker-skinned individuals, a trend confirmed by MIT’s Gender Shades project.

Ko’s work on requirements gathering emphasizes how these problems often start early in the design process. When development teams fail to include diverse perspectives, their systems are more likely to reflect and reinforce existing inequities. These errors aren’t just technical—they erode trust in the institutions deploying these technologies, particularly for marginalized communities.

 

Bias in Hiring: When AI Shuts People Out

Hiring decisions should be fair and unbiased, but AI-powered hiring tools can have the opposite effect. Take Amazon’s AI recruiting tool, which was abandoned after it was found to penalize resumes with terms like “women’s chess club”. As Reuters reported, the system had been trained on resumes from a male-dominated industry, leading it to favor male candidates over equally qualified women.

This isn’t just an Amazon problem—it’s a widespread issue. Many companies rely on similar systems without fully understanding how biases in training data can lead to discriminatory outcomes. Ko’s focus on sustainable software design highlights the long-term consequences of these decisions. In hiring, biased AI doesn’t just harm individuals—it shapes entire workplace cultures, reinforcing homogeneity and stifling diversity.

When Healthcare AI Misses the Mark

AI’s potential in healthcare is huge, but it’s also fraught with risk. A 2019 study published in Science found that a widely used healthcare algorithm systematically underestimated the needs of Black patients. By prioritizing healthcare costs over patient outcomes, the system effectively deprioritized care for those with complex or costly conditions, such as sickle cell anemia, which disproportionately affects Black communities.

This isn’t just about biased data—it’s about what metrics these systems are designed to prioritize. Ko’s work on inclusivity in design underscores the importance of involving diverse stakeholders in the development process. Without their input, critical needs can be overlooked, perpetuating inequities in access to care.

 

Fixing AI: What Needs to Change

So, how do we fix this? The answers aren’t simple, but they’re clear. While there are promising solutions, they face significant challenges—both technical and systemic. Let’s explore these solutions and the barriers to implementing them effectively:

1.      Diversify Data and Design Teams
AI systems need training data that represents all communities fairly. Equally important is having diverse teams involved in designing these systems. When developers come from a wide range of backgrounds, they bring perspectives that help prevent blind spots and biases. Organizations like OpenAI have made transparency and inclusivity central to their mission, involving external researchers and publishing ethical guidelines to ensure fairness.

But the road to diversification isn’t without obstacles. Companies often struggle to recruit and retain talent from underrepresented groups due to systemic inequities in hiring and workplace cultures. Even when diverse teams are in place, they often lack the authority to drive meaningful change. The controversy surrounding Google’s firing of Timnit Gebru is a striking example. Gebru, a prominent AI ethics researcher, was let go after raising concerns about the risks of large language models and advocating for greater inclusion in the industry. Her dismissal exposed how corporate priorities can clash with ethical goals, silencing critical voices in favor of maintaining profit-driven narratives.

This tension reflects a broader issue: companies often prioritize short-term gains over the long-term value of building trust with stakeholders. Failing to invest in diversity and accountability may cut costs in the moment but risks alienating users and losing credibility in the long run.

2.      Bias Audits and Regulation
Independent audits and regulatory frameworks, like the EU’s Artificial Intelligence Act, are vital tools for identifying and mitigating bias in AI systems. The Act, for instance, categorizes AI applications by risk level, imposing stricter requirements on those used in high-stakes areas like healthcare and law enforcement.

Yet, these measures face resistance. Companies argue that compliance is costly and could slow innovation. The tech industry has also lobbied extensively against such regulations, as seen with  pushback on the EU AI Act, claiming it could make Europe less competitive globally. These concerns, while not entirely unfounded, often prioritize profit over the public good. Moreover, without global standards, inconsistent regulations create loopholes, allowing companies to sidestep accountability by operating in less regulated markets.

3.      Shift Industry Priorities
The tech industry’s "move fast and break things" culture has often prioritized speed and profitability over fairness and accountability. Shifting this mindset is essential. Organizations like OpenAI have taken steps toward embedding ethics into their development processes, establishing committees to oversee the societal impact of their innovations.

However, even companies with public-facing ethical initiatives face internal conflicts. Google’s involvement in military AI contracts led to widespread employee protests, with many staff members resigning in opposition. This incident underscores a recurring issue: when corporate interests clash with ethical considerations, companies often prioritize profits, even at the expense of public trust.

As Amy Ko’s research suggests, ethics shouldn’t just be an afterthought. It’s not just about avoiding harm—it’s about creating systems that people can trust and that truly serve the diverse communities they impact. Ethical design isn’t just good for society—it’s good business.

 

Why It’s So Hard to Adapt These Solutions

While the solutions may seem straightforward, implementing them is anything but. Companies face several systemic challenges:

• Financial Pressures: The cost of diversifying datasets, conducting audits, and building inclusive teams often clashes with the relentless focus on quarterly profits. Ethical development requires long-term investment, which isn’t always compatible with short-term shareholder demands.
• Internal Resistance: Within many organizations, ethics teams lack the authority to push back against business priorities. The Timnit Gebru case highlighted how critical voices are often stifled when their findings challenge the status quo.
• Regulatory Gaps: Without consistent global standards, companies can exploit regulatory loopholes, moving operations to regions with less oversight. This weakens the effectiveness of well-intentioned frameworks like the EU AI Act.
• Cultural Inertia: The industry’s culture of rapid innovation often overlooks the time and care needed to build systems that are truly fair and inclusive. This “speed-first” mentality has created a status quo that’s hard to change.

These barriers show that fixing AI isn’t just about better technology—it’s about addressing the underlying systems and values driving the industry.

 

Why This Matters

AI has the potential to transform society for the better, but only if we’re intentional about how it’s developed and deployed. Without meaningful change, these systems risk perpetuating and amplifying inequalities, undermining trust, and leaving the most vulnerable behind.

The critical question remains: Are we willing to prioritize fairness, accountability, and ethics over speed and profit? The answer will shape not only the future of AI but also the kind of society we want to live in.