Data Ethics: Making Data-Driven Decisions

Welcome to Complete Guide to Data Ethics from Courses Buddy!

As AI increasingly influences bank loans, job interviews, and health insurance approvals, organisations must confront a critical question: Do they have an ethical obligation to explain these decisions to customers?

The answer is a resounding yes. Transparency builds trust, ensures fairness, and helps companies comply with regulations like GDPR and AI ethics guidelines.

Designing AI Systems Without Bias

To create fair and unbiased AI, companies must:

✔ Audit Data Sources – Ensure training data doesn’t reflect historical biases.
✔ Implement Explainable AI (XAI) – Design models that clearly show how decisions are made.
✔ Regular Bias Testing – Continuously monitor and refine AI models to prevent gender, racial, or socioeconomic bias.
✔ Human Oversight – AI should assist, not replace, human judgment in high-stakes decisions.

Who Is Responsible?

These choices aren’t just made in boardrooms. They happen in everyday meetings with project managers, analysts, directors, and developers—the very people designing these systems. Ethical AI isn’t just a technical issue; it’s a business and societal responsibility.

How does your organisation ensure AI-driven decisions are fair and explainable?

Ethical Decision-Making 

In today’s world, machine learning and AI-driven systems influence hiring, lending, and customer ratings—often with little transparency. This raises critical ethical challenges for businesses.

Do Decisions Need to Be Traceable?

Customers may have a right to know how AI impacts them. Should they be told why they were denied a loan, job, or service? Ethical AI requires explainability—systems should provide clear, understandable reasons for decisions.

Balancing Objectivity & Fairness

AI models are trained on historical data, which can reflect biases related to income, race, or gender. Should organisations modify data to ensure fairness, or adhere strictly to objectivity?

The challenge is balancing:
✔ Business Interests – AI improves efficiency, but is it fair to all users?
✔ Transparency – Users should understand how decisions are made.
✔ Ethics – Does the system reflect organisational values?

The Path Forward: Ethical AI in Action

AI shouldn’t just make fast decisions—it should make fair ones. This means:

1. Explainable AI (XAI) for clear decision-making. 
2. Bias audits to ensure fairness.
3. Human oversight for critical choices.

How does your company handle AI-driven decisions? Let’s discuss!

One of the earliest ethical debates appears in Plato’s Republic. His older brother, Glaucon, recounts the story of Gyges, a poor shepherd who discovers a magic ring that grants him invisibility. With this newfound power, Gyges seduces the local queen, murders the king, and seizes control of the kingdom. Glaucon argues that any person, no matter how virtuous, would act similarly when given unchecked power. The ability to act without consequence, he claims, leads inevitably to corruption.

This theme has influenced literature for centuries, appearing in works such as H.G. Wells’ The Invisible Man and J.R.R. Tolkien’s The Lord of the Rings. These narratives explore how invisibility might free individuals from moral constraints, allowing them to act purely in self-interest.

Modern organisations have created their own version of the Ring of Gyges through the power of data collection. Companies can now invisibly monitor customer behaviour, using this insight to manipulate decision-making. This raises a crucial ethical question: should companies exploit this power for their own gain, or exercise restraint and prioritise long-term trust?

Some organisations amass as much data as possible, leveraging customer vulnerabilities to drive sales or even selling data to third parties. Others choose a more ethical approach, gathering only necessary data and using it responsibly. While this may lead to missed profit opportunities, it fosters trust and long-term customer loyalty. Ultimately, how a company wields its power determines its reputation and ethical standing. Just as the people of Gyges’ kingdom would have condemned his deceitful rise to power, customers today react negatively to unethical data practices. Ethical decisions made within a company—often in routine meetings—shape its culture and long-term success.

How to Approach Ethics?

Bringing people together in an organisation inevitably results in diverse moral perspectives. However, to function effectively, organisations must establish a shared ethical framework—an unwritten set of rules governing interactions and decision-making. These collectively agreed principles define morality within the company.

Ethics, or moral philosophy, is the study of these principles. It seeks to understand different moral beliefs and how they can lead to better behaviour. While ethics applies across many domains—business, religion, government—this discussion focuses on the ethical challenges surrounding data collection and usage.

With unprecedented access to personal information, companies must navigate complex moral dilemmas. What constitutes acceptable data use? How can organisations ensure ethical decision-making in data management? Some companies fail to establish ethical guidelines, prioritising short-term gains over long-term trust. However, ethical integrity provides a competitive advantage. Employees prefer to work for ethical companies, and customers are more likely to trust businesses that prioritise morality. Once lost, trust is nearly impossible to regain, making ethical decision-making crucial for long-term success.

Start with Ethical Objectivism

A major challenge in ethics is reaching consensus on what is morally right. Some argue that morality is entirely subjective—dependent on individual perspectives and situations. However, ethical relativism is impractical for organisations, which require clear moral guidelines to make decisions.

Ethical objectivism offers structured approaches to ethical dilemmas. Three key theories provide guidance:

1. Deontology (Immanuel Kant) – Kantian ethics emphasises duty. Moral laws are universal and objective, requiring individuals to act according to duty, regardless of consequences. While this approach ensures consistency, critics argue it can be overly rigid.
2. Utilitarianism (Jeremy Bentham & John Stuart Mill) – This consequentialist theory prioritises actions that maximise overall happiness. Unlike deontology, utilitarianism considers outcomes, aiming to achieve the greatest good for the greatest number.
3. Virtue Ethics (Aristotle) – This approach focuses on moral character. Instead of strict rules, virtue ethics asks what a virtuous person would do in a given situation. It blends elements of deontology and utilitarianism by valuing both moral character and positive consequences.

These theories provide valuable frameworks for ethical decision-making, particularly when handling data.

Think About Your Categorical Imperatives

Organisations frequently face ethical conflicts between what they want to do and what they ought to do. For example, a driver may want to run a red light but feels duty-bound to stop. Kant referred to these obligations as categorical imperatives—absolute moral rules that must be followed regardless of outcomes.

Kant outlined three key formulations of the categorical imperative:

1. Universalizability Principle – If an action would create problems if universally adopted, it is likely immoral. For instance, if everyone ignored traffic lights, chaos would ensue, making the act inherently unethical.
2. Rule of Humanity – People should never be treated merely as a means to an end. Unlike objects, humans have intrinsic value and must be respected accordingly.
3. Formula of Autonomy – Ethical actions should preserve individuals’ autonomy and decision-making ability. For example, dishonesty undermines a person’s ability to make informed choices.

Many ethical issues in data collection conflict with these principles, particularly transparency and fairness in handling customer information.

What Would a Virtuous Person Do?

Aristotle argued that happiness comes from living a life of virtue. His Virtue Ethics framework distinguishes between intellectual virtues (such as reason and knowledge) and moral virtues (such as honesty, justice, and empathy). A virtuous person consistently chooses the right path based on character, not rules or consequences.

Consider a scenario where an organisation allows users to upload and share short videos. A popular influencer posts a video promoting e-cigarettes, claiming they are safer than traditional cigarettes. The company must decide whether to remove the video.

1. A deontological perspective may argue that the company has no duty to regulate content, as e-cigarettes are legal and the influencer is an adult.
2. A utilitarian approach would consider broader consequences, arguing that removing the video benefits public health and promotes overall happiness.
3. A virtue ethics perspective would ask what a morally upright person would do. A virtuous individual would prioritise protecting young audiences from harmful influences, advocating for the video’s removal.

Each ethical framework offers a different perspective on the same dilemma. Recognising these perspectives helps organisations foster meaningful discussions and arrive at principled decisions.

As companies gain unprecedented access to data, they must carefully consider the ethical implications of their actions. Ethical decision-making is not just about avoiding scandals—it shapes corporate culture, influences employee satisfaction, and determines customer trust. By applying ethical theories such as deontology, utilitarianism, and virtue ethics, organisations can navigate moral dilemmas effectively and sustain long-term success.

Data ethics challenges come in many forms, varying across industries and organizations. No single course can cover all potential challenges, but my goal is to equip you with the tools to navigate them effectively. To help you remember key data ethics challenges, consider the mnemonic POTOMAC:

1. P – Privacy
2. O – Ownership
3. T – Algorithmic Traceability
4. O – Objectivity
5. M – Misuse
6. A – Accuracy
7. C – Consent

You are likely to encounter at least some of these challenges in any data-driven organization. In a previous data ethics course, we explored privacy and data ownership:

• Privacy: Concerns about how much data can be collected and shared externally.
• Data Ownership: Ethical considerations around who truly owns the data.

Well, we will focus on algorithmic traceability and data objectivity:

• Algorithmic Traceability: Modern data-processing technologies make decisions in milliseconds. If an algorithm denies a customer a loan, does your organization have an ethical responsibility to explain why? Should decisions be traceable, or can companies simply defer to the algorithm?
• Data Objectivity: Most collected data describes human activities, which are inherently subjective and biased. Should companies attempt to correct biases in data, and if so, how?

We will not cover them in depth now, but the final three elements of POTOMAC—misuse, accuracy, and consent—present additional ethical dilemmas:

1. Misuse: Should insurance companies access credit card data to assess an individual’s lifestyle?
2. Accuracy: Is it the company’s responsibility to verify data accuracy, or should it rely on user-provided information?
3. Consent: Customers may click “agree” when using a service, but how can organizations ensure true informed consent?

Each of these categories presents unique ethical considerations. A solid understanding of ethical theories will help you develop effective strategies for addressing your organization’s data ethics challenges.

Computer algorithms are sets of instructions designed to solve specific problems. Increasingly, we rely on these algorithms to make decisions for us. Your fitness watch alerts you when it is time to move, your navigation software directs you to your destination, and your computer reminds you of upcoming appointments. Throughout the day, you might receive hundreds of algorithmic suggestions, which quickly become second nature to follow.

The better these algorithms become at their tasks, the easier it is to trust them without question. Navigation software, for example, has improved to the point where most people follow its suggested routes without hesitation. Similarly, when a fitness tracker recommends additional activity, we often comply instinctively.

Algorithmic Influence in Organisations

This reliance on algorithms extends beyond personal habits—it also plays a crucial role in organisations. Algorithms are continuously analysing and predicting customer behaviour, guiding business decisions in ways that employees may follow without questioning. Over time, this automated decision-making process becomes so ingrained that tracing back the steps taken by the algorithm becomes increasingly difficult.

This challenge is known as algorithmic decision traceability and is one of the seven key data ethics issues organisations must address. These issues—often referred to as the POTOMAC principles—include:

1. Privacy
2. Data Ownership
3. Decision Traceability
4. Objectivity
5. Data Misuse
6. Accuracy
7. Consent

The Right to Explanation

One of the most pressing concerns regarding decision traceability is the customer’s right to an explanation. What obligation does an organisation have to explain algorithmic decisions to its customers?

Consider how companies like Facebook and Google categorise users into advertising groups, known as affinity groups. These groups allow businesses to target customers based on shared characteristics. If one person in a group likes a particular product, it is assumed that others in the group might like it too.

Your organisation might use a similar technique to classify customers. For instance, a customer may be labelled as risk-averse or financially irresponsible, leading to their categorisation as a high-risk loan candidate. However, the customer may have no knowledge of how they were placed in this group or how to change their classification.

The Ethical Challenge of Algorithmic Decisions

The ethical dilemma revolves around the level of transparency regarding how these algorithms make decisions. When organisations rely on algorithms to make important choices about customers, they must consider how much explanation is necessary.

For example, a customer might be denied a car loan but remain unaware of the specific algorithmic factors that contributed to this decision. Similarly, they might be charged a higher insurance premium without understanding why they were classified as high-risk. Without transparency, customers are left powerless, unable to challenge or alter the outcomes of these decisions.

Ensuring Ethical Algorithmic Practices

As algorithms become increasingly complex, retracing the steps behind a decision becomes more challenging. This raises the fundamental question: What right of explanation does your organisation owe its customers?

Businesses must develop ethical guidelines to ensure customers understand the key factors influencing algorithmic decisions. By prioritising decision traceability, organisations can foster trust, enhance accountability, and ensure fairness in algorithm-driven processes.

Algorithmic traceability presents two main challenges. The first is accessibility—who can have access to the decision-making process? The second is comprehensibility—if someone does get access to algorithmic decision-making, will they understand how the machine made the decision? In machine learning, this is often referred to as the black box problem.

The Challenge of Credit Scoring

Imagine that you work for a new credit reporting agency. This agency not only assesses bank accounts, loans, and credit cards but also considers social media usage and real-time spending. Your agency combines this data to generate a comprehensive credit score. Your clients—banks and credit card companies—rely on these scores to evaluate potential borrowers.

Many of your clients appreciate the depth of these credit reports, as they provide a real-time snapshot of an individual’s financial health. However, your company has recently come under scrutiny. The public is questioning how these scores are generated and demanding greater algorithmic transparency.

Ethical Considerations in Decision Traceability

You call a meeting to discuss how to explain the decision-making behind your scoring process. The first issue to consider is whether individuals have the right to access their own credit reports. This relates to the first challenge of decision traceability—should you grant access to the algorithm’s decision-making process?

One approach is to consider your duties towards your customers. However, in this scenario, your customers are banks and credit card companies, not the individuals being scored. In a sense, the people being scored are your product, raising an ethical dilemma—do you have an obligation to explain your decision-making to them?

From a Kantian ethics perspective, treating people merely as a means to an end is considered immoral. Since those being scored do not directly benefit, one could argue that they are being used unfairly. Conversely, a utilitarian approach might justify the system by arguing that overall, it facilitates access to credit, benefiting more people in the long run.

The Complexity of Comprehensibility

Even if you decide to grant individuals access to their data, the second challenge remains—can they understand it? Your organisation analyses vast big data sets, and machine learning algorithms often identify patterns that may be difficult for humans to interpret. Additionally, your company may not want to disclose the full range of data sources and algorithmic processes involved in generating a score.

One possible solution is to develop a simplified explanation of how the comprehensive credit score is calculated. This approach aligns with virtue ethics, demonstrating that your organisation is acting transparently and ethically. This could be especially valuable in addressing concerns from individuals who are dissatisfied with their credit scores.

One of the key challenges in decision traceability is determining who has access to decision-making processes. This issue is known as the problem of decision accessibility. In the modern data-driven world, individuals often become the product, as companies collect data about groups of people and sell it to other organisations. One of the biggest hurdles in data accessibility is distinguishing between the product and the customer.

The Role of Data in Decision-Making

Imagine a human resources company that develops software to identify top talent. This software scans the top 5% of employees working for Fortune 500 companies and analyses their biographical data to detect common patterns. When a new job applicant matches these patterns, the software assigns them a higher score. In essence, it predicts future top performers before they even start their job.

The software becomes so effective that many Fortune 500 companies rely on it to make hiring decisions. As a result, an applicant’s candidate score can significantly impact their career prospects.

Ethical Dilemmas in Data Accessibility

One day, the company receives a letter from Morehouse College, stating that many of its graduates have received low scores from the software. The manager requests an investigation to determine if a bug is affecting the system.

Upon analysis, the data science team discovers that Morehouse graduates are indeed scoring lower. However, this is not due to a bug; rather, it is because fewer Morehouse graduates are represented among the top 5% of Fortune 500 employees. The software is simply recognising patterns from existing data.

This raises an ethical dilemma: does Morehouse College have a right to an explanation? Should they have access to the data and decision-making process?

Ethical Perspectives on Data Access

1. Deontological Ethics: From this perspective, the company has a duty to provide transparency. Withholding this information limits Morehouse College’s ability to understand and improve its graduates’ placement rates.
2. Utilitarian Ethics: The manager argues that the real issue is the lack of diversity in Fortune 500 companies. Revealing this information may put the company in a difficult position, as Morehouse College could use the data to challenge their clients’ hiring practices. From a utilitarian standpoint, withholding the information might serve the greater good by maintaining business relationships and avoiding potential backlash.
3. Virtue Ethics: A virtuous organisation should strive to promote equality and diversity. By revealing the truth, the company could help address systemic biases, even if it risks harming its own customers.

The Responsibility of Organisations

As data-driven decision-making becomes increasingly prevalent, organisations must navigate complex ethical challenges. They must balance business interests with social responsibility, ensuring that their algorithms do not inadvertently reinforce biases or restrict opportunities for underrepresented groups.

Ultimately, companies must decide how much transparency they owe to individuals affected by their decisions and whether they should prioritise ethical integrity over corporate convenience.

In algorithmic decision traceability, two major challenges arise: accessibility and comprehensibility. Organisations must determine whether they have a moral obligation to provide individuals with access to data and decision-making processes. Additionally, they must assess their responsibility in ensuring that these decisions are understandable.

The Challenge of Comprehensibility

A significant issue in comprehending algorithmic decisions is the widespread use of machine learning. Unlike traditional decision-making processes, where explicit human reasoning determines outcomes, machine learning models identify patterns in vast datasets and make predictions based on those patterns.

For instance, in the past, a health insurance company might have determined higher health risks based on explicit factors such as smoking habits or a family history of disease. These decisions were made by humans who could justify their reasoning and develop measurable criteria, such as a health score. However, modern machine learning algorithms do not require explicit programming. Instead, they construct predictive models that assess customers based on vast amounts of data, identifying correlations that may be imperceptible to humans.

This shift has led to the problem of the black box—a scenario where the algorithm makes decisions in a way that is opaque to human observers. These machines detect complex patterns in data that even the programmers and managers overseeing them may not fully understand.

The Complexity of Machine Learning Decisions

Consider a machine learning model that determines health insurance premiums. The algorithm might find a correlation suggesting that an individual has a higher risk of respiratory infections. However, it does not identify or explain the cause of this correlation; it merely adjusts the insurance premium accordingly. This creates a situation where even human agents within the company cannot fully explain the basis of a particular decision.

As machine learning becomes more prevalent, untangling these decision-making processes will become increasingly difficult. Currently, customers have little to no visibility into how such algorithms arrive at conclusions that impact their lives. They cannot trace the steps leading to a decision, making accountability a major ethical concern.

The Ethical Responsibility of Organisations

With the growing reliance on machine learning, organisations must address the ethical implications of decision traceability. One key challenge is that these algorithms are designed to uncover patterns that humans would not naturally detect. Consequently, explaining these patterns to people who lack technical expertise can be akin to translating a complex poem into a language they do not understand.

Moreover, organisational decision-making may become so intricate that even internal teams struggle to comprehend it fully. In such cases, providing external stakeholders with meaningful explanations becomes an even greater challenge. As algorithmic decision-making grows in complexity, organisations must consider their responsibility in ensuring transparency and accessibility while maintaining the effectiveness of their systems.

So, addressing the black box problem requires balancing technological advancement with ethical responsibility. Organisations must explore ways to improve decision traceability and ensure that affected individuals have both access to and an understanding of how decisions are made. This will not only foster trust but also help mitigate potential biases and unintended consequences in algorithmic decision-making.

One of the biggest challenges with decision traceability is dealing with the mystery of the machine learning black box. Machine learning algorithms are designed to see patterns that are nearly impossible for humans to understand. These algorithms are used to make decisions all the time, and not knowing exactly how these decisions are made creates this black box problem. These black box decisions make it very difficult to enforce a right of explanation. If you can’t understand how the machine made the decision, then there’s no way to explain that decision to someone else.

Over the last few years, there’s been a big push to start adopting explainable artificial intelligence (XAI). These systems put the human in the loop when making decisions with machine learning algorithms. They design algorithms that can be explained by human experts. XAI emphasises fairness, accountability, and transparency (FAT systems). These systems should be fair in how they make decisions, accountable to a human being, and transparent for customers.

XAI systems are becoming increasingly popular, with organisations recognising an ethical obligation for decision traceability. In 2017, a company called the SAS Institute developed software that helped school districts evaluate their teaching staff. The software used machine learning algorithms to assign teachers a comprehensive teaching score. The Houston School District used the software to evaluate 8,000 teachers, assigning each a numerical score without explaining how it was determined. Many teachers were promoted or dismissed based on this score.

Teachers successfully sued the district, arguing that the lack of decision traceability violated their rights. Because the algorithm was not explainable, they couldn’t understand how they were evaluated or how to improve. There was no way to ensure fairness in the evaluation process, leading the court to rule against the school district.

Now, consider a bank using machine learning algorithms to review mortgage applications. The system analyses various data sources to assess a person’s ability to repay a loan. However, all decisions occur within a black box—applicants receive a simple approval or rejection with no explanation.

As a result, many customers feel frustrated because they don’t know why they were rejected. Customer service representatives lack information to help them improve future applications. An XAI system would provide traceability for these decisions. Instead of simply rejecting an application, the system could state:

1. “If your income was $5,000 higher, you would have been approved.”
2. “If you had held your current job for more than two years, you would have been approved.”

With an XAI system, customer service representatives could trace algorithmic decisions and provide clear, actionable explanations. This level of transparency fosters trust between organisations and their customers while ensuring that AI-driven decisions remain ethical, fair, and accountable.

In the 1960s, an English philosopher named Phillipa Foot came up with an ethical thought experiment commonly known as the trolley problem. Imagine you see a runaway trolley. There’s a fork in the tracks, and in one direction, five people are sleeping on the track, unaware of the oncoming train. On the other track, there is one lone workman with his back turned to the trolley. Just a few feet in front of you, there’s a lever that controls which track the trolley runs on. You can pull the lever and decide the correct moral answer.

If you don’t do anything, the trolley will continue toward the five people. You can save them by pulling the lever, but then you’d kill the lone workman. The ethical dilemma arises: what is the correct moral action? Different ethical theories provide different answers.

1. Deontology (Kant’s approach): Killing is always morally wrong, regardless of consequences. So, pulling the lever would be unethical.
2. Utilitarianism: The right action is the one that maximises overall happiness. Here, pulling the lever saves more lives, so it’s the moral choice.
3. Virtue Ethics: A virtuous person would act with courage and compassion. This might lead them to pull the lever, accepting the guilt of one death to save five lives.

This theoretical problem has now become a real-world data ethics challenge with self-driving cars. Imagine you’re in a self-driving car, and a truck suddenly pulls out in front of you. The software must make a split-second decision:

1. Veer into a building wall, potentially injuring you.
2. Swerve into a bike lane, potentially harming a cyclist.

When Google was asked about this issue, they stated that their self-driving software was programmed to run into the smaller object—bad news for cyclists. This raises important ethical questions: Should drivers be aware of these programmed choices? Should they be allowed to customise them? How traceable and comprehensible should these decisions be? As self-driving cars become more prevalent, these dilemmas will only grow more pressing.

Self-driving cars present some complex ethical challenges. Imagine you’re in a car when suddenly a truck swerves in front of you. The car must either swerve to the right into a wall or left into a cyclist. How does it choose?

We know Google has designed its self-driving cars to veer into the smaller object, but other car companies have not been so forthcoming about how they make their driving decisions. Now, imagine you’re a project manager working for a company testing its own version of a self-driving car. In a meeting, you must decide how your cars should handle some variation of this problem.

A Deontological Approach

If you consider the perspective of Immanuel Kant and deontology, then the car must be programmed to avoid killing others at all costs. Deontology focuses on ethical rules, and one fundamental rule is never to take a life. This principle must be applied universally, regardless of the consequences. Kant’s categorical imperative states that ethical rules must be universal, meaning you cannot justify a rule that permits killing—even if it is to save more lives.

Google’s Approach: Prioritising the Driver

Another option is to mimic Google’s approach and prioritise protecting the driver. The car would always attempt to crash into the smallest object, assuming it would cause the least damage. However, this raises ethical concerns—should the life of a pedestrian or cyclist be sacrificed for the safety of the driver?

The Utilitarian Perspective

A utilitarian approach would seek to maximise overall happiness and minimise harm. This could mean that, in some scenarios, the self-driving car may decide to sacrifice itself to save more lives. For instance, it might drive off a bridge rather than hit a group of pedestrians. However, this would be bad news for the driver, who might not have consented to such a decision.

The Challenge of Virtue Ethics

Virtue ethics presents the most difficult challenge to programme into a self-driving car. It requires considering what a virtuous person would do in each scenario. A truly virtuous individual might prioritise protecting others at all costs, meaning the car would always sacrifice itself to save others. However, not all drivers may share this moral outlook, which creates further ethical dilemmas.

The Issue of Decision Traceability

A key challenge across all these ethical frameworks is decision traceability. Do customers have the right to know how their self-driving car is programmed to react in a crash? Is there an ethical obligation to inform them if their car might sacrifice itself to save a greater number of people?

One possible solution is allowing customers to customise their car’s ethical decision-making. This way, drivers could select an approach aligned with their moral beliefs. However, most car manufacturers have kept their algorithms a closely guarded secret. Even if they were made accessible, many customers might struggle to comprehend the complexities of these ethical choices.

Self-driving cars bring philosophy into real-world decision-making, and the question remains: should ethics be programmed, or should humans always have the final say?

Let’s start with an old English riddle and see if you can guess the right answer.

A father and a son are in a car together when they get into a terrible accident. Two different ambulances take them to two separate hospitals. When the son arrives at the hospital, the nurses determine that he needs urgent surgery. They contact the head of surgery, who arrives to see the young man on the operating table and exclaims, “I can’t operate on him. This boy is my son.”

How is this possible?

Some people guess that the son has two fathers or that the head of surgery is a stepfather. However, the most straightforward answer is that the head of surgery is the boy’s mother. Yet, most people fail to arrive at this conclusion.

In a study conducted at Boston University, only 14% of students correctly identified the doctor as the boy’s mother, and this percentage remained consistent regardless of whether the students were male or female. The reason for this difficulty is implicit bias—many people subconsciously associate the role of a surgeon with a middle-aged white man.

Bias in Data and Machine Learning

This unconscious bias is not just limited to riddles—it appears in organisational data all the time, and machine learning algorithms can amplify these biases. The challenge is that remaining objective means presenting raw data as it is, without manipulation based on personal assumptions. However, ensuring both objectivity and fairness can sometimes be a complex ethical dilemma.

Data objectivity is one of the seven key ethical concerns in organisations, which can be remembered using the POTOMAC framework:

1. Privacy
2. Ownership (who controls the data)
3. Traceability (decision accountability)
4. Objectivity
5. Misuse of data
6. Accuracy
7. Consent

The Ethical Dilemma in Hospital Data

Now, imagine you work for a hospital that uses a patient evaluation system. Patients rate their experiences using a simple survey, and the hospital bases promotions and salary increases on these ratings.

One of the survey questions asks: “Did this doctor meet your expectations?”

As a project manager, you begin to notice a trend—middle-aged men consistently receive higher ratings. As a result, the hospital is more likely to promote and give pay raises to male specialists. Over time, this leads to a significant salary disparity between male and female doctors.

However, when questioned about this inequality, the hospital administration simply states that they are following the data. They argue that the ratings objectively indicate that male doctors provide better care, so they naturally deserve promotions.

Should Bias Be Corrected?

As a project manager, you must confront a difficult ethical question: Do you have an obligation to correct potential bias in the data?

From the earlier riddle, we know that people have preconceived ideas about what a doctor should look like.

1. On one hand, the data appears to be objective—it’s a simple five-star rating system. But how could you remove the bias? Would it be fair to subtract half a star from men simply because they are men? Should women receive an extra point just to balance out historical bias?
2. On the other hand, acting blindly on this data would be unethical. Why should male doctors earn more than female doctors when they are performing the same job, simply because of patient perception?

These are some of the toughest ethical challenges when dealing with data objectivity. The question remains: should organisations act on data without interpretation, or is there a responsibility to account for hidden biases?

As you can imagine, a company like Amazon receives hundreds of applications for each job opening. In 2014, to streamline the hiring process, Amazon developed a human resources system that would rate candidates on a five-star scale.

The idea was simple and elegant:

1. Managers rated current employees using a five-star system.
2. The system analysed patterns in the applications of five-star employees.
3. A machine learning algorithm then looked for similar patterns in new job applications.

By 2015, Amazon had started using this system to filter new job applicants.

The Unintended Bias

The problem? Around 60% of Amazon employees were men. Since the majority of five-star employees were male, the algorithm learned that male applicants were preferable.

As a result, the system strongly favoured men and began penalising applicants who:

1. Listed affiliations like “Women’s Chess Club”
2. Attended women’s colleges

This reinforced gender bias, leading to the majority of new five-star applicants being men. The bias became so severe that Amazon eventually shut down the system.

Analysing the Ethics of Bias

Let’s imagine Amazon hired you as an ethics consultant. How should they think about the ethical implications?

1. The Deontological Approach (Kantian Ethics)

Immanuel Kant’s categorical imperative states that one must always tell the truth, regardless of consequences.

1. The algorithm wasn’t lying—it simply learned the existing bias in Amazon’s hiring process and applied it consistently.
2. Some might argue that removing this bias would be manipulating the truth, limiting Amazon’s ability to understand its own hiring patterns.
3. Additionally, penalising men to make the system “fairer” might violate autonomy, another key principle in Kantian ethics.

So, from a deontological perspective, fixing the bias could be seen as distorting objective data.

2. The Utilitarian Approach

A utilitarian focuses on the greatest good for the greatest number.

1. Studies show that diverse workplaces are less prone to groupthink and have fewer blind spots.
2. A utilitarian might argue that Amazon should fix the algorithm to increase diversity, ultimately benefiting the company and society.

So, in this approach, the best decision would be to modify the algorithm to ensure fairer hiring practices.

3. Virtue Ethics

Virtue ethics considers the moral character and motivations of a virtuous person.

1. A virtuous person would try to eliminate inequality simply because it’s the right thing to do.
2. This means Amazon should immediately stop using the algorithm until it ensures fairness for all applicants.

Amazon ultimately chose to discard the system rather than attempt to fix it. However, the ethical debate remains:

1. Should companies prioritise objective data, even if it reflects systemic bias?
2. Do businesses have a moral responsibility to adjust their systems for fairer outcomes?

These are crucial questions for any organisation using AI-driven decision-making.

Data objectivity and bias do not always appear in obvious ways. Sometimes, bias emerges where you least expect it.

In 2011, the city of Boston introduced an innovative solution to fix potholes. They developed an app called Street Bump, designed to detect potholes automatically.

How the Street Bump App Worked

1. Users installed the Street Bump app on their smartphones.
2. The phone’s accelerometer detected when the car hit a pothole.
3. The app sent GPS coordinates of the pothole to city officials.
4. Over time, this created a detailed map of all the potholes in the city.

At first, the system seemed effective. However, a city employee noticed a pattern—most reported potholes were in wealthier neighbourhoods.

The Hidden Bias in the Data

The bias was not intentional, but it was built into the system:

1. In 2011, smartphones were not as common as they are today.
2. Wealthier individuals were more likely to own smartphones.
3. As a result, potholes in wealthier areas were reported more frequently.
4. The city prioritised repairs based on this data, leading to an imbalance—potholes in lower-income areas were neglected.

The Ethical Dilemma

Boston city officials faced a difficult decision:

1. Remain objective and continue using raw data, even if it favoured wealthier areas.
2. Correct the bias by adjusting how the data was used.
3. Abandon the app and return to manual pothole reporting.

Some data scientists argued that their role was to report the data, not alter it. If wealthier citizens were the ones reporting potholes, should their data be ignored?

Boston’s Ethical Approach

Boston’s Office of New Urban Mechanics decided to address the issue through the lens of virtue ethics.

1. Virtue ethics focuses on fairness and equality.
2. It would not be virtuous to use a finite amount of resources to benefit only the wealthiest neighbourhoods.
3. Although the data itself was objective, the outcome was unfair.

To correct this, Boston partnered with academics to adjust the weighting of the data. This allowed them to prioritise repairs more equitably.

Alternative Ethical Perspectives

Boston chose virtue ethics, but other ethical frameworks might disagree:

1. Deontological Ethics (Kantian Approach)

1. Immanuel Kant might argue that altering the data is dishonest.
2. The raw data tells the truth—changing it manipulates reality.
3. From this view, all potholes should be treated equally, regardless of where they are reported.

2. Utilitarianism 

1. A utilitarian might argue that prioritising wealthier areas could be justifiable.
2. Perhaps these areas have more traffic, meaning repairs benefit more people overall.
3. If fixing potholes in wealthier areas results in fewer accidents, wouldn’t that be the best outcome?

Which Ethical Approach Is Most Logical?

Boston chose to adjust the data to ensure a fairer distribution of repairs. But was this the right decision?

1. Was it fair to give lower-income areas priority based on ethical concerns?
2. Or should all potholes be treated equally, regardless of where they are reported?

These are the difficult questions that arise when dealing with bias in data-driven decision-making.

In the United States, sentencing decisions often rely on risk assessment algorithms. One such system, COMPAS, analyses past court cases and uses a short questionnaire to predict whether a person is likely to reoffend.

How COMPAS Works

1. The system assigns a risk score to the defendant.
2. If the software determines that the individual has a high risk of reoffending, they receive a higher score.
3. Judges often use this score to justify longer sentences.

However, there is a fundamental problem—COMPAS is owned by a private company, meaning:

1. No transparency: The algorithm’s decision-making process is not disclosed.
2. No accountability: The company protects its formula as a trade secret.
3. No clear oversight: The legal system trusts the algorithm without fully understanding it.

The Ethical Challenge of Data Bias

A major concern with COMPAS is its bias against African Americans. Studies have shown that:

African Americans are twice as likely to be labelled as repeat offenders. Incorrect risk scores lead to longer sentences, reinforcing racial inequalities. The software is trained on biased data, reflecting existing prejudices in the justice system.

The Root of the Problem: Biased Data

The raw data used by COMPAS is not neutral:

Studies show that juries are more likely to convict African Americans for similar crimes. If the algorithm is purely objective, it simply mirrors this injustice. By reporting the data “as is”, the company is perpetuating systemic bias.

This raises a serious ethical dilemma—should the company alter the data to correct the bias, or should it remain purely objective?

Ethical Perspectives on Data Objectivity

1. The Content Perspective: Objectivity Above All

The algorithm reflects reality—if juries are biased, the data will show that. From this perspective, objectivity is not the problem; it’s the justice system itself. However, this approach reinforces existing discrimination, making the company complicit in systemic bias.

2. Utilitarianism: The Greatest Good for Society

A utilitarian approach argues that data should be adjusted to reduce harm. If perpetuating bias harms society, then the company has an ethical duty to correct it. However, changing the data could raise concerns about manipulating legal outcomes. States may reject the software if they believe it is overriding jury verdicts.

3. Virtue Ethics: Doing the Right Thing

A virtuous person would not allow someone to serve a longer sentence simply because of their race. This perspective suggests that the company must intervene to ensure fairness. However, this means sacrificing objectivity in favour of moral responsibility.

Can Data Ever Be Truly Objective?

Most organisations strive for objective data, but cases like COMPAS reveal a deeper issue—can true objectivity exist when the data itself is biased?

As a product manager at this company, how would you approach this dilemma?

1. Would you tweak the data to make it more fair?
2. Or would you leave the data unchanged, ensuring it remains objective but unjust?

This is the challenge that many data-driven systems face—sometimes, the greatest ethical dilemmas arise not from bias itself, but from being too objective.

When discussing data objectivity and bias, it’s easy to assume that these issues only affect a few high-tech companies. In reality, bias in data is a widespread and significant problem—one that often goes unnoticed.

For example:

1. 55% of organisations report using algorithms to process job applications.
2. Many companies rely on rating systems to assess employee performance and customer satisfaction.
3. If your organisation is using machine learning, your data may already be biased, even if you’re unaware of it.

Where Does Bias Come From?

Bias can be introduced in unexpected ways:

1. If your app requires the latest smartphone, it excludes people with lower incomes.
2. If your algorithm analyses human behaviour, it may reflect societal prejudices.
3. If your system prioritises certain patterns, it may reinforce existing inequalities.

Once you start looking, you’ll see the struggle between objectivity and fairness in almost every large dataset. But even if you identify bias, the next challenge is defining fairness.

How Do We Define Fairness?

What would unbiased data actually look like? Different organisations have tackled this question in different ways:

Case 1: Amazon’s HR System

1. Amazon’s hiring algorithm favoured male applicants, reflecting historical biases.
2. The company scrapped the system, but did that actually make hiring fairer?
3. Should women be given an advantage to correct past inequalities?

Case 2: Boston’s Pothole App

1. The app collected objective data, but wealthier areas received more repairs.
2. The city adjusted the algorithm to prioritise lower-income neighbourhoods.
3. But was this a fair solution, or did it unfairly tip the scales?

Fairness vs. Objectivity

Society doesn’t define fairness the same way a computer algorithm does. People don’t like the idea of race, gender, or income being treated as numerical values that can be added or subtracted. However, if you want to move beyond pure objectivity, these discussions are inevitable.

Virtue ethics suggests we should correct the data to promote fairness and equality.
Utilitarianism argues for maximising the greatest good, even if it means adjusting the data.

Kantian ethics insists that objectivity and truth must take precedence, regardless of the consequences.

Ethical Decision-Making in Data Science

Thank you for engaging with this course. I hope it has helped you understand the ethical complexities of data-driven decision-making.

Key takeaways:

1. Ensure your organisation’s decisions are transparent and explainable. 
2. Be aware that human bias exists in all data.
3. Decide whether your organisation has an ethical duty to remove bias.
4. Consider whether your brand reputation depends on being perceived as fair.
5. Think about whether utilitarian, virtue-based, or Kantian ethics best suit your approach.

Next time you’re making a data-driven decision, take a moment to ask: What does fairness really mean?

“Data shapes the future—but ethics shape data. Choose fairness, question bias, and make decisions that don’t just inform, but inspire.”