Understanding the "Why" Behind AI Decisions
DATA4800: Artificial Intelligence and Machine Learning
Kaplan Business School
DATA4800: Artificial Intelligence and Machine Learning
Kaplan Business School
By the end of this workshop, you will be able to:
Sarah applies for a $50,000 business loan at Global Bank. She has:
The bank's AI system rejects her application.
When Sarah asks why, the loan officer says: "Our AI model determined you're high risk. That's all I can tell you."
What's wrong with this picture?
Sarah needs to know what she can improve. Is it her debt-to-income ratio? Her business age? Without explanation, she can't take action to improve her chances next time.
The bank manager sees profitable applicants being rejected. Without understanding the model's logic, she can't verify if it's making sound business decisions or if it needs adjustment.
Financial regulators must ensure the AI isn't discriminating based on protected characteristics. A "black box" makes compliance verification impossible.
Some AI decisions have serious consequences that demand transparency:
Example: AI recommends surgery vs. medication
Why explanation matters: Doctors need to understand the reasoning to make informed recommendations to patients
Example: AI screens job applications
Why explanation matters: Must ensure decisions aren't based on biased factors like age, gender, or ethnicity
Example: AI approves/denies loans
Why explanation matters: Legal requirement in many jurisdictions; customers have right to know
Example: AI predicts recidivism risk
Why explanation matters: Decisions affect people's freedom and must be justifiable in court
Understanding HOW the model works overall
Analogy: Seeing the Recipe
You can see all the ingredients and steps needed to bake a cake. You understand the entire process from start to finish.
Example: A simple scoring formula where you can see exactly how credit score, income, and debt combine to produce a risk score.
Understanding WHY a specific decision was made
Analogy: Explaining the Outcome
You might not know the full recipe, but you can explain why THIS cake came out dense: "We used whole wheat flour instead of all-purpose flour."
Example: For Sarah's rejected loan, explaining "Your debt-to-income ratio of 45% was the primary factor" even if we can't show the full model.
You can see inside and understand exactly how inputs become outputs.
Examples:
Advantage: Complete transparency - you can explain both HOW and WHY
Limitation: May not capture complex patterns
Internal workings are complex and opaque. You see inputs and outputs but not the process.
Examples:
Advantage: Can capture very complex patterns and often more accurate
Limitation: Difficult to understand and explain decisions
Imagine calculating a restaurant bill. The formula is simple and transparent:
Example Calculation:
β’ 4 people having dinner
β’ 2 orders of appetizers
β’ 3 desserts
Total = $15 + (4 Γ $25) + (2 Γ $8) + (3 Γ $6) = $149
Why is this interpretable?
You can explain EXACTLY why the bill is $149. Each person adds $25, each appetizer adds $8, etc. There's no mystery!
Price = $50,000 + (1,500 Γ $120) + (3 Γ $15,000) + (2 Γ $8,000) = $311,000
β’ Base value: $50,000
β’ Square footage: +$180,000 β (largest impact)
β’ Bedrooms: +$45,000 β
β’ Garage: +$16,000 β
We can see EXACTLY how each feature contributes to the final price!
Logistic regression is like a scoring system that converts points into a 0-100% probability.
Calculate a score based on applicant features:
Then convert score to probability:
If Score = 2.5, then:
Probability of Approval = 92%
Still Interpretable!
We can see how each feature (credit score, income, debt ratio) contributes to the final probability.
Linear models assume each feature contributes independently. But real-world features often interact!
The model says:
But in reality: A house with a huge basement (2,000 sq ft) but tiny first floor (800 sq ft) is oddly shaped and less valuable than these numbers suggest!
The features interact - you can't just add them up independently!
This is where more complex (black-box) models can capture patterns that simple linear models miss.
More complex models (black-box) can capture intricate patterns and achieve higher accuracy, but become harder to interpret. Simpler models (white-box) are easy to understand but may miss complex relationships.
The choice depends on your business priorities: accuracy vs. transparency.
| Situation | Recommended Approach | Reasoning |
|---|---|---|
| Simple problem (2-3 features) |
White-Box | Linear relationships are likely sufficient; transparency is valuable |
| High regulatory requirements (banking, healthcare) |
White-Box | Must be able to explain and justify every decision |
| Complex problem (100+ features with interactions) |
Black-Box + Explanation Tools | Need complex model for accuracy, use SHAP/LIME for explanations |
| Image/text data (computer vision, NLP) |
Black-Box (Deep Learning) | Linear models can't handle these data types effectively |
| Internal analytics (no external stakeholders) |
Either | Choose based on accuracy needs vs. debugging convenience |
Business Question: "How does credit score affect loan approval probability, holding everything else constant?"
"On average, increasing credit score from 650 to 750 increases approval probability by 25 percentage points."
You've isolated the effect of ONE feature in your black-box model!
Imagine a group project where three students work together:
Try all possible team combinations:
SHAP averages across ALL these combinations to determine each student's fair share of the +25 point improvement.
The model's baseline approval rate (average across all applicants): 65%
Sarah's predicted approval probability: 32%
β’ Credit Score (720): +8% (helps approval)
β’ Annual Income ($45K): -15% (hurts approval)
β’ Debt-to-Income Ratio (45%): -20% (hurts approval)
β’ Business Age (3 years): -6% (hurts approval)
Total: +8% - 15% - 20% - 6% = -33% β
Primary issue: Debt-to-income ratio (-20%). This is what Sarah should focus on improving!
Imagine a restaurant with 200 menu items. Understanding how the chef prices EVERY dish is complex. But you just want to understand why YOUR specific order of pasta costs $18.
LIME's Approach: "Zoom in" on just the pasta dish and nearby similar dishes (other pasta, similar ingredients). Build a simple model for JUST THAT area of the menu.
| Aspect | SHAP | LIME |
|---|---|---|
| What it explains | Fair contribution of each feature | Local approximation with simple model |
| Computation time | Slower (tries all combinations) | Faster (approximates locally) |
| Consistency | Same explanation every time | Can vary slightly between runs |
| Theoretical foundation | Game theory (Shapley values) | Local approximation |
| Best for | High-stakes decisions needing precise attribution | Quick explanations, exploring many instances |
| Example use case | Explaining denied loan to regulator | Internal model debugging and validation |
In practice: Both are valuable tools! SHAP for precision, LIME for speed and ease of use.
LendingClub processes 10,000+ loan applications per month. They need to:
What increases loan approval chances?
π Home Ownership:
Owning a home β +30% approval chance
Interpretation: Homeowners are seen as more stable and creditworthy
π° Annual Income:
Each additional $10,000 β +5% approval chance
Interpretation: Higher income provides greater repayment capacity
π Debt-to-Income Ratio:
Each 10% increase β -25% approval chance
Interpretation: Higher existing debt obligations increase risk
β Credit Score:
100-point increase β +115% better approval odds
Interpretation: Credit score is the strongest predictor
Now that we understand the business meaning, here's the actual logistic regression formula:
Ξ²β (intercept) = -8.2
Ξ²β (home ownership) = 1.1 β exp(1.1) = 3.0 times better odds
Ξ²β (income per $10K) = 0.05 β exp(0.05) = 1.05 times better odds
Ξ²β (debt-to-income per 10%) = -1.4 β exp(-1.4) = 0.25 times the odds
Ξ²β (credit score per 100 pts) = 0.77 β exp(0.77) = 2.15 times better odds
Key Point: Notice how we presented business meaning FIRST, then the formula. This aids understanding!
A doctor tests 100 patients for cancer. 30 actually have cancer, 70 don't.
| Actual Reality | Doctor's Prediction | |
|---|---|---|
| Predicted: Cancer | Predicted: No Cancer | |
| Actually has cancer (30) | 25 True Positive (TP) β Correctly detected |
5 False Negative (FN) β Missed cancer! |
| Actually healthy (70) | 8 False Positive (FP) β False alarm |
62 True Negative (TN) β Correctly cleared |
| Actual Outcome | Model's Prediction | |
|---|---|---|
| Predicted: Good Loan | Predicted: Will Default | |
| Good loan (9,704) | 8,200 True Positive β Approved good customer |
1,504 False Negative β Rejected good customer π° Lost revenue! |
| Default (2,586) | 415 False Positive β Approved bad customer π° Will lose money! |
2,171 True Negative β Correctly rejected |
Business Question: Which error is more costly?
FP is ~5x more costly than FN! Our model should err on the side of caution.
β οΈ Warning: Can be misleading with imbalanced data! If 95% of loans are good, predicting "all good" gives 95% accuracy but is useless.
Business meaning: We're capturing 84.5% of revenue opportunities. Missing 15.5%.
Business meaning: We're avoiding 84% of potential defaults. 16% slip through.
Business meaning: When we approve a loan, we're right 95.2% of the time. High confidence in approvals!
Business meaning: Harmonic mean of precision and recall. Useful when you want a single score that balances both.
It depends on business priorities:
Logistic regression gives us probabilities (0-100%). We need to choose a cutoff threshold (Z) to make binary approve/reject decisions.
Approve only if probability > 85%
Result: Very few defaults (high precision) but miss many good customers (low recall)
Risk: Business stagnation from lost customers
Approve if probability > 75%
Result: Moderate defaults, good customer capture
Sweet spot: Sustainable growth with acceptable risk
Approve if probability > 50%
Result: Capture most good customers but many defaults too
Risk: Bankruptcy from excessive defaults
Receiver Operating Characteristic curve shows True Positive Rate (Recall) vs. False Positive Rate at different thresholds.
X-axis: False Positive Rate (bad loans approved)
Y-axis: True Positive Rate (good loans approved)
Area Under the Curve = 0.66
β’ 0.5 = Random guessing (diagonal line)
β’ 1.0 = Perfect prediction
β’ 0.66 = Decent, but room for improvement
Take a moment to answer these questions (write notes, discuss with a neighbor):
π Resource: Orange Explain Model widget documentation at
orangedatamining.com/widget-catalog/explain/explain-model/
Try this Google Colab notebook for hands-on SHAP/LIME practice:
π Colab Notebook:
colab.research.google.com
(Notebook link will be provided via Canvas)
Note: Focus on interpreting the outputs rather than the code syntax. The goal is understanding explanations, not becoming a Python expert!
Build a predictive model that:
Define the decision being made and why it matters. Who needs to understand the model's decisions (managers, customers, regulators)?
Did you choose a white-box or black-box model? Justify this choice based on your business context. If black-box, which explanation techniques will you use?
Show which features are most important overall. Use feature importance scores, SHAP summary plots, or coefficients (for linear models).
Pick 2-3 specific examples and explain WHY the model made those predictions. Show how features contributed to each decision.
Justify your metric selection based on business costs. Is FP or FN more costly? Did you adjust the decision threshold accordingly?
Address potential biases. Could your model discriminate unfairly? How did you check for this?
Interpretability is understanding HOW a model works overall. Explainability is understanding WHY a specific decision was made. Both are valuable in different contexts.
Simple, transparent models (white-box) are easy to explain but may miss complex patterns. Complex models (black-box) can be more accurate but require explanation techniques like SHAP, LIME, or Partial Dependence Plots.
Accuracy isn't everything! Choose metrics (precision, recall, F-score) based on the relative costs of different error types in your business problem.
SHAP, LIME, and PDPs help explain black-box models but don't make biased models fair or bad models good. They're diagnostic tools that reveal how models make decisions.
In high-stakes domains, being able to explain WHY a decision was made is not just nice to haveβit's essential for regulatory compliance, customer trust, and model debugging.
We'll explore Time Series Forecasting - predicting future values based on historical patterns. Topics include seasonality, trends, and ARIMA models.
Questions? Office hours or post on the discussion forum!
Week 11: Model Interpretability & Explainability
Remember: The best model is one you can explain and trust.
Next Steps:
1. Complete the Orange hands-on exercises
2. Start planning your Assessment 3 interpretability approach
3. Review key concepts using the practice quiz on Canvas