1 / 20

Assessment 1: Skills-Building Project

Predictive and Prescriptive Analytics for Business Decision Making

DATA5000 - Artificial Intelligence Programming in Business Analytics

25 Marks | 25% of Final Grade | Due Week 5

Webinar Objectives

By the end of this session, you will understand:

Assessment Structure

  • What you need to submit
  • How marks are allocated
  • Report structure (6 sections)
  • Word limits and formatting

Success Strategies

  • What markers are looking for
  • Common mistakes to avoid
  • Tips for high marks
  • Using GenAI appropriately

Key Message: This assessment tests your ability to translate technical AI analysis into business recommendations that executives can act upon.

Assessment Overview

Component Details
Type Individual Business Report
Word Limit 1,200 words maximum (+/- 10% acceptable)
Marks 25 marks total (25% of final grade)
Due Date Week 5, Tuesday 23:55 AEST
Submission Turnitin via MyKBS
GenAI Policy Level 2 - Optional use with documentation

Important: Only the business report (.docx) is marked. The notebook is for reference only.

The Business Scenario

CloudMetrics Inc. - SaaS Analytics Platform

You are a Business Analytics Consultant hired by CloudMetrics, a growing SaaS company experiencing customer retention challenges.

The Business Problem

  • Customer churn affecting growth
  • Need to predict at-risk customers
  • Want to understand churn drivers
  • Testing retention campaign effectiveness

Executive Questions

  • Which customers will churn?
  • What factors drive churn?
  • Does our retention campaign work?
  • What actions should we take?

Your Task: Analyse their customer data using AI techniques and provide actionable business recommendations.

What You Submit

Google Colab Notebook

Reference Only - Not Marked

  • Pre-built code provided
  • Run all cells to generate outputs
  • Saves 11 visualisations
  • Share link with your report

Time needed: 30-45 minutes

Business Report (.docx)

This Is What Gets Marked!

  • 1,200 words maximum
  • 6 structured sections
  • 3-5 visualisations from notebook
  • Professional executive format

Time needed: 3-4 hours

Workflow: Complete the notebook FIRST, then write your report using the outputs and insights generated.

The Google Colab Notebook

The notebook contains all the code you need. Your job is to run it and interpret the results.

Section 1A: Data Prep

  • Load 15,000 customers
  • Summary statistics
  • 4 visualisations
  • Handle missing values

Section 1B: Prediction

  • LightGBM churn model
  • TFT revenue forecasting
  • Performance metrics
  • Confusion matrix

Section 1C: SHAP

  • Feature importance
  • Summary plot
  • Waterfall plot
  • Dependence plots

Section 1D: Causal Analysis

  • EconML DRLearner
  • ATE calculation
  • CATE by segments
  • Treatment effects

Section 2: Summary

  • Key findings checklist
  • List of saved images
  • Report writing tips
  • Submission reminders

Report Structure: 6 Sections

Section 1

Business Problem & Data

4 marks
~250 words

Section 2

Predictive Analytics

4 marks
~400 words

Section 3

Explainable AI

4 marks
~200 words

Section 4

Causal Analysis

4 marks
~150 words

Section 5

Recommendations

5 marks
~250 words

Section 6

Communication

4 marks
Overall quality

Total: 25 marks | 1,200 words maximum | 3-5 visualisations required

Section 1: Business Problem & Data Overview 4 marks

Word target: ~250 words | Purpose: Set the business context

What Markers Want

  • Clear problem statement
  • Specific data statistics (e.g., "15,000 customers, 7.7% churn rate")
  • Key variables identified
  • Business impact stated

Common Mistakes

  • Generic descriptions without numbers
  • Not mentioning churn rate
  • Copying dataset description verbatim
  • Too much technical jargon

Example of Good Opening:

"CloudMetrics faces a significant retention challenge with a 7.7% churn rate across its 15,000-customer base. Analysis of 26 customer attributes including engagement metrics, contract types, and payment patterns reveals opportunities for targeted intervention..."

Section 2: Predictive Analytics Findings 4 marks

Word target: ~400 words | Purpose: Present prediction results with business meaning

LightGBM Churn Model

  • Report accuracy, precision, recall
  • Explain what metrics mean for business
  • Include confusion matrix insights
  • Identify false positive/negative trade-offs

TFT Revenue Forecasting

  • Report MAE, RMSE, MAPE
  • Explain forecast accuracy
  • Describe revenue trends
  • Business planning implications

Critical Requirement:

You must cover BOTH models (LightGBM AND TFT). Reports discussing only one model will lose marks.

Tip: Don't just list numbers. Explain what "85% accuracy" means: "The model correctly identifies 85 out of 100 customers' churn status, enabling targeted retention efforts."

Section 3: Explainable AI Insights 4 marks

Word target: ~200 words | Purpose: Explain what drives churn using SHAP

Required: Top 5 SHAP Features

You MUST identify and explain the top 5 features from your SHAP analysis. Example:

  1. Customer Health Score - Lower scores strongly predict churn
  2. Days Since Last Login - Longer gaps increase churn risk
  3. Contract Type - Month-to-month customers churn more
  4. Payment Failures - Each failure increases churn probability
  5. Feature Adoption Score - Low adoption signals disengagement

Good Approach

"SHAP analysis reveals customer_health_score as the strongest predictor, with scores below 40 increasing churn probability by 35%."

Weak Approach

"SHAP shows various features are important for predicting churn in the model."

Section 4: Causal Analysis & Treatment Effects 4 marks

Word target: ~150 words | Purpose: Show if the retention campaign actually works

Key Concepts to Report

ATE (Average Treatment Effect)

Overall impact of retention campaign on churn. Example: "The campaign reduces churn by 18% on average."

CATE (Conditional Treatment Effect)

How effect varies by segment. Example: "Enterprise customers show 30% reduction vs 12% for SMB."

Correlation vs Causation

Critical Distinction: SHAP tells us what predicts churn (correlation). EconML tells us what causes changes in churn (causation). Your recommendations should focus on causal factors because those are actionable.

Section 5: Recommendations 5 marks

Word target: ~250 words | Purpose: Synthesise all findings into actionable strategy

This is the highest-weighted section! It tests whether you can connect all your analysis into coherent business strategy.

Strong Recommendation

"Based on CATE analysis showing 30% churn reduction for high-value customers, CloudMetrics should prioritise retention campaigns for Enterprise accounts with health scores below 50. Expected ROI: $240,000 annually."

Weak Recommendation

"CloudMetrics should focus on reducing churn and improving customer satisfaction through better service."

Strong recommendations include:

  • Specific actions (who, what, when)
  • Evidence from your analysis (cite specific findings)
  • Expected business impact (quantified where possible)
  • Prioritisation based on causal effects, not just correlations

Section 6: Business Communication 4 marks

Purpose: Overall report quality, structure, and professional presentation

Writing Quality

  • Clear, concise language
  • No jargon without explanation
  • Logical flow between sections
  • Executive-appropriate tone

Visual Elements

  • 3-5 charts from notebook
  • Charts support key points
  • Proper labels/captions
  • Professional formatting

Structure

  • All 6 sections present
  • Within word limit
  • Notebook link included
  • Proper .docx format

Visualisation Selection Tips

Choose charts that directly support your narrative. Good choices: SHAP summary plot (for Section 3), CATE by segments (for Section 4), confusion matrix or feature importance (for Section 2).

Understanding the Mark Distribution

Grade Band Mark Range What It Looks Like
High Distinction 21.25 - 25 Exceptional synthesis, specific evidence, quantified business impact
Distinction 18.75 - 21.24 Strong analysis with clear business application, good evidence use
Credit 16.25 - 18.74 Competent analysis, some business insight, adequate evidence
Pass 12.50 - 16.24 Basic understanding, limited business application, some missing elements
Fail 0 - 12.49 Missing sections, no evidence from data, generic statements

Key to High Marks: Every claim must be supported by specific data from YOUR notebook analysis. Generic statements without evidence will not score well.

Tips for High Marks

Do This

  • Be specific: "7.7% churn rate" not "high churn"
  • Cite your analysis: "SHAP shows..." or "The ATE of -0.18 indicates..."
  • Explain business meaning: What does 85% accuracy mean for CloudMetrics?
  • Connect sections: Recommendations should flow from your findings
  • Quantify impact: "Expected to save $X annually"
  • Choose visuals wisely: 3-5 charts that support your narrative

Avoid This

  • Generic statements: "Churn is a problem for businesses"
  • Theory without application: Explaining what SHAP is without using results
  • Missing sections: Skipping TFT or causal analysis
  • Over word limit: Penalties apply for exceeding 1,320 words
  • Technical jargon: Write for executives, not data scientists
  • Disconnected recommendations: Suggestions not based on your analysis

Top 10 Common Mistakes to Avoid

  1. Only covering LightGBM - You need BOTH models
  2. Fewer than 3 visuals - Minimum required
  3. No specific numbers - Every section needs data
  4. Confusing correlation with causation - SHAP vs EconML
  5. Vague recommendations - "Improve customer service"
  1. Missing notebook link - Required for reference
  2. Wrong file format - Must be .docx
  3. Over word limit - 5% penalty per 10% over
  4. Not running all notebook cells - Missing outputs
  5. Copy-pasting technical output - Interpret, don't copy

Word Count Penalties

Over LimitPenalty
1-10% (1,201-1,320 words)-0.5 marks
11-20% (1,321-1,440 words)-1.0 marks
>20% (1,441+ words)-2.0 marks

Using Generative AI (GenAI Level 2)

Level 2: Optional Use with Documentation

You MAY use GenAI tools (ChatGPT, Claude, etc.) to help with this assessment, but you MUST document your use.

Appropriate GenAI Use

  • Explaining technical concepts
  • Improving grammar and clarity
  • Structuring your report
  • Understanding error messages
  • Generating initial drafts (then edit significantly)

Inappropriate GenAI Use

  • Generating entire report without editing
  • Not documenting GenAI use
  • Using GenAI output without verification
  • Submitting AI analysis of different data
  • Asking GenAI to interpret YOUR notebook results*

*Critical: GenAI cannot see your notebook. If you ask "what does my SHAP plot show?", it will make up an answer. Always interpret your own results.

Recommended Timeline

Week 4

Review materials

Read brief & rubric

→
Week 4-5

Complete notebook

30-45 minutes

→
Week 5

Write report

3-4 hours

→
Before Due

Review & submit

Check all requirements

Pre-Submission Checklist

  • All 6 sections complete
  • Word count under 1,200 (check in Word)
  • 3-5 visualisations included
  • Notebook link in report
  • Saved as .docx format
  • GenAI use documented (if applicable)

Resources Available to You

Assessment Materials

  • Assessment Brief (PDF)
  • Student Marking Rubric
  • Google Colab Notebook
  • Dataset files (CSV)

Course Materials

  • Week 2: LightGBM slides
  • Week 3: Deep Learning
  • Week 4: Causal ML
  • Workshop notebooks

Support

  • Workshop Q&A time
  • Discussion forums
  • Email your facilitator
  • Academic Success Centre

Where to Find Materials

All assessment materials are available on MyKBS under the Assessment 1 section. The notebook link and datasets will be provided in Week 4.

Questions?

Key Takeaways

  • Only the business report is marked (not the notebook)
  • Every claim needs specific evidence from your analysis
  • Section 5 (Recommendations) is worth the most - synthesise everything
  • Write for executives, not data scientists
  • Run the notebook first, then write your report

Due: Week 5, Tuesday 23:55 AEST

Good luck with your assessment!