1 / 38

DATA4400: Data-Driven Forecasting

Week 6: Advanced Time Series Analysis

Stationarity, Random Walks, and ARMA Models

Understanding the foundations of modern time series modeling

Kaplan Business School Australia
Advanced Forecasting Techniques

Workshop Learning Outcomes

Foundation: Building on smoothing techniques to advanced statistical models

Recap: Smoothing Techniques

What is Smoothing?

Smoothing techniques reduce noise and irregular fluctuations in time series data to reveal underlying patterns and trends.

Key Benefits:

  • Reduces random variation
  • Highlights trends and patterns
  • Improves forecast accuracy
  • Simplifies data interpretation

Common Applications:

  • Sales forecasting
  • Economic indicators
  • Stock price analysis
  • Seasonal adjustments

Moving Average (MA) Technique

MA(k) = (Yt + Yt-1 + ... + Yt-k+1) / k

Advantages

  • Simple to calculate
  • Effective noise reduction
  • Good for trend identification
  • No parameter estimation needed

Limitations

  • Lags behind actual data
  • All observations weighted equally
  • Poor at capturing recent changes
  • Requires sufficient historical data

Exponential Smoothing (ES) Technique

Ft+1 = αYt + (1-α)Ft

Key Features

  • α (alpha): Smoothing parameter (0 < α < 1)
  • Higher α: More responsive to recent data
  • Lower α: More smoothing effect
  • Weighted average: Recent data gets more weight

When to Use

  • Data with no clear trend
  • Short-term forecasting
  • Rapidly changing conditions
  • Limited historical data

Exercise: Fill in the Calculations

Compare 3-Period MA vs Exponential Smoothing (α=0.3)

Month Actual Sales MA(3) Forecast MA Error SES (α=0.3) SES Error
1 120 - - 120 -
2 135 - -
3 118 - -
4 142
5 128

Understanding Stationarity

What is Stationarity?

A time series is stationary when its statistical properties (mean, variance, autocorrelation) remain constant over time.

Stationary Series

  • Constant mean over time
  • Constant variance over time
  • Autocorrelation depends only on lag
  • No systematic change in structure

Non-Stationary Series

  • Trending mean
  • Changing variance
  • Seasonal patterns
  • Structural breaks

Stationary vs Non-Stationary: Visual Examples

Stationary Series

✓ Constant mean and variance
✓ Predictable behavior

Non-Stationary Series

✗ Trending behavior
✗ Changing variance

Why Stationarity Matters in Forecasting

Critical Importance

Many statistical models (like ARMA) assume stationarity because it ensures reliable parameter estimation and stable predictions.

With Stationarity

  • Stable Models: Parameters remain meaningful
  • Reliable Forecasts: Confidence intervals valid
  • Simplified Analysis: No time dependencies
  • Better Performance: Consistent accuracy

Without Stationarity

  • Unstable Models: Parameters change over time
  • Poor Forecasts: Unreliable predictions
  • Spurious Relationships: False correlations
  • Model Failure: Breakdown in assumptions

Random Walk Models

Simple Random Walk: Yt = Yt-1 + εt

Key Concept

A random walk is a process where the next value equals the current value plus a random shock. Today's best forecast for tomorrow is simply today's value.

Real-World Examples:

  • Stock Prices: Efficient market hypothesis
  • Exchange Rates: Currency fluctuations
  • Economic Indicators: GDP, unemployment rates

Random Walk with Drift

Random Walk with Drift: Yt = δ + Yt-1 + εt

Simple Random Walk

  • No systematic direction
  • Mean is stationary
  • Variance increases over time
  • δ = 0

Random Walk with Drift

  • Systematic upward/downward trend
  • Mean is non-stationary
  • Both trend and variance increase
  • δ ≠ 0 (drift parameter)

Key Insight: Drift (δ) represents the average change per period

Random Walk Properties: Visual Demonstration

Stationarity Properties

  • Simple Random Walk: Stationary in differences, not in levels
  • Random Walk with Drift: Non-stationary in both levels and differences
  • Variance: Increases linearly with time for both types

Autoregressive (AR) Models

AR(1): Yt = μ + α(Yt-1 - μ) + εt

Simple Explanation

Autoregressive models predict today's value based on yesterday's values. Think of it as learning from the recent past to predict the immediate future.

Intuitive Understanding:

  • Weather: Tomorrow's temperature depends on today's
  • Stock Returns: Today's performance influences tomorrow's
  • Sales: This month's sales predict next month's

Parameter α: Strength of relationship with past values (-1 < α < 1 for stationarity)

Moving Average (MA) Models in Time Series

MA(1): Yt = μ + εt + β1εt-1

Key Difference from MA Smoothing

Unlike moving average smoothing, MA models in time series focus on past forecast errors rather than past values themselves.

MA Smoothing

  • Uses past actual values
  • Simple averaging technique
  • Data preprocessing method
  • No error term modeling

MA Models

  • Uses past forecast errors
  • Statistical modeling approach
  • Accounts for shock persistence
  • Models error autocorrelation

ARMA Models: Combining AR and MA

ARMA(p,q): Yt = μ + Σαi(Yt-i - μ) + εt + Σβjεt-j

Best of Both Worlds

ARMA models combine the memory of past values (AR component) with the impact of past shocks (MA component).

Model Components:

  • AR(p): p past values influence current value
  • MA(q): q past errors influence current value
  • Common: ARMA(1,1), ARMA(2,1), ARMA(1,2)

When to Use Each Model: Comparison Guide

Model Type Best Used When Key Characteristics Limitations
Random Walk Efficient markets, no predictable patterns Current value = best forecast No learning from patterns
AR Models Clear dependency on recent values Uses past values directly Ignores error patterns
MA Models Short-term shock effects dominate Uses past forecast errors No direct value dependency
ARMA Models Complex patterns with both dependencies Combines both approaches More complex to estimate

Knowledge Check: Quiz 1

Which of the following best describes why stationarity is important for ARMA models?
A) It makes the data easier to visualize and understand
B) It reduces the computational complexity of model estimation
C) It ensures constant statistical properties needed for reliable parameter estimation and stable predictions
D) It eliminates the need for diagnostic testing of model residuals
Correct! Stationarity ensures that the mean, variance, and autocorrelation structure remain constant over time, which is crucial for ARMA models to provide reliable parameter estimates and stable forecasts.

Model Identification: ACF and PACF

Diagnostic Tools

ACF (Autocorrelation Function) and PACF (Partial Autocorrelation Function) help us identify the appropriate ARMA model structure.

ACF (Autocorrelation Function)

  • Measures correlation at different lags
  • Includes both direct and indirect effects
  • Helps identify MA order (q)
  • Shows overall correlation structure

PACF (Partial Autocorrelation Function)

  • Measures direct correlation only
  • Controls for intermediate lags
  • Helps identify AR order (p)
  • Shows unique lag contributions

Pattern Recognition Rules

Model Identification Guidelines:

Model Type ACF Pattern PACF Pattern
AR(p) Gradual decay Cuts off after lag p
MA(q) Cuts off after lag q Gradual decay
ARMA(p,q) Gradual decay after lag (q-p) Gradual decay after lag (p-q)

Advanced Tip: If AR and MA terms seem to cancel each other's effects, try models with one fewer AR term and one fewer MA term.

ACF and PACF: Visual Examples

AR(1) Process

✓ PACF cuts off after lag 1
✓ ACF shows gradual decay

MA(1) Process

✓ ACF cuts off after lag 1
✓ PACF shows gradual decay

MA Process Deep Dive

MA(1) Process

Yt = μ + εt + β1εt-1
  • Memory: Only 1 period
  • ACF: Non-zero at lag 1 only
  • PACF: Decays exponentially
  • Use case: Short-term shocks

MA(2) Process

Yt = μ + εt + β1εt-1 + β2εt-2
  • Memory: 2 periods
  • ACF: Non-zero at lags 1 and 2
  • PACF: Complex decay pattern
  • Use case: Medium-term effects

MA Process: Correlation Patterns

Key Observations:

  • MA(1): Strong correlation at lag 1, then cuts off sharply
  • MA(2): Strong correlations at lags 1 and 2, then cuts off
  • Cut-off behavior: Distinguishing feature of MA processes

Knowledge Check: Quiz 2

If you observe that the PACF cuts off sharply after lag 2, while the ACF shows a gradual decay pattern, which model would you most likely choose?
A) MA(2) - because there are 2 significant lags
B) AR(2) - because PACF cuts off after lag 2
C) ARMA(2,2) - because both ACF and PACF show patterns
D) Random Walk - because the patterns are unclear
Correct! When PACF cuts off after lag p, it suggests an AR(p) model. The gradual decay in ACF is consistent with this interpretation.

Autocorrelation Case Study

Practical Application

Let's analyze real sales data to understand how autocorrelation patterns guide our model selection process.

Case: Monthly Retail Sales

A retail chain has 60 months of sales data. We need to:

  • Calculate and interpret the autocorrelation function
  • Identify appropriate lag structure
  • Select the best ARMA model
  • Validate our choice using residual analysis

Autocorrelation Function: Technical Details

ACF(k) = Corr(Yt, Yt-k) = Cov(Yt, Yt-k) / √Var(Yt)Var(Yt-k)

Key Properties

  • Range: -1 ≤ ACF(k) ≤ 1
  • Symmetry: ACF(k) = ACF(-k)
  • Lag 0: ACF(0) = 1 always
  • Significance: Usually plot lags 1-20

Interpretation Guidelines

  • |ACF| > 0.3: Strong correlation
  • 0.1 < |ACF| < 0.3: Moderate correlation
  • |ACF| < 0.1: Weak correlation
  • Confidence bands: ±1.96/√n

Case Study: ACF Pattern Analysis

Observations from the Plot:

  • Lag 1: Strong positive correlation (0.68)
  • Lag 2: Moderate correlation (0.31)
  • Lag 3+: Gradual decay within confidence bands
  • Pattern: Suggests AR(2) or ARMA(2,1) model

Practical Uses of ACF Analysis

Model Identification

  • Determine ARMA order (p,q)
  • Guide initial model selection
  • Compare competing models
  • Identify seasonal patterns

Residual Diagnostics

  • Check for remaining patterns
  • Validate model adequacy
  • Detect specification errors
  • Ensure white noise residuals

White Noise Check

After fitting an ARMA model, residuals should show ACF ≈ 0 for all lags, indicating the model has captured all systematic patterns.

Knowledge Check: Quiz 3

After fitting an ARMA model to your data, you examine the ACF of the residuals and find significant correlations at lags 1 and 3. What does this suggest?
A) The model is perfect and ready for forecasting
B) The model is inadequate and hasn't captured all patterns in the data
C) This is normal behavior and can be ignored
D) The data needs to be transformed before modeling
Correct! Significant autocorrelations in residuals indicate that the model hasn't captured all systematic patterns. You should consider adding more AR or MA terms or trying a different model specification.

Comprehensive Model Selection Workflow

1. Data Preparation

Check stationarity, transform if needed

2. ACF/PACF Analysis

Identify potential model orders

3. Model Estimation

Fit candidate ARMA models

4. Model Validation

Check residuals and goodness-of-fit

Iterative Process

Model selection is often iterative. If residual diagnostics fail, return to step 2 and consider alternative specifications.

Looking Ahead: ARIMA Models

Next Week Preview

ARIMA models extend ARMA by including Integration (I) to handle non-stationary data through differencing.

ARMA Models

  • Require stationary data
  • Work with levels of data
  • Limited to stationary processes
  • Foundation for ARIMA

ARIMA Models

  • Handle non-stationary data
  • Work with differences
  • More flexible framework
  • Include seasonal patterns

Practical Implementation Guidelines

Software Implementation:

  • Python: statsmodels.tsa.arima.model
  • R: forecast package, auto.arima()
  • Excel: Limited ARMA capabilities
  • MATLAB: Econometrics Toolbox

Best Practices

  • Always check stationarity first
  • Start with simple models
  • Use information criteria (AIC, BIC)
  • Validate with out-of-sample testing

Common Pitfalls

  • Ignoring non-stationarity
  • Over-fitting with too many parameters
  • Not checking residual diagnostics
  • Extrapolating too far ahead

Comparison: ARMA vs Prophet vs Holt-Winters

Aspect ARMA Models Prophet Holt-Winters
Data Requirements Stationary time series Daily/weekly data with trends Data with seasonality
Complexity High - requires expertise Low - automated features Medium - interpretable parameters
Seasonality Limited seasonal handling Multiple seasonal patterns Single seasonal pattern
Trend Handling Requires differencing Automatic trend detection Linear/exponential trends
Missing Data Poor tolerance Handles gaps well Requires complete data
Interpretability Good for statistical insight Business-friendly components Clear seasonal decomposition
Best Use Case Short-term, stationary data Business forecasting with growth Regular seasonal patterns

Key Takeaway: Choose based on data characteristics, required expertise, and forecasting horizon.

Model Limitations and Considerations

ARMA Limitations

  • Linear only: Cannot capture non-linear patterns
  • Constant parameters: No time-varying coefficients
  • Normal errors: Assumes Gaussian distributions
  • Short memory: Limited to recent past influence

When to Consider Alternatives

  • Non-linear patterns: Use GARCH, threshold models
  • Structural breaks: Use regime-switching models
  • Long memory: Use ARFIMA models
  • Multiple series: Use VAR models

Remember: ARMA models are powerful but not universal. Always validate assumptions and consider context.

Key Formulas: Quick Reference

Random Walk: Yt = Yt-1 + εt
Random Walk with Drift: Yt = δ + Yt-1 + εt
AR(1): Yt = μ + α(Yt-1 - μ) + εt
MA(1): Yt = μ + εt + β1εt-1
ARMA(1,1): Yt = μ + α(Yt-1 - μ) + εt + β1εt-1

Model Identification: Quick Guide

Pattern Observed Suggested Model Next Steps
PACF cuts off at lag p AR(p) Estimate and validate
ACF cuts off at lag q MA(q) Estimate and validate
Both ACF and PACF decay ARMA(p,q) Try multiple combinations
No clear pattern Random Walk Consider differencing

Remember: These are guidelines, not rigid rules. Always validate with residual analysis!

Hands-On Workshop Activities

Activity 1: FOURWEEK_SALES Analysis

  • Fit MA(1) model to fortnightly sales data
  • Calculate RMSE and forecast accuracy
  • Generate 1-step and 2-step ahead forecasts
  • Construct 68% prediction intervals

Activity 2: Moving Average Modeling

  • Import and detrend time series data
  • Model MA(1) and MA(2) processes
  • Perform in-sample forecasting
  • Visualize results and compare performance

Assessment and Preparation for Next Week

Key Concepts to Master

  • Stationarity testing and interpretation
  • Random walk properties and applications
  • AR, MA, and ARMA model structures
  • ACF and PACF pattern recognition

Next Week: ARIMA Models

  • Integration and differencing
  • Seasonal ARIMA (SARIMA)
  • Model selection criteria
  • Advanced diagnostic testing

Recommended Reading

Review course materials on stationarity testing, practice ACF/PACF interpretation, and prepare for next week's ARIMA modeling session.

Questions & Discussion

Key Discussion Points

  • How do you decide between AR, MA, or ARMA models in practice?
  • What are the most common challenges in stationarity testing?
  • How do you handle ambiguous ACF/PACF patterns?
  • When would you choose a random walk over ARMA models?

Practical Tips for Success:

  • Always start with data visualization and exploration
  • Don't rely solely on one diagnostic tool
  • Practice with real data to build intuition
  • Compare multiple models before making final decisions

Session Summary

What We've Covered Today

  • ✓ Reviewed smoothing techniques and their applications
  • ✓ Mastered stationarity concepts and their importance
  • ✓ Explored random walk models and their properties
  • ✓ Understood AR, MA, and ARMA model structures
  • ✓ Learned ACF/PACF pattern recognition for model identification
  • ✓ Applied autocorrelation analysis to real case studies

Ready for Next Week:

You now have the foundation to tackle ARIMA models, seasonal adjustments, and advanced forecasting techniques!

Thank you for your participation!