Risk & ML models: Stress, Scenario Testing & Evaluation

Learn how to evaluate risk in machine models

This 6-module program geared towards practitioners!

Delivered by QuantUniversity in partnership with PRMIA

LIVE/ONLINE/ONDEMAND

REGISTER

Weekly Course Schedule

01

Introduction to Machine Learning, AI & Risk

Machine Learning in Finance : A tour key methods used in Machine learning
Defining risk in ML models

Concept drift, data drift, model drift

Stress, Scenario Testing & Evaluation

Key metrics

The role of Algorithmic auditors for ML models

Motivation: Case study Covid -19

Scenario generation and testing with Synthetic data

02

Stress Testing and Scenario Generation

How are AI/ML models different from traditional models?

Scenario stress testing

Reverse stress testing

- Identifying and assessing tail risk scenarios
Scenario generation

- Role of Synthetic data and data augmentation

The ML life cycle and risks

Case study:
Stress testing of an ML-based forecasting model under different regimes

03

Metrics and Evaluation for risk in ML models

Metrics for Quantifying risks in ML models
Working with sensitive data

Detecting Data leakage
Monitoring and retuning/retraining
ML Risk reporting

Case study:
- A dashboard for measuring and evaluating risk in ML models

04

Anomalies and outliers

Detecting and addressing anomalies
Explainability & Outlier analysis
Methods for generating and testing for anomalies
Checks for plausibility
Data techniques and Ensemble methods to address anomalies

Case study:
Anomaly detection in time-series datasets using GANS

05

Model Validation of Machine learning models

Verification vs Validation of ML models

Benchmarking ML models

Challenger models

Backup models

Issues when adopting machine learning models:-

- Model selection challenges
- Interpretability and Explainability

Case study:
Validating an ML model for credit risk

06

Frontier topics & wrap up

Operationalizing evaluation of risk in ML models.

- Real-time & near-realtime risk evaluation
- Architecture choices for scaling risk calculations
- Issues with integrating traditional and ML models

- Governance Mechanisms to address Risk in ML models
- Algorithm auditing & issues of Bias and fairness
- Adversarial attacks, Sensitive Data and unknown risks
Frontier topics

- Deep learning and other ML innovations
- Technologies and trends to look out for

Summary

The use of AI and machine learning in finance has grown significantly in the last few years. As more and more AI and ML applications are being deployed in enterprises, concerns are growing about the increased complexity of models, the growing ecosystem of untested frameworks and products, potential for AI accidents, model and reputation risk. As the debate about explainability, fairness, bias, and privacy grows, there is increased attention to understanding how the models work and whether the models are designed and thoroughly tested to address potential issues.

The growth of data-driven applications have changed the financial industry. AI and ML models have accelerated business transformation, reduced turn-around times and have enabled applications that weren’t feasible just a few years ago. Institutions have ramped up the adoption of ML models and are seeing significant benefits through the growing portfolios of ML based decision making models. While the interest is huge, the challenges of comprehensively testing and evaluating ML models remain. AI accidents and the risks associated with algorithmic decision making is challenging enterprises to innovate and adopt risk management techniques factoring the new realities!

In this QuantUniversity Course, we will discuss the key aspects of risks in ML models and discuss key techniques in stress, scenario testing and evaluation of machine learning models. Through examples and case studies, we will discuss the state-of-the-art in testing and evaluation of ML-based models and how to comprehensively address risk when developing, deploying and monitoring ML applications. By the end of the course, participants will have a clear idea of the challenges, best practices and pragmatic tools that can be used to address risks in machine learning models

Hands-on examples and case studies through QuSandbox will be provided to reinforce concepts.

COURSE details

The financial industry has been adopting AI and machine learning at a rapid pace. Alternative datasets including text analytics, cloud computing, algorithmic trading are game changers for many firms exploring novel modeling methods to augment their traditional investment and decision workflows. As more and more open-source technologies penetrate enterprises, quants and data scientists have a plethora of choices for building, testing and scaling models. While there is significant enthusiasm, model risk professionals and risk managers are concerned about the onslaught of new technologies, programming languages, and data sets that are entering the enterprise. With little formal guidance from regulators on how to validate models and quantify model risk, organizations are developing their own home-cooked methods to address model risk management challenges.

In this course, we aim to bring clarity on some of the model risk management and validation challenges with data science and machine learning models in the enterprise. We will discuss key drivers of model risk in today’s environment and how the scope of model risk management is changing. We will introduce key concepts and discuss aspects to be considered when developing a model risk management framework incorporating data science techniques and machine learning methodologies in a pragmatic way.

Learning Objectives
Upon completion of this course, you will be able to:
• Role of Machine Learning and AI in financial services
• Model Risk Management challenges and best practices for machine learning models
• Validating machine learning models: Quantifying risk, best practices and templates
• Regulatory guidance and the future
• Practical case studies with sample code

Delivery:

Session: 1.5 hours/session
Duration: 5 weeks + Practical Exercise (2 weeks)
Case study + Labs using the Qu.Academy

*If you would like an invoice for your payment for reimbursement or related questions on alternative payment methods, please contact info@qusandbox.com

Who should attend?

Model Risk professionals, Model validators, Regulators and Financial professionals new to data-driven methodologies
Quantitative analysts, investment professionals, Machine learning enthusiasts interested in understanding model risk and governance aspects in fintech, insurance and financial organizations

Practical Exercise:
Participants will go through a guided exercise to perform model validation on a chosen machine learning model of their choice. Guidance will be provided in scoping and implementing the project.

Week 1

module 1

Machine Learning and AI: A Model Risk Perspective

Drivers of Model Risk in the age of data science and AI
Machine Learning vs Traditional quant models: How has the world changed?
A tour of Machine Learning and AI methods
Supervised vs Unsupervised Learning (Regression, Neural Networks, XGBoost, PCA, Clustering)
Deep Learning & Reinforcement Learning (Keras, Tensorflow, PyTorch)
Automatic Machine Learning & Machine Learning APIs (Google,
Comprehend, Watson)
ML on the cloud vs On-prem

Models redefined: Data, Modeling environment, Modeling tools, Modeling process

Week 2

MODULE 2

Model Risk Management for Machine Learning Models - Part 1

-ML Life cycle management
-Tracking
-Metadata management
-Scaling
-Reproducibility
-Interpretability
-Testing
-Measurement

The Decalogue: Ten key aspects to factor when developing your model risk management framework when integrating Machine Learning models:
1. Models redefined: It’s not just input, process and output
2. Governing the Machine Learning process
3. Model Verification and Validation for Machine Learning Models
4. Performance Metrics and Evaluation criteria
5. Model Inventory and tracking

Week 3

module 3

Model Risk Management for Machine Learning Models - Part 2

The Decalogue: Ten things to think about when developing your model governance framework when integrating Machine Learning models (cont’d):
6. Integrating Data Governance and Model Governance
7. Development Models vs Production Models
8. Fairness, Reproducibility, Auditability, Explainability, Interpretability & Bias
a. How do we objectively measure these?
b. Review of the Apple-Goldman Sachs credit card debacle
9. Machine Learning options and considerations
a. AutoML (Data Robot, H20.ai, etc.), ML as a service (Google, Comprehend, Watson) and home-cooked custom models
10. ML and Governance: Roles and Responsibilities redefined

Managing models in the day of Covid19

- Perspectives on point-forecasts, validation and fat-tails!

Week 4

module 4

Pragmatic Model Risk Management for AI/ML models

Challenges and best practices for pragmatic model management within the enterprise
Working with open source projects
Working with vendor models and machine learning APIs
Quantifying model risk for machine learning models
Model risk management for deep-learning models
Validation criteria and best practices
Templates for Model Validation for machine learning models

Synthetic data for Model Risk Management
Use of Synthetic datasets

WEEK 5

Module 5

Hands-on Case study

Validating a Credit-risk machine learning model

A case study illustrating a model validation of a credit risk model involving machine learning
Working with Regression, Neural Networks, and Random Forest models
Development models vs Production models
Sample templates and worksheets will be provided
Roadmap for the MRM team to upskill and keep abreast of changes in the AI and ML landscape
Training, education, and expectation setting
Future outlook: Regulation, Sandboxes, Frameworks
Review of recent regulatory efforts
How should companies proactively plan for changes and the future?

Week 6

module 6

Guided Exercise, Part 1: Scoping and design

Put your newly learned skills to practice while being mentored through the process. Participants will go through a guided exercise to perform model validation on a machine learning model of their choice. Guidance will be provided in scoping and implementing the project.

Week 7

module 7

Guided Exercise, part 2: Demonstrate your skills

Participants will have the opportunity to share their findings

Course instructor:

Sri Krishnamurthy, CFA

Chief Data Scientist, QuantUniversity

Sri Krishnamurthy is the founder of QuantUniversity, a data and Quantitative Analysis Company and the creator of the Analytics Certificate program and Fintech Certificate program. Sri has more than two decades of experience in analytics, quantitative analysis, statistical modeling and designing large-scale applications.

Prior to starting QuantUniversity, Sri has worked at Citigroup, Endeca, MathWorks and with more than 25 customers in the financial services and energy industries. He has trained more than 1000 students in quantitative methods, analytics and big data in the industry and at Babson College, Northeastern University and Hult International Business School.

Sri earned an MS in Computer Systems Engineering and another MS in Computer Science, both from Northeastern University and an MBA with a focus on Investments from Babson College.

Who should attend?

- Risk Professionals

- Model Validators

- Model Auditors

- Data Scientists

- ML Professionals

- ML Ops and Software professionals

Preview the class here

Learn how to evaluate risk in machine models

This 6-module program geared towards practitioners!

Delivered by QuantUniversity in partnership with PRMIA

LIVE/ONLINE/ONDEMAND

Learn how to evaluate risk in machine models

This 6-module program geared towards practitioners!

Delivered by QuantUniversity in partnership with PRMIA

LIVE/ONLINE/ONDEMAND

Attendees

hosted by

QuantUniversity

QuantUniversity (www.quantuniversity.com) is a quantitative analytics and machine learning advisory based in Boston, Massachusetts. QuantUniversity runs various data science and machine learning workshops in Boston, New York, Chicago, San Francisco and online. The company offers an Analytics Certificate Program and the Fintech Certificate program along with multiple workshops in its Explore-Experience-Excel series. Contact us at info@qusandbox.com

FIND OUT MORE

Supported by

PRMIA

The Professional Risk Managers International Association (PRMIA) is a professional organization focused on the "promotion of sound risk management standards and practices globally", and "the integration of practice and theory".It provides certification and credentialing for professional risk managers, as well as other educational programs and resources.

FIND OUT MORE

Stephen D. Cutler Center for Investments and Finance

The Stephen D. Cutler Center for Investments and Finance is provides programs and cutting-edge resources that enrich the student learning experience, support faculty research, and engage our alumni community. We’re committed to furthering Babson’s innovative and practical approach to finance education and enabling industry practitioners, faculty, and students to collaborate and learn from one another.

FIND OUT MORE

Thank You to Our Supporters

Past Attendees

Past Attendees of QuantUniversity workshops include Assette, Baruch College, Bentley College, Bloomberg, BNY Mellon, Boston University, Datacamp, Fidelity, Ford, Goldman Sachs, IBM, J.P. Morgan Chase, MathWorks, Matrix IFS, MIT Lincoln Labs, Morgan Stanley, Nataxis Global, Northeastern University, NYU, Pan Agora, Philips Health, Stevens Institute, T.D. Securities and many more..

We are entering a time where the speed at which decisions are made is critical. Ensuring your models work with comprehensive testing and evaluation is key for successful ML projects