MathWorks recently hosted a webinar on Model Monitoring and Drift Detection, where Paul Peeling presented strategies for maintaining the health and fairness of deployed models using the MathWorks Modelscape solution. The webinar was also followed up with a Technical Article reinforcing some of the key points presented.

Whether you attended or missed the webinar, here’s your chance to catch up on some of the information shared. I’d encourage you to read the entire technical article or watch the webinar recording if you’d like to get more detailed information on this topic.

The Imperative of Model Monitoring

Effective model monitoring is not optional—it’s a necessity. With environments and data constantly changing, ML models must be monitored continuously to ensure their accuracy and effectiveness. The COVID-19 pandemic underscored the need for adaptable models, as pre-pandemic assumptions were upended, necessitating timely adjustments beyond periodic revalidations.

Advancing Model Integrity

Model integrity hinges on fairness and the ability to detect drift—changes in data patterns over time. Global regulations, including the Monetary Authority of Singapore’s FEAT principles and the European AI Act, underscore the importance of these measures. The Federal Reserve in the U.S. is also focusing on fairness in AI and ML models, as highlighted in an EY white paper on responsible AI.

The Role of Dashboards in Dynamic Model Risk Management (MRM)

Dashboards are pivotal in model risk management, offering real-time insights and acting as an early warning system. They enable the detection of data drift and help ensure models operate without bias. A centralized dashboard view aids in coordinating responses to issues across interconnected models.

The Importance of Timely Alerts

Automated alerts are essential for timely interventions, ensuring that any deviations are addressed promptly. These alerts must be embedded within the governance and development processes, allowing for a proactive approach to maintaining model integrity. Automated alert systems can notify relevant stakeholders for immediate action. These systems should facilitate the production and consumption of alerts, such as those indicating model bias or data drift.

Drift and Fairness Monitoring

Drift refers to the changes in data patterns over time, which can be sudden or gradual, affecting the model’s performance. Fairness, on the other hand, involves assessing the model’s decisions across different groups, potentially divided by protected attributes like gender or race, to ensure unbiased outcomes. Monitoring for drift and fairness is key to maintaining the trustworthiness of models. Active observation and adjustment for these factors can significantly improve model integrity and fairness.

Integrating DevOps in AI/ML Model Monitoring

DevOps plays a critical role in model monitoring, with a focus on operational aspects such as execution time and data throughput. Continuous monitoring and measurement of model properties over time are essential. Tools like Prometheus can be used to collect telemetry data, allowing for real-time queries about model performance. Ultimately, the observability of models—a concept
borrowed from DevOps—combines both instrumentation and telemetry, enabling DevOps teams to monitor models effectively and provide valuable feedback to business users and developers,
aligning with model governance and development protocols.

Modelscape: A Comprehensive Solution

MathWorks’ Modelscape provides an organization-wide overview of all models, integrating alerts and enabling the creation of dashboards for high-level insights and detailed analysis. It supports model refinement based on production data and informs governance decisions, enhancing risk management and operational efficiency.

Modelscape addresses the challenges of model fairness and data drift, offering features like threshold setting, alerts, and dashboard creation for performance review. It supports the entire modeling lifecycle, from validation in live environments to preproduction testing and deployment, ensuring the health and fairness of deployed models.

Learn more about how Modelscape can enhance your model monitoring and drift detection efforts at MathWorks Modelscape.