Data Lake Engineering Strategies for Financial Risk Monitoring Systems

In a data-driven world, financial institutions face relentless external pressure to constantly innovate their product offerings. This need for innovation has always been at odds with enterprise risk management. However, recent events in the financial markets indicate that there is more than just a desire for innovation that pressures banks into rushing these changes. The new interest rate regime has made legacy products unprofitable, increased the demand for new products with new features that are difficult to replicate with older pricing models, and has provided an incentive to experiment with these new products, exposing financial institutions to risks that should be monitored on a continuous basis.

Configuring a set of collaborative systems that allow quantitative finance teams to monitor financial risk on a continuous basis, detect breaches when they occur, and backtest these models against historical data empowers improved risk decision making and rapid product innovation. A new design philosophy that treats the enterprise data ecosystem as a data lake in which data ingestion is engineered as a first-class citizen, combined with an emphasis on metadata, enables the implementation of a financial risk monitoring system at the scale of a Tier 1 investment bank.