<b>Review of Data Engineering Frameworks (Trino and Kubernetes) for Implementing Generative AI in Financial Risk</b>
This paper discusses a comprehensive Data Engineering Full Stack Framework for implementing Generative AI (GenAI) in financial risk management. We examine the challenges, propose solutions, and explore the application of various data tools directly relevant to the integration of GenAI in financial risk. In particular, we highlight how GenAI is transforming financial risk management by utilizing deep learning models to generate synthetic data for stress testing, anomaly detection, and risk modeling. We explore full-stack big data frameworks such as Apache Spark, Trino, Kubernetes, AWS SageMaker, and Databricks for scaling GenAI model deployment. Additionally, the paper provides a literature review on AI and GenAI applications in Trino, emphasizing query performance, optimization techniques, and AI-driven analytics. We investigate the bidirectional synergy between Trino and GenAI—how AI enhances Trino’s capabilities and how Trino facilitates advanced AI workloads. Key contributions include insights into Trino's scalability, AI-powered query execution, and the trade-offs between generative models and computational efficiency. Furthermore, we discuss deployment strategies leveraging AWS, Kubernetes, and Apache Spark for scalable risk computation. This paper also reviews the latest advancements in Azure-based AI agents and Kubernetes orchestration for AI/ML workloads, categorizing developments in agentic workflows, AI/ML orchestration, and AI-driven DevOps. Most of the work we have cited is very recent and less than 12 months old.
Publication Details
- Journal
- Figshare
- Published
- 2025-01-01
- DOI
- https://doi.org/10.6084/m9.figshare.28378547.v1
- Primary Topic
- Financial Distress and Bankruptcy Prediction
- Type
- article
- Field-Weighted Citation Impact
- 0.00