Exploring Transparency and AI Assessment in LLM-Assisted Research Applications

In this work, we further investigate the utility of using a large language model as a research assistant to identify research grant funding opportunities that are best suited for a user-defined natural language set of capabilities. The use case is a United States Department of Defense Small Business Innovation Research Broad Agency Announcement. To explore principles of responsible/ethical artificial intelligence and accountability in the context of large language model-driven applications, we perform clustering on embeddings and apply a suite of metrics to compare cluster quality against Latent Semantic Indexing. Further, we use visualization techniques to depict the contents of funding opportunities that lie at the intersection of multiple capabilities. Finally, we show the importance of maintaining the human in the loop for vetted data quality.