Cloud vs. Local:A Study on Practical LLMBasedDocument Classification in Infosec

Information security (infosec) is an ever more pressing issue for companies and institutions to managewith the expanding digital landscape. Coinciding with this are the large improvements to LLM researchand development makes the implementation of a system which utilizes it attractive for companies. Animportant aspect within infosec is document classification. With the rapid developments of LLMmodels that can be run locally and different services offered by distributors which gives differentapproaches on how to implement and develop an LLM system. The aim of this paper is to develop andevaluate two different practical approaches of a LLM system that assesses and classifies documents asthey relate to an internal classification policy. This is to illuminate the question of whether selecting alarger model via pay-to-use services offers better classification performance and capabilities ratherthan a smaller model hosted locally on readily available hardware. This paper considers and takesinspiration from similar approaches to LLM classification but changes the approach to reducedevelopment time of fine-tunes and instead utilizes a hierarchical system of descriptive explanation ofeach category of classification label. The results show that a small model can give accurate, consistent,and qualitative classifications just as well or for some documents better than a larger model, in thiscase Microsoft's Copilot. It provides an alternative to external providers who offer a pay-to-use costfor their model, due the solution provided which utilizes a smaller model makes it is easily and costeffective to scale up.