Classification of Bid Notice Sections using BERT, DistilBERT and ModernBERT

This work describes the process of training a classifier specialised in classifying document segments related to the procurement of goods and/or services by the Brazilian government. Correctly identifying the sections of these types of documents is an essential task for the auditing process of government contracts and is consequently a necessary step for automated strategies to prevent and combat fraud. The classifier provides the initial input for automated auditing processes, aiming to identify problems that could harm the public administration. As part of the chosen approach, a database was compiled consisting of the characteristic text of the document sections existing in government procurement procedures, annotated with a label characteristic of the respective text. 13 labels discriminating essential sections of the documents were selected for training the classifier. Three models were evaluated, including the original version of BERT. DistilBERT was chosen because it is a compact, lightweight model that prioritises efficiency. At the other extreme, ModernBERT was evaluated, an approach that uses up-to-date training methodologies and allows long sequences to be used as model input. Finally, the results comparing the models are presented.