Text Representations for Property Prediction of Organic Molecules Using RoBERTa

Recent work has shown that pre-trained large language models (LLMs) and transformers can be used in downstream prediction tasks in materials science and chemistry.Density is often a key property of organic molecules, for example in energetic materials.Transformers rely on text representations of inputs to encode and decode sequences and the choice of text representation may impact model performance on downstream tasks.Here, we pre-train RoBERTa using SMILES, SELFIES, and InCHI text representations on the PubChem10M dataset of organic molecules.We then perform downstream regression training to predict the density of organic molecules.Impact of the regression training dataset size is also examined.We evaluate performance on both a test set of 1000 molecules from the PubChem10M dataset as well as a dataset of energetic materials curated by Elton et al.We find that performance generally increases with increasing regression training dataset size, but that there is much room for improvement.SMILES generally performs the worst of the three descriptors while SELFIES and InCHI perform similarly.Areas of improvement are also discussed.