SwishFormer for robust firmness and ripeness recognition of fruits using visual tactile imagery

The accurate assessment of fruit ripeness is a critical task in the agricultural industry. It affects the fruit quality, shelf-life, and consumer satisfaction. Traditional methods for estimating fruit ripeness rely on subjective human judgment and invasive sampling techniques, which are both infeasible and time-consuming. This paper presents a novel method for estimating firmness and ripeness of fruits using their palpation motion encoded within the visual tactile scans. Moreover, these tactile scans are passed to the proposed SwishFormer model coupled with Random Forest head to predict the fruits firmness, which is later used in classifying the fruits ripeness stage. SwishFormer, unlike the existing state-of-the-art models, encompasses hardswish activation as a token mixer which allows it to generate distinctive set of features from the candidate tactile scans. These rich feature representations are then fed to the Random Forest regressor to robustly estimate the fruit firmness values and the estimated firmness values are then used in accurately predicting the ripeness level of the fruits. Apart from this, SwishFormer is extensively evaluated on the proposed dataset, containing the palpation visual tactile scans, and it outperforms state-of-the-art works by achieving 4.77%, 4.09%, 13.69%, and 4.65% better performance in terms of MSE, RMSE, R 2 , and MAE scores, while possessing 2.02 times less parameters, and 2.09 times lesser GMACs. Additionally, the ripeness recognition performance of the proposed system is thoroughly tested through real-world experiments using a Stretch Robot, where it achieves a success rate of 96.6%, 98.3%, and 93.3% for recognizing avocados as underripe, ripe, and overripe, respectively. To the best of our knowledge, this paper introduces a first non-destructive approach to estimate fruit firmness and ripeness using off-the-shelf vision-based tactile information. Moreover, the proposed dataset and source code of this work is available at https://mashood3624.github.io/SwishFormer/ . • SwishFormer: A VBTS-based architecture for non-destructive fruit firmness estimation. • SwishFormer is 2.02x smaller, 2.09x faster (GMACs), and outperforms prior methods. • Deployed on a Stretch Robot, achieving 96.1% success in avocado ripeness classification. • SwishFormer improves firmness prediction with 4.77% better MSE, 4.09% RMSE, and 13.69% R2. • Proposed a dataset with VBTS scans of avocados and kiwis for firmness estimation.