Addressing Modality Mismatch in Real Estate Recommendation Using Image Captioning

In this work, we consider the real estate recommendation system, which aims to recommend houses that match the specifications and interests of the user. While it is not difficult to retrieve houses that match the user's specifications (for example, number of rooms, location), showing houses that match the description provided by the user is complicated due to a modality mismatch. Typically, a house is represented in the form of specifications and images. An accurate textual description (like interiors and aesthetics) of the house is seldom available. On the other hand, the users provide their desired description of the house in the form of natural language and not images. Therefore, it is crucial to address this mismatch to provide relevant recommendations to the user. Moreover, the recommendations should be provided in real-time to the user, thereby requiring faster inference. In this work, we propose an improved recommendation pipeline that efficiently addresses the modality mismatch and provides faster recommendations. Through experimental studies, we demonstrate the superiority of our proposed algorithm.