Architecture of the Software for Updating Physical Knowledge Using a Combination of ClickHouse and HDFS

The analysis of the global patent database, journals on physical subjects, dissertations involves processing large amounts of text and graphic data, working with various sources and formats. Thus, the task of organizing effective accumulation, storage and access to large volumes of text and graphic data is urgent. The article describes the development of the concept and architecture of an automated system for updating physical knowledge for information support of search design using a centralized data warehouse and Apache Hadoop/Spark components. The concept and architecture of storing a unified knowledge base in the field of physics has been formed, using the associated storage of bibliographic information in the ClickHouse DBMS, large blocks of data (text fields of patents, journal articles, images) in the HDFS distributed file system. The approbation was carried out using the example of sampling based on US patent documents (USPTO). Verification of the correctness of the patent sample formation was carried out by comparing the results of clustering of textual patent information with the belonging of patents to IPC classes.