Deep Web Search: Design and Implementation

This article provides a novel practical approach for designing and implementing Deep Web Search applications, including search engine and search results aggregation and delivery system. In the beginning we present the specifics of Deep Web Data Sources and how the desired information is searched for and delivered. Not all Websites are designed and implemented the same: some of them provide free access to information and others implement various access restrictions and data protection mechanisms, leading to what we know as Surface Web and Deep Web, respectively. Deep Web Search, or DWS is topic-oriented and we expect a specific approach for each topic. However, there are principles and common design and implementation guidelines we present in this paper that apply to a broad range of DWS applications. Our goal is to help developers build such applications, also known as Data-as-a-Service, or DaaS. We’ll focus on a novel practical approach of designing the architecture of a DWS engine and the search strategy for mining multiple, autonomous and heterogeneous Data Sources. The extracted information is further exposed to different level of automatic interpretation according to application requirements. Finally, the search results are aggregated, consolidated and delivered in a customizable format based on user-defined templates. The guidelines outlined in this work have been applied to design and implement a DWS application, Advanced Title Search, for Title Search Industry in US. The guidelines presented can easily be applied to other DWS applications specific to other topics.