The Application of Generative AI for Sketch-to-Image Conversion

Sketches have historically been the primary way to identify individuals when images are unavailable. Most commonly, this appears in a police investigation, where the police release a sketch to the public in an attempt to find a suspect. However, sketches occasionally do not provide much detail due to the limitations of both the witness and the artist. A poor sketch greatly hinders the public’s ability to identify someone, decreasing the chances they will be found. To improve upon this inefficient way of identifying individuals, we propose a new pipeline that leverages deep generative artificial intelligence models to create detailed images of them. Our approach is a two-step process: the initial formulation of an image and the iterative improvement of it. The pipeline first utilizes a sketch and a text description to generate an initial image. Then, the user can continually make changes to this image, modifying facial features, skin color, hair color, etc. until the image matches their mental picture. This enables both a stronger match between the image and the user’s memory as well as an increase in identifiability, as an image provides more details than a sketch. The pipeline was evaluated on the Illinois Department of Corrections (IDOC) dataset, which contains images and text descriptions of around 70,000 individuals. With our sketch-to-image model, we achieved a Fréchet inception distance (FID) score of 56.86, a distance metric between generated and real images. We additionally received overwhelmingly positive feedback during user testing. This work can be utilized by government agencies to make communities safer and search and rescue organizations to find lost individuals.