A Flexible Workspace: Using the Cloud to Stage & Ingest New Content

At CDL we work with librarians and archivists from across the ten University of California campuses. More specifically, as the team that maintains and augments our digital preservation repository, Merritt, we help depositors with their digital preservation projects. During recent collaborations, our team identified a need to be able to assist partner libraries with limited resources to stage their content ahead of ingest into the repository. This content is typically that which was generated as a result of work with a digitization vendor and resides on HDDs or other transfer devices. To facilitate staging, validation and ingest of HDD content, we created a workspace that incorporates AWS S3, while simultaneously incorporating shell scripts and other resources to manipulate content once it is uploaded to the cloud. To this end, we implemented Amazon FSx for Lustre as a bridge between an EC2 instance and S3. The system also incorporates a Lambda function for submission automation purposes, as well as configuration management via Sceptre. This workspace has allowed us to streamline the staging and ingest of newly digitized content while working with our library partners, via a collaborative solution that minimizes impact to resources of all teams involved.