Automating Document Migration and File Management

Manufacturers are required by law to keep all documents related to their product sales, including materials components and line generation, for many years. Our manufacturing client used an older electronic document management system which was no longer supported. This system was not searchable as multiple documents were filed together in one scan.

Our client needed to correct this filing issue and set up a new management system which would correctly tag documents so they could be searched. SphereGen was able to build an RPA automation which handled the separation and migration of documents to SharePoint, as well as establish a front end to their SharePoint system which allows new documents to be indexed correctly as they are filed. The automation cut months off their document migration timeline and reduced new document uploading by 7-8 minutes per document.

Challenge​

Our client faced two challenges. Their document management system was not searchable as multiple documents were combined into one scan. As a result, they needed to migrate all documents to a new document management system. This meant breaking apart the scans into multiple documents and tagging them for search. In addition, they needed an easier way to load and tag new documents to their management system.

Solution​

Manually breaking scans into individual documents for upload to a new management system was incredibly time consuming. SphereGen created an automation which read index markers on the old system, used those markers to identify a specific document, and then created a pdf of the individual document. The pdf is catalogued and loaded to SharePoint. Any new documents which must be filed are indexed and loaded into the proper SharePoint folder using a front-end application which incorporates Intelligent Document Processing to create searchable tags based on document data.

In order to migrate and tag individual documents correctly, large .tiff and .pdf scans had to be broken into multiple pieces for each document. Trying to accomplish this manually presented an overwhelming task with a never-ending timeline.

Instead, the automated migration process was built to create new SharePoint folders, identify .tiff or .pdf documents in the old system and create new pdfs, that could be mapped and uploaded to the correct SharePoint folder creating searchable tags for each document.

After the migration was complete, our client still needed an efficient way to load and tag new documents so they could also be filed in the appropriate folder.

APPROACH

Building SharePoint Folder System

Before migration took place, the SharePoint Folder system was created. Using a Rules index which contains information for document types and mappable columns, an automation ran which created all necessary SharePoint folders according to the established rules. These rules included document name and all tagging information which needed to be assigned to a document that would reside in that folder.

This rules index is used throughout all other processes.

Migration

After the SharePoint folders were created, the migration automation handled the document conversion and upload to SharePoint.

The migration started by retrieving documents from the old system according to the indexes. These indexes contain the path of the document, the document type and what pages it resides on. The pages were an important piece of information as a document could be shared across many folders, with pages located in different folders. The migration bot used the document path and pages to read the old .tiff and .pdf files and select all pages of the document for conversion to a new pdf.

To correctly match the file to a SharePoint folder, the bot used the indexing information and the SharePoint rules to retrieve the appropriate folder name based on the document type. Upon a successful match, the file is correctly tagged according to the rules and uploaded to SharePoint.

The information for every transaction is written to a log. Any unsuccessful matches or files which could not be processed are recorded in the log with all pertinent information regarding the error and how to locate the file which did not process.

The migration was able to run 24/7 to migrate all files. The bots were monitored to make sure they continued to actively run.

New Document Management

To assist with new document filing, a front-end indexing automation was written. New pdfs are uploaded to a folder based on document type. The bot reads the folder, picks up new documents and loads them to the document management system using the rules-based logic from the migration. During this process, the bot uses Intelligent Document Processing to read the pdf and find the necessary field values required to assign searchable tags.

This required the training and testing of the bots to ensure that documents could be identified correctly.

Reduced Processing Time

The process of manually loading documents into the old document management system required 7-8 minutes per document. With the new automation, documents can be uploaded and correctly indexed and filed in less than 1 minute, improving processing time by 90%.