1 Perform Preliminary Training
RECENSEO Predictive Coding begins with a few hundred documents from the review population. Sample selection can be either random or judgmental (based on subject matter expert search results), or a combination of the two. This preliminary training allows the Subject Matter Expert (SME) to get acclimated to the review set before reviewing the foundational Control Batch, and perhaps more importantly, it allows for training to begin as soon as a substantive portion of the review population is in the review set.
2 Review the Control Batch
The Control Batch is a critical part of the training process. It is typically 500 to 1,000 documents. After the SME has reviewed the documents in the Control Batch, RECENSEO evaluates performance against it, throughout the training process. Completing the Control Batch at this stage in the process enables the iControl ESI Analytics Team to add value by:
Providing Initial Review Estimates – Estimated responsive documents in the review population and the estimated total number of documents the review team will need to put eyes on to complete the review
Tracking Training Progress – Performance against the Control Batch enables the iControl ESI team to estimate when continued training stops, adding significant value to the end result
3 Conduct Subject Matter Expert Training
With the Control Batch complete, SME training continues through a series of small batches. While judgmental and random samples are possible, generally RECENSEO generates training samples using Active Learning (Uncertainty Sampling). Essentially, this means RECENSEO focuses on the documents that the machine learning is least certain about. By reviewing these documents, the SME helps RECENSEO continuously adjust the line between responsive and non-responsive documents.
4 Perform Process Validation
While the Control Batch is extremely useful in making estimates and monitoring the completeness of the training process, it does not sufficiently demonstrate the effectiveness of the trained predictive coding engine. An effective validation process typically involves SME review of another 1,000 to 1,500 documents for comparison against the Predictive Coding engine’s responsiveness decisions. The iControl ESI Analytics Team provides detailed reports on the Quality Control Batches and insight to evaluate the results.
5 Complete Review & Production
Strictly speaking, attorney review of the predicted responsive documents is an optional exercise. That said, in most cases, some level of attorney review is still standard on most of our projects. This may be a review of potentially privileged documents or a full pre-production review of all predicted responsive documents. As mentioned above, review and production can be held until the Predictive Coding process is complete, or run in parallel, depending entirely on the legal team’s preference.