Discovery-led refinement in e-discovery investigations: sensemaking, cognitive ergonomics and system design

Abstract

Given the very large numbers of documents involved in e-discovery investigations, lawyers face a considerable challenge of collaborative sensemaking. We report findings from three exploratory workplace studies which looked at different aspects of how this challenge was met. From a sociotechnical perspective, the studies aimed to understand how investigators collectively and individually worked with information to support sensemaking and decision making. Here, we focus on discovery-led refinement; specifically, how engaging with the materials of the investigations led to discoveries that supported refinement of the problems and new strategies for addressing them. These refinements were essential for tractability. We begin with observations which showing how new lines of enquiry were recursively embedded. We then analyse the conceptual structure of a line of enquiry and consider how reflecting this in ediscovery support systems might support scalability and group collaboration. We then focus on the individual activity of manual document review where refinement corresponded with the inductive identification of classes of irrelevant and relevant documents within a collection. Our observations point to the effects of priming on dealing with these efficiently and to issues of cognitive ergonomics at the human-computer interface. We use these observations to consider how visualisations might enable reviewers to deal with such refinements more efficiently.

Afterword: data, knowledge, and e-discovery

“…Statistical, data-driven methods are already finding great success, but I believe both Empiricism and Rationalism have large roles to play. In this special issue, both Conrad (2010) and Ashley and Bridewell (2010) reach not dissimilar conclusions, and at least two of the systems reported can be said to combine statistical and knowledge-based approaches (Attfield and Blandford 2010;Hogan et al 2010). E-discovery problems are likely to attract attention from many quarters of the research community, particularly if more realistic data sets can be made available.…”

“…Supervised learning methods are already in use by many e-discovery software vendors and service providers (Kershaw and Howie 2010). Several systems using machine learning are discussed in this special issue (Attfield and Blandford 2010;Hogan et al 2010;Privault et al 2010), and many have been reported on at the TREC and DESI meetings.…”

Collaborative sense-making during simulated Intelligence Analysis Exercises

“…While the Intelligence Cycle might begin with 'Direction', this only gives a high-level sense of what the analyst might be looking for. As 'Collection' and 'Processing' progresses, new problem opportunities arise through 'discovery-led refinement' (Attfield and Blandford, 2010). Thus, one could read figure 1 in terms of a 'Direction' providing a tightly specified frame (so that the analyst will only collect and process data which are directly relevant to this frame), or in terms of a familiar problem (so the frame could be based on previous experience of similar cases), or in terms of a problem opportunity (so combinations of data would suggest particular frames which could be expanded and explained).…”

Conceptual Design for Sensemaking

“…Since lawyers expend considerable energy in filtering, selecting and structuring otherwise unstructured evidence collections (such as emails) into temporally ordered narratives, they would benefit from visualisations that can automatically perform that kind of visual structuring for them. This was the rationale behind a representation called ThreadsVI [2] which aimed to associate emails in a meaningful way by representing them visually in the context of their discussion threads and in terms of senders and receiver such that the user could gain informal visual impressions of levels of activity (similar to centrality) within a semantically filtered social network over time (Figure 4).…”