Sheffield Digital Justice: Finding AI Solutions for Law Firms

Anish Bailin-Saunders, Shinidine Ceraphin, Jessica Hell, Rubiyah Mubashir, Emma White, Hannah Wilson and Alice Xie

Off

Introduction

Digital technologies and Generative Artificial Intelligence (‘genAI’) are transforming professional industries, including the legal sector. Law firms are increasingly exploring AI-driven solutions for legal research, drafting, and administrative work. However, questions remain regarding the reliability, regulation, and integration of such technologies within legal practice.

The extra-curricular research programme ‘Sheffield Digital Justice: Finding AI Solutions for Law Firms’ aimed to bridge the gap between the technological and legal worlds. Led by the School of Law at the University of Sheffield and in collaboration with Sheffield-based tech company etiCloud, the project brought students from law, criminology, and computer science together to explore the potential applications of AI within the legal sector. The project aims were:

  • To investigate what legal professions in local law firms want from AI and try to develop AI solutions to meet those needs. 
  • To bridge the gap between technological advancement and the legal profession.

Through interviews with legal professionals, the programme identified key areas where AI tools could support legal practice. These findings then informed the testing and evaluation of a range of genAI tools. Overall, this project generated insights into the opportunities and limitations of AI within legal practice.   

Interviews with Legal Professionals 

Four semi-structured qualitative interviews were conducted with legal professionals to explore their understanding, use and concerns regarding AI, particularly genAI. Interviewees were contacted through personal and professional networks. They were provided with information sheets and consent forms prior to participation, ensuring informed engagement and confidentiality. Accordingly, all participant details have been anonymised within this report. Despite the small sample size, interviewees represented a range of roles across different law firms, offering diverse perspectives on technology within the legal sector. 

The interview question template was informed by the project aims. The questions were targeted in order to get the most useful responses. This also meant that there was consistency in the research. Questions focused on participants’ roles within their firms, organisational attitudes towards digital technology, and opportunities as well as challenges associated with genAI. Following the interviews, volunteers conducted a thematic analysis of the findings. Key themes and patterns were identified and then discussed within the group and during a meeting with etiCloud.

The findings suggested a generally positive attitude towards digital technology and a good level of awareness of potential genAI applications, such as legal drafting. However, there was some uncertainty around the full potential of how such applications could be used to enhance legal work. Participants expressed a number of concerns regarding genAI, particularly accountability and ‘hallucinations’, where AI systems generate misleading or false information. This issue was recently highlighted when the US law firm Sullivan and Cromwell formally apologised over hallucinations in a court document relating to high profile proceedings.

Other key concerns included data protection and security, confidentiality, and client discomfort. Due to these concerns, participants explained that they verified the work of AI, limiting any time-saving benefits.. 

Issues also arose regarding technological integration with current tech systems already in use This explained the seeming popularity of Microsoft Co-Pilot as the genAI tool most frequently used, given it is already integrated within firms’ systems. Several participants referred to not wanting to be dependent on the tools when producing work for a client. More junior participants also raised concerns around how best to develop their own skill sets if genAI was increasingly relied upon. 

While participants voiced various worries about genAI use within legal research and client interactions, most interviewees identified strong potential for genAI within administrative tasks, horizon scanning tasks and the drafting of standard legal documents. While they were reluctant to rely upon it as a research tool, although acknowledged it could be used to identify missing information.

Analysis of AI Tools 

Based on this identified potential for using AI in legal drafting, volunteers analysed existing AI tools and their applications within this area. This included looking at specific legal drafting tools on considering key features of the tool / pricing / other useful information about the tool. The purpose of this was to discover what was already out in the market and whether there was a large range of tools available. 

Publicly available genAI chatbots, such as ChatGPT, Google Gemini, Claude, and Character AI, were then tested, using two example legal drafting scenarios. Additionally, specialised legal AI platforms were compared based on their advertised capabilities, features, and pricing.

The analysis revealed that publicly available GenAi chatbots were highly user-friendly and capable of creating easily understandable outcomes within seconds. Some tools, such as Claude, even asked follow-up questions, demonstrating more advanced capabilities. Despite varying accuracy and quality, most tools generated inconsistencies in formatting and accuracy. Thus, a process of verification remained necessary. Moreover, many of these platforms failed to address concerns relating to data protection and confidentiality. These findings suggest that publicly available genAI chatbots are currently not reliable enough for unsupervised legal work.

Some key findings from testing publicly-available genAI chatbots

  • It is fairly easy to use and sign up to the AI tools  
  • They can generate easy to understand results results within seconds / minutes 
  • You have to ask follow up questions to get sufficiently comprehensive outputs. 
  • There are questions over the accuracy of the outputs generated. There are fairly high levels of accuracy but there are some discrepancies which make the outputs difficult to rely on. 
  • The outputs not always given in the correct formatting to be used under the law of England and Wales 
  • There are issues over data protection and security, with potential uncertainty over whether information entered into chatbots will be stored and used to train the platform.

As part of the analysis, volunteers also experimented with creating legal drafting assistants using chatbots. While not all chatbots were able to create such assistants, created assistants produced more adaptable and tailored results. So, public AI systems may be more valuable when designed for specific legal tasks.

Rather than relying on these publicly available AI tools, large commercial law firms are increasingly adopting specialised legal AI platforms with features such as contract drafting and legal research. Although these tools could not be directly tested due to subscription requirements, their promised features and pricing were compared. Pricing ranged from approximately £200 to over £1,200 per user per month, with many providers not offering any pricing information. This revealed effective, but expensive options with accessibility barriers for smaller firms.

Finally, volunteers were introduced to Qanooni, a highly tailored AI drafting, reviewing, and legal research tool. The platform is directly integrated into Microsoft Office applications and emphasises the secure handling of data. Contrary to most AI tools, Qanooni is trained using the individual firm’s internal data, allowing outcomes to reflect the firm’s tone and style. The platform seeks to address ‘hallucinations’, by providing certified sources and verification mechanisms.

The analysis of all these different AI tools revealed the growing potential and the current limitations of AI within legal practice.

The wider importance of the project

The project has engaged directly with one of the most current and rapidly evolving trends in the legal sector: the integration of AI into legal practice. Overall, there is great potential for extending the use of AI in law firms and the legal sector more generally. However there are still major challenges to be overcome, which was recognised and acknowledged by interviewees.

This trend has been seen by the pattern of larger firms integrating it into their workflows and beginning innovation themselves, for example:

This demonstrates that there are a range of different ways in which AI is already currently utilised by firms. There is also potential for smaller firms to develop its usage to improve efficiency.  However, at the same time, there are practical barriers to adoption, for example, where the tools are not currently good enough, not available or too expensive. There is also a concern around over-reliance on such tools. 

Individual legal professionals hold a range of different attitudes towards AI. They may be keen and open to learning about how AI can be used to enhance the precision of their work and improve overall efficiency. However, there may also be challenges in understanding AI and effectively leveraging it to enhance work, necessitating additional training in this area.

Conclusion and Learning Experience 

In conclusion, this project demonstrated both the growing use of AI within legal practice and the barriers limiting its expansion. While interviewees identified benefits of AI for administrative work and document drafting, concerns regarding data protection, accountability, and hallucinations remain relevant. A comparison of publicly available and specialised AI tools also highlighted issues of cost and accessibility.

This suggests that AI should be seen as a tool to support legal experts rather than as a replacement for them. Even though AI can increase productivity by expediting processes like document review, case summary, document review, legal research, human judgement is still necessary to guarantee accuracy and the privacy of clients. 

Beyond the research findings themselves, the programme provided valuable practical and academic experience to volunteers. Conducting interviews and analysing AI tools developed participants’ communicative, analytical, and collaborative skills. For example, interviewing required active listening and adaptability/flexibility, as it was essential to adapt and tailor follow up questions in interviews based on responses provided. It also necessitated engaging with individuals from diverse backgrounds and expressing views to them clearly and coherently.

The project also allowed the volunteers to develop a nuanced understanding of the topic, aiding them as they seek to begin applications in the legal sector by providing the ability to discuss real-world developments in a clear and informed way.

Furthermore, the project provided insights into the current and potential relationship between law and AI. It highlighted the gap between the potential of AI in law and its actual day-to-day use, allowing interviewees to reflect on how they could better incorporate AI into their legal work as well as signalling the key objectives for the improvement of legal AI platforms going forward. It highlighted the increasing importance of technological literacy within the legal profession and the need for interdisciplinary collaboration to address emerging challenges of AI adoption.