The High Cost of Quality in Legal Apps

We seem to be awash with law-oriented apps. And we clearly could use even more of them. So this may not be the best time to raise awkward questions about their quality. But inconvenient truths deserve attention.

(I’m using ‘apps’ here as shorthand for software applications of all kinds, not just those built for the native operating systems on smart phones. You might also think of them as ‘appliances.’)

A recently published American Bar Association study identified more than 320 digital tools that help people act on a legal problem. Despite the impressive number, that list only skims the surface. I know of one service, not explicitly called out, that alone serves up several thousand highly interactive modules.

Also, most of the tools in the ABA study are aimed at consumers. The broader ecosystem is considerably more expansive. It includes highly specialized tools for experienced professionals, and document assemblers for startups proffered by law firms as loss-leaders. Some are aimed at small business users. See e.g. https://fitsmallbusiness.com/reviews/legal-service-reviews. And in a movement sometimes referred to as Apps 4 Justice law students around the world are creating software to help people deal with legal issues. (My own latest course in that spirit is described here.)

These tools boast a broad spectrum of competencies. They can ask probing questions, perform analyses, render customized guidance, generate documents, and interact with other parties on your behalf. Doing any of these things requires some level of codified expertise. Even basic rules in a law office’s calendaring or matter management system can reflect significant forms of knowledge expression.

Since I can’t do justice to a representative sample of actual applications in this piece, I won’t mention any by name. Plus that would divert attention from the many features they share, and the common challenges they pose. Which are considerable.

At a high level, all of these systems use some combination of texts, graphics, images, video, and sounds to (1) get information (by asking questions, noticing behavior, and accessing previously stored data), (2) give information (inform, advise, suggest, warn), (3) deliver artifacts (documents and other kinds of files), and (4) do other things (send emails, update databases).

In the course of all this they almost always communicate purported knowledge about law and the legal system, both explicitly and implicitly.

The Many Faces of Quality

There are many qualities we care about in legal apps. They should be reasonably priced, if not free. They should be easy to access and to use. They should be unbiased and respect user privacy. Most importantly, they should be effective in helping users deal with the legal problems and opportunities they face. (In that respect it is encouraging to see renewed attention to outcome measurement in the field.)

One key determinant of effectiveness is correctness. That is, an app should properly reflect the rules, forms, and considerations that apply to the situation at hand. The legal substance should be solid.

Of course an app that is totally correct, but unusable, doesn’t do anyone much good. A punctiliously accurate app with an awful UX, or one that disgorges material that an inexperienced consumer can’t effectively use, is of little value. Yet the opposite is also true: wonderfully designed apps that give wrong guidance or generate bad documents can be worse than worthless.

When we use an automated teller machine, or Microsoft Office, we expect it to work as promised, and maybe even behave semi-intelligently. We have a sense of its features and functions. We’re pleasantly surprised when it works and irritated when it doesn’t. But we don’t expect such devices to communicate knowledge about the world. Legal apps are different — they should work, but we also expect them to give valid guidance and generate appropriate documents.

The Costs of Correctness

All existing and potential legal applications entail considerable expense. It is costly to achieve high ‘substantive’ quality in such applications. It can also be quite expensive when an app doesn’t exhibit adequate quality. Even the absence of a needed application represents a cost.

Quality raises high costs however you look at it. The difference is mainly who bears the expense — the provider, the user, or broader society.

The lack of substantive quality in an app of course can be costly to those who rely on it to their detriment. Their finances, family relationships, and even their liberty can suffer.

And you might think of failure to create an app as a theoretical ‘opportunity cost,’ but the cost is real if a readily buildable tool that could produce more net benefit to humanity than it would cost to develop is not tackled.

It’s Complicated

Programs are complex functions, mapping combinations and sequences of inputs to outputs. They define exquisite dances of user and system statements. You can think of an app as a speech act apparatus. Both its development and its execution involve sophisticated communicative acts.

It’s not sufficient just to focus on the explicit ‘utterances’ of an app. People for instance make assumptions based on the absence of questions or documents. (“They didn’t ask whether my spouse is a foreign national, so I guess that must not matter.”)

Effective assistance can also require affective assistance — understanding and attending to the user’s emotions and attitudes, not just their circumstances and goals.

The many software platforms and process paradigms behind today’s apps offer myriad ways to deal with these issues. We just aren’t yet doing a great job in my view.

Especially in the nonprofit world, projects tend to be underfunded and understaffed. They are often driven by short-term grants that don’t stimulate long-term investments. The amount of work funded can be grossly inadequate to the task of doing a thorough job. Perfect applications, though, can require an infinity of time and money, which is not realistic in the present budget climate.

No Escape

So, we have to deal with substantive quality one way or another. We have choices, but can’t avoid making them.

=> When a potential app would produce more benefit than the cost of fielding it, there’s a real cost to not developing it. We can strive to get the necessary resources to build and deliver it. Or we can deprive the world of it entirely, or do something less than its full scope. (The Florida Justice Technology Center suggests that we apply the famous Drake Equation — a formula for calculating the number of intelligent civilizations in our galaxy — to this kind of determination in the access-to-justice context.)

=> Once building an app, we can choose to make it a high-quality one. Or we can release it with inadequate or unknown quality and accept the risks of serious harms that may cause.

Compromises and tradeoffs are inevitable. The trick is to choose wisely.

In recent months there’s been heated debate about an ABA resolution proposed by the New York State BarAssociation (since withdrawn) that would call upon app providers to forgo warranty disclaimers and otherwise follow a set of prescribed practices. Legislation to that effect is already in place in North Carolina, and similar measures are under consideration in Washington State. (My own view has long been that legal software, like other works of authorship, deserves robust protection under the First Amendment, and that suppressing it is not only bad social policy, but offensive to constitutional values. See Liberty, Justice, and Legal Automata and Are We Free to Code the Law?) ‘Best practices’ compiled by bar associations may be good for virtue-signaling, but not have much impact on actual practices, while raising suspicions about true motivations, as well as legitimate concerns about anti-competitive behavior.

In any event those of us in the business of building apps and teaching others about them need to up our game. Legal knowledge engineers should also be trustworthiness engineers. There are many existing tools and techniques we can employ more systematically. And plenty of new ones we can help catalyze. I and others are talking about this in more depth elsewhere, for instance in this recent webinar on Ensuring Quality in Online Legal Applications.

Perhaps we can look forward to quality audit tools for legal apps that we can harness for drive-by verification. X-ray-like devices that reveal what an app ‘thinks’ it ‘knows.’ You might call that radio-epistemology. In the meantime there’s a lot of plain old human work to be done.