Interview with Denis Merigoux and Liane Huttner on the CATALA project

The creation of the CATALA language

Denis Merigoux: The starting point of our project is a category of computer programs which do what the law tells them to do, ie. their decisions are interpretations of legal dispositions. These programs are at work in numerous businesses and administrations where they take care of the application of regulations concerning pay, human resources, finances, or tax calculation for instance. Since the 1980’s, fiscal authorities have developed and implemented algorithms which centralize and simplify such calculations. Funds for family allowance, retirement and unemployment followed suit. Unlike statistics-based software, these programs are based on rules written by humans and encoded by them in computer languages. These rules are meant to capture some interpretation of a given law. Consider the case of income tax calculation: legal dispositions determine income brackets and applicable rates which are then calculated for each constituent by a program. When programs interpret legal dispositions, they are called critical programs. This means that the decisions they make have significant impacts on users. The problem is that there are bugs in these programs (ie. unexpected malfunctions or glitches as the program operates). Unfortunately, these bugs are fairly common: they have been noted in unemployment and pay, for instance. And, of course, these situations can have serious consequences for people who depend on these critical programs. Here, the interruption of social security benefits in particular comes to mind. Consequences of this sort are well-documented in the literature. But Liane and I noticed that very little attention was paid to the way in which these programs were made — little attention from scholars and businesses, but also very few means to make sure that the programs work accurately when implementing law. 

Liane Huttner: It must be pointed out that it is impossible to understand an algorithm out of its context since it results from several types of constraints, like production constraints, or technical and organizational constraints… Over the last five years, it became common in American scholarship to refer to algorithms as “socio-technical objects”. That is really what we had in mind when we created CATALA. We wanted to create a system to make sure that critical programs accurately implement laws and that they are in fact doing what they are meant to be doing. This is where the concept of fidelity comes in. In short, we started from the specific dysfunction of critical programs concerning adaptations to income tax calculations or to social security benefits for constituents, and from their lacking a verification system. With the creation of a computer language designed as a verifiable technical object, we aim to solve these problems. Our project includes both the development of the language itself, and of methodological requirements for software production. I insist on the fact that CATALA is a multidisciplinary project involving close collaboration between legal experts and computer scientists. More information is available on the project’s website: https://catala-lang.org/fr/.

Denis Merigoux: Laws involving calculations are transcribed into a computer language. But the selection of our computer language makes a difference in the way we encode law. For instance it is very difficult for computer code to correspond exactly to legal texts. There are technical reasons for this. But the core reasons, as we know since the 1980’s, have to do with the logical structure of legal texts. Sarah Lansky’s work has shown that the structure of legal texts belongs to a paradigm opposite to the one programming languages belong to. When we read laws — and especially these laws which involve calculations — we start from a basic, default case, which we then contrast with exceptions. Programmation, on the other hand, follows the reverse logical order: we start with specific cases, and then proceed to more general ones. Due to these structural differences, some elements are lost in the translation of legal texts into computer languages. 

Liane Huttner: There is another reason calling for a new programming language: the software must be as easy to understand as possible for those who write laws. It is complicated to teach Python to a lawyer. So we tried to find some common ground between the knowledge of legal experts and computer science, and to design a language which does not require formal programmation training to be used.

Denis Merigoux: I specialize in the study of programming languages and in coding. Liane is a lawyer specializing in digital law. We divided the work according to our expertise: I took care of the logical function of the CATALA language, while Liane was in charge of what we call the surface function, that is, to develop the language’s vocabulary into a form which could be used and read by lawyers. Our language is interoperable, which means that it can be translated in other computer languages, while being understandable by lawyers. The compiler acts as a translator, and allows us to translate CATALA into other languages.

A human intelligence project meeting the legal definition of AI systems

Denis Merigoux: When we are asked how AI is involved in our project, I like to answer that it depends on the definition of AI. Our project — in my understanding — rather falls under the definition of human intelligence. We write the code, and the program is the way we wrote it. We control it. We consider a number of situations before they happen. And it is because we imagine all possible scenarios and because we think ahead that we obtain a robust program which accurately translates the law.

Liane Huttner: It is true that, according to the definition of AI in the proposed legislation on AI, our project is not AI because it does not involve machine learning. We have no dataset on which to train our program. Whenever laws change, we change the program. Legally speaking, however, according to the definition put forward by the European Commission in its own proposed legislation on AI, our project qualifies as a kind of AI system. But I completely agree with Denis on the broader issue: the CATALA language is similar to human intelligence supported by good tools. What matters most in our project is methodology, which takes us closer to the work of philosopher Ivan Illich.

Steps in designing a dedicated proof-oriented language for critical programs

Denis Merigoux: There are of course several ways to think about the design of a dedicated proof-oriented language for critical programs. Our project had four steps.

• The very first step is to meet with the future users of our programming language, and to ask them what they want. This allows us to identify needs and required language functionalities. Based on this, we obtain technical specifications.
• Then comes a step in which we clarify the sometimes ambiguous needs of users to match them with the logical foundations which act as the basis of our language. These foundations can be an object-oriented programming language or a functional programming language. We can then do syntax highlighting or develop static analysis tools to assist programmers.
• The third step is language formalization. At this stage, we treat the program like a mathematical object and write a formal description of what a program in this language is and how it is executed. We then write a proof for a theorem showing that the language behaves approprietely: this is type safety. The language core will be consistent and calculate things in the way we want, and without bugs.
• We finally arrive at the compilation step. The compiler will be able to translate our language into previously known languages, and is also meant to meet the criteria established at the previous step.

It is especially important to remember that the conception of programming languages involves a lot of planning. It takes time. It is a social activity as well: we must keep in touch with the users during the entire process, because these are tough design choices and hard decisions to make.

The legal fidelity of CATALA algorithms

Liane Huttner: The CATALA language is a tool, and we can use it in many ways. The method we have developed guarantees fidelity to legal texts. It includes these requirements.

• The first is pair programming. A computer scientist and a legal expert work together, with the former doing programmation, and the latter reading and explaining the dispositions in a given legal text, and the entire writing process is as interactive as possible. Nothing is coded without the legal expert’s approval. In this way, we try to ensure that the program is loyal to the legal text.
• The method’s second requirement is literate programming. Concretely, we always write the CATALA version of a rule near or under the original version. For instance, if an article of law says that family benefits start with the second child, we write the code version underneath. This means that anybody can verify the translation of the legal text into CATALA.
• Finally, the CATALA language is open source: it is transparent and anyone can access it to suggest corrections or propose alternatives. The runtime system of CATALA is open source as well, which means it can be replicated outside of administrative instances. The code’s public availability makes our algorithm transparent a priori.

Denis Merigoux: I want to point out that while this method guarantees legal fidelity, it is not infallible. Coding in CATALA is not enough — following the method is necessary as well. We still obtain a guarantee of legal fidelity as high as 99%. To reach that, we use formal methods part of a branch of computer science which treats algorithms like mathematical objects about which we can reason and write theorems we can prove. With these methods, we can show that the program has high legal fidelity. These kinds of methods are notably used in the industry for critical computer systems like those used in nuclear power plants or aeronautics. Elements guaranteeing reliability have also been implemented. Furthermore, we discuss and develop our work in weekly online meetings with other research groups, comprising legal experts and computer scientists mainly from France and the USA — since our program is used to study the American tax code in academic environments. The code for the CATALA language is also hosted on the Github platform.

CATALA as a mouth of the law

Liane Huttner: Whenever the interpretation of a legal disposition or text is unclear, we signal it with an insert. We use it to explicate the interpretative difficulty with notes and by referring to a variety of sources. It is then up to users to decide what is the best interpretation in their opinion. We do not claim to have any interpretative authority: we are merely mouths for the law. And we present users with a choice only when distinct options are put forward in the text. For instance, in civil law liability, judges can use equivalence in condition or adequate causality. The two methods are correct in this kind of cases. We must also point out that CATALA decisions are not definitive and can disputed by other administrators. Users’ choices are neither arbitrary nor unjustified. Besides, users can run simulations and verify in advance. Other administrative instances can also reject an interpretation as invalid. When this happens, there must be available recourses for abuse of power. We encountered an example of such situations: a deduction depending on payable taxes was related to another, capped deduction. We resolved the issue by creating two related variables. Indeed, our project allows us to use an infinite amount of variables. For instance, we can use different variables for one word (like “child”) defined in different ways in Title I and Title III of a legal text. Finally, this has led us to identify inconsistencies or inaccuracies with legal texts we were working with. These issues were reported to competent authorities.

The limits of encoding law

Liane Huttner: It is not neutral to translate law into a computer language. I would even go as far as to say that I stand against it on principle. For I think that law is the work of men and courtrooms — and not the work of computer scientists. Our project, however, concerns solely the application to individual cases of legislation involving calculations. We create nothing new. We simply have a better system to offer than the one used for the past forty years, which is opaque in its operations. For we do need algorithms, since it is impossible to calculate the income tax for 65 million French citizens manually. In the context, it seems justified to me to encode legal texts, and that it is proportionate to use algorithms. What is even more interesting is that such tools can be used and verified by both the public and administrative instances. I must point out, though, that some thinkers reject the automation of law in all its forms, and that their reasoning is based on valid grounds. Yet, if we follow this line of thought, we encounter many questions. Whose work would it be to calculate income tax? Should we recruit public servants to go through these tedious calculations? Would that lead to simplifications in the tax code? We would need to think thoroughly about options other than automation. Yet, in my opinion our project cannot be expanded to legal fields where no calculations are involved — except if it were justified by the principle of proportionality. It follows that the CATALA project should not be used for the penal code or the civil code. If we do want to translate legal texts into code, we must reflect on the matter beforehand and make sure that ethical and technical feasability conditions are met. We must also think of production methods which are loyal, ethical, and which include a system of checks and balances.