www.researchgate.net
Open in
urlscan Pro
2606:4700::6811:2169
Public Scan
URL:
https://www.researchgate.net/publication/355216002_A_Survey_on_Legal_Question_Answering_Systems
Submission: On February 07 via manual from AT — Scanned from DE
Submission: On February 07 via manual from AT — Scanned from DE
Form analysis 3 forms found in the DOM
GET search
<form method="GET" action="search" class="lite-page__header-search-input-wrapper"><input type="hidden" name="context" readonly="" value="publicSearchHeader"><input placeholder="Search for publications, researchers, or questions" name="q"
autocomplete="off" class="lite-page__header-search-input"><button
class="nova-legacy-c-button nova-legacy-c-button--align-center nova-legacy-c-button--radius-full nova-legacy-c-button--size-s nova-legacy-c-button--color-white nova-legacy-c-button--theme-ghost nova-legacy-c-button--width-square lite-page__header-search-button"
type="submit" width="square"><span class="nova-legacy-c-button__label"><svg aria-hidden="true"
class="nova-legacy-e-icon nova-legacy-e-icon--size-s nova-legacy-e-icon--theme-bare nova-legacy-e-icon--color-inherit nova-legacy-e-icon--luminosity-medium">
<use xlink:href="/m/493903042280588/images/icons/nova/icon-stack-s.svg#magnifier-s"></use>
</svg></span></button></form>Name: loginForm — POST https://www.researchgate.net/login?_sg=NZSX3-VZCmFiWxDRuWYUvQYlkxGqKRZCl9gV6VaP7angHv4mPlXb3o_rVz8XToi46yu1IvXhQSYlSw
<form method="post" action="https://www.researchgate.net/login?_sg=NZSX3-VZCmFiWxDRuWYUvQYlkxGqKRZCl9gV6VaP7angHv4mPlXb3o_rVz8XToi46yu1IvXhQSYlSw" name="loginForm" id="headerLoginForm"><input type="hidden" name="request_token"
value="aad-AKCozDfKJJQ8VEgHNaJRCHuLqsxFzKJV/03H0MEFd6XOzxNvIllJfqwNKN8dFRW0J4sq3HAMBHMDlLPgzFKKvyduCQtwDARVs1Bm4eflODzvaq8f6iG+48GK4pRQ+UJQBiRXy3rs++gDoNn9zRqgJ0CRbqPw7i3IWwK1pkwrk89729zWS0QEJPDOtev072FRlIEb/g18leSenyrcwWXrdI3RtYQjvB/34wEmpJ354mO87jmfyUiePkzgm7nS4k6rskIHoBd5lH6Fs4kp5OU="><input
type="hidden" name="urlAfterLogin" value="publication/355216002_A_Survey_on_Legal_Question_Answering_Systems"><input type="hidden" name="invalidPasswordCount" value="0"><input type="hidden" name="headerLogin" value="yes">
<div class="lite-page__header-login-item"><label class="nova-legacy-e-text nova-legacy-e-text--size-m nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-none nova-legacy-e-text--color-inherit lite-page__header-login-label"
for="input-header-login">Email <div class="lite-page-tooltip "><svg aria-hidden="true" class="nova-legacy-e-icon nova-legacy-e-icon--size-s nova-legacy-e-icon--theme-bare nova-legacy-e-icon--color-inherit nova-legacy-e-icon--luminosity-medium">
<use xlink:href="/m/493903042280588/images/icons/nova/icon-stack-s.svg#info-circle-s"></use>
</svg>
<div class="lite-page-tooltip__content lite-page-tooltip__content--above">
<div class="nova-legacy-e-text nova-legacy-e-text--size-s nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-none nova-legacy-e-text--color-inherit"><b>Tip:</b> Most researchers use their institutional email address as their
ResearchGate login</div>
<div class="lite-page-tooltip__arrow lite-page-tooltip__arrow--above">
<div class="lite-page-tooltip__arrow-tip"></div>
</div>
</div>
</div></label></div><input type="email" required="" placeholder="" id="input-header-login" name="login" autocomplete="email" tabindex="1"
class="nova-legacy-e-input__field nova-legacy-e-input__field--size-m lite-page__header-login-item nova-legacy-e-input__ambient nova-legacy-e-input__ambient--theme-default">
<div class="lite-page__header-login-item"><label class="lite-page__header-login-label"
for="input-header-password">Password</label><a class="nova-legacy-e-link nova-legacy-e-link--color-blue nova-legacy-e-link--theme-bare lite-page__header-login-forgot" href="application.LostPassword.html">Forgot password?</a></div><input
type="password" required="" placeholder="" id="input-header-password" name="password" autocomplete="current-password" tabindex="2"
class="nova-legacy-e-input__field nova-legacy-e-input__field--size-m lite-page__header-login-item nova-legacy-e-input__ambient nova-legacy-e-input__ambient--theme-default"><label
class="nova-legacy-e-checkbox lite-page__header-login-checkbox"><input type="checkbox" class="nova-legacy-e-checkbox__input" aria-invalid="false" name="setLoginCookie" tabindex="3" value="yes" checked=""><span
class="nova-legacy-e-checkbox__checkmark"></span><span class="nova-legacy-e-checkbox__label"> Keep me logged in</span></label>
<div
class="nova-legacy-l-flex__item nova-legacy-l-flex nova-legacy-l-flex--gutter-m nova-legacy-l-flex--direction-column@s-up nova-legacy-l-flex--align-items-stretch@s-up nova-legacy-l-flex--justify-content-center@s-up nova-legacy-l-flex--wrap-nowrap@s-up">
<div class="nova-legacy-l-flex__item"><button
class="nova-legacy-c-button nova-legacy-c-button--align-center nova-legacy-c-button--radius-m nova-legacy-c-button--size-m nova-legacy-c-button--color-blue nova-legacy-c-button--theme-solid nova-legacy-c-button--width-full" type="submit"
width="full" tabindex="4"><span class="nova-legacy-c-button__label">Log in</span></button></div>
<div class="nova-legacy-l-flex__item nova-legacy-l-flex__item--align-self-center@s-up">
<div class="nova-legacy-e-text nova-legacy-e-text--size-s nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-none nova-legacy-e-text--color-inherit">or</div>
</div>
<div class="nova-legacy-l-flex__item">
<div
class="nova-legacy-l-flex__item nova-legacy-l-flex nova-legacy-l-flex--gutter-m nova-legacy-l-flex--direction-column@s-up nova-legacy-l-flex--align-items-center@s-up nova-legacy-l-flex--justify-content-flex-start@s-up nova-legacy-l-flex--wrap-nowrap@s-up">
<div class="nova-legacy-l-flex__item">
<a href="connector/google"><div style="display:inline-block;width:247px;height:40px;text-align:left;border-radius:2px;white-space:nowrap;color:#444;background:#4285F4"><span style="margin:1px 0 0 1px;display:inline-block;vertical-align:middle;width:38px;height:38px;background:url('images/socialNetworks/logos-official-2019-05/google-logo.svg') transparent 50% no-repeat"></span><span style="color:#FFF;display:inline-block;vertical-align:middle;padding-left:15px;padding-right:42px;font-size:16px;font-family:Roboto, sans-serif">Continue with Google</span></div></a>
</div>
</div>
</div>
</div>
</form>Name: loginForm — POST https://www.researchgate.net/login?_sg=NZSX3-VZCmFiWxDRuWYUvQYlkxGqKRZCl9gV6VaP7angHv4mPlXb3o_rVz8XToi46yu1IvXhQSYlSw
<form method="post" action="https://www.researchgate.net/login?_sg=NZSX3-VZCmFiWxDRuWYUvQYlkxGqKRZCl9gV6VaP7angHv4mPlXb3o_rVz8XToi46yu1IvXhQSYlSw" name="loginForm" id="modalLoginForm"><input type="hidden" name="request_token"
value="aad-AKCozDfKJJQ8VEgHNaJRCHuLqsxFzKJV/03H0MEFd6XOzxNvIllJfqwNKN8dFRW0J4sq3HAMBHMDlLPgzFKKvyduCQtwDARVs1Bm4eflODzvaq8f6iG+48GK4pRQ+UJQBiRXy3rs++gDoNn9zRqgJ0CRbqPw7i3IWwK1pkwrk89729zWS0QEJPDOtev072FRlIEb/g18leSenyrcwWXrdI3RtYQjvB/34wEmpJ354mO87jmfyUiePkzgm7nS4k6rskIHoBd5lH6Fs4kp5OU="><input
type="hidden" name="urlAfterLogin" value="publication/355216002_A_Survey_on_Legal_Question_Answering_Systems"><input type="hidden" name="invalidPasswordCount" value="0"><input type="hidden" name="modalLogin" value="yes">
<div class="nova-legacy-l-form-group nova-legacy-l-form-group--layout-stack nova-legacy-l-form-group--gutter-s">
<div class="nova-legacy-l-form-group__item nova-legacy-l-form-group__item--width-auto@m-up"><label
class="nova-legacy-e-text nova-legacy-e-text--size-m nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-xxs nova-legacy-e-text--color-inherit nova-legacy-e-label" for="input-modal-login-label"><span
class="nova-legacy-e-label__text">Email <div class="lite-page-tooltip "><span class="nova-legacy-e-text nova-legacy-e-text--size-m nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-none nova-legacy-e-text--color-grey-500">·
Hint</span>
<div class="lite-page-tooltip__content lite-page-tooltip__content--above">
<div class="nova-legacy-e-text nova-legacy-e-text--size-s nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-none nova-legacy-e-text--color-inherit"><b>Tip:</b> Most researchers use their institutional email address as
their ResearchGate login</div>
<div class="lite-page-tooltip__arrow lite-page-tooltip__arrow--above">
<div class="lite-page-tooltip__arrow-tip"></div>
</div>
</div>
</div></span></label><input type="email" required="" placeholder="Enter your email" id="input-modal-login" name="login" autocomplete="email" tabindex="1"
class="nova-legacy-e-input__field nova-legacy-e-input__field--size-m nova-legacy-e-input__ambient nova-legacy-e-input__ambient--theme-default"></div>
<div class="nova-legacy-l-form-group__item nova-legacy-l-form-group__item--width-auto@m-up">
<div class="lite-page-modal__forgot"><label class="nova-legacy-e-text nova-legacy-e-text--size-m nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-xxs nova-legacy-e-text--color-inherit nova-legacy-e-label"
for="input-modal-password-label"><span
class="nova-legacy-e-label__text">Password</span></label><a class="nova-legacy-e-link nova-legacy-e-link--color-blue nova-legacy-e-link--theme-bare lite-page-modal__forgot-link" href="application.LostPassword.html">Forgot password?</a>
</div><input type="password" required="" placeholder="" id="input-modal-password" name="password" autocomplete="current-password" tabindex="2"
class="nova-legacy-e-input__field nova-legacy-e-input__field--size-m nova-legacy-e-input__ambient nova-legacy-e-input__ambient--theme-default">
</div>
<div class="nova-legacy-l-form-group__item nova-legacy-l-form-group__item--width-auto@m-up"><label class="nova-legacy-e-checkbox"><input type="checkbox" class="nova-legacy-e-checkbox__input" aria-invalid="false" checked="" value="yes"
name="setLoginCookie" tabindex="3"><span class="nova-legacy-e-checkbox__checkmark"></span><span class="nova-legacy-e-checkbox__label"> Keep me logged in</span></label></div>
<div class="nova-legacy-l-form-group__item nova-legacy-l-form-group__item--width-auto@m-up"><button
class="nova-legacy-c-button nova-legacy-c-button--align-center nova-legacy-c-button--radius-m nova-legacy-c-button--size-m nova-legacy-c-button--color-blue nova-legacy-c-button--theme-solid nova-legacy-c-button--width-full" type="submit"
width="full" tabindex="4"><span class="nova-legacy-c-button__label">Log in</span></button></div>
<div class="nova-legacy-l-form-group__item nova-legacy-l-form-group__item--width-auto@m-up">
<div
class="nova-legacy-l-flex__item nova-legacy-l-flex nova-legacy-l-flex--gutter-m nova-legacy-l-flex--direction-column@s-up nova-legacy-l-flex--align-items-center@s-up nova-legacy-l-flex--justify-content-flex-start@s-up nova-legacy-l-flex--wrap-nowrap@s-up">
<div class="nova-legacy-l-flex__item">
<div class="nova-legacy-e-text nova-legacy-e-text--size-s nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-none nova-legacy-e-text--color-inherit">or</div>
</div>
<div class="nova-legacy-l-flex__item">
<div
class="nova-legacy-l-flex__item nova-legacy-l-flex nova-legacy-l-flex--gutter-m nova-legacy-l-flex--direction-column@s-up nova-legacy-l-flex--align-items-center@s-up nova-legacy-l-flex--justify-content-flex-start@s-up nova-legacy-l-flex--wrap-nowrap@s-up">
<div class="nova-legacy-l-flex__item">
<a href="connector/google"><div style="display:inline-block;width:247px;height:40px;text-align:left;border-radius:2px;white-space:nowrap;color:#444;background:#4285F4"><span style="margin:1px 0 0 1px;display:inline-block;vertical-align:middle;width:38px;height:38px;background:url('images/socialNetworks/logos-official-2019-05/google-logo.svg') transparent 50% no-repeat"></span><span style="color:#FFF;display:inline-block;vertical-align:middle;padding-left:15px;padding-right:42px;font-size:16px;font-family:Roboto, sans-serif">Continue with Google</span></div></a>
</div>
</div>
</div>
<div class="nova-legacy-l-flex__item">
<div class="nova-legacy-e-text nova-legacy-e-text--size-s nova-legacy-e-text--family-sans-serif nova-legacy-e-text--spacing-none nova-legacy-e-text--color-grey-500" align="center">No account?
<a class="nova-legacy-e-link nova-legacy-e-link--color-blue nova-legacy-e-link--theme-decorated" href="signup.SignUp.html?hdrsu=1&_sg%5B0%5D=BiXArZ-UnjfZwdRLTxV6Jgmrrn202CokkpD_A6EmkDOcEQo6L3V_FtUyq92GeZShRczG1Qs8XvceGSzEXf5bYF2Hns0">Sign up</a>
</div>
</div>
</div>
</div>
</div>
</form>Text Content
WE VALUE YOUR PRIVACY
We and our partners store and/or access information on a device, such as cookies
and process personal data, such as unique identifiers and standard information
sent by a device for personalised ads and content, ad and content measurement,
and audience insights, as well as to develop and improve products. With your
permission we and our partners may use precise geolocation data and
identification through device scanning. You may click to consent to our and our
partners’ processing as described above. Alternatively you may click to refuse
to consent or access more detailed information and change your preferences
before consenting.
Please note that some processing of your personal data may not require your
consent, but you have a right to object to such processing. Your preferences
will apply to a group of websites. You can change your preferences at any time
by returning to this site or visit our privacy policy.
MORE OPTIONSDISAGREEAGREE
PreprintPDF Available
A SURVEY ON LEGAL QUESTION ANSWERING SYSTEMS
* October 2021
Authors:
Jorge Martinez-Gil
* Software Competence Center Hagenberg
Download file PDFRead file
Preprints and early-stage research may not have been peer reviewed yet.
Download file PDF
Read file
Download citation
Copy link Link copied
--------------------------------------------------------------------------------
Read file
Download citation
Copy link Link copied
References (95)
ABSTRACT
Many legal professionals think that the explosion of information about local,
regional, national, and international legislation makes their practice more
costly, time-consuming, and even error-prone. The two main reasons for this are
that most legislation is usually unstructured, and the tremendous amount and
pace with which laws are released causes information overload in their daily
tasks. In the case of the legal domain, the research community agrees that a
system allowing to generate automatic responses to legal questions could
substantially impact many practical implications in daily activities. The degree
of usefulness is such that even a semi-automatic solution could significantly
help to reduce the workload to be faced. This is mainly because a Question
Answering system could be able to automatically process a massive amount of
legal resources to answer a question or doubt in seconds, which means that it
could save resources in the form of effort, money, and time to many
professionals in the legal sector. In this work, we quantitatively and
qualitatively survey the solutions that currently exist to meet this challenge.
Discover the world's research
* 20+ million members
* 135+ million publications
* 700k+ research projects
Join for free
Public Full-text 1
Content uploaded by Jorge Martinez-Gil
Author content
All content in this area was uploaded by Jorge Martinez-Gil on Oct 15, 2021
Content may be subject to copyright.
arXiv:2110.07333v1 [cs.IR] 12 Oct 2021
A Survey on Legal Question Answering Systems
Jorge Martinez-Gil
Software Competence Center Hagenberg
Softwarepark 32a, 4232
Hagenberg im Muhlkreis, Austria
jorge. martinez-gil@ scch. at
Abstract
Many legal professionals think that the explosion of information about local,
regional, national, and
international legislation makes their practice more costly, time-consuming, and
even error-prone. The
two main reasons for this are that most legislation is usually unstructured, and
the tremendous amount
and pace with which laws are released causes information overload in their daily
tasks. In the case
of the legal domain, the research community agrees that a system allowing to
generate automatic
responses to legal questions could substantially impact many practical
implications in daily activities.
The degree of usefulness is such that even a semi-automatic solution could
significantly help to reduce
the workload to be faced. This is mainly because a Question Answering system
could be able to auto-
matically process a massive amount of legal resources to answer a question or
doubt in seconds, which
means that it could save resources in the form of effort, money, and time to many
professionals in
the legal sector. In this work, we quantitatively and qualitatively survey the
solutions that currently
exist to meet this challenge.
Keywords: Knowledge Engineering, Data Engineering, Legal Intelligence, Legal
Information
Processing, Legal Question Answering
Preprint submitted to Computer Science Review October 15, 2021
1. Introduction
Legal Intelligence (LI) is a broad umbrella covering a group of technologies
aimed at automating
tasks that formerly required human intelligence, communication, and
decision-making in the legal
domain. Many applications use LI technology to support services, including legal
decision support,
adaptive interfaces, and mining litigation data for strategic legal advice, etc.
In the context of this
work, we focus on a subset of the Question Answering (QA) challenge; the design
of solutions for
automatic Legal Question Answering (LQA). The LQA challenge is about building
methods and tools
that involve accurately ingesting the query information from a person as an
input and automatically
offering a response tailored to the person’s needs as an output.
The LQA challenge is far from being trivial since one of the legal domain’s
significant problems
is that newly generated legislation is usually formatted unstructured. Moreover,
the vast volume
and speed at which legislation is made available usually lead to an information
overload problem.
Therefore, finding practical solutions is considered a significant research
challenge, and it is a matter
of intense research. In fact, in recent times, we have witnessed an explosion in
the number of solutions
for LQA.
The truth is that this domain is a perfect candidate for data and knowledge
scientists and prac-
titioners to validate their systems because it meets all the requirements that
make the problem at-
tractive: there is a massive amount of information, which also grows and grows
every day (e.g., the
vast amount of legislation at local, regional, national and international levels
that is generated every
day), and this information often lacks structure (almost all public
administrations publish their laws
in unstructured formats and without annotation, at most with some metadata). As
a result, people
in the sector face severe difficulties navigating this sea of information.
Numerous solutions claim to solve the problem to overcome the above-mentioned
problems (with
varying degrees of success). This is mainly because these solutions are
explicitly designed to automat-
ically process many legal sources to answer a question or doubt in a matter of
seconds, which means
2
it could save resources in the form of effort, money, and time for many legal
professionals. Therefore,
both academia and industry have a strong interest in presenting their solutions
as an excellent candi-
date to overcome some of the limitations in the legal field by trying to
automatically provide answers
to given questions, rather than presenting the professionals with a list of
related items (Kolomiyets &
Moens, 2011).
If we look at the historical evolution of this type of solution, we can realize
that in the past, most
techniques were based on the development or refinement of information retrieval
(IR) techniques that
would allow legal texts to be processed to answer questions. However, in recent
times, both academia
and industry are turning more to neural network-based solutions. However, both
have significant
advantages and disadvantages. For example, while classic IR techniques require
very long processing
times, which can be mitigated with high-performance hardware architectures,
neural network-based
techniques require a lot of training time, but once trained, they are much
faster, and they usually
deliver good results. However, the significant problems associated with these
neural architectures
are the low interpretability of the final models, the tremendous amount of
resources needed for their
training, and the great difficulty in transferring the learned models.
The tremendous impact that QA systems can have on a wide range of academic
disciplines and
business models has attracted much attention from both the research community
and the legal indus-
try. As a result, there already several surveys covering several aspects of this
challenge, e.g., (Wang,
2006; Kolomiyets & Moens, 2011; H¨offner et al., 2017; Diefenbach et al., 2018;
Franco et al., 2020;
Dimitrakis et al., 2020; da Silva et al., 2020). However, the present survey is
different from the previous
ones in many ways. First, it is, to the best of our knowledge, the first attempt
to compile the existing
literature to address the LQA problem. Moreover, we intend to do it informative
and transverse,
without focusing on a specific computational method or family of information
sources. Finally, it is
the most updated to date. We are collecting the latest advances related to the
advances in neural
computation and knowledge graphs.
3
Therefore, to articulate this survey’s organization, we intend to answer the five
questions that we
consider key to meeting the challenge. This means that the contribution of the
present work is based
on an objective and depth answer to the following five main research questions:
•RQ1. What are the most significant LQA research works to date?
•RQ2. How do the primary data sources on which LQA relies look like?
•RQ3. How do the current LQA solutions address the problem of interpretability?
•RQ4. How is the quality of LQA systems currently assessed?
•RQ5. What are the research challenges that remain open in this context?
The rest of the work is structured as follows: Section 2 addresses RQ1 by
presenting the state-of-
the-art LQA methods categorized by the different conceptual approaches they are
based on. Section
3 addresses RQ2 by explaining the primary sources of data, information, or
knowledge from which
existing solutions are usually sourced. Section 4 addresses RQ3, introducing the
concept of inter-
pretability, explaining its paramount importance in the legal domain, and
analyzing which solutions
are the most advanced in this respect. Section 5 addresses RQ4, explaining which
are the most
common methods for the evaluation and quality assessment of the different LQA
systems. Section 6
addresses RQ5 explaining the challenges that industry and academia will face
soon if they want their
solutions adopted by professionals in the sector. Finally, we note the
conclusions and lessons learned
that could be drawn from this work.
2. Related Works
In the context of this work, we assume that a question usually comes in the form
of a natural
language sentence, which usually starts with an interrogative word and expresses
some information
need of the user, although sometimes a question could also take a form of an
imperative construct
4
and starts with a verb. The answer to a question, which may or may not be known
in advance,
would be a word, number, date, sentence, or paragraph that completely satisfies
the information need
expressed by the question. The standard process of automatically answering a
question involves three
fundamental phases: Query Understanding (to extract the core elements that make
up the question),
Sources Retrieval (to identify a few documents that may contain the answer from
a large source pool),
and Answer Extraction (to find a short phrase or sentence that answers the user’s
question directly).
However, not all methods implement the different phases similarly, as we will see
throughout the
section.
The problem of automatically match an answer to a question is widely assumed to
be an impor-
tant research challenge that tries to address one aspect of Artificial
Intelligence (AI) that will allow
computers to perform routine and tedious tasks. Nevertheless, automatically
answering questions is
genuinely complex and encompasses many challenges and nuances that make it
unfeasible to be cov-
ered by just an application. This means that most of the associated challenges
could have an impact
on both industry and academia. This is because having information systems that
can correctly answer
questions asked by a person opens a wide range of possibilities to impact, from
the most basic research
to the most advanced business models.
For this reason, there are numerous solutions to the problem of QA. While in the
past, most
techniques were based on the development or refinement of natural language
techniques that would
allow legal texts to be processed to answer questions, in recent times, both
academia and industry
are turning more to neural network-based solutions inspired in the seminal work
of Mikolov et al.
(Mikolov et al., 2013). For example, BERT (Devlin et al., 2019) and ELMo (Peters
et al., 2018)
or USE (Cer et al., 2018). These approaches have tremendous advantages and
disadvantages. For
example, while traditional techniques require properly tuning up different
workflow pipelines, neural
network-based techniques can be trained once and use many times later. The
problem is the lack of
interpretability, i.e., the limited capacity a person might have to understand
why the methods work
so well and the number of resources required for training and adaptation to new
cases.
5
In this survey, we focus strictly on existing LQA work, i.e., questions and
answers restricted to the
world of law. Table 1 shows a summary of the features that make them different
from other question
answering systems. First of all, in LQA, we work in a closed environment.
Moreover, the questions
can only be related to the legal domain. In addition, all types of questions are
allowed, whether they
are definition, comparison, confirmation, etc. The analysis of the texts, both
questions and sources to
extract the answers, is not limited. Nor are the types of sources that can be
used to find the answer
limited. Last but not least, in the context of the current survey, we do not
include dialogue systems
but only question answering solutions.
Several different classifications schemes can be potentially established for
categorizing the main
families of systems built in this domain. QA is a form of information retrieval,
and this means that
solutions are defined by how they represent the input information and the
retrievable entity as well
as by the ranking function. To properly articulate this section, we have focused
on the different
computational approaches for building LQA solutions. Techniques of this kind
have been widely
used in various forms of research on Yes/No Answers, Multiple Choice Question
Answering, Classical
Solutions from the Information Retrieval field, Ontology-based solutions, Neural
solutions, and Other
approaches.
•Yes/No Answers
•Legal Multiple Choice Question Answering
•Classical IR-based Solution
•Big Data solutions
•Ontology-based solutions
•Neural solutions
•Other approaches, e.g., Knowledge Graphs
6
Table 1: Kind of QA systems covered within this survey
Features of Legal Question Answering
Environment [Open] Closed
[Closed]
Domain
[General-purpose] Legal
[Mathematical]
[Legal]
[Other]
Kind of Questions [Factoid] All
[Definition]
[Confirmation]
[Causal]
[Comparison]
Kind of Analysis
[Morphological] All
[Syntactical]
[Semantic]
[Pragmatic]
Sources [Structured] All
[Semi-structured]
[Unstructured]
Kind of dialogue
[Question Answering] Question Answering
[Dialog]
[Spoken Dialog]
7
Each of the different branches of research has its peculiarities. Thus, for
example, the first types
based on Yes/No Answers and Multiple Choice Question Answering assume a
simplified version of the
problem where it is unnecessary to worry about generating candidate answers,
only about confirming
or not the veracity of a sentence or constructing a ranking of answers.
Classical Information Retrieval
techniques, in principle, should be one the most numerous families since they
are one of the natural
ways of tackling the problem. On the other hand, Big Data techniques, even if
they do not represent
an apparent scientific breakthrough, do represent a considerable advance in
technical characteristics,
where even unfeasible but straightforward techniques on a regular computer
(think, for example of
the calculation of regular expressions) can have a good performance. Finally,
the novel techniques
based on neural architectures with the best-expected results and other
approaches come to fill some
of the gaps related to neural networks, such as, for example, the lack of
interpretability. We then
explain what search strategy we followed to capture the primary sources of the
literature, we review
the different approaches, and we conclude the section with a summary.
2.1. Data Sources and Search Strategy
To choose the primary sources of this survey, we have built a search string so
that it includes
two major search terms: ’Method’ and ’Field’. The first major search term
represents the employed
methodology, namely, ’question answering’, whereas the second major search term
illustrates the fields
in which the method should have been utilized, i.e., ’legal’. This term includes
all sorts of technologies
and synonyms of legal in which that application should occur. Terms like ’legal
domain’, ’legal sector’,
’legal industry’, ’legal area’, ’legal documents’, ’legal material’,
’legislation’, and ’regulatory’. In
addition, through snowballing, we have come up with articles that contain terms
such as ’due diligence’.
Queries were run on Google Scholar1, DBLP2, Microsoft Academic3, and Semantic
Scholar4.
1https://scholar.google.com/
2https://dblp.org/
3https://academic.microsoft.com/home
4https://www.semanticscholar.org/
8
Table 2: Inclusion and exclusion criteria for work selection
Inclusion Exclusion
Written in English Written in other languages rather than English
Published in journal, conference or workshop Preprints when the paper is
available
Described an algorithm, technique or system Thesis, editorials, prefaces, or
summaries
Uses, at least, one LQA dataset QA method but no application in LQA
Published from January 1990 Published by a predatory publisher
Published until June 2021 Patents or patent applications
Moreover, we have had to use some criteria to decide whether a particular work
should be consid-
ered. Table 2 shows the criteria we have followed in deciding whether a paper
should be considered
for inclusion or exclusion in this survey. This table follows the recommendation
stated by (Soares &
Parreiras, 2020) for the study of QA systems.
As a result of the screening process, we have obtained 65 primary sources
analyzed and categorized
below within their corresponding family of solutions.
2.2. Yes/No Answers
This approach is the simplest of all. Given a question and a known answer, one
tries to determine
whether the associated answer is true or false. In this way, the systems do not
need to generate
candidate answers. Nevertheless, answering yes/no questions in the legal area is
very different from
other domains. One of its inherent features is that legal statements need a
thorough examination of
predicate-argument structures and semantic abstraction in these statements. The
methods from this
family usually check if there is a high degree of co-occurrence between
questions and answers in corpora
of a legal nature. Some variants might result in searching for some kinds of
regular expressions.
Works based on this approach, such as (Kim et al., 2013) have developed different
approaches
to answer yes/no questions relevant to civil laws in legal bar exams. A bar
examination is intended
9
Table 3: Summary of approaches based on Yes/No
Work Approach
(Kim et al., 2013) Antonym detection and Semantic Analysis
(Kim et al., 2014) TF-IDF, LDA and SVM
(Kim et al., 2016) Paraphrasing detection
(Taniguchi & Kano, 2016) Case-role analysis
(Kano et al., 2017) Using linguistic structures
(Taniguchi et al., 2018) Using Framenet
to determine whether a candidate is qualified to practice law in a given
jurisdiction. There is a
recurring concern in the literature on whether it is possible to pass this type
of test without human
supervision. The same author (Kim et al., 2014) has also implemented some
unsupervised baseline
models (TF-IDF and Latent Dirichlet Allocation (LDA)) and a supervised model,
Ranking SVM, for
facing the challenges. The model features are a set of words and scores of an
article based on the
corresponding baseline models. In addition, a final enhancement includes the use
of paraphrasing
(Kim et al., 2016). Other authors (Taniguchi & Kano, 2016; Taniguchi et al.,
2018) has addressed the
problem by case-role analysis and Framenet5, respectively. Finally, Kano et al.
has explored, for the
first time, linguistic structures (Kano et al., 2017). Table 3 summarizes all
these works.
This branch of research was one of the first to yield results. Its strength is
based on a superficial
understanding of the problem with solutions that tend to work well. The major
problem of this model
is that questions in human language express a well-defined information
requirement on the one hand,
but they also convey more information than an essential list of relevant
keywords since they represent
syntactic and semantic links between those keywords on the other. Therefore,
solutions tend to work
well only to a limited extent, i.e., when the questions are simple.
5https://framenet.icsi.berkeley.edu/fndrupal/
10
Pros: High interpretability. Methods can be used in many other domains for
fact-checking
Cons: The operating model is trivial and leads to poor results at the present
time
2.3. Legal Multiple Choice Question Answering
Legal Multiple Choice Question Answering (LMQA) consists of correctly answer
questions in a
scenario where the possible answers are already given beforehand. In this model,
there are two clear
facilitators. It is unnecessary to generate candidate answers since this family
of methods assumes
that candidate answers are a different problem and should be studied apart.
Furthermore, it is known
that the correct answer is among the given ones (even in situations where None
of above answers are
allowed). Therefore, the goal is to learn a scoring function S(F, z ) with a
normalization parameter z
(whereby zor the normalization factor is usually used so that the values
associated with each answer
are in the same numerical range) such that the score of the correct choice is
higher than the score of
the other hypotheses and their corresponding probabilities.
Some works have used the classical Latent Semantic Analysis to obtain good
results in this con-
text (Deerwester et al., 1990). In general, all techniques based on
co-occurrence are very appropriate
because it is usually sufficient to check the number of times that the question
and each of the answers
appear together in some textual corpora of a legal nature (Li et al., 2003). Of
course, one improvement
is to use semantic similarity detection techniques to detect different
formulations of the same informa-
tion (Han et al., 2013). In addition, crowdsourcing techniques have also been
very successful (Aydin
et al., 2014). There are variations of the above that attempt to compute
co-occurrence in a more
sophisticated way so that text windows and regular expressions are taken into
account (Martinez-Gil
et al., 2019b). More recently, (Chitta & Hudek, 2019) has proposed an
improvement through multi-
class classifiers to identify the answer to the question. The key idea is to
handle imbalanced data
by generating synthetic instances of the minority answer categories. Table 4
chronologically lists the
works that we have mentioned above.
11
Table 4: Summary of approaches based on Multiple Choice
Work Approach
(Deerwester et al., 1990) Latent Semantic Analysis
(Li et al., 2003) Co-occurrence
(Han et al., 2013) Synonym detection
(Aydin et al., 2014) Crowdsourcing
(Martinez-Gil et al., 2019b) Reinforced Co-occurrence
(Chitta & Hudek, 2019) Synthetic minority oversampling
One of the fundamental keys for LMCQA to work well is the choice of the legal
corpus to be worked
on, both in terms of extension and quality since most methods require the
calculation of semantic
similarity, textual co-occurrences, etc.
Pros: High speed. High interpretability. Many existing methods for calculating
rankings
Cons: Knowing all answers beforehand is not a realistic situation in the real
world
2.4. Classical IR-based Solutions
In this context of LQA, classical IR-based solutions usually represent words in
the form of discrete
and atomic units. This family of methods assumes that it is vital that the
solution appropriately
identify either the exact answer or source and specific paragraph containing the
relevant information.
The lawyer then should read and interpret the excerpt for the client. For
example, the first approach
(and the simplest) could query the number of Google results for a specific
question and a given
answer together. However, this solution has brought several problems like the
lack of context for the
formulated question.
To overcome the problem of the lack of context, word processing models such as
LSA (Deerwester
et al., 1990) and term frequency-inverse document frequency (TF-IDF) partially
solve these ambigu-
ities by using terms that appear in a similar context based on their vector
representation. Then they
12
group the semantic space into the same semantic cluster. Within this family of
methods, one of the
best-known QA systems is IBM Watson (Ferrucci et al., 2013), which is very
popular for its victory
in the televised show Jeopardy (Tesauro et al., 2013). Although in recent times,
IBM Watson has
become a generic umbrella that includes other business analytics capabilities.
If we restrict ourselves to LQA, one of the most popular initiatives is
ResPubliQA. This evaluation
task was proposed at the Cross-Language Evaluation Forum (CLEF) (Pe˜nas et al.,
2009, 2010) a
consists of given a pool of independent questions in natural language about
European legislation,
proposed systems should return a passage (but not an exact response) that
answers each question.
This initiative was a remarkable success. In fact, thanks to ResPubliQA, this
family of methods has
been the subject of intensive research during the last decade.
In relation to the works of this family, it is possible to identify,
Brueninghaus and Ashley that have
addressed the problem for the first time with a focus on information extraction
methods (Br¨uninghaus
& Ashley, 2001). Quaresma et al. focused on Portuguese legislation using a
classical IR pipeline
(Quaresma & Rodrigues, 2005). Some years after, (Maxwell & Schafer, 2008) went a
step further
beyond by using conceptual and contextual search. Another strategy was proposed
by (Monroy
et al., 2008) that returns several relevant passages extracted from different
legal sources. The relevant
passages allows generating answers to questions formulated in plain language
using graph algorithms.
A qualitative improvement was presented (Monroy et al., 2009) to lemmatizing and
using manual
and automatic thesauri for improving question-based document retrieval. For the
construction of
the graph, the authors followed the approach of representing all the articles as
a graph structure. In
parallel, (Tran et al., 2013) proposed mining reference information. Last but
not least, (Rodrigo et al.,
2013) proposed a new IR-based system with the lessons learned from ResPubliQA.
More recently, (Carvalho et al., 2016) have introduced a new approach for
lexical to discourse-level
corpus modeling, and (Bach et al., 2017) proposes a novel solution based on
Conditional Random
Fields (CRFs), which are methods that resorts on statistical modeling in order
to segment and label
13
sequence data. Furthermore, (van Kuppevelt & van Dijck, 2017) decided to explore
how to perform
network analysis and visualization as a proper way to deal with Dutch case law.
Meanwhile, (Delfino
et al., 2018) has brought for the first time the Portuguese version of the
thesaurus OpenWordNet6to
the table.
The last works in this direction are those from (Hoshino et al., 2018) by using
predicate-argument
structure, (McElvain et al., 2019a,b) over plain text and focusing on
non-factoid kind of questions.
Non-factoid questions are those whose answer is not directly accessible in the
target document, which
demands some inference and perhaps extra processing to obtain an answer. Wehnert
et al. have
tried a new application of the popular BM25 approach (Wehnert et al., 2019), and
the latest to date,
(Verma et al., 2020) has had a focus on relevant subsection retrieval to answer
the questions of legal
nature. The story so far can be summarized in Table 5.
As can be seen, this family is extensive. Moreover, in its bosom, many and very
diverse proposals
have been born to face the problem. Before the latest advances in Deep Learning,
it was the dominant
family, and its solutions were considered good until those times. However, in
recent times very tough
competitors have emerged.
Pros: High performance, High interpretability
Cons: In recent times, accuracy has been overtaken by neural solutions
2.5. Big Data solutions
In recent times, the arrival of novel Big Data approaches has come up with many
advantages and
challenges for the legal sector. The possibilities of Big Data like the effective
and efficient processing
of massive amounts of data have introduced many opportunities and challenges.
Nevertheless, some
of the limitations that have traditionally burdened this area of knowledge
remain. Perhaps the most
illustrative of these limitations is the inability of the systems to understand
the question instead of
6https://github.com/own-pt/openWordnet-PT
14
Table 5: Summary of approaches based on classical and novel approaches for
Information Retrieval
Work Approach
(Br¨uninghaus & Ashley, 2001) Information extraction methods
(Quaresma & Rodrigues, 2005) Classical IR pipeline
(Maxwell & Schafer, 2008) Conceptual and contextual search
(Monroy et al., 2008) Exploitation of Graph-based algorithms
(Monroy et al., 2009) Exploitation of Graph-based algorithms
(Tran et al., 2013) Mining reference information
(Rodrigo et al., 2013) Lessons learned from ResPubliQA
(Carvalho et al., 2016) Lexical to discourse level corpus modeling
(Kim & Goebel, 2017) Cascaded textual entailment
(Bach et al., 2017) Conditional Random Fields
(van Kuppevelt & van Dijck, 2017) Network Analysis
(Delfino et al., 2018) Exploitation of word senses and relations
(Hoshino et al., 2018) Predicate Argument Structure
(McElvain et al., 2019a) Non-Factoid questions
(McElvain et al., 2019b) Non-Factoid questions
(Wehnert et al., 2019) BM25 and Elasticsearch
(Verma et al., 2020) Relevant subsection retrieval
15
Table 6: Summary of approaches based on Big Data
Work Approach
(Bennett et al., 2017) Scalable architecture by design
(Mimouni et al., 2017) Approximate Answers and Rich Semantics
(Martinez-Gil et al., 2019a) Mutual Information Exchange
merely using its huge computational potential to search for statistical patterns
in corpora of a legal
nature.
For this reason, the existing approaches are focused on developing good ideas
that allow to answer
the questions asked and obviate the performance details since these can be
solved with these new forms
of high-performance computing. Some outstanding works in this direction are
proposed by Bennet
et al. with a focus on the scalability of the solution by design (Bennett et
al., 2017), or Mimouni
et al. for handling the problem of complex queries by working with approximate
answers and richer
semantics (Mimouni et al., 2017). Last but not least, there is also a novel
approach based on the
concept of mutual information exchange (Martinez-Gil et al., 2019a), which, when
applied to large
volumes of data, have demonstrated better performance than classic co-occurrence
methods. Table 6
lists chronologically these works.
On the other hand, these applications entail processing and handling large
amounts of data quickly,
and they facilitate the automation of specific low-level computing operations. So
in this way, most
technical limitations disappear, and therefore, methods that were previously
considered too expensive
in terms of resource consumption in the form of time are no longer a problem.
For example, the
execution of queries based on the concept of regular expression that is very
expensive in terms of
computing time. However, the scientific difficulties to model appropriate
inter-dependencies between
the concepts from the questions remain, so the results are not optimal yet.
16
Pros: Very good performance, no limitations of a technical nature
Cons: Inability of the systems to properly understand the meaning of a complex
question
2.6. Ontology-based solutions
The capacity to construct a semantic representation of the data paired with the
accompanying
domain knowledge is one of the significant benefits of adopting domain ontologies.
Ontologies can also
be used to establish connections between various sorts of semantic knowledge. As
a result, ontologies
can be utilized to develop various data-search strategies. The legal domain has
not remained oblivious
to this line of research.
Examples of works that deal with the first approach are Lame et al. proposed for
the first time
the integration of NLP and ontologies to solve LQA problems (Lame, 2004).
Moreover, Xu et al.
proposed a system for relation extraction and textual evidence(Xu et al., 2016).
In addition, Fawei
et al. (Fawei et al., 2018), in the first instance, proposed a criminal law and
procedure ontology.
Meanwhile, in the second instance, proposed a way to semi-automatically
construct ontologies for the
legal section (Fawei et al., 2019). Another approach has consisted of Semantic
Role Labeling (Veena
et al., 2019), and last but not least, Kourtin et al. have opted for an
RDF-SPARQL tailored design
(Kourtin et al., 2021). Table 7 shows us a summary of the works mentioned above.
Solutions based on ontological foundations have always promised to provide the
correct answer to
a question. However, in practice, things are not rather complicated. There are
very few ontologies,
and those that exist are hardly reusable. So it is costly and error-prone to
develop legal ontological
models from scratch. Not to mention the learning curve that dealing with
description logics present,
and the difficulty of performing large-scale reasoning in reasonable time.
Pros: Logical reasoning provides accurate answers. High interpretability
Cons: Models very expensive to build, very difficult to reuse, slow reasoning
times
17
Table 7: Summary of approaches based on ontologies
Work Approach
(Lame, 2004) Combination of NLP and ontologies
(Xu et al., 2016) Relation extraction and textual evidence
(Fawei et al., 2018) Using a criminal law and procedure ontology
(Fawei et al., 2019) Using semi-automated ontology construction
(Veena et al., 2019) Semantic Role Labeling
(Kourtin et al., 2021) RDF-SPARQL tailored design
2.7. Neural solutions
Solutions based on architectures inspired by neural models have adequately and
consistently solved
most benchmarks in the QA field. These results are outstanding and have made most
of the community
decide to use their resources to explore this direction. In recent times,
several works have been
presented. There are two significant lines of research: A branch that tries to
develop solutions that
exploit structured resources that have led to a family that we will see later.
Furthermore, another
branch that tries to develop methods capable of exploiting unstructured
information.
Exemplary works that operate with unstructured text using the most modern
machine learning
techniques are, for example, works such as (Kim et al., 2015; Do et al., 2017)
used for the first time a
convolutional neural network to LQA. A few years after, the same authors
proposed an improvement
using deep siamese networks (Kim et al., 2017). Other authors decide to explore
new kinds of systems
using LSTM (John et al., 2016), the concept of neural attention (Morimoto et
al., 2017). The concept
of neural attention consists of freeing the encoder-decoders architecture from a
fixed-length internal
representation. Furthermore, the exploitation of a Multi-task Convolutional
Neural Network (Xiao
et al., 2017) has also been proposed.
Some recent works such as (Cer et al., 2018; Devlin et al., 2019; Peters et al.,
2018) have made
some of the most critical advances in question answering in recent times. These
approaches have con-
18
sistently achieved top results in most academic competitions with a clear
general-purpose orientation.
Moreover, its performance in the legal sector is more than remarkable.
Futhermore, (Collarana et al., 2018) introduced for the first time a new
technique based on Match-
LSTM and a Pointer Layer, whereas (Nicula et al., 2018) proposed to use Deep
Learning over candidate
contexts. (Liu & Luo, 2018) built a system using Deep Neural Networks. In
parallel, Ravichander
et al. proposed a new strategy based on the combination of computational and
legal perspectives
(Ravichander et al., 2019). And last, but not least, Zhong et al. proposed two
novel alternatives using
Reinforcement Learning and Deep Learning respectively (Zhong et al., 2020b,c).
Among the latest published works in this family of methods, we can highlight the
following: The
Neural Attentive Text Representation from (Kien et al., 2020), the introduction
of Few-shot Learning
in the legal domain (Wu et al., 2020). Novel techniques for language modeling
(Huang et al., 2020b).
Deep learning restricted to the world of building (Zhong et al., 2020a). And
diverse applications of
the successful BERT to implement LQA solutions (Holzenberger et al., 2020; Zhang
& Xing, 2021).
Table 8 shows a chronological summary of the aforementioned solutions.
The advances in the numeric representation of words has opened a complete family
of methods.
Many different training methods are able to generate word embeddings from
unstructured data, mak-
ing the novel semantic analysis models achieve state-of-art performance. As for
the legal point of
view, it is critical to find out an efficient way to represent the semantic meaning
of the longer pieces
of text, such as phrases, sentences, or documents, to achieve a more reasonable
interpretation.
On the other hand, one major limiting factor in this context is that there are
usually no legal
datasets to train neural models at a scale properly. Hence, the performance
expected for an LQA
system is usually worse than in a generic one.
Pros: These methods are capable of achieving the best results to date
Cons: Very poor interpretability. Huge amount of data needed for training
As a final note, it is worth mentioning that although the lack of
interpretability of neural solutions is
19
Table 8: Summary of LQA approaches based on neural networks
Work Approach
(Kim et al., 2015) Convolutional Neural Network
(John et al., 2016) Long-Short Term Memory
(Do et al., 2017) Convolutional Neural Network
(Kim et al., 2017) Deep Siamese Networks
(Morimoto et al., 2017) Neural Attention
(Xiao et al., 2017) Multi-task Convolutional Neural Network
(Cer et al., 2018) Universal Sentence Encoder
(Devlin et al., 2019) Bidirectional Encoder Representations from Transformers
(Peters et al., 2018) Embeddings from Language Model
(Collarana et al., 2018) Match-LSTM + Pointer Layer
(Nicula et al., 2018) Deep Neural Networks over candidate contexts
(Liu & Luo, 2018) Deep Neural Networks
(Ravichander et al., 2019) Combination of computational and legal perspectives
(Zhong et al., 2020b) Reinforcement Learning
(Zhong et al., 2020c) Deep Neural Networks
(Kien et al., 2020) Neural Attentive Text Representation
(Wu et al., 2020) Few-shot Learning
(Holzenberger et al., 2020) Based on BERT
(Huang et al., 2020b) Language Modeling
(Zhong et al., 2020a) Deep Neural Networks over building regulation
(Zhang & Xing, 2021) Based on BERT
20
always mentioned, an effort is being made in recent times to develop frameworks
that help the human
operator (Ribeiro et al., 2016; Lundberg & Lee, 2017). These frameworks are
intended to enable
machine learning engineers and relevant domain specialists to analyze the
end-to-end solutions and
discover differences that could lead to a sub-optimal performance concerning the
desired objectives.
2.8. Towards a new family of solutions
Without detracting from the merit of neural solutions, many legal professionals
are not satisfied
with just an answer to their question and demand much more. They demand to
understand why
the solution has opted for such an answer and not another alternative. This
brings up one of the
ghosts that has always been associated with neural computing, that is, that its
behavior is similar
to that of a black-box since it is possible to give it an input and obtain an
output. However, it is
humanly impossible to understand how the connections of thousands of neurons
have worked to give
rise to such an answer, or even the real meaning behind the feature vectors
obtained after subjecting
the model to the training of a neural nature. For example, the base configuration
of BERT (Devlin
et al., 2019) requires a configuration consisting of 12 layers of neurons and 12
windows of attention.
Therefore, there are usually issues related to the interpretability of the
resulting model. This is where
this family of methods comes in.
While Linked Data is now well-established and has a strong community behind it
that has been
studying it for several years, there is a lack of work on its application to the
legal domain. It is true,
however, that some proposals have begun to be put on the table. Nevertheless,
what really promises
to impact this area is the combination of Knowledge Graphs and Machine Learning
to create a new
generation of LQA systems.
To date, some authors thought that exploring the Linked Data route was the way
to go (He et al.,
2013; Filtz, 2017). However, the solutions are still unrealistic, as most of the
legislation is still not
published in a structured way. Furthermore, automatically structuring it is
currently unfeasible. In
addition, the learning curve for formulating questions in graph-oriented
languages has led to the search
21
Table 9: Summary of LQA approaches based on other approaches
Work Approach
(He et al., 2013) Linked Data
(Filtz, 2017) Linked Data
(Tong et al., 2019) Legal Knowledge Graph
(Sovrano et al., 2020) Legal Knowledge Graph
(Huang et al., 2020a) Legal Knowledge Graph
(Filtz et al., 2021) Legal Knowledge Graph
for alternative approaches. In recent times, several works have also emerged
that use for the first time
the potential of knowledge graphs to overcome some of the gaps of neural
solutions, such as the lack of
interpretability and the need for large amounts of data for training (Tong et
al., 2019; Sovrano et al.,
2020; Huang et al., 2020a; Filtz et al., 2021). In addition, technologies such
as GPT-37have been
developed to convert questions formulated in natural language to their
equivalent in graph-oriented
language. Table 9 lists the works above.
Legal Knowledge Graphs has attracted much attention in recent times due to that
an increasing
number of researchers point out that they can bring a high degree of accuracy
together with the
highest interpretability that can be achieved to date. However, the other side
of the coin says that
this approach can offer good results at a high cost (in terms of money, time, and
effort needed).
This is mainly because this technology faces some obstacles in its development
related to the amount
of engineering work to build graphs properly and the necessary improvement of
natural language
translators to graph query language.
Pros: High accuracy. High interpretability. High performance
Cons: No good or cheap methods for automatizing the whole process
7https://openai.com/
22
2.9. Summary
It is clear from the literature that working with information concerning
legislation and case law
has always been attractive to computer scientists and practitioners applying for
the latest advances in
language and information technologies. These technologies have proven to be very
useful for solving
several problems that have traditionally affected legal information processing.
In practice, the daily
activities of these legal professionals requires checking vast amounts of legal
material necessary to assess
the relevant information pieces and identify the correct fragment they need.
With this objective in
mind, the LQA discipline was born and has been developed.
We provide a general summary of the different methods followed to build LQAs in
the last decades.
Figure 1 shows how, at present, of the 65 primary sources analyzed, the vast
majority belong to the
families of neural solutions and IR-based methods, although some families, such
as other approaches,
e.g., Legal Knowledge Graphs, are getting traction in recent times.
Neural Solutions
32.3%
Information Retrieval
26.1%
Yes-No Answers
9.2%
LMCQA
9.2%
Ontologies
9.2% Other approaches
9.2%
Big Data
4.8%
Figure 1: Summary of existing families to meet the challenge of LQA
Next, we will look at the sources of data, information, and knowledge that are
commonly used
when building LQA solutions.
23
3. Data, Information and Knowledge Sources
LQA solutions are a kind of system that analyzes one or various data sources to
answer a given
question automatically. These data sources are usually structured or
unstructured. The first ones
are commonly referred to as legal corpora, while the latter are usually
considered Legal Knowledge
Bases (LKBs). Please note that in this domain, data is considered the atomic
unit with which the
systems work. If this data is structured to facilitate its interpretation, it is
considered information.
Finally, when relationships are created to facilitate understanding and
communication, it is considered
knowledge.
Depending on the sources to be exploited, two techniques for tackling the
problem are working with
unstructured legal corpora or working LKBs. Sometimes a third option is also
considered concerning
semi-structured sources (although this scenario is usually considered structured
as well). Each one
has its own set of benefits and drawbacks. Working with structured LKBs, for
example, let system
designers take advantage of the knowledge represented by using so-called
inference engines to infer
new information and answer questions. The fact is that not easy to implement
these systems, so they
have been progressively replaced by more efficient systems based on lighter
knowledge models such
as knowledge graphs and other enhanced lexical semantic models (Yih et al.,
2013). However, at the
beginning of this survey, we mentioned that legislation usually comes in an
unstructured form, so it is
more realistic to design QA systems being prepared to ingest vast amounts of
data from unstructured
sources. We will now focus on the two fundamental kinds of data sources that LQA
solutions typically
use to find the appropriate answers to the questions asked. We divide these
sources into unstructured
and structured.
3.1. Unstructured sources
As we have seen repeatedly in this survey, most legislation is generated in an
unstructured manner
and at high speed. This means that designing methods capable of working with
unstructured text is
usually the norm when dealing with legal information. LQAs solutions exploiting
unstructured sources
24
have several practical benefits as most of them have been specifically designed to
efficiently process
vast amounts of textual data (usually represented in legal language). These
enormous amounts of data
come from existing documents, legal databases, and so on. Most of the methods of
the Information
Retrieval and Neural Solutions families are based on this strategy.
The impossibility of having structured legal information on a large scale, of
high quality, and
covering all the current legislation, makes researchers and practitioners
consider using methods to
exploit unstructured resources of a legal nature. For this reason, the most
frequent type of LQA
solution needs to identify the relevant legal documents or information pieces
and split them into
candidate passages, and then, face the main key to the problem that consists of
extracting the answer
from the candidate passages.
The current generation of LQA solutions has evolved to extract answers from a
wide range of dif-
ferent plain machine-readable resources. These LQA solutions exploit the massive
set of unstructured
information available on some sources to retrieve information about any
particular question. It is
important to note that these QA systems are possible mainly due to the
extraction functions that
are usually either heuristic or learned from a dataset such the Stanford
Question Answering Dataset
(SQuAD) (Rajpurkar et al., 2016) or JEC-QA (Zhong et al., 2020c) in this
particular case. Moreover,
since these extraction methods can process questions about different domains and
topics, they are
highly reusable.
Concrete examples in the legal domain are ResPubliQA 2010 collection, i.e., a
subset of the JRC-
ACQUIS Multilingual Parallel Corpus and the EUROPARL collection which are
multilingual parallel
collections. The JRC-ACQUIS is a legal corpus representing the complete body of
the European Union
documents, and it is commonly used to build systems. Other interesting examples
are the House of
Lords judgments corpus8, etc. Moreover, some recent works such as (Chalkidis &
Kampas, 2019)
show us one of the clearest examples of how LQA solutions can benefit from
unstructured corpora to
8https://publications.parliament.uk/pa/ld/ldjudgmt.htm
25
build successful systems.
Pros: Information can be easily added or updated
Cons: Reliability of answers is low, paraphrasing is frequent, problematic
representation of
unstructured data
3.2. Structured sources
The most popular form of the structured source is Legal Knowledge Bases (LKBs)
(Bench-Capon &
Coenen, 1992). The LKBs are knowledge management systems being able to describe
legal entities and
the relationships between them. Legal entities are usually represented as nodes,
while the relationships
between them are represented as edges. Each legal entity can be classified using
one or more classes.
At the same time, classes can be organized hierarchically. The same is valid for
relationships. Nodes
also stand for literals. Moreover, most LKBs can perform reasoning enabling the
derivation of inferred
facts. Most of ontology-based, Linked Data, and Legal Knowledge Graphs use these
kinds of structured
sources.
For example, an LQA solution consuming knowledge from a LKB might be focused on
translating
a question formulated by a user to a specific query of an LKB, such as in
(Kourtin et al., 2021). This
makes these LQA solutions perfect candidates for answering factoid formulations
with high accuracy.
The reason is that factoid formulations are simple questions that could be
answered with a precise
fact, so the corresponding answer can be obtained merely by knowing an entity
and a property in the
LKB. Nevertheless, many problems may arise when many facts need to be
discovered, connected, and
combined to elaborate.
Furthermore, there may be several reasons why the generation of structured
information is not so
popular in this domain, but in general, it is widely assumed that generating
such information might
be too expensive in terms of resource consumption, it might be subject to many
errors what it would
make it difficult and expensive to be maintained (Martinez-Gil, 2015).
26
Despite these problems, general-purpose datasets to test solutions have been
developed in recent
times. For example, the series of Question Answering over Linked Data (QALD)
(Cimiano et al.,
2013). However, and unfortunately, the development of structured sources of
legal content is much
more limited. Although the process on how the extraction function is usually
learned is well described
in (Zou et al., 2014). The good news is that the use of Knowledge Graphs of a
legal nature is being
investigated to obtain better results (Filtz, 2017; Huang et al., 2020a; Filtz
et al., 2021).
Pros: Reliability of answers is high, not need of complex natural language
processing
Cons: Limited information stored, sourcing is error-prone and expensive to
build, difficult
interoperability in data sources
4. The problem of the interpretability
Historically, the legal sector is as interested in knowing the answer to a
question as in knowing
its associated explanation. It is something inherent to this domain that
differentiates it from many
disciplines where a black-box would have no significant problems operating. In
the legal industry,
apart from the scenario whereby the system is assumed to have the required
information and the user
is willing to accept the answers, another critical factor comes into play: the
need for explanation.
Therefore, the problem of interpretability is highly relevant for legal
professionals.
Methods and techniques for answering specific questions are in high demand, and
as a result,
several solutions for LQA have been developed to respond to this need. System
designers claim that
the capability to answer questions through computers automatically could help
alleviate a problem
involving tedious tasks such as an extensive information search that is, in
general, time-consuming.
Moreover, therefore, by automatically providing hints concerning a vast number
of legal topics, lots
of resources in the form of effort, money, and time could be saved.
The problem is that designing a LQA solution with a high degree of accuracy,
interpretability,
and performance at the same time is far from being trivial. So one often has to
choose at most
27
two characteristics out of the three possible ones. Interpretability being, in
the legal sector, almost
a mandatory requirement. In addition, if it is taken into account that, as we
have seen, black-box
solutions currently dominate this application domain, it is possible to deduce
that one of the root
causes that is limiting the LQA growth is the fact that the automatic generation
of explanations is
still not very satisfactory for the legal professionals.
There are many proposals to work with legal texts; some of these proposals are
based on variations
of the concept of the distributional assumption terms that appear in a similar
context base, the
calculation of synonyms, others are based on the co-occurrence of words in the
same textual corpus,
different neural architectures, different legal sources, etc. In principle, it is
challenging to discern
which approach could perform better than the others. This always depends on the
use case and the
context in which they are applied. But there is at least one factor that can be
known in advance: the
solution’s interpretability. That is, a human operator can fully understand the
model used to answer
questions. A system that is worthy of people’s trust must not only be effective
and efficient; it must
also be able to communicate the reason for the outputs it produces (Atkinson et
al., 2020).
If we did not have a fundamental requirement, the interpretability of the
solution, it is clear that
the family of neural solutions might be a perfect candidate to build the best
LQA solutions to date.
However, understanding the model, which is an essential aspect in the legal
domain, is not the only
obstacle. For example, finding large data sets representing solved cases is very
difficult or expensive,
or the neural model is usually difficult to be reused in problems of a similar
nature. In this respect,
some of the latest IR systems are superior because they do not require training,
and they can always
be reused over any corpora.
For all these reasons we have detailed, explaining the answers offered is very
important. Moreover,
explanations should be contrastive. This means that in addition to explaining
why a particular
outcome has been offered, a good explanation should communicate why other
outcomes have been
discarded. According to recent work (Doshi-Velez & Kim, 2017), the
interpretability of a model is
28
measured according to three levels:
•Application level interpretability is the level at which a domain expert could
understand how the
model works. Therefore, a good knowledge of the legal area and how the quality
of the result
can be determined is needed. This level should be sufficient for specialized
professionals in the
sector.
•Human level interpretability is a more superficial level than the application
level because it is
not necessary to rely on an expert to examine the model but an average person.
This makes the
experiments cheaper and more straightforward because it is possible to find many
more people
for the validation phase of the model. In addition, it could help the widespread
adoption of such
solutions.
•Function level interpretability does not need people. It is assumed that anyone
could interpret
the resulting model. For example, everyone can understand how decision trees
work. However,
the limitation of functional level interpretability when working with text is
limited (Martinez-Gil
& Chaves-Gonzalez, 2020).
To start working, a system with good interpretability at the application level
should be sufficient.
At least for a professional in the sector. However, this is often not enough
because not all professionals
are specialized in the same legal field. So the aim here is to reach high levels
of interpretability
regarding the application level, human level, and function level at the same
time, as well as the need
for many fewer resources for training and the possibility of a simple transfer
learning stage. If not,
the user will not necessarily assume that the system has a good understanding of
the domain, which
could be used to explain the predictions. As a result, it will choose not to use
it.
This create big problems since the simplest regression-based models are easy for
users to interpret.
However, they do not lead to the best results. The opposite is true for models
of a neural nature.
Therefore, accuracy and interpretability are often considered orthogonal goals.
Therefore, building a
solution that reconciles the two features is far from being trivial.
29
To date, no fully interpretable LQA solution delivery highly accurate results
have been achieved.
However, the use of Legal Knowledge Graphs seems promising because they combine
the compu-
tational power of machine learning with the structured nature of knowledge bases
(in the form of
graphs). Time will tell if this family of methods can deliver what it promises.
In the meantime, users
of highly accurate neural LQA solutions can count on some frameworks that allow
them to get great
clues about how systems make the decisions they do (Ribeiro et al., 2016;
Lundberg & Lee, 2017).
5. Evaluation
Several datasets Sovrano et al. (2021) and performance metrics have been
proposed to evaluate
the results of a given QA system. According to Rodrigo et al. (Rodrigo & Pe˜nas,
2017) the method
to obtain the final score depends on the evaluation measure selected. Each
measure evaluates a
different set of features. Hence, researchers must select the measure depending
on the objectives of
the evaluation.
The most relevant metrics to evaluate the quality of LQA solutions are the
following: Without
ranking, ranking by question, ranking for all questions, cardinal confidence
self-score, and other met-
rics, for example, to evaluate the quality of the artificially generated text. We
are going to see them
in detail below.
5.1. Without Ranking
Although accuracy is a simple and intuitive measure, it only acknowledges
correct answers and
does not consider the amount of incorrect answers. However, its application is
very popular because
it is easy to understand.
accuracy =correct answers
total answers (1)
30
5.2. Ranking by question
When the system offers several answers in order of priority and what is of
interest is to average
the correct answer in the ordered list, the following Mean Average Precision
(MAP) is usually used.
The MAP compares the ground truth of correct answers to the answers offered by
the system and
returns a score. The higher the score, the more accurate the LQA solution is.
MAP =PQ
q=1 AveP (q)
Q(2)
whereby Qis the number of questions and AveP is defined as
AveP =Pn
k=1 P(k)×rel(k)
number of relevant documents (3)
whereby rel(k) is a function returning 1 if the answer is correct
It is also usual to use the Mean Reciprocal Rank (MRR). MRR is a statistic
measure for evaluating
any process that produces a list of possible answers to a set of queries,
ordered by the probability of
correctness.
MRR =1
|Q|
|Q|
X
i=1
1
ranki
(4)
When there is a list of questions, precision and recall are evaluated based on
the complete list
of known distinct instances of the answers. Precision is the fraction of answers
that are correct in
relation to the found ones:
precision ={f ound answers} ∩ {correct answers}
{found answers}(5)
The recall is the fraction of the found answers in relation to the correct ones.
31
recall ={f ound answers} ∩ {correct answers}
{correct answers}(6)
F-measure combines precision and recall into a harmonic mean to summarize both
metrics.
f−measure =2·precision ·recall
precision +recall (7)
Precision can be optimized at the expense of recall and vice versa. For this
reason, it is convenient
to report the two measures together, or at least, the combination of them
through an F-measure.
5.3. Ranking for all questions
The most popular metric for this scenario is the so-called Confidence Weighted
Score (CWS),
which is based on the notion of average precision. CWS requires LQA solution to
return only one
answer per question and rank all the answers according to the background truth.
Then, the CWS
metrics reward solutions returning correct answers at top positions in the
ranking.
CW S =1
|Q|
|Q|
X
i=1
C(i)
i(8)
whereby Qis the number of questions and Cis defined as
C=
i
X
j=1
I(j) (9)
being I(j) the function that returns 1 if answer j is correct and 0 if it is
not.
5.4. Cardinal Confidence Self-Score
The most popular metrics in this context are K and K1 (Peters et al., 2005). K
and K1 are based
on a utility function that returns -1 if the answer is incorrect and 1 if it is
correct. Both measures
weigh this value with the confidence score given by the LQA solution. A positive
value does not
32
indicate more correct answers than incorrect ones, but that the sum of scores
from correct answers is
higher than the sum of scores from incorrect answers (Rodrigo & Pe˜nas, 2017).
5.5. Unanswered different from incorrect
In the context of LQA solutions, most of the time, it is preferable not to
respond incorrectly
(Pe˜nas & Rodrigo, 2011). This is because it is usually unsuitable for confusing
the user with incorrect
answers that undermine the system’s trustworthiness. This idea is not new, but
despite several
previous attempts, there is no commonly accepted measure to assess non-response.
To deal with this situation, the metric Correctness at one c@1 has been
proposed.
c@1 = 1
total questions ·(correct answers +correct answers
total questions ·unanswered questions) (10)
5.6. Other metrics
In recent times, new alternative metrics have emerged to evaluate the quality of
responses (Chen
et al., 2019). These metrics are of particular relevance for some families of
LQA construction methods,
especially those that aim to generate a textual sentence and not just answer a
factual question. A
summary of these alternative metrics is shown below.
5.6.1. BERT Score and its variants
BERTScore is a family of alternative metrics that first tries to get the BERT
embeddings (Devlin
et al., 2019) of each term and calculate the current answer and the ground truth
through a BERT
model separately. Then, a mapping is computed between candidate and reference
terms by means of
pairwise cosine similarity. This mapping is then aggregated into precision and
recall scores, and then
into a harmonic mean.
33
5.6.2. Sentence Mover’s Similarity
SMS is another alternative metric based on mover’s distance to evaluate textual
information such as
machine-generated text. Its application in LQA assessment is done by firstly
computing an embedding
for each sentence in an answer as an average of the ELMo embeddings (Peters et
al., 2018). Then, a
mapping function is used to obtain the distance of moving a candidate answer’s
sentences to match
the reference answer.
5.6.3. N-gram based metrics
These are metrics developed for evaluating machine translation, e.g. BLEU,
ROUGE, etc. The
key idea is to evaluate a candidate sentence by determining the n-grams in the
candidate sentence
that also appear in the reference sentence (Chen et al., 2019). In the context
of LQA solutions, these
metrics can be adapted to measure the quality of the generated answer in
relation to the answer
specified in the ground truth.
5.7. Summary
In the following, we show a summary of the different families of methods to build
LQAs and the
most appropriate ways to automatically determine their quality. Table 10 shows a
tabular summary
of these correspondences.
For example, metrics such as BERTScore, SMS, and N-Gram do not make sense in the
Yes/No
Answer or LMCQA family of methods since the answers are known in advance and are
unnecessary
to generate them. Alternatively, when working with ontologies, the measure of
recall is much more
important than precision. Because if the answer was wrong, it is due to the
access to a wrong ontology
model rather than to the reasoning algorithm used to obtain the answer.
Moreover, ranking measures
do not make much sense either.
34
Table 10: Most useful evaluation methods to determine the accuracy of LQA
solutions
Family of Methods Evaluation
Yes/No Answers Acc, KK1, c@1
LMCQA Acc, PR, MAP, MRR, CWS, KK1, c@1
Classical IR-based solutions Acc, PR, MAP, MRR, CWS, KK1, c@1, BScore, SMS,
NGram
Big Data solutions Acc, PR, MAP, MRR, CWS, KK1, c@1
Ontology-based solutions Acc, Recall, KK1, c@1
Neural solutions Acc, PR, MAP, MRR, CWS, KK1, c@1, BScore, SMS, NGram
Other approaches Acc, PR, MAP, MRR, CWS, KK1, c@1, BScore, SMS, NGram
6. Open Research Challenges
Given the state-of-the-art and the open problems that need to be investigated,
it is possible to
identify some significant gaps that should be filled to facilitate the adoption of
LQA solutions at
scale. Among the Open Research Challenges (ORC) that can be identified are the
lack of proposals
for transfer learning in the legal sector, the lack of solutions to deal with
the complicated nuances of
legal language, the improvement of the integration of distributed and
heterogeneous data sources, as
well as the importance of the multilingual capabilities of the different
solutions. In the following, we
review each of them.
6.1. ORC1. Transfer Learning
One important concept related to knowledge representation is Transfer Learning.
These methods
are based on the possibility of using a given model to export it to other
scenarios of analogous
nature. In other words, the knowledge representation that has been learned to
complete one task
can be generalized to help complete other tasks. A suitable knowledge
representation method must
determine which factors and features will be exploited and thus reuse them in
another task.
Most traditional machine learning approaches can be used to build models capable
of addressing
35
various problems given enough data and time. However, in practice, the amount of
data and time
available are frequently limited. For this reason, transfer learning has
received considerable attention
from such communities as Deep Learning and Big Data communities to date. These
communities’
problems are very resource-intensive in the form of data and time. This fact
explains why strategies
of this kind are often seen as a way to alleviate such issues.
To date, transfer learning methods in the field of LQA solutions have been only
superficially
explored. Only the work of Yan et al. has attempted to bring solutions to the
table (Yan et al., 2020).
It will be necessary for the community to intensify its research in this area in
the near future. It is a
great way to boost the accuracy of solutions while reusing training data and
computational resources.
6.2. ORC2. Nuances of legal language
While in many application domains of QA systems, there is an apparent problem of
ambiguity,
i.e., most of the time, the questions are formulated so that a single statement
may assume different
meanings. In legal language, the problem is just the opposite. The language is
so precise that it
cannot give rise to multiple interpretations. Although this may seem a
facilitator a priori, this fact
brings a series of disadvantages that should be faced.
Classical stemming techniques cannot be freely used to obtain the root of the
terms. Thus, for
example, the sentence the contract is void is very different from the sentence
the contract is voidable.
Furthermore, even though the terms void and voidable contain the same root, a
similar meaning could
not be assumed.
This forces us to have optimal performance methods to really understand the
meaning of the
information being processed. In the near future, we will have to continue
working in this direction.
This would be facilitated by exploring legal texts’ inherent characteristics to
utilize these features for
building LQA solutions. For example, properties such as references between
documents or structured
relationships in legal statements should be investigated since they are often a
great help in processing
those different nuances that legal language brings.
36
6.3. ORC3. Distributed Knowledge
We have seen that although approaches based on structured corpora often yield
good results, it is
often difficult to use them in practice mainly due to the cost when building such
structures (i.e., it
is expensive in terms of effort, money, and time needed) and it is often
complicated to find experts
with enough knowledge for curating them. Therefore, it is usually desirable to
access all possible
sources to maximize the chances of success in answering a question while
minimizing the expenditure
of resources.
The problem is that the different data representations in the sources make
automatic aggregation,
interlinking, and integration very difficult. Therefore, the development of
efficient methods to prop-
erly access differently structured sources of legal nature, which can serve as a
knowledge source for
computer systems in search of the ideal answer to a given question, is a
critical challenge that should
be faced in the near future.
For these reasons, further research is necessary to make the task easier, by
different interlinking
corpora based on diverse matching methodologies or by transforming and properly
integrating other
data source types into novel knowledge models, e.g., linked data, knowledge,
graphs, etc. using some
semantic labeling techniques, or even creating a unique global representation
such as the universal
schema such as the one proposed in (Riedel et al., 2013).
6.4. ORC4. Multilingualism
Working with several languages simultaneously is a recurring problem, not only
in the design of
LQA solutions. Nevertheless, the truth is that in this domain, it is of
fundamental importance since
there is important legislation and regulations that are not always written in
the same language spoken
in the territories where they apply.
The good news in this regard is the latest advances in word vectorization, which
brings with it the
possibility of working with abstract representations that are
language-independent (Grave et al., 2018).
37
In this sense, representing words by vectors can be considered one of the main
recent breakthroughs
in deep learning for natural language processing (Goldberg, 2017). The benefits
of adopting vectors
are multiple since, in addition to working with abstract representations of
words, they have other
associated advantages, such as low computational complexity, because these
vectors are efficient to
compute using simple operations.
The advances in word representation have a substantial impact on semantic
analysis. The future
semantic analysis models of legal text will improve LQA performance thanks to
various deep learning
methods to build multilingual word embeddings from unstructured text. However,
to achieve a better
performance, it is necessary to explore practical approaches describing the
meaning of textual pieces,
such as sentences, paragraphs, or even documents, from a legal perspective.
7. Conclusions
QA technology is becoming an essential solution in many areas overloaded by the
constant gener-
ation of large amounts of information. Automatically answering specific questions
can contribute to
alleviating the problem of dealing with those vast amounts of data. In the legal
domain, good LQA
solutions are in high demand mainly due to the problems of information overload
that legal profession-
als have to face in their daily activities. As a result, several solutions for
LQA have been developed as
a response to that situation. The primary reason for that is that the capability
to answer questions
through computers automatically could help alleviate a problem involving tedious
tasks such as an
extensive information search that is, in general, time-consuming and
error-prone.
We have surveyed the different LQA solutions that have been proposed in the
literature to date.
To do that, we have proceeded with the analysis of the essential methods in LQA
over the last
decades. We have identified the data sources on which existing LQA solutions have
been implemented
to automatically answer questions of legal nature. We have seen that there is no
such solution that
can provide high accuracy, interpretability, and performance at the same time.
Nevertheless, there are
38
usually strategies capable of optimizing two of them at the expense of the
other, being interpretability
of significant importance in the legal domain since the community has always
found more useful
systems whereby a few hundredths more precision does not compensate for the
other disadvantages of
interpretability and performance. Furthermore, we have made an overview of the
most popular ways
to determine the quality of LQA solutions from a strictly quantitative point of
view. Last but not
least, we have made a review of the research challenges that are still open
today and that will have
to be faced by the community soon.
This concludes this survey of more than two decades of research efforts in the
field of LQA.
It is expected that soon, more and more proposals for highly accurate,
interpretable, and efficient
QA systems specialized in legal matters will see the light of day. It seems also
reasonable to think
that LQA solutions will not replace legal experts and their unique, specialized
knowledge. However,
all indications are that these solutions will undoubtedly and significantly
transform the traditional
delivery of legal services in the near future.
Competing interest
The author has no competing interest to declare.
Acknowledgments
This research work has been partially supported by the Austrian Ministry for
Transport, Innovation
and Technology, the Federal Ministry of Science, Research and Economy, and the
Province of Upper
Austria in the frame of the COMET center SCCH.
39
References
Atkinson, K., Bench-Capon, T. J. M., & Bollegala, D. (2020). Explanation in AI
and law: Past, present
and future. Artif. Intell.,289 , 103387. URL:
https://doi.org/10.1016/j.artint.2020.103387.
doi:10.1016/j.artint.2020.103387.
Aydin, B. I., Yilmaz, Y. S., Li, Y., Li, Q., Gao, J., & Demirbas, M. (2014).
Crowd-
sourcing for multiple-choice question answering. In C. E. Brodley, & P. Stone
(Eds.),
Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July
27 -
31, 2014, Qu´ebec City, Qu´ebec, Canada (pp. 2946–2953). AAAI Press. URL:
http://www.aaai.org/ocs/index.php/IAAI/IAAI14/paper/view/8634.
Bach, N. X., Cham, L. T. N., Thien, T. H. N., & Phuong, T. M. (2017). Question
analysis for
vietnamese legal question answering. In T. T. Nguyen, A. Le, S. Tojo, L. Nguyen,
& X. Phan (Eds.),
9th International Conference on Knowledge and Systems Engineering, KSE 2017,
Hue, Vietnam,
October 19-21, 2017 (pp. 154–159). IEEE. URL:
https://doi.org/10.1109/KSE.2017.8119451.
doi:10.1109/KSE.2017.8119451.
Bench-Capon, T. J. M., & Coenen, F. (1992). The maintenance of legal knowledge
based
systems. Artif. Intell. Rev.,6, 129–143. URL:
https://doi.org/10.1007/BF00150230.
doi:10.1007/BF00150230.
Bennett, Z., Russell-Rose, T., & Farmer, K. (2017). A scalable approach to legal
question an-
swering. In J. Keppens, & G. Governatori (Eds.), Proceedings of the 16th edition
of the Inter-
national Conference on Artificial Intelligence and Law, ICAIL 2017, London,
United Kingdom,
June 12-16, 2017 (pp. 269–270). ACM. URL:
https://doi.org/10.1145/3086512.3086547.
doi:10.1145/3086512.3086547.
Br¨uninghaus, S., & Ashley, K. D. (2001). Improving the representation of legal
case texts with infor-
mation extraction methods. In R. P. Loui (Ed.), Proceedings of the Eigths
International Conference
40
on Artificial Intelligence and Law, ICAIL 2001, St. Louis, Missouri, USA, May
21-25, 2001 (pp.
42–51). ACM. URL: https://doi.org/10.1145/383535.383540.
doi:10.1145/383535.383540.
Carvalho, D. S., Tran, V. D., Van Tran, K., Lai, V. D., & Nguyen, M.-L. (2016).
Lexical to discourse-
level corpus modeling for legal question answering. In Tenth International
Workshop on Juris-
Informatics (JURISIN).
Cer, D., Yang, Y., Kong, S., Hua, N., Limtiaco, N., John, R. S., Constant, N.,
Gua jardo-Cespedes,
M., Yuan, S., Tar, C., Strope, B., & Kurzweil, R. (2018). Universal sentence
encoder for en-
glish. In E. Blanco, & W. Lu (Eds.), Proceedings of the 2018 Conference on
Empirical Methods
in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels,
Belgium, Oc-
tober 31 - November 4, 2018 (pp. 169–174). Association for Computational
Linguistics. URL:
https://doi.org/10.18653/v1/d18-2029. doi:10.18653/v1/d18-2029.
Chalkidis, I., & Kampas, D. (2019). Deep learning in law: early adaptation and
le-
gal word embeddings trained on large corpora. Artif. Intell. Law,27 , 171–198.
URL:
https://doi.org/10.1007/s10506-018-9238-9. doi:10.1007/s10506-018-9238-9.
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019). Evaluating question
answering eval-
uation. In A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, & D. Chen (Eds.),
Proceedings of
the 2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019,
Hong
Kong, China, November 4, 2019 (pp. 119–124). Association for Computational
Linguistics. URL:
https://doi.org/10.18653/v1/D19-5817. doi:10.18653/v1/D19-5817.
Chitta, R., & Hudek, A. K. (2019). A reliable and accurate multiple choice
question answering
system for due diligence. In Proceedings of the Seventeenth International
Conference on Artificial
Intelligence and Law, ICAIL 2019, Montreal, QC, Canada, June 17-21, 2019 (pp.
184–188). ACM.
URL: https://doi.org/10.1145/3322640.3326711. doi:10.1145/3322640.3326711.
Cimiano, P., L´opez, V., Unger, C., Cabrio, E., Ngomo, A. N., & Walter, S.
(2013). Multilin-
41
gual question answering over linked data (QALD-3): lab overview. In P. Forner,
H. M¨uller,
R. Paredes, P. Rosso, & B. Stein (Eds.), Information Access Evaluation.
Multilinguality, Multi-
modality, and Visualization - 4th International Conference of the CLEF
Initiative, CLEF 2013,
Valencia, Spain, September 23-26, 2013. Proceedings (pp. 321–332). Springer
volume 8138 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-642-40802-1_30.
doi:10.1007/978-3-642-40802-1\_30.
Collarana, D., Heuss, T., Lehmann, J., Lytra, I., Maheshwari, G., Nedelchev, R.,
Schmidt, T., &
Trivedi, P. (2018). A question answering system on regulatory documents. In
JURIX (pp. 41–50).
Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Harshman, R.
A. (1990).
Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci.,41 , 391–407.
Delfino, P., Cuconato, B., Passos, G. P., Zaverucha, G., & Rademaker, A. (2018).
Using openwordnet-
pt for question answering on legal domain. In F. Bond, P. Vossen, & C. Fellbaum
(Eds.),
Proceedings of the 9th Global Wordnet Conference, GWC 2018, Nanyang
Technological Univer-
sity (NTU), Singapore, January 8-12, 2018 (pp. 105–112). Global Wordnet
Association. URL:
https://www.aclweb.org/anthology/2018.gwc-1.13/.
Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: pre-training of
deep bidirectional transformers for language understanding, . (pp. 4171–4186).
URL:
https://doi.org/10.18653/v1/n19-1423. doi:10.18653/v1/n19-1423.
Diefenbach, D., L´opez, V., Singh, K. D., & Maret, P. (2018). Core techniques of
question
answering systems over knowledge bases: a survey. Knowl. Inf. Syst.,55 ,
529–569. URL:
https://doi.org/10.1007/s10115-017-1100-y. doi:10.1007/s10115-017-1100-y.
Dimitrakis, E., Sgontzos, K., & Tzitzikas, Y. (2020). A survey on question
answer-
ing systems over linked data and documents. J. Intell. Inf. Syst.,55 , 233–259.
URL:
https://doi.org/10.1007/s10844-019-00584-7. doi:10.1007/s10844-019-00584-7.
42
Do, P., Nguyen, H., Tran, C., Nguyen, M., & Nguyen, M. (2017). Legal question
answer-
ing using ranking SVM and deep convolutional neural network. CoRR,abs/1703.05320
. URL:
http://arxiv.org/abs/1703.05320.arXiv:1703.05320.
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable
machine learning.
arXiv preprint arXiv:1702.08608 , .
Fawei, B., Pan, J. Z., Kollingbaum, M. J., & Wyner, A. Z. (2018). A methodology
for a criminal law
and procedure ontology for legal question answering. In R. Ichise, F. L´ecu´e,
T. Kawamura, D. Zhao,
S. Muggleton, & K. Kozaki (Eds.), Semantic Technology - 8th Joint International
Conference, JIST
2018, Awaji, Japan, November 26-28, 2018, Proceedings (pp. 198–214). Springer
volume 11341 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-030-04284-4_14.
doi:10.1007/978-3-030-04284-4\_14.
Fawei, B., Pan, J. Z., Kollingbaum, M. J., & Wyner, A. Z. (2019). A
semi-automated on-
tology construction for legal question answering. New Gener. Comput.,37 ,
453–478. URL:
https://doi.org/10.1007/s00354-019-00070-2. doi:10.1007/s00354-019-00070-2.
Ferrucci, D. A., Levas, A., Bagchi, S., Gondek, D., & Mueller, E. T. (2013).
Watson: Beyond jeop-
ardy! Artif. Intell.,199-200 , 93–105. URL:
https://doi.org/10.1016/j.artint.2012.06.009.
doi:10.1016/j.artint.2012.06.009.
Filtz, E. (2017). Building and processing a knowledge-graph for legal data. In
E. Blomqvist,
D. Maynard, A. Gangemi, R. Hoekstra, P. Hitzler, & O. Hartig (Eds.), The
Semantic Web -
14th International Conference, ESWC 2017, Portoroˇz, Slovenia, May 28 - June 1,
2017, Pro-
ceedings, Part II (pp. 184–194). volume 10250 of Lecture Notes in Computer
Science. URL:
https://doi.org/10.1007/978-3-319-58451- 5_13.
doi:10.1007/978-3-319-58451-5\_13.
Filtz, E., Kirrane, S., & Polleres, A. (2021). The linked legal data landscape:
linking legal data across
different countries. Artificial Intelligence and Law, (pp. 1–55).
43
Franco, W., Avila, C. V. S., Oliveira, A., Maia, G., Brayner, A., Vidal, V. M.
P., Carvalho,
F., & Pequeno, V. M. (2020). Ontology-based question answering systems over
knowledge
bases: A survey. In J. Filipe, M. Smialek, A. Brodsky, & S. Hammoudi (Eds.),
Proceed-
ings of the 22nd International Conference on Enterprise Information Systems,
ICEIS 2020,
Prague, Czech Republic, May 5-7, 2020, Volume 1 (pp. 532–539). SCITEPRESS. URL:
https://doi.org/10.5220/0009392205320539. doi:10.5220/0009392205320539.
Goldberg, Y. (2017). Neural Network Methods for Natural Language Process-
ing. Synthesis Lectures on Human Language Technologies. Morgan & Clay-
pool Publishers. URL: https://doi.org/10.2200/S00762ED1V01Y201703HLT037.
doi:10.2200/S00762ED1V01Y201703HLT037.
Grave, E., Bojanowski, P., Gupta, P., Joulin, A., & Mikolov, T. (2018). Learning
word vectors for
157 languages. arXiv preprint arXiv:1802.06893 , .
Han, L., Kashyap, A. L., Finin, T., Mayfield, J., & Weese, J. (2013). Umbc
ebiquity-core: Se-
mantic textual similarity systems. In M. T. Diab, T. Baldwin, & M. Baroni
(Eds.), Proceedings
of the Second Joint Conference on Lexical and Computational Semantics, *SEM
2013, June 13-
14, 2013, Atlanta, Georgia, USA (pp. 44–52). Association for Computational
Linguistics. URL:
https://www.aclweb.org/anthology/S13-1005/.
He, S., Liu, S., Chen, Y., Zhou, G., Liu, K., & Zhao, J. (2013). Casia@qald-3: A
question answering system over linked data. In P. Forner, R. Navigli, D. Tufis, &
N. Ferro (Eds.), Working Notes for CLEF 2013 Conference , Valencia, Spain,
Septem-
ber 23-26, 2013 . CEUR-WS.org volume 1179 of CEUR Workshop Proceedings. URL:
http://ceur-ws.org/Vol-1179/CLEF2013wn-QALD3-HeEt2013.pdf.
H¨offner, K., Walter, S., Marx, E., Usbeck, R., Lehmann, J., & Ngomo, A. N.
(2017). Survey
on challenges of question answering in the semantic web. Semantic Web,8,
895–920. URL:
https://doi.org/10.3233/SW-160247. doi:10.3233/SW-160247.
44
Holzenberger, N., Blair-Stanek, A., & Van Durme, B. (2020). A dataset for
statutory reasoning in
tax law entailment and question answering. arXiv preprint arXiv:2005.05257, .
Hoshino, R., Taniguchi, R., Kiyota, N., & Kano, Y. (2018). Question answering
system for le-
gal bar examination using predicate argument structure. In K. Kojima, M.
Sakamoto, K. Mi-
neshima, & K. Satoh (Eds.), New Frontiers in Artificial Intelligence - JSAI-isAI
2018 Workshops,
JURISIN, AI-Biz, SKL, LENLS, IDAA, Yokohama, Japan, November 12-14, 2018,
Revised Se-
lected Papers (pp. 207–220). Springer volume 11717 of Lecture Notes in Computer
Science. URL:
https://doi.org/10.1007/978-3-030-31605- 1_16.
doi:10.1007/978-3-030-31605-1\_16.
Huang, W., Jiang, J., Qu, Q., & Yang, M. (2020a). AILA: A question answering
system
in the legal domain. In C. Bessiere (Ed.), Proceedings of the Twenty-Ninth
International
Joint Conference on Artificial Intelligence, IJCAI 2020 (pp. 5258–5260).
ijcai.org. URL:
https://doi.org/10.24963/ijcai.2020/762. doi:10.24963/ijcai.2020/762.
Huang, W., Liao, X., Xie, Z., Qian, J., Zhuang, B., Wang, S., & Xiao, J.
(2020b). Generating reason-
able legal text through the combination of language modeling and question
answering. In C. Bessiere
(Ed.), Proceedings of the Twenty-Ninth International Joint Conference on
Artificial Intelligence,
IJCAI 2020 (pp. 3687–3693). ijcai.org. URL:
https://doi.org/10.24963/ijcai.2020/510.
doi:10.24963/ijcai.2020/510.
John, A., Caro, L. D., Boella, G., & Bartolini, C. (2016). An approach to in-
formation retrieval and question answering in the legal domain. In Proceedings
of the 10th International Workshop on Juris-informatics (JURISIN 2016). URL:
http://orbilu.uni.lu/bitstream/10993/29385/1/paper%2018.pdf.
Kano, Y., Hoshino, R., & Taniguchi, R. (2017). Analyzable legal yes/no question
answering system
using linguistic structures. In K. Satoh, M. Kim, Y. Kano, R. Goebel, & T.
Oliveira (Eds.), COL-
IEE 2017. 4th Competition on Legal Information Extraction and Entailment, held
in conjunction
with the 16th International Conference on Artificial Intelligence and Law (ICAIL
2017) in King’s
45
College London, UK (pp. 57–67). EasyChair volume 47 of EPiC Series in Computing
. URL:
https://easychair.org/publications/paper/zdP.
Kien, P. M., Nguyen, H., Bach, N. X., Tran, V., Nguyen, M. L., & Phuong, T. M.
(2020). Answering
legal questions by learning neural attentive text representation. In D. Scott,
N. Bel, & C. Zong
(Eds.), Proceedings of the 28th International Conference on Computational
Linguistics, COLING
2020, Barcelona, Spain (Online), December 8-13, 2020 (pp. 988–998).
International Commit-
tee on Computational Linguistics. URL:
https://doi.org/10.18653/v1/2020.coling-main.86.
doi:10.18653/v1/2020.coling-main.86.
Kim, M., & Goebel, R. (2017). Two-step cascaded textual entailment for legal bar
exam question
answering. In J. Keppens, & G. Governatori (Eds.), Proceedings of the 16th
edition of the In-
ternational Conference on Artificial Intelligence and Law, ICAIL 2017, London,
United Kingdom,
June 12-16, 2017 (pp. 283–290). ACM. URL:
https://doi.org/10.1145/3086512.3086550.
doi:10.1145/3086512.3086550.
Kim, M., Lu, Y., & Goebel, R. (2017). Textual entailment in legal bar exam
question answering using
deep siamese networks. In S. Arai, K. Kojima, K. Mineshima, D. Bekki, K. Satoh,
& Y. Ohta
(Eds.), New Frontiers in Artificial Intelligence - JSAI-isAI Workshops, JURISIN,
SKL, AI-Biz,
LENLS, AAA, SCIDOCA, kNeXI, Tsukuba, Tokyo, Japan, November 13-15, 2017, Revised
Se-
lected Papers (pp. 35–48). Springer volume 10838 of Lecture Notes in Computer
Science. URL:
https://doi.org/10.1007/978-3-319-93794- 6_3.
Kim, M., Xu, Y., & Goebel, R. (2014). Legal question answering using ranking SVM
and syn-
tactic/semantic similarity. In T. Murata, K. Mineshima, & D. Bekki (Eds.), New
Frontiers in
Artificial Intelligence - JSAI-isAI 2014 Workshops, LENLS, JURISIN, and GABA,
Kanagawa,
Japan, October 27-28, 2014, Revised Selected Papers (pp. 244–258). Springer
volume 9067 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-662-48119-6_18.
doi:10.1007/978-3-662-48119-6\_18.
46
Kim, M., Xu, Y., & Goebel, R. (2015). Applying a convolutional neural network to
legal ques-
tion answering. In M. Otake, S. Kurahashi, Y. Ota, K. Satoh, & D. Bekki (Eds.),
New Fron-
tiers in Artificial Intelligence - JSAI-isAI 2015 Workshops, LENLS, JURISIN, AAA,
HAT-
MASH, TSDAA, ASD-HR, and SKL, Kanagawa, Japan, November 16-18, 2015, Revised Se-
lected Papers (pp. 282–294). volume 10091 of Lecture Notes in Computer Science.
URL:
https://doi.org/10.1007/978-3-319-50953- 2_20.
doi:10.1007/978-3-319-50953-2\_20.
Kim, M., Xu, Y., Goebel, R., & Satoh, K. (2013). Answering yes/no questions in
legal bar ex-
ams. In Y. I. Nakano, K. Satoh, & D. Bekki (Eds.), New Frontiers in Artificial
Intelligence
- JSAI-isAI 2013 Workshops, LENLS, JURISIN, MiMI, AAA, and DDS, Kanagawa, Japan,
October 27-28, 2013, Revised Selected Papers (pp. 199–213). Springer volume 8417
of Lec-
ture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-319-10061-6_14.
doi:10.1007/978-3-319-10061-6\_14.
Kim, M., Xu, Y., Lu, Y., & Goebel, R. (2016). Question answering of bar exams by
para-
phrasing and legal text analysis. In S. Kurahashi, Y. Ohta, S. Arai, K. Satoh, &
D. Bekki
(Eds.), New Frontiers in Artificial Intelligence - JSAI-isAI 2016 Workshops,
LENLS, HAT-
MASH, AI-Biz, JURISIN and SKL, Kanagawa, Japan, November 14-16, 2016, Revised
Selected
Papers (pp. 299–313). Springer volume 10247 of Lecture Notes in Computer
Science. URL:
https://doi.org/10.1007/978-3-319-61572- 1_20.
doi:10.1007/978-3-319-61572-1\_20.
Kolomiyets, O., & Moens, M. (2011). A survey on question answering technol-
ogy from an information retrieval perspective. Inf. Sci.,181 , 5412–5434. URL:
https://doi.org/10.1016/j.ins.2011.07.047. doi:10.1016/j.ins.2011.07.047.
Kourtin, I., Mbarki, S., & Mouloudi, A. (2021). A legal question answering
ontology-based system. In
Formalising Natural Languages: Applications to Natural Language Processing and
Digital Humani-
ties: 14th International Conference, NooJ 2020, Zagreb, Croatia, June 5–7, 2020,
Revised Selected
Papers 14 (pp. 218–229). Springer International Publishing.
47
van Kuppevelt, D. E., & van Dijck, G. (2017). Answering legal research questions
about dutch
case law with network analysis and visualization. In A. Z. Wyner, & G. Casini
(Eds.), Legal
Knowledge and Information Systems - JURIX 2017: The Thirtieth Annual Conference,
Lux-
embourg, 13-15 December 2017 (pp. 95–100). IOS Press volume 302 of Frontiers in
Artifi-
cial Intelligence and Applications. URL:
https://doi.org/10.3233/978-1-61499-838-9-95.
doi:10.3233/978-1-61499-838-9-95.
Lame, G. (2004). Using NLP techniques to identify legal ontology components:
Concepts and re-
lations. Artif. Intell. Law,12 , 379–396. URL:
https://doi.org/10.1007/s10506-005-4160-3.
doi:10.1007/s10506- 005-4160-3.
Li, Y., Bandar, Z., & McLean, D. (2003). An approach for measuring semantic
similarity between
words using multiple information sources. IEEE Trans. Knowl. Data Eng.,15 ,
871–882. URL:
https://doi.org/10.1109/TKDE.2003.1209005. doi:10.1109/TKDE.2003.1209005.
Liu, L., & Luo, J. (2018). A question answering system based on deep learning.
In D. Huang,
M. M. Gromiha, K. Han, & A. Hussain (Eds.), Intelligent Computing Methodologies
- 14th
International Conference, ICIC 2018, Wuhan, China, August 15-18, 2018,
Proceedings, Part
III (pp. 173–181). Springer volume 10956 of Lecture Notes in Computer Science.
URL:
https://doi.org/10.1007/978-3-319-95957- 3_19.
doi:10.1007/978-3-319-95957-3\_19.
Lundberg, S. M., & Lee, S. (2017). A unified approach to interpreting model
predictions. In I. Guyon,
U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, & R.
Garnett (Eds.),
Advances in Neural Information Processing Systems 30: Annual Conference on
Neural Information
Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA (pp.
4765–4774). URL:
https://proceedings.neurips.cc/paper/2017/.
Martinez-Gil, J. (2015). Automated knowledge base management: A survey. Com-
put. Sci. Rev.,18, 1–9. URL: https://doi.org/10.1016/j.cosrev.2015.09.001.
doi:10.1016/j.cosrev.2015.09.001.
48
Martinez-Gil, J., & Chaves-Gonzalez, J. M. (2020). A novel method based on
symbolic regres-
sion for interpretable semantic similarity measurement. Expert Syst. Appl.,160,
113663. URL:
https://doi.org/10.1016/j.eswa.2020.113663. doi:10.1016/j.eswa.2020.113663.
Martinez-Gil, J., Freudenthaler, B., & Tjoa, A. M. (2019a). A general frame-
work for multiple choice question answering based on mutual information and
rein-
forced co-occurrence. Trans. Large Scale Data Knowl. Centered Syst.,42, 91–110.
URL:
https://doi.org/10.1007/978-3-662-60531- 8_4. doi:10.1007/978-3-662-60531-8\_4.
Martinez-Gil, J., Freudenthaler, B., & Tjoa, A. M. (2019b). Multiple choice
question answering
in the legal domain using reinforced co-occurrence. In S. Hartmann, J. K¨ung, S.
Chakravarthy,
G. Anderst-Kotsis, A. M. Tjoa, & I. Khalil (Eds.), Database and Expert Systems
Applications
- 30th International Conference, DEXA 2019, Linz, Austria, August 26-29, 2019,
Proceedings,
Part I (pp. 138–148). Springer volume 11706 of Lecture Notes in Computer
Science. URL:
https://doi.org/10.1007/978-3-030-27615- 7_10.
doi:10.1007/978-3-030-27615-7\_10.
Maxwell, K. T., & Schafer, B. (2008). Concept and context in legal information
retrieval. In
E. Francesconi, G. Sartor, & D. Tiscornia (Eds.), Legal Knowledge and
Information Systems -
JURIX 2008: The Twenty-First Annual Conference on Legal Knowledge and
Information Systems,
Florence, Italy, 10-13 December 2008 (pp. 63–72). IOS Press volume 189 of
Frontiers in Arti-
ficial Intelligence and Applications. URL:
https://doi.org/10.3233/978-1-58603-952-3-63.
doi:10.3233/978-1-58603-952-3-63.
McElvain, G., Sanchez, G., Matthews, S., Teo, D., Pompili, F., & Custis, T.
(2019a). West-
search plus: A non-factoid question-answering system for the legal domain. In B.
Pi-
wowarski, M. Chevalier, ´
E. Gaussier, Y. Maarek, J. Nie, & F. Scholer (Eds.), Proceedings
of the 42nd International ACM SIGIR Conference on Research and Development in
Informa-
tion Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019 (pp. 1361–1364).
ACM. URL:
https://doi.org/10.1145/3331184.3331397. doi:10.1145/3331184.3331397.
49
McElvain, G., Sanchez, G., Teo, D., & Custis, T. (2019b). Non-factoid question
answering in the
legal domain. In B. Piwowarski, M. Chevalier, ´
E. Gaussier, Y. Maarek, J. Nie, & F. Scholer (Eds.),
Proceedings of the 42nd International ACM SIGIR Conference on Research and
Development in
Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019 (pp.
1395–1396). ACM. URL:
https://doi.org/10.1145/3331184.3331431. doi:10.1145/3331184.3331431.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013).
Distributed representations
of words and phrases and their compositionality. In Advances in Neural
Information Processing
Systems 26: 27th Annual Conference on Neural Information Processing Systems
2013. Proceedings
of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States. (pp.
3111–3119).
Mimouni, N., Nazarenko, A., & Salotti, S. (2017). Answering complex queries on
legal networks:
A direct and a structured IR approaches. In U. Pagallo, M. Palmirani, P.
Casanovas, G. Sar-
tor, & S. Villata (Eds.), AI Approaches to the Complexity of Legal Systems -
AICOL Inter-
national Workshops 2015-2017: AICOL-VI@JURIX 2015, AICOL-VII@EKAW 2016, AICOL-
VIII@JURIX 2016, AICOL-IX@ICAIL 2017, and AICOL-X@JURIX 2017, Revised Selected
Pa-
pers (pp. 451–464). Springer volume 10791 of Lecture Notes in Computer Science.
URL:
https://doi.org/10.1007/978-3-030-00178- 0_31.
doi:10.1007/978-3-030-00178-0\_31.
Monroy, A., Calvo, H., & Gelbukh, A. F. (2008). Using graphs for shallow
question answering
on legal documents. In A. F. Gelbukh, & E. F. Morales (Eds.), MICAI 2008:
Advances in
Artificial Intelligence, 7th Mexican International Conference on Artificial
Intelligence, Atizap´an
de Zaragoza, Mexico, October 27-31, 2008, Proceedings (pp. 165–173). Springer
volume 5317 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-540-88636-5_15.
doi:10.1007/978-3-540-88636-5\_15.
Monroy, A., Calvo, H., & Gelbukh, A. F. (2009). NLP for shallow question
answering of legal
documents using graphs. In A. F. Gelbukh (Ed.), Computational Linguistics and
Intelligent Text
Processing, 10th International Conference, CICLing 2009, Mexico City, Mexico,
March 1-7, 2009.
50
Proceedings (pp. 498–508). Springer volume 5449 of Lecture Notes in Computer
Science. URL:
https://doi.org/10.1007/978-3-642-00382- 0_40.
doi:10.1007/978-3-642-00382-0\_40.
Morimoto, A., Kubo, D., Sato, M., Shindo, H., & Matsumoto, Y. (2017). Legal
question answering
system using neural attention. In K. Satoh, M. Kim, Y. Kano, R. Goebel, & T.
Oliveira (Eds.),
COLIEE 2017. 4th Competition on Legal Information Extraction and Entailment,
held in conjunc-
tion with the 16th International Conference on Artificial Intelligence and Law
(ICAIL 2017) in
King’s College London, UK (pp. 79–89). EasyChair volume 47 of EPiC Series in
Computing.
URL: https://easychair.org/publications/paper/pqm.
Nicula, B., Ruseti, S., & Rebedea, T. (2018). Improving deep learning for
multiple choice ques-
tion answering with candidate contexts. In G. Pasi, B. Piwowarski, L. Azzopardi,
& A. Han-
bury (Eds.), Advances in Information Retrieval - 40th European Conference on IR
Research, ECIR
2018, Grenoble, France, March 26-29, 2018, Proceedings (pp. 678–683). Springer
volume 10772 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-319-76941-7_62.
doi:10.1007/978-3-319-76941-7\_62.
Pe˜nas, A., Forner, P., Rodrigo, ´
A., Sutcliffe, R. F. E., Forascu, C., & Mota, C. (2010). Overview of
respubliqa 2010: Question answering evaluation over european legislation. In M.
Braschler, D. Har-
man, & E. Pianta (Eds.), CLEF 2010 LABs and Workshops, Notebook Papers, 22-23
Septem-
ber 2010, Padua, Italy. CEUR-WS.org volume 1176 of CEUR Workshop Proceedings.
URL:
http://ceur-ws.org/Vol-1176/CLEF2010wn-MLQA10-PenasEt2010.pdf.
Pe˜nas, A., Forner, P., Sutcliffe, R. F. E., Rodrigo, ´
A., Forascu, C., Alegria, I., Giampiccolo, D.,
Moreau, N., & Osenova, P. (2009). Overview of respubliqa 2009: Question
answering evaluation
over european legislation. In C. Peters, G. M. D. Nunzio, M. Kurimo, T. Mandl,
D. Mostefa,
A. Pe˜nas, & G. Roda (Eds.), Multilingual Information Access Evaluation I. Text
Retrieval Ex-
periments, 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009,
Corfu, Greece,
September 30 - October 2, 2009, Revised Selected Papers (pp. 174–196). Springer
volume 6241 of
51
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-642-15754-7_21.
doi:10.1007/978-3-642-15754-7\_21.
Pe˜nas, A., & Rodrigo, ´
A. (2011). A simple measure to assess non-response. In D. Lin, Y. Mat-
sumoto, & R. Mihalcea (Eds.), The 49th Annual Meeting of the Association for
Computational
Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24
June, 2011,
Portland, Oregon, USA (pp. 1415–1424). The Association for Computer Linguistics.
URL:
https://www.aclweb.org/anthology/P11-1142/.
Peters, C., Clough, P. D., Gonzalo, J., Jones, G. J. F., Kluck, M., & Magnini,
B. (Eds.) (2005).
Multilingual Information Access for Text, Speech and Images, 5th Workshop of the
Cross-Language
Evaluation Forum, CLEF 2004, Bath, UK, September 15-17, 2004, Revised Selected
Papers volume
3491 of Lecture Notes in Computer Science. Springer. URL:
https://doi.org/10.1007/b138934.
doi:10.1007/b138934.
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., &
Zettlemoyer, L. (2018).
Deep contextualized word representations. In Proceedings of NAACL-HLT (pp.
2227–2237).
Quaresma, P., & Rodrigues, I. (2005). A question-answering system for portuguese
juridical doc-
uments. In G. Sartor (Ed.), The Tenth International Conference on Artificial
Intelligence and
Law, Proceedings of the Conference, June 6-11, 2005, Bologna, Italy (pp.
256–257). ACM. URL:
https://doi.org/10.1145/1165485.1165536. doi:10.1145/1165485.1165536.
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). Squad: 100,000+
questions for machine
comprehension of text. In Proceedings of the 2016 Conference on Empirical
Methods in Natural
Language Processing (pp. 2383–2392).
Ravichander, A., Black, A. W., Wilson, S., Norton, T. B., & Sadeh, N. M. (2019).
Question answering
for privacy policies: Combining computational and legal perspectives. In K.
Inui, J. Jiang, V. Ng,
& X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in
Natural Language
52
Processing and the 9th International Joint Conference on Natural Language
Processing, EMNLP-
IJCNLP 2019, Hong Kong, China, November 3-7, 2019 (pp. 4946–4957). Association
for Computa-
tional Linguistics. URL: https://doi.org/10.18653/v1/D19-1500.
doi:10.18653/v1/D19-1500.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). ”why should I trust you?”:
Explaining the predictions
of any classifier. In B. Krishnapuram, M. Shah, A. J. Smola, C. C. Aggarwal, D.
Shen, & R. Rastogi
(Eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge
Discovery
and Data Mining, San Francisco, CA, USA, August 13-17, 2016 (pp. 1135–1144).
ACM. URL:
https://doi.org/10.1145/2939672.2939778. doi:10.1145/2939672.2939778.
Riedel, S., Yao, L., McCallum, A., & Marlin, B. M. (2013). Relation extraction
with matrix factoriza-
tion and universal schemas. In Proceedings of the 2013 Conference of the North
American Chapter
of the Association for Computational Linguistics: Human Language Technologies
(pp. 74–84).
Rodrigo, ´
A., & Pe˜nas, A. (2017). A study about the future evaluation of
question-answering sys-
tems. Knowl. Based Syst.,137 , 83–93. URL:
https://doi.org/10.1016/j.knosys.2017.09.015.
doi:10.1016/j.knosys.2017.09.015.
Rodrigo, ´
A., P´erez-Iglesias, J., Pe˜nas, A., Garrido, G., & Araujo, L. (2013). An-
swering questions about european legislation. Expert Syst. Appl.,40 , 5811–5816.
URL:
https://doi.org/10.1016/j.eswa.2013.05.008. doi:10.1016/j.eswa.2013.05.008.
da Silva, J. W. F., Venceslau, A. D. P., Sales, J. E., Maia, J. G. R., Pinheiro,
V. C. M.,
& Vidal, V. M. P. (2020). A short survey on end-to-end simple question answering
sys-
tems. Artif. Intell. Rev.,53 , 5429–5453. URL:
https://doi.org/10.1007/s10462-020-09826-5.
doi:10.1007/s10462- 020-09826-5.
Soares, M. A. C., & Parreiras, F. S. (2020). A literature review on question
answering tech-
niques, paradigms and systems. J. King Saud Univ. Comput. Inf. Sci.,32 ,
635–646. URL:
https://doi.org/10.1016/j.jksuci.2018.08.005. doi:10.1016/j.jksuci.2018.08.005.
53
Sovrano, F., Palmirani, M., Distefano, B., Sapienza, S., & Vitali, F. (2021). A
dataset for evaluating
legal question answering on private international law. In J. Maranh˜ao, & A. Z.
Wyner (Eds.), ICAIL
’21: Eighteenth International Conference for Artificial Intelligence and Law,
S˜ao Paulo Brazil,
June 21 - 25, 2021 (pp. 230–234). ACM. URL:
https://doi.org/10.1145/3462757.3466094.
doi:10.1145/3462757.3466094.
Sovrano, F., Palmirani, M., & Vitali, F. (2020). Legal knowledge extraction for
knowledge graph
based question-answering. In V. Serena, J. Harasta, & P. Kremen (Eds.), Legal
Knowledge and
Information Systems - JURIX 2020: The Thirty-third Annual Conference, Brno,
Czech Republic,
December 9-11, 2020 (pp. 143–153). IOS Press volume 334 of Frontiers in
Artificial Intelligence
and Applications. URL: https://doi.org/10.3233/FAIA200858.
doi:10.3233/FAIA200858.
Taniguchi, R., Hoshino, R., & Kano, Y. (2018). Legal question answering system
using framenet.
In K. Kojima, M. Sakamoto, K. Mineshima, & K. Satoh (Eds.), New Frontiers in
Artificial
Intelligence - JSAI-isAI 2018 Workshops, JURISIN, AI-Biz, SKL, LENLS, IDAA,
Yokohama,
Japan, November 12-14, 2018, Revised Selected Papers (pp. 193–206). Springer
volume 11717 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-030-31605-1_15.
doi:10.1007/978-3-030-31605-1\_15.
Taniguchi, R., & Kano, Y. (2016). Legal yes/no question answering system using
case-role analysis.
In S. Kurahashi, Y. Ohta, S. Arai, K. Satoh, & D. Bekki (Eds.), New Frontiers in
Artificial Intelli-
gence - JSAI-isAI 2016 Workshops, LENLS, HAT-MASH, AI-Biz, JURISIN and SKL,
Kanagawa,
Japan, November 14-16, 2016, Revised Selected Papers (pp. 284–298). Springer
volume 10247 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-319-61572-1_19.
doi:10.1007/978-3-319-61572-1\_19.
Tesauro, G., Gondek, D., Lenchner, J., Fan, J., & Prager, J. M. (2013). Analysis
of
watson’s strategies for playing jeopardy! J. Artif. Intell. Res.,47, 205–251.
URL:
https://doi.org/10.1613/jair.3834. doi:10.1613/jair.3834.
54
Tong, P., Zhang, Q., & Yao, J. (2019). Leveraging domain context for question
answering over
knowledge graph. Data Science and Engineering,4, 323–335.
Tran, O. T., Bach, N. X., Nguyen, M. L., & Shimazu, A. (2013). Answering legal
questions by mining
reference information. In Y. I. Nakano, K. Satoh, & D. Bekki (Eds.), New
Frontiers in Artificial
Intelligence - JSAI-isAI 2013 Workshops, LENLS, JURISIN, MiMI, AAA, and DDS,
Kanagawa,
Japan, October 27-28, 2013, Revised Selected Papers (pp. 214–229). Springer
volume 8417 of
Lecture Notes in Computer Science. URL:
https://doi.org/10.1007/978-3-319-10061-6_15.
doi:10.1007/978-3-319-10061-6\_15.
Veena, G., Gupta, D., Anil, A., & Akhil, S. (2019). An ontology driven question
answering system for
legal documents. In 2019 2nd International Conference on Intelligent Computing,
Instrumentation
and Control Technologies (ICICICT) (pp. 947–951). IEEE volume 1.
Verma, A., Morato, J., Jain, A., & Arora, A. (2020). Relevant subsection
retrieval for law domain
question answer system. In Data Visualization and Knowledge Engineering (pp.
299–319). Springer.
Wang, M. (2006). A survey of answer extraction techniques in factoid question
answering. Computa-
tional Linguistics,1, 1–14.
Wehnert, S., Hoque, S. A., Fenske, W., & Saake, G. (2019). Threshold-based
retrieval and textual
entailment detection on legal bar exam questions. arXiv preprint
arXiv:1905.13350 , .
Wu, J., Liu, J., & Luo, X. (2020). Few-shot legal knowledge question answering
system for covid-19
epidemic. In 2020 3rd International Conference on Algorithms, Computing and
Artificial Intelli-
gence (pp. 1–6).
Xiao, G., Mo, J., Chow, E., Chen, H., Guo, J., & Gong, Z. (2017). Multi-task CNN
for classification
of chinese legal questions. In O. Hussain, L. Jiang, X. Fei, C. Lan, & K. Chao
(Eds.), 14th IEEE
International Conference on e-Business Engineering, ICEBE 2017, Shanghai, China,
November 4-
55
6, 2017 (pp. 84–90). IEEE Computer Society. URL:
https://doi.org/10.1109/ICEBE.2017.22.
doi:10.1109/ICEBE.2017.22.
Xu, K., Reddy, S., Feng, Y., Huang, S., & Zhao, D. (2016). Question answering on
freebase via relation
extraction and textual evidence. In Proceedings of the 54th Annual Meeting of
the Association for
Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume
1: Long Pa-
pers. The Association for Computer Linguistics. URL:
https://doi.org/10.18653/v1/p16-1220.
doi:10.18653/v1/p16-1220.
Yan, M., Zhang, H., Jin, D., & Zhou, J. T. (2020). Multi-source meta transfer
for low resource
multiple-choice question answering. In D. Jurafsky, J. Chai, N. Schluter, & J.
R. Tetreault (Eds.),
Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, ACL
2020, Online, July 5-10, 2020 (pp. 7331–7341). Association for Computational
Linguistics. URL:
https://doi.org/10.18653/v1/2020.acl-main.654.
doi:10.18653/v1/2020.acl-main.654.
Yih, W., Chang, M., Meek, C., & Pastusiak, A. (2013). Question answering using
enhanced lexical se-
mantic models. In Proceedings of the 51st Annual Meeting of the Association for
Computational Lin-
guistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers (pp.
1744–1753). The
Association for Computer Linguistics. URL:
https://www.aclweb.org/anthology/P13-1171/.
Zhang, N.-N., & Xing, Y. (2021). Questions and answers on legal texts based on
bert-bigru. In Journal
of Physics: Conference Series (p. 012035). IOP Publishing volume 1828.
Zhong, B., He, W., Huang, Z., Love, P. E. D., Tang, J., & Luo, H. (2020a). A
building regulation
question answering system: A deep learning methodology. Adv. Eng. Informatics,46
, 101195. URL:
https://doi.org/10.1016/j.aei.2020.101195. doi:10.1016/j.aei.2020.101195.
Zhong, H., Wang, Y., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020b). Iteratively
questioning and
answering for interpretable legal judgment prediction. In The Thirty-Fourth AAAI
Conference on
Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of
Artificial Intelli-
56
gence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in
Artificial
Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 (pp. 1250–1257).
AAAI Press.
URL: https://aaai.org/ojs/index.php/AAAI/article/view/5479.
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020c). JEC-QA: A
legal-domain
question answering dataset. In The Thirty-Fourth AAAI Conference on Artificial
Intelligence,
AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence
Conference,
IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial
Intelligence,
EAAI 2020, New York, NY, USA, February 7-12, 2020 (pp. 9701–9708). AAAI Press.
URL:
https://aaai.org/ojs/index.php/AAAI/article/view/6519.
Zou, L., Huang, R., Wang, H., Yu, J. X., He, W., & Zhao, D. (2014). Natural
language question
answering over RDF: a graph data driven approach. In C. E. Dyreson, F. Li, & M.
T. ¨
Ozsu
(Eds.), International Conference on Management of Data, SIGMOD 2014, Snowbird,
UT, USA,
June 22-27, 2014 (pp. 313–324). ACM. URL:
https://doi.org/10.1145/2588555.2610525.
doi:10.1145/2588555.2610525.
57
CITATIONS (0)
REFERENCES (95)
ResearchGate has not been able to resolve any citations for this publication.
The linked legal data landscape: linking legal data across different countries
Article
Full-text available
* Dec 2021
* Artif Intell Law
* Erwin Filtz
* Sabrina Kirrane
* Axel Polleres
The European Union is working towards harmonizing legislation across Europe, in
order to improve cross-border interchange of legal information. This goal is
supported for instance via standards such as the European Law Identifier (ELI)
and the European Case Law Identifier (ECLI), which provide technical
specifications for Web identifiers and suggestions for vocabularies to be used
to describe metadata pertaining to legal documents in a machine readable format.
Notably, these ECLI and ELI metadata standards adhere to the RDF data format
which forms the basis of Linked Data, and therefore have the potential to form a
basis for a pan-European legal Knowledge Graph. Unfortunately, to date said
specifications have only been partially adopted by EU member states. In this
paper we describe a methodology to transform the existing legal information
system used in Austria to such a legal knowledge graph covering different steps
from modeling national specific aspects, to population, and finally the
integration of legal data from other countries through linked data. We
demonstrate the usefulness of this approach by exemplifying practical use cases
from legal information search, which are not possible in an automated fashion so
far.
View
Show abstract
Answering Legal Questions by Learning Neural Attentive Text Representation
Conference Paper
Full-text available
* Jan 2020
* Phi Manh Kien
* Ha-Thanh Nguyen
* Ngo Xuan Bach
* Tu Minh Phuong
View
Legal Knowledge Extraction for Knowledge Graph Based Question-Answering
Chapter
Full-text available
* Dec 2020
* Francesco Sovrano
* Monica Palmirani
* Fabio Vitali
This paper presents the Open Knowledge Extraction (OKE) tools combined with
natural language analysis of the sentence in order to enrich the semantic of the
legal knowledge extracted from legal text. In particular the use case is on
international private law with specific regard to the Rome I Regulation EC
593/2008, Rome II Regulation EC 864/2007, and Brussels I bis Regulation EU
1215/2012. A Knowledge Graph (KG) is built using OKE and Natural Language
Processing (NLP) methods jointly with the main ontology design patterns defined
for the legal domain (e.g., event, time, role, agent, right, obligations,
jurisdiction). Using critical questions, underlined by legal experts in the
domain, we have built a question answering tool capable to support the
information retrieval and to answer to these queries. The system should help the
legal expert to retrieve the relevant legal information connected with topics,
concepts, entities, normative references in order to integrate his/her searching
activities.
View
Show abstract
Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering
Conference Paper
Full-text available
* Jan 2020
* Ming Yan
* Hao Zhang
* Di Jin
* Joey Tianyi Zhou
View
A dataset for evaluating legal question answering on private international law
Conference Paper
* Jun 2021
* Francesco Sovrano
* Monica Palmirani
* Biagio Distefano
* Fabio Vitali
View
Few-Shot Legal Knowledge Question Answering System for COVID-19 Epidemic
Conference Paper
* Dec 2020
* Jiaye Wu
* Jie Liu
* Xudong Luo
View
A building regulation question answering system: A deep learning methodology
Article
* Nov 2020
* ADV ENG INFORM
* Botao Zhong
* Wanlei He
* Ziwei Huang
* Hanbin Luo
Regulations play an important role in assuring the quality of a building’s
construction and minimizing its adverse environmental impacts. Engineers and the
like need to retrieve regulatory information to ensure a building conforms to
specified standards. Despite the availability of search engines and digital
databases that can be used to store regulations, engineers, for example, are
unable to retrieve information for domain-specific needs in a timely manner. As
a consequence, users often have to deal with the burden of browsing and
filtering information, which can be a time-consuming process. This research
develops a robust end-to-end methodology to improve the efficiency and
effectiveness of retrieving queries pertaining to building regulations. The
developed methodology integrates information retrieval with a deep learning
model of Natural Language Processing (NLP) to provide precise and rapid answers
to user’s questions from a collection of building regulations. The methodology
is evaluated and a prototype system to retrieve queries is developed. The
paper’s contribution is therefore twofold as it develops a: (1) methodology that
combines NLP and deep learning to be able to address queries raised about the
building regulations; and (2) chatbot of question answering system, which we
refer to as QAS4CQAR. Our proposed methodology has powerful feature
representation and learning capability and therefore can potentially be adopted
to building regulations in other jurisdictions.
View
Show abstract
Explanation in AI and law: Past, present and future
Article
* Dec 2020
* ARTIF INTELL
* Katie Atkinson
* Trevor Bench-Capon
* Danushka Bollegala
Explanation has been a central feature of AI systems for legal reasoning since
their inception. Recently, the topic of explanation of decisions has taken on a
new urgency, throughout AI in general, with the increasing deployment of AI
tools and the need for lay users to be able to place trust in the decisions that
the support tools are recommending. This paper provides a comprehensive review
of the variety of techniques for explanation that have been developed in AI and
Law. We summarise the early contributions and how these have since developed. We
describe a number of notable current methods for automated explanation of legal
reasoning and we also highlight gaps that must be addressed by future systems to
ensure that accurate, trustworthy, unbiased decision support can be provided to
legal professionals. We believe that insights from AI and Law, where explanation
has long been a concern, may provide useful pointers for future development of
explainable AI.
View
Show abstract
AILA: A Question Answering System in the Legal Domain
Conference Paper
* Jul 2020
* Weiyi Huang
* Jiahao Jiang
* Qiang Qu
* Min Yang
Question answering (QA) in the legal domain has gained increasing popularity for
people to seek legal advice. However, existing QA systems struggle to comprehend
the legal context and provide jurisdictionally relevant answers due to the lack
of domain expertise. In this paper, we develop an Artificial Intelligence Law
Assistant (AILA) for question answering in the domain of Chinese laws. AILA
system automatically comprehends users' natural language queries with the help
of the legal knowledge graph (KG) and provides the best matching answers for
given queries. In addition, AILA provides visual cues to interpret the input
queries and candidate answers based on the legal KG. Experimental results on a
large-scale legal QA corpus show the effectiveness of AILA. To the best of our
knowledge, AILA is the first Chinese legal QA system which integrates the domain
knowledge from legal KG to comprehend the questions and answers for ranking QA
pairs. AILA is available at http://bmilab.ticp.io:48478/.
View
Show abstract
A novel method based on symbolic regression for interpretable semantic
similarity measurement
Article
* Jun 2020
* EXPERT SYST APPL
* Jorge Martinez-Gil
* Jose M. Chaves-González
The problem of automatically measuring the degree of semantic similarity between
textual expressions is a challenge that consists of calculating the degree of
likeness between two text fragments that have none or few features in common
according to human judgment. In recent times, several machine learning methods
have been able to establish a new state-of-the-art regarding the accuracy, but
none or little attention has been paid to their interpretability, i.e. the
extent to which an end-user could be able to understand the cause of the output
from these approaches. Although such solutions based on symbolic regression
already exist in the field of clustering (Lensen et al., 2019), we propose here
a new approach which is being able to reach high levels of interpretability
without sacrificing accuracy in the context of semantic textual similarity.
After a complete empirical evaluation using several benchmark datasets, it is
shown that our approach yields promising results in a wide range of scenarios.
View
Show abstract
Show more
RECOMMENDATIONS
Discover more
Project
MPCENERGY
* Georgios C. Chasparis
* Thomas Natschläger
* Thomas Grubinger
* [...]
* Christa Illibauer
View project
Project
ACCURATE AND EFFICIENT PROFILE MATCHING IN KNOWLEDGE BASES
* Jorge Martinez-Gil
Accurate matching of job offers and applicant profiles
View project
Project
SEMANTIC SIMILARITY MEASUREMENT
* Jorge Martinez-Gil
View project
Project
ONTOLOGY MATCHING AND META-MATCHING
* Jorge Martinez-Gil
* Jose F Aldana Montes
* Ismael Navas Delgado
View project
Article
AUGMENTED AND CHALLENGING DATASETS WITH MULTI-STEP REASONING AND MULTI-SPAN
QUESTIONS FOR CHINESE JU...
December 2022
* Qingye Meng
* Ziyue Wang
* Hang Chen
* [...]
* Shijin Wang
The existing judicial reading comprehension datasets are relatively simple, and
the answers to the questions can be obtained through single-step reasoning.
However, the content of legal documents in actual scenarios is complex, making
it problematic to infer correct results merely by single-step reasoning. To
solve this type of issue, we promote the difficulties of questions included in
Chinese ... [Show full abstract] Judicial Reading Comprehension (CJRC) dataset
and propose two augmented versions, CJRC2.0 and CJRC3.0. These datasets are
derived from Chinese judicial judgment documents in different fields and
annotated by judicial professionals. Compared to CJRC, there are more types of
judgment documents in the two datasets, and the questions become are more
challenging to answer. For CJRC2.0, we only preserve complex questions that
require to be solved by multi-step reasoning. Besides, we provide additional
supporting facts to the answers. For CJRC3.0, we introduce a new question type,
the multi-span question, which should be answered by extracting and combining
multiple spans in the documents. We implement two powerful baselines to evaluate
the difficulty of our proposed datasets. Our proposed datasets fill gaps in the
field of explainable legal machine reading comprehension.
Read more
Conference Paper
Full-text available
QUESTION ANSWERING OF BAR EXAMS BY PARAPHRASING AND LEGAL TEXT ANALYSIS
July 2017 · Lecture Notes in Computer Science
* mi-young Kim
* Ying Xu
* Yao Lu
* Randy Goebel
Our legal question answering system combines legal information retrieval and
textual entailment, and exploits paraphrasing and sentence-level analysis of
queries and legal statutes. We have evaluated our system using the training data
from the competition on legal information extraction/entailment (COLIEE)-2016.
The competition focuses on the legal information processing required to answer
yes/no ... [Show full abstract] questions from Japanese legal bar exams, and it
consists of three phases: legal ad-hoc information retrieval (Phase 1), textual
entailment (Phase 2), and a combination of information retrieval and textual
entailment (Phase 3). Phase 1 requires the identification of Japan civil law
articles relevant to a legal bar exam query. For this phase, we have used an
information retrieval approach using TF-IDF and a Ranking SVM. Phase 2 requires
decision on yes/no answer for previously unseen queries, which we approach by
comparing the approximate meanings of queries with relevant articles. Our
meaning extraction process uses a selection of features based on a kind of
paraphrase, coupled with a condition/conclusion/exception analysis of articles
and queries. We also identify synonym relations using word embedding, and detect
negation patterns from the articles. Our heuristic selection of attributes is
used to build an SVM model, which provides the basis for ranking a decision on
the yes/no questions. Experimental evaluation show that our method outperforms
previous methods. Our result ranked highest in the Phase 3 in the COLIEE-2016
competition.
View full-text
Chapter
COLIEE-2018: EVALUATION OF THE COMPETITION ON LEGAL INFORMATION EXTRACTION AND
ENTAILMENT
October 2019
* Yoshinobu Kano
* mi-young Kim
* Masaharu Yoshioka
* [...]
* Ken Satoh
We summarize the evaluation of the 5th Competition on Legal Information
Extraction/Entailment 2018 (COLIEE-2018). The COLIEE-2018 tasks include two
tasks in each of statute law and case law. The case law component includes an
information retrieval (Task 1), and the confirmation of an entailment relation
between an existing case and an unseen case (Task 2). The statute law component
includes ... [Show full abstract] information retrieval (Task 3) and
entailment/question answering (Task 4). Participation was open to any group
based on any approach. 13 teams participated in the case law competition, and we
received results from 7 teams where 6 submissions to Task 1 (12 runs), and 4
submissions to Task 2 (8 runs). Regarding the statute law, there were
submissions of 17 runs from 8 teams (including 2 organizers’ runs) for Task 3
and 7 runs from 3 teams for Task 4. We describe each team’s approaches, our
official evaluation, and analysis on our data and submission results. We also
discuss possibilities for future competition tasks.
Read more
Chapter
MULTIPLE CHOICE QUESTION ANSWERING IN THE LEGAL DOMAIN USING REINFORCED
CO-OCCURRENCE
August 2019
* Jorge Martinez-Gil
* Bernhard Freudenthaler
* A Min Tjoa
Nowadays, the volume of legal information available is continuously growing. As
a result, browsing and querying this huge legal corpus in search of specific
information is currently a tedious task exacerbated by the fact that data
presentation does not usually meet the needs of professionals in the sector. To
satisfy these ever-increasing needs, we have designed an appropriate solution to
provide ... [Show full abstract] an adaptive and intelligent solution for the
automatic answer of questions of legal content based on the computation of
reinforced co-occurrence, i.e. a very demanding type of co-occurrence that
requires large volumes of information but guarantees good results. This solution
is based on the pattern-based methods that have been already successfully
applied in information extraction research. An empirical evaluation over a
dataset of legal questions seems to indicate that this solution is promising.
Read more
Discover the world's research
Join ResearchGate to find the people and research you need to help your work.
Join for free
ResearchGate iOS App
Get it from the App Store now.
Install
Keep up with your stats and more
Access scientific knowledge from anywhere
or
Discover by subject area
* Recruit researchers
* Join for free
* Login
Email
Tip: Most researchers use their institutional email address as their
ResearchGate login
PasswordForgot password?
Keep me logged in
Log in
or
Continue with Google
Welcome back! Please log in.
Email
· Hint
Tip: Most researchers use their institutional email address as their
ResearchGate login
PasswordForgot password?
Keep me logged in
Log in
or
Continue with Google
No account? Sign up
Company
About us
News
Careers
Support
Help Center
Business solutions
Advertising
Recruiting
© 2008-2023 ResearchGate GmbH. All rights reserved.
* Terms
* Privacy
* Copyright
* Imprint