arxiv.org Open in urlscan Pro
151.101.3.42  Public Scan

Form analysis
2 forms found in the DOM

GET https://arxiv.org/search

<form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
  <div class="field has-addons">
    <div class="control">
      <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms">
      <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
    </div>
    <div class="control">
      <div class="select is-small">
        <select name="searchtype" aria-label="Field to search">
          <option value="all" selected="selected">All fields</option>
          <option value="title">Title</option>
          <option value="author">Author</option>
          <option value="abstract">Abstract</option>
          <option value="comments">Comments</option>
          <option value="journal_ref">Journal reference</option>
          <option value="acm_class">ACM classification</option>
          <option value="msc_class">MSC classification</option>
          <option value="report_num">Report number</option>
          <option value="paper_id">arXiv identifier</option>
          <option value="doi">DOI</option>
          <option value="orcid">ORCID</option>
          <option value="author_id">arXiv author ID</option>
          <option value="help">Help pages</option>
          <option value="full_text">Full text</option>
        </select>
      </div>
    </div>
    <input type="hidden" name="source" value="header">
    <button class="button is-small is-cul-darker">Search</button>
  </div>
</form>

GET https://arxiv.org/search

<form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
  <div class="field has-addons">
    <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms">
    <input type="hidden" name="source" value="header">
    <input type="hidden" name="searchtype" value="all">
    <button class="button">GO</button>
  </div>
</form>

Text Content

Skip to main content
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
> cs > arXiv:1906.08237

Help | Advanced Search

All fields Title Author Abstract Comments Journal reference ACM classification
MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help
pages Full text
Search
open search
GO
open navigation menu


QUICK LINKS

 * Login
 * Help Pages
 * About


COMPUTER SCIENCE > COMPUTATION AND LANGUAGE

arXiv:1906.08237 (cs)
[Submitted on 19 Jun 2019 (v1), last revised 2 Jan 2020 (this version, v2)]


TITLE:XLNET: GENERALIZED AUTOREGRESSIVE PRETRAINING FOR LANGUAGE UNDERSTANDING

Authors:Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan
Salakhutdinov, Quoc V. Le
View a PDF of the paper titled XLNet: Generalized Autoregressive Pretraining for
Language Understanding, by Zhilin Yang and 5 other authors
View PDF

> Abstract:With the capability of modeling bidirectional contexts, denoising
> autoencoding based pretraining like BERT achieves better performance than
> pretraining approaches based on autoregressive language modeling. However,
> relying on corrupting the input with masks, BERT neglects dependency between
> the masked positions and suffers from a pretrain-finetune discrepancy. In
> light of these pros and cons, we propose XLNet, a generalized autoregressive
> pretraining method that (1) enables learning bidirectional contexts by
> maximizing the expected likelihood over all permutations of the factorization
> order and (2) overcomes the limitations of BERT thanks to its autoregressive
> formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the
> state-of-the-art autoregressive model, into pretraining. Empirically, under
> comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a
> large margin, including question answering, natural language inference,
> sentiment analysis, and document ranking.

Comments: Pretrained models and code are available at this https URL Subjects:
Computation and Language (cs.CL); Machine Learning (cs.LG) Cite as:
arXiv:1906.08237 [cs.CL]   (or arXiv:1906.08237v2 [cs.CL] for this version)  
https://doi.org/10.48550/arXiv.1906.08237
Focus to learn more
arXiv-issued DOI via DataCite


SUBMISSION HISTORY

From: Zhilin Yang [view email]
[v1] Wed, 19 Jun 2019 17:35:48 UTC (264 KB)
[v2] Thu, 2 Jan 2020 12:48:08 UTC (2,662 KB)

Full-text links:


ACCESS PAPER:

   View a PDF of the paper titled XLNet: Generalized Autoregressive Pretraining
   for Language Understanding, by Zhilin Yang and 5 other authors
 * View PDF
 * TeX Source
 * Other Formats

view license
Current browse context:
cs.CL
< prev   |   next >

new | recent | 2019-06
Change to browse by:
cs
cs.LG



REFERENCES & CITATIONS

 * NASA ADS
 * Google Scholar
 * Semantic Scholar




7 BLOG LINKS

(what is this?)


DBLP - CS BIBLIOGRAPHY

listing | bibtex
Zhilin Yang
Zihang Dai
Yiming Yang
Jaime G. Carbonell
Ruslan Salakhutdinov
…
a export BibTeX citation Loading...


BIBTEX FORMATTED CITATION

×
loading...
Data provided by:


BOOKMARK

Bibliographic Tools


BIBLIOGRAPHIC AND CITATION TOOLS

Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)



Code, Data, Media


CODE, DATA AND MEDIA ASSOCIATED WITH THIS ARTICLE

Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Links to Code Toggle
Papers with Code (What is Papers with Code?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)






Demos


DEMOS

Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)



Related Papers


RECOMMENDERS AND SEARCH TOOLS

Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Core recommender toggle
CORE Recommender (What is CORE?)

 * Author
 * Venue
 * Institution
 * Topic





About arXivLabs


ARXIVLABS: EXPERIMENTAL PROJECTS WITH COMMUNITY COLLABORATORS

arXivLabs is a framework that allows collaborators to develop and share new
arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and
accepted our values of openness, community, excellence, and user data privacy.
arXiv is committed to these values and only works with partners that adhere to
them.

Have an idea for a project that will add value for arXiv's community? Learn more
about arXivLabs.



Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help

 * contact arXivClick here to contact arXiv Contact
 * subscribe to arXiv mailingsClick here to subscribe Subscribe

 * Copyright
 * Privacy Policy

 * Web Accessibility Assistance

 * arXiv Operational Status
   Get status notifications via email or slack