papers.nips.cc Open in urlscan Pro
198.202.70.94  Public Scan

Form analysis
1 forms found in the DOM

GET /papers/search

<form class="form-inline my-2 my-lg-0" method="get" role="search" action="/papers/search">
  <input class="form-control mr-sm-2" type="text" name="q" placeholder="Search" aria-label="Search" id="navsearch">
  <button class="btn btn-outline-success my-2 my-sm-0" type="submit">Search</button>
</form>

Text Content

NeurIPS Proceedings
 * 
 * 

Search


NEURAL WORD EMBEDDING AS IMPLICIT MATRIX FACTORIZATION

Part of Advances in Neural Information Processing Systems 27 (NIPS 2014)

Bibtex Metadata Paper Reviews


AUTHORS

Omer Levy, Yoav Goldberg

ABSTRACT



We analyze skip-gram with negative-sampling (SGNS), a word embedding method
introduced by Mikolov et al., and show that it is implicitly factorizing a
word-context matrix, whose cells are the pointwise mutual information (PMI) of
the respective word and context pairs, shifted by a global constant. We find
that another embedding method, NCE, is implicitly factorizing a similar matrix,
where each cell is the (shifted) log conditional probability of a word given its
context. We show that using a sparse Shifted Positive PMI word-context matrix to
represent words improves results on two word similarity tasks and one of two
analogy tasks. When dense low-dimensional vectors are preferred, exact
factorization with SVD can achieve solutions that are at least as good as SGNS's
solutions for word similarity tasks. On analogy questions SGNS remains superior
to SVD. We conjecture that this stems from the weighted nature of SGNS's
factorization.




NAME CHANGE POLICY

×

Requests for name changes in the electronic proceedings will be accepted with no
questions asked. However name changes may cause bibliographic tracking issues.
Authors are asked to consider this carefully and discuss it with their
co-authors prior to requesting a name change in the electronic proceedings.

Use the "Report an Issue" link to request a name change.


Report an Issue    |    Name Change Policy
Do not remove: This comment is monitored to verify that the site is working
properly