pypi.org Open in urlscan Pro
2a04:4e42:400::223  Public Scan

Submitted URL: https://pypi.python.org/pypi/Unidecode
Effective URL: https://pypi.org/project/Unidecode/
Submission: On July 21 via automatic, source links-suspicious — Scanned from DE

Form analysis 3 forms found in the DOM

/search/

<form class="search-form search-form--primary" action="/search/" role="search">
  <label for="search" class="sr-only">Search PyPI</label>
  <input id="search" class="search-form__search" type="text" name="q" placeholder="Search projects" value="" autocomplete="off" autocapitalize="off" spellcheck="false" data-controller="search-focus"
    data-action="keydown@window->search-focus#focusSearchField" data-search-focus-target="searchField">
  <button type="submit" class="search-form__button">
    <i class="fa fa-search" aria-hidden="true"></i>
    <span class="sr-only">Search</span>
  </button>
</form>

/search/

<form class="search-form search-form--fullwidth" action="/search/" role="search">
  <label for="mobile-search" class="sr-only">Search PyPI</label>
  <input id="mobile-search" class="search-form__search" type="text" name="q" placeholder="Search projects" value="" autocomplete="off" autocapitalize="off" spellcheck="false">
  <button type="submit" class="search-form__button">
    <i class="fa fa-search" aria-hidden="true"></i>
    <span class="sr-only">Search</span>
  </button>
</form>

/locale/

<form action="/locale/">
  <ul>
    <li>
      <button class="language-switcher__selected" name="locale_id" value="en" type="submit"> English </button>
    </li>
    <li>
      <button name="locale_id" value="es" type="submit"> español </button>
    </li>
    <li>
      <button name="locale_id" value="fr" type="submit"> français </button>
    </li>
    <li>
      <button name="locale_id" value="ja" type="submit"> 日本語 </button>
    </li>
    <li>
      <button name="locale_id" value="pt_BR" type="submit"> português (Brasil) </button>
    </li>
    <li>
      <button name="locale_id" value="uk" type="submit"> українська </button>
    </li>
    <li>
      <button name="locale_id" value="el" type="submit"> Ελληνικά </button>
    </li>
    <li>
      <button name="locale_id" value="de" type="submit"> Deutsch </button>
    </li>
    <li>
      <button name="locale_id" value="zh_Hans" type="submit"> 中文 (简体) </button>
    </li>
    <li>
      <button name="locale_id" value="zh_Hant" type="submit"> 中文 (繁體) </button>
    </li>
    <li>
      <button name="locale_id" value="ru" type="submit"> русский </button>
    </li>
    <li>
      <button name="locale_id" value="he" type="submit"> עברית </button>
    </li>
    <li>
      <button name="locale_id" value="eo" type="submit"> esperanto </button>
    </li>
  </ul>
</form>

Text Content

Skip to main content Switch to mobile version
Warning Some features may not work without JavaScript. Please try enabling it if
you encounter problems.



Search PyPI Search
 * Help
 * Sponsors
 * Log in
 * Register

Menu
 * Help
 * Sponsors
 * Log in
 * Register

Search PyPI Search


UNIDECODE 1.3.4

pip install Unidecode Copy PIP instructions

Latest version

Released: Mar 10, 2022

ASCII transliterations of Unicode text




NAVIGATION

 * Project description
 * Release history
 * Download files


STATISTICS

View statistics for this project via Libraries.io, or by using our public
dataset on Google BigQuery


META

License: GNU General Public License v2 or later (GPLv2+) (GPL)

Author: Tomaz Solc

Requires: Python >=3.5


MAINTAINERS

avian bbangert


CLASSIFIERS

 * License
   * OSI Approved :: GNU General Public License v2 or later (GPLv2+)
 * Programming Language
   * Python
   * Python :: 3
   * Python :: 3.5
   * Python :: 3.6
   * Python :: 3.7
   * Python :: 3.8
   * Python :: 3.9
   * Python :: Implementation :: CPython
   * Python :: Implementation :: PyPy
 * Topic
   * Text Processing
   * Text Processing :: Filters

IBM is a Contributing sponsor of the Python Software Foundation.
PSF Sponsor · Served ethically
 * Project description
 * Project details
 * Release history
 * Download files


PROJECT DESCRIPTION

It often happens that you have text data in Unicode, but you need to represent
it in ASCII. For example when integrating with legacy code that doesn’t support
Unicode, or for ease of entry of non-Roman names on a US keyboard, or when
constructing ASCII machine identifiers from human-readable Unicode strings that
should still be somewhat intelligible. A popular example of this is when making
an URL slug from an article title.

Unidecode is not a replacement for fully supporting Unicode for strings in your
program. There are a number of caveats that come with its use, especially when
its output is directly visible to users. Please read the rest of this README
before using Unidecode in your project.

In most of examples listed above you could represent Unicode characters as ???
or \\15BA\\15A0\\1610, to mention two extreme cases. But that’s nearly useless
to someone who actually wants to read what the text says.

What Unidecode provides is a middle road: the function unidecode() takes Unicode
data and tries to represent it in ASCII characters (i.e., the universally
displayable characters between 0x00 and 0x7F), where the compromises taken when
mapping between two character sets are chosen to be near what a human with a US
keyboard would choose.

The quality of resulting ASCII representation varies. For languages of western
origin it should be between perfect and good. On the other hand transliteration
(i.e., conveying, in Roman letters, the pronunciation expressed by the text in
some other writing system) of languages like Chinese, Japanese or Korean is a
very complex issue and this library does not even attempt to address it. It
draws the line at context-free character-by-character mapping. So a good rule of
thumb is that the further the script you are transliterating is from Latin
alphabet, the worse the transliteration will be.

Generally Unidecode produces better results than simply stripping accents from
characters (which can be done in Python with built-in functions). It is based on
hand-tuned character mappings that for example also contain ASCII approximations
for symbols and non-Latin alphabets.

Note that some people might find certain transliterations offending. Most common
examples include characters that are used in multiple languages. A user expects
a character to be transliterated in their language but Unidecode uses a
transliteration for a different language. It’s best to not use Unidecode for
strings that are directly visible to users of your application. See also the
Frequently Asked Questions section for more info on common problems.

This is a Python port of Text::Unidecode Perl module by Sean M. Burke
<sburke@cpan.org>.


MODULE CONTENT

This library contains a function that takes a string object, possibly containing
non-ASCII characters, and returns a string that can be safely encoded to ASCII:

>>> from unidecode import unidecode
>>> unidecode('kožušček')
'kozuscek'
>>> unidecode('30 \U0001d5c4\U0001d5c6/\U0001d5c1')
'30 km/h'
>>> unidecode('\u5317\u4EB0')
'Bei Jing '


You can also specify an errors argument to unidecode() that determines what
Unidecode does with characters that are not present in its transliteration
tables. The default is 'ignore' meaning that Unidecode will ignore those
characters (replace them with an empty string). 'strict' will raise a
UnidecodeError. The exception object will contain an index attribute that can be
used to find the offending character. 'replace' will replace them with '?' (or
another string, specified in the replace_str argument). 'preserve' will keep the
original, non-ASCII character in the string. Note that if 'preserve' is used the
string returned by unidecode() will not be ASCII-encodable!:

>>> unidecode('\ue000') # unidecode does not have replacements for Private Use Area characters
''
>>> unidecode('\ue000', errors='strict')
Traceback (most recent call last):
...
unidecode.UnidecodeError: no replacement found for character '\ue000' in position 0


A utility is also included that allows you to transliterate text from the
command line in several ways. Reading from standard input:

$ echo hello | unidecode
hello


from a command line argument:

$ unidecode -c hello
hello


or from a file:

$ unidecode hello.txt
hello


The default encoding used by the utility depends on your system locale. You can
specify another encoding with the -e argument. See unidecode --help for a full
list of available options.


REQUIREMENTS

Nothing except Python itself. Unidecode supports Python 3.5 or later.

You need a Python build with “wide” Unicode characters (also called “UCS-4
build”) in order for Unidecode to work correctly with characters outside of
Basic Multilingual Plane (BMP). Common characters outside BMP are bold, italic,
script, etc. variants of the Latin alphabet intended for mathematical notation.
Surrogate pair encoding of “narrow” builds is not supported in Unidecode.

If your Python build supports “wide” Unicode the following expression will
return True:

>>> import sys
>>> sys.maxunicode > 0xffff
True


See PEP 261 for details regarding support for “wide” Unicode characters in
Python.


INSTALLATION

To install the latest version of Unidecode from the Python package index, use
these commands:

$ pip install unidecode


To install Unidecode from the source distribution and run unit tests, use:

$ python setup.py install
$ python setup.py test



FREQUENTLY ASKED QUESTIONS

German umlauts are transliterated incorrectly Latin letters “a”, “o” and “u”
with diaeresis are transliterated by Unidecode as “a”, “o”, “u”, not according
to German rules “ae”, “oe”, “ue”. This is intentional and will not be changed.
Rationale is that these letters are used in languages other than German (for
example, Finnish and Turkish). German text transliterated without the extra “e”
is much more readable than other languages transliterated using German rules. A
workaround is to do your own replacements of these characters before passing the
string to unidecode(). Japanese Kanji is transliterated as Chinese Same as with
Latin letters with accents discussed in the answer above, the Unicode standard
encodes letters, not letters in a certain language or their meaning. With
Japanese and Chinese this is even more evident because the same letter can have
very different transliterations depending on the language it is used in. Since
Unidecode does not do language-specific transliteration (see next question), it
must decide on one. For certain characters that are used in both Japanese and
Chinese the decision was to use Chinese transliterations. If you intend to
transliterate Japanese, Chinese or Korean text please consider using other
libraries which do language-specific transliteration, such as Unihandecode.
Unidecode should support localization (e.g. a language or country parameter,
inspecting system locale, etc.) Language-specific transliteration is a
complicated problem and beyond the scope of this library. Changes related to
this will not be accepted. Please consider using other libraries which do
provide this capability, such as Unihandecode. Unidecode should automatically
detect the language of the text being transliterated Language detection is a
completely separate problem and beyond the scope of this library. Unidecode
should use a permissive license such as MIT or the BSD license. The maintainer
of Unidecode believes that providing access to source code on redistribution is
a fair and reasonable request when basing products on voluntary work of many
contributors. If the license is not suitable for you, please consider using
other libraries, such as text-unidecode. Unidecode produces completely wrong
results (e.g. “u” with diaeresis transliterating as “A 1/4 “) The strings you
are passing to Unidecode have been wrongly decoded somewhere in your program.
For example, you might be decoding utf-8 encoded strings as latin1. With a
misconfigured terminal, locale and/or a text editor this might not be
immediately apparent. Inspect your strings with repr() and consult the Unicode
HOWTO. Why does Unidecode not replace \u and \U backslash escapes in my strings?
Unidecode knows nothing about escape sequences. Interpreting these sequences and
replacing them with actual Unicode characters in string literals is the task of
the Python interpreter. If you are asking this question you are very likely
misunderstanding the purpose of this library. Consult the Unicode HOWTO and
possibly the unicode_escape encoding in the standard library. I’ve upgraded
Unidecode and now some URLs on my website return 404 Not Found. This is an issue
with the software that is running your website, not Unidecode. Occasionally, new
versions of Unidecode library are released which contain improvements to the
transliteration tables. This means that you cannot rely that unidecode() output
will not change across different versions of Unidecode library. If you use
unidecode() to generate URLs for your website, either generate the URL slug once
and store it in the database or lock your dependency of Unidecode to one
specific version.

Some of the issues in this section are discussed in more detail in this blog
post.


PERFORMANCE NOTES

By default, unidecode() optimizes for the use case where most of the strings
passed to it are already ASCII-only and no transliteration is necessary (this
default might change in future versions).

For performance critical applications, two additional functions are exposed:

unidecode_expect_ascii() is optimized for ASCII-only inputs (approximately 5
times faster than unidecode_expect_nonascii() on 10 character strings, more on
longer strings), but slightly slower for non-ASCII inputs.

unidecode_expect_nonascii() takes approximately the same amount of time on ASCII
and non-ASCII inputs, but is slightly faster for non-ASCII inputs than
unidecode_expect_ascii().

Apart from differences in run time, both functions produce identical results.
For most users of Unidecode, the difference in performance should be negligible.


SOURCE

You can get the latest development version of Unidecode with:

$ git clone https://www.tablix.org/~avian/git/unidecode.git


There is also an official mirror of this repository on GitHub at
https://github.com/avian2/unidecode


CONTACT

Please make sure to read the Frequently asked questions section above before
contacting the maintainer.

Bug reports, patches and suggestions for Unidecode can be sent to
tomaz.solc@tablix.org.

Alternatively, you can also open a ticket or pull request at
https://github.com/avian2/unidecode


COPYRIGHT

Original character transliteration tables:

Copyright 2001, Sean M. Burke <sburke@cpan.org>, all rights reserved.

Python code and later additions:

Copyright 2022, Tomaž Šolc <tomaz.solc@tablix.org>

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 2 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
Street, Fifth Floor, Boston, MA 02110-1301 USA. The programs and documentation
in this dist are distributed in the hope that they will be useful, but without
any warranty; without even the implied warranty of merchantability or fitness
for a particular purpose.


PROJECT DETAILS


STATISTICS

View statistics for this project via Libraries.io, or by using our public
dataset on Google BigQuery


META

License: GNU General Public License v2 or later (GPLv2+) (GPL)

Author: Tomaz Solc

Requires: Python >=3.5


MAINTAINERS

avian bbangert


CLASSIFIERS

 * License
   * OSI Approved :: GNU General Public License v2 or later (GPLv2+)
 * Programming Language
   * Python
   * Python :: 3
   * Python :: 3.5
   * Python :: 3.6
   * Python :: 3.7
   * Python :: 3.8
   * Python :: 3.9
   * Python :: Implementation :: CPython
   * Python :: Implementation :: PyPy
 * Topic
   * Text Processing
   * Text Processing :: Filters



RELEASE HISTORY RELEASE NOTIFICATIONS | RSS FEED

This version


1.3.4

Mar 10, 2022

1.3.3

Feb 19, 2022

1.3.2

Sep 16, 2021

1.3.1 yanked

Sep 9, 2021

1.3.0 yanked

Sep 6, 2021

1.2.0

Feb 5, 2021

1.1.2

Dec 20, 2020

1.1.1

Jun 21, 2019

1.1.0

Jun 14, 2019

1.0.23

Nov 19, 2018

1.0.22

Jan 5, 2018

0.04.21

Jun 28, 2017

0.04.20

Jan 9, 2017

0.04.19

Jan 21, 2016

0.04.18

Jun 13, 2015

0.04.17

Dec 18, 2014

0.04.16

May 11, 2014

0.04.14

Sep 20, 2013

0.04.13

May 30, 2013

0.04.12

Jan 28, 2013

0.04.11

Jan 17, 2013

0.04.10

Dec 1, 2012

0.04.9

Sep 23, 2011

0.04.8

Sep 22, 2011

0.04.7

Apr 4, 2011

0.04.6

Feb 16, 2011

0.04.1

Jun 16, 2009


DOWNLOAD FILES

Download the file for your platform. If you're not sure which to choose, learn
more about installing packages.


SOURCE DISTRIBUTION

Unidecode-1.3.4.tar.gz (192.3 kB view hashes)

Uploaded Mar 10, 2022 source


BUILT DISTRIBUTION

Unidecode-1.3.4-py3-none-any.whl (235.9 kB view hashes)

Uploaded Mar 10, 2022 py3

Close


HASHES FOR UNIDECODE-1.3.4.TAR.GZ

Hashes for Unidecode-1.3.4.tar.gz Algorithm Hash digest SHA256
8e4352fb93d5a735c788110d2e7ac8e8031eb06ccbfe8d324ab71735015f9342 Copy MD5
9cc5ae6c5d6ee1324091dfc54be727ac Copy BLAKE2-256
4116ee78864c2c2ba9bddba3978baa378f83270c8b7810eb7900f84e62a9ffac Copy

Close
Close


HASHES FOR UNIDECODE-1.3.4-PY3-NONE-ANY.WHL

Hashes for Unidecode-1.3.4-py3-none-any.whl Algorithm Hash digest SHA256
afa04efcdd818a93237574791be9b2817d7077c25a068b00f8cff7baa4e59257 Copy MD5
437c72205fb41ee2387bde63c06f8db8 Copy BLAKE2-256
f95b7603add7f192252916b85927263b598c74585f82389e6e42318a6278159b Copy

Close


HELP

 * Installing packages
 * Uploading packages
 * User guide
 * FAQs


ABOUT PYPI

 * PyPI on Twitter
 * Infrastructure dashboard
 * Package index name retention
 * Our sponsors


CONTRIBUTING TO PYPI

 * Bugs and feedback
 * Contribute on GitHub
 * Translate PyPI
 * Development credits


USING PYPI

 * Code of conduct
 * Report security issue
 * Privacy policy
 * Terms of use

--------------------------------------------------------------------------------

Status: All Systems Operational

Developed and maintained by the Python community, for the Python community.
Donate today!

© 2022 Python Software Foundation
Site map

Switch to desktop version
 * English
 * español
 * français
 * 日本語
 * português (Brasil)
 * українська
 * Ελληνικά
 * Deutsch
 * 中文 (简体)
 * 中文 (繁體)
 * русский
 * עברית
 * esperanto

Supported by

AWS Cloud computing Datadog Monitoring Facebook / Instagram PSF Sponsor Fastly
CDN Google Object Storage and Download Analytics Huawei PSF Sponsor Microsoft
PSF Sponsor NVIDIA PSF Sponsor Pingdom Monitoring Salesforce PSF Sponsor Sentry
Error logging StatusPage Status page