luigi.readthedocs.io Open in urlscan Pro
2606:4700::6811:2152  Public Scan

Submitted URL: http://luigi.readthedocs.io/
Effective URL: https://luigi.readthedocs.io/en/stable/
Submission: On May 30 via api from GB — Scanned from GB

Form analysis 2 forms found in the DOM

GET search.html

<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
  <input type="text" name="q" placeholder="Search docs">
  <input type="hidden" name="check_keywords" value="yes">
  <input type="hidden" name="area" value="default">
</form>

GET //readthedocs.org/projects/luigi/search/

<form id="flyout-search-form" class="wy-form" target="_blank" action="//readthedocs.org/projects/luigi/search/" method="get">
  <input type="text" name="q" aria-label="Search docs" placeholder="Search docs">
</form>

Text Content

Luigi
stable

 * Example – Top Artists
 * Building workflows
 * Tasks
 * Parameters
 * Running Luigi
 * Using the Central Scheduler
 * Execution Model
 * Luigi Patterns
 * Configuration
 * Configure logging
 * Design and limitations

 * luigi package
 * luigi.contrib package
 * luigi.tools package
 * luigi.local_target module

LaunchDarkly Galaxy Live on Tour 2022 - Connecting the brightest stars in
feature management.
Ad by EthicalAds   ·   Monetize your site
Luigi
 * Docs »
 * Getting Started
 * Edit on GitHub

--------------------------------------------------------------------------------

Luigi is a Python (2.7, 3.6, 3.7 tested) package that helps you build complex
pipelines of batch jobs. It handles dependency resolution, workflow management,
visualization, handling failures, command line integration, and much more.


GETTING STARTED¶

Run pip install luigi to install the latest stable version from PyPI.
Documentation for the latest release is hosted on readthedocs.

Run pip install luigi[toml] to install Luigi with TOML-based configs support.

For the bleeding edge code, pip install
git+https://github.com/spotify/luigi.git. Bleeding edge documentation is also
available.


BACKGROUND¶

The purpose of Luigi is to address all the plumbing typically associated with
long-running batch processes. You want to chain many tasks, automate them, and
failures will happen. These tasks can be anything, but are typically long
running things like Hadoop jobs, dumping data to/from databases, running machine
learning algorithms, or anything else.

There are other software packages that focus on lower level aspects of data
processing, like Hive, Pig, or Cascading. Luigi is not a framework to replace
these. Instead it helps you stitch many tasks together, where each task can be a
Hive query, a Hadoop job in Java, a Spark job in Scala or Python, a Python
snippet, dumping a table from a database, or anything else. It’s easy to build
up long-running pipelines that comprise thousands of tasks and take days or
weeks to complete. Luigi takes care of a lot of the workflow management so that
you can focus on the tasks themselves and their dependencies.

You can build pretty much any task you want, but Luigi also comes with a toolbox
of several common task templates that you use. It includes support for running
Python mapreduce jobs in Hadoop, as well as Hive, and Pig, jobs. It also comes
with file system abstractions for HDFS, and local files that ensures all file
system operations are atomic. This is important because it means your data
pipeline will not crash in a state containing partial data.


VISUALISER PAGE¶

The Luigi server comes with a web interface too, so you can search and filter
among all your tasks.




DEPENDENCY GRAPH EXAMPLE¶

Just to give you an idea of what Luigi does, this is a screen shot from
something we are running in production. Using Luigi’s visualiser, we get a nice
visual overview of the dependency graph of the workflow. Each node represents a
task which has to be run. Green tasks are already completed whereas yellow tasks
are yet to be run. Most of these tasks are Hadoop jobs, but there are also some
things that run locally and build up data files.




PHILOSOPHY¶

Conceptually, Luigi is similar to GNU Make where you have certain tasks and
these tasks in turn may have dependencies on other tasks. There are also some
similarities to Oozie and Azkaban. One major difference is that Luigi is not
just built specifically for Hadoop, and it’s easy to extend it with other kinds
of tasks.

Everything in Luigi is in Python. Instead of XML configuration or similar
external data files, the dependency graph is specified within Python. This makes
it easy to build up complex dependency graphs of tasks, where the dependencies
can involve date algebra or recursive references to other versions of the same
task. However, the workflow can trigger things not in Python, such as running
Pig scripts or scp’ing files.


WHO USES LUIGI?¶

We use Luigi internally at Spotify to run thousands of tasks every day,
organized in complex dependency graphs. Most of these tasks are Hadoop jobs.
Luigi provides an infrastructure that powers all kinds of stuff including
recommendations, toplists, A/B test analysis, external reports, internal
dashboards, etc.

Since Luigi is open source and without any registration walls, the exact number
of Luigi users is unknown. But based on the number of unique contributors, we
expect hundreds of enterprises to use it. Some users have written blog posts or
held presentations about Luigi:

 * Spotify (presentation, 2014)
 * Foursquare (presentation, 2013)
 * Mortar Data (Datadog) (documentation / tutorial)
 * Stripe (presentation, 2014)
 * Asana (blog, 2014)
 * Buffer (blog, 2014)
 * SeatGeek (blog, 2015)
 * Treasure Data (blog, 2015)
 * Growth Intelligence (presentation, 2015)
 * AdRoll (blog, 2015)
 * 17zuoye (presentation, 2015)
 * Custobar (presentation, 2016)
 * Blendle (presentation)
 * TrustYou (presentation, 2015)
 * Groupon / OrderUp (alternative implementation)
 * Red Hat - Marketing Operations (blog, 2017)
 * GetNinjas (blog, 2017)
 * voyages-sncf.com (presentation, 2017)
 * Open Targets (blog, 2017)
 * Leipzig University Library (presentation, 2016) / (project)
 * Synetiq (presentation, 2017)
 * Glossier (blog, 2018)
 * Data Revenue (blog, 2018)
 * Uppsala University (tutorial) / (presentation, 2015) / (slides, 2015) /
   (poster, 2015) / (paper, 2016) / (project)
 * GIPHY (blog, 2019)
 * xtream (blog, 2019)
 * CIAN (presentation, 2019)

Some more companies are using Luigi but haven’t had a chance yet to write about
it:

 * Schibsted
 * enbrite.ly
 * Dow Jones / The Wall Street Journal
 * Hotels.com
 * Newsela
 * Squarespace
 * OAO
 * Grovo
 * Weebly
 * Deloitte
 * Stacktome
 * LINX+Neemu+Chaordic
 * Foxberry
 * Okko
 * ISVWorld
 * Big Data
 * Movio
 * Bonnier News
 * Starsky Robotics
 * BaseTIS
 * Hopper
 * VOYAGE GROUP/Zucks
 * Textpert
 * Whizar
 * xtream
 * Skyscanner
 * Jodel
 * Mekar
 * M3

We’re more than happy to have your company added here. Just send a PR on GitHub.


EXTERNAL LINKS¶

 * Mailing List for discussions and asking questions. (Google Groups)
 * Releases (PyPI)
 * Source code (GitHub)
 * Hubot Integration plugin for Slack, Hipchat, etc (GitHub)


AUTHORS¶

Luigi was built at Spotify, mainly by Erik Bernhardsson and Elias Freider. Many
other people have contributed since open sourcing in late 2012. Arash Rouhani is
currently the chief maintainer of Luigi.


TABLE OF CONTENTS¶

 * Example – Top Artists
   * Step 1 - Aggregate Artist Streams
   * Running this Locally
   * Step 1b - Aggregate artists with Spark
   * Step 2 – Find the Top Artists
   * Step 3 - Insert into Postgres
   * Using the Central Planner
 * Building workflows
   * Target
   * Task
   * Parameter
   * Dependencies
 * Tasks
   * Task.requires
   * Requiring another Task
   * Task.output
   * Task.run
   * Task.input
   * Dynamic dependencies
   * Task status tracking
   * Events and callbacks
   * But I just want to run a Hadoop job?
   * Task priority
   * Namespaces, families and ids
   * Instance caching
 * Parameters
   * Instance caching
   * Insignificant parameters
   * Parameter visibility
   * Parameter types
   * Setting parameter value for other classes
   * Parameter resolution order
 * Running Luigi
   * Running from the Command Line
   * Running from Python code
   * Response of luigi.build()/luigi.run()
   * Luigi on Windows
 * Using the Central Scheduler
   * The luigid server
   * Enabling Task History
 * Execution Model
   * Workers and task execution
   * Scheduler
   * Triggering tasks
 * Luigi Patterns
   * Code Reuse
   * Triggering Many Tasks
   * Triggering recurring tasks
   * Efficiently triggering recurring tasks
   * Backfilling tasks
   * Propagating parameters with Range
   * Batching multiple parameter values into a single run
   * Tasks that regularly overwrite the same data source
   * Avoiding concurrent writes to a single file
   * Decreasing resources of running tasks
   * Monitoring task pipelines
   * Atomic Writes Problem
   * Sending messages to tasks
 * Configuration
   * Parameters from config Ingestion
   * Configurable options
   * [core]
   * [cors]
   * [worker]
   * [elasticsearch]
   * [email]
   * [batch_notifier]
   * [hadoop]
   * [hdfs]
   * [hive]
   * [kubernetes]
   * [mysql]
   * [postgres]
   * [redshift]
   * [resources]
   * [retcode]
   * [scalding]
   * [scheduler]
   * [sendgrid]
   * [smtp]
   * [spark]
   * [task_history]
   * [execution_summary]
   * [webhdfs]
   * [datadog]
   * Per Task Retry-Policy
   * Retry-Policy Fields
 * Configure logging
   * Config options:
   * Config section
   * Luigid CLI options:
   * Worker CLI options:
   * Configuration options resolution order:
 * Design and limitations


API REFERENCE¶

luigi Package containing core luigi functionality. luigi.contrib Package
containing optional and-on functionality. luigi.tools Sort of a standard library
for doing stuff with Tasks at a somewhat abstract level. luigi.local_target
LocalTarget provides a concrete implementation of a Target class that uses files
on the local file system


INDICES AND TABLES¶

 * Index
 * Module Index
 * Search Page

Next

--------------------------------------------------------------------------------

© Copyright 2011-2020, The Luigi Authors Revision 54a34736.

Built with Sphinx using a theme provided by Read the Docs.
Read the Docs v: stable
Versions latest stable Downloads PDF HTML Epub On Read the Docs Project Home
Builds Downloads On GitHub View Search


--------------------------------------------------------------------------------

Hosted by Read the Docs · Privacy Policy