www.comet.com Open in urlscan Pro
50.16.93.68  Public Scan

Form analysis
1 forms found in the DOM

GET /site/

<form method="get" class="searchform" action="/site/">
  <label>
    <span class="screen-reader-text">Search</span>
    <input type="search" class="field" name="s" placeholder="Search" autocomplete="off">
  </label>
  <button type="submit" class="searchform-submit"><span class="ticon ticon-search" aria-hidden="true"></span><span class="screen-reader-text">Submit</span></button>
</form>

Text Content

skip to Main Content
Comet Launches Kangas, an Open Source Data Analysis, Exploration and Debugging
Tool for Machine Learning.
Learn More
 * Enterprise
 * Products
   * Experiment Management
   * Artifacts
   * Model Registry
   * Model Production Monitoring
 * Customers
 * Resources
 * Learn
   * Docs
   * Blog
   * Heartbeat
   * Deep Learning Weekly
 * Pricing
 * Company
   * About Us
   * News and Events
     * Events
     * Press Releases
   * Careers
   * Contact Us
   * Leadership
 * Login
 * Create a Free Account


Search Submit
Home › Customers › Uber


HOW SCIENTISTS AT UBER USE COMET TO MANAGE MACHINE LEARNING EXPERIMENTS

Scale is an interesting, often over-simplified challenge in machine learning.
Intuitively, most everyone understands that bigger models require large amounts
of resources (large datasets, computational firepower, etc.), but cost is just
one piece of ML’s scale problem. For example, the complexity of managing offline
ML experiments at scale is a particularly thorny problem.

In contrast with online A/B tests, offline ML experiments are attempts to
improve the accuracy or performance of an ML model by experimenting with changes
to the model offline.

At Uber, where ML is fundamental to most products, a mechanism to manage offline
experiments easily is needed to improve developer velocity.  To solve for this,
Uber AI was looking for a solution that will potentially complement and extend
its in-house experiment management and collaboration capabilities.

Video Player
/site/wp-content/uploads/2022/06/Uber-Recording-4-1080p-2X.mp4

00:00
00:00
00:24
Use Up/Down Arrow keys to increase or decrease volume.



WHAT IS HARD ABOUT MANAGING EXPERIMENTS AT UBER-SCALE?

Take a ML-powered feature of Uber, say estimating the ETA for every ride and
delivery. Imagine every market where Uber is active, and then every product
offering within each market. The sheer quantity and diversity of product and
market segments present logistical challenges right out of the gate. As Olcay
Cirit, research scientist at Uber AI, says, “We can’t simply look at a global
accuracy metric, rather we must use sliced analysis to gauge the improvements or
degradations in each market and product category.”

This results in a huge volume of experiments, conducted by different teams with
different processes, all of which must be compared in order to make optimal
improvements to the model. Performing deep analysis on model results where each
model might behave differently across segments is extremely difficult.

BUILDING EXPERIMENT MANAGEMENT INFRASTRUCTURE AT UBER

Comet supports most of the ML experiment tracking workload out-of-the-box. As
Cirit explains, “As my team piloted Comet, it organized my team’s deep learning
experiments by tracking all of our hyperparameters, metrics, and code changes…
Developers working on improvements to ML infrastructure can more easily gauge,
for example, whether improvements in training speed might adversely impact model
convergence.”

> “We found that Comet had these critical customizability features, including
> code panels, and were also quite willing to engage with us during our
> evaluation.”

For Uber, customizability and extensibility are critical concerns, as Uber AI
has many different teams working on state-of-the-art models in different
domains. As Cirit explains:

 * “We are working on integrating Comet into our in-house framework so that all
   users of the platform can benefit from experiment tracking… With Comet, it
   will be very easy for product team members to add new, project-specific
   metrics to track and visualize without having to make changes to the
   platform…”
 *  “Uber has its own internally developed Bayesian Optimization package that we
   use to tune hyperparameters. Comet lets us visualize all of the trials using
   panels and parallel coordinate charts, so that we can derive high-level
   insights about the hyperparameter search space in addition to getting a tuned
   set of hyperparameters.”

What you need, at that stage, are tools with which you can build ML
infrastructure suited to your specific needs.

UBER

Mobility-as-a-service provider

INDUSTRY

Technology – Transportation

TECHNOLOGIES

Custom-built MLOps platform


SCHEDULE A DEMO.

 To learn more about Comet Enterprise or to schedule
a demo, contact us. We will reply within 24 hours.

Create Free AccountContact Sales


PRODUCTS

 * Experiment Management
 * Artifacts
 * Model Registry
 * Model Production Monitoring


LEARN

 * Documentation
 * Resources
 * Comet Blog
 * Deep Learning Weekly
 * Heartbeat


COMPANY

 * About Us
 * News and Events
 * Careers
 * Contact Us


GET STARTED

 * Pricing
 * Create a Free Account
 * Contact Sales

Follow Us
LinkedInTwitterYoutubeFacebook
© 2023 Comet ML, Inc. - All Rights Reserved
 * Terms of Service
 * Privacy Policy
 * CCPA Privacy Notice
 * Cookie Settings

Back To Top

We use cookies to collect statistical usage information about our website and
its visitors and ensure we give you the best experience on our website. Please
refer to our Privacy Policy to learn more.OkPrivacy policy
×Close mobile menu
 * Enterprise
 * Products
   * Experiment Management
   * Artifacts
   * Model Registry
   * Model Production Monitoring
 * Customers
 * Resources
 * Learn
   * Docs
   * Blog
 * Pricing
 * Company
   * About Us
   * News and Events
     * Events
     * Press Releases
   * Careers
   * Leadership
   * Contact Us
 * Login
 * Create a Free Account

1