www.eckerson.com Open in urlscan Pro
104.21.58.60  Public Scan

Form analysis
3 forms found in the DOM

POST /members

<form data-type="html" data-action="ajax:success->menu#onPostSuccess ajax:error->menu#onPostError" class="contact_form" action="/members" accept-charset="UTF-8" data-remote="true" method="post"><input type="hidden" name="authenticity_token"
    value="S0ayNdbqKcTK47TRPzqki+GthXdhbkJ1igHqF9OmtCAkeJrJ4loVOHAVUn4QouwCweHmJFGbQ6vlb84lO+bR8g=="><input class="form-control email" required="required" placeholder="Email" hide_label="true" type="email" name="member[email]" id="member_email">
  <input value="Newsletter Signup" type="hidden" name="member[last_updated_source_page]" id="member_last_updated_source_page">
  <input type="submit" name="commit" value="Sign Up" class="submit" data-disable-with="Sign Up">
  <div class="message error-message hidden" data-menu-target="alertBoxErrorMessage">
    <div class="alert alert-danger"> We are sorry but something went wrong. We have been notified about this error. Please retry or contact us at info@eckerson.com </div>
  </div>
  <div class="message success-message hidden" data-menu-target="alertBoxSuccessMessage">
    <div class="alert alert-success"> Thank you for joining our mailing list. You will now be informed about new Eckerson Group activities and content. </div>
  </div>
</form>

POST /members

<form data-type="html" data-action="ajax:success->alerts-aside#onPostSuccess ajax:error->alerts-aside#onPostError" class="newsletter-subscribe form-inline" action="/members" accept-charset="UTF-8" data-remote="true" method="post"><input type="hidden"
    name="authenticity_token" value="Q4TFGaXefFn/Omko47O6rBGPt8pVEVk0+QuE7rPw0rEsuu3lkW5ApUXMj4fMK/IlMcPUmWXkWOqWZaDcW7C3Yw==">
  <div class="mb-4" data-alerts-aside-target="alertBoxInputs">
    <div class="form-group">
      <input class="form-control email" required="required" placeholder="Email" hide_label="true" data-action="input->alerts-aside#validateEmail" data-alerts-aside-target="emailInput" type="email" name="member[email]" id="member_email">
      <input value="Content Page - Why Are Enterprise Analytics and AI So Painful? The Case for Data Pipeline Observability" type="hidden" name="member[last_updated_source_page]" id="member_last_updated_source_page">
    </div>
    <input type="submit" name="commit" value="Sign Up" class="submit" data-alerts-aside-target="submitBtn" disabled="disabled" data-disable-with="Sign Up">
  </div>
  <div class="message error-message hidden" data-alerts-aside-target="alertBoxErrorMessage">
    <div class="alert alert-danger"> We are sorry but something went wrong. We have been notified about this error. Please retry or contact us at info@eckerson.com </div>
  </div>
  <div class="message success-message hidden" data-alerts-aside-target="alertBoxSuccessMessage">
    <div class="alert alert-success"> Thank you for joining our mailing list. You will now be informed about new Eckerson Group activities and content. </div>
  </div>
</form>

POST /members

<form data-type="html" data-action="ajax:success->member-form#onPostSuccess ajax:error->member-form#onPostError" action="/members" accept-charset="UTF-8" data-remote="true" method="post"><input type="hidden" name="authenticity_token"
    value="lYUn75GZ1vHbdq0kS2FOKA4cLU/isEcsKu2VVzeLecj6uw8TpSnqDWGAS4tk+QahLlBOHNJFRvJFg7Fl38scGg==">
  <div class="modal-header">
    <h3 class="modal-title"> Contact Eckerson Group </h3>
  </div>
  <div class="modal-body">
    <div data-member-form-target="formBody" id="formBody">
      <div class="row">
        <div class="col-md-12">
          <div class="subtitle_1715276699">
            <style media="screen">
              .subtitle_1715276699 {
                position: absolute !important;
                top: -9999px;
                left: -9999px;
              }
            </style><label for="member_subtitle">If you are a human, ignore this field</label><input type="text" name="member[subtitle]" id="member_subtitle" autocomplete="off" tabindex="-1" class="form-control"><input type="hidden" name="spinner"
              value="a8464d35600567a2a6b60185f15ff03e">
          </div>
        </div>
      </div>
      <div class="row">
        <div class="col-md-6">
          <input class="form-control" placeholder="First name" required="required" data-member-form-target="firstNameInput" data-action="debounced:input->member-form#validateInput" type="text" name="member[first_name]" id="member_first_name">
          <i class="fa fa-check input-success hidden"></i>
        </div>
        <div class="col-md-6">
          <input class="form-control" placeholder="Last name" required="required" data-member-form-target="lastNameInput" data-action="debounced:input->member-form#validateInput" type="text" name="member[last_name]" id="member_last_name">
          <i class="fa fa-check input-success hidden"></i>
        </div>
      </div>
      <div class="row">
        <div class="col-md-6">
          <input class="form-control" placeholder="Company" required="required" data-member-form-target="companyInput" data-action="debounced:input->member-form#validateInput" type="text" name="member[company]" id="member_company">
          <i class="fa fa-check input-success hidden"></i>
        </div>
        <div class="col-md-6">
          <select class="reg-form-select subscribe-form-select form-control" data-member-form-target="titleInput" data-action="debounced:input->member-form#validateInput" name="member[title]" id="member_title">
            <option value="">Which best describes your role?</option>
            <option value="Executive: CDO, CAO, CIO or equivalent">Executive: CDO, CAO, CIO or equivalent</option>
            <option value="VP/Director: BI, Analytics, AI/ML, Data Management or equivalent">VP/Director: BI, Analytics, AI/ML, Data Management or equivalent</option>
            <option value="Manager: BI, Analytics, AI/ML, Data Management or equivalent">Manager: BI, Analytics, AI/ML, Data Management or equivalent</option>
            <option value="Architect: Data, Analytics, AI/ML, or equivalent">Architect: Data, Analytics, AI/ML, or equivalent</option>
            <option value="Engineer: Data, Analytics, AI/ML, or equivalent">Engineer: Data, Analytics, AI/ML, or equivalent</option>
            <option value="Analyst: Data, Analytics, AI/ML, or equivalent">Analyst: Data, Analytics, AI/ML, or equivalent</option>
            <option value="Consultant">Consultant</option>
            <option value="Vendor">Vendor</option>
            <option value="Academic">Academic</option>
            <option value="Other">Other</option>
          </select>
          <i class="fa fa-check input-success hidden mr-3"></i>
        </div>
      </div>
      <br>
      <div class="row">
        <div class="col-md-12">
          <input class="form-control" placeholder="Email address *" required="required" data-member-form-target="emailInput" data-action="debounced:input->member-form#validateInput" type="email" name="member[email]" id="member_email">
          <i class="fa fa-check input-success hidden" data-member-form-target="emailValidIcon"></i>
          <i class="fa fa-times input-fail hidden" data-member-form-target="emailInvalidIcon"></i>
          <div class="loader-container">
            <div class="loader hidden" data-member-form-target="emailValidationLoader"></div>
          </div>
        </div>
      </div>
      <br>
      <div class="row">
        <div class="col-md-12">
          <select class="reg-form-select subscribe-form-select form-control" data-member-form-target="titleInput" data-action="debounced:input->member-form#validateInput" name="member[referral_source]" id="member_referral_source">
            <option value="">How did you hear about us?</option>
            <option value="I've known Eckerson Group for a long time">I've known Eckerson Group for a long time</option>
            <option value="Search engine">Search engine</option>
            <option value="LinkedIn">LinkedIn</option>
            <option value="Colleague referral">Colleague referral</option>
            <option value="A webinar">A webinar</option>
            <option value="A physical event">A physical event</option>
            <option value="Other">Other</option>
          </select>
        </div>
      </div>
      <br>
      <div class="row">
        <div class="col-md-12">
          <textarea class="form-control" placeholder="Question or Request" required="required" rows="5" data-action="debounced:input->member-form#validateInput input->member-form#validateInput" minlength="10" name="member[message]"
            id="member_message"></textarea>
          <i class="fa fa-check input-success hidden"></i>
        </div>
      </div>
      <input value="Content Page - Why Are Enterprise Analytics and AI So Painful? The Case for Data Pipeline Observability" type="hidden" name="member[last_updated_source_page]" id="member_last_updated_source_page">
      <div class="row">
        <div class="form-group">
          <div class="col-sm-10">
            <div class="checkbox subscribe">
              <input name="member[subscribed]" type="hidden" value="0"><input type="checkbox" value="1" checked="checked" name="member[subscribed]" id="member_subscribed"> Eckerson Group may send me insights about emerging trends, techniques, and
              technologies in data and analytics. <br>
              <a class="privacy-policy" target="_blank" href="/privacy-policy">Privacy policy</a>
            </div>
          </div>
        </div>
      </div>
    </div>
    <!-- Show error message if Neverbounce finds out an invalid email -->
    <div class="message error-message hidden" data-member-form-target="emailInvalidErrorMessage">
      <div class="alert alert-danger"> The email you entered is not a valid email. Please enter a valid email to continue. </div>
    </div>
    <div class="error-message hidden" data-member-form-target="errorMessage">
      <div class="alert alert-danger"> We are sorry but something went wrong. We have been notified about this error. Please retry or contact us at info@eckerson.com </div>
    </div>
    <div class="error-message hidden" data-member-form-target="recaptchaErrorMessage">
      <div class="alert alert-danger"> Please, make sure to check recaptcha before submitting the form </div>
    </div>
    <div class="hidden" data-member-form-target="successMessage">
      <div class="alert alert-success"> Thank you for contacting Eckerson Group. A representative will be in contact with you soon regarding your question or request. </div>
    </div>
  </div>
  <div class="ml-4 mb-3" data-member-form-target="formRecaptcha">
    <script src="https://www.recaptcha.net/recaptcha/api.js" async="" defer=""></script>
    <div data-sitekey="6Ld4ql4gAAAAAGaJn_zwz3rcY7QkTbp2tR_l2WQ_" class="g-recaptcha ">
      <div style="width: 304px; height: 78px;">
        <div><iframe title="reCAPTCHA" width="304" height="78" role="presentation" name="a-giwnnkv97lap" frameborder="0" scrolling="no"
            sandbox="allow-forms allow-popups allow-same-origin allow-scripts allow-top-navigation allow-modals allow-popups-to-escape-sandbox allow-storage-access-by-user-activation"
            src="https://www.recaptcha.net/recaptcha/api2/anchor?ar=1&amp;k=6Ld4ql4gAAAAAGaJn_zwz3rcY7QkTbp2tR_l2WQ_&amp;co=aHR0cHM6Ly93d3cuZWNrZXJzb24uY29tOjQ0Mw..&amp;hl=en&amp;v=V6_85qpc2Xf2sbe3xTnRte7m&amp;size=normal&amp;cb=h66zpta3b4g4"></iframe>
        </div><textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response"
          style="width: 250px; height: 40px; border: 1px solid rgb(193, 193, 193); margin: 10px 25px; padding: 0px; resize: none; display: none;"></textarea>
      </div>
    </div>
    <noscript>
      <div>
        <div style="width: 302px; height: 422px; position: relative;">
          <div style="width: 302px; height: 422px; position: absolute;">
            <iframe src="https://www.recaptcha.net/recaptcha/api/fallback?k=6Ld4ql4gAAAAAGaJn_zwz3rcY7QkTbp2tR_l2WQ_" name="ReCAPTCHA" style="width: 302px; height: 422px; border-style: none; border: 0; overflow: hidden;">
            </iframe>
          </div>
        </div>
        <div style="width: 300px; height: 60px; border-style: none;
                bottom: 12px; left: 25px; margin: 0px; padding: 0px; right: 25px;
                background: #f9f9f9; border: 1px solid #c1c1c1; border-radius: 3px;">
          <textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response" style="width: 250px; height: 40px; border: 1px solid #c1c1c1;
                  margin: 10px 25px; padding: 0px; resize: none;">                </textarea>
        </div>
      </div>
    </noscript>
  </div>
  <div class="modal-footer">
    <div data-member-form-target="formFooter" id="formFooter">
      <input type="submit" name="commit" value="Contact" class="button bordered-bot green" data-member-form-target="submitBtn" disabled="disabled" data-action="click->member-form#submit" data-disable-with="Contact">
      <span>
        <a class="red bordered-bot button" data-action="click->member-form#closeForm" data-dismiss="modal" data-member-form-target="cancelBtn">Cancel</a>
        <a class="gray button hidden" data-action="click->member-form#closeForm" data-dismiss="modal" data-member-form-target="closeBtn">Close</a>
      </span>
    </div>
  </div>
</form>

Text Content

GET MORE VALUE FROM YOUR DATA

Free Consultation Mailing List

I'd like to receive Eckerson Group insights on the latest trends, technologies,
and techniques in data and analytics.

Close
We are sorry but something went wrong. We have been notified about this error.
Please retry or contact us at info@eckerson.com
Thank you for joining our mailing list. You will now be informed about new
Eckerson Group activities and content.

NO RESULTS FOUND

View All Results
 * Home
 * Research
 * Consulting
 * Events
 * Topics
   A Guide to Data Products Self-Service Analytics Data Governance Modern Data
   Architecture DataOps
 * Blogs
   Wayne Eckerson Dave Wells Kevin Petrie Jay Piscioneri
 * Glossary
 * About

NOV 13, 2020 / BY KEVIN PETRIE WAYNE ECKERSON/ IN DECODING DATA SOFTWARE


WHY ARE ENTERPRISE ANALYTICS AND AI SO PAINFUL? THE CASE FOR DATA PIPELINE
OBSERVABILITY


 * 
   
 * 
 * 

Subscribe to the Decoding Data Software blog

Explosive data growth makes data pipelines complex on every dimension. This
complexity makes it impossible for data teams to monitor and control thousands
of data pipeline components and services.

Driven by business demand, a growing population of enterprise data consumers
seeks to use new data from new sources to address new use cases. This prompts
data teams to adopt new tools, run workloads on new platforms, and migrate to
hybrid and multi-cloud infrastructures. As inter-dependent technologies
accumulate, and data volumes rise, enterprises struggle to operate their data
pipelines successfully with their analytics & AI projects. To regain control,
data teams need data pipeline observability: the ability to monitor,
automatically detect, predict, and resolve issues, from source to consumption,
across the enterprise.

This blog, the first in a series, examines the evolution of enterprise data
environments, rising pain of complexity, and resulting requirements for data
pipeline observability. The next blog will define this new paradigm and its
relationship to established disciplines such as DataOps, APM and ITOps. The
final blog will explore best practices for data pipeline observability, based on
enterprise successes and lessons learned.


ARCHITECTURAL EVOLUTION

To understand the problem, let’s review how architectures have changed.

Until recently, enterprise data pipelines served a staid, predictable,
on-premises world. A handful of ETL and change data capture (CDC) tools ingested
structured data from databases and applications, then transformed and stored
that data in monolithic data warehouses. Traditional business intelligence
software created dashboards and reports based on batch analytics workloads in
the data warehouse.

But architectures evolved. To manage increasing data and processing needs,
enterprises had to rapidly adopt new technologies. Architects and data engineers
now use ELT tools, CDC, APIs and event streaming systems such as Apache Kafka.
These tools ingest structured, semi-structured and unstructured data from
sources that include social media, IT logs and Internet of Things (IoT) sensors.
That data is transformed and stored in data warehouses, data lakes, NoSQL, and
stream data stores. Later, it is delivered for consumption in dashboards, BI
tools, AI and advanced analytics.

A final layer of complexity: More and more of these pipelines rely on elastic
cloud object stores and cloud compute nodes. This leads to hybrid and
multi-cloud environments that still must integrate with legacy on-premises
systems. All told, data pipelines become fragile webs of many interconnected
elements.




SYMPTOMS OF OVERLOAD

This complexity and rising tide of data can overwhelm enterprise teams that
manage the infrastructure, data pipeline and consumption layers.

Infrastructure layer. Platform engineers and site reliability engineers (SREs)
struggle to support data pipelines at scale with distributed compute and storage
resources. They use open source or commercial tools to monitor resource
availability, utilization and performance in isolation. But they often cannot
correlate those metrics across heterogeneous environments or gauge their impact
on data pipeline flows. This lack of visibility leads to issues, outages and
broken SLAs.

Data pipeline layer. Architects and data engineers struggle to diagnose and
remediate bottlenecks. They monitor data processing flows and performance with
Apache Spark, Apache Kafka and various commercial tools. But once again, those
isolated views cannot explain how issues relate across heterogeneous components.
They do not see the role of underlying resources – compute, storage, etc. – or
the impact of pipeline latency and throughput on actual analytics consumption.
Consequently, data timeliness and quality suffer.

Consumption layer. BI analysts, data scientists and business managers struggle
to make decisions with less timely and reliable analytics output. They escalate
issues to the VP of Analytics, Chief Data Officer or business executives – who
turn to infrastructure and data teams that cannot provide conclusive answers or
a path to improvement. Application performance management (APM) tools provide
insufficient visibility into the root cause of issues.

These enterprise teams lack the time and skills to stitch together multiple
tools or develop custom full-stack views themselves. Teams communicate with each
other but lack a common language and platform to collaborate.

So far enterprises responded to these problems with a mix of incomplete
responses. They rented elastic cloud compute resources to ease bottlenecks. They
patched together monitoring views by customizing multiple tools. And they
applied fast new engines like Apache Arrow in-memory, columnar processing to key
parts of their pipelines. This is like putting a band-aid on a tumor. The
problem may be hidden temporarily, but it’s only going to get worse. Ultimately,
it can only be addressed at the root cause.


WHAT’S NEXT?

It is time to take a comprehensive and holistic look at the issue.

Enterprises need data pipeline observability to achieve full-stack monitoring
and control of all the elements that drive AI and analytics data workloads. They
need to share common and intuitive views of data pipelines, and collaborate to
anticipate, prevent and resolve issues. They need to observe data pipelines
across the infrastructure, data and consumption layers, and across heterogeneous
components.

This data pipeline observability can help platform engineers and site
reliability engineers monitor and ensure infrastructure reliability, efficiency
and capacity. It can help architects and data engineers improve data access,
quality and lineage. It can help data consumers understand why issues arise –
and bolster their confidence that such issues can be resolved.



That is the intended value of data pipeline observability. In the next blog,
we’ll unpack what data pipeline observability means in practice, assess its
feasibility, and compare it with current solutions for DataOps, APM and ITOps.

To learn more in the meantime, you can register for Acceldata’s webinar, “The
Role of Observability for Analytics & AI,” on Tuesday, November 17, 2020, at 10
am PT / 1 pm ET.

YOU MIGHT ALSO LIKE

 * DataOps for Generative AI Data Pipelines, Part II: Must-Have Characteristics
 * DataOps for Generative AI Data Pipelines, Part I: What and Why
 * The Data Leader’s Guide to Generative AI, Part I: Models, Applications, and
   Pipelines
 * Analyst Series: Should AI Bots Build Your Data Pipelines?
 * The New Data Pipeline for Generative AI: Where and How It Works

Previous post by expert Next post by expert

KEVIN PETRIE



Kevin is the VP of Research at BARC US, where he writes and speaks about the
intersection of AI, analytics, and data management. For nearly three decades
Kevin has deciphered...

More About Kevin Petrie



CURATED RESEARCH ON DATA ANALYTICS TOPICS

 * A Guide to Data Products
 * Self-Service Analytics
 * Data Governance
 * Modern Data Architecture
 * DataOps

ECKERSON GROUP ALERTS

Join our mailing list to get tips and insights from our experts

We are sorry but something went wrong. We have been notified about this error.
Please retry or contact us at info@eckerson.com
Thank you for joining our mailing list. You will now be informed about new
Eckerson Group activities and content.

NEED A MODERN DATA STRATEGY?

Our veteran consultants can help

Contact Us Eckerson Consulting

YOU MIGHT ALSO LIKE

 * DataOps for Generative AI Data Pipelines, Part II: Must-Have Characteristics
 * DataOps for Generative AI Data Pipelines, Part I: What and Why
 * The Data Leader’s Guide to Generative AI, Part I: Models, Applications, and
   Pipelines
 * Analyst Series: Should AI Bots Build Your Data Pipelines?
 * The New Data Pipeline for Generative AI: Where and How It Works

NEED DEEPER EXPERT INSIGHT?

Meet Our Experts



CONTACT ECKERSON GROUP

If you are a human, ignore this field

Which best describes your role? Executive: CDO, CAO, CIO or equivalent
VP/Director: BI, Analytics, AI/ML, Data Management or equivalent Manager: BI,
Analytics, AI/ML, Data Management or equivalent Architect: Data, Analytics,
AI/ML, or equivalent Engineer: Data, Analytics, AI/ML, or equivalent Analyst:
Data, Analytics, AI/ML, or equivalent Consultant Vendor Academic Other



How did you hear about us? I've known Eckerson Group for a long time Search
engine LinkedIn Colleague referral A webinar A physical event Other


Eckerson Group may send me insights about emerging trends, techniques, and
technologies in data and analytics.
Privacy policy
The email you entered is not a valid email. Please enter a valid email to
continue.
We are sorry but something went wrong. We have been notified about this error.
Please retry or contact us at info@eckerson.com
Please, make sure to check recaptcha before submitting the form
Thank you for contacting Eckerson Group. A representative will be in contact
with you soon regarding your question or request.

Cancel Close

BOOKS BY OUR EXPERTS



Read the Secrets of Business Analytics Leaders by: Wayne Eckerson

Wayne Eckerson reveals the secrets of success of seven top business intelligence
and analytics leaders in this unconventional book that combines Wayne’s insights
with verbatim dialogue from the leaders.

Buy Now

Performance Dashboards: Measuring, Monitoring, and Managing Your Business by:
Wayne Eckerson

Hailed by many as the definitive guide to dashboards and scorecards, this book
is a must read.

Buy Now

Data Warehousing, Data Mining, and OLAP (Data Warehousing/Data Management) by:
Alex Berson, Stephen J. Smith

This reference provides strategic, theoretical and practical insight into three
information management technologies: data warehousing, online analytical
processing (OLAP), and data mining.

Buy Now

Building Data Mining Applications for CRM by: Alex Berson, Stephen J. Smith,
Berson, Kurt Thearling

This text provides comparison and contrast to different approaches and tools
available for contemporary data mining. It offers a step-by-step plan to help
readers develop a personalized approach.

Buy Now

Get in the Stream by: Dewayne Washington

This book is the authority on customer adoption written by a veteran in the
business who has implemented real world tactics.

Buy Now
Previous Next

MENU

CONSULTING

RESEARCH

ABOUT

CAREERS

CONTACT

CONTACT

Eckerson Group
617-653-5957
info@eckerson.com Privacy Policy

ABOUT

Eckerson Group helps organizations get more value from data and analytics
through thought leadership, full-service consulting, and educational workshops.
Our experts each have more than 25-years of experience in the field.



View All Results
 * 
 * Home
 * Research
 * Consulting
 * Blogs
   * 
   * Wayne Eckerson
   * Dave Wells
   * Kevin Petrie
   * Jay Piscioneri
   * Frameworks for Data Leaders
 * Webinars
 * About
 * Glossary