www.bigdatawire.com Open in urlscan Pro
172.67.182.132  Public Scan

Form analysis
2 forms found in the DOM

GET https://www.bigdatawire.com/

<form method="get" id="searchform" class="form-inline" action="https://www.bigdatawire.com/">
  <div><input type="text" value="Search this site" name="s" id="s" onfocus="if(this.value==this.defaultValue)this.value='';" onblur="if(this.value=='')this.value=this.defaultValue;"><input type="submit" id="searchsubmit" class="btn" value="Search">
  </div>
</form>

POST //translate.googleapis.com/translate_voting?client=te

<form id="goog-gt-votingForm" action="//translate.googleapis.com/translate_voting?client=te" method="post" target="votingFrame" class="VIpgJd-yAWNEb-hvhgNd-aXYTce"><input type="text" name="sl" id="goog-gt-votingInputSrcLang"><input type="text"
    name="tl" id="goog-gt-votingInputTrgLang"><input type="text" name="query" id="goog-gt-votingInputSrcText"><input type="text" name="gtrans" id="goog-gt-votingInputTrgText"><input type="text" name="vote" id="goog-gt-votingInputVote"></form>

Text Content

▼

Translation Disclaimer


 * About
 * Resources
 * Subscribe

Follow BigDATAwire:




Menu

 * Home
 * COVID-19
 * Features
   * Articles
   * This Just In
   * Big Data Executive Videos
   * People to Watch
     * 2024
     * 2023
     * 2022
     * 2021
     * 2020
     * 2019
     * 2018
     * 2017
   * Readers’ Choice Awards
     * 2024
     * 2023
     * 2022
     * 2021
     * 2020
     * 2019
     * 2018
     * 2017
     * 2016
   * Decade of Datanami
 * Sectors
   * Academia
   * Biosciences
   * Financial Services
   * Government
   * Healthcare
   * Manufacturing
   * Retail
   * Science
   * Other
 * Applications
   * Artificial Intelligence
   * Complex Event Processing
   * Data Mining
   * Enterprise Analytics
   * Predictive Analytics
   * Research Analytics
   * Visualization
 * Technologies
   * Cloud
   * Frameworks
   * Middleware
   * Network
   * Processors
   * Storage
   * Systems
 * Vendors
 * Job Bank
 * Events
   * Events
   * Advanced Scale Forum
   * HPC + AI Wall St
 * Advertise

October 8, 2024


STILL TOO MUCH DUCT TAPE IN DATA TRANSFORMATION, DBT LABS’ HANDY SAYS

Alex Woodie

Tristan Handy at the Coalesce conference in San Diego, October 23

While real progress has been made in streamlining some aspects of big data
analytics workflows, there is still too much duct tape keeping it all together,
according to Tristan Handy, the founder and CEO of dbt Labs, which today
unveiled a slew of enhancements to dbt Cloud at its annual user conference.

Dbt has emerged as one of the most popular tools for preparing data for
analytics. Instead of writing raw SQL code, data engineers write dbt’s syntax to
create models that define the data transformations that need to be performed,
while respecting dependencies up and down the stack. At runtime, a dbt user
calls one model or series of models to execute a transformation in a defined,
declarative manner. It’s DevOps discipline meets data engineering, or DataOps.

The DataOps approach of dbt has resonated with millions of workers who use dbt,
or analytyics engineers, as dbt Labs likes to call them. When data
transformations are coded in dbt, it brings other benefits, like fewer lines of
code, automated documentation, visual lineage, and pipeline break notifications.

However, even with these data benefits in hand, it doesn’t mean we have solved
all data problems, Handy says.

“The data industry has made real progress towards maturity over the past
decade,” Handy says in a press release. “But real problems persist. Siloed data.
Lack of trust. Too much ‘duct tape’ in our operational systems.”

Handy elaborated on his thoughts in a blog post last month.

dbt Labs envisions DataOps as the future of analytics engineering (Image source:
dbt Labs)

“We can observe from dbt product instrumentation data that a large majority of
companies that transition to the cloud adopt at least some elements of a mature
analytics workflow–particularly related to data transformations. But what about
the other layers of the analytics stack?” he wrote.

There are sticking points in those other layers, he says. For instance, Handy
asks whether notebooks and dashboards are well-tested and have provable SLAs.
“Do your ingestion pipelines have clear versioning? Do they have processes to
roll back schema changes? Do they support multiple environments?”

“Can data consumers request support and declare incidents directly from within
the analytical systems they interact with?” he asks. “Do you have on-call
rotations? Do you have a well-defined incident management process? The answer to
these questions, for almost every company out there, is ‘no,’” he writes.

While it’s unlikely that any one company or product could supply all those
capabilities, the folks at dbt Labs are making a go out of filling the gaps and
ripping off that duct tape. To that end, dbt Labs today announced a series of
enhancements in dbt Cloud, its enterprise offering for analytics professionals.
The company says these enhancements represent the “One dbt” vision of creating a
single dbt experience across multiple data personas and data platforms as part
of what it calls the analytics development lifecycle, or ADLC.

The company today unveiled several enhancements to dbt Cloud that it says will
help customers build better data pipelines. That includes dbt Copilot that will
automate repetitive manual work around things like creating tests, writing
documentation, and creating semantic models. Dbt Labs is also building a chatbot
that lets users ask questions of their data using natural language.

Dbt Labs is building on the data mesh that it launched at last year’s Coalesce,
which allowed cross-project dbt references, with a new cross-platform mesh. The
new offering uses Apache Iceberg to create portable data tables that can be read
across different platforms. Benefits include the ability to centrally define and
maintain data governance standards, to see end-to-end lineage across various
data platforms, and find, reference, and re-use existing data assets instead of
rebuilding, dbt Labs says.

Dbt Cloud customers are also getting a new low-code, drag-and-drop environment
for building and exploring dbt models. The company says this new environment
(which is currently in beta) will allow a new group of less-technical users to
develop analytics code themselves.

It will be easier to catch bugs in dbt code before they go into production using
the new Advanced CI (continuous integration) offering. Dbt Labs says Advanced CI
will make it easier for users to compare code changes as part of the CI process
and catch any unexpected behavior before the new code is merged into production.
“This improves code quality and helps organizations optimize compute spend by
only materializing correct models,” the company says.

Other improvements dbt Labs is making to dbt Cloud include:

 * Data health tiles that can be embedded into any downstream app to provide
   real-time info about their data, including freshness and quality, directly in
   tools where users work;
 * Auto-exposures with Tableau, a new feature that automatically incorporates
   Tableau dashboards into dbt lineage, boosting data freshness;
 * Semantic layer integration with Power BI;
 * New supported adapters, including Teradata (preview) and AWS Athena (GA).

Related Items:

AI Impacting Data Engineering Faster Than Expected, dbt Labs’ Handy Says

Tristan Handy’s Audacious Vision of the Future of Data Engineering

Semantic Layer Belongs in Middleware, and dbt Wants to Deliver It


Applications: Data Management
Technologies: Cloud, Middleware
Sectors: Retail
Vendors: dbt Labs
Tags: big data, data silos, data transformation, Tristan Handy

Comments are closed.


 * This Just In
 * Most Read
   

NOVEMBER 14, 2024

 * NHL Partners with VAST Data to Accelerate and Streamline Media Production
   Operations
 * Hakkōda Announces General Availability of New AI-Driven Services on AWS
 * Cloudera to Acquire Octopai’s Platform to Deliver Trusted Data Across Entire
   Hybrid Cloud Data Estate
 * AMD Enhances Data Center and Edge Performance with Versal Premium Series Gen
   2 SoCs
 * OSI and the Eclipse Foundation to Collaborate on Shaping Open Source AI
   Public Policy
 * ‘Playtime is Over’ for GenAI: NTT DATA Research Shows Organizations Shifting
   From Experiments to Investments
 * ThoughtSpot Launches Spotter, the Autonomous Agent for Analytics
 * Hammerspace Introduces Tier 0 Storage Solution to Maximize GPU Server
   Efficiency
 * Arcitecta and Wasabi Partner to Simplify Cloud Storage Access Across
   Workflows
 * New MLPerf Training v4.1 Benchmarks Highlight Industry’s Focus on New Systems
   and GenAI Applications

NOVEMBER 13, 2024

 * Connecty AI Raises $1.8M to Solve Enterprise Data’s Three-Dimensional Problem
 * Nutanix Extends AI Platform to Public Cloud
 * Snowflake Expands Capabilities for Enterprises to Deliver Trustworthy AI into
   Production
 * DataRobot Announces AI-Ready Data to Empower AI Teams
 * Hitachi Vantara Expands Hybrid Cloud Storage Platform with Object Storage,
   All-QLC Flash and Advancing Cloud Integration
 * CTERA Delivers Next-Generation Data Observability with CTERA Insight Service
 * Snowflake’s Unistore Unifies Transactional and Analytical Data with the
   General Availability of Hybrid Tables
 * DataRobot Announces Industry-First Generative AI Tooling to Secure AI
   Outcomes
 * SAS Acquires Hazy Synthetic Data Software to Boost Generative AI Portfolio
 * RelationalAI Knowledge Graph Coprocessor is Generally Available as a
   Snowflake Native App

 * More This Just In…






SPONSORED PARTNER CONTENT

 * DESIGNING A COPILOT FOR DATA TRANSFORMATION

 * GET YOUR DATA AI READY – CELEBRATE ONE YEAR OF DEEP DISH DATA VIRTUAL SERIES!

 * SUPERCHARGE YOUR DATA LAKE WITH SPARK 3.3


LEADING SOLUTION PROVIDERS




TABOR NETWORK















SPONSORED WHITEPAPERS

 * IDC SPOTLIGHT: BOOSTING AI IMPACT WITH DATA PRODUCTS

 * BUILDING A TRUSTED DATA FOUNDATION FOR AI/ML AND BUSINESS INTELLIGENCE (BI)

 * View the Whitepaper Library


SPONSORED MULTIMEDIA

THE POWER OF DATAOPS: BRING AUTOMATION TO LIFE
NO COMMENTS

TACTICAL STEPS FOR CLOUD MIGRATION
NO COMMENTS

IMMUTA DATA ACCESS PLATFORM
NO COMMENTS

DATA MESH: FACT OR FICTION?
NO COMMENTS

‹ ›




CONTRIBUTORS

Tiffany Trader
Editorial Director



Alex Woodie
Managing Editor



Douglas Eadline
Contributing Editor



Jamie Hampton
Contributing Editor



Kevin Jackson
Contributing Editor



Ali Azhar
Contributing Editor



John Russell
Contributing Editor



Steve Conway
Contributing Editor



Drew Jolly
Assistant Editor








© 2024 BigDATAwire. All Rights Reserved. A Tabor Communications Publication
 * Back to Top
 * Contact
 * Privacy Policy
 * Cookie Policy
 * About BigDATAwire
 * Update Subscription Preferences
 * California Consumers

BigDATAwire
This website uses cookies to improve your experience. We'll assume you're ok
with this, but you can opt-out if you wish.Accept Read More
Privacy & Cookies Policy
Close

PRIVACY OVERVIEW

This website uses cookies to improve your experience while you navigate through
the website. Out of these, the cookies that are categorized as necessary are
stored on your browser as they are essential for the working of basic
functionalities of the ...
Necessary
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly.
This category only includes cookies that ensures basic functionalities and
security features of the website. These cookies do not store any personal
information.
Non-necessary
Non-necessary
Any cookies that may not be particularly necessary for the website to function
and is used specifically to collect user personal data via analytics, ads, other
embedded contents are termed as non-necessary cookies. It is mandatory to
procure user consent prior to running these cookies on your website.
SAVE & ACCEPT





SHARE

Blogger
Delicious
Digg
Email
Facebook
Facebook messenger
Flipboard
Google
Hacker News
Line
LinkedIn
Mastodon
Mix
Odnoklassniki
PDF
Pinterest
Pocket
Print
Reddit
Renren
Short link
SMS
Skype
Telegram
Tumblr
Twitter
VKontakte
wechat
Weibo
WhatsApp
X
Xing
Yahoo! Mail


COPY SHORT LINK

Copy link

Original text

Rate this translation
Your feedback will be used to help improve Google Translate