redpanda.com
Open in
urlscan Pro
76.76.21.21
Public Scan
Submitted URL: https://app.go.singlestore.com/e/er?s=1387486446&lid=5713&elqtrackid=7e41d28725c747908b522bcaf9c12f56&elq=45e806ef99964e8fb7bdb...
Effective URL: https://redpanda.com/blog/build-real-time-data-analysis-singlestore-redpanda?utm_medium=email&utm_source=singlestore&...
Submission: On July 28 via api from US — Scanned from DE
Effective URL: https://redpanda.com/blog/build-real-time-data-analysis-singlestore-redpanda?utm_medium=email&utm_source=singlestore&...
Submission: On July 28 via api from US — Scanned from DE
Form analysis 1 forms found in the DOM
POST https://forms.hsforms.com/submissions/v3/public/submit/formsnext/multipart/7733588/cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca
<form id="hsForm_cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" method="POST" accept-charset="UTF-8" enctype="multipart/form-data" novalidate=""
action="https://forms.hsforms.com/submissions/v3/public/submit/formsnext/multipart/7733588/cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"
class="hs-form-private hsForm_cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca hs-form-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca hs-form-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca_bae29192-3849-4587-a3a0-411402d48425 hs-form stacked"
target="target_iframe_cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" data-instance-id="bae29192-3849-4587-a3a0-411402d48425" data-form-id="cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" data-portal-id="7733588">
<fieldset class="form-columns-3">
<div class="hs_blog_signup hs-blog_signup hs-fieldtype-booleancheckbox field hs-form-field">
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input">
<ul class="inputs-list">
<li class="hs-form-booleancheckbox"><label for="blog_signup-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="hs-form-booleancheckbox-display"><input id="blog_signup-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="hs-input" type="checkbox"
name="blog_signup" value="true"><span>Blog</span></label></li>
</ul>
</div>
</div>
<div class="hs_newsletter_signup hs-newsletter_signup hs-fieldtype-booleancheckbox field hs-form-field">
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input">
<ul class="inputs-list">
<li class="hs-form-booleancheckbox"><label for="newsletter_signup-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="hs-form-booleancheckbox-display"><input id="newsletter_signup-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="hs-input"
type="checkbox" name="newsletter_signup" value="true"><span>Newsletter</span></label></li>
</ul>
</div>
</div>
<div class="hs_email hs-email hs-fieldtype-text field hs-form-field"><label id="label-email-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your Email" for="email-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>Email</span><span
class="hs-form-required">*</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input id="email-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" name="email" required="" placeholder="Email" type="email" class="hs-input" inputmode="email" autocomplete="email" value=""></div>
</div>
</fieldset>
<fieldset class="form-columns-3">
<div class="hs_utm_medium hs-utm_medium hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_medium-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your UTM Medium"
for="utm_medium-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>UTM Medium</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_medium" class="hs-input" type="hidden" value="email"></div>
</div>
<div class="hs_utm_source hs-utm_source hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_source-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your UTM Source"
for="utm_source-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>UTM Source</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_source" class="hs-input" type="hidden" value="singlestore"></div>
</div>
<div class="hs_utm_campaign hs-utm_campaign hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_campaign-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your UTM Campaign"
for="utm_campaign-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>UTM Campaign</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_campaign" class="hs-input" type="hidden" value="july-newsletter-2023"></div>
</div>
</fieldset>
<fieldset class="form-columns-2">
<div class="hs_utm_content hs-utm_content hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_content-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your UTM Content"
for="utm_content-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>UTM Content</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_content" class="hs-input" type="hidden" value=""></div>
</div>
<div class="hs_utm_term hs-utm_term hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_term-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your UTM Term"
for="utm_term-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>UTM Term</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_term" class="hs-input" type="hidden" value=""></div>
</div>
</fieldset>
<fieldset class="form-columns-3">
<div class="hs_utm_assettype hs-utm_assettype hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_assettype-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your utm_assettype"
for="utm_assettype-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>utm_assettype</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_assettype" class="hs-input" type="hidden" value=""></div>
</div>
<div class="hs_utm_assetname hs-utm_assetname hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_assetname-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your utm_assetname"
for="utm_assetname-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>utm_assetname</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_assetname" class="hs-input" type="hidden" value=""></div>
</div>
<div class="hs_utm_version hs-utm_version hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-utm_version-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your utm_version"
for="utm_version-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>utm_version</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="utm_version" class="hs-input" type="hidden" value=""></div>
</div>
</fieldset>
<fieldset class="form-columns-1">
<div class="hs_last_marketing_touch__c hs-last_marketing_touch__c hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-last_marketing_touch__c-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class=""
placeholder="Enter your Last Marketing Touch" for="last_marketing_touch__c-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>Last Marketing Touch</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="last_marketing_touch__c" class="hs-input" type="hidden" value="Newsletter & Blog Sign Up"></div>
</div>
</fieldset>
<fieldset class="form-columns-1">
<div class="hs_gclid hs-gclid hs-fieldtype-text field hs-form-field" style="display: none;"><label id="label-gclid-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" class="" placeholder="Enter your GCLID"
for="gclid-cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca"><span>GCLID</span></label>
<legend class="hs-field-desc" style="display: none;"></legend>
<div class="input"><input name="gclid" class="hs-input" type="hidden" value=""></div>
</div>
</fieldset>
<div class="hs_submit hs-submit">
<div class="hs-field-desc" style="display: none;"></div>
<div class="actions"><input type="submit" class="hs-button primary large" value="SIGN ME UP"></div>
</div><input name="hs_context" type="hidden"
value="{"embedAtTimestamp":"1690564116556","formDefinitionUpdatedAt":"1672779053116","lang":"en","embedType":"REGULAR","disableCookieSubmission":"true","clonedFromForm":"38a50b00-c1f6-4158-8d09-6a7f96ef6485","renderRawHtml":"true","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.5790.110 Safari/537.36","pageTitle":"Unleashing the power of real-time analytics with SingleStore and Redpanda","pageUrl":"https://redpanda.com/blog/build-real-time-data-analysis-singlestore-redpanda?utm_medium=email&utm_source=singlestore&utm_campaign=july-newsletter-2023&campaignid=july-newsletter-2023","urlParams":{"utm_medium":"email","utm_source":"singlestore","utm_campaign":"july-newsletter-2023","campaignid":"july-newsletter-2023"},"isHubSpotCmsGeneratedPage":false,"formTarget":"#blog-post-form-updates","locale":"en","timestamp":1690564116654,"originalEmbedContext":{"portalId":"7733588","formId":"cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca","region":"na1","target":"#blog-post-form-updates","isBuilder":false,"isTestPage":false,"isPreview":false,"isMobileResponsive":true},"correlationId":"bae29192-3849-4587-a3a0-411402d48425","renderedFieldsIds":["blog_signup","newsletter_signup","email","utm_medium","utm_source","utm_campaign","utm_content","utm_term","utm_assettype","utm_assetname","utm_version","last_marketing_touch__c","gclid"],"captchaStatus":"NOT_APPLICABLE","emailResubscribeStatus":"NOT_APPLICABLE","isInsideCrossOriginFrame":false,"source":"forms-embed-1.3479","sourceName":"forms-embed","sourceVersion":"1.3479","sourceVersionMajor":"1","sourceVersionMinor":"3479","_debug_allPageIds":{},"_debug_embedLogLines":[{"clientTimestamp":1690564116630,"level":"INFO","message":"Retrieved pageContext values which may be overriden by the embed context: {\"pageTitle\":\"Unleashing the power of real-time analytics with SingleStore and Redpanda\",\"pageUrl\":\"https://redpanda.com/blog/build-real-time-data-analysis-singlestore-redpanda?utm_medium=email&utm_source=singlestore&utm_campaign=july-newsletter-2023&campaignid=july-newsletter-2023\",\"userAgent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.5790.110 Safari/537.36\",\"urlParams\":{\"utm_medium\":\"email\",\"utm_source\":\"singlestore\",\"utm_campaign\":\"july-newsletter-2023\",\"campaignid\":\"july-newsletter-2023\"},\"isHubSpotCmsGeneratedPage\":false}"},{"clientTimestamp":1690564116633,"level":"INFO","message":"Retrieved countryCode property from normalized embed definition response: \"DE\""}]}"><iframe
name="target_iframe_cfc119e8-f8e5-4bd6-94b4-dcbc4bb4e3ca" style="display: none;"></iframe>
</form>Text Content
* PLATFORM
* Overview
Technical tour of the Redpanda platform
* Core capabilities
Explore all features and functions
* Platform TCO
6x lower TCO - see how
* Redpanda Console
An interactive web UI for Kafka
* Deployment options
Compare self-hosted and cloud options
* CLOUD
* CUSTOMERS
* BLOG
* LEARN
* Documentation
Dive deep into features and APIs.
* Redpanda University
Guided courses on building real-time apps.
* Resources
Tech talks, workshops, reports, and more.
* Community Slack
Ask our dev community your burning questions.
* COMPANY
* About us
Meet our team and investors.
* Careers
Explore Redpanda opportunities and culture.
* CONTACT
* Request demo
Request private tech talks, pricing, product resources.
* Join our Slack
Get answers from our community.
* Support request
Submit technical support ticket.
* Try Redpanda
*
* 7.5K
Blog
Ecosystem
UNLEASHING THE POWER OF REAL-TIME ANALYTICS WITH SINGLESTORE AND REDPANDA
Learn how to build a modern clickstream analytics system using Redpanda and
SingleStore
ByManish KumarChris LarsenonJune 20, 2023
Copy link
The ability to use real-time data to make business decisions is critical in
today’s world. In the modern business landscape, data has become the new oil,
fueling growth and innovation across industries. Yet, as crucial as data is, its
true value lies not only in the volume accumulated but in the speed at which we
can process and analyze it effectively and efficiently to drive business
outcomes.
The sheer magnitude of data generated every day requires tools that facilitate
high-speed transactions, real-time analytics, and robust data streaming.
Recognizing these requirements, two platforms stand out: SingleStoreDB and
Redpanda. While each is powerful in its own right, integrating the two can
revolutionize your data management strategy by bringing together the best of
data streaming and real-time analytics.
Let's take an example of a clickstream data. Real-time clickstream analysis is a
pervasive challenge in today's world of online interactions. It involves
tracking and analyzing the sequence of clicks that a user makes while navigating
a website or application, which is leveraged for improving user experiences,
personalizing content and optimizing marketing strategies.
The complexity of dealing with high volume, velocity, and variety of clickstream
data make its real-time processing extra challenging. Users typically use an
event streaming platform like Kafka to process the clickstream, but have
historically struggled with the end database on which they need to do complex
analysis.
Building such a real-time analytics system requires both ingesting large amounts
of events and serving the analytical needs of the application as quickly as
possible. Such integrations involve plumbing multiple tools and custom solutions
which makes it hard to manage and scale. This is where Redpanda’s integration
with SingleStoreDB provides a best-in-class option for customers building such
applications.
WHAT'S REDPANDA?
If you're new here, Redpanda is a modern streaming platform that acts as a
drop-in replacement for Apache Kafka®. Kafka was once a streaming data
superpower, but it struggles with modern data-intensive requirements--sparking a
need for leaner, faster streaming data alternatives.
Redpanda is designed to offer higher performance with a simpler, more
developer-friendly architecture compared to Kafka. It's fully API compatible
with Kafka, which makes it highly suitable for building such applications and
also using existing Kafka applications without any code changes.
This makes it easy to integrate with SingleStore pipelines, which can ingest
massive amounts of data into SingleStore and make it queryable in real time.
Additionally, this can be done in a simple SQL-like interface, which makes using
these configurations seamless for developers.
WHAT'S SINGLESTORE?
SingleStore is a distributed, SQL database that unifies transactions and
analytics in a single engine to drive low-latency access to large datasets,
simplifying the development of fast, modern applications. Built for developers
and architects, SingleStoreDB is based on a distributed SQL architecture,
delivering 10-100 millisecond performance on complex queries, while ensuring
businesses can effortlessly scale.
In this blog post, we will show how to set up a Redpanda cluster that receives
clickstream data from a variety of sources. This data can then be ingested into
a SingleStoreDB cluster running on AWS to provide rich insights in real time.
THE CHALLENGES OF BUILDING A MODERN CLICKSTREAM ANALYTICS SYSTEM
Building a modern clickstream analytics system involves collecting, processing,
analyzing, and storing large amounts of data in real time. Let's take an example
of the user operation doing multiple clicks online. All of the clickstream data
needs to be processed to drive business value or predictive analytics. At a high
level, it involves four steps:
1. Collect data from variety of sources
2. Ingest data
3. Store the data
4. Drive business insights by consuming the data
All of these operations need to happen in real time while ensuring the
architecture is robust for enterprise readiness, scalable based on business
needs, and at a reasonable cost.
While using real-time stream processing systems such as Apache Flink and Apache
Spark can provide a low latency solution, quick ingestion into a database, and
quick processing-—Redpanda's integration with SingleStore provides the best
parts of both tools.
Typical databases struggle with the speed of ingestion and have to rely on
external tools. However, SingleStore supports native capability called Pipelines
which help in super-fast ingestion from Kafka.
BUILD A REAL-TIME ANALYTICS SYSTEM IN THREE STEPS
Now we'll show a simple setup of ingesting a large amount of clickstream data
from Redpanda to SingleStoreDB cloud running on AWS.
Clickstream Kafka pipelines to SingleStore
Redpanda is easy to deploy in the cloud using one of two options: Dedicated
Cloud (provisioned in Redpanda’s tenant, AWS in this case) or Bring Your Own
Cloud (BYOC - provisioned in your tenant yet still fully managed with Redpanda’s
unified control plane). The solution in this tutorial was built using Redpanda’s
BYOC model.
To build the connection with SingleStoreDB Cloud, customers can set up a Kafka.
Now that you have SingleStoreDB cluster running, we'll create a pipeline that
can capture the incoming stream of data natively into SingleStore. All you need
is three steps:
1 - Set up the actual pipeline using SingleStore Kafka pipelines.
CREATE OR REPLACE PIPELINE `<Pipeline_name> `
AS LOAD DATA KAFKA 'Redpanda_topic_1,
Redpanda_topic_2,
Redpanda_topic_3,
CONFIG '{
"sasl.username": "<user_name> ",
"sasl.mechanism": "SCRAM-SHA-256",
"security.protocol": "SASL_SSL"
}'
CREDENTIALS '{
"sasl.password": "REDACTED"
}'
DISABLE OUT_OF_ORDER OPTIMIZATION
INTO TABLE <table_name>
FORMAT JSON
(
field_1<- value_1,
field_2<- value_2,
field_3<- value_3,
field_4<- value_4,
field_5<- value_5,
)
ON DUPLICATE KEY UPDATE
field_1= VALUES(value_1),
field_2= VALUES(value_2),
field_3= VALUES(value_3),
field_4= VALUES(value_4),
field_5= VALUES(value_5),
;
2 - Once we have the pipeline created you can check the pipeline by checking the
sample data
> TEST PIPELINE `<Pipeline_name> `
3 - Once you have verified that the pipeline works perfectly fine through the
sample data, you can start the pipeline and see that the data should start
flowing.
> START PIPELINE `<Pipeline_name> `
Just like that, you can start getting the data into SingleStore, which is
instantly available for queries and provide quick insights.
EMPOWER YOUR REAL-TIME DATA STRATEGY
The combination of SingleStore and Redpanda offers a best-in-class solution for
organizations seeking real-time analytics and high-speed data processing. By
harnessing the power of these platforms, businesses can stay ahead in today’s
data-driven landscape. In this step-by-step blog, we demonstrated how to set up
the connection between Redpanda and SingleStoreDB running on AWS.
Redpanda provides a real-time ingestion platform that can be a drop-in
replacement for Kafka. SingleStore provides unique capability to solve both
transactional and analytical needs of your application and makes data available
for query as soon as it's loaded. This makes building real-time applications
simple and efficient—a perfect choice for storing real-time data, streamed from
Redpanda.
Interested in trying out SingleStore and Redpanda? Learn how to get started
building with SingleStoreDB and try Redpanda for free. You can also browse the
Redpanda blog for step-by-step tutorials and real-world customer stories. If you
have questions or just want to chat with fellow Redpanda users, join the
Redpanda Community on Slack.
ADDITIONAL RESOURCES
* Learn more about what’s new in SingleStore’s latest release 8.1
* Learn how SingleStore and Redpanda can work together to solve the operational
complexity of global logistics
* Redpanda vs. Kafka: comparing performance and total cost of ownership
* Develop real-time apps faster and simpler with Redpanda
LET'S KEEP IN TOUCH
Subscribe and never miss another blog post, announcement, or community event. We
hate spam and will never sell your contact information.
* Blog
* Newsletter
Email*
UTM Medium
UTM Source
UTM Campaign
UTM Content
UTM Term
utm_assettype
utm_assetname
utm_version
Last Marketing Touch
GCLID
RELATED ARTICLES
View all posts
EcosystemTutorial
GETTING STARTED WITH REDPANDA USING NODE.JS
Mateo Carvajal •
December 15, 2020
Ecosystem
KAFKA STREAMS VS. SPARK STREAMING
Idowu Odesanmi •
March 28, 2023
Ecosystem
MODERN EVENTING WITH CQRS, REDPANDA, AND ZILLA (PART 2)
John Fallows •
May 16, 2023
View all posts
The streaming data platform for developers.
5758 Geary Blvd. #153
San Francisco, CA 94121
WHAT IS REDPANDA?
* Capabilities
* Redpanda University
* Kafka vs Redpanda
* Jepsen report
* Quick start guide
COMPANY
* Contact
* About us
* Careers
* Press
* Scholarship
QUICK LINKS
* Pricing
* Docs
* Blog
* Support
* Privacy policy
* Terms of use
* Security
* Cloud SLA
--------------------------------------------------------------------------------
© 2023 Redpanda Data Inc.
Cookie preferences
*
*
*
*
*
This website uses cookies to enhance the user experience and to better
understand how our site is used. By continuing to use this site or clicking on
Accept, you are consenting to our use of cookies and our privacy policy.
Cookies Settings Accept All Cookies
1