blog.avenuecode.com
Open in
urlscan Pro
2606:2c40::c73c:67e1
Public Scan
Submitted URL: https://info.avenuecode.com/e3t/Btc/WY+113/cjhhH04/VVYL6c1Lc6msN5knfL8yBfvRVZjc4S4z8XvGN3Gk3893lLBGV1-WJV7CgJT8W4pVms82pBkSB...
Effective URL: https://blog.avenuecode.com/data-lakes-the-key-to-data-modernization?utm_medium=email&_hsmi=175727050&_hsenc=p2ANqtz-8d8whtE...
Submission: On October 28 via manual from US — Scanned from DE
Effective URL: https://blog.avenuecode.com/data-lakes-the-key-to-data-modernization?utm_medium=email&_hsmi=175727050&_hsenc=p2ANqtz-8d8whtE...
Submission: On October 28 via manual from US — Scanned from DE
Form analysis 2 forms found in the DOM
POST https://forms.hsforms.com/submissions/v3/public/submit/formsnext/multipart/2564010/589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4
<form novalidate="" accept-charset="UTF-8" action="https://forms.hsforms.com/submissions/v3/public/submit/formsnext/multipart/2564010/589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4" enctype="multipart/form-data"
id="hsForm_589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" method="POST"
class="hs-form stacked hs-form-private hsForm_589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4 hs-form-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4 hs-form-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_cca08088-a23c-4b43-8651-8009c7f9b953"
data-form-id="589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4" data-portal-id="2564010" target="target_iframe_589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" data-reactid=".hbspt-forms-1">
<div class="hs_firstname hs-firstname hs-fieldtype-text field hs-form-field" data-reactid=".hbspt-forms-1.1:$0"><label id="label-firstname-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="" placeholder="Enter your First Name"
for="firstname-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" data-reactid=".hbspt-forms-1.1:$0.0"><span data-reactid=".hbspt-forms-1.1:$0.0.0">First Name</span><span class="hs-form-required" data-reactid=".hbspt-forms-1.1:$0.0.1">*</span></label>
<legend class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-1.1:$0.1"></legend>
<div class="input" data-reactid=".hbspt-forms-1.1:$0.$firstname"><input id="firstname-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="hs-input" type="text" name="firstname" required="" value="" placeholder="" autocomplete="given-name"
data-reactid=".hbspt-forms-1.1:$0.$firstname.0" inputmode="text"></div>
</div>
<div class="hs_lastname hs-lastname hs-fieldtype-text field hs-form-field" data-reactid=".hbspt-forms-1.1:$1"><label id="label-lastname-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="" placeholder="Enter your Last Name"
for="lastname-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" data-reactid=".hbspt-forms-1.1:$1.0"><span data-reactid=".hbspt-forms-1.1:$1.0.0">Last Name</span></label>
<legend class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-1.1:$1.1"></legend>
<div class="input" data-reactid=".hbspt-forms-1.1:$1.$lastname"><input id="lastname-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="hs-input" type="text" name="lastname" value="" placeholder="" autocomplete="family-name"
data-reactid=".hbspt-forms-1.1:$1.$lastname.0" inputmode="text"></div>
</div>
<div class="hs_email hs-email hs-fieldtype-text field hs-form-field" data-reactid=".hbspt-forms-1.1:$2"><label id="label-email-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="" placeholder="Enter your Email"
for="email-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" data-reactid=".hbspt-forms-1.1:$2.0"><span data-reactid=".hbspt-forms-1.1:$2.0.0">Email</span><span class="hs-form-required" data-reactid=".hbspt-forms-1.1:$2.0.1">*</span></label>
<legend class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-1.1:$2.1"></legend>
<div class="input" data-reactid=".hbspt-forms-1.1:$2.$email"><input id="email-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="hs-input" type="email" name="email" required="" placeholder="" value="" autocomplete="email"
data-reactid=".hbspt-forms-1.1:$2.$email.0" inputmode="email"></div>
</div>
<div class="hs_website hs-website hs-fieldtype-text field hs-form-field" data-reactid=".hbspt-forms-1.1:$3"><label id="label-website-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="" placeholder="Enter your Website"
for="website-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" data-reactid=".hbspt-forms-1.1:$3.0"><span data-reactid=".hbspt-forms-1.1:$3.0.0">Website</span></label>
<legend class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-1.1:$3.1"></legend>
<div class="input" data-reactid=".hbspt-forms-1.1:$3.$website"><input id="website-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="hs-input" type="text" name="website" value="" placeholder="" data-reactid=".hbspt-forms-1.1:$3.$website.0"
inputmode="url"></div>
</div>
<div class="hs_comment hs-comment hs-fieldtype-textarea field hs-form-field" data-reactid=".hbspt-forms-1.1:$4"><label id="label-comment-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="" placeholder="Enter your Comment"
for="comment-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" data-reactid=".hbspt-forms-1.1:$4.0"><span data-reactid=".hbspt-forms-1.1:$4.0.0">Comment</span><span class="hs-form-required" data-reactid=".hbspt-forms-1.1:$4.0.1">*</span></label>
<legend class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-1.1:$4.1"></legend>
<div class="input" data-reactid=".hbspt-forms-1.1:$4.$comment"><textarea id="comment-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class="hs-input" name="comment" required="" placeholder="" data-reactid=".hbspt-forms-1.1:$4.$comment.0"></textarea>
</div>
</div>
<div class="hs_lifecyclestage hs-lifecyclestage hs-fieldtype-radio field hs-form-field" style="display:none;" data-reactid=".hbspt-forms-1.1:$5"><label id="label-lifecyclestage-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" class=""
placeholder="Enter your Lifecycle Stage" for="lifecyclestage-589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" data-reactid=".hbspt-forms-1.1:$5.0"><span data-reactid=".hbspt-forms-1.1:$5.0.0">Lifecycle Stage</span></label>
<legend class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-1.1:$5.1"></legend>
<div class="input" data-reactid=".hbspt-forms-1.1:$5.$lifecyclestage"><input name="lifecyclestage" class="hs-input" type="hidden" value="subscriber" data-reactid=".hbspt-forms-1.1:$5.$lifecyclestage.0"></div>
</div><noscript data-reactid=".hbspt-forms-1.2"></noscript>
<div class="hs_recaptcha hs-recaptcha field hs-form-field" data-reactid=".hbspt-forms-1.3">
<div class="input" data-reactid=".hbspt-forms-1.3.0">
<div class="grecaptcha-badge" data-style="inline" style="width: 256px; height: 60px; box-shadow: gray 0px 0px 5px;">
<div class="grecaptcha-logo"><iframe title="reCAPTCHA"
src="https://www.google.com/recaptcha/enterprise/anchor?ar=1&k=6Ld_ad8ZAAAAAAqr0ePo1dUfAi0m4KPkCMQYwPPm&co=aHR0cHM6Ly9ibG9nLmF2ZW51ZWNvZGUuY29tOjQ0Mw..&hl=de&v=YhkYx1k-yvvb8OonJPmOpoJY&size=invisible&badge=inline&cb=9rc6n7vbr406"
width="256" height="60" role="presentation" name="a-j17zy73jkkfa" frameborder="0" scrolling="no" sandbox="allow-forms allow-popups allow-same-origin allow-scripts allow-top-navigation allow-modals allow-popups-to-escape-sandbox"></iframe>
</div>
<div class="grecaptcha-error"></div><textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response"
style="width: 250px; height: 40px; border: 1px solid rgb(193, 193, 193); margin: 10px 25px; padding: 0px; resize: none; display: none;"></textarea>
</div><iframe style="display: none;"></iframe>
</div><input type="hidden" name="g-recaptcha-response" id="hs-recaptcha-response" value="" data-reactid=".hbspt-forms-1.3.1">
</div>
<div class="hs_submit hs-submit" data-reactid=".hbspt-forms-1.5">
<div class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-1.5.0"></div>
<div class="actions" data-reactid=".hbspt-forms-1.5.1"><input type="submit" value="Submit Comment" class="hs-button primary" data-reactid=".hbspt-forms-1.5.1.0"></div>
</div><noscript data-reactid=".hbspt-forms-1.6"></noscript><input name="hs_context" type="hidden"
value="{"rumScriptExecuteTime":2251.800000190735,"rumServiceResponseTime":2468.300000190735,"rumFormRenderTime":1.8000001907348633,"rumTotalRenderTime":2470.800000190735,"rumTotalRequestTime":215.59999990463257,"embedAtTimestamp":"1635393966906","formDefinitionUpdatedAt":"1533138964229","pageUrl":"https://blog.avenuecode.com/data-lakes-the-key-to-data-modernization?utm_medium=email&_hsmi=175727050&_hsenc=p2ANqtz-8d8whtE0r437M1InI-L9A_ay0HoElQxGwKM1OXDl3QWxrAZq57CzQmkSzbDsQfTbZnT4KIKxqITEYpk1tPm56c5jJ3ng&utm_content=175727050&utm_source=hs_email","pageTitle":"Data Lakes: The Key to Data Modernization","source":"FormsNext-static-5.393","timestamp":1635393966908,"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","originalEmbedContext":{"portalId":"2564010","formId":"589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4","pageId":"58071379878","region":"na1","pageName":"Data Lakes: The Key to Data Modernization","contentType":"blog-post","formsBaseUrl":"/_hcms/forms/","target":"#hs_form_target_589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4","type":"BLOG_COMMENT","submitButtonClass":"hs-button primary","formInstanceId":"8109"},"canonicalUrl":"https://blog.avenuecode.com/data-lakes-the-key-to-data-modernization","pageId":"58071379878","pageName":"Data Lakes: The Key to Data Modernization","formInstanceId":"8109","urlParams":{"utm_medium":"email","_hsmi":"175727050","_hsenc":"p2ANqtz-8d8whtE0r437M1InI-L9A_ay0HoElQxGwKM1OXDl3QWxrAZq57CzQmkSzbDsQfTbZnT4KIKxqITEYpk1tPm56c5jJ3ng","utm_content":"175727050","utm_source":"hs_email"},"renderedFieldsIds":["firstname","lastname","email","website","comment"],"formTarget":"#hs_form_target_589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4","correlationId":"b8afbccc-1eea-4ef5-b64e-2dfeca511a79","contentType":"blog-post","hutk":"5634db99b1225cf0b539462d617c2526","useRecaptchaEnterprise":true,"isHostedOnHubspot":true}"
data-reactid=".hbspt-forms-1.7"><iframe name="target_iframe_589cf7bc-db7b-4a6f-bf0c-e6e5d5d8dfc4_8109" style="display:none;" data-reactid=".hbspt-forms-1.8"></iframe>
</form>POST https://forms.hsforms.com/submissions/v3/public/submit/formsnext/multipart/2564010/baf4c424-d1cd-4e62-84d7-995d9eba60cf
<form novalidate="" accept-charset="UTF-8" action="https://forms.hsforms.com/submissions/v3/public/submit/formsnext/multipart/2564010/baf4c424-d1cd-4e62-84d7-995d9eba60cf" enctype="multipart/form-data" id="hsForm_baf4c424-d1cd-4e62-84d7-995d9eba60cf"
method="POST" class="hs-form stacked hs-form-private hsForm_baf4c424-d1cd-4e62-84d7-995d9eba60cf hs-form-baf4c424-d1cd-4e62-84d7-995d9eba60cf hs-form-baf4c424-d1cd-4e62-84d7-995d9eba60cf_43514637-698d-4f05-a758-5c3beb0be371"
data-form-id="baf4c424-d1cd-4e62-84d7-995d9eba60cf" data-portal-id="2564010" target="target_iframe_baf4c424-d1cd-4e62-84d7-995d9eba60cf" data-reactid=".hbspt-forms-0">
<div class="hs_email hs-email hs-fieldtype-text field hs-form-field" data-reactid=".hbspt-forms-0.1:$0"><label id="label-email-baf4c424-d1cd-4e62-84d7-995d9eba60cf" class="" placeholder="Enter your " for="email-baf4c424-d1cd-4e62-84d7-995d9eba60cf"
data-reactid=".hbspt-forms-0.1:$0.0"><span data-reactid=".hbspt-forms-0.1:$0.0.0"></span></label>
<legend class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-0.1:$0.1"></legend>
<div class="input" data-reactid=".hbspt-forms-0.1:$0.$email"><input id="email-baf4c424-d1cd-4e62-84d7-995d9eba60cf" class="hs-input" type="email" name="email" required="" placeholder="Email*" value="" autocomplete="email"
data-reactid=".hbspt-forms-0.1:$0.$email.0" inputmode="email"></div>
</div><noscript data-reactid=".hbspt-forms-0.2"></noscript>
<div class="hs_submit hs-submit" data-reactid=".hbspt-forms-0.5">
<div class="hs-field-desc" style="display:none;" data-reactid=".hbspt-forms-0.5.0"></div>
<div class="actions" data-reactid=".hbspt-forms-0.5.1"><input type="submit" value="Submit" class="hs-button primary large" data-reactid=".hbspt-forms-0.5.1.0"></div>
</div><noscript data-reactid=".hbspt-forms-0.6"></noscript><input name="hs_context" type="hidden"
value="{"rumScriptExecuteTime":2223,"rumServiceResponseTime":2431.2000002861023,"rumFormRenderTime":2.3999996185302734,"rumTotalRenderTime":2437.5999999046326,"rumTotalRequestTime":207.10000038146973,"renderRawHtml":"true","embedAtTimestamp":"1635393966893","formDefinitionUpdatedAt":"1479490395291","pageUrl":"https://blog.avenuecode.com/data-lakes-the-key-to-data-modernization?utm_medium=email&_hsmi=175727050&_hsenc=p2ANqtz-8d8whtE0r437M1InI-L9A_ay0HoElQxGwKM1OXDl3QWxrAZq57CzQmkSzbDsQfTbZnT4KIKxqITEYpk1tPm56c5jJ3ng&utm_content=175727050&utm_source=hs_email","pageTitle":"Data Lakes: The Key to Data Modernization","source":"FormsNext-static-5.393","timestamp":1635393966893,"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","originalEmbedContext":{"portalId":"2564010","formId":"baf4c424-d1cd-4e62-84d7-995d9eba60cf","target":"#hbspt-form-1635393966756-32666956"},"canonicalUrl":"https://blog.avenuecode.com/data-lakes-the-key-to-data-modernization","pageId":"58071379878","urlParams":{"utm_medium":"email","_hsmi":"175727050","_hsenc":"p2ANqtz-8d8whtE0r437M1InI-L9A_ay0HoElQxGwKM1OXDl3QWxrAZq57CzQmkSzbDsQfTbZnT4KIKxqITEYpk1tPm56c5jJ3ng","utm_content":"175727050","utm_source":"hs_email"},"renderedFieldsIds":["email"],"formTarget":"#hbspt-form-1635393966756-32666956","correlationId":"0831b4d9-83db-4759-8e0d-212b7bef9e33","contentType":"blog-post","hutk":"5634db99b1225cf0b539462d617c2526","useRecaptchaEnterprise":true,"isHostedOnHubspot":true}"
data-reactid=".hbspt-forms-0.7"><iframe name="target_iframe_baf4c424-d1cd-4e62-84d7-995d9eba60cf" style="display:none;" data-reactid=".hbspt-forms-0.8"></iframe>
</form>Text Content
This website utilizes cookies to improve site performance and personalize your experience. If disabled, certain features of the site may be unavailable. Dismiss Disable Cookies AvenueCode.com News Contact menu * Agile * Back-end * Cloud * Front-end * Industry * Integration * Mobile * Testing * Agile * Back-end * Front-end * Industry * Integration * Mobile * Testing DATA LAKES: THE KEY TO DATA MODERNIZATION * * Share * perm_identity Frederico Caram schedule 10/27/21 2:00 PM Nearly every business is facing new challenges in how to store and effectively utilize data. Today we'll look at how data lakes solve these challenges and why GCP is the available, scalable, cost-efficient data lake hosting solution your business needs. DATA CHALLENGES IN TODAY'S BUSINESS CONTEXT In today's environment, businesses face new challenges arising from the sheer volume of data, the increase in the generation and consumption of unstructured data, the need to have real or near real-time data, and the trend toward migrating data from on-premise servers to the cloud. Legacy data warehouses are unable to adequately handle unstructured data since they usually depend on fixed schemas to perform well, and that requires structured data formats. They also have limitations when it comes to scaling horizontally, making it more expensive to handle high volumes of data or high ingestion rates. Ultimately, moving from on-premise servers to the cloud, alongside computational evolutions, made it much cheaper to store data than to process it. So it became easier to store multiple copies of transformed data than to process it all at once, resulting in new paradigms, such as moving from ETL (extract-transform-load) to ELT (extract-load-transform). DATA LAKES AND DATA WAREHOUSES To overcome some of these challenges, a new concept arose: Data Lakes. A Data Lake is a centralized place to store data, allowing us to ingest, store, transform, analyze, and model data in a secure, cost-effective, easily organized, and manageable way. Data Lakes aren’t supposed to replace Data Warehouses (check out our previous post on Data Warehouses), but they can integrate or incorporate Data Warehouses to make the best use of each solution. It's important to note that a Data Lake is an architectural concept and not a tool. Even though it is very common for vendors to relate a Data Lake to a specific tool, Data Lakes usually utilize a variety of tools to load, store, transform, and expose data. This data concept is not only about moving and transforming your data through data pipelines, but it's also about doing it in a traceable way, managing the data lifecycle and its lineage, identifying sensitive information and making sure that the data only gets to people who are authorized to see it. DATA LAKE ORGANIZATION Data Lakes are usually organized into layers. As data moves from one layer to another through data pipelines, it gets cleaner, more informative, more trustworthy, better curated, and altogether more meaningful for the business. One common way to structure Data Lake layers is to categorize them as Bronze, Silver, and Gold layers, as displayed in the following figure: Image courtesy of Medium. WHY USE GCP FOR YOUR DATA LAKE One easy and effective way to organize your Data Lake is through Google Cloud Platform (GCP). GCP provides tools to cover most of your Data Lake needs with easily scalable and manageable serverless components so that the team can focus on what brings value to the business instead of focusing on how the infrastructure will be managed for the most common scenarios: Batch Ingestion: GCP provides many tools to copy and receive data from other cloud providers or on-premise structures, such as: Transfer Appliance, Transfer Service, and GSutil. In scenarios where data must be received from or called by an API, we can also use computing tools such as Cloud Functions, Cloud Run, App Engine, Compute Engine or even Cloud Data Fusion. Stream/Real Time Ingestion: Usually the best tools for stream ingestion in GCP are Cloud Pub/Sub combined Dataflow, where Pub/Sub is responsible for receiving the data and Dataflow is responsible for processing it and moving it to a persistent storage. Change Data Capture: If you are using MySQL or Oracle databases, Datastream allows you to stream your changing data to cloud storage. Landing Zone and Raw Data: The most common use case is to use Cloud storage for your landing data and raw data, but in certain scenarios, BigQuery, BigTable, or Pub/Sub can also be used. Data Warehouse and Data Marts: The most common tool for these scenarios is BigQuery, but GCP also offers alternatives, such as DataProc and Databricks for companies that prefer or need to use Apache Hadoop Stack. GCP Architecture provides the following reference as an example of what's possible: Image courtesy of Google Cloud. THE AVAILABLE, SCALABLE, COST-EFFICIENT DATA SOLUTION In summary, GCP can help you achieve the data availability and scalability your business needs at a competitive cost and with a complete ecosystem to support the most common business scenarios; and you won't have to worry about the underlying infrastructure or operations. At Avenue Code, we have several Google Cloud Platform experts who can help you modernize your Data Warehouse to be highly available, scalable, and cost-efficient. Don't hesitate to reach out to discuss your project! LEARN MORE ABOUT DATA MODERNIZATION Want to learn more about data modernization and analysis? Be sure to check out the other blogs in our series: The 6 Pillars of Data Modernization Success, 4 Strategies to Boost Sales with Data Mining, and Modernizing Your Data Warehouse with Big Query. -------------------------------------------------------------------------------- AUTHOR FREDERICO CARAM Frederico Caram is a Data Architect at Avenue Code. He enjoys reading historical fantasy novels, ballroom dancing, and playing video games. -------------------------------------------------------------------------------- RELATED POSTS MODERNIZING YOUR DATA WAREHOUSE WITH BIGQUERY READ MORE LEAVE A COMMENT! First Name* Last Name Email* Website Comment* Lifecycle Stage AVENUE CODE SOCIAL NEWSLETTER Want to stay on top of all tips and news from Avenue Code? POPULAR SNIPPETS * Directives to the Point: Manipulating the DOM with Angular * How To Build Headless Browser Testing and Integrate With Jenkins * Enabling Rich Previews of Shared Links * Android Basics - Activities & Fragments * Callback Hell, Promises, and Async/Await ABOUT US * Who We Are * What We Do * Portfolio * Partners * News * Events * Blog * Contact OUR OFFICES San Francisco +1 415 766 4178 ac.inquiries@avenuecode.com Belo Horizonte +55 31 2516 1448 brazil.info@avenuecode.com São Paulo +55 11 3205 3232 brazil.info@avenuecode.com WE'RE HIRING! * Belo Horizonte * New York * San Francisco * São Paulo -------------------------------------------------------------------------------- ©2015 - 2017 Avenue Code Please enable JavaScript to view the comments powered by Disqus. © 2021 Avenue Code