community.cloudera.com
Open in
urlscan Pro
2600:9000:214f:2200:1a:6cae:29c0:93a1
Public Scan
URL:
https://community.cloudera.com/t5/Community-Articles/Sizing-CML-Workspaces-Must-Knows-for-properly-planning/ta-p/372168
Submission: On July 12 via api from US — Scanned from DE
Submission: On July 12 via api from US — Scanned from DE
Form analysis
1 forms found in the DOMName: form_d5333febe646b — POST https://community.cloudera.com/t5/tkb/v2_4/articlepage.searchformv32.form.form
<form enctype="multipart/form-data" class="lia-form lia-form-inline SearchForm" action="https://community.cloudera.com/t5/tkb/v2_4/articlepage.searchformv32.form.form" method="post" id="form_d5333febe646b" name="form_d5333febe646b">
<div class="t-invisible"><input value="blog-id/CommunityArticles/article-id/6699" name="t:ac" type="hidden"><input value="search/contributions/page" name="t:cp" type="hidden"><input
value="0kidEIX_A714h-BfMLPioNw_QjPqPRBEgyZODd1r0F93-kTpHgCc9YadS2dOx7gvGNKr9j7PPa-wbbkoqkwyWae7qIFv6DaSIXKqDKnMdIrSvCiA8SebwvFArcR0hHXc5nCKl1595GK3nhrJ71nKfTTMRp44pn_esXnt6Jwa45-8n0vj6Uwaui_WfgI21Oh9yC4ErqdazyJuCYCC5TGnjaH70Wht0epW1uF3nn5xzexllvc3kuEnKvFympT_BNth9tb93_H-S1rZiubTULFZm4HxaXS-X_gbDV9LibkemKUEfbprFBXyzAU2_cBgz85dPq09GrC9U1EjR8tXxyp_F_QyswPPkNKUoFH6FKLP_7A5ZDadsivKyheqiwddKVcUiLNZLF991f1LO9IT6oFf1xAoM9iidFuqA3QqBwAo3lRuTtsIf-EQ3Oq0uUFd_czR4rt1YL70ytmhByO0eMpjah-8gNzI9bt7ttgFoSiA7-L3uWpbO7_7ZvhBpR1zFX6zIX6npjyV_10Yuz6w2nCpnZrDvvfTw-i2bhdwuv1FzonWQksFq5jm7SFXunHVXPWO4H-tjC6rbW3UUJvA8Hmm9PjnkGllSEl_KDcddWTeFCNGsDvPaZ0RaXE2mav-ymR6Jc7JST9SkMxfSaOzlbGEQV3NomlxKi73W5bFvPJlbWMHmNFcc_uZFJgND-bpKLeYquCF_l5Vm2evv8jAMr08TcqaiM4Ilv6SM7Z2I6DBMEAp61ZzY3hz3DJ_Up4z5hxOmczjZXrgj4N1jXFNzCPeDxWG3SPYDKjBjkwvWAbCytPp1N7lAtHBjYQxGCA2t65h0kfGIaOKLVY7XY-Y2KFIdTyEozbNcDf0CJHC9GAvxhhNLnnxBtc9h2KGx9fdESxsRZ3-t5YIIDoo8BZjPGhRjxVY7Nn5ydWCvMe3O6aRifpyjm2sWT2yK8Xe1LeytIuSCi_QF4JyjMoStBxywki2DHPXBtZJ5KvXK5yNZ11sp8600sbIFMg334Dy1W7is-pVWh5basG1xcXRQSfcK1Aykq1l1rFyA0ApDvsAe65MJY9ytkgwNf8C-07AfteVA70CdQd5Mjj7Kx0iofzH5rveWA.."
name="lia-form-context" type="hidden"><input value="TkbArticlePage:blog-id/CommunityArticles/article-id/6699:searchformv32.form:" name="liaFormContentKey" type="hidden"><input
value="5DI9GWMef1Esyz275vuiiOExwpQ=:H4sIAAAAAAAAALVSTU7CQBR+krAixkj0BrptjcpCMSbERGKCSmxcm+kwlGrbqTOvFDYexRMYL8HCnXfwAG5dubDtFKxgYgu4mrzvm3w/M+/pHcphHQ4kI4L2dMo9FLYZoM09qbeJxQ4V0+XC7e/tamqyBPEChwgbh1JAjQtLIz6hPaYh8ZlEMaxplAvm2KZmEsm0hhmBhOKpzZzOlsEw8LevR5W3zZfPEqy0oJIYc+eCuAyh2rolfaI7xLN0I8rjWfWBj7CuzJvf5osmbxRN3hacMimNwHRtKSOr0XNnv/vx+FoCGPjhMRzljhNLYHrEt9kA5T08ACCsKvREoYuqxqLl8BLO84q4UcMITcG49y/QOGs1pYyESl5p6V6qwRW086rinVmoxMZsiZud/zBUTc6gmVc4kExkJafmcYG1GM9+wfIsCkf2OP54hal5EjnG54z8h0XhjfcF7wQUs5Kz0GTjU2rOjc/llTT4Au07pDOcBQAA"
name="t:formdata" type="hidden"></div>
<div class="lia-inline-ajax-feedback">
<div class="AjaxFeedback" id="feedback_d5333febe646b"></div>
</div>
<input value="ahM9FXk8_kFWDipIITiux6zzC06hl_1ItNTRM0BLa2U." name="lia-action-token" type="hidden">
<input value="form_d5333febe646b" id="form_UIDform_d5333febe646b" name="form_UID" type="hidden">
<input value="" id="form_instance_keyform_d5333febe646b" name="form_instance_key" type="hidden">
<span class="lia-search-granularity-wrapper">
<select title="Search Granularity" class="lia-search-form-granularity search-granularity" aria-label="Search Granularity" id="searchGranularity_d5333febe646b" name="searchGranularity">
<option title="All community" value="xgkfq28377|community">All community</option>
<option title="This category" value="Support|category">This category</option>
<option title="Community Articles" selected="selected" value="CommunityArticles|tkb-board">Community Articles</option>
<option title="Users" value="user|user">Users</option>
</select>
</span>
<span class="lia-search-input-wrapper">
<span class="lia-search-input-field">
<span class="lia-button-wrapper lia-button-wrapper-secondary lia-button-wrapper-searchForm-action"><input value="searchForm" name="submitContextX" type="hidden"><input class="lia-button lia-button-secondary lia-button-searchForm-action"
value="Search" id="submitContext_d5333febe646b" name="submitContext" type="submit"></span>
<input placeholder="Search the Community" aria-label="Search" title="Search" class="lia-form-type-text lia-autocomplete-input search-input lia-search-input-message" value="" id="messageSearchField_d5333febe646b_0" name="messageSearchField"
type="text" aria-autocomplete="both" autocomplete="off">
<div class="lia-autocomplete-container" style="display: none; position: absolute;">
<div class="lia-autocomplete-header">Enter a search word</div>
<div class="lia-autocomplete-content">
<ul></ul>
</div>
<div class="lia-autocomplete-footer">
<a class="lia-link-navigation lia-autocomplete-toggle-off lia-link-ticket-post-action lia-component-search-action-disable-auto-complete" data-lia-action-token="jMH5XTZRE6KdlHvZrZA1eqh0QwPE3zxDeVdkvYPFpAU." rel="nofollow" id="disableAutoComplete_d5333feeaf065" href="https://community.cloudera.com/t5/tkb/v2_4/articlepage.disableautocomplete:disableautocomplete?t:ac=blog-id/CommunityArticles/article-id/6699&t:cp=action/contributions/searchactions">Turn off suggestions</a>
</div>
</div>
<input placeholder="Search the Community" aria-label="Search" title="Search" class="lia-form-type-text lia-autocomplete-input search-input lia-search-input-tkb-article lia-js-hidden" value="" id="messageSearchField_d5333febe646b_1"
name="messageSearchField_0" type="text" aria-autocomplete="both" autocomplete="off">
<div class="lia-autocomplete-container" style="display: none; position: absolute;">
<div class="lia-autocomplete-header">Enter a search word</div>
<div class="lia-autocomplete-content">
<ul></ul>
</div>
<div class="lia-autocomplete-footer">
<a class="lia-link-navigation lia-autocomplete-toggle-off lia-link-ticket-post-action lia-component-search-action-disable-auto-complete" data-lia-action-token="g1JEynPgMlqmn5p8QdhSgq1JzVlRx8cBQ5io1oEGpJk." rel="nofollow" id="disableAutoComplete_d5333ff4cd651" href="https://community.cloudera.com/t5/tkb/v2_4/articlepage.disableautocomplete:disableautocomplete?t:ac=blog-id/CommunityArticles/article-id/6699&t:cp=action/contributions/searchactions">Turn off suggestions</a>
</div>
</div>
<input placeholder="Search the Community" ng-non-bindable="" title="Enter a user name or rank" class="lia-form-type-text UserSearchField lia-search-input-user search-input lia-js-hidden lia-autocomplete-input"
aria-label="Enter a user name or rank" value="" id="userSearchField_d5333febe646b" name="userSearchField" type="text" aria-autocomplete="both" autocomplete="off">
<div class="lia-autocomplete-container" style="display: none; position: absolute;">
<div class="lia-autocomplete-header">Enter a user name or rank</div>
<div class="lia-autocomplete-content">
<ul></ul>
</div>
<div class="lia-autocomplete-footer">
<a class="lia-link-navigation lia-autocomplete-toggle-off lia-link-ticket-post-action lia-component-search-action-disable-auto-complete" data-lia-action-token="2mGtrdeVREcpmAQa73VN_SiD6ECTbEkswpoxz6nZK6M." rel="nofollow" id="disableAutoComplete_d5333ff6b9163" href="https://community.cloudera.com/t5/tkb/v2_4/articlepage.disableautocomplete:disableautocomplete?t:ac=blog-id/CommunityArticles/article-id/6699&t:cp=action/contributions/searchactions">Turn off suggestions</a>
</div>
</div>
<input title="Enter a search word" class="lia-form-type-text NoteSearchField lia-search-input-note search-input lia-js-hidden lia-autocomplete-input" aria-label="Enter a search word" value="" id="noteSearchField_d5333febe646b_0"
name="noteSearchField" type="text" aria-autocomplete="both" autocomplete="off">
<div class="lia-autocomplete-container" style="display: none; position: absolute;">
<div class="lia-autocomplete-header">Enter a search word</div>
<div class="lia-autocomplete-content">
<ul></ul>
</div>
<div class="lia-autocomplete-footer">
<a class="lia-link-navigation lia-autocomplete-toggle-off lia-link-ticket-post-action lia-component-search-action-disable-auto-complete" data-lia-action-token="rhyTmBMaSBpE7DYOT3PAO_J5pVJ0c6F99gY0Vck944c." rel="nofollow" id="disableAutoComplete_d5333ff8bd6c4" href="https://community.cloudera.com/t5/tkb/v2_4/articlepage.disableautocomplete:disableautocomplete?t:ac=blog-id/CommunityArticles/article-id/6699&t:cp=action/contributions/searchactions">Turn off suggestions</a>
</div>
</div>
<input title="Enter a search word" class="lia-form-type-text ProductSearchField lia-search-input-product search-input lia-js-hidden lia-autocomplete-input" aria-label="Enter a search word" value="" id="productSearchField_d5333febe646b"
name="productSearchField" type="text" aria-autocomplete="both" autocomplete="off">
<div class="lia-autocomplete-container" style="display: none; position: absolute;">
<div class="lia-autocomplete-header">Enter a search word</div>
<div class="lia-autocomplete-content">
<ul></ul>
</div>
<div class="lia-autocomplete-footer">
<a class="lia-link-navigation lia-autocomplete-toggle-off lia-link-ticket-post-action lia-component-search-action-disable-auto-complete" data-lia-action-token="kQKUTEZiBoOEBG2P6zFKD3sQvmV4BCeTIYrEIStrpbo." rel="nofollow" id="disableAutoComplete_d5333ffad893d" href="https://community.cloudera.com/t5/tkb/v2_4/articlepage.disableautocomplete:disableautocomplete?t:ac=blog-id/CommunityArticles/article-id/6699&t:cp=action/contributions/searchactions">Turn off suggestions</a>
</div>
</div>
<input class="lia-as-search-action-id" name="as-search-action-id" type="hidden">
</span>
</span>
<span class="lia-cancel-search">cancel</span>
</form>
Text Content
* Community * Training * Partners * Support * Log In * Register CLOUDERA COMMUNITY * Get Started * Browse * Contribute * About the Community * What’s New Ask a Question COMMUNITY ARTICLES Find and share helpful community-sourced technical articles. All communityThis categoryCommunity ArticlesUsers Enter a search word Turn off suggestions Enter a search word Turn off suggestions Enter a user name or rank Turn off suggestions Enter a search word Turn off suggestions Enter a search word Turn off suggestions cancel Turn on suggestions Showing results for Show only | Search instead for Did you mean: Advanced Search * Cloudera Community * : * Support * : * Community Articles * : * Sizing CML Workspaces: Must-Knows for properly pla... Announcements Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub. Options * Subscribe to RSS Feed * * Mark as New * Mark as Read * * Bookmark * Subscribe * * Printer Friendly Page * Report Inappropriate Content Options * Subscribe to RSS Feed * * Mark as New * Mark as Read * * Bookmark * Subscribe * * Printer Friendly Page * Report Inappropriate Content SIZING CML WORKSPACES: MUST-KNOWS FOR PROPERLY PLANNING WORKSPACE RESOURCES Labels (2) Labels: * CLOUDERA DATA PLATFORM (CDP) * CLOUDERA MACHINE LEARNING (CML) pauldefusco Cloudera Employee Created on 06-05-2023 08:45 PM - edited on 06-08-2023 02:43 AM by VidyaSargur The Cloudera Data Platform (CDP) is a hybrid data platform designed to deliver faster and easier data management, analytics and AI at enterprise scale. Cloudera Machine Learning (CML) is one of CDP’s Cloud Native Data Services designed to enable secure, governed Data Science. With immediate access to enterprise data pipelines, scalable compute resources, and access to preferred tools, Data Scientists and Engineers use CML to streamline the process of getting analytic workloads into production and intelligently manage machine learning use cases and MLOps processes. While CML Data Scientists spend most of their time prototyping and productionizing models in CML Projects, the CML Admin needs to be familiar with the CML Workspace, its basic architecture and how it allocates Cloud resources. With this article we will share some foundational concepts in order to help CML Administrators better understand how to size Workspaces. What is a Workspace? CML workloads are executed within Workspaces and in turn within Projects and Teams. To the CML User, the Workspace is a high-level construct to create CML Projects, store CML Runtimes, and perform other administrative tasks such as creating Resource Profiles. However, under the covers, the Workspace is better defined as an auto-scaling service for Data Science leveraging Kubernetes. The Kubernetes cluster runs in Cloud Virtual Machines using AKS and EKS in the Public Cloud or OCP and Cloudera ECS in the Private Cloud. The CML Administrator or Data Scientist is not required to know or handle Kubernetes in any way. CML automatically deploys and manages the infrastructure resources for you in your CDP Environment of choice. When a Workspace is created for the first time a node is deployed to the underlying infrastructure. This is a fixed resource that is required to run at all times for a small cost. Subsequently, when a CML User runs a workload such as a Notebook, a Model API Endpoint, or a Batch Job, the CML Workspace provisions the necessary Pod(s) thus requesting a second node from the underlying infrastructure. As mentioned above, the auto-scaling process is fully automated and does not require any supervision. Auto-scaling events are fast and designed so that CML Users are not aware of them. Running workloads are not affected by the auto-scaling event e.g. running Sessions will continue undisturbed. If needed, any pending workloads such as new CML Sessions or previously scheduled CML Jobs will be queued automatically until new resources are deployed. At a high level, the pods carve out resources from the node(s) which is then released when the workload is complete. Thus, the CML Customer is only charged on the go as cloud resources are consumed and then discarded. The CML User explicitly picks the amount of CPU, Memory and optionally GPU resources when launching the workload. This amount is called a Resource Profile (e.g. 1 CPU / 2 GiB Mem) and it is predefined by the CML Admin at the Workspace level in order to provide an approval process and prevent Data Scientists from consuming too many resources without control. Sizing Considerations When deploying the Workspace for the first time, the user is prompted to select an instance type and an Autoscale Range (see image below). In the Public Cloud, these are AWS or Azure instances. The Autoscale Range is simply a min and max boundary of the instances that can be deployed by the Service. Typically, the more CPU, Memory, and GPU resources available per instance, the higher the hourly cost to run them but the more CML workloads can be deployed per instance without requiring the autoscaler to deploy an additional node. Because a typical workload such as a Data Exploration Notebook only requires a small Resource Profile, it is not uncommon to have multiple users working concurrently within the same node and thus at a fairly limited hourly cost. This means that instance types of relatively small size can be chosen when deploying a workspace. In the event of more horsepower being required, the Workspace will simply autoscale by adding as many instances as required and allowed by the Workspace Autoscale Range. However, if you plan on running workloads that cannot horizontally scale in a distributed fashion with frameworks such as Spark, TensorFlow, etc., then it may make sense to choose a more powerful instance type. This could be the case in Time Series Machine Learning where algorithms cannot always be distributed. Finally, it’s important to note that CML Instance Types and autoscale ranges can be changed even after a Workspace has been deployed. Cost Management Considerations Instance hourly rates are publicly available on the Cloudera Pricing Site. In addition, your Cloudera Account Team can provide additional recommendations to plan and size your Workspace according to your use cases. CML is designed to allow the Administrator to closely monitor and limit usage in order to prevent runaway cloud charges. As mentioned above, Resource Profiles are whitelisted by the CML Admin in order to prevent CML Users from requesting resources without supervision. To be specific, the CML User will only be able to launch Jobs, Sessions, Applications, etc. with the CPU/Mem/GPU profiles designated in the Runtime menu as shown below. Furthermore, CML Users are also users at the CDP Environment level. In other words, each Workspace can grant or deny access to a particular CDP User. Finally, within each Workspace, the CML Admin can create Quotas to directly limit a User’s maximum amount of CPU, Memory, and GPU use across all workloads at any given time. Quota consumption is only a subset of the Workspace Autoscale ranges which can be viewed as a second option for managing costs at the global level. Using Multiple Workspaces It is common practice to create multiple CML Workspaces as each additional Workspace can provide workload isolation and a quick second option in case of failure. CML Customers typically deploy them based on scope such as Use Case, Business Organization, or function e.g. DEV vs QA vs PROD. The additional workspace(s) can be created in the same CDP Environment or in a separate CDP Environment. In the former case, the Workspaces will share the same SDX Data Lake and thus their users will be able to access and transform the same datasets while being governed and secured by the same Atlas and Ranger services. In the latter case, creating Workspaces in different CDP Environments will guarantee that they won’t be adversely affected in case of a failure at the CDP Environment level. For example, the below image shows two workspaces deployed in the same CDP Environment while a third one is in a separate one. Notice the first Workspace is undergoing a change in instance types and autoscale range. Additionally, CML supports MLFlow Registry which allows you to deploy models from one Workspace to another. As a result, multiple workspaces can support DevOps pipelines across multiple CDP Environments and even allow you to deploy models from Public to Private Cloud and vice versa (Hybrid Machine Learning). Although each Workspace comes with a small fixed hourly charge, another advantage is that you will be able to select different instance types and autoscale ranges for each deployment which in turn could allow you to save money by enforcing stricter limitations on particular business functions or user groups. A Sizing Exercise Example With all these considerations in mind, we recommend you go through a similar exercise as below when planning your Workspace deployment. Step 1: Estimate the number of CML Users and optionally whether these will be working within the same or different Teams, Use Cases, and CDP Data Lakes. Step 2: Estimate average and peak CPU, Memory, and optionally GPU consumption per User. If planning on more than one Team, determine if the average and peak dramatically varies between them. Step 3: Decide if you need more than one workspace. Try to group users into Teams and Use Cases as much as reasonably possible based on similarities in Data Lake Access, average and peak consumption. Other factors may include whether users need GPUs, special Ranger ACLs, and types of workloads (e.g. primarily hosting API Model Endpoints vs Exploratory Data Science in Notebooks vs Spark ETL in CML Jobs). Step 4: Sum up all CPU, Memory, and GPU required per workspace at peak and average, then add 20%. Step 5: Look up CPU, Memory, and GPU resources per AWS or Azure Instance types and estimate how many instances would be required to fit the sum from Step 4. Pick an Instance Type that will fit most of your average workloads with a reasonable instance count (i.e. within the 3-6 range) and your peak workloads with no more than 10 instances. If this is not possible, divide the workload further into two separate workspaces where one has the same or smaller instance types and the other has larger instance types. Conclusions In this article, we highlighted some of the most fundamental considerations for sizing a CML Workspace. In summary: * CML Workspaces are autoscaling Kubernetes clusters providing Workload Isolation. CML automatically deploys and manages the infrastructure resources for you and requires no knowledge or interaction with the Kubernetes resources under the hood. * When planning for the deployment of Workspaces it is important to keep in mind that multiple Workspaces can and should be deployed based on Use Case, Team, Function, and Resource Consumption estimates. * Generally, sizing a Workspace consists of an exercise of estimating average and peak consumption in terms of CML Resource Profiles and mapping the estimates to AWS or Azure Instance Types. Additional considerations such as workload type, Data Lake access and SLAs should be prioritized as decision factors. 129 Views 1 Kudo Take a Tour of the Community Community Browser * Explore the Community * Groups * Cloudera Data Analytics (CDA) * Cloudera Data Analytics (CDA) Forum * Cloudera Data Analytics (CDA) Blogs * Cloudera Data Analytics (CDA) Articles * Announcements * Community Announcements * Product Announcements * Support Announcements * What's New @ Cloudera * Support * Support Questions * Code Repositories * Community Articles * Using the Community * Intros and Suggestions * Community Tips Have a Cloudera Account? Sign In Don't have an account? Register Your experience may be limited. Sign in to explore more. Announcements What's New @ Cloudera Cloudera Operational Database (COD) supports enabling custom recipes using CDP CLI Beta What's New @ Cloudera Cloudera Streaming Analytics (CSA) 1.10 introduces new built-in widget for data visualization and has been rebased onto Apache Flink 1.16 Product Announcements CDP Public Cloud: June 2023 Release Summary What's New @ Cloudera Cloudera Data Engineering (CDE) 1.19 in Public Cloud introduces interactive Spark development sessions What's New @ Cloudera Cloudera DataFlow 2.5 supports latest NiFi version, new flow metric based auto-scaling, new Designer capabilities and in-place upgrades are now GA View More Announcements Version history Last update: 06-08-2023 02:43 AM Updated by: VidyaSargur Contributors * pauldefusco * VidyaSargur Top Kudoed Authors User Count TimothySpann 763 sshimpi 379 KuldeepK 316 cstanca 309 abajwa 270 View Full Leaderboard Recommendations * ML WORKSPACE PAGE DOES NOT OPEN * CLOUDERA MACHINE LEARNING ADDS ABILITY FOR ADMINS ... * CLOUDERA MACHINE LEARNING WORKSPACE PROVISIONING P... * HOW TO PROVISION WORKSPACES IN CLOUDERA MACHINE LE... * CLOUDERA MACHINE LEARNING (CML) - QUESTIONS & ANSW... * CLOUDBREAK WORKSPACES: MOVE CLUSTERS * SPARK IN CML: RECOMMENDATIONS FOR USING SPARK IN C... * SPARK AUTO-SCALING WITH KUBERNETES IN CLOUDERA DAT... * HOW TO SETUP CUSTOM JUPYTER KERNEL IN CML * PROCESSING FILES WITHIN THE CML EFS (BETTER I/O PE... Top * My Cloudera Home * Community * Documentation * Knowledge Base * Partners * Support * Products * Cloudera Data Platform * Data Hub * Data Warehouse * Machine Learning * CDP Private Cloud * Data Flow * Data Science Workbench * Enterprise Data Hub * Solutions * Customer Analytics * IoT / Connected Products * Security, Risk, & Compliance * Industry Solutions * Solutions Gallery * Downloads * Cloudera Data Flow * CDP Private Cloud * Cloudera Director * More * About * News & Blogs * Investor Relations * Events * Customers * Careers * Security * Community Guidelines * Community FAQs CONTACT US United States: +1 888 789 1488 Outside the US: +1 650 362 0488 © 2023 Cloudera, Inc. All rights reserved. Terms & Conditions Privacy Policy and Data Policy Unsubscribe / Do Not Sell My Personal Information Supported Browsers Policy Apache Hadoop and associated open source project names are trademarks of the Apache Software Foundation . For a complete list of trademarks, click here . Cloudera uses cookies to provide and improve our site's services. By using this site, you consent to use of cookies as outlined in Cloudera's Privacy and Data Policies. Accept Cookies Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.