status.topdesk.com
Open in
urlscan Pro
65.9.95.118
Public Scan
Submitted URL: https://stspg.io/9bkr4lkgxv33
Effective URL: https://status.topdesk.com/incidents/4pbtb55hrwv0
Submission: On July 12 via api from NL — Scanned from NL
Effective URL: https://status.topdesk.com/incidents/4pbtb55hrwv0
Submission: On July 12 via api from NL — Scanned from NL
Form analysis 1 forms found in the DOM
POST /subscriptions/incident.json
<form class="modal-content" id="subscribe-form-4pbtb55hrwv0" action="/subscriptions/incident.json" accept-charset="UTF-8" data-remote="true" method="post">
<input type="hidden" name="incident_code" id="incident_code" value="4pbtb55hrwv0" autocomplete="off">
<div class="modal-header">
<a href="#" data-dismiss="modal" class="close" data-js-hook="incident-modal-close" aria-label="close" role="button">×</a>
<h4 id="incident-subscription-dialog-header"> Subscribe to Incident </h4>
</div>
<div class="modal-body">
<p style="margin-bottom:25px" id="incident-subscription-dialog-description"> Subscribe to updates for <strong>RESOLVED: Disruption in hosting location NL3. Action Sequences are not functioning.</strong> via email and/or text message. You'll
receive email notifications when incidents are updated, and text message notifications whenever TOPdesk SaaS Status page <strong>creates</strong> or <strong>resolves</strong> an incident. </p>
<div class="control-group">
<label for="email-4pbtb55hrwv0">VIA EMAIL:</label>
<div class="controls">
<input type="text" name="email" id="email-4pbtb55hrwv0" data-js-hook="email" class="full-width">
</div>
</div>
<div class="control-group">
<label class="phone-country-code" for="phone-country-4pbtb55hrwv0">VIA SMS:</label>
<div class="controls phone-number">
<div class="row">
<div id="phone-number-code-4pbtb55hrwv0" class="phone-country-wrapper" data-otp-enabled="false">
<select name="phone_country" id="phone-country-4pbtb55hrwv0" data-js-hook="phone-country" class="span6 phone-country-dropdown">
<option value="af" data-otp-enabled="false">Afghanistan (+93)</option>
<option value="al" data-otp-enabled="false">Albania (+355)</option>
<option value="dz" data-otp-enabled="false">Algeria (+213)</option>
<option value="as" data-otp-enabled="false">American Samoa (+1)</option>
<option value="ad" data-otp-enabled="false">Andorra (+376)</option>
<option value="ao" data-otp-enabled="false">Angola (+244)</option>
<option value="ai" data-otp-enabled="false">Anguilla (+1)</option>
<option value="ag" data-otp-enabled="false">Antigua and Barbuda (+1)</option>
<option value="ar" data-otp-enabled="false">Argentina (+54)</option>
<option value="am" data-otp-enabled="false">Armenia (+374)</option>
<option value="aw" data-otp-enabled="false">Aruba (+297)</option>
<option value="au" data-otp-enabled="false">Australia/Cocos/Christmas Island (+61)</option>
<option value="at" data-otp-enabled="false">Austria (+43)</option>
<option value="az" data-otp-enabled="false">Azerbaijan (+994)</option>
<option value="bs" data-otp-enabled="false">Bahamas (+1)</option>
<option value="bh" data-otp-enabled="false">Bahrain (+973)</option>
<option value="bd" data-otp-enabled="false">Bangladesh (+880)</option>
<option value="bb" data-otp-enabled="false">Barbados (+1)</option>
<option value="by" data-otp-enabled="false">Belarus (+375)</option>
<option value="be" data-otp-enabled="false">Belgium (+32)</option>
<option value="bz" data-otp-enabled="false">Belize (+501)</option>
<option value="bj" data-otp-enabled="false">Benin (+229)</option>
<option value="bm" data-otp-enabled="false">Bermuda (+1)</option>
<option value="bo" data-otp-enabled="false">Bolivia (+591)</option>
<option value="ba" data-otp-enabled="false">Bosnia and Herzegovina (+387)</option>
<option value="bw" data-otp-enabled="false">Botswana (+267)</option>
<option value="br" data-otp-enabled="false">Brazil (+55)</option>
<option value="bn" data-otp-enabled="false">Brunei (+673)</option>
<option value="bg" data-otp-enabled="false">Bulgaria (+359)</option>
<option value="bf" data-otp-enabled="false">Burkina Faso (+226)</option>
<option value="bi" data-otp-enabled="false">Burundi (+257)</option>
<option value="kh" data-otp-enabled="false">Cambodia (+855)</option>
<option value="cm" data-otp-enabled="false">Cameroon (+237)</option>
<option value="ca" data-otp-enabled="false">Canada (+1)</option>
<option value="cv" data-otp-enabled="false">Cape Verde (+238)</option>
<option value="ky" data-otp-enabled="false">Cayman Islands (+1)</option>
<option value="cf" data-otp-enabled="false">Central Africa (+236)</option>
<option value="td" data-otp-enabled="false">Chad (+235)</option>
<option value="cl" data-otp-enabled="false">Chile (+56)</option>
<option value="cn" data-otp-enabled="false">China (+86)</option>
<option value="co" data-otp-enabled="false">Colombia (+57)</option>
<option value="km" data-otp-enabled="false">Comoros (+269)</option>
<option value="cg" data-otp-enabled="false">Congo (+242)</option>
<option value="cd" data-otp-enabled="false">Congo, Dem Rep (+243)</option>
<option value="cr" data-otp-enabled="false">Costa Rica (+506)</option>
<option value="hr" data-otp-enabled="false">Croatia (+385)</option>
<option value="cy" data-otp-enabled="false">Cyprus (+357)</option>
<option value="cz" data-otp-enabled="false">Czech Republic (+420)</option>
<option value="dk" data-otp-enabled="false">Denmark (+45)</option>
<option value="dj" data-otp-enabled="false">Djibouti (+253)</option>
<option value="dm" data-otp-enabled="false">Dominica (+1)</option>
<option value="do" data-otp-enabled="false">Dominican Republic (+1)</option>
<option value="eg" data-otp-enabled="false">Egypt (+20)</option>
<option value="sv" data-otp-enabled="false">El Salvador (+503)</option>
<option value="gq" data-otp-enabled="false">Equatorial Guinea (+240)</option>
<option value="ee" data-otp-enabled="false">Estonia (+372)</option>
<option value="et" data-otp-enabled="false">Ethiopia (+251)</option>
<option value="fo" data-otp-enabled="false">Faroe Islands (+298)</option>
<option value="fj" data-otp-enabled="false">Fiji (+679)</option>
<option value="fi" data-otp-enabled="false">Finland/Aland Islands (+358)</option>
<option value="fr" data-otp-enabled="false">France (+33)</option>
<option value="gf" data-otp-enabled="false">French Guiana (+594)</option>
<option value="pf" data-otp-enabled="false">French Polynesia (+689)</option>
<option value="ga" data-otp-enabled="false">Gabon (+241)</option>
<option value="gm" data-otp-enabled="false">Gambia (+220)</option>
<option value="ge" data-otp-enabled="false">Georgia (+995)</option>
<option value="de" data-otp-enabled="false">Germany (+49)</option>
<option value="gh" data-otp-enabled="false">Ghana (+233)</option>
<option value="gi" data-otp-enabled="false">Gibraltar (+350)</option>
<option value="gr" data-otp-enabled="false">Greece (+30)</option>
<option value="gl" data-otp-enabled="false">Greenland (+299)</option>
<option value="gd" data-otp-enabled="false">Grenada (+1)</option>
<option value="gp" data-otp-enabled="false">Guadeloupe (+590)</option>
<option value="gu" data-otp-enabled="false">Guam (+1)</option>
<option value="gt" data-otp-enabled="false">Guatemala (+502)</option>
<option value="gn" data-otp-enabled="false">Guinea (+224)</option>
<option value="gy" data-otp-enabled="false">Guyana (+592)</option>
<option value="ht" data-otp-enabled="false">Haiti (+509)</option>
<option value="hn" data-otp-enabled="false">Honduras (+504)</option>
<option value="hk" data-otp-enabled="false">Hong Kong (+852)</option>
<option value="hu" data-otp-enabled="false">Hungary (+36)</option>
<option value="is" data-otp-enabled="false">Iceland (+354)</option>
<option value="in" data-otp-enabled="false">India (+91)</option>
<option value="id" data-otp-enabled="false">Indonesia (+62)</option>
<option value="iq" data-otp-enabled="false">Iraq (+964)</option>
<option value="ie" data-otp-enabled="false">Ireland (+353)</option>
<option value="il" data-otp-enabled="false">Israel (+972)</option>
<option value="it" data-otp-enabled="false">Italy (+39)</option>
<option value="jm" data-otp-enabled="false">Jamaica (+1)</option>
<option value="jp" data-otp-enabled="false">Japan (+81)</option>
<option value="jo" data-otp-enabled="false">Jordan (+962)</option>
<option value="ke" data-otp-enabled="false">Kenya (+254)</option>
<option value="kr" data-otp-enabled="false">Korea, Republic of (+82)</option>
<option value="xk" data-otp-enabled="false">Kosovo (+383)</option>
<option value="kw" data-otp-enabled="false">Kuwait (+965)</option>
<option value="kg" data-otp-enabled="false">Kyrgyzstan (+996)</option>
<option value="la" data-otp-enabled="false">Laos (+856)</option>
<option value="lv" data-otp-enabled="false">Latvia (+371)</option>
<option value="lb" data-otp-enabled="false">Lebanon (+961)</option>
<option value="ls" data-otp-enabled="false">Lesotho (+266)</option>
<option value="lr" data-otp-enabled="false">Liberia (+231)</option>
<option value="ly" data-otp-enabled="false">Libya (+218)</option>
<option value="li" data-otp-enabled="false">Liechtenstein (+423)</option>
<option value="lt" data-otp-enabled="false">Lithuania (+370)</option>
<option value="lu" data-otp-enabled="false">Luxembourg (+352)</option>
<option value="mo" data-otp-enabled="false">Macao (+853)</option>
<option value="mk" data-otp-enabled="false">Macedonia (+389)</option>
<option value="mg" data-otp-enabled="false">Madagascar (+261)</option>
<option value="mw" data-otp-enabled="false">Malawi (+265)</option>
<option value="my" data-otp-enabled="false">Malaysia (+60)</option>
<option value="mv" data-otp-enabled="false">Maldives (+960)</option>
<option value="ml" data-otp-enabled="false">Mali (+223)</option>
<option value="mt" data-otp-enabled="false">Malta (+356)</option>
<option value="mq" data-otp-enabled="false">Martinique (+596)</option>
<option value="mr" data-otp-enabled="false">Mauritania (+222)</option>
<option value="mu" data-otp-enabled="false">Mauritius (+230)</option>
<option value="mx" data-otp-enabled="false">Mexico (+52)</option>
<option value="mc" data-otp-enabled="false">Monaco (+377)</option>
<option value="mn" data-otp-enabled="false">Mongolia (+976)</option>
<option value="me" data-otp-enabled="false">Montenegro (+382)</option>
<option value="ms" data-otp-enabled="false">Montserrat (+1)</option>
<option value="ma" data-otp-enabled="false">Morocco/Western Sahara (+212)</option>
<option value="mz" data-otp-enabled="false">Mozambique (+258)</option>
<option value="na" data-otp-enabled="false">Namibia (+264)</option>
<option value="np" data-otp-enabled="false">Nepal (+977)</option>
<option value="nl" data-otp-enabled="false" selected="">Netherlands (+31)</option>
<option value="nz" data-otp-enabled="false">New Zealand (+64)</option>
<option value="ni" data-otp-enabled="false">Nicaragua (+505)</option>
<option value="ne" data-otp-enabled="false">Niger (+227)</option>
<option value="ng" data-otp-enabled="false">Nigeria (+234)</option>
<option value="no" data-otp-enabled="false">Norway (+47)</option>
<option value="om" data-otp-enabled="false">Oman (+968)</option>
<option value="pk" data-otp-enabled="false">Pakistan (+92)</option>
<option value="ps" data-otp-enabled="false">Palestinian Territory (+970)</option>
<option value="pa" data-otp-enabled="false">Panama (+507)</option>
<option value="py" data-otp-enabled="false">Paraguay (+595)</option>
<option value="pe" data-otp-enabled="false">Peru (+51)</option>
<option value="ph" data-otp-enabled="false">Philippines (+63)</option>
<option value="pl" data-otp-enabled="false">Poland (+48)</option>
<option value="pt" data-otp-enabled="false">Portugal (+351)</option>
<option value="pr" data-otp-enabled="false">Puerto Rico (+1)</option>
<option value="qa" data-otp-enabled="false">Qatar (+974)</option>
<option value="re" data-otp-enabled="false">Reunion/Mayotte (+262)</option>
<option value="ro" data-otp-enabled="false">Romania (+40)</option>
<option value="ru" data-otp-enabled="false">Russia/Kazakhstan (+7)</option>
<option value="rw" data-otp-enabled="false">Rwanda (+250)</option>
<option value="ws" data-otp-enabled="false">Samoa (+685)</option>
<option value="sm" data-otp-enabled="false">San Marino (+378)</option>
<option value="sa" data-otp-enabled="false">Saudi Arabia (+966)</option>
<option value="sn" data-otp-enabled="false">Senegal (+221)</option>
<option value="rs" data-otp-enabled="false">Serbia (+381)</option>
<option value="sc" data-otp-enabled="false">Seychelles (+248)</option>
<option value="sl" data-otp-enabled="false">Sierra Leone (+232)</option>
<option value="sg" data-otp-enabled="false">Singapore (+65)</option>
<option value="sk" data-otp-enabled="false">Slovakia (+421)</option>
<option value="si" data-otp-enabled="false">Slovenia (+386)</option>
<option value="za" data-otp-enabled="false">South Africa (+27)</option>
<option value="es" data-otp-enabled="false">Spain (+34)</option>
<option value="lk" data-otp-enabled="false">Sri Lanka (+94)</option>
<option value="kn" data-otp-enabled="false">St Kitts and Nevis (+1)</option>
<option value="lc" data-otp-enabled="false">St Lucia (+1)</option>
<option value="vc" data-otp-enabled="false">St Vincent Grenadines (+1)</option>
<option value="sd" data-otp-enabled="false">Sudan (+249)</option>
<option value="sr" data-otp-enabled="false">Suriname (+597)</option>
<option value="sz" data-otp-enabled="false">Swaziland (+268)</option>
<option value="se" data-otp-enabled="false">Sweden (+46)</option>
<option value="ch" data-otp-enabled="false">Switzerland (+41)</option>
<option value="tw" data-otp-enabled="false">Taiwan (+886)</option>
<option value="tj" data-otp-enabled="false">Tajikistan (+992)</option>
<option value="tz" data-otp-enabled="false">Tanzania (+255)</option>
<option value="th" data-otp-enabled="false">Thailand (+66)</option>
<option value="tg" data-otp-enabled="false">Togo (+228)</option>
<option value="to" data-otp-enabled="false">Tonga (+676)</option>
<option value="tt" data-otp-enabled="false">Trinidad and Tobago (+1)</option>
<option value="tn" data-otp-enabled="false">Tunisia (+216)</option>
<option value="tr" data-otp-enabled="false">Turkey (+90)</option>
<option value="tc" data-otp-enabled="false">Turks and Caicos Islands (+1)</option>
<option value="ug" data-otp-enabled="false">Uganda (+256)</option>
<option value="ua" data-otp-enabled="false">Ukraine (+380)</option>
<option value="ae" data-otp-enabled="false">United Arab Emirates (+971)</option>
<option value="gb" data-otp-enabled="false">United Kingdom (+44)</option>
<option value="us" data-otp-enabled="false">United States (+1)</option>
<option value="uy" data-otp-enabled="false">Uruguay (+598)</option>
<option value="uz" data-otp-enabled="false">Uzbekistan (+998)</option>
<option value="ve" data-otp-enabled="false">Venezuela (+58)</option>
<option value="vn" data-otp-enabled="false">Vietnam (+84)</option>
<option value="vg" data-otp-enabled="false">Virgin Islands, British (+1)</option>
<option value="vi" data-otp-enabled="false">Virgin Islands, U.S. (+1)</option>
<option value="ye" data-otp-enabled="false">Yemen (+967)</option>
<option value="zm" data-otp-enabled="false">Zambia (+260)</option>
<option value="zw" data-otp-enabled="false">Zimbabwe (+263)</option>
</select>
</div>
</div>
</div>
<label class="sub-label" for="phone-number-4pbtb55hrwv0">Enter mobile number</label>
<div class="controls phone-number" data-js-hook="modal-open-incident-subscribe-phone-number">
<div class="flex-container">
<input type="text" name="phone_number" id="phone-number-4pbtb55hrwv0" data-js-hook="phone-number">
<input type="hidden" name="incident-code-4pbtb55hrwv0" id="incident-code-4pbtb55hrwv0" value="4pbtb55hrwv0" autocomplete="off">
<a class="btn-change-number-incident" id="btn-change-number-incident-4pbtb55hrwv0" style="display: none;" data-incident-code="4pbtb55hrwv0" href="#">
Edit number
</a> <a class="btn-subcriber-send-otp" id="btn-subcriber-send-otp-4pbtb55hrwv0" style="display: none;" data-incident-code="4pbtb55hrwv0" href="#">
Send OTP
</a>
</div>
</div>
<div class="sms-atl-error" id="sms-atl-error-4pbtb55hrwv0"></div>
<div class="opt-container-section" id="otp-container-incident-4pbtb55hrwv0" style="display:none">
<label class="sub-label" for="otp">Enter the OTP sent</label>
<div class="flex-container">
<input name="otp" id="otp-field-4pbtb55hrwv0" type="text" class="prepend" disabled="">
<a class="resend-otp" id="resend-otp-4pbtb55hrwv0" style="display: none;" data-incident-code="4pbtb55hrwv0" href="#">
Resend OTP
</a> <span class="timer-incident" id="timer-incident-4pbtb55hrwv0" style="display:none"> in <span id="countdown-incident-4pbtb55hrwv0">30</span> seconds</span>
</div>
<div class="info-row flex-container">
<div class="info-icon"><svg width="2vh" height="2vh" viewBox="0 0 21 21" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" id="status-embed-svg-info-icon">
<defs>
<path
d="M2,12 C2,6.47666667 6.47666667,2 12,2 C17.5233333,2 22,6.47666667 22,12 C22,17.5233333 17.5233333,22 12,22 C6.47666667,22 2,17.5233333 2,12 Z M4,12 C4,16.4187638 7.58123617,20 12,20 C16.4187638,20 20,16.4187638 20,12 C20,7.58123617 16.4187638,4 12,4 C7.58123617,4 4,7.58123617 4,12 Z M11,11.0029293 C11,10.4490268 11.4438648,10 12,10 C12.5522847,10 13,10.4378814 13,11.0029293 L13,15.9970707 C13,16.5509732 12.5561352,17 12,17 C11.4477153,17 11,16.5621186 11,15.9970707 L11,11.0029293 Z M12,9 C11.4477153,9 11,8.55228475 11,8 C11,7.44771525 11.4477153,7 12,7 C12.5522847,7 13,7.44771525 13,8 C13,8.55228475 12.5522847,9 12,9 Z"
id="path-info"></path>
</defs>
<g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<g transform="translate(-702.000000, -945.000000)">
<g id="global/info" transform="translate(700.500000, 943.500000)">
<mask id="mask-info" fill="white">
<use xlink:href="#path-info"></use>
</mask>
<use id="Combined-Shape" fill="#42526E" fill-rule="nonzero" xlink:href="#path-info"></use>
<g id="Neutral-/-N000" mask="url(#mask-info)" fill-rule="evenodd">
<polygon points="0 24 24 24 24 0 0 0"></polygon>
</g>
</g>
</g>
</g>
</svg>
</div>
<div>To receive SMS updates, please verify your number. To proceed with just email click ‘Subscribe’ </div>
</div>
</div>
</div>
</div>
<div class="modal-footer incident-subscribe">
<!-- tests fail if static width isn't present ¯\_(ツ)_/¯ -->
<button name="button" type="submit" class="flat-button cpt-button incident-subscribe-btn-captcha" id="subscribe-btn-4pbtb55hrwv0" onclick="submitCaptchaIncidentSubscribe(event)">Subscribe to Incident</button>
<input type="hidden" name="captcha_error" id="captcha_error" value="false" autocomplete="off">
<input type="hidden" name="g-recaptcha-response" id="g-recaptcha-response" value="false" autocomplete="off">
<div class="terms_and_privacy_information left small">Message and data rates may apply. By subscribing you agree to our <a target="_blank" rel="noopener" href="https://www.topdesk.com/us/privacy-policy/">Privacy Policy</a>, the Atlassian
<a target="_blank" rel="noopener" href="https://www.atlassian.com/legal/product-specific-terms#statuspage-specific-terms">Terms of Service</a>, and the Atlassian
<a target="_blank" rel="noopener" href="https://www.atlassian.com/legal/privacy-policy">Privacy Policy</a>. This site is protected by reCAPTCHA and the Google
<a target="_blank" rel="noopener" href="https://policies.google.com/privacy">Privacy Policy</a> and <a target="_blank" rel="noopener" data-js-hook="captcha-terms-of-service-link" href="https://policies.google.com/terms">Terms of Service</a>
apply.</div>
</div>
</form>Text Content
RESOLVED: Disruption in hosting location NL3. Action Sequences are not functioning. Incident Report for TOPdesk SaaS Status page Postmortem ROOT CAUSE ANALYSIS (RCA) Disruption in NL3 Hosting Location Impacting Action Sequences DATE OF THE INCIDENT First symptoms: 21.03.2024 - Major TDR24 03 7280 Second round of symptoms: 26.03.2024 - Major TDR24 03 9021 WHAT HAPPENED On our NL3 hosting location, we experienced issues with respect to the performance of the messaging system. These issues resulted in disruptions on the RabbitMQ, which mainly caused issues for the execution in the action sequences. WHY IT HAPPENED The message-queues of TOPdesk instances in NL3 hosting location got into a state where they were multiplying their connections to the messaging server (from here on RabbitMQ). How the TOPdesk instances got into that state was unclear, and we were unable to get them in that state by force. However, the instances kept increasing the amount of connections. This initially caused RabbitMQ to run out of page files/file descriptors, which in turn caused the system to run out of memory and nodes to start to fall over. When this was increased, TOPdesk which got in an erroneous state, started to indefinitely try and recreate connections, which again in turn caused the RabbitMQ to run out of memory. DETAILED TIMELINE * 21.03.2024 | RabbitMQ cluster on NL3 crashed for the first time. To get it out of the crash loop, memory was increased. * 26.03.2024 | The cluster crashed for the second time. To stabilize the system, memory was increased even further. * 27.03.2024 | The amount of Erlang processes and memory consumption were going up, approaching the max limit. To prevent that from happening, system limits were increased one more time. As the memory consumption and the number of channels were still growing, the cluster broke again later in the night. Memory was increased again. Restarting the instances resulted in a drop in the number of channels and connections. * 28.03.2024 | Even though the restarts from the previous day improved the situation, the number of channels, connections and Erlang processes were still in steady increase. Due to a hick-up on the load balancer, the traffic dropped temporarily. Monitoring and investigation activities continued. * 29.03.2024 | Due to the growing number of channels and node sync issues, emergency maintenance was planned for the Easter weekend to minimize the potential customer impact. * 30.03.2024 | Starting from the early morning, the number of connections was rapidly increasing. As part of the emergency maintenance, two nodes that seemed to be really busy, were taken out of the RabbitMQ cluster and recreated. Following the maintenance, customer environments were restarted. * 01.04.2024 | To stabilize the number of growing channels and connections, a max amount of channels limit per connection was implemented. * 02.04.2024 | Not all the message queues were being replicated in all the nodes. Measures were taken to ensure queue replication in all nodes. * 03.04.2024 | Due to all the measures taken and limits implemented throughout the process, the system was stable again. THE MEASURES WE TOOK To mitigate the symptoms and implement a permanent solution, we gradually took some actions, both during and after the major, while also closely monitoring the system. Below please see an overview of the actions taken: * Increased the capacity of our messaging system to handle the amount of processes * Restarted the environments in NL3 hosting location to eliminate the faulty connections within the messaging system * Implemented a max amount of channel per connection limit * Implemented monitoring on disk usage, queue replication and system metrics * As an addition to the already existing alerts, implemented new alerts so that early warning mechanism is more comprehensive * Improved and centralized the logging so that it is easier to track any potential odd behavior and troubleshoot issues * Fixed the remaining queue replication issues even though there was no visible impact on the service’s ability to consume messages * Updated the RabbitMQ version so that the servers run on the most recent version The following was also adjusted in TOPdesk code by development: * Improved the messaging system so it properly names their connections. This will help to figure out which instances are causing issues. * Fixed issues with reconnecting to the messaging system in TOS. FINAL REMARKS Through the above described infrastructure, development and monitoring/logging improvements, not only the messaging system was brought back to its stable state where it could successfully process messages, but the health of the system has also been improved so that we can avert similar issues from occurring in the future. Posted 22 minutes ago. Jul 12, 2024 - 15:25 CEST -------------------------------------------------------------------------------- Resolved We're pleased to inform you that the recent major incident impacting our services is now fully resolved. The back-end has stabilized and since two days, the load on the machines has been normalized to the values before any issues arose. All systems are now working optimally and we are confident in the stability of our current operations. However, we understand the importance of identifying the root cause to prevent potential future disruptions. In light of this, we have assembled a dedicated team of skilled engineers to conduct a thorough investigation into the root cause and further increasing the resilience of the messaging system and improving our monitoring. Because of the complexity of this problem the Root Cause Analysis (RCA) will take some time. Until we have the full RCA we will give you biweekly updates in your incident. Your understanding throughout this situation has been greatly appreciated. We apologize for any inconvenience caused. Posted 3 months ago. Apr 04, 2024 - 16:35 CEST Update We've been keeping an eye on things throughout the day, and all systems are functioning as expected. All features are also operating normally. As we continue our investigation into the root cause, we plan to enhance our monitoring measures and have significantly scaled up our systems to prevent any impact on our customers. Even with this new enhanced setup, we've noticed that since the restarts last night, all connections have maintained their stability and the performance load of the messaging system is back to normal, as it was before the first issues emerged on the 21st of March. We will continue to keep you informed with any further developments. We sincerely apologize for any inconvenience caused and we appreciate your patience during this time. Posted 3 months ago. Apr 03, 2024 - 17:26 CEST Update After the restarts we conducted last night, we're pleased to inform you that our messaging system is in a healthy state and all functionalities have returned to normal. We will continue to maintain a close watch and closely monitor throughout the day. Please note that the investigation into the root cause is still ongoing. We aim to provide updates when we have significant new information. Your patience and understanding during this time are greatly appreciated. Posted 3 months ago. Apr 03, 2024 - 10:40 CEST Update We would like to provide an extended update on recent events within our NL3 data center, particularly regarding our messaging system. This morning, we observed that action sequences were not being processed. Our engineers promptly reset and resolved the issues, and the queue was fully caught up by 12:30 CET. On Saturday, emergency maintenance was conducted on faulty parts of the messaging system. On Monday, a change was made to the back-end to increase the resilience of the messaging system. Today, a configuration issue, unrelated to the action sequence issues, was found and promptly rectified. However, to fully realize the effects of these implementations, we will be restarting all environments within NL3, which is expected to eliminate most, if not all, current faulty connections within the messaging system. Despite our messaging system not being in the desired state, it remains stable and operational. We will continue to actively monitor the messaging system until we are confident that the underlying issue is resolved. Posted 3 months ago. Apr 02, 2024 - 17:05 CEST Update The queue within our messaging system has caught up, suggesting that the delay should not be significant. We are monitoring the situation, while our engineers will continue working on mitigating the symptoms and implementing a permanent solution to address the underlying issue. If you have any questions or require support, please don't hesitate to reach out to us at https://my.topdesk.com/. We will continue to monitor the situation closely and provide updates as soon as they are available. Posted 3 months ago. Apr 02, 2024 - 12:19 CEST Update Our engineers have implemented changes to our messaging system. We can now confirm that action sequences are once again executing with a delay, while our messaging system catches up with the queue. Now that the messaging system is functioning properly, our engineers will work on mitigating the symptoms and implementing a permanent solution to address the underlying issue. If you have any questions or require support, please don't hesitate to reach out to us at https://my.topdesk.com/. We will continue to monitor the situation closely and provide updates as soon as they are available. Posted 3 months ago. Apr 02, 2024 - 10:44 CEST Update We're currently receiving reports of delayed action sequences. Despite the changes made by our engineers, the issues persist, and the system's stability remains a challenge. Our team is actively working to get the messaging system stable and resolve the underlying issues. We are committed to identifying and addressing the root cause of the problems. This update marks the resumption of regular communication after the holiday period. We'll keep you informed of any significant developments as we work towards a resolution. If you have any questions or require support, please don't hesitate to reach out to us at https://my.topdesk.com/. Thank you for your understanding as we work to maintain the functionality and stability of our messaging system. Posted 3 months ago. Apr 02, 2024 - 09:35 CEST Update After our engineers have implemented the planned changes, we found that not all symptoms have been mitigated. We have taken extra measures to ensure that the system stays in a stable state. Due to these measures, performance will remain degraded throughout the weekend. Customers might still experience a delayed execution of action sequences as a result. We will continue our efforts to establish the root cause. Unless the situation changes, this will be the last update in this weekend. We will follow up on Tuesday. We apologize for the inconvenience and thank you for your continued patience and understanding. Posted 3 months ago. Mar 30, 2024 - 22:43 CET Update Our engineers are still busy implementing the changes and expect that it will take an additional 2 hours before they are fully implemented. This means that performance will be degraded until 20:00 (UTC). Thank you for your understanding as we work diligently to maintain the functionality and stability of our messaging system. Posted 3 months ago. Mar 30, 2024 - 18:29 CET Update Our engineers are still hard at work implementing the changes which are taking longer to implement than expected. Based on the current progress we expect it to take 3 hours longer than originally plannend meaning that performance will be degraded until 18:00 (UTC). Posted 3 months ago. Mar 30, 2024 - 14:49 CET Update While our messaging system's performance remains stable and is operating as expected, the underlying issue persists. To address these ongoing concerns, this weekend, during the public holiday period, we're seizing the opportunity to tackle the issues. Our standby developers will be actively working on recreating the nodes in our messaging system to flush out these issues. We will start implementing the changes Saturday 30th of March between 10am-15pm (UTC) we expect degraded performances during these hours. As it's a weekend and Monday is a public holiday in many countries, we'll be limiting communication on the status page to critical updates only during this period. Thank you for your understanding as we work diligently to maintain the functionality and stability of our messaging system. Posted 3 months ago. Mar 29, 2024 - 16:41 CET Update In our ongoing efforts to maintain the stability of our messaging system on NL3, we conducted a restart for the "Group 1" and "Test" upgrade groups last night. While this action resulted in a positive change with the numbers, it wasn't the outcome we anticipated. Despite this, our messaging system's performance remains stable and is operating as expected. We will continue to closely monitor our messaging system's performance to ensure minimal disruption and will provide another update later today to share further details on our progress. If you have any questions or require support, please don't hesitate to reach out to us at https://my.topdesk.com/. Thank you for your understanding as we work to maintain the functionality and stability of our messaging system. Posted 4 months ago. Mar 29, 2024 - 10:17 CET Update In our ongoing efforts to maintain the stability of our messaging system, We are scheduling a restart for the "Group 1" and "Test" upgrade groups during the maintenance window. This action is aimed at ensuring that our message system maintains its standard functionality. We will closely monitor all traffic passing through our messaging system to ensure minimal disruption. If you have any questions or require support, please don't hesitate to reach out to us at https://my.topdesk.com/. We will continue to monitor the situation closely and provide another update tomorrow morning to provide further details on our progress. Posted 4 months ago. Mar 28, 2024 - 16:54 CET Update Our messaging system continues to function, maintaining its standard functionality. Despite ongoing stability, the symptoms persist. Our team is actively investigating to identify the root cause. While services remain operational, we will keep monitoring the situation to ensure minimal disruption. Our developers are working to mitigate the symptoms and implement a permanent solution to address the underlying issue. If you have any questions or require support, please don't hesitate to reach out to us at https://my.topdesk.com/. We will continue to monitor the situation closely and provide updates as soon as they become available. Thank you for your patience and understanding. Posted 4 months ago. Mar 28, 2024 - 14:03 CET Update As normal operations resumed on our NL3 datacenter last night, our monitoring systems indicated that the messaging component was running into a bottleneck. Our engineers worked through the night, increasing resources where necessary. This has significantly improved the situation, and while the metrics have not fully returned to normal, they are looking healthy. Please note that almost all NL3 environments underwent a restart last night, some of which were outside the standard maintenance window. This was an essential step to ensure a return to optimal functionality. While we have managed to alleviate the problematic symptoms for now, our engineers are continuing their investigations to fully resolve the issue. They will continue to closely monitor the situation to ensure that customer experience is not impacted. In addition, we feel it's important to stress that the recent stability issues occurred only in this particular datacenter. Therefore, we will be performing a thorough Root Cause Analysis in the coming days to determine what caused this behavior and how we can address it to prevent future outages. We sincerely apologize for any inconvenience this may have caused. If you have any further questions or concerns, please feel free to contact our support department. Posted 4 months ago. Mar 28, 2024 - 09:10 CET Update The measurements implemented have provided an indication that the situation is improving, but we are still closely monitoring it as we have not yet observed a full return to normalcy. The issue at hand has our highest priority and we will continue monitoring this closely and will provide further updates tomorrow morning. We apologize for any inconvenience this is causing. Posted 4 months ago. Mar 27, 2024 - 16:56 CET Monitoring The adjustments implemented have given us an idea that the situation is returning to normal. We anticipate that things are normalizing, but we are closely monitoring the situation for any further developments. Customers are encouraged to contact support if they continue to experience issues. Thank you for your patience and our apologies for the inconvenience this issue has caused. Posted 4 months ago. Mar 27, 2024 - 14:35 CET Update The adjustments have not yielded the result we were striving for, leaving the messaging system still affected. We will continue our investigation into the issue. The next update will be provided when we have any relevant news on the issue or have resolved it. Posted 4 months ago. Mar 27, 2024 - 12:46 CET Update Our engineers have increased the capacity of our messaging system to handle the amount of processes. A restart of the system has been initiated and we're monitoring the results. We will post another update as soon as possible. Posted 4 months ago. Mar 27, 2024 - 11:50 CET Update The measures implemented have not yielded the expected results, and unfortunately, issues with automated actions persist. We are continuing our investigation and implementing further measures to resolve the issue as soon as possible. Posted 4 months ago. Mar 27, 2024 - 11:14 CET Update We are currently investigating the root cause of the message delivery delays. However, the measures we have implemented appear to have addressed the issue, and we anticipate that things will return to normal soon. We are now awaiting confirmation of resolution. Posted 4 months ago. Mar 27, 2024 - 10:54 CET Update We're currently addressing issues at our NL3 hosting location, impacting action sequences. Our Technical Support and Development teams are actively investigating the issue and trying to implement measurements to solve the issue. We apologize for any inconvenience caused. Further updates will be provided on issue as soon as they become available. Posted 4 months ago. Mar 27, 2024 - 10:32 CET Investigating We are currently experiencing problems on the NL3 hosting location. As a result your action sequences might not work. We are aware of the problem and are working on a solution. Our apologies for the inconvenience. At the time of writing this we are not able to give you an estimate on when the problem will be resolved. We aim to update this status page every 30 minutes until the issue has been resolved. E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates. To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident TDR24 03 9021. Posted 4 months ago. Mar 27, 2024 - 10:04 CET This incident affected: NL3 SaaS hosting location. ← Current Status Powered by Atlassian Statuspage × SUBSCRIBE TO INCIDENT Subscribe to updates for RESOLVED: Disruption in hosting location NL3. Action Sequences are not functioning. via email and/or text message. You'll receive email notifications when incidents are updated, and text message notifications whenever TOPdesk SaaS Status page creates or resolves an incident. VIA EMAIL: VIA SMS: Afghanistan (+93) Albania (+355) Algeria (+213) American Samoa (+1) Andorra (+376) Angola (+244) Anguilla (+1) Antigua and Barbuda (+1) Argentina (+54) Armenia (+374) Aruba (+297) Australia/Cocos/Christmas Island (+61) Austria (+43) Azerbaijan (+994) Bahamas (+1) Bahrain (+973) Bangladesh (+880) Barbados (+1) Belarus (+375) Belgium (+32) Belize (+501) Benin (+229) Bermuda (+1) Bolivia (+591) Bosnia and Herzegovina (+387) Botswana (+267) Brazil (+55) Brunei (+673) Bulgaria (+359) Burkina Faso (+226) Burundi (+257) Cambodia (+855) Cameroon (+237) Canada (+1) Cape Verde (+238) Cayman Islands (+1) Central Africa (+236) Chad (+235) Chile (+56) China (+86) Colombia (+57) Comoros (+269) Congo (+242) Congo, Dem Rep (+243) Costa Rica (+506) Croatia (+385) Cyprus (+357) Czech Republic (+420) Denmark (+45) Djibouti (+253) Dominica (+1) Dominican Republic (+1) Egypt (+20) El Salvador (+503) Equatorial Guinea (+240) Estonia (+372) Ethiopia (+251) Faroe Islands (+298) Fiji (+679) Finland/Aland Islands (+358) France (+33) French Guiana (+594) French Polynesia (+689) Gabon (+241) Gambia (+220) Georgia (+995) Germany (+49) Ghana (+233) Gibraltar (+350) Greece (+30) Greenland (+299) Grenada (+1) Guadeloupe (+590) Guam (+1) Guatemala (+502) Guinea (+224) Guyana (+592) Haiti (+509) Honduras (+504) Hong Kong (+852) Hungary (+36) Iceland (+354) India (+91) Indonesia (+62) Iraq (+964) Ireland (+353) Israel (+972) Italy (+39) Jamaica (+1) Japan (+81) Jordan (+962) Kenya (+254) Korea, Republic of (+82) Kosovo (+383) Kuwait (+965) Kyrgyzstan (+996) Laos (+856) Latvia (+371) Lebanon (+961) Lesotho (+266) Liberia (+231) Libya (+218) Liechtenstein (+423) Lithuania (+370) Luxembourg (+352) Macao (+853) Macedonia (+389) Madagascar (+261) Malawi (+265) Malaysia (+60) Maldives (+960) Mali (+223) Malta (+356) Martinique (+596) Mauritania (+222) Mauritius (+230) Mexico (+52) Monaco (+377) Mongolia (+976) Montenegro (+382) Montserrat (+1) Morocco/Western Sahara (+212) Mozambique (+258) Namibia (+264) Nepal (+977) Netherlands (+31) New Zealand (+64) Nicaragua (+505) Niger (+227) Nigeria (+234) Norway (+47) Oman (+968) Pakistan (+92) Palestinian Territory (+970) Panama (+507) Paraguay (+595) Peru (+51) Philippines (+63) Poland (+48) Portugal (+351) Puerto Rico (+1) Qatar (+974) Reunion/Mayotte (+262) Romania (+40) Russia/Kazakhstan (+7) Rwanda (+250) Samoa (+685) San Marino (+378) Saudi Arabia (+966) Senegal (+221) Serbia (+381) Seychelles (+248) Sierra Leone (+232) Singapore (+65) Slovakia (+421) Slovenia (+386) South Africa (+27) Spain (+34) Sri Lanka (+94) St Kitts and Nevis (+1) St Lucia (+1) St Vincent Grenadines (+1) Sudan (+249) Suriname (+597) Swaziland (+268) Sweden (+46) Switzerland (+41) Taiwan (+886) Tajikistan (+992) Tanzania (+255) Thailand (+66) Togo (+228) Tonga (+676) Trinidad and Tobago (+1) Tunisia (+216) Turkey (+90) Turks and Caicos Islands (+1) Uganda (+256) Ukraine (+380) United Arab Emirates (+971) United Kingdom (+44) United States (+1) Uruguay (+598) Uzbekistan (+998) Venezuela (+58) Vietnam (+84) Virgin Islands, British (+1) Virgin Islands, U.S. (+1) Yemen (+967) Zambia (+260) Zimbabwe (+263) Enter mobile number Edit number Send OTP Enter the OTP sent Resend OTP in 30 seconds To receive SMS updates, please verify your number. To proceed with just email click ‘Subscribe’ Subscribe to Incident Message and data rates may apply. By subscribing you agree to our Privacy Policy, the Atlassian Terms of Service, and the Atlassian Privacy Policy. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.