Our cloud service provider identified and resolved a networking issue that temporarily affected connectivity in the Jakarta data center. Between 11:00 and 13:00 UTC, customers in the impacted region may have experienced intermittent packet loss and connection timeouts.
Our cloud service provider identified and resolved a networking issue that temporarily affected connectivity in the London data center. Between 14:00 and 16:00 UTC, customers in the impacted region may have experienced intermittent packet loss and connection timeouts.
Our cloud service provider resolved a networking issue that affected internal connectivity in the Chicago data center. Customers using the storage we provide in this region may have experienced reduced performance or intermittent errors when accessing files between 08:00 and 10:00 UTC.
Our cloud service provider identified and resolved a networking issue that impacted connectivity at the Mumbai and Chennai data centers. Customers in the affected regions may have experienced packet loss and connection timeouts between 17:00 and 18:00 UTC.
Our cloud service provider identified and resolved a networking issue that impacted connectivity at the Miami and Milan data centers. Customers in the affected regions may have experienced intermittent connectivity disruptions between 17:40 and 18:20 UTC. However, in most cases, the impact was minimal and limited to slight slowdowns, with no noticeable service interruption for the majority of users.
Our cloud service provider investigated and resolved an issue that was affecting high-performance Object Storage endpoints in the following data centers:
Clients with deployments in these regions using the affected storage endpoints may have experienced upload and download errors when interacting with the storage bundled with our SaaS offerings. Clients using their own cloud storage were not affected. Affected customers were notified via email, and mitigations were implemented where applicable to minimize impact.
Our cloud service provider investigated and resolved networking issues affecting the Los Angeles data center. Customers in the Los Angeles region experienced intermittent connectivity disruptions between 18:40 and 19:00 UTC.
Our cloud service provider investigated and resolved an issue affecting the Object Storage service in the Miami data center. Customers with deployments in the Miami region may have experienced intermittent access issues specifically with our provided Object Storage between 10:26 and 14:29 UTC.
Connectivity issues in the Frankfurt and Stockholm data centers, caused by a networking incident, were resolved by our cloud provider. The disruption began at 20:15 UTC and was resolved by approximately 22:05 UTC at the Stockholm data center and by 22:30 UTC at the Frankfurt data center.
Our cloud service provider investigated and resolved networking issues affecting the Miami data center. Customers in the Miami region experienced intermittent connectivity disruptions between 19:25 and 19:45 UTC.
Our cloud service provider investigated and resolved networking issues affecting the Amsterdam data center. Two of our customers in Amsterdam were unable to access the services during the time frame of 17:45–20:10 UTC.
On March 6th at 10:57 UTC we received warnings from our monitoring system regarding excessive memory usage and upload errors for an installation in the Miami data center. After investigation, we found that the issues were related to incorrect settings for multi-part uploads. These are advanced settings and it was our mistake to leave them configurable in the WebAdmin UI. We contacted our customer and corrected the configuration.
Timeline:
Postmortem:
On March 8, we rolled out an update to all of our SaaS instances to hide advanced settings that could confuse users and are closely related to the resources allocated to the instances. These settings are now automatically configured based on the instance type.
Our cloud service provider investigated and resolved an issue involving the Object Storage service in the Los Angeles data center. For our Los Angeles deployments, clients may have experienced intermittent errors during uploads and downloads to/from our provided storage during the time frame of 19:04–21:08 UTC.
SFTPGo