After extended monitoring and analysis, we have not observed any new occurrences of the reported connectivity issues. All systems are stable and functioning as expected.
The incident is now considered resolved. We will continue to monitor our infrastructure as part of our normal operations.
Thank you for your patience and understanding
Posted Mar 26, 2025 - 10:12 UTC
Monitoring
After further investigation, we have identified that the remaining connectivity issues were caused by an automatic rule update process on one of our newly deployed firewalls. This process was inadvertently clearing active sessions during each update cycle, resulting in micro-outages for some users.
To recap:
The initial issue was related to aggressive ARP timeout settings on our leaf switches, which we resolved by increasing the timeout to 8 hours.
Despite that fix, some partners continued to report intermittent disruptions.
We traced these to the firewall behavior, which has now been adjusted to preserve active sessions during rule updates.
We are currently monitoring the infrastructure closely to ensure full stability. Thank you for your patience.
Posted Mar 25, 2025 - 21:10 UTC
Investigating
Following the previously applied changes to our network infrastructure, we are still receiving reports from some partners regarding intermittent micro-outages.
Our engineering team is actively investigating these new reports, as the root cause is not yet fully identified. We are working to gather more data and closely monitoring the situation.
Further updates will be provided as we make progress.
Posted Mar 25, 2025 - 09:58 UTC
Monitoring
We have identified the root cause of the connectivity issues as a misconfiguration in the ARP timeout settings on our leaf switches. Our configuration included a periodic ARP table clear every 15 minutes. As a result, if there was no active traffic within that timeframe, the leaf switches would lose the MAC address, causing intermittent connectivity or session drops. This behavior was successfully replicated in our lab environment. We have now removed the aggressive ARP timeout and increased the ARP entry lifetime to 8 hours, aligning it with a standard workday to prevent further unexpected disconnections. We are currently monitoring the system to ensure stability
Posted Mar 24, 2025 - 23:21 UTC
Identified
We have adjusted internal routing paths towards upstream internet providers in order to improve traffic ingress into our cloud infrastructure. These changes are expected to improve connectivity for affected users, especially those relying on French ISPs that were previously not learning our BGP routes properly.
We continue to monitor the situation closely and will share further updates as needed.
Posted Mar 24, 2025 - 21:18 UTC
Update
We have identified that some Internet Service Providers (ISPs) in France are currently not learning our BGP routes correctly. This may result in intermittent connectivity issues when accessing our cloud infrastructure from certain networks within the region.
Our team is actively working with our upstream providers to ensure proper route propagation and restore full connectivity for all affected users.
We will provide further updates as progress is made.
Posted Mar 24, 2025 - 21:00 UTC
Investigating
We are currently investigating reports of intermittent connectivity issues affecting customer access to our cloud infrastructure from the Paris region. Our engineering team is actively working to identify the root cause. Updates will be provided as more information becomes available.
Posted Mar 24, 2025 - 18:45 UTC
This incident affected: Servers (Paris) and Remote Desktop (Paris).