Stability and performance improvements – We’re ready for Black Friday

2020 is going to be a record year for ecommerce, partially due to the impact of COVID-19. Research shows online stores expect an extra increase in turnover of more than 20% compared to the same period last year. In a proactive approach for this soon to be record-breaking holiday season, we’ve made a number of progressive stability and performance improvements. We’re glad to share them in this blogpost.

How we made Hypernode steadier than a rock

We’ve randomized the Let’s Encrypt Renewal Cronjobs

The cron we suggest for renewing Let’s Encrypt certificates is widely used across our Hypernode platform. Lately, we’ve noticed an increasing amount of failed requests to renew certificates. This is caused by the Let’s Encrypt servers being under too much load, due to the large volume of requests.

In hindsight, the amount of times this cron checks whether or not there is a new certificate available, is a bit excessive. That’s why we made the cron a weekly check, instead of a daily check. If you have this cron configured, it will now perform the check on a random weekday, somewhere between 0 and 7 AM UTC.

We automatically block requests coming from Petalbot

Earlier this year we informed you about the increasing impact of the Aspiegel/Petalbot on our Hypernode platform. Since the pattern in requests has not changed, there are still millions of daily requests coming from this bot. We previously applied a ratelimit to reduce the impact caused by the Pedalbot.

However, the ratelimiter didn’t solve the problem entirely because we still were getting complaints about the Petalbot impacting the performance of Hypernodes. That’s why we now automatically block all requests coming from the Petalbot when they cross our threshold of malicious requests.

We’ve pinned the version of Composer to version 1

Composer, a commonly-used dependency manager for Magento, has released a new version: Composer 2.0. At the release day, we’ve received many complaints about the Composer 2.0 not working as intended. Because we want to make sure Composer isn’t causing any issues during the busy and important holiday season, we are enforcing Composer 1. We’ve also prevented the composer from automatically self-updating to the newest version. After the holiday season, we will have another look at Composer 2.0.

We’ve set a maximum amount of requests for PHP-FPM workers

PHP-FPM workers are the processes that run Magento. In short, these workers handle requests and send responses accordingly. Once a request is finished, the workers wait for the next requests. The memory used to serve a request is eventually returned to the system.

If there is a memory leak in PHP, for example caused by PHP or an extension, a portion of the memory used to serve a request is not returned to the system. Over time, a significant amount of memory can be leaked and then is not available to the system anymore. By setting a maximum amount of requests for PHP-FPM workers and spawning new workers if they reach this amount, we set a limit on possible memory leaks. This way we ensure that the system’s available memory is used to serve shop requests.

If you’re looking for more technical details about these stability improvements, please check out this changelog: Release 7786: Ensuring stability on Hypernode.

How we made Hypernode faster than the speed of light

We’ve changed the PHP-FPM workers to CPU core ratio

Previously, our ratio of PHP-FPM workers to CPU cores was 5 PHP-FPM workers to 1 CPU core. After experimenting with different ratios in benchmarks, it turned out that this ratio was suboptimal. Therefore we changed the ratio to 4 PHP-FPM workers to 1 CPU core which leads to an average response time improvement of 5.9%!

We’ve given CPU priority to Nginx and Varnish

While benchmarking our platform, we noticed that when we applied a high load to the Hypernode, the load was eventually bottlenecked on the CPU. When this happens, the CPU scheduler has to make a decision where to spend its resources on.

When your Hypernode is experiencing a high CPU load, for example caused by traffic peaks in the holiday season, you want to ensure that most requests still succeed. In order to do so, you want to make use of as many cached and static requests as possible. They have a less impact on the CPU. Since cached and static requests come directly from Varnish and Nginx, we increased the priority of these processes in the CPU scheduler.

We’ve started with caching Hypernode monitoring requests

Every two minutes three of our heartbeat servers do a series of checks to make sure all critical services are still running on your Hypernode. This monitoring is critical for our platform to ensure our systems can act upon any issues that are found on your Hypernode. However, when your shop is experiencing a high load, we want to reserve all resources for your visitors. The last thing we want is to cause any more load with our monitoring. To prevent this from happening, we started caching monitoring results of all system checks. By caching monitoring requests we decrease the impact but still make sure all critical services are running on your Hypernode.

If you’re looking for more technical details about these performance improvements, please check out this changelog: Release 7785: Increasing performance on Hypernode.

Have you become interested in the stability and performance of our hosting platform? Take a peek at our website or order a 14-day free trial to find out what Hypernode can bring to your eCommerce store.