Over the past months we’ve been keeping a close eye on the impact of the Amazonbot on our servers. The Amazonbot is Amazon’s web crawler used to collect data for their search index and improve their services, such as enabling Alexa to answer (even) more questions for their customers.
We’re always on top of these kinds of activities since bots, crawlers and unwanted traffic can have a negative impact on the performance of online stores. In this article we gladly share the insights we gained about the Amazonbot.
Here’s what the past has taught us
Over the past 20 years, we’ve seen many bots come and go. In recent years, the demand for data to fuel search engines and artificial intelligence (AI) has reached unprecedented heights, and we have yet to reach the peak of this trend… Bots have advanced significantly, enabling them to crawl through millions of pages at an incredible speed while extracting product information and prices along the way.
Two years ago, we posted an article about Huawei’s AspiegelBot aggressively crawling and indexing websites to gather information for their search engine app. This bot flooding led to sites and online services being unavailable due to being overwhelmed by AspiegelBot’s traffic. On our platform we saw an insane increase in traffic of 460% in a month’s time!
Don’t underestimate the impact of bots and crawlers
As you understand by now, the impact of bots and crawlers shouldn’t be taken lightly. As the bots crawl through a website, it can cause a significant increase in server load, leading to slower load times and poor user experience for other visitors. This can result in frustrated customers and lost sales, but it could also affect search engine rankings, as page speed is an important ranking factor.
The impact of Amazonbot on the Hypernode platform
The graph below shows the activity of the Amazonbot across our Hypernode hosting platform. Interestingly, the traffic from the Amazonbot differs a lot from week to week.
The traffic decreased by 44% between March 21st and March 27th. However, between April 4th and April 14th, there was an increase of 47%. While it is still too early to draw any conclusions based on these numbers, it is advisable to monitor the situation closely if you are an owner or employee of an online store.
Is the Amazonbot worth utilizing your server’s CPU cycles?
This is a question we can’t answer for you. We have already seen some cases where 10%+ of the servers’ compute capacity was used by the Amazonbot to collect webshop data. This could be way too much, or not, if your business heavily relies on Amazon’s search index. We therefore recommend you to check with your SEO specialist if the Amazonbot is worth the server load it generates and block them otherwise.
How you easily block bad bots in the Hypernode Control Panel
We offer you the possibility to block bad user agents with a few mouse clicks, via our Control Panel. Simply log in to the Control Panel, select a Hypernode and then click on Bot Blocklist in the sidebar. More information can be found in our support documentation about How to Block Bad Bots via the Control Panel.
If you have experiences about the Amazonbot you would like to share, please get in contact with us!
Hi! My name is Dion, Account Manager at Hypernode
Want to know more about Hypernode's Managed E-commerce Hosting? Schedule your online meeting.
schedule one-on-one meeting +31 (0) 648362102
