A traffic spike might be a joy for marketing department after successful advertising placement or a viral campaign. However, you will not be able to use up that traffic if your website is not loading properly. And that is quite common: no one likes to pay for more hardware than it is necessary during “normal operation”. I have seen major local news sites shutting down during such spikes, and had experienced such spikes myself as well. And it is up to whole IT team to solve the issue as fast as possible. So, what to do?
1. Identify what causes server problems and what one can do to improve. Typical bottlenecks are disk access, database query amount, http and mysql process limits, and memory amount available for all the processes.
2. Check what pages are hit heavily. This can be done by examining http server logs, or (at worst case) setting Google analytics date to current day.
3. Check if caching works as intended. Check if mysql cache is turned on (This is fast one, and quite important), and if enough memory is assigned to mysql cache.
4. If it is possible, upgrade today. VDS upgrades can be done instantaneous. That is one of reasons I have started to love them. In other cases, it is not so fast, but you might get more memory installed quite quickly.
5. Tune the http process children amount. You do not want to have more of them than fit into memory, remaining from system processes and mysql. Swapping these processes is a really bad idea. This might need to be repeated if memory amount changes.
6. After doing easy stuff, time for more complex ones. You might have to implement easy version of page caching on heavily hit pages if this is feasible. If you code with PHP, this might be done with output buffer module.
7. Done easy caching but still no improvement? If the problem is database, time to dig into database queries. Turn database logging on for minute or two, then revert for version without logging. Also, it is worth checking slow queries with slow query log setting at mysql. You have to identify a) Slow queries b) queries that repeat multiple times during the page load c) queries that return same information for majority of pages (for example, some static block like recent articles). First type of queries should be rewritten if possible. Second type of queries should be eliminated by reusing the information. Third type of queries means that fragment of pages needs pre-cached elements.
8. As long as you look at the queries, check if there are multiple versions of the same queries. Rewrite them to look the same. Mysql can not cache queries if they are written differently (even when only case differs).
9. Also, you might want to turn some things off during peak times. Like page view counter (in worst case), some cron jobs.
10. Disk still heavily in use? Check if you can reduce how many http child processes are required to display the page. Move some elements to sprites. Or move images to different webserver if possible. Use single css file instead of multiple. And so on.
Anything else? Please comment bellow.
A traffic spike might be a joy for marketing department after successful advertising placement or a viral campaign. However, you will not be able to use up that traffic if your website is not loading properly. And that is quite common: no one likes to pay for more hardware than it is necessary during “normal operation”. I have seen major local news sites shutting down during such spikes, and had experienced such spikes myself as well. And it is up to whole IT team to solve the issue as fast as possible. So, what to do?
- Identify what causes server problems and what one can do to improve. Typical bottlenecks are disk access, database query amount, http and mysql process limits, and memory amount available for all the processes.
- Check what pages are hit heavily. This can be done by examining http server logs, or (at worst case) setting Google analytics date to current day. There are log analytic software, but you will probably see with the naked eye what pages occur most often. “grep” might help in this case too, by excluding images from your search.
- Check if caching works as intended. Check if mysql cache is turned on (This is fast one, and quite important), and if enough memory is assigned to mysql cache. Mysql caching is switched off by default.
- If it is possible, upgrade today. VDS upgrades can be done instantaneous. That is one of reasons I have started to love them. In other cases, it is not so fast, but you might get more memory installed quite quickly.
- Tune the http process children amount (if you use Apache). You do not want to have more of them than fit into memory, remaining from system processes and mysql. Swapping these processes is a really bad idea. This might need to be adjusted if memory amount changes.
- After doing easy stuff, time for more complex ones. You might have to implement easy version of page caching on heavily hit pages if this is feasible. If you code with PHP, this might be done with output buffer control module.
- Done easy caching but still no improvement? If the problem is database, time to dig into database queries. Turn database logging on for minute or two, then revert for version without logging. Also, it is worth checking slow queries with slow query log at mysql. You have to identify a) Slow queries b) queries that repeat multiple times during the page load c) queries that return same information for majority of pages (for example, some static block like recent articles). First type of queries should be rewritten if possible. Second type of queries should be eliminated by reusing the information. Third type of queries means that fragment of pages needs pre-cached elements.
- As long as you look at the queries, check if there are multiple versions of the same queries. Rewrite them to look the same. Mysql can not cache queries if they are written differently (even when only case differs).
- Also, you might want to turn some things off during peak times. Like page view counter (in worst case), some cron jobs.
- Disk still heavily in use? Check if you can reduce how many http child processes are required to display the page. Move some elements to sprites. Or move images to different webserver if possible. Use single css file instead of multiple. And so on. It is worth considering using lighttpd for showing images instead of apache.
Overall, it is better to be prepared for traffic spike than be surprised by it. However, in lot of cases even on dire server loads there are things, that can be done so users would not notice any downtime and marketing campaign would not go in vain.
Anything else? Please comment bellow.
0 Comments