If anyone owns a blog or site that they suspect may appear on HackerNews (especially if you're posting it), then please take the small amount of time to put an instance of Varnish in front of the site.
Then, ensure that Varnish is actually caching every element of the page, and that you are seeing the cache being hit consistently.
You should expect over 10,000 unique visitors within 24 hours, with most coming in the 30 minutes to 2 hours after you've hit the front page on HN.
You need not do your whole site... but definitely ensure that the key landing page can take the strain.
Unless you've put something like Varnish in front of your web servers, there's a good chance your web server is going down, especially if your pages are quite dynamic and require any processing.
A few weeks ago I got on the front page and within a 24 hour period was hit with 29,000 unique visitors with 38,000 page views. The page itself is image heavy with 1.3 MB on first load. I'm running Wordpress with the Quick Cache plugin by PriMoThemes. I'm hosted on a shared 1and1 server.
I've been hit before and went down, that's when I installed the Quick Cache plugin. Also 1and1 moved me to another server at some point but I'm not sure if it was before or after. Either the cache plugin is really good or I'm on a rockin server all by my lonesome. Or both.
If you're self hosting a wordpress site grab the Hyper Cache plugin or the very very simple Quick Cache plugin by PriMoThemes.
Thank you. This is pretty much exactly what I read the post for and was a little disappointed not to find. People talk about getting frontpaged on HN, Slashdot or Reddit, and how you need to be sure you can handle the load, but never give any useful figures on what that load is.
Knowing that I can serve 10 requests per second and likely withstand a frontpage on HN is more useful to me than knowing that I need a way out if and when my web-servers are being totally overwhelmed.
Agreed. I self-hosted a wordpress blog for a video-game fan site. I had a server status page that was a normal wordpress page that imported a file from the filesystem to embed in the middle (reported status of the game servers).
During some large events, I was seeing 5-7 page views per second and Wordpress did just fine (I think I had 120k page views in 24 hours). But I made sure to test as much of my site as possible to make sure it performed well. Using various page analyzers to make sure the proper headers were returned so objects were cached on the client. Tuning wordpress to cache pages properly. Turning on PHP APC (opcode cache). Running various stress tests on the site (apache benchmark and loadimpact).
I'd say the page analysis tools and extended apache-benchmark runs really helped me tune my OS and services properly so I could handle a huge load.
The standard WP caching plugin most people use is W3 Total Cache. It can require some effort to setup and get everything working but once done it will make orders of magnitude improvements to your performance.
WP Super Cache (using the htaccess option) got a client's site through 56,000 uniques / 78,000 page views on a HostGator VPS with 1.3GB RAM. The site unexpectedly got picked up by Japanese blogs as it was being taken live. Google Analytics showed up to 400 active visitors. Fun times.
I'll be very happy to help you and HN'ers set Varnish up on their server (not looking for compensation for this) and get you through HN traffic on your launch day.
Plug: we've built several products around Varnish so we have a good handle of how/where Varnish can be leveraged. Here's a list of varnish things we've built at unixy:
Varnish is very much like a programmable device. A VCL that works for one website can break the other. So it's important to know what you're up against in order to cook up the right VCL.
You could come up with a generic VCL that works for most websites out there but its cache effectiveness diminishes as you try to account for the most common corner cases. In fact, we did come up with such VCL. We distribute it with the cPanel Varnish plugin.
If you ever have a question or need a hand with Varnish/VCL drop me an email. I'll be more than happy to help out.
Cucumbertown co-founder here. Nginx was serving the cache and our sense was it was caching. But then the day before we put in csrf validation to the login form and it was bypassing the caching.
So in theory we were positioned to serve from Nginx cache.
The blog mentions that they did have caching on with nginx (which is what Varnish does, isn't it?). The problems were because of nginx not caching the frontpage (configuration issue) and because there was an unexpected hit on solr.
The unexpected hit was just users using the full text search, right? That probably should have been one of the first things tested as I assume that would be a bottleneck in speed in all stages of development...or at least it has been in my experience.
I often try to think of ways to not have full text search...this case is a little more difficult, but why not create a list of common recipe names (i.e. "grilled cheese" would be a facet for all grilled cheese sandwiches) and store it as a static json? It would take some taxonomy work on the backend but the list as JsON could easily be less than a MB and the. You wouldn't have to worry about full search as much...full search could still be an option, just not a top of the front page option.
Then, ensure that Varnish is actually caching every element of the page, and that you are seeing the cache being hit consistently.
You should expect over 10,000 unique visitors within 24 hours, with most coming in the 30 minutes to 2 hours after you've hit the front page on HN.
You need not do your whole site... but definitely ensure that the key landing page can take the strain.
Unless you've put something like Varnish in front of your web servers, there's a good chance your web server is going down, especially if your pages are quite dynamic and require any processing.