PHP vs HHVM - Silverstripe

Posted by Twisted Bytes on 11 May 2015

How much faster is HHVM running a real PHP framework? That's what we look at. A few weeks ago we wrote a blog about how HHVM can be an interesting option for websites and applications. Now we are looking into the difference when using HHVM with a framwork. The framework we tested this time is Silverstripe.

The last blog was a simple comparison between PHP and HHVM. It showed that HHVM can be 70% to 800% faster. That was a small test of some PHP functions, but real websites will produce different results.

Real PHP sites work with a database, reads files, have external connections to APIs, and a lot of PHP files are read and parsed. These activities have a different effect on the speed and that's where the strength and quality of HHVM lies.

TL;DR, to the results.

Setup

The test is performed using the demo project of SilverStripe. This project consists of a working SilverStripe site with about 15 pages, content and pictures. The result is a site which is representative of a simple PHP site or application.

For the test we made an Apache JMeter plan. This test creates an increasing number of threads, from 15 to 300 in increments of 15, which continuously request pages of the site. The translation of threads to users is not a simple calculation, but an estimate is that each thread is about 10 to 40 users. A thread does not wait before requesting a new page, but users do.

We put the demo site on a serverand only switch between PHP en HHVM during the tests. No changes to anything else. The PHP version tested is 5.6.7 and HHVM version 3.6.0. After testing with PHP and HHVM we upgraded the server to a faster one to see how it scales with more CPU and memory resources. We tested with these configurations:

• 2 cpu cores & 2G memory
• 4 cpu cores & 8G memory
• 8 cpu cores & 64G memory

We're only testing content generated by PHP and no static content. Static content is no problem for an Apache in event-mode or Nginx and therefore not of interest for this test.

Results

The test results are a list of graphs:

• response time
• How fast is the site responding "now"
• hits/second & transactions per second
• How many answers does the server give each second
• response latencies over time & response times vs threads
• How much time does a request need, compared to the number of threads
• transaction throughput vs threads
• How many answers does the server give compared to the number of threads

In total 6 graphs. We do not want to show all of them here so we will only talk about the most interesting ones: response time and transactions per seconde.

2 cpu’s and 2GB memory

These are the results of HHVM and PHP running with 2 cpu's and 2GB of memory. It's not very clear but PHP has a maximum reposetime of about 5500 milliseconds, 5.5 seconds. PHP gets in trouble at about 240 concurrent threads. HHVM shows its doing things different. It's maximum is about 5000 milliseconds and the result are not all over the place. The difference in maximum response time is about 10%.

Someting to note is that the HHVM graphs show just one line, PHP shows 3 lines. This is a difference in response time for individual tested pages. Some pages need more time with PHP but for HHVM there is almost no difference.

PHP can handle about 45 requests per second while HHVM can do 60. That's a difference of 33%. The start of the HHVM graph shows HHVM needs to build up speed. That's the compiler and JIT warming up. Something else to notice is PHP producing error pages at the same moment the reposonse latencies get al over the place. That's the pink line at the bottom of the transactions per second page.

We do know HHVM is faster, but a new observation is the different response to many requests. PHP is starting to fail a lot and HHVM just needs more time.

4CPU’s en 8GB memory

The test with 4 cpu's and 8GB memory show the same results. Of course its faster. PHP still has problems with many threads and HHVM keeps going. PHP does show it's using the extra resources to be more stable at many threads. 

8CPU’s en 64GB memory

The last set of tests are done with 8 CPU's and 64GB of memery. PHP performes at a maximum of 1700 millisecond and HHVM at 1100 milisecond for a request. A difference of about 55%. And PHP keeps having problems when hitting the site with 240 or more threads per second.

The number of requests rises with the number of CPU's. PHP is at about 175 hits per second. HHVM is at about 210. A difference of about 54%.

Overview of all results

 

Test	PHP	HHVM	Difference
Max latency 2cpu & 2GB	5500 ms	5000 ms	10%
Max latency 4cpu & 8GB	4000 ms	2800 ms	43%
Max latency 8cpu & 64GB	1700 ms	1100 ms	55%
TX / second 2cpu & 2GB	45	60	33%
TX / second 4cpu & 8GB	75	110	47%
TX / second 8cpu & 64GB	175	270	54%

 

PHP and HHVM - use of memory and processes

An important result during this test was HHVM being more stable than PHP. At the same time we saw that HHVM does not need more memory when serving a lot of requests at the same time. PHP start using a lot of memory when hitting it with a lot of requests. Sadly we do not have not recorded the memory usage during the test.

The difference in handling requests between HHVM and PHP needs some explanation. The way PHP handles requests is by spawning extra processes. PHP can run in different modes: CGI, mod_php and php-fpm are some of them. We are not going into detail of this, but each has PHP processing requests in a single process for each request. And each process has the full PHP runtime in memory and after a while the process is recycled to free the memory. When processing 50 requests in parallel there are 50 processes in memory. During our tests each PHP process needed about 25 Megabyte of memory, so 50x25=1250 megabyte of memory.

HHVM is thread based. This means it does not spawn an extra process to handle a request. It spawns an extra thread. HHVM does some caching and compiling and keeps the results in memory and available for each thread. The runtime, data, caching and the results of the compilation are shared between the threads. Initially this needs extra memory. During our tests we saw 180MB to 350MB. That seems a lot compared to PHP, but this is all the memory it is going to need. After this HHVM did not need more memory.

This difference is one of the reasons why PHP and HHVM show such different graphs. When PHP handling a lot of requests the server is very busy handling the processes and memory. And in the end PHP is running out of memory. 

During the first test with just 2GB of memory PHP ran out of memory. at about 200 jmeter threads PHP needed 100 processes to handle it. And at that time PHP needed more memory than was available. That's the reason PHP was failing at 200+ jmeter threads. In later tests we had a lot more memory. This fixed the momery problem, but the OS became very busy with handling all PHP processes.

This is an extra reason to use HHVM instead of PHP. It is easier to handle memory and process limits.

Conclusion

There is a short and long conclusion. The short one first:

• HHVM is faster than PHP when running a real website. The difference is about 40%-60%. This is not as much as simple tests show, but it's still a lot.
• PHP uses more memory when handling 8-12 parallel requests. So small sites a better off with PHP.
• With busy sites HHVM can be a good choice because its more balanced in servicing the user requests and it will go, slower but still answering requests.

Now the longer conclusing.

HHVM is faster than PHP. And the change it will kill you site because of traffic is less. PHP will run out of memory or processes while HHVM just becomes slower for each individual request. When running PHP with a lot of requests these are the possibilities:

• The number of processes is limitted: users need to wait a long time to get an answer.
• The number of processes is not limitted: the memory will run out and users are going to get error messages.
• The number of processes is not limitted and you have a lot of memory: The server is going to be very busy handling all PHP processes, and PHP is going slower because of that.

Of all these option the last one is probably the best, but you need a big server for that. So there is a trade-off. And it only happens with big sites. HHVM will get you further on the same "hardware" and is an interessting option for those types of sites and application. Especially when sometimes you have a lot of users and do not want to run out of memory.

A good story has 2 sides and this one does too. HHVM is not (yet) fully compatible with PHP and it needs an other approach to set it up on a server. But it's a good tool to have when trying to get a better performancing site. 

If you are interessted in this article or want to test HHVM, please contact us: [email protected].