kenjis/php-framework-benchmark

PHP Framework Benchmark

This project attempts to measure minimum overhead (minimum bootstrap cost) of PHP frameworks in real world.

So I think the minimum applications to benchmark should not include:

cost of template engine (HTML output)
cost of database manipulation
cost of debugging information

Components like Template engine or ORM/Database library are out of scope in this project.

Benchmarking Policy

This is master branch.

Install a framework according to the official documentation.
Use the default configuration.
- Don't remove any components/configurations even if they are not used.
- With minimum changes to run this benchmark.
Set environment production/Turn off debug mode.
Run optimization which you normally do in your production environment, like composer's --optimize-autoloader.
Use controller or action class if a framework has the functionality.

Some people may think using default configuration is not fair. But I think a framework's default configuration is an assertion of what it is. Default configuration is a good starting point to know a framework. And I can't optimize all the frameworks. Some frameworks are optimized, some are not, it is not fair. So I don't remove any components/configurations.

But if you are interested in benchmarking with optimization (removing components/configurations which are not used), See optimize branch.

If you find something wrong with my code, please feel free to send Pull Requests. But please note optimizing only for "Hello World" is not acceptable. Building fastest "Hello World" application is not the goal in this project.

Frameworks to Benchmark

Phalcon 2.0.8
Ice 1.0.34
Tipsy 0.10.3
FatFree 3.5.0
Slim 2.6.2
CodeIgniter 3.0.2
NoFussFramework 1.2.3
Slim 3.0.0-RC2
BEAR.Sunday 1.0.1
Lumen 5.1.6
zend-expressive 1.0.0rc2 (FastRoute + zend-servicemanager)
Radar 1.0.0-dev
Yii 2.0.6
Silex 1.3.4
Cygnite 1.3.1
FuelPHP 1.8-dev
PHPixie 3.2
Aura 2.0.2
CakePHP 3.1.4
Symfony 2.7.7
Laravel 5.1.24
Zend Framework 2.5.2
TYPO3 Flow 3.0.0

Results

Benchmarking Environment

CentOS 6.6 64bit (VM; VirtualBox)
- PHP 5.5.30 (Remi RPM)
  - Zend OPcache v7.0.4-dev
- Apache 2.2

Hello World Benchmark

These are my benchmarks, not yours. I encourage you to run on your (production equivalent) environments.

(2015/11/29)

framework	requests per second	relative	peak memory	relative
phalcon-2.0	1,746.90	64.7	0.27	1.0
ice-1.0	1,454.79	53.9	0.26	1.0
tipsy-0.10	1,425.66	52.8	0.32	1.2
fatfree-3.5	1,106.20	41.0	0.42	1.6
slim-2.6	880.24	32.6	0.47	1.8
ci-3.0	810.99	30.1	0.43	1.7
nofuss-1.2	672.16	24.9	0.40	1.5
slim-3.0	534.16	19.8	0.61	2.4
bear-1.0	442.63	16.4	0.76	2.9
lumen-5.1	412.36	15.3	0.95	3.7
ze-1.0	391.97	14.5	0.80	3.1
radar-1.0-dev	369.79	13.7	0.70	2.7
yii-2.0	379.77	14.1	1.37	5.3
silex-1.3	383.66	14.2	0.86	3.3
cygnite-1.3	385.16	14.3	0.76	2.9
fuel-1.8-dev	346.33	12.8	0.65	2.5
phpixie-3.2	236.58	8.8	1.31	5.1
aura-2.0	233.80	8.7	0.89	3.5
cake-3.1	207.27	7.7	1.37	5.3
symfony-2.7	101.99	3.8	3.21	12.4
laravel-5.1	91.59	3.4	2.76	10.7
zf-2.5	81.13	3.0	3.02	11.7
typo3f-3.0	26.98	1.0	6.50	25.2

Note(1): All the results are run on php with phalcon.so and ice.so. If you don't load phalcon.so or ice.so, the rps except for Phalcon or Ice probably increase a bit.

How to Benchmark

Install source code as http://localhost/php-framework-benchmark/:

$ git clone https://github.com/kenjis/php-framework-benchmark.git
$ cd php-framework-benchmark
$ sh setup.sh

Run benchmarks:

$ sh benchmark.sh

See http://localhost/php-framework-benchmark/.

If you want to benchmark some frameworks:

$ sh setup.sh fatfree-3.5/ slim-3.0/ lumen-5.1/ silex-1.3/
$ sh benchmark.sh fatfree-3.5/ slim-3.0/ lumen-5.1/ silex-1.3/

Linux Kernel Configuration

I added below in /etc/sysctl.conf

# Added
net.netfilter.nf_conntrack_max = 100000
net.nf_conntrack_max = 100000
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

and run sudo sysctl -p.

If you want to see current configuration, run sudo sysctl -a.

Apache Virtual Host Configuration

<VirtualHost *:80>
  DocumentRoot /home/vagrant/public
</VirtualHost>

References

Aura (@auraphp)
BEAR.Sunday (@BEARSunday)
CakePHP (@cakephp)
CodeIgniter (@CodeIgniter)
Cygnite (@cygnitephp)
FatFree (@phpfatfree)
FuelPHP (@fuelphp)
Ice (@iceframework)
- See https://github.com/kenjis/php-framework-benchmark/pull/17#issuecomment-98244668
Laravel (@laravelphp)
Lumen
NoFussFramework
Phalcon (@phalconphp)
PHPixie (@phpixie)
Radar
Silex
Slim (@slimphp)
Symfony (@symfony)
- How to Deploy a Symfony Application
Tipsy
TYPO3 Flow (@neoscms)
Yii (@yiiframework)
zend-expressive (@zfdevteam)
Zend Framework (@zfdevteam)

Failed to load latest commit information.
aura-2.0	Extract the peak memory output code to a file	Oct 25, 2015
bear-0.10	Extract the peak memory output code to a file	Oct 25, 2015
bear-1.0	composer update	Nov 29, 2015
benchmarks	Add included file count to results	Oct 28, 2015
bin	Add script to output current framework list as array	Dec 6, 2015
cake-3.0	Extract the peak memory output code to a file	Oct 25, 2015
cake-3.1	composer update	Nov 29, 2015
ci-3.0	Rename folder name from codeigniter-3.0 to ci-3.0	Oct 25, 2015
cygnite-1.3	composer update	Nov 29, 2015
fatfree-3.5	Add Hello controller for fatfree-3.5	Oct 28, 2015
fuel-1.8-dev	composer update	Nov 29, 2015
fuel-2.0-dev	Extract the peak memory output code to a file	Oct 25, 2015
ice-1.0	Extract the peak memory output code to a file	Oct 25, 2015
laravel-4.2	Extract the peak memory output code to a file	Oct 25, 2015
laravel-5.0	Extract the peak memory output code to a file	Oct 25, 2015
laravel-5.1	composer update	Nov 29, 2015
libs	Change graph size	Nov 29, 2015
lumen-5.0	Extract the peak memory output code to a file	Oct 25, 2015
lumen-5.1	composer update	Nov 29, 2015
no-framework	Extract the peak memory output code to a file	Oct 25, 2015
nofuss-1.2	Fix index.php for nofuss-1.2	Oct 25, 2015
output	Update README and results	Nov 29, 2015
phalcon-1.3	Extract the peak memory output code to a file	Oct 25, 2015
phalcon-2.0	Extract the peak memory output code to a file	Oct 25, 2015
phpixie-3.2	composer update	Nov 29, 2015
radar-1.0-dev	Extract the peak memory output code to a file	Oct 25, 2015
silex-1.2	Extract the peak memory output code to a file	Oct 25, 2015
silex-1.3	composer update	Nov 29, 2015
slim-2.6	Extract the peak memory output code to a file	Oct 25, 2015
slim-3.0	composer update	Nov 29, 2015
symfony-2.5	Extract the peak memory output code to a file	Oct 25, 2015
symfony-2.6	Extract the peak memory output code to a file	Oct 25, 2015
symfony-2.7	composer update	Nov 29, 2015
tipsy-0.10	Fix coding style	Nov 29, 2015
typo3f-2.3	Rename folder name from typo3-flow-2.3 to typo3f-2.3	Oct 25, 2015
typo3f-3.0	Revert composer.lock of type3f-3.0	Nov 29, 2015
yii-2.0	composer update	Nov 29, 2015
ze-1.0	composer update	Nov 29, 2015
zf-2.4	Extract the peak memory output code to a file	Oct 25, 2015
zf-2.5	composer update	Nov 29, 2015
.gitignore	Update gitignore	Oct 15, 2015
README.md	Update README	Dec 9, 2015
benchmark.sh	Improve bash scripts	Oct 14, 2015
index.php	Add simple theme functionality	Oct 29, 2015
list.sh	Update list.sh	Nov 29, 2015
setup.sh	Add script to composer update	Oct 25, 2015
update.sh	Add script to composer update	Oct 24, 2015