Skyrock, quite unknown platform

ParisWeb 2009 is the French speaking european community event around Web technologies. This year, I gave a talk about HTTP, this not-so-known technology. Among the 400 conference attendees, there were 30 Web developers of skyrock (The internal Web Team is around 50 persons including moderators, managers, etc). This social network outside of the French speaking world is not very well known. Let’s start with a few details.

Skyrock Summary

Skyrock has been created in December 2002 and is now the 7th largest social network in the world. In July 2009, it was said to have 26 millions blogs, 16.6 millions profiles, 650 millions of articles and almost 4 billions of comments. In December 2007, the blogs brought more than 15 millions euros in revenues according to its CEO Pierre Bellanger.

Skyrock Platform

The information is scarse about skyrock, but searching here and there, we can find a few details. In 2005, skyrock was serving

around 3 millions pages a day on two gigabit links. The balancing rules are quite complex since the content is splitted on multiple servers and SANs. We use software load balancers: Zeus ZXTM [zeus.com] on Gentoo Linux. The nice thing about software load balancers is that you can easily replace the hardware if it fails. Having a spare PC is way cheaper than a spare load balancer. We are very pleased with ZXTM so far. Very reliable, fast, and very flexible. It uses a PHP-like scripting language to process requests and you can really handle any specific backend architecture with that.

and we can discover a bit more information from 2004

We have a bunch of static servers for static HTML, CSS, images, etc. They run minimal Apache servers, designed for speed, with NPTL and the worker MPM. Non-forking servers like thttpd or lighttpd is also an option. The static servers are mainly old P3 machines, with only 512 Mb RAM.

Then, we have servers for PHP. The Apache they are running is huge (our web sites need a lot of modules), the hosts are dual 3 Ghz Xeon with 2 Gb RAM and there are some other specific tweaks.

Content differentiation is important. It’s a waste to spawn huge Apache process to serve static stuff, just because the same host should also be able to serve PHP. Also, tuning (esp. NFS) is very different for static and dynamic content. And as a specialized server often serves the same files, caching is more efficient.

We run Gentoo Linux on all web servers, plus one DragonFlyBSD (mostly for testing).

The same content differentiation is made for SQL server. One SQL server serves one sort of thing, so that caching is efficient. Also don’t forget that on x86, Linux and MySQL can hardly use more than 2 Gb of RAM. So with big tables, this is really annoying. We are switching SQL servers to Transtec Opteron-based servers for that.

On high traffic infrastructures, the I/O is often the bottleneck especially if you serve a lot of different content.

For our blog service, we had to buy a Storagetek disk array with 56 disks (fiber channel, 15k) in RAID 10. As NFS would introduce too much delay, we directly plugged two web servers to the controller of the disk array. These web servers are the NFS servers for the PHP servers, but they also directly serve the static content.

The access time of hard disk is really annoying. For shared data, but also for databases. We found that RAID 5 was way too slow (even with the high-end Storagetek/LSI controller) since we have about 1 write for 5 reads. So we had to switch everything to RAID 10. It really performs better, but it’s obviously more expensive.

Another bottleneck was the share of PHP sessions between all load-balanced PHP server. We first used a MySQL/InnoDB-based solution, but it poorly scaled. That’s why I had to write specific software : Sharedance http://sharedance.pureftpd.org/ [pureftpd.org]

If you have more information about this platform, please share with us. It would be an excellent opportunity for an article on High Scalibility blog.

Update 2009-10-19: That was quick ;) information from 2007 by the dev team.

9 loadbalancers Zeus Layer 7 ZXTM: machines diverses;

50 frontaux Web Apache 1.3.x: bi dual-core, 6 Go de ram, PHP 5.2, Xcache, Nginx pour les élements statiques;

50 serveurs MySQL 5.0.x: bi dual-core, 16 Go de ram, 8 disques SAS RAID 10 73 Go, en réplication master/master;

30 serveurs Memcached 1.2.x: bi dual-core (complètement sous exploité actuellement :), 16 go de ram, 3 instances memcached de 4 Go par machine;

2 serveur de sessions sharedance: stockage tmpfs 6 go (on en utilise grosso modo la moitié), un en master, un en secours;

1 cluster de stockage Isilon: 40 nodes de 12 disques SATA pour le stockage; A coté, divers types de stockages encore un peu utilisés (principalement des restes d’avant Isilon), avec SAN Sata Transtec, SAN Fiber Chanel Transtec, SAN Fiber Chanel Storagtek, SCSI Interne, SATA Interne… Oui, on a essayé beaucoup de systèmes différents par le passé :-)

Le tout tourne sous Linux x86_64, généralement du debian avec nos paquets.

Le Grand Club Platform

At Pheromone, we are developing a social network for RDS, Le Grand Club. It’s why we are always eager to learn about the features, tools and options that other sites have been using. The lead developer for this social network is Benoit Goyette (A Web developer team is working on it every day). The site runs on Ruby On Rails with postgres.