new doc on HA and recommended deployments

admin_manual/conf.py

@@ -286,3 +286,4 @@ epub_copyright = u'2012-2016, The ownCloud developers'
 # Include todos?
 todo_include_todos = True
 
+

admin_manual/contents.rst

@@ -9,6 +9,7 @@ Table of Contents
     release_notes
     whats_new_admin
     installation/index
+    deployment_recommendations/index
     configuration_server/index
     configuration_user/index
     configuration_files/index    

admin_manual/deployment_recommendations/deployment_recommendations.rst

@@ -0,0 +1,513 @@
+===================================
+ownCloud Deployment Recommendations
+===================================
+
+What is the best way to install and maintain ownCloud? The answer to that, of 
+course, is *"it depends"* because every ownCloud customer has their own 
+particular needs and IT infrastructure. ownCloud and the LAMP stack are 
+highly-configurable, so we will present three typical scenarios and make 
+best-practice recommendations for both software and hardware.
+
+General Recommendations
+-----------------------
+
+.. note:: Whatever the size of your organization, always keep one thing in mind: 
+   the amount of data stored in ownCloud will only grow. Plan ahead.
+
+The amount of data stored in an ownCloud instance continually grows. Plan ahead. 
+Consider setting up a scale-out deployment, or using Federated Cloud Sharing to 
+keep individual ownCloud instances to a manageable size.
+
+.. comment: Federating instances seems the best way organically grow in an 
+   enterprise. A lookup server to tie all the instances together under a single 
+   domain is being worked on.
+
+* Operating system: Linux.
+* Webserver: Apache 2.4.
+* Database: MySQL/MariaDB.
+* PHP 5.5+. PHP 5.4 is the minimum supported version; note that it reached 
+  end-of-life in September 2015 and is no longer supported by the PHP team. 
+  Some Linux vendors, such as Red Hat, still support PHP 5.4.
+  5.6+ is recommended. ``mod_php`` is the recommended Apache module because it 
+  provides the best performance.
+
+.. comment: mod_php is easier to set up, php-fpm with apache event MPM seems to 
+   scale better under load and limited RAM restrictions: 
+   http://blog.bitnami.com/2014/06/performance-enhacements-for-apache-and.html
+
+Small Workgroups or Departments
+-------------------------------
+
+* Number of users
+   Up to 150 users.
+
+* Storage size
+   100 GB to 10TB.
+
+* High availability level
+   Nightly interruption of service for backup, component failure leads to 
+   interruption of service.
+   
+*Image is missing*
+
+Recommended System Requirements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. comment: ***this image is missing*** ![OC small deployment 
+   scenario](http://yuml.me/diagram/scruffy/class/OC small deployment scenario, 
+   [Web server|Apache;DB Server;local storage], [Web server]-[LDAP Server])
+
+One machine running the application server, Webserver, database server and 
+local storage.
+
+Authentication via an existing LDAP or Active Directory server.
+
+* Components
+   One server with at least 2 CPU cores, 16GB RAM, local storage as needed.
+
+* Operating system
+   Enterprise grade Linux distribution with full support from OS vendor. Red 
+   Hat Enterprise Linux or SUSE Linux Enterprise Server 12 are recommended.
+
+* SSL Configuration
+   The SSL termination is done in Apache. A standard SSL certificate is 
+   needed, installed according to the Apache documentation.
+
+* Load Balancer
+   None. 
+
+* Database
+   MySQL, MariaDB or PostgreSQL.
+ 
+.. comment: We currently recommend MySQL / MariaDB, as our customers have 
+   had good experiences when moving to a Galera cluster to scale the DB.
+
+* Backup
+   Automatic nightly backups:
+   
+   * Shut down Apache.
+   * Create database dump.
+   * Push data directory to backup.
+   * Push database dump to backup.
+   * Start Apache.
+   * Optionally rsync to a backup storage and tape backup. (See the 
+     `Maintenance`_ section of the Administration manual for tips on backups 
+     and restores.)
+
+* Authentication
+   User authentication via one or several LDAP or Active Directory servers. (See
+   `User Authentication with LDAP`_ for information on configuring ownCloud to 
+   use LDAP and AD.)
+
+* Session Management
+   Local session management on the application server. PHP sessions are stored 
+   in a tmpfs mounted at the operating system-specific session storage 
+   location. You can find out where that is by running ``grep -R 
+   'session.save_path` /etc/php5`` and then add it to the ``/etc/fstab`` file, 
+   for example: 
+   ``echo "tmpfs /var/lib/php5/pool-www tmpfs defaults,noatime,mode=1777 0 0" 
+   >> /etc/fstab``.
+
+* Caching
+   A memcache speeds up server performance, and ownCloud supports four 
+   memcaches; refer to `Configuring Memory Caching`_ for information on 
+   selecting and configuring a memcache.
+
+* Storage
+   Local storage.
+
+* ownCloud Edition
+   Standard Edition. (See `ownCloud Server or Enterprise Edition`_ for 
+   comparisons of the ownCloud editions.)
+
+Mid-sized Enterprises
+---------------------
+
+* Number of users
+   150 to 1,000 users.
+   
+* Storage size
+   Up to 200TB.
+   
+* High availability level
+   Every component is fully redundant and can fail without service interruption. 
+   Backups without service interruption
+   
+.. figure:: images/deprecs-2.png
+   :alt: Network diagram for mid-sized enterprise.
+
+Recommended System Requirements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+2 to 4 application servers.
+
+A cluster of two database servers.
+
+Storage on an NFS server.
+
+Authentication via an existing LDAP or Active Directory server.
+
+* Components
+   2 to 4 application servers with 4 sockets and 32GB RAM.
+   2 DB servers with 4 sockets and 64GB RAM
+   1 HAproxy load balancer with 2 sockets and 16GB RAM.
+   NFS storage server as needed.
+
+* Operating system
+   Enterprise grade Linux distribution with full support from OS vendor. Red 
+   Hat Enterprise Linux or SUSE Linux Enterprise Server 12 are recommended.
+
+* SSL Configuration
+   The SSL termination is done in the HAProxy Load Balancer. A standard SSL 
+   certificate is needed, installed according to the `HAProxy 
+   documentation <http://www.haproxy.org/#docs>`_.)
+
+* Load Balancer
+   HAProxy running on a dedicated server in front of the application servers. 
+   Sticky session needs to be used because of local session management on the 
+   application servers. 
+
+.. comment: (please add configuration details here)   
+.. comment: why sticky sessions? the nice thing about haproxy is that it can 
+   send requests to the application server with the least load. redis or 
+   memcached seem more appropriate. this is mid size already. the software 
+   stack should be the same as for L`_
+   Frank: Yes. But this only works if haproxy can read the http stream which 
+   means that we have to terminate SSL in the haproxy instead of the webserver. 
+   Totally possible. Whatever you prefer :-)
+   Jörn: AFAIK you need to do SSL offloading to do sticky sessions, because the 
+   load balancer has to look into the http stream or rely on the client IP to 
+   determine the web server for the session. Not doing SSL offloading instead 
+   requires you to use a shared session (via memcached or redis) because the 
+   requests are distributed via round robin or least load. It allows you to 
+   scale out the ssl load by adding more applicaton servers. So ... I think it 
+   is exactly the other way round.
+
+* Database
+   MySQL/MariaDB Galera cluster with master-master replication.
+
+* Backup
+   Minimum daily backup without downtime. All MySQL/MariaDB statements should 
+   be replicated to a backup MySQL/MariaDB slave instance.
+   
+    * Create a snapshot on the NFS storage server. 
+    * At the same time stop the MySQL replication.
+    * Create a MySQL dump of the backup slave.
+    * Push the NFS snapshot to the backup.
+    * Push the MySQL dump to the backup.
+    * Delete the NFS snapshot.
+    * Restart MySQL replication.
+
+* Authentication
+   User authentication can be used via one or several LDAP or AD directories. 
+   (See `User Authentication with LDAP`_  for information on configuring 
+   ownCloud to use LDAP and AD.)
+   
+* LDAP 
+   Read-only slaves should be deployed on every application server for 
+   optimal scalability
+
+* Session Management
+   Local Session management on the application server. PHP sessions are stored 
+   in /tmp which is mounted as tmpfs. (please add configuration details here)
+
+* Caching
+   A memcache speeds up server performance, and ownCloud supports four 
+   memcaches; refer to `Configuring Memory Caching`_ for information on 
+   selecting and configuring a memcache.
+   
+* Storage
+   An off-the-shelf NFS solution should be used. Examples are IBM Elastic 
+   Storage or RedHat CEPH.
+   
+* ownCloud Edition
+   Enterprise Edition. (See `ownCloud Server or Enterprise Edition`_ for 
+   comparisons of the ownCloud editions.)
+
+Large Enterprises and Service Providers
+---------------------------------------
+
+* Number of users
+   5,000 to >100,000 users.
+   
+* Storage size
+   Up to 1 petabyte.
+   
+* High availabily level
+   Every component is fully redundant and can fail without service interruption.
+   Backups without service interruption
+   
+.. figure:: images/deprecs-3.png
+   :scale: 60%
+   :alt: Network diagram for large enterprise.   
+
+.. comment: ![Not pretty ... but needs discussion anyway] 
+   (http://yuml.me/cfeebddd)   
+   
+Recommended System Requirements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+4 to 20 application/Webservers.
+
+A cluster of two or more database servers.
+
+Storage is an NFS server, or an object store that is S3 compatible.
+
+Cloud federation for a distributed setup over several data centers.
+
+Authentication via an existing LDAP or Active Directory server, or SAML.
+
+* Components
+   4 to 20 application servers with 4 sockets and 64GB  RAM.
+   4 DB servers with 4 sockets and 128GB RAM
+   2 Hardware load balancer, for example BIG IP from F5
+   NFS storage server as needed.
+
+* Operating system
+   RHEL 7 with latest service packs.
+
+* SSL Configuration
+   The SSL termination is done in the Load Balancer. A standard SSL certificate 
+   is needed, installed according to the Load Balancer documentation. 
+
+* Load Balancer
+   A redundant hardware load-balancer with heartbeat, for example `F5 Big-IP`_. 
+   This runs two load balancers in front of the application servers.
+
+* Database
+   MySQL/MariaDB Galera Cluster with 4x master -- master replication.
+
+* Backup
+   Minimum daily backup without downtime. All MySQL/MariaDB statements should 
+   be replicated to a backup MySQL/MariaDB slave instance.
+   
+    * Create a snapshot on the NFS storage server. 
+    * At the same time stop the MySQL replication.
+    * Create a MySQL dump of the backup slave.
+    * Push the NFS snapshot to the backup.
+    * Push the MySQL dump to the backup.
+    * Delete the NFS snapshot.
+    * Restart MySQL replication.
+    
+* Authentication
+   User authentication via one or several LDAP or Active Directory 
+   servers, or SAML/Shibboleth. (See `User Authentication with LDAP`_ and 
+   `Shibboleth Integration`_.) 
+
+* LDAP
+   Read-only slaves should be deployed on every application server for 
+   optimal scalability.
+
+* Session Management
+   Redis should be use for the session management storage (see `Configuring 
+   Memory Caching`_).
+
+* Caching
+   Redis for distributed in-memory caching (see `Configuring Memory 
+   Caching`_).
+   
+*  Storage
+    An off-the-shelf NFS solution should be used. Examples are IBM Elastic 
+    Storage or RedHAT CEPH. Optionally, an S3 compatible object store can also 
+    be used. 
+
+    .. comment:  (please add more meat here)   
+
+* ownCloud Edition
+   Enterprise Edition. (See `ownCloud Server or Enterprise Edition`_ for 
+   comparisons of the ownCloud editions.)
+   
+Hardware Considerations
+-----------------------
+
+* Solid-state drives (SSDs) for I/O.
+* Separate hard disks for storage and database, SSDs for DBs.
+* Multiple network interfaces to distribute server synchronisation and backend 
+  traffic across multiple subnets.
+
+Single Machine / Scale-Up Deployment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Widely used in the community. Lowest end hardware: RaspberryPI.
+
+Pros:
+
+* Easy setup (no session storage daemon-> use tmpfs for performance, local 
+  storage).
+* No network latency to consider.
+* To scale buy bigger CPU, memory, or hard drive.
+
+Cons:
+
+* No high availability
+* Amount of data in oC tends to never shrink -> sooner rather than later a 
+  single machine will not scale (multiple up- and downloads kill I/O, even with 
+  SSD)
+
+Scale-Out Deployment
+^^^^^^^^^^^^^^^^^^^^
+
+Provider setup:
+
+* DNS round robin to HAProxy servers (2-n, SSL offloading, cache static 
+  resources)
+* Least load to Apache servers (2-n)
+* Memcached/Redis for shared session storage (2-n)
+* Database cluster with single Master, multiple slaves and proxy to split 
+  requests accordingly (2-n)
+* GPFS or Ceph via phprados (2-n, 3 to be safe, Ceph 10+ nodes to see speed 
+  benefits under load)
+
+Pros:
+
+* Components can be scaled as needed.
+* High availability.
+* Test migrations easier.
+
+Cons:
+
+* More complicated to setup.
+* Network becomes the bottleneck (10GB Ethernet recommended).
+* Currently DB filecache table will grow rapidly, making migrations painful in 
+  case the table is altered.
+
+What About Nginx / PHP-FPM?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Could be used instead of HAproxy as the load balancer.
+But on uploads stores the whole file on disk before handing it over to PHP-FPM.
+
+A Single Master DB is Single Point of Failure, Does Not Scale
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When master fails another slave can become master. Multi-master has the risk of 
+split brain and is a more complicated. Can run into deadlocks which oC tries 
+to solve with high level file locking -> here be dragons.
+
+Software Considerations
+-----------------------
+
+Operating System
+^^^^^^^^^^^^^^^^
+
+We are dependent on distributions that offers an easy way to install the 
+various components in an up-to-date version. Debian is loved by administrators 
+for its stability and can integrate recent versions of PHP fairly easy.
+
+That being said, ownCloud has a partnership with RedHat and SUSE for customers 
+who need commercial support.
+
+Webserver
+^^^^^^^^^
+
+Taking Apache and Nginx as the contenders, Apache with mod_php is currently the 
+best option, as Nginx does not support all features necessary for enterprise 
+deployments. Mod_php is recommended instead of PHP_FPM, because in scale-out 
+deployments separate PHP pools are simply not necessary.
+
+
+.. comment: Nginx currently does not integrate with Shibboleth, which prevents 
+   SSO. Nevertheless, the Shibboleth community seems to be investigating how to 
+   integrate with Nginx.
+
+.. comment:  Nginx stores uploaded files on disk before handing them to php-fpm 
+   which is a performance problem with GB-sized files. There seems to be an 
+   Nginx fork from China that handles that better.
+
+Relational Database
+^^^^^^^^^^^^^^^^^^^
+
+More often than not the customer already has an opinion on what database to 
+use. In general, the recommendation is to use what their database administrator 
+is most familiar with. Taking into account what GCX is seeing at customer 
+deployments, we recommend MySQL/MariaDB in a master-slave deployment with a 
+MySQL proxy in front of them to send updates to master, and selects to the 
+slave(s).
+
+.. comment: MySQL locks tables for schema updates and might even have to copy 
+   the whole table. That is pretty much a non-starter for migrations unless you 
+   are using a scale out deployment where you can apply the schema changes to 
+   each slave individually. Even the each migration might take several hours. 
+   Make sure you have enough disk space. You have been warned.
+
+.. comment: Currently, ownCloud uses the utf8 character set with utf8_bin 
+   collation on MySQL installations. As a result 4 byte UTF characters like 
+   emojis cannot be used. This can be fixed by [moving to 
+   utf8mb4/utf8mb4_bin](https://github.com/owncloud/core/issues/7030).
+
+The second best option is PostgreSQL (alter table does not lock table - makes 
+migration less painful) although we have yet to find a customer who uses a 
+master-slave setup.
+
+.. comment: PostgreSQL may produce excessive amounts of dead tuples due to 
+   owncloud transactions preventing the execution of the autovacum process.
+
+What about the other DBMS?
+
+* Sqlite is adequate for simple testing, and for low-load single-user 
+  deployments. It is not adequate for production systems.
+* MSSQL is not automatically tested.
+* Oracle is a pain, but the de facto standard at huge enterprises. Developers 
+  need to be aware of the 30 char identifier limit, empty string equals null 
+  and varchar2 can only be made 4000 chars wide.
+
+File Storage
+------------
+
+This is what separates us from WordPress or typo3. Our main use case is up- and 
+download of files. Sooner or later that requires scale-out storage. Currently, 
+the options are GPFS or an object store like Ceph/s3 or Openstack/Swift. GPFS 
+is expensive, and our s3 and Swift implementations use temp files which 
+prevents them from scaling adequately.
+
+.. comment: A proof of concept implementation based on 
+   [phprados](https://github.com/ceph/phprados) that talks directly to a 
+   [ceph](http://ceph.com/) cluster without having to use temp files is [in 
+   development](https://github.com/owncloud/objectstore/pull/26).
+
+.. comment: NFS can be used but needs to be micro-managed to distribute users 
+   on multiple storages. If you want to go that route configure ldap to provide 
+   a custom home folder location. That allows you to move each users data 
+   folder to different nfs mounts.
+
+Session Storage
+---------------
+
+* Redis (persistent, nice graphical inspection tools available, ownCloud high 
+  level locking supported)
+* If Shibboleth is a requirement you must use Memcached, as it can also be used 
+  to scale-out shibd session storage (see `Memcache StorageService`_).
+
+.. comment: High Availability / Failover deployment
+   Use Case: site replication -> different problem
+
+References
+----------
+
+`Database High Availability`_
+   
+`Performance enhancements for Apache and PHP`_   
+
+.. _Maintenance: 
+   https://doc.owncloud.org/server/9.0/admin_manual/maintenance/index.html
+.. _User Authentication with LDAP:
+   https://doc.owncloud.org/server/9.0/admin_manual/configuration_user/    
+   user_auth_ldap.html
+.. _Configuring Memory Caching:   
+   https://doc.owncloud.org/server/9.0/admin_manual/configuration_server/ 
+   caching_configuration.html
+.. _ownCloud Server or Enterprise Edition:  
+   https://owncloud.com/owncloud-server-or-enterprise-edition/
+.. _F5 Big-IP: https://f5.com/products/big-ip/
+
+.. _Shibboleth Integration: 
+   https://doc.owncloud.org/server/9.0/admin_manual/enterprise_user_management/
+   user_auth_shibboleth.html
+.. _Memcache StorageService:  
+   https://wiki.shibboleth.net/confluence/display/SHIB2/
+   NativeSPStorageService#NativeSPStorageService-MemcacheStorageService
+.. _Database High Availability: 
+   http://www.severalnines.com/blog/become-mysql-dba-blog-series-database-high-
+   availability
+.. _Performance enhacements for Apache and PHP: 
+   http://blog.bitnami.com/2014/06/performance-enhacements-for-apache-and.html  

admin_manual/deployment_recommendations/index.rst

@@ -0,0 +1,8 @@
+===================================
+ownCloud Deployment Recommendations
+===================================
+
+.. toctree::
+    :maxdepth: 2
+
+    deployment_recommendations