Changeset View
Changeset View
Standalone View
Standalone View
src/docs/user/cluster/cluster.diviner
@title Clustering Introduction | @title Clustering Introduction | ||||
@group cluster | @group cluster | ||||
Guide to configuring Phabricator across multiple hosts for availability and | Guide to configuring Phorge across multiple hosts for availability and | ||||
performance. | performance. | ||||
Overview | Overview | ||||
======== | ======== | ||||
WARNING: This feature is a prototype. Installs should expect a challenging | WARNING: This feature is a prototype. Installs should expect a challenging | ||||
adventure when deploying clusters. In the best of times, configuring a | adventure when deploying clusters. In the best of times, configuring a | ||||
cluster is complex and requires significant operations experience. | cluster is complex and requires significant operations experience. | ||||
Phabricator can be configured to run on multiple hosts with redundant services | Phorge can be configured to run on multiple hosts with redundant services | ||||
to improve its availability and scalability, and make disaster recovery much | to improve its availability and scalability, and make disaster recovery much | ||||
easier. | easier. | ||||
Clustering is more complex to setup and maintain than running everything on a | Clustering is more complex to setup and maintain than running everything on a | ||||
single host, but greatly reduces the cost of recovering from hardware and | single host, but greatly reduces the cost of recovering from hardware and | ||||
network failures. | network failures. | ||||
Each Phabricator service has an array of clustering options that can be | Each Phorge service has an array of clustering options that can be | ||||
configured somewhat independently. Configuring a cluster is inherently complex, | configured somewhat independently. Configuring a cluster is inherently complex, | ||||
and this is an advanced feature aimed at installs with large userbases and | and this is an advanced feature aimed at installs with large userbases and | ||||
experienced operations personnel who need this high degree of flexibility. | experienced operations personnel who need this high degree of flexibility. | ||||
The remainder of this document summarizes how to add redundancy to each | The remainder of this document summarizes how to add redundancy to each | ||||
service and where your efforts are likely to have the greatest impact. | service and where your efforts are likely to have the greatest impact. | ||||
For additional guidance on setting up a cluster, see "Overlaying Services" | For additional guidance on setting up a cluster, see "Overlaying Services" | ||||
Show All 18 Lines | |||||
| **Fulltext Search** | Minimal | Low | No Risk | Low | | **Fulltext Search** | Minimal | Low | No Risk | Low | ||||
See below for a walkthrough of these services in greater detail. | See below for a walkthrough of these services in greater detail. | ||||
Preparing for Clustering | Preparing for Clustering | ||||
======================== | ======================== | ||||
To begin deploying Phabricator in cluster mode, set up `cluster.addresses` | To begin deploying Phorge in cluster mode, set up `cluster.addresses` | ||||
in your configuration. | in your configuration. | ||||
This option should contain a list of network address blocks which are considered | This option should contain a list of network address blocks which are considered | ||||
to be part of the cluster. Hosts in this list are allowed to bend (or even | to be part of the cluster. Hosts in this list are allowed to bend (or even | ||||
break) some of the security and policy rules when they make requests to other | break) some of the security and policy rules when they make requests to other | ||||
hosts in the cluster, so this list should be as small as possible. See "Cluster | hosts in the cluster, so this list should be as small as possible. See "Cluster | ||||
Whitelist Security" below for discussion. | Whitelist Security" below for discussion. | ||||
If you are deploying hardware in EC2, a reasonable approach is to launch a | If you are deploying hardware in EC2, a reasonable approach is to launch a | ||||
dedicated Phabricator VPC, whitelist the whole VPC as a Phabricator cluster, | dedicated Phorge VPC, whitelist the whole VPC as a Phorge cluster, | ||||
and then deploy only Phabricator services into that VPC. | and then deploy only Phorge services into that VPC. | ||||
If you have additional auxiliary hosts which run builds and tests via Drydock, | If you have additional auxiliary hosts which run builds and tests via Drydock, | ||||
you should //not// include them in the cluster address definition. For more | you should //not// include them in the cluster address definition. For more | ||||
detailed discussion of the Drydock security model, see | detailed discussion of the Drydock security model, see | ||||
@{article:Drydock User Guide: Security}. | @{article:Drydock User Guide: Security}. | ||||
Most other clustering features will not work until you define a cluster by | Most other clustering features will not work until you define a cluster by | ||||
configuring `cluster.addresses`. | configuring `cluster.addresses`. | ||||
Cluster Whitelist Security | Cluster Whitelist Security | ||||
======================== | ======================== | ||||
When you configure `cluster.addresses`, you should keep the list of trusted | When you configure `cluster.addresses`, you should keep the list of trusted | ||||
cluster hosts as small as possible. Hosts on this list gain additional | cluster hosts as small as possible. Hosts on this list gain additional | ||||
capabilities, including these: | capabilities, including these: | ||||
**Trusted HTTP Headers**: Normally, Phabricator distrusts the load balancer | **Trusted HTTP Headers**: Normally, Phorge distrusts the load balancer | ||||
HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be | HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be | ||||
client-controlled and can be set to arbitrary values by an attacker if no load | client-controlled and can be set to arbitrary values by an attacker if no load | ||||
balancer is deployed. In particular, clients can set `X-Forwarded-For` to any | balancer is deployed. In particular, clients can set `X-Forwarded-For` to any | ||||
value and spoof traffic from arbitrary remotes. | value and spoof traffic from arbitrary remotes. | ||||
These headers are trusted when they are received from a host on the cluster | These headers are trusted when they are received from a host on the cluster | ||||
address whitelist. This allows requests from cluster loadbalancers to be | address whitelist. This allows requests from cluster loadbalancers to be | ||||
interpreted correctly by default without requiring additional custom code or | interpreted correctly by default without requiring additional custom code or | ||||
Show All 33 Lines | |||||
Cluster: Databases | Cluster: Databases | ||||
================= | ================= | ||||
Configuring multiple database hosts is moderately complex, but normally has the | Configuring multiple database hosts is moderately complex, but normally has the | ||||
highest impact on availability and resistance to data loss. This is usually the | highest impact on availability and resistance to data loss. This is usually the | ||||
most important service to make redundant if your focus is on availability and | most important service to make redundant if your focus is on availability and | ||||
disaster recovery. | disaster recovery. | ||||
Configuring replicas allows Phabricator to run in read-only mode if you lose | Configuring replicas allows Phorge to run in read-only mode if you lose | ||||
the master and to quickly promote the replica as a replacement. | the master and to quickly promote the replica as a replacement. | ||||
For details, see @{article:Cluster: Databases}. | For details, see @{article:Cluster: Databases}. | ||||
Cluster: Repositories | Cluster: Repositories | ||||
===================== | ===================== | ||||
Configuring multiple repository hosts is complex, but is required before you | Configuring multiple repository hosts is complex, but is required before you | ||||
can add multiple daemon or web hosts. | can add multiple daemon or web hosts. | ||||
Repository replicas are important for availability if you host repositories | Repository replicas are important for availability if you host repositories | ||||
on Phabricator, but less important if you host repositories elsewhere | on Phorge, but less important if you host repositories elsewhere | ||||
(instead, you should focus on making that service more available). | (instead, you should focus on making that service more available). | ||||
The distributed nature of Git and Mercurial tend to mean that they are | The distributed nature of Git and Mercurial tend to mean that they are | ||||
naturally somewhat resistant to data loss: every clone of a repository includes | naturally somewhat resistant to data loss: every clone of a repository includes | ||||
the entire history. | the entire history. | ||||
Repositories may become a scalability bottleneck, although this is rare unless | Repositories may become a scalability bottleneck, although this is rare unless | ||||
your install has an unusually heavy repository read volume. Slow clones/fetches | your install has an unusually heavy repository read volume. Slow clones/fetches | ||||
▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Lines | |||||
For details, see @{article:Cluster: Notifications}. | For details, see @{article:Cluster: Notifications}. | ||||
Cluster: Fulltext Search | Cluster: Fulltext Search | ||||
======================== | ======================== | ||||
Configuring search services is relatively simple and has no pre-requisites. | Configuring search services is relatively simple and has no pre-requisites. | ||||
By default, Phabricator uses MySQL as a fulltext search engine, so deploying | By default, Phorge uses MySQL as a fulltext search engine, so deploying | ||||
multiple database hosts will effectively also deploy multiple fulltext search | multiple database hosts will effectively also deploy multiple fulltext search | ||||
hosts. | hosts. | ||||
Search indexes can be completely rebuilt from the database, so there is no | Search indexes can be completely rebuilt from the database, so there is no | ||||
risk of data loss no matter how fulltext search is configured. | risk of data loss no matter how fulltext search is configured. | ||||
For details, see @{article:Cluster: Search}. | For details, see @{article:Cluster: Search}. | ||||
Overlaying Services | Overlaying Services | ||||
=================== | =================== | ||||
Although hosts can run a single dedicated service type, certain groups of | Although hosts can run a single dedicated service type, certain groups of | ||||
services work well together. Phabricator clusters usually do not need to be | services work well together. Phorge clusters usually do not need to be | ||||
very large, so deploying a small number of hosts with multiple services is a | very large, so deploying a small number of hosts with multiple services is a | ||||
good place to start. | good place to start. | ||||
In planning a cluster, consider these blended host types: | In planning a cluster, consider these blended host types: | ||||
**Everything**: Run HTTP, SSH, MySQL, notifications, repositories and daemons | **Everything**: Run HTTP, SSH, MySQL, notifications, repositories and daemons | ||||
on a single host. This is the starting point for single-node setups, and | on a single host. This is the starting point for single-node setups, and | ||||
usually also the best configuration when adding the second node. | usually also the best configuration when adding the second node. | ||||
▲ Show 20 Lines • Show All 74 Lines • Show Last 20 Lines |
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under Apache 2.0 or other open source licenses. · CC BY-SA 4.0 · Apache 2.0