Differential D25007 Diff 19 src/docs/user/cluster/cluster.diviner

Changeset View

Standalone View

src/docs/user/cluster/cluster.diviner

	@title Clustering Introduction			@title Clustering Introduction
	@group cluster			@group cluster

	Guide to configuring Phabricator across multiple hosts for availability and			Guide to configuring Phorge across multiple hosts for availability and
	performance.			performance.


	Overview			Overview
	========			========

	WARNING: This feature is a prototype. Installs should expect a challenging			WARNING: This feature is a prototype. Installs should expect a challenging
	adventure when deploying clusters. In the best of times, configuring a			adventure when deploying clusters. In the best of times, configuring a
	cluster is complex and requires significant operations experience.			cluster is complex and requires significant operations experience.

	Phabricator can be configured to run on multiple hosts with redundant services			Phorge can be configured to run on multiple hosts with redundant services
	to improve its availability and scalability, and make disaster recovery much			to improve its availability and scalability, and make disaster recovery much
	easier.			easier.

	Clustering is more complex to setup and maintain than running everything on a			Clustering is more complex to setup and maintain than running everything on a
	single host, but greatly reduces the cost of recovering from hardware and			single host, but greatly reduces the cost of recovering from hardware and
	network failures.			network failures.

	Each Phabricator service has an array of clustering options that can be			Each Phorge service has an array of clustering options that can be
	configured somewhat independently. Configuring a cluster is inherently complex,			configured somewhat independently. Configuring a cluster is inherently complex,
	and this is an advanced feature aimed at installs with large userbases and			and this is an advanced feature aimed at installs with large userbases and
	experienced operations personnel who need this high degree of flexibility.			experienced operations personnel who need this high degree of flexibility.

	The remainder of this document summarizes how to add redundancy to each			The remainder of this document summarizes how to add redundancy to each
	service and where your efforts are likely to have the greatest impact.			service and where your efforts are likely to have the greatest impact.

	For additional guidance on setting up a cluster, see "Overlaying Services"			For additional guidance on setting up a cluster, see "Overlaying Services"
	Show All 18 Lines
	\| Fulltext Search \| Minimal \| Low \| No Risk \| Low			\| Fulltext Search \| Minimal \| Low \| No Risk \| Low

	See below for a walkthrough of these services in greater detail.			See below for a walkthrough of these services in greater detail.


	Preparing for Clustering			Preparing for Clustering
	========================			========================

	To begin deploying Phabricator in cluster mode, set up `cluster.addresses`			To begin deploying Phorge in cluster mode, set up `cluster.addresses`
	in your configuration.			in your configuration.

	This option should contain a list of network address blocks which are considered			This option should contain a list of network address blocks which are considered
	to be part of the cluster. Hosts in this list are allowed to bend (or even			to be part of the cluster. Hosts in this list are allowed to bend (or even
	break) some of the security and policy rules when they make requests to other			break) some of the security and policy rules when they make requests to other
	hosts in the cluster, so this list should be as small as possible. See "Cluster			hosts in the cluster, so this list should be as small as possible. See "Cluster
	Whitelist Security" below for discussion.			Whitelist Security" below for discussion.

	If you are deploying hardware in EC2, a reasonable approach is to launch a			If you are deploying hardware in EC2, a reasonable approach is to launch a
	dedicated Phabricator VPC, whitelist the whole VPC as a Phabricator cluster,			dedicated Phorge VPC, whitelist the whole VPC as a Phorge cluster,
	and then deploy only Phabricator services into that VPC.			and then deploy only Phorge services into that VPC.

	If you have additional auxiliary hosts which run builds and tests via Drydock,			If you have additional auxiliary hosts which run builds and tests via Drydock,
	you should //not// include them in the cluster address definition. For more			you should //not// include them in the cluster address definition. For more
	detailed discussion of the Drydock security model, see			detailed discussion of the Drydock security model, see
	@{article:Drydock User Guide: Security}.			@{article:Drydock User Guide: Security}.

	Most other clustering features will not work until you define a cluster by			Most other clustering features will not work until you define a cluster by
	configuring `cluster.addresses`.			configuring `cluster.addresses`.


	Cluster Whitelist Security			Cluster Whitelist Security
	========================			========================

	When you configure `cluster.addresses`, you should keep the list of trusted			When you configure `cluster.addresses`, you should keep the list of trusted
	cluster hosts as small as possible. Hosts on this list gain additional			cluster hosts as small as possible. Hosts on this list gain additional
	capabilities, including these:			capabilities, including these:

	Trusted HTTP Headers: Normally, Phabricator distrusts the load balancer			Trusted HTTP Headers: Normally, Phorge distrusts the load balancer
	HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be			HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be
	client-controlled and can be set to arbitrary values by an attacker if no load			client-controlled and can be set to arbitrary values by an attacker if no load
	balancer is deployed. In particular, clients can set `X-Forwarded-For` to any			balancer is deployed. In particular, clients can set `X-Forwarded-For` to any
	value and spoof traffic from arbitrary remotes.			value and spoof traffic from arbitrary remotes.

	These headers are trusted when they are received from a host on the cluster			These headers are trusted when they are received from a host on the cluster
	address whitelist. This allows requests from cluster loadbalancers to be			address whitelist. This allows requests from cluster loadbalancers to be
	interpreted correctly by default without requiring additional custom code or			interpreted correctly by default without requiring additional custom code or
	Show All 33 Lines
	Cluster: Databases			Cluster: Databases
	=================			=================

	Configuring multiple database hosts is moderately complex, but normally has the			Configuring multiple database hosts is moderately complex, but normally has the
	highest impact on availability and resistance to data loss. This is usually the			highest impact on availability and resistance to data loss. This is usually the
	most important service to make redundant if your focus is on availability and			most important service to make redundant if your focus is on availability and
	disaster recovery.			disaster recovery.

	Configuring replicas allows Phabricator to run in read-only mode if you lose			Configuring replicas allows Phorge to run in read-only mode if you lose
	the master and to quickly promote the replica as a replacement.			the master and to quickly promote the replica as a replacement.

	For details, see @{article:Cluster: Databases}.			For details, see @{article:Cluster: Databases}.


	Cluster: Repositories			Cluster: Repositories
	=====================			=====================

	Configuring multiple repository hosts is complex, but is required before you			Configuring multiple repository hosts is complex, but is required before you
	can add multiple daemon or web hosts.			can add multiple daemon or web hosts.

	Repository replicas are important for availability if you host repositories			Repository replicas are important for availability if you host repositories
	on Phabricator, but less important if you host repositories elsewhere			on Phorge, but less important if you host repositories elsewhere
	(instead, you should focus on making that service more available).			(instead, you should focus on making that service more available).

	The distributed nature of Git and Mercurial tend to mean that they are			The distributed nature of Git and Mercurial tend to mean that they are
	naturally somewhat resistant to data loss: every clone of a repository includes			naturally somewhat resistant to data loss: every clone of a repository includes
	the entire history.			the entire history.

	Repositories may become a scalability bottleneck, although this is rare unless			Repositories may become a scalability bottleneck, although this is rare unless
	your install has an unusually heavy repository read volume. Slow clones/fetches			your install has an unusually heavy repository read volume. Slow clones/fetches
	▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Lines
	For details, see @{article:Cluster: Notifications}.			For details, see @{article:Cluster: Notifications}.


	Cluster: Fulltext Search			Cluster: Fulltext Search
	========================			========================

	Configuring search services is relatively simple and has no pre-requisites.			Configuring search services is relatively simple and has no pre-requisites.

	By default, Phabricator uses MySQL as a fulltext search engine, so deploying			By default, Phorge uses MySQL as a fulltext search engine, so deploying
	multiple database hosts will effectively also deploy multiple fulltext search			multiple database hosts will effectively also deploy multiple fulltext search
	hosts.			hosts.

	Search indexes can be completely rebuilt from the database, so there is no			Search indexes can be completely rebuilt from the database, so there is no
	risk of data loss no matter how fulltext search is configured.			risk of data loss no matter how fulltext search is configured.

	For details, see @{article:Cluster: Search}.			For details, see @{article:Cluster: Search}.


	Overlaying Services			Overlaying Services
	===================			===================

	Although hosts can run a single dedicated service type, certain groups of			Although hosts can run a single dedicated service type, certain groups of
	services work well together. Phabricator clusters usually do not need to be			services work well together. Phorge clusters usually do not need to be
	very large, so deploying a small number of hosts with multiple services is a			very large, so deploying a small number of hosts with multiple services is a
	good place to start.			good place to start.

	In planning a cluster, consider these blended host types:			In planning a cluster, consider these blended host types:

	Everything: Run HTTP, SSH, MySQL, notifications, repositories and daemons			Everything: Run HTTP, SSH, MySQL, notifications, repositories and daemons
	on a single host. This is the starting point for single-node setups, and			on a single host. This is the starting point for single-node setups, and
	usually also the best configuration when adding the second node.			usually also the best configuration when adding the second node.
	▲ Show 20 Lines • Show All 74 Lines • Show Last 20 Lines