Page MenuHomePhorge
Differential D25007 Diff 23 src/docs/user/cluster/cluster_partitioning.diviner

Changeset View

Standalone View

View Options

src/docs/user/cluster/cluster_partitioning.diviner

@title Cluster: Partitioning and Advanced Configuration @title Cluster: Partitioning and Advanced Configuration
@group cluster @group cluster
Guide to partitioning Phabricator applications across multiple database hosts. Guide to partitioning Phorge applications across multiple database hosts.
Overview Overview
======== ========
You can partition Phabricator's applications across multiple databases. For You can partition Phorge's applications across multiple databases. For
example, you can move an application like Files or Maniphest to a dedicated example, you can move an application like Files or Maniphest to a dedicated
database host. database host.
The advantages of doing this are: The advantages of doing this are:
- moving heavily used applications to dedicated hardware can help you - moving heavily used applications to dedicated hardware can help you
scale; and scale; and
- you can match application workloads to hardware or configuration to make - you can match application workloads to hardware or configuration to make
operating the cluster easier. operating the cluster easier.
This configuration is complex, and very few installs will benefit from pursuing This configuration is complex, and very few installs will benefit from pursuing
it. Phabricator will normally run comfortably with a single database master it. Phorge will normally run comfortably with a single database master
even for large organizations. even for large organizations.
Partitioning generally does not do much to increase resilience or make it Partitioning generally does not do much to increase resilience or make it
easier to recover from disasters, and is primarily a mechanism for scaling and easier to recover from disasters, and is primarily a mechanism for scaling and
operational convenience. operational convenience.
If you are considering partitioning, you likely want to configure replication If you are considering partitioning, you likely want to configure replication
with a single master first. Even if you choose not to deploy replication, you with a single master first. Even if you choose not to deploy replication, you
should review and understand how replication works before you partition. For should review and understand how replication works before you partition. For
details, see @{article:Cluster: Databases}. details, see @{article:Cluster: Databases}.
Databases also support some advanced configuration options. Briefly: Databases also support some advanced configuration options. Briefly:
- `persistent`: Allows use of persistent connections, reducing pressure on - `persistent`: Allows use of persistent connections, reducing pressure on
outbound ports. outbound ports.
See "Advanced Configuration", below, for additional discussion. See "Advanced Configuration", below, for additional discussion.
What Partitioning Does What Partitioning Does
====================== ======================
When you partition Phabricator, you move all of the data for one or more When you partition Phorge, you move all of the data for one or more
applications (like Maniphest) to a new master database host. This is possible applications (like Maniphest) to a new master database host. This is possible
because Phabricator stores data for each application in its own logical because Phorge stores data for each application in its own logical
database (like `phabricator_maniphest`) and performs no joins between databases. database (like `phorge_maniphest`) and performs no joins between databases.
If you're running into scale limits on a single master database, you can move If you're running into scale limits on a single master database, you can move
one or more of your most commonly-used applications to a second database host one or more of your most commonly-used applications to a second database host
and continue adding users. You can keep partitioning applications until all and continue adding users. You can keep partitioning applications until all
heavily used applications have dedicated database servers. heavily used applications have dedicated database servers.
Alternatively or additionally, you can partition applications to make operating Alternatively or additionally, you can partition applications to make operating
the cluster easier. Some applications have unusual workloads or requirements, the cluster easier. Some applications have unusual workloads or requirements,
Show All 17 Lines
each `replica` database follows. Here's a simple example config: each `replica` database follows. Here's a simple example config:
```lang=json ```lang=json
... ...
"cluster.databases": [ "cluster.databases": [
{ {
"host": "db001.corporation.com", "host": "db001.corporation.com",
"role": "master", "role": "master",
"user": "phabricator", "user": "phorge",
"pass": "hunter2!trustno1", "pass": "hunter2!trustno1",
"port": 3306, "port": 3306,
"partition": [ "partition": [
"default" "default"
] ]
}, },
{ {
"host": "db002.corporation.com", "host": "db002.corporation.com",
"role": "replica", "role": "replica",
"user": "phabricator", "user": "phorge",
"pass": "hunter2!trustno1", "pass": "hunter2!trustno1",
"port": 3306, "port": 3306,
"master": "db001.corporation.com:3306" "master": "db001.corporation.com:3306"
}, },
{ {
"host": "db003.corporation.com", "host": "db003.corporation.com",
"role": "master", "role": "master",
"user": "phabricator", "user": "phorge",
"pass": "hunter2!trustno1", "pass": "hunter2!trustno1",
"port": 3306, "port": 3306,
"partition": [ "partition": [
"file", "file",
"passphrase", "passphrase",
"slowvote" "slowvote"
] ]
}, },
{ {
"host": "db004.corporation.com", "host": "db004.corporation.com",
"role": "replica", "role": "replica",
"user": "phabricator", "user": "phorge",
"pass": "hunter2!trustno1", "pass": "hunter2!trustno1",
"port": 3306, "port": 3306,
"master": "db003.corporation.com:3306" "master": "db003.corporation.com:3306"
} }
], ],
... ...
``` ```
Show All 12 Lines
After you have configured partitioning, it needs to be committed to the After you have configured partitioning, it needs to be committed to the
databases. This writes a copy of the configuration to tables on the databases, databases. This writes a copy of the configuration to tables on the databases,
preventing errors if a webserver accidentally starts with an old or invalid preventing errors if a webserver accidentally starts with an old or invalid
configuration. configuration.
To commit the configuration, run this command: To commit the configuration, run this command:
``` ```
phabricator/ $ ./bin/storage partition phorge/ $ ./bin/storage partition
``` ```
Run this command after making any partition or clustering changes. Webservers Run this command after making any partition or clustering changes. Webservers
will not serve traffic if their configuration and the database configuration will not serve traffic if their configuration and the database configuration
differ. differ.
Launching a new Partition Launching a new Partition
========================= =========================
To add a new partition, follow these steps: To add a new partition, follow these steps:
- Set up the new database host or hosts. - Set up the new database host or hosts.
- Add the new database to `cluster.databases`, but keep its "partition" - Add the new database to `cluster.databases`, but keep its "partition"
configuration empty (just an empty list). If this is the first time you configuration empty (just an empty list). If this is the first time you
are partitioning, you will need to configure your existing master as the are partitioning, you will need to configure your existing master as the
new "default". This will let Phabricator interact with it, but won't send new "default". This will let Phorge interact with it, but won't send
any traffic to it yet. any traffic to it yet.
- Run `bin/storage partition`. - Run `bin/storage partition`.
- Run `bin/storage upgrade` to initialize the schemata on the new hosts. - Run `bin/storage upgrade` to initialize the schemata on the new hosts.
- Stop writes to the applications you want to move by putting Phabricator - Stop writes to the applications you want to move by putting Phorge
in read-only mode, or shutting down the webserver and daemons, or telling in read-only mode, or shutting down the webserver and daemons, or telling
everyone not to touch anything. everyone not to touch anything.
- Dump the data from the application databases on the old master. - Dump the data from the application databases on the old master.
- Load the data into the application databases on the new master. - Load the data into the application databases on the new master.
- Reconfigure the "partition" setup so that Phabricator knows the databases - Reconfigure the "partition" setup so that Phorge knows the databases
have moved. have moved.
- Run `bin/storage partition`. - Run `bin/storage partition`.
- While still in read-only mode, check that all the data appears to be - While still in read-only mode, check that all the data appears to be
intact. intact.
- Resume writes. - Resume writes.
You can do this with a small, rarely-used application first (on most installs, You can do this with a small, rarely-used application first (on most installs,
Slowvote might be a good candidate) if you want to run through the process Slowvote might be a good candidate) if you want to run through the process
end-to-end before performing a larger, higher-stakes migration. end-to-end before performing a larger, higher-stakes migration.
How Partitioning Works How Partitioning Works
====================== ======================
If you have multiple masters, Phabricator keeps the entire set of schemata up If you have multiple masters, Phorge keeps the entire set of schemata up
to date on all of them. When you run `bin/storage upgrade` or other storage to date on all of them. When you run `bin/storage upgrade` or other storage
management commands, they generally affect all masters (if they do not, they management commands, they generally affect all masters (if they do not, they
will prompt you to be more specific). will prompt you to be more specific).
When the application goes to read or write normal data (for example, to query a When the application goes to read or write normal data (for example, to query a
list of tasks) it only connects to the master which the application it is list of tasks) it only connects to the master which the application it is
acting on behalf of is assigned to. acting on behalf of is assigned to.
In most cases, a masters will not have any data in most the databases which are In most cases, a masters will not have any data in most the databases which are
not assigned to it. If they do (for example, because they previously hosted the not assigned to it. If they do (for example, because they previously hosted the
application) the data is ignored. This approach (of maintaining all schemata on application) the data is ignored. This approach (of maintaining all schemata on
all hosts) makes it easier to move data and to quickly revert changes if a all hosts) makes it easier to move data and to quickly revert changes if a
configuration mistake occurs. configuration mistake occurs.
There are some exceptions to this rule. For example, all masters keep track There are some exceptions to this rule. For example, all masters keep track
of which patches have been applied to that particular master so that of which patches have been applied to that particular master so that
`bin/storage upgrade` can upgrade hosts correctly. `bin/storage upgrade` can upgrade hosts correctly.
Phabricator does not perform joins across logical databases, so there are no Phorge does not perform joins across logical databases, so there are no
meaningful differences in runtime behavior if two applications are on the same meaningful differences in runtime behavior if two applications are on the same
physical host or different physical hosts. physical host or different physical hosts.
Advanced Configuration Advanced Configuration
====================== ======================
Separate from partitioning, some advanced configuration is supported. These Separate from partitioning, some advanced configuration is supported. These
options must be set on database specifications in `cluster.databases`. You can options must be set on database specifications in `cluster.databases`. You can
configure them without actually building a cluster by defining a cluster with configure them without actually building a cluster by defining a cluster with
only one master. only one master.
`persistent` //(bool)// Enables persistent connections. Defaults to off. `persistent` //(bool)// Enables persistent connections. Defaults to off.
With persistent connections enabled, Phabricator will keep a pool of database With persistent connections enabled, Phorge will keep a pool of database
connections open between web requests and reuse them when serving subsequent connections open between web requests and reuse them when serving subsequent
requests. requests.
The primary benefit of using persistent connections is that it will greatly The primary benefit of using persistent connections is that it will greatly
reduce pressure on how quickly outbound TCP ports are opened and closed. After reduce pressure on how quickly outbound TCP ports are opened and closed. After
a TCP port closes, it normally can't be used again for about 60 seconds, so a TCP port closes, it normally can't be used again for about 60 seconds, so
rapidly cycling ports can cause resource exhaustion. If you're seeing failures rapidly cycling ports can cause resource exhaustion. If you're seeing failures
because requests are unable to bind to an outbound port, enabling this option because requests are unable to bind to an outbound port, enabling this option
is likely to fix the issue. This option may also slightly increase performance. is likely to fix the issue. This option may also slightly increase performance.
The cost of using persistent connections is that you may need to raise the The cost of using persistent connections is that you may need to raise the
MySQL `max_connections` setting: although Phabricator will make far fewer MySQL `max_connections` setting: although Phorge will make far fewer
connections, the connections it does make will be longer-lived. Raising this connections, the connections it does make will be longer-lived. Raising this
setting will increase MySQL memory requirements and may run into other limits, setting will increase MySQL memory requirements and may run into other limits,
like `open_files_limit`, which may also need to be raised. like `open_files_limit`, which may also need to be raised.
Persistent connections are enabled per-database. If you always want to use Persistent connections are enabled per-database. If you always want to use
them, set the flag on each configured database in `cluster.databases`. them, set the flag on each configured database in `cluster.databases`.
Next Steps Next Steps
========== ==========
Continue by: Continue by:
- returning to @{article:Clustering Introduction}. - returning to @{article:Clustering Introduction}.
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under Apache 2.0 or other open source licenses. · CC BY-SA 4.0 · Apache 2.0