Differential D25007 Diff 23 src/docs/user/configuration/configuring_file_storage.diviner

Changeset View

Standalone View

src/docs/user/configuration/configuring_file_storage.diviner

@title Configuring File Storage		@title Configuring File Storage
@group config		@group config

Setup file storage and support for large files.		Setup file storage and support for large files.

Overview		Overview
========		========

This document describes how to configure Phabricator to support large file		This document describes how to configure Phorge to support large file
uploads, and how to choose where Phabricator stores files.		uploads, and how to choose where Phorge stores files.

There are two major things to configure:		There are two major things to configure:

- set up PHP and your HTTP server to accept large requests;		- set up PHP and your HTTP server to accept large requests;
- choose and configure a storage engine.		- choose and configure a storage engine.

The following sections will guide you through this configuration.		The following sections will guide you through this configuration.


How Phabricator Stores Files		How Phorge Stores Files
============================		============================

Phabricator stores files in "storage engines", which are modular backends		Phorge stores files in "storage engines", which are modular backends
that implement access to some storage system (like MySQL, the filesystem, or		that implement access to some storage system (like MySQL, the filesystem, or
a cloud storage service like Amazon S3).		a cloud storage service like Amazon S3).

Phabricator stores large files by breaking them up into many chunks (a few		Phorge stores large files by breaking them up into many chunks (a few
megabytes in size) and storing the chunks in an underlying storage engine.		megabytes in size) and storing the chunks in an underlying storage engine.
This makes it easier to implement new storage engines and gives Phabricator		This makes it easier to implement new storage engines and gives Phorge
more flexibility in managing file data.		more flexibility in managing file data.

The first section of this document discusses configuring your install so that		The first section of this document discusses configuring your install so that
PHP and your HTTP server will accept requests which are larger than the size of		PHP and your HTTP server will accept requests which are larger than the size of
one file chunk. Without this configuration, file chunk data will be rejected.		one file chunk. Without this configuration, file chunk data will be rejected.

The second section discusses choosing and configuring storage engines, so data		The second section discusses choosing and configuring storage engines, so data
is stored where you want it to be.		is stored where you want it to be.


Configuring Upload Limits		Configuring Upload Limits
=========================		=========================

File uploads are limited by several pieces of configuration at different layers		File uploads are limited by several pieces of configuration at different layers
of the stack. Generally, the minimum value of all the limits is the effective		of the stack. Generally, the minimum value of all the limits is the effective
one.		one.

To upload large files, you need to increase all the limits to at least		To upload large files, you need to increase all the limits to at least
32MB. This will allow you to upload file chunks, which will let Phabricator		32MB. This will allow you to upload file chunks, which will let Phorge
store arbitrarily large files.		store arbitrarily large files.

The settings which limit file uploads are:		The settings which limit file uploads are:

HTTP Server: The HTTP server may set a limit on the maximum request size.		HTTP Server: The HTTP server may set a limit on the maximum request size.
If you exceed this limit, you'll see a default server page with an HTTP error.		If you exceed this limit, you'll see a default server page with an HTTP error.
These directives limit the total size of the request body, so they must be		These directives limit the total size of the request body, so they must be
somewhat larger than the desired maximum filesize.		somewhat larger than the desired maximum filesize.

- Apache: Apache limits requests with the Apache `LimitRequestBody`		- Apache: Apache limits requests with the Apache `LimitRequestBody`
directive.		directive.
- nginx: nginx limits requests with the nginx `client_max_body_size`		- nginx: nginx limits requests with the nginx `client_max_body_size`
directive. This often defaults to `1M`.		directive. This often defaults to `1M`.
- lighttpd: lighttpd limits requests with the lighttpd		- lighttpd: lighttpd limits requests with the lighttpd
`server.max-request-size` directive.		`server.max-request-size` directive.

Set the applicable limit to at least 32MB. Phabricator can not read these		Set the applicable limit to at least 32MB. Phorge can not read these
settings, so it can not raise setup warnings if they are misconfigured.		settings, so it can not raise setup warnings if they are misconfigured.

PHP: PHP has several directives which limit uploads. These directives are		PHP: PHP has several directives which limit uploads. These directives are
found in `php.ini`.		found in `php.ini`.

- post_max_size: Maximum POST request size PHP will accept. If you		- post_max_size: Maximum POST request size PHP will accept. If you
exceed this, Phabricator will give you a useful error. This often defaults		exceed this, Phorge will give you a useful error. This often defaults
to `8M`. Set this to at least `32MB`. Phabricator will give you a setup		to `8M`. Set this to at least `32MB`. Phorge will give you a setup
warning about this if it is set too low.		warning about this if it is set too low.
- memory_limit: For some uploads, file data will be read into memory		- memory_limit: For some uploads, file data will be read into memory
before Phabricator can adjust the memory limit. If you exceed this, PHP		before Phorge can adjust the memory limit. If you exceed this, PHP
may give you a useful error, depending on your configuration. It is		may give you a useful error, depending on your configuration. It is
recommended that you set this to `-1` to disable it. Phabricator will		recommended that you set this to `-1` to disable it. Phorge will
give you a setup warning about this if it is set too low.		give you a setup warning about this if it is set too low.

You may also want to configure these PHP options:		You may also want to configure these PHP options:

- max_input_vars: When files are uploaded via HTML5 drag and drop file		- max_input_vars: When files are uploaded via HTML5 drag and drop file
upload APIs, PHP parses the file body as though it contained normal POST		upload APIs, PHP parses the file body as though it contained normal POST
parameters, and may trigger `max_input_vars` if a file has a lot of		parameters, and may trigger `max_input_vars` if a file has a lot of
brackets in it. You may need to set it to some astronomically high value.		brackets in it. You may need to set it to some astronomically high value.
- upload_max_filesize: Maximum file size PHP will accept in a raw file		- upload_max_filesize: Maximum file size PHP will accept in a raw file
upload. This is not normally used when uploading files via drag-and-drop,		upload. This is not normally used when uploading files via drag-and-drop,
but affects some other kinds of file uploads. If you exceed this,		but affects some other kinds of file uploads. If you exceed this,
Phabricator will give you a useful error. This often defaults to `2M`. Set		Phorge will give you a useful error. This often defaults to `2M`. Set
this to at least `32MB`.		this to at least `32MB`.

Once you've adjusted all this configuration, your server will be able to		Once you've adjusted all this configuration, your server will be able to
receive chunk uploads. As long as you have somewhere to store them, this will		receive chunk uploads. As long as you have somewhere to store them, this will
enable you to store arbitrarily large files.		enable you to store arbitrarily large files.


Storage Engines		Storage Engines
===============		===============

Phabricator supports several different file storage engines:		Phorge supports several different file storage engines:

\| Engine \| Setup \| Cost \| Notes \|		\| Engine \| Setup \| Cost \| Notes \|
\|--------\|-------\|------\|-------\|		\|--------\|-------\|------\|-------\|
\| MySQL \| Automatic \| Free \| May not scale well. \|		\| MySQL \| Automatic \| Free \| May not scale well. \|
\| Local Disk \| Easy \| Free \| Does not scale well. \|		\| Local Disk \| Easy \| Free \| Does not scale well. \|
\| Amazon S3 \| Easy \| Cheap \| Scales well. \|		\| Amazon S3 \| Easy \| Cheap \| Scales well. \|
\| Custom \| Hard \| Varies \| Implement a custom storage engine. \|		\| Custom \| Hard \| Varies \| Implement a custom storage engine. \|

You can review available storage engines and their configuration by navigating		You can review available storage engines and their configuration by navigating
to {nav Applications > Files > Help/Options > Storage Engines} in the web UI.		to {nav Applications > Files > Help/Options > Storage Engines} in the web UI.

By default, Phabricator is configured to store files up to 1MB in MySQL, and		By default, Phorge is configured to store files up to 1MB in MySQL, and
reject files larger than 1MB. To store larger files, you can either:		reject files larger than 1MB. To store larger files, you can either:

- increase the MySQL limit to at least 8MB; or		- increase the MySQL limit to at least 8MB; or
- configure another storage engine.		- configure another storage engine.

Doing either of these will enable the chunk storage engine and support for		Doing either of these will enable the chunk storage engine and support for
arbitrarily large files.		arbitrarily large files.

▲ Show 20 Lines • Show All 45 Lines • ▼ Show 20 Lines	To enable file storage in S3, set these keys:
- `amazon-s3.region`: Your AWS S3 region.		- `amazon-s3.region`: Your AWS S3 region.
- `amazon-s3.endpoint`: Your AWS S3 endpoint.		- `amazon-s3.endpoint`: Your AWS S3 endpoint.
- `storage.s3.bucket`: S3 bucket name where files should be stored.		- `storage.s3.bucket`: S3 bucket name where files should be stored.

Testing Storage Engines		Testing Storage Engines
=======================		=======================

You can test that things are correctly configured by dragging and dropping		You can test that things are correctly configured by dragging and dropping
a file onto the Phabricator home page. If engines have been configured		a file onto the Phorge home page. If engines have been configured
properly, the file should upload.		properly, the file should upload.

Migrating Files Between Engines		Migrating Files Between Engines
===============================		===============================

If you want to move files between storage engines, you can use the `bin/files`		If you want to move files between storage engines, you can use the `bin/files`
script to perform migrations. For example, suppose you previously used MySQL but		script to perform migrations. For example, suppose you previously used MySQL but
recently set up S3 and want to migrate all your files there. First, migrate one		recently set up S3 and want to migrate all your files there. First, migrate one
file to make sure things work:		file to make sure things work:

phabricator/ $ ./bin/files migrate --engine amazon-s3 F12345		phorge/ $ ./bin/files migrate --engine amazon-s3 F12345

If that works properly, you can then migrate everything:		If that works properly, you can then migrate everything:

phabricator/ $ ./bin/files migrate --engine amazon-s3 --all		phorge/ $ ./bin/files migrate --engine amazon-s3 --all

You can use `--dry-run` to show which migrations would be performed without		You can use `--dry-run` to show which migrations would be performed without
taking any action. Run `bin/files help` for more options and information.		taking any action. Run `bin/files help` for more options and information.

Next Steps		Next Steps
==========		==========

Continue by:		Continue by:

- reviewing at-rest encryption options with		- reviewing at-rest encryption options with
@{article:Configuring Encryption}; or		@{article:Configuring Encryption}; or
- returning to the @{article:Configuration Guide}.		- returning to the @{article:Configuration Guide}.