Page MenuHomePhorge
New Question
Ponder Q135

Files larger than 8MB fails to upload for some S3-compatible apis using upload form at /file/upload (but works using Drag & Drop)
OpenPublic
Actions

Asked by revi on May 3 2024, 09:06.
Tags
None
Referenced Files
F2191615: manual-administration_of_archives_2023.pdf
Jun 27 2024, 15:41
Subscribers
Cigaryno
Matthew
valerio.bozzolan
Tokens
"Grey Medal" token, awarded by valerio.bozzolan.

Details

Version info at the line 2.

P36 errors
1[Fri May 03 08:22:35.871070 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] [2024-05-03 17:22:35] EXCEPTION: (Exception) No configured storage engine can store this file. See "Configuring File Storage" in the documentation for information on configuring storage engines. at [<phorge>/src/applications/files/storage/PhabricatorFile.php:335]
2[Fri May 03 08:22:35.871351 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] arcanist(head=stable, ref.master=5bc53cfe53d0, ref.stable=ca72430916af), phorge(head=stable, ref.master=1b49165ddd16, ref.stable=d519f75dfdee, custom=1)
3[Fri May 03 08:22:35.871359 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #0 <#2> PhabricatorFile::buildFromFileData(string, array) called at [<phorge>/src/applications/files/storage/PhabricatorFile.php:449]
4[Fri May 03 08:22:35.871363 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #1 <#2> PhabricatorFile::newFromFileData(string, array) called at [<phorge>/src/applications/files/storage/PhabricatorFile.php:231]
5[Fri May 03 08:22:35.871366 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #2 <#2> PhabricatorFile::newFromPHPUpload(array, array) called at [<phorge>/src/applications/files/controller/PhabricatorFileUploadController.php:26]
6[Fri May 03 08:22:35.871369 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #3 <#2> PhabricatorFileUploadController::handleRequest(AphrontRequest) called at [<phorge>/src/aphront/configuration/AphrontApplicationConfiguration.php:284]
7[Fri May 03 08:22:35.871372 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #4 phlog(Exception) called at [<phorge>/src/aphront/handler/PhabricatorDefaultRequestExceptionHandler.php:41]
8[Fri May 03 08:22:35.871374 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #5 PhabricatorDefaultRequestExceptionHandler::handleRequestThrowable(AphrontRequest, Exception) called at [<phorge>/src/aphront/configuration/AphrontApplicationConfiguration.php:751]
9[Fri May 03 08:22:35.871377 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #6 AphrontApplicationConfiguration::handleThrowable(Exception) called at [<phorge>/src/aphront/configuration/AphrontApplicationConfiguration.php:296]
10[Fri May 03 08:22:35.871380 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #7 AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phorge>/src/aphront/configuration/AphrontApplicationConfiguration.php:204]
11[Fri May 03 08:22:35.871383 2024] [php:notice] [pid 414798] [client 127.0.0.1:0] #8 AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phorge>/webroot/index.php:35]

I use Cloudflare R2's S3-compatible API; and…

  • Apache LimitRequestBody at 2147483648 (2 GiB)
  • php.ini post_max_size and upload_max_filesize also both at 100MB
  • memory_limit = -1
  • max_input_vars =1(12 zeros)

Given that I've satisfied all the criteria @ the docs I thought it would do chunked uploads, but seems like it is not…

I'm sort of suspecting R2 is at fault here but not sure if this is really the case, so…

Update:

It looks like /file/upload path cannot handle big files, since it does not do a chunked upload.

For example I was able to upload this file using drag & drop:

manual-administration_of_archives_2023.pdf17 MBDownload
<< uploads in editor upload form; but fails when tried via https://we.phorge.it/file/upload/

Show 1 Comment

Answers

This answer has been hidden.

valerio.bozzolan
Updated 146 Days Ago
Actions

So you say that using drag & drop it works. That is probably thanks to the HTML5 JavaScript that splits in chunks, so the parts are under 8MB.

Note that there is an hardcoded limit to 8MB here, that may have sense to deeply encourage the chunk logic (and discourage one single big upload):

https://we.phorge.it/source/phorge/browse/master/src/applications/files/engine/PhabricatorFileStorageEngine.php;899abf7a2b534131d3d4a34016526558494d1fdc$79-113

I think that it's still safe, however, to consider the object-storage as something without limits, or, at least it should not be limited to 8MB. If this makes sense, try this dirty patch:

diff --git a/src/applications/files/engine/PhabricatorS3FileStorageEngine.php b/src/applications/files/engine/PhabricatorS3FileStorageEngine.php
index 95cdfc737b..79f5404250 100644
--- a/src/applications/files/engine/PhabricatorS3FileStorageEngine.php
+++ b/src/applications/files/engine/PhabricatorS3FileStorageEngine.php
@@ -38,6 +38,13 @@ final class PhabricatorS3FileStorageEngine
       phutil_nonempty_string($region);
   }
 
+  /**
+   * S-3 object storage probably has not an hardcoded limit.
+   * Note that the parent class is hardcoded to 8MB.
+   */
+  public function hasFilesizeLimit() {
+    return false;
+  }
 
 /* -(  Managing File Data  )------------------------------------------------- */

So you override the default and you say that your object storage should be used also for files bigger than 8MB.

IMPORTANT: The above one is a dirty patch and MAY completely NUKE your Phorge 💀 💀 💀 💀 💀 💀
NOTE: Or, probably, Phorge should, from PHP, receive your file, and chunks that, faking the frontend HTML5 logics that already chunk stuff. But this does not make any sense and seems super-inefficient. Even if probably this may be more simple to implement as plan B.
NOTE: Or, probably, Phorge should not show a "traditional" file input, and should introduce some frontend drag & drop that can support chunks. This seems interesting and it's maybe a good plan C.

But for now, you can try that dirty workaround with all the possible dark consequences.

Add a Comment

New Answer

Answer

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under Apache 2.0 or other open source licenses. · CC BY-SA 4.0 · Apache 2.0