Side note: indeed PHP allows threads and async. For example in the Config page, Phorge executes 6+ git commands in async.

I'm not sure what is being suggested.

Arc isn't needed for technical reasons - it's used because it's extremely useful on its own. landing from the server side is actually supported (although hard to setup). There's at least one proof-of-concept for the git push for/ref implementation.

There's also support for things like DOM patching - for example, the preview window bellow the comment area is done server-side, and is updated using AJAX.

Are you suggesting that Phorge currently is "distributed"? I'd describe it as extremely monolithic. There's really just a single namespace for almost all code (All server side, all daemons, arcanist), plus a bunch of client code and 3rd party stuff (which is not something that can be removed). The only "alien" part is Aphlict, which is needed because PHP doesn't (didn't?) do websockets.

Also, I don't know for sure, but I don't believe the technical issues had a serious effect on Phacility's shutdown - I believe it was the business of trying to compete with GitHub in the low end, and with Phabricator on the high-end - Phabricator/Phorge is at its best when it's being managed in-house, with customizations for integration with the rest of the business tools. Using it as a sealed product is missing half the fun.

It all comes from the fact that PHP was designed for a web 1.0

I don't follow this logic.

1. PHP modernized with the rest of the web via php-fpm for process management and opcache for optimizing memory across process invocations. Phorge's system requirements are comparatively low compared to other popular forges like gitlab/github

2. The reason for Phacility closing down was not due to lack of technical ability to scale. Contrarily my best guess is the lack of demand in order to scale. Phacility ran hosted instances on AWS which scaled rather well - in fact if you clone/pull from a repository hosted on secure.phabricator.com you'll see additional messages clarifying what server you were routed to. Phabricator has support for clustering and if Phacility managed hosting services with 2-3 people I'm led to the different conclusion that Phabricator/Phorge scales quite well. I do believe Phacility had tooling to assist with this which was not released open source.

If the desire is to make a hosting service I suggest clarifying the pains you are having or foreseeing so we can identify the gap in managing this scalability and develop the tooling to address it. I believe Phacility implemented their management as extensions to arcanist, similar to their https://secure.phabricator.com/source/secure/ which contained extensions applied to secure.phabricator.com.

Thanks for your replies!

I realize that I failed to communicate my ideas, which only confirms my lack of skill in this department :) I am grateful for the fact that you entertain the discussion nevertheless!

@valerio.bozzolan - I think we rather different understanding of "async" and "threading" concepts. Based on my limited understanding of the codebase Phorge is executing things in separate OS processes, which technically is asynchronous in the broad definition of the term, but is quite different from nodejs promises or golang goroutines. Is us much closer to python's multiprocessing approach https://docs.python.org/3/library/multiprocessing.html (python added true async in 3.5). This may sound like a technicality but it has several crucial differences: interface with a separate OS thread is rather limited, you only have stdin/stdout pipes and an exit code, you are forced to serialize/de-serialize data, it is much harder to debug(due to the fact that it is a different OS process) and the very fact that this is a separate OS process incurs a heavy tax on the OS - spawning thousands OS processes is a good way to thrash your server, unlike spawning async functions in nodes, goroutines in go, processes in erlang etc.
In some cases like running git multiprocessing is a fine choise (an probably the only), but in many other cases it is not, like running a background thread/co-routine that dynamically updates user's page when some event happens (another user comments, or mergability check finishes, etc.)
If I am greatly out of touch with reality and PHP does have true in-process async or multi-threading stick a link to my face with an educational material, I will be grateful!

@avivey sorry for confusing message, I am not currently suggesting any concrete actions. I am trying to communicate the idea of a complexity associated with a distributed system. I know this is a bit of a wild view, and by most classic definitions Phorge is a monolith, but nevertheless I think looking at it as a distributed system has some merit that allows us to extract some practical observations.

Lets look through things through the prism of this definition: "distributed system is one that relies on multiple threads of execution communicating correctly ". By this definition even a single process, 2-threaded program is a distributed system, and if you look at the synchronization problems of multi-threading you will see that it is no different from synchronization issues in distributed system. The only difference is that there is no shared memory, but there is still shared state.

Looking at Phorge through this lens I boldly state that it is highly distributed system: at a very minimum it requires 3 different pieces of software working together: a web server(apache/nginx), php engine, and a database(mysql). All 3 components may even reside on different hardware. While having a separate database for production setup is essential complexity in my mind, separate web server is not (because many other languages and frameworks offer a webserver as a part of the application, allowing better vertical integration, removing IPC overhead, etc.). Without going any further I want to pause here and look at practical implications. Having 3 components in the system makes it harder to:

• distribute the application (there is no single tar.gz/binary that I can download and simply run phorge, it hinders trial and exploration)
• setup development environment (increases barrier for entry for new developers)

On top of that there are accidental complexity penalties: setup instructions for apache/nginx/mysql differ for OSes and distributions, each system uses it's own configuration syntax (xml for apach, nginx config, ini-like for mysql, json for phorge itself). As good counter-examples I want to bring fossil or gitea. Both consist of a single executable file that can be downloaded easily and run with minimal to no configuration (it is possible to run gitea with sqlite to avoid setting up any databases)

Now add Phorge's daemons on top of this for emails, notifications, and whatnot where each requires separate configuration (+ extra configuration of the webserver in some cases) and you get even more accidental complexity.
I am not saying it is impossible to add features in this case. I am asking you - do you think it is easy work to add features like server-side merging today? Does it require creating more daemon processes? - if yes then this increases the complexity of the system and makes it more distributed. Why can't we make Aphlict run in the main application?

So possible practical questions (which I have asked in various forms before) are these:

• Is it possible to make Phorge use a different database? Adding SQLite support could greatly reduce complexity and lower the barrier for entry for new developers and allow running Phorge as a standalone app.

• Same goes for the webserver - is it possible to remove reliance on the web server? PHP has it's own web server. Can we eliminate cases that mandate a third-party web server?

• Can we move daemons and aphlict into the main app?

• What else can we do to make it easier to setup/run/develop Phorge?

• Is it possible to make Phorge use a different database? Adding SQLite support could greatly reduce complexity and lower the barrier for entry for new developers and allow running Phorge as a standalone app.

Phorge/Phab support a variety of different setups, including support of clustered MySQL/MariaDB databases. Moving to SQLite would be a large undertaking and would not support clustering/replication without major overhauls. For a move like this I can only conceive of many downsides and no benefits.

• Same goes for the webserver - is it possible to remove reliance on the web server? PHP has it's own web server. Can we eliminate cases that mandate a third-party web server?

The built in web server is not designed for production use

From https://www.php.net/manual/en/features.commandline.webserver.php

WARNING: This web server is designed to aid application development. It may also be useful for testing purposes or for application demonstrations that are run in controlled environments. It is not intended to be a full-featured web server. It should not be used on a public network.

Whereas using nginx + php-fpm is fully intended to be used in production/public setups. Additionally while there are a large number of web server implementations most any public-facing web application will be behind a reverse proxy of one of the big three battle-tested/hardened web servers anyways, Apache, Nginx, or MS IIS. Doing so has numerous benefits in that those are well-known and well-documented in addition to being so widely used in public-facing setups. Additionally there are numerous extensions/plugins for e.g. security and otherwise which are widely available. Whereas any other web server is likely not as feature-rich or documented. The language/environment of the web server runtime needs to (securely) support TLS whereas apache/nginx/iis have been doing this for ages and can terminate TLS while working as a reverse proxy. If it's going to run behind a reverse proxy then moving to a different web server only makes things more complex, for no benefits that I can think of.

Designing a web application to have a web server "built-in" for the sake of avoiding mult-process setups also has downsides. If something goes wrong with the webserver such as DDoS taking it down then the whole application is kaput, including background tasks, etc. Whereas if apache/nginx/iis go down then all the daemons/background tasks will continue to run.

• Can we move daemons and aphlict into the main app?

Daemons are likely to require more memory and if they encounter an error then having them as a separate process makes them more isolated and easier to manage. It would be cool to not have a dependency on nodejs for aphlict but moving to a separate daemon-like php process is likely not worthwhile.

In all, this is not the direction Phorge should be moving towards. Moving everything into a single process might make things easy from a sysadmin perspective but architecturally there are only downsides. It would not simplify the code architecture or provide any system benefits.

• What else can we do to make it easier to setup/run/develop Phorge?

This appears to be the crux of what you're driving towards and I think where discussion should be re-focused to, to understand specific pain points. We can improve on supplying installation/deploy scripts or packaging, as well as providing quick setups for local testing. See also T15011. I have been working on a docker-compose setup to get a development environment quickly running, currently in https://github.com/neandrake/phab-dev.

@speck I think I can see now where we have fundamental differences in our views.

It comes down to what we think Phorge should be. As @avivey mentioned before - Phab tried to compete with Github and failed. There is no point in competing with Github. In my opinion Phab/Phorge shines best for small and medium sized focused teams that need one stop shop for their development needs. It only makes sense to focus on what makes Phorge unique/different from the rest of the competition, and I don't think this PHP, apache or mysql clustering. For me 90% of the value comes from trunk based development model and stackable patches (instead of non-stackable pull requests), a workflow that enforces small clean commits, easy and incremental merges of big changes - that is what making Phab/Phorge stand out and worth our time.

My idea was to have provide isolated Phorge instances for small/medium teams. Small and medium teams get successful because they focus above all else on their primary task. They don't want setup and maintain LAMP stack plus a dozen of supplementary daemons. They don't need mysql clustering. Also nobody wants to setup PHP these days either. This is not something that FreeBSD community wants, and not what our team at work wants. That is why we are looking at gitea and similar products. Focused teams want quick and easy way to get started and low operational overhead. Easy way to hack on the project - easy way to spin up a local dev env to write a patch. DB clustering and production grade DDoS protection only becomes priority if one wants to compete with Github.
Unfortunately my calculations indicate that operational overhead with current setup is going to be too high. There are simply too many moving parts for a relatively small app.

...
Additionally some of your comments indicate that you are missing my point. Yes, in production public-facing setups admins will put nginx/apache in front of the app, but it should not be a mandatory requirement to launch the app on my laptop. Same way you are not flying a helicopter to the grocery store that is a couple of blocks away from your house - requiring nginx just to get started is an overkill. Additionally I know a lot of Rust, Golang and Scala web servers that perform better than nginx/apache for production setups, the state of the technology has advanced since 2010 when apache/nginx was the only safe and good choice for production.

The built in web server is not designed for production use

It does seem like it is not designed for development use either..

Please consider this issue exhausted.

It does seem like it is not designed for development use either..

Why do you say so?

Phorge works perfectly AFAIK with the development webserver. That is just a single command, without having to install Apache.

https://we.phorge.it/book/contrib/article/running_builtin_php_webserver/

(I think I'm still interested in continuing some of this discussion, but I'm far behind on reading it already. I might resurrect [parts of] it later...)

About offering hosting services massively, AFAIK Phorge is already ready for multi-configuration. I mean, you can quickly offer multi-hosting of a Phorge from a very single installation, but having completely different and separated environments, based for example on the incoming request (domain).

Tips:

https://we.phorge.it/book/phorge/article/advanced_configuration/

I do a similar thing for WordPress (apt install wordpress), where I have several configuration files in /etc/wordpress visible only to the respective Unix users. So I have 80+ WordPress websites online using only a single application, and, while the binary files are the same, the installations are on separated databases and resources.

Ok, the discussion here got a little out of focus. Maybe some parts of this can be extracted to their own discussion.

I would like to answer some of the explicit questions though:

do you think it is easy work to add features like server-side merging today?

I personally implemented this twice, and Evan made a big implementation (That I don't fully understand, and I think it requires Harbormaster + Drydock + I don't know what else), so I think I can implement it again without too much trouble.

One implementation was in-process during the POST response handler - it really wasn't slow enough to be a problem; The second implementation just triggered Jenkins, which was easier (maybe via Harbormaster for feedback).

The difficulties here are more in the terms of user interaction than actual technical issues (how to report errors, how to notify that the action was done, how to allow custom branches...).

Does it require creating more daemon processes?

We don't really do that any more. In the current architecture, the user-side creates a Task object and adds it to a pool (in DB), and a daemon picks it up and executes it.
In theory, we might never have to write "another daemon" ever again.

There's an exception to this rule - the Pull daemon, which has write-access to the local disk. This daemon should run under a different system user from the web-server, for security reasons.

Would new code, that runs outside of the webserver need to be written? yes. A new daemon process? No, we have abstract daemons now.

Is it possible to make Phorge use a different database?

This comes up every couple of years - Q45 is the latest one. Short answer - "Not insanely hard", but hard.

is it possible to remove reliance on the web server?

Well, no. Not in any reasonable way. We can cheat and package a small webserver - I like https://caddyserver.com/ right now.

Can we move daemons and aphlict into the main app?

Ahhh... maaaayybe. I don't fully understand the details, but there's a long-running process that's alive all the time, and it might be able to control the daemons. No idea if that's a thing.
And another maybe for the Aphlict - if PHP can do Websockets, then it might make sense to merge it in.

What else can we do to make it easier to setup/run/develop Phorge?

Those are 3 great questions!
The way we normally go about these things is trying to understand the pain points and root causes, and try to come with a plan that will work to solve them. But we tend to be sticklers about the "understand" part - that is, it's obvious there's things that trouble you (or other users), and you know what they are, but we don't feel them and we need very detailed explanations. Unfortunately, that's the point where many people give up on trying to explain and switch to other products.
"We" (the "core team") are generally ok with the setup/managing of Phorge because we're used to it. We genuinely don't know what exactly is bothering you, because we're already used to what it is.

For the "Develop" part, we have T15011 - it's the only one of the 3 that has an open general ticket, because it's the one we felt was most urgent.
T15364 is a great example on how we can improve things - it's a list of specific things we can address, that if resolved will make users happier.

I guess the summary for all these is "figure out what hurts users, and then see if we can solve it".

My idea was to have provide isolated Phorge instances for small/medium teams.

I think there's lots we can do here without making any architectural changes. Of the top of my head, we can provide a VM image (like GHE) or a Docker image or a K8s Helm chart that has everything packaged up and can be easily deployed in the user's environment. But the key here is that each of these helps a specific group - a team that likes this particular deployment mode (granted, K8s probably covers 99% of the world today).

what we think Phorge should be

We actually discussed this before we started the Phorge org/project - we basically captured it in L1. I might have the actual discussion recorded somewhere too.
Some of the people that were in that discussion haven't been around for a while, and other people have joined since, so I guess it could be re-opened if there's a desire. I'm not sure where @ton sits with that vision (The vision is explicitly against us running a hosted service, but it's not against somebody else - including team members - running one).

On re-writing in a better language

Not explicitly asked, but I'm already 2 pages in. TL;DR: Can't happen.

PHP sucks. It's a really bad language. It competes with JavaScript for "worst language in common use".

But we're invested in really great infrastructures that do work - and work really well and really fast.
Rebuilding an equivalent infrastructure in any language will take literally years, and I don't see anybody paying for that.

We're also heavily invested in a lot of application code - we have lots of it, and much of it is actually used.

There are comparable infrastructures out there, written in other languages, sure - but migrating over across framework and language is an extreme effort. It's one of those projects you hear about in The Daily WTF, where a company hires contractors for millions of dollars and multiple years and ends up with nothing.

If someone wants "the same thing like Phorge, but written in a good language", they'll pretty much have to start over from square one and build something new that's inspired by Phorge (There's plenty of GitHub clones out there, so maybe there's room for a Phorge clone too.).

@valerio.bozzolan

It does seem like it is not designed for development use either..

Why do you say so?

Phorge works perfectly AFAIK with the development webserver. That is just a single command, without having to install Apache.

I can't find the link ATM, but I remember seeing comments from Evan saying that phab makes requests to itself via http in some cases, which will hang the built-in webserver.

Nevertheless, this piece of doc would be nice to have in the main installation/configuration guide (or some sort of quick start guide)