Page MenuHomePhorge

Update Python-related linters for modern workflows
Open, Needs TriagePublic

Description

Modern python workflows generally use virtualenvs and venv managers (poetry, pipenv), unfortunately, arc lint and (more importantly) Harbormaster arc lint + arc unit buildplan does not know how to use these.

This results in linting tasks in python failing because dependencies are either missing, or the virtualenv that has all the relevant dependencies is not active.

arc lint should probably use the dependency installation and environment activation functions of a project's venv manager (easily detectable by the existence of files like poetry.lock and Pipfile) for python-related linters

Event Timeline

Should this be the responsibility of the linter/tester and not part of Arcanist itself? Updating arcanist to handle the many different environment-dependent systems for languages would mean accounting for nearly every language like Javascript/npm, Ruby/rubyenv, etc. right?

Right now,arc lint doesn't really allow for the activation of a virtual env. That's okay when I'm just running arc lint locally, because I can just activate the environment myself and run it in that environment.

Outside of arcanist, poetry run arc pylint would be how one would invoke pylint in a poetry-managed virtualenv. My issue is, that there is no way to tell Phorge or Arcanist that the linter needs to be invoked that way. And arcanist just knowing that is "as simple as" detecting the existence of poetry.lock (Pipenv.lock in Pipenv's case)

There's also the whole other question of running a linter while project's dependencies are not installed, which results in the linter complaining about unresolved imports.

A workaround to all that for now would be to use the script-and-regex linter, but arcanist already understands pylint and others - would be nice if it also understood the other tools that are often necessary to actually be useful.

Modern(ish) linters support a separate "interpreter" config - if that's set, they run $interpreter $binary $args rather then just $binary $args. Can this be utilized?

It looks like some python projects would need poetry run pylint and some others would need maybe venv/bin/python pylint or something like that - just adding a config in .arcconfig might be the sanest solution.

Alternatively, a general and extensible "guess the environment" feature might make sense these days too.

I think that's roughly what I ended up doing in https://secure.phabricator.com/D14632 for launching separate Java linters, where java had to be configured as the "interpreter" and checkstyle.jar (or whatever) configured as the "binary" (https://secure.phabricator.com/D15067 added the ability to pass flags to the "interpreter").

Checking the source in Arcanist repo, it seems like none of the python linters are actually configured to use an interpreter. (If I attempt to specify one for Pylint anyway, it fails with Got unexpected parameters: interpreter)

Also, this doesn't really solve the problem that the virtualenv might not exist/be in the correct state to run the linter correctly.

Yeah, I think we'd like to try and update arcanist to account for this with a general solution if possible and not making updates for each individual linter which might necessitate this. There's probably some general restructuring that needs to happen to account for the same odd scenario with Java and so forth.

Might be worth it having the linter classes inherit from a language-specific class that would handle things like environment initialization and dependency installation.