Audit User Guide

Phorge Administrator and User Documentation (Application User Guides)

Guide to using Phorge to audit published commits.

Overview

Phorge supports two code review workflows, "review" (pre-publish) and "audit" (post-publish). To understand the differences between the two, see User Guide: Review vs Audit.

How Audit Works

The audit workflow occurs after changes have been published. It provides ways to track, discuss, and resolve issues with commits that are discovered after they go through whatever review process you have in place (if you have one).

Two examples of how you might use audit are:

Fix Issues: If a problem is discovered after a change has already been published, users can find the commit which introduced the problem and raise a concern on it. This notifies the author of the commit and prompts them to remedy the issue.

Watch Changes: In some cases, you may want to passively look over changes that satisfy some criteria as they are published. For example, you may want to review all Javascript changes at the end of the week to keep an eye on things, or make sure that code which impacts a subsystem is looked at by someone on that team, eventually.

Developers may also want other developers to take a second look at things if they realize they aren't sure about something after a change has been published, or just want to provide a heads-up.

You can configure Herald rules and Owners packages to automatically trigger audits of commits that satisfy particular criteria.

Audit States and Actions

The audit workflow primarily keeps track of two things:

• Commits and their audit state (like "Not Audited", "Approved", or "Concern Raised").
• Audit Requests which ask a user (or some other entity, like a project or package) to audit a commit. These can be triggered in a number of ways (see below).

Users interact with commits by leaving comments and applying actions, like accepting the changes or raising a concern. These actions change the state of their own audit and the overall audit state of the commit. Here's an example of a typical audit workflow:

• Alice publishes a commit containing some Javascript.
• This triggers an audit request to Bailey, the Javascript technical lead on the project (see below for a description of trigger mechanisms).
• Later, Bailey logs into Phorge and sees the audit request. She ignores it for the moment, since it isn't blocking anything. At the end of the week she looks through her open requests to see what the team has been up to.
• Bailey notices a few minor problems with Alice's commit. She leaves comments describing improvements and uses "Raise Concern" to send the commit back into Alice's queue.
• Later, Alice logs into Phorge and sees that Bailey has raised a concern (usually, Alice will also get an email). She resolves the issue somehow, maybe by making a followup commit with fixes.
• After the issues have been dealt with, she uses "Request Verification" to return the change to Bailey so Bailey can verify that the concerns have been addressed.
• Bailey uses "Accept Commit" to close the audit.

In Diffusion → Browse Commits, you can review commits and query for commits with certain audit states. The default "Active Audits" view shows all of the commits which are relevant to you given their audit state, divided into buckets:

• Needs Attention: These are commits which you authored that another user has raised a concern about: for example, maybe they believe they have found a bug or some other problem. You should address the concerns.
• Needs Verification: These are commits which someone else authored that you previously raised a concern about. The author has indicated that they believe the concern has been addressed. You should verify that the remedy is satisfactory and accept the change, or raise a further concern.
• Ready to Audit: These are commits which someone else authored that you have been asked to audit, either by a user or by a system rule. You should look over the changes and either accept them or raise concerns.
• Waiting on Authors: These are commits which someone else authored that you previously raised a concern about. The author has not responded to the concern yet. You may want to follow up.
• Waiting on Auditors: These are commits which you authored that someone else needs to audit.

You can use the query constraints to filter this list or find commits that match certain criteria.

Audit Triggers

Audit requests can be triggered in a number of ways:

• You can add auditors explicitly from the web UI, using either "Edit Commit" or the "Change Auditors" action. You might do this if you realize you are not sure about something that you recently published and want a second opinion.
• If you put Auditors: username1, username2 in your commit message, it will trigger an audit request to those users when you push it to a tracked branch.
• You can create rules in Herald that trigger audits based on properties of the commit -- like the files it touches, the text of the change, the author, etc.
• You can create an Owners package and enable automatic auditing for the package.

Audits in Small Teams

If you have a small team and don't need complicated trigger rules, you can set up a simple audit workflow like this:

• Create a new Project, "Code Audits".
• Create a new global Herald rule for Commits, which triggers an audit by the "Code Audits" project for every commit where "Differential Revision" "does not exist" (this will allow you to transition partly or fully to review later if you want).
• Have every engineer join the "Code Audits" project.

This way, everyone will see an audit request for every commit, but it will be dismissed if anyone approves it. Effectively, this enforces the rule "every commit should have someone look at it".

Once your team gets bigger, you can refine this ruleset so that developers see only changes that are relevant to them.

Audit Tips

• When viewing a commit, audit requests you are responsible for are highlighted. You are responsible for a request if it's a user request and you're that user, or if it's a project request and you're a member of the project, or if it's a package request and you're a package owner. Any action you take will update the state of all the requests you're responsible for.
• You can leave inline comments by clicking the line numbers in the diff.
• You can leave a comment across multiple lines by dragging across the line numbers.
• Inline comments are initially saved as drafts. They are not submitted until you submit a comment at the bottom of the page.
• Press "?" to view keyboard shortcuts.

Audit Maintenance

The bin/audit command allows you to perform several maintenance operations. Get more information about a command by running:

phorge/ $ ./bin/audit help <command>

Supported operations are:

Delete Audits: Delete audits that match certain parameters with bin/audit delete.

You can use this command to forcibly delete requests which may have triggered incorrectly (for example, because a package or Herald rule was configured in an overbroad way).

Synchronize Audit State: Synchronize the audit state of commits to the current open audit requests with bin/audit synchronize.

Normally, overall audit state is automatically kept up to date as changes are made to an audit. However, if you manually update the database to make changes to audit request state, the state of corresponding commits may no longer be consistent.

This command will update commits so their overall audit state reflects the cumulative state of their actual audit requests.

Update Owners Package Membership: Update which Owners packages commits belong to with bin/audit update-owners.

Normally, commits are automatically associated with packages when they are imported. You can use this command to manually rebuild this association if you run into problems with it.

Next Steps

• Learn more about Herald at Herald User Guide.