Allow CodeQL CLI version (not path) to be specified in workspace settings

[dbartol]

Is your feature request related to a problem? Please describe.
The current settings schema for the extension lets you specify an arbitrary path to the CodeQL CLI. For security reasons, this setting is ignored when specified in a workspace settings file, because a workspace settings file is usually pulled down as part of the source code. The makes it difficult to specify which version of the CLI a workspace is compatible with. I have this problem to some extent locally, but for customers with multiple query authors working in a repo that depends on a particular version of the CLI and standard libraries, the lack of a way to ensure a consistent version of the CLI is even more frustrating.

Describe the solution you'd like
I'd like a new extension setting, CodeQL > Cli: Version, which allows you the specify the version tag of a public CLI release, and will ensure that the extension downloads and uses that specific release of the CLI. Hopefully, restricting the source of the CLI to our own release feed is sufficiently secure that we can automatically download from there even if it's in the workspace settings. If that is still too scary, though, we could at least put up a prompt saying "this workspace requests version v2.3.6 of the CodeQL CLI. Do you want to download and use it?".

Describe alternatives you've considered
I've considered making my User-level CodeQL CLI path point to a script that tries to figure out the workspace and associated version, but that seems cumbersome for actual customers.

[aeisenberg]

I think this makes sense, especially if this helps our users to migrate away from unsupported products. Our downloading and distribution logic is already quite complex, so I'd want to implement this carefully.

Some questions (and answers):

• Presumably, this setting takes precedence over a codeql.cli.executablePath setting.
• Should we deprecate codeql.cli.executablePath since that would become largely superfluous with the new setting? I realize that this would make using custom builds more challenging, but is that really an issue for our customers or is it only internal people who would need this? I prefer deprecation (without any intention of ever actually removing it) just so we can advertise a single, simpler way of doing things.

[dbartol]

I think executablePath is necessary both for internal developers and for external developers who choose to install the CLI in a particular local path. The simplest example is a developer who uses both the extension and runs the CLI directly. They probably want the CLI on their path, and want to use exactly the same CLI from within the extension.

I would have expected the settings to interact as follows:

• If codeql.cli.executablePath is set, then use the CLI at that path, else
• If codeql.cli.cliVersion is set, download and use that version, else
• If codeql.cli.includePrerelease is set, download and use the latest prerelease version, else
• Download and use the latest non-prerelease version

Basically, codeql.cli.cliVersion would just affect which version was searched for when doing an update, and changes to codeql.cli.cliVersion would force an update check.

[aeisenberg]

Makes sense about not deprecating executablePath. Though, I'd think that the first two bullets should be switched. Even if a user is using a custom, external cli version specified by an executablePath, when a workspace is downloaded and requires a specific cliVersion, I would think that takes precedence.

[dbartol]

Good point. That might not work for the scenario where I want to try out a self-built CLI on a workspace that normally expects a certain CLI version, but at that point I'm happy to modify the workspace settings locally anyway.

[aeisenberg]

As part of a larger work in cli discovery, we should make the cli version being used explicit. So, a user can choose one of

1. latest release
2. hard coded path
3. path variable
4. version number

in their settings and the appropriate cli will be used (or failure if it doesn't exist). And workspaces can override that and specify a version number.

[dbartol]

From Slack:

After a bit more research, it looks like TypeScript has a clever workaround for the security issue. You can specify a TypeScript path in the workspace settings, but it won't be used by default. Instead, it uses whatever the default version would be based on the user settings, but clicking on that version number in the status bar lets you choose "Use workspace version", which then forces TypeScript to use the path set in the workspace settings. There might also be a notification when you open such a workspace. I don't know if we need to go quite that far, but if we think it would help, it seems like a pretty good experience that still requires user consent before running a potentially untrusted TypeScript (or CodeQL CLI) binary.

There are probably some ideas to mine in there.

[aeisenberg]

I was hoping this issue could be a quick win where we use a setting to specify the cli version (currently, the cli version is hard coded here https://github.com/aeisenberg/vscode-codeql/blob/3ceaba9acafaf35252bf46168822cfb05c2cd3c7/extensions/ql-vscode/src/distribution.ts#L43-L43).

Changing this to be a setting, sort of works. The proper cli version is downloaded. However, the way the extension works now is that at most one extension managed distribution can be managed at a time. When a new version of the cli is downloaded, the old version is removed. This will not work for users who need to manage multiple cli versions for different workspaces.

So, a minimal version of this feature will need to:

1. allow for multiple downloaded clis to co-exist
2. provide a way for a user to remove old cli versions:
   • It could be as simple as adding a command that opens the extension global storage directory in a finder/explorer window that contains all the clis so that they can be removed manually.
   • Users should be given the option to remove the previous version after successfully downloading a new version.
   • Users should be given the option to manage versions (see above) after successfully downloading a new version.
3. In order to make management easier, each cli version should be stored in a folder that contains its version number. Doing it this way will help users know which versions they already have. It will also help the extension know about each version without needing to actually issue a cli command. Right now cli folders are named distribution-<num> where <num> is a number that increments each time a new cli is downloaded.
4. The distribution management is a bit tangled right now with all the different ways a cli can be discovered. To implement this feature, we will need to work on distentangling things so that we can more easily make the changes we need.