CHANGELOG.md

CodeQL CLI changelog

Release 2.2.3 (2020-06-15)

CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new version number because the v2.2.2 folder on the download site originally contained the 2.2.0 binaries instead of the correct 2.2.2 ones.

If you have downloaded release 2.2.2, and codeql --version correctly identifies itself as being that version, you don't need to upgrade to 2.2.3.

Release 2.2.2 (2020-06-12)

• The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.

Improvements

• Query evaluations that time out due to a --timeout option are no longer silently discarded. Instead codeql will terminate with exit code 33. Commands that evaluate multiple queries will produce as much output as they can even if one of the queries times out.

Release 2.2.1

There is no CodeQL CLI version 2.2.1. This version number was used internally to work around restrictions in the CodeQL for VS Code extension.

Release 2.2.0 (2020-05-29)

• The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
• Starting with this release, the CodeQL CLI can be downloaded either as a single codeql.zip file containing the CLI for all supported platforms, or as a codeql-PLATFORM.zip that contains the files for just one platform. The single-platform zips are faster to download.

QL language improvement

• QL now supports the definition of new types as type unions. This feature currently allows unions of branches from an already existing algebraic data type and unions of database types.

Release 2.1.4 (2020-05-26)

This release corresponds to release 1.24.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.2.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available.

Features added

• A new codeql query format command exposes the QL autoformatter for use on the command line.

Bugs fixed

• -J command-line options that contain spaces now ought to work on Windows. They still do not work reliably on Linux or MacOS, though.

Release 2.1.3 (2020-05-13)

Bugs fixed

• Fixes a bug in codeql execute cli-server (a helper used by the VS Code extension) which would sometimes cause query compilation to fail until the extension was restarted.
• Fixes a bug in codeql database upgrade which could lead to performance losses if the upgraded database was subsequently used with LGTM or the legacy Semmle Core product.
• Fixes a bug in the QL evaluator that would sometimes lead to crashes for queries that use the new unique aggregate added in release 2.1.0.
• The value of the --compilation-cache-size option is now correctly interpreted as a number of megabytes rather than a number of bytes.

Release 2.1.2 (2020-05-06)

• Updated license terms to allow CI use with GitHub Actions for open-source software.

Potentially breaking changes

• In query suite definitions, filter instructions that filter on the query path pseudo-tag will now always see the relative path to the query expressed with / as a directory separator, independently of the platform. Previously they erroneously used the platform's directory separator, meaning that query suites developed on Windows would not work correctly on Unix systems (and vice versa) if they used query path. Existing suite definitions developed on Windows may need to be updated to match the new behavior.

Features added

• A new codeql test accept subcommand helps automate updating the expected output for unit tests after a desired change in query behavior. This can also be done by the new --learn option for codeql test run.

Bugs fixed

• codeql database create will now report an explicit error if given a --command argument that specifies an empty string. Previously this would be accepted initially, leading to confusing failures later.

Release 2.1.1 (2020-04-20)

• The bundled extractors are updated to match the versions currently used on LGTM.com.

Features added

• codeql resolve queries accepts a --format=bylanguage option. This is used to help automated workflows determine which languages to create databases for, from the queries that are available to run.
• It is now possible to attempt to execute .ql files that are not in a QL pack. This is used by a few specialized internal workflows. However, standalone queries cannot import any of the dependencies that you would usually declare in a qlpack.yml file, so will not be useful in most cases.

Release 2.1.0 (2020-03-27)

• The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.23) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.23 instance, you need to create them with release 2.0.1. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Potentially breaking changes

• If you pass a directory name as a command-line argument to codeql test run, it will now consider all .ql or .qlref files found under that directory to be test queries, even if they have no accompanying .expected file. Tests that lack an .expected file will fail, but will generate an .actual file that you can rename to .expected if you want to use the results.

  The goal of this change is to support existing workflows of experienced CodeQL users, and also to provide clear error indications if an .expected file is accidentally lost, renamed, or misspelled.

  However, if you invoke codeql test run on a directory tree that contains both tests and non-test queries, you will now encounter errors if any of the .ql files can't be processed as test queries. If you're affected by this change, you can suppress these errors by:

  • Adding a tests property to this QL pack to define specify which directories contain only test queries and associated test code. For more information, see About QL packs.
  • Running codeql test run with a new --strict-test-discovery option.

  In the longer term, we recommend that you reorganize the queries so that test queries are stored in a directory tree that's separate from actual queries.

• codeql database create and codeql database finalize will no longer recognize a --no-duplicate-code option. This option has never had any effect, and its positive variant --duplicate-code previously led to a fatal error.

Features added

• A new XML extractor is included. It is not intended to be used as a stand-alone extractor, but rather to augment the data produced by other extractors. In particular, the C# and Java extractors invoke it during database creation to include information relevant to the analysis of those languages, much like LGTM.com does.
• Two new plumbing commands codeql database index-files and codeql resolve files have been added for support of invoking the XML extractor support. These commands are generally only of interest for extractor authors.
• Two new plumbing commands have been added to codeql dataset. The measure subcommand can be used to collect size information from a dataset, and the check subcommand can scan a dataset for database inconsistencies. These commands are useful when developing a new CodeQL extractor.
• The QL evaluator contains a number of features in support of an internal experiment with using machine-learning techniques to identify functions in unknown codebases as sources or sinks of taint. This includes new command-line options --ml-model-path and --native-library-path to several subcommands. As the new features are not yet ready for general use, these new options should be ignored by external CodeQL users.

Bugs fixed

• Fixes a bug that could result in empty databases for C/C++. Previously, extraction would mistakenly be skipped for source files compiled with the Clang compiler, if the -fintegrated-cc1 option was specified.
• codeql database create and codeql database init will now, as they have always been documented, refuse to create a database whose parent directory doesn't already exist.
• codeql test run will no longer leave .actual files from previous runs in the file system after a test passes.

QL language improvements

• QL now supports set literals, and the QL extractor can identify them with the SetLiteral class. For more information, see Set literal expressions in the QL language reference.

• QL now supports a uniqueness aggregate. This can express constraints that there is precisely one value. The syntax is taken from previous aggregates such as min and max.

  unique(int x | x = 4 or x = 2 * 2 | x)
  

Release 2.0.6 (2020-03-16)

Bugs fixed

• Fixes a problem preventing codeql database create from working with Python 3 on macOS.
• Fixes a problem preventing codeql database create from finding locally installed Python packages.

Release 2.0.5 (2020-03-13)

• The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Features added

• codeql test run has a new --slice option that can be used to parallelize tests over more machines.

Release 2.0.4 (2020-02-21)

• The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Features added

• Subcommands that execute queries (such as codeql database analyze) now have a --timeout option that can be used to set a timeout to automatically cancel query evaluations that appear to diverge.
• A new plumbing command codeql query decompile can display the DIL intermediate representations that is included in the output of codeql query compile --dump-qlo --include-dil-in-qlo. This is useful mainly for certain internal workflows; the information produced is the same as what codeql query compile --dump-dil already outputs.

Bugs fixed

• The --debug and --tuple-counting options to codeql test run erroneously had no effect. Now they ought to work.

Release 2.0.3 (2020-02-12)

Bugs fixed

• Fixes a bug where codeql test run would fail with the message CatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it when running tests against the master branch of the CodeQL libraries for certain languages.
• Otherwise identical to release 2.0.2.

Release 2.0.2 (2020-02-05)

• The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.
• The parent and sibling directories of the unpacked CLI are no longer searched recursively for QL packs. QL packs will only be found if there's a qlpack.yml or .codeqlmanifest.json directly in a parent or sibling directory. This should eliminate the very long disk-scanning delays experienced by users who unpacked earlier versions of the CLI in their home directory.
• Parent and sibling directories of the unpacked CLI will now be searched for QL packs as a last resort, even if you give an explicit --search-path option. This means, for example, that you can define a search path in the per-user configuration file without it depending on where the CLI is unpacked. In particular, the setting can now be meaningfully used by users who let the CodeQL for VS Code extension manage the downloading and unpacking of the CLI.

Security updates

• The codeql database create command and its relatives will no longer attempt to find extractors located in the parent and sibling directories of the unpacked CLI. This closes a security risk for users who unpacked the CodeQL CLI in their home directory. This could've resulted in arbitrary code execution if the user unpacked a file archive containing a malicious extractor anywhere in the home directory. Extractors will now only be found within the unpacked CLI itself, or in directories explicitly listed in the --search-path. It is expected that users will only point --search-path to locations they trust at least as much as the CLI download itself.

Features added

• This release supports executing query regression tests using the codeql test command. For further information, see Testing custom queries.
• The error message if you try executing a query against a database that needs to be upgraded (which can happen routinely if you're using a fresh master checkout of the CodeQL libraries with the bundled extractors) will now explicitly suggest a codeql database update command to run. The database is not automatically upgraded, as this may make it irreversibly incompatible with older versions of the CodeQL libraries. This allows users who want to compare behavior of different versions of the libraries against the same database to make a copy before they upgrade it.

Release 2.0.1 (2019-12-17)

• Corresponds to LGTM Enterprise release 1.23.
• The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the extractor versions used in LGTM Enterprise.
• No other changes to the core CLI.

Release 2.0.0 (2019-11-14)

• First public release.