CodeQL CLI changelog

Release 2.20.0 (2024-12-09)

Known issues

The Windows executable for this release is labeled with an incorrect version number within its properties: the version number should be 2.20.0 rather than 2.19.4. codeql version reports the correct version number.

New features

The QlBuiltins::BigInt type of arbitrary precision integers is generally available and no longer hidden behind the --allow-experimental=bigint CLI feature flag.

Miscellaneous

Backslashes are now escaped when writing output in the Graphviz DOT format (--format=dot).
The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version 21.0.5.

Release 2.19.4 (2024-12-02)

Improvements

CodeQL now supports passing values containing the equals character (=) to extractor options via the --extractor-option flag. This allows cases like --extractor-option opt=key=value, which sets the extractor option opt to hold the value key=value, whereas previously that would have been rejected with an error.
The codeql pack bundle command now sets the numeric user and group IDs of entries in the generated tar archive to 0. This avoids failures like IllegalArgumentException: user id '7111111' is too big ( > 2097151 ) when the numeric user ID is too large.

Bugs fixed

On MacOS, arch -arm64 commands no longer fail when they are executed via codeql database create --command, via codeql database trace-command, or are run after codeql database init --begin-tracing. Note that build commands invoked this way still will not normally be traced, so this is useful only for running ancillary commands which are incidental to building your code.
Fixed a bug where codeql test run would not preserve test databases on disk after a test failed.

Release 2.19.3 (2024-11-07)

Bugs fixed

Fixed a bug where using codeql database import to combine multiple non-empty databases may produce a corrupted database. (The bug does not affect using codeql database finalize --additional-dbs to combine multiple databases.)
Fixed a bug where uses of a QlBuiltins::ExtensionId variable that was not bound to a value could be incorrectly accepted in some cases. In many cases, this would result in a crash.
CodeQL would sometimes refuse to run with more than around 1,500 GB of RAM available, complaining that having so much memory was "unrealistic". The amount of memory CodeQL is able to make any meaningful use of still tops out at about that value, but it will now gracefully accept that so large computers do in fact exist.
Fixed a bug in command-line parsing where a misspelled option could sometimes be misinterpreted as, e.g., the name of a query to run. Now every command-line argument that begins with a dash is assumed to be intended as an option (unless it comes after the -- separator), and an appropriate error is emitted if that is not a recognized one.

The build command in codeql database trace-command is exempted from this for historical reasons, but we strongly recommend putting a -- before the entire build command there, in case a future codeql version starts recognizing options that you intended to be part of the build command.

Miscellaneous

The CodeQL Bundle is now available as an artifact that is compressed using Zstandard. This artifact is smaller and faster to decompress than the original, gzip-compressed bundle. The CodeQL bundle is a tar archive containing tools, scripts, and various CodeQL-specific files.

If you are currently using the CodeQL Bundle, you may want to consider switching to the Zstandard variant of the bundle. You can download the new form of the CodeQL Bundle from the codeql-action releases page by selecting the appropriate bundle with the .zst extension. The gzip-compressed bundles will continue to be available for backwards compatibility.

Release 2.19.2 (2024-10-21)

Potentially breaking changes

The Python extractor will no longer extract the standard library by default, relying instead on models of the standard library. This should result in significantly faster extraction and analysis times, while the effect on alerts should be minimal. It will for a while be possible to force extraction of the standard library by setting the environment variable CODEQL_EXTRACTOR_PYTHON_EXTRACT_STDLIB to 1.

Bugs fixed

The 2.19.1 release contained a bug in the query evaluator that under rare conditions could lead to wrong alerts or resource exhaustion. Although we have never seen the problem outside of internal testing, we encourage users on 2.19.1 to upgrade to 2.19.2.

Miscellaneous

The database relation sourceLocationPrefix is changed for databases created with codeql test run. Instead of containing the path of the enclosing qlpack, it now contains the actual path of the test, similar to if one had run codeql database create on the test folder. For example, for a test such as <checkout>/cpp/ql/test/query-tests/Security/CWE/CWE-611/XXE.qlref we now populate sourceLocationPrefix with <checkout>/cpp/ql/test/query-tests/Security/CWE/CWE-611/ instead of <checkout>/cpp/ql/test/. This change typically impacts calls to File.getRelativePath(), and may as a result change the expected test output.

Release 2.19.1 (2024-10-04)

New Features

The command codeql generate query-help now supports Markdown help files. The Markdown help format is commonly used in custom CodeQL query packs. This new feature allows us to generate SARIF reporting descriptors for CodeQL queries that include Markdown help directly from a query Markdown help file.
Added a new command, codeql resolve packs. This command shows each step in the pack search process, including what packs were found in each step. With the --show-hidden-packs option, it can also show details on which packs were hidden by packs found earlier in the search sequence. codeql resolve packs is intended as a replacement for most uses of codeql resolve qlpacks, whose output is both less detailed and less accurate.

Release 2.19.0 (2024-09-18)

Improvements

codeql database analyze and codeql database interpret-results now support the --sarif-run-property option. You can provide this option when using a SARIF output format to add a key-value pair to the property bag of the run object.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version 21.0.4.

Release 2.18.4 (2024-09-12)

New Features

C# support for build-mode: none is now out of beta, and generally available.
Go 1.23 is now supported.

Release 2.18.3 (2024-08-28)

There are no user-facing changes in this release.

Release 2.18.2 (2024-08-13)

Deprecations

Swift analysis on Ubuntu is no longer supported. Please migrate to macOS if this affects you.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version 21.0.3.

Release 2.18.1 (2024-07-25)

Security Updates

Resolves CVE-2023-4759, an arbitrary file overwrite in Eclipse JGit that can be triggered when using untrusted third-party queries from a git repository. See the security advisory for more information.
The following dependencies have been updated. These updates include security fixes in the respective libraries that prevent out-of-bounds accesses or denial-of-service in scenarios where untrusted files are processed. These scenarios are not likely to be encountered in most uses of CodeQL and code scanning, and only apply to advanced use cases where precompiled query packs, database ZIP files, or database TRAP files are obtained from untrusted sources and then processed on a trusted machine.
- airlift/aircompressor is updated to version 0.27.
- Apache Ant is updated to version 1.10.11.
- Apache Commons Compress is updated to version 1.26.0.
- Apache Commons IO is updated to version 2.15.1.
- Apache Commons Lang3 is updated to version 3.14.0.
- jsoup is updated to version 1.15.3.
- Logback is updated to version 1.2.13.
- Snappy is updated to version 0.5.

New features

The experimental type QlBuiltins::BigInt of arbitrary-precision integers has been introduced. To opt in to this API, compile your queries with --allow-experimental=bigint. Big integers can be constructed using the .toBigInt() methods of int and string. The built-in operations are:
- comparisons: =, !=, <, <=, >, >=,
- conversions: .toString(), .toInt(),
- arithmetic: binary +, -, *, /, %, unary -,
- bitwise operations: .bitAnd(BigInt), .bitOr(BigInt), .bitXor(BigInt), .bitShiftLeft(int), .bitShiftRightSigned(int), .bitNot(),
- aggregates: min, max, (strict)sum, (strict)count, avg, rank, unique, any.
- other: .pow(int), .abs(), .gcd(BigInt), .minimum(BigInt), .maximum(BigInt).
codeql test run now supports postprocessing of test results. When .qlref files specify a path to a postprocess query, then this is evaluated after the test query to transform the test outputs prior to concatenating them into the actual results.

Improvements

The 30% QL query compilation slowdown noted in 2.18.0 has been fixed.

Release 2.18.0 (2024-07-11)

Breaking changes

A number of breaking changes have been made to the C and C++ CodeQL test environment as used by codeql test run:
- The test environment no longer defines any GNU-specific builtin macros. If these macros are still needed by a test, please define them via semmle-extractor-options.
- The --force-recompute option is no longer directly supported by semmle-extractor-options. Instead, --edg --force-recompute should be specified.
- The --gnu_version and --microsoft_version options that can be specified via semmle-extractor-options are now synonyms, and only one should be specified as part of semmle-extractor-options. Furthermore, is also no longer possible to specify these options via the following syntax.
  - --edg --gnu_version --edg <version number>, and
  - --edg --microsoft_version --edg <version number>
  The shorter --gnu_version <version number> and --microsoft_version <version number> should be used.
The --build_error_dir and --predefined_macros command line options have been removed from the C/C++ extractor. It has never been possible to pass these options through the CLI, but some customers with advanced setups may have been passing them through internal undocumented interfaces. Passing the option --build_error_dir did not have any effect, and it is safe to remove the option. The --predefined_macros option should have been unnecessary, as long as the extractor was invoked with the --mimic option.

Regressions

Compilation of QL queries is about 30% slower than in previous releases. This only affects users who write custom queries, and only at compilation time, not at run time. This regression will be fixed in the upcoming 2.18.1 release.

Improvements

Introduced the --include-logs option to the codeql database bundle command. This new feature allows users to include logs in the generated database bundle, allowing for a more complete treatment of the bundle, and bringing the tool capabilities up-to-speed with the documentation.
codeql database init and codeql database create now support the --force-overwrite option. When this option is specified, the command will delete the specified database directory even if it does not look like a database directory. This option is only recommended for automation. For directcommand line commands, it is recommended to use the --overwrite option, which includes extra protection and will refuse to delete a directory that does not look like a database directory.
Extract .xsaccess, *.xsjs and *.xsjslib files for SAP HANA XS as Javascript.
We have updated many compiler error messages and warnings to improve their readability and standardize their grammar. Where necessary, please use the --learn option for the codeql test run command.

Bugs fixed

Where a MacOS unsigned binary cannot be signed, CodeQL will now continue trying to trace compiler invocations created by that process and its children. In particular this means that Bazel builds on MacOS are now traceable.
Fixed a bug where test discovery would fail if there is a syntax error in a qlpack file. Now, a warning message will be printed and discovery will continue.

Release 2.17.6 (2024-06-27)

New features

Beta support is now available for analyzing C# codebases without needing a working build. To use this, pass the --build-mode none option to codeql database create.

Improvements

The --model-packs option is now publicly available. This option allows commands like codeql database analyze to accept a list of model packs that are used to augment the analysis of all queries involved in the analysis.

Release 2.17.5 (2024-06-12)

Breaking changes

All the commands that output SARIF will output a minified version to reduce the size. The codeql database analyze, codeql database interpret-results, codeql generate query-help, and codeql bqrs interpret commands support the option --no-sarif-minify to output a pretty printed SARIF file.
A number of breaking changes have been made to the semmle-extractor-options functionality available for C and C++ CodeQL tests.
- The Arm, Intel, and CodeWarrior compilers are no longer supported and the --armcc, --intel, --codewarrior flags are now ignored, as are all the flags that only applied to those compilers.
- The --threads and -main-file-name options, which did not have any effect on tests, are now ignored. Any specification of these options as part of semmle-extractor-options should be removed.
- Support for --linker, all flags that would only invoke the preprocessor, and the /clr flag have been removed, as those flags would never produce any usable test output.
- Support for the --include_path_environment flag has been removed. All include paths should directly be specified as part of semmle-extractor-options.
- Microsoft C/C++ compiler response files specified via @some_file_name are now ignored. Instead, all options should directly be specified as part of semmle-extractor-options.
- Support for Microsoft #import preprocessor directive has been removed, as support depends on the availability of the Microsoft C/C++ compiler, and availability cannot be guaranteed on all platforms while executing tests.
- Support for the Microsoft /EHa, /EHs, /GX, /GZ, /Tc, /Tp, and /Zl flags, and all /RTC flags have been removed. Any specification of these options as part of semmle-extractor-options should be removed.
- Support for the Apple-specific -F and -iframework flags has been removed. The -F flag can still be used by replacing -F <directory> by --edg -F --edg <directory>. Any occurrence of -iframework <arg> should be replaced by --edg --sys_framework --edg <arg>.
- Support for the /TC, /TP, and -x flags has been removed. Please ensure all C, respectively C++, source files have a .c, respectively .cpp, extension.
- The --build_error_dir, -db, --edg_base_dir, --error_limit, --src_archive, --trapfolder, and --variadic_macros flags are now ignored.
The above changes do not affect the creation of databases through the CodeQL CLI, or when calling the C/C++ extractor directly with the --mimic or --linker flags. Similar functionality continues to be supported in those scenarios, except for CodeWarrior and the --edg_base_dir, --include_path_environment, /Tc, and /Tp flags, which were never supported.

Improvements

codeql generate log-summary now reports completed pipeline runs that are part of an incomplete recursive predicate.

Miscellaneous

The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal documentation generation commands has been updated to version 20240325.1.

Release 2.17.4 (2024-06-03)

New features

CodeQL package management is now generally available, and all GitHub-produced CodeQL packages have had their version numbers increased to 1.0.0.

Release 2.17.3 (2024-05-17)

Improvements

The language server that our IDE integration is built on now defaults to fine-grained dependency tracking for incremental error-checking after file changes. This slightly improves the latency of refreshing errors after local source code edits and will enable significant speedups in the future.
We now properly handle globs (such as folder/**/*.py) in paths configuration to specify what files to include for Python analysis (see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#specifying-directories-to-scan).
TRAP import (a part of codeql database create and codeql database finalize) now supports allocating 2^32 IDs during the import process. The previous limit was 2^31 IDs.

Release 2.17.2 (2024-05-07)

Known issues

The beta support for analyzing Swift in this release and all previous releases requires g++-13 when running on Linux. Users analyzing Swift using the ubuntu-latest, ubuntu-22.04, or ubuntu-20.04 runner images for GitHub Actions should update their workflows to install g++-13. For more information, see the runner images announcement.

Improvements

When uploading a SARIF file to GitHub using codeql github upload-results, the CodeQL CLI now waits for the file to be processed by GitHub. If any errors occurred during processing of the analysis results, the command will log these and return a non-zero exit code. To disable this behaviour, pass the --no-wait-for-processing flag.

By default, the command will wait for the SARIF file to be processed for a maximum of 2 minutes, however this is configurable with the --wait-for-processing-timeout option.
The build tracer is no longer enabled when using the none build mode to analyze a compiled language, thus improving performance.

Release 2.17.1 (2024-04-24)

Deprecations

The --mode option and -m alias to codeql database create, codeql database cleanup, and codeql dataset cleanup has been deprecated. Instead, use the new --cache-cleanup option, which has identical behavior.

Improvements

Improved the diagnostic message produced when no code is processed when creating a database. If a build mode was specified using --build-mode, the message is now tailored to your build mode.

Miscellaneous

The scc tool used by the CodeQL CLI to calculate source code baseline information has been updated to version 3.2.0.

Release 2.17.0 (2024-04-04)

Deprecations

The --[no-]analysis-summary-v2 and --[no-]new-analysis-summary options that were used to enable (or disable) improved summary information printed at the end of a codeql database analyze invocation are no longer supported. Improved summary information is now enabled for all invocations.
Support for overwriting default CodeQL SARIF run properties using the --sarif-run-property command line option has been removed. This removes the ability to overwrite the semmle.formatSpecifier, metricResults, and codeqlConfigSummary properties in the SARIF run file.

Improvements

TRAP import (a part of codeql database create and codeql database finalize) now performs better in low-memory situations. (Put another way, it now needs less RAM to achieve the same performance as before.)
The worst-case performance of transitive closure computation (using the + or * postfix operators or the fastTC higher-order primitive in QL) has been greatly improved.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version 21.0.2.

Release 2.16.6 (2024-03-26)

Bugs fixed

Fixes a bug where extractor logs would be output at a lower than expected verbosity level when using the codeql database create command.

Release 2.16.5 (2024-03-21)

New features

Beta support has been added for analyzing Java codebases without needing a working build. To enable this, pass the --build-mode none option to codeql database create.

Release 2.16.4 (2024-03-11)

Potentially breaking changes

A number of internal command line options (--builtin_functions_file, --clang_builtin_functions, --disable-objc-default-synthesize-properties, --list_builtin_functions, --memory-limit-bytes, --mimic_config, and --objc) has been removed from the C/C++ extractor. It has never been possible to pass these options through the CLI itself, but some customers with advanced setups may have been passing them through internal undocumented interfaces. All of the removed options were already no-ops, and will now generate errors.

The --verbosity command line option has also been removed. The option was an alias for --codeql-verbosity, which should be used instead.

Improvements

The frontend of the C/C++ extractor has been updated, improving the extractor's reliability and increasing its ability to extract source code.

Bugs fixed

When parsing user-authored YAML files such as codeql-pack.yml, qlpack.yml, codeql-workspace.yml, and any YAML file defining a data extension, unquoted string values starting with a * character are now correctly interpreted as YAML aliases. Previously, they were interpreted as strings, but with the first character skipped.

If you see a parse error similar to while scanning an alias... unexpected character found *(42),it likely means that you need to add quotes around the indicated string value. The most common cause is unquoted glob patterns that start with *, such as include: **/*.yml, which will need to be quoted as include: "**/*.yml".

Release 2.16.3 (2024-02-22)

Security patches

Fixes CVE-2024-25129, a limited data exfiltration vulnerability that could be triggered by untrusted databases or QL packs. See the security advisory for more information.

New Features

A new extractor option has been added to the Python extractor: python_executable_name. You can use this option to override the default process the extractor uses to find and select a Python executable. Pass one of --extractor-option python_executable_name=py or --extractor-option python_executable_name=python or --extractor-option python_executable_name=python3 to commands that run the extractor, for example: codeql database create.

On Windows machines, the Python extractor will expect to find py.exe on the system PATH by default. If the Python executable has a different name, you can set the new extractor option to override this value and look for python.exe or python3.exe.

For more information about using the extractor option with the CodeQL CLI, see Extractor options.

Bugs fixed

Fixed a bug where CodeQL may produce an invalid database when it exhausts all available ID numbers. Now it detects the condition and reports an error instead.

Release 2.16.2 (2024-02-12)

There are no user-facing changes in this release.

Release 2.16.1 (2024-01-25)

Improvements

When executing the codeql database init command, the CodeQL runner executable path is now stored in the CODEQL_RUNNER environment variable. Users of indirect tracing on MacOS with System Integrity Protection enabled who previously had trouble with indirect tracing should prefix their build command with this path. For example, $CODEQL_RUNNER build.sh.

QL language improvements

Name clashes between weak aliases (i.e. aliases that are not final aliases of non-final entities) of the same target no longer cause ambiguity errors.

Release 2.16.0 (2024-01-16)

New Features

Users specifying extra tracing configurations may now use the GetRegisteredMatchers(languageId) Lua function to retrieve the existing table of matchers registered to a given language.

Improvements

The Experimental flag has been removed from all packaging and related commands.
The RA pretty-printer omits names of internal RA nodes and pretty-prints binary unions with nested internal unions as n-ary unions. VS Code extension v1.11.0 or newer is required to compute join order badness metrics in VS Code for the new RA format.

Potentially breaking changes

The Python extractor will no longer extract dependencies by default. See https://github.blog/changelog/2023-07-12-code-scanning-with-codeql-no-longer-installs-python-dependencies-automatically-for-new-users/ for more context. In versions until 2.17.0, it will be possible to restore the old behavior by setting CODEQL_EXTRACTOR_PYTHON_FORCE_ENABLE_LIBRARY_EXTRACTION_UNTIL_2_17_0=1.
The --ram option to codeql database run-queries and other commands that execute queries is now interpreted more strictly. Previously it was mostly a rough hint for how much memory to use, and the actual memory footprint of the CodeQL process could be hundreds of megabytes higher. From this release, CodeQL tries harder to keep its total memory consumption during evaluation below the given limit.

The new behavior yields more predictable memory use, but since it works by allocating less RAM, it can lead to more use of disk storage for intermediate results compared to earlier releases with the same --ram value, and consequently a slight performance loss. In rare cases, for large databases, analysis may fail with a Java OutOfMemoryError.

The cure for this is to increase --ram to be closer to the amount of memory actually available for CodeQL. As a rule of thumb, it will usually be possible to increase the value of --ram by 700 MB or more, without actually using more resources than release 2.15.x would with the old setting. An exact amount cannot stated, however, since the actual memory footprint in earlier releases depended on factors such as the size of the databases that were not fully taken into account.

If you use the CodeQL Action, you do not need to do anything unless you have manually overridden the Action's RAM setting. The Action will automatically select a --ram setting that matches the version of the CLI it uses.

Release 2.15.5 (2023-12-20)

New features

A new extractor option has been added to the JavaScript/TypeScript extractor. Set the environment variable CODEQL_EXTRACTOR_JAVASCRIPT_OPTION_SKIP_TYPES to true to skip the extraction of types in TypeScript files. Use this to speed up extraction if your codebase has a high volume of TypeScript type information that causes a noticeable bottleneck for TypeScript extraction. The majority of analysis results should be preserved even when no types are extracted.

Bugs fixed

Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. This affected the human-readable summary produced by codeql database analyze and codeql database interpret-results, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page.
When analyzing Swift codebases, CodeQL build tracing will now ignore the codesign tool. This prevents errors in build commands or workflows on macOS that include both CodeQL and code signing.

Release 2.15.4 (2023-12-11)

New features

Java 21 is now fully supported, including support for new language features such as pattern switches and record patterns.

Improvements

Parallelism in the evaluator has been improved, resulting in faster analysis when running with many threads, particularly for large databases.

Release 2.15.3 (2023-11-22)

New features

codeql database analyze now defaults to include markdown query help for all custom queries with help files available. To change the default behaviour you can pass the new flag --sarif-include-query-help, which provides the options always (which includes query help for all queries), custom_queries_only (the default) and never (which does not include query help for any query). The existing flag --sarif-add-query-help has been deprecated and will be removed in a future release.
The new (advanced) command-line option --[no-]linkage-aware-import disables the linkage-awareness phase of codeql dataset import, as a quick fix (at the expense of database completeness) for C++ projects where this part of database creation consumes too much memory. This option is available in the commands database create, database finalize, database import, dataset import, test extract, and test run.
The CodeQL language server now provides basic support for Rename, and you can now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The current Rename support is less a refactoring tool and more a labor-saving device. You may have to perform some manual edits after using Rename, but it should still be faster and less work than renaming a symbol manually.

Improvements

The Find References feature in the CodeQL language server now supports all CodeQL identifiers and offers improved performance compared to CodeQL CLI 2.14 releases.
The compiler generates shorter human-readable DIL and RA relation names. Due to use of an extended character set, full VS Code support for short relation names requires VS Code extension 1.9.4 or newer.
codeql database create and codeql database finalize now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk.

Bugs fixed

Fixed an internal error in the compiler when arguments to the codePointCount string primitive were not bound.
Fixed a bug where codeql database finalize would fail if a database under construction was moved between machines between codeql database init and codeql database finalize. This should now work, as long as both commands are run by the same release of the CodeQL CLI and the extractors used are the ones bundled with the CLI.
Fixed a bug where codeql database run-queries would fail in some circumstances when the database path included an @.

Release 2.15.2 (2023-11-13)

Breaking changes

C++ extraction has been updated to output more accurate C++ value categories. This may cause unexpected alerts on databases extracted with an up-to-date CodeQL when the queries are part of a query pack that was compiled with an earlier CodeQL. To resolve this, please recompile the query pack with the latest CodeQL.

New features

codeql database analyze and codeql database interpret-results can now output human-readable analysis summaries in a new format. This format provides file coverage information and improves the way that diagnostic messages are displayed. The new format also includes a link to the tool status page when the GITHUB_SERVER_URL and GITHUB_REPOSITORY environment variables are set. Note that that page only exists on GitHub.com, or in GitHub Enterprise Server version 3.9.0 or later. To enable this new format, pass the --analysis-summary-v2 flag.
CodeQL now supports distinguishing file coverage information between related languages C and C++, Java and Kotlin, and JavaScript and TypeScript. By default, file coverage information for each of these pairs of languages is grouped together. To enable specific file coverage information for these languages, pass the --sublanguage-file-coverage flag when initializing the database (with codeql database create or codeql database init) and when analyzing the database (with codeql database analyze or codeql database interpret-results). If you are uploading results to a GitHub instance, this flag requires GitHub.com or GitHub Enterprise Server version 3.12 or later.
All CLI commands now support --common-caches, which controls the location of the cached data that is persisted between several runs of the CLI, such as downloaded QL packs and compiled query plans.

Improvements

Model packs that are used in an analysis will now be included in an output SARIF results file. All model packs now include the isCodeQLModelPack: true property in their tool component property bag.
The default formatting of DIL now more closely resembles equivalent QL code.

Bugs fixed

Fixed a bug where codeql github upload-results would report a 403 error when attempting to upload to a GitHub Enterprise Server instance.
Fixed a bug in Python extraction where UTF-8 characters would cause logging to fail on systems with non-UTF-8 default system encoding (for example, Windows systems).
The resolve qlpacks --kind extension command no longer resolves extensions packs from the search path. This matches the behavior of resolve extensions-by-pack and will ensure that extensions which are resolved by resolve qlpacks --kind extension can also be resolved by resolve extensions-by-pack.

Release 2.15.1 (2023-10-19)

Potentially Breaking Changes

The query server's evaluation/trimCache command was previously equivalent to the codeql database cleanup --mode=gentle CLI command, but is now equivalent to using --mode=normal. The new meaning of the command is to clear the entire evaluation cache of a database except for predicates annotated with the cached keyword.

Deprecations

The accepted values of the --mode option for codeql database cleanup have been renamed to bring them in line with what they are called in the VSCode extension and the query server:
- --mode=brutal is now --mode=clear.
- --mode=normal is now --mode=trim.
- --mode=light is now --mode=fit.
- The old names are deprecated, but will be accepted for backwards-compatibility reasons until further notice.

Improvements

The list of failed tests at the end of a codeql test run is now sorted lexicographically.
The syntax of DIL now more closely resembles the QL source code that it is compiled from. In particular, conjunctions and disjunctions now use the familiar and and or keywords, and clauses are enclosed in curly braces.

Bugs fixed

Fixed a bug where the $CODEQL_JAVA_HOME environment variable was erroneously ignored for certain subsidiary Java processes started by codeql.
Fixed a bug in the CodeQL build tracer on Apple Silicon machines that prevented database creation if System Integrity Protection was disabled.

Release 2.15.0 (2023-10-11)

Deprecations

pragma[assume_small_delta] is now deprecated. The pragma has no effect and should be removed.
Missing override annotations on class fields now raise errors rather than warnings. This is to avoid confusion with the shadowing behavior in the presence of final fields.
The CodeQL CLI no longer supports ML-powered alerts. For more information, including details of our work in the AI-powered security technology space, see "CodeQL code scanning deprecates ML-powered alerts."

New Features

The output of codeql version --format json now includes a features property. Each key in the map identifies a feature of the CodeQL CLI. The value for a key is always true. Going forward, whenever a significant new feature is added to the CodeQL CLI, a corresponding entry will be added to the features map. This is intended to make it easier for tools that invoke the CodeQL CLI to know if the particular version of the CLI they are invoking supports a given feature, without having to know exactly what CLI version introduced that feature.

Improvements

You can now specify the CodeQL languages C/C++, Java/Kotlin, and JavaScript/TypeScript using --language c-cpp, --language java-kotlin, and --language javascript-typescript respectively. These new CodeQL language names convey more clearly what languages each CodeQL language will analyze.

You can also reference these CodeQL languages via their secondary language names (C/C++ via --language c or --language cpp, Java/Kotlin via --language java or --language kotlin, and JavaScript/TypeScript via --language javascript or --language typescript), however we recommend you refer to them via the new primary CodeQL language names for improved clarity.
CodeQL now respects custom home directories set by the $HOME environment variable on MacOS and Linux and %USERPROFILE% on Windows. When set, CodeQL will use the variable's value to change the default location of downloaded packages and the global compilation cache.
This release improves the quality of file coverage information for repositories that vendor their dependencies. This is currently supported for Go and JavaScript projects.

Bugs fixed

Fixed an issue with analyzing Python projects using Python 3.12.

QL language improvements

The QL language now has two new methods codePointAt and codePointCount on the string type. The methods both return integers and act the same as the similarly named Java methods on strings. For example, "abc".codePointAt(2) is 99 and ("a" + 128512.toUnicode() + "c").codePointAt(1) is a 128512.

Release 2.14.6 (2023-09-26)

Bugs fixed

The tracking of RAM usage has been improved. This fixes some cases where CodeQL uses more RAM than requested.

Release 2.14.5 (2023-09-14)

Bugs fixed

Fixed a JavaScript extractor crash that was introduced in 2.14.4.

Release 2.14.4 (2023-09-12)

Potentially breaking changes

The CodeQL CLI no longer supports the SEMMLE_JAVA_ARGS environment variable. All previous versions of the CodeQL CLI perform command substitution on the SEMMLE_JAVA_ARGS value (for example, replacing '$(echo foo)' with 'foo') when starting a new Java virtual machine, which, depending on the execution environment, may have security implications. Users are advised to check their environments for possible SEMMLE_JAVA_ARGS misuse.

New Features

The Java extractor now supports files that use Lombok.

Bugs fixed

codeql database init (and github/codeql-action/init@v2 on GitHub Actions) should no longer hang or crash for traced languages on 64-bit Windows machines when certain antivirus software is installed.
During codeql pack create and codeql pack publish, a source version of a pack coming from --additional-packs can explicitly be used to override a requested pack version even if this source version is incompatible with the requested version in the pack file. Previously, this would fail with a confusing error message.
Fixed a bug where codeql database interpret-results hangs when a path query produces a result that has no paths from source to sink.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.8.

Release 2.14.3 (2023-08-25)

Breaking changes

The <run>.tool.extensions property in the SARIF generated by codeql database analyze now contains the following packs:
- The containing query pack for each query that was evaluated.
- Each model pack that was specified via the --model-packs option, regardless of whether that model pack affected any of the evaluated queries.
Library packs are no longer included in the list.

Previously, this property contained every query and library pack that was available on the search path, regardless of whether that pack was used during the evaluation.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.8.
When codeql test generates .actual files, they will in some cases list the query predicates in a different order than past versions. There is no need to update .expected files, as codeql test sorts their results accordingly before diffing. However, when there are genuine changes in expected results, the generated .actual file can show additional changes against the .expected due to the reordering.

Release 2.14.2 (2023-08-11)

Breaking changes

The functionality provided by the codeql execute query-server subcommand has been removed. The subcommand now responds to all JSON RPC requests with an error response. Correspondingly, this release is no longer compatible with versions of the CodeQL extension for Visual Studio Code prior to 1.7.6.

This change also breaks third-party CodeQL IDE integrations that still rely on the codeql execute query-server subcommand. Maintainers of such CodeQL IDE integrations should migrate to the codeql execute query-server2 subcommand at the earliest opportunity.

Improvements

Switched from prefix filtering of autocomplete suggestions in the language server to client-side filtering. This improves autocomplete suggestions in contexts with an autocompletion prefix.
The CodeQL language server now checks query metadata for errors. This allows Visual Studio Code users to see errors in their query metadata without needing to compile the query.

Bugs fixed

Fixed bug that made the --warnings=hide option do nothing in codeql database analyze and other commands that evaluate queries.

Release 2.14.1 (2023-07-27)

There are no user-facing changes in this release.

Release 2.14.0 (2023-07-13)

Potentially breaking changes

The legacy option --search-path will now be used, if provided, when searching for the dependencies of packages that have no lock file.
CodeQL query packs that specify their dependencies using the legacy libraryPathDependencies property in qlpack.yml/codeql-pack.yml files are no longer permitted to contain a codeql-pack.lock.yml lock file. This will lead to a compilation error. This change is intended to prevent confusing behavior arising from a mix of legacy (unversioned) and modern (versioned) package dependencies. To fix this error, either delete the lock file, or convert libraryPathDependencies to dependencies.
CodeQL CLI commands that create packages or update package lock files, such as codeql pack publish and codeql pack create, will no longer work on query packs that specify their dependencies using the legacy libraryPathDependencies property. To fix this error, convert libraryPathDependencies to dependencies.

Deprecations

Missing override annotations on class member predicates now raise errors rather than warnings. This is to avoid confusion with the shadowing behaviour in the presence of final member predicates.

class Foo extends Base {
  final predicate foo() { ... }

  predicate bar() { ... }
}

class Bar extends Foo {
  // This method shadows Foo::foo.
  predicate foo() { ... }

  // This used to override Foo::bar with a warning, now raises error.
  predicate bar() { ... }
}

Improvements

Unqualified imports can now be marked as deprecated to indicate that the import may be removed in the future. Usage of names only reachable through deprecated imports will generate deprecation warnings.
Classes declared inside a parameterized modules can final extend parameters of the module as well as types that are declared outside the parameterized module.
Fields are fully functional when extending types from within a module instantiation.
Files with a .yaml extension will now be included in compiled CodeQL packs. Previously, files with this extension were excluded even though .yml files were included.
When interpreting results (e.g., using bqrs interpret or database interpret-results), extra placeholders in alert messages are treated as normal text. Previously, results with more placeholders than placeholder values were skipped.
Windows users of the CodeQL extension for VS Code will see faster start times.
In VS Code, errors in the current file are rechecked when dependencies change.
In VS Code, autocomplete in large QL files is now faster.
Member predicates can shadow final member predicates of the same arity even when the signatures are not fully matching.

Bugs fixed

Fixed super calls on final base classes (or final aliases) so that they are now dispatched the same way as super calls on instanceof supertypes.
Fixed a bug where running codeql database finalize with a large number of threads would fail due to running out of file descriptors.
Fixed a bug where codeql database create --overwrite would not work with database clusters.
Fixed a bug where the CodeQL documentation coverage statistics were incorrect.
Fixed a bug where the generated CodeQL libarary documentation could generate invalid uris on windows.

Release 2.13.5 (2023-07-05)

New Features

The Swift extractor now supports Swift 5.8.1.

Release 2.13.4 (2023-06-19)

New features

Temporary files and folders created by the CodeQL CLI will now be cleaned up when each CLI command (and its internal JVM) shuts down normally.

Bugs fixed

Fixed an issue where indirect build tracing did not work in Azure DevOps pipeline jobs in Windows containers. To use indirect build tracing in such environments, ensure both the --begin-tracing and --trace-process-name=CExecSvc.exe arguments are passed to codeql database init.
Improved the error message for the codeql pack create command when the pack being published has a dependency with no scope in its name.

Release 2.13.3 (2023-05-31)

New features

This release enhances our preliminary Swift support, setting the stage for the upcoming public beta.
The codeql database bundle command now supports the --[no]-include-temp option. When enabled, this option will include the temp folder of the database directory in the zip file of the bundled database. This folder includes generated packages and queries, and query suites.
The structured log produced by codeql generate log-summary now includes a Boolean isCached field for predicate events, where a true value indicates the predicate is a wrapper implementing the cached annotation on another predicate. The wrapper depends on the underlying predicate that the annotation was found on, and will usually have the same name, but it has a separate raHash.

Bugs fixed

Fixed a bug that could cause the compiler to infer incorrect binding sets for non-direct calls to overriding member predicates that have stronger binding sets than their root definitions.
Fixed a bug that could have caused the compiler to incorrectly infer that a class matched a type signature. The bug only affected classes with overriding member predicates that had stronger binding sets than their root definitions.
Fixed a bug where a query could not be run from VS Code when there were packs nested within sibling directories of the query.

Release 2.13.2

This release was skipped.

Release 2.13.1 (2023-05-03)

Bugs fixed

Fixed a bug in codeql database upload-results where the subcommand would fail with "A fatal error occurred: Invalid SARIF.", reporting an InvalidDefinitionException. This issue occurred when the SARIF file contained certain kinds of diagnostic information.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.7.

Release 2.13.0 (2023-04-20)

Known issues

We recommend that customers using the CodeQL CLI in a third party CI system do not upgrade to this release, due to an issue with codeql github upload-results. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. For more information, see the "Known issues" section for CodeQL 2.12.6.

Potentially breaking changes

In codeql pack add, the dependency that is added to the qlpack.yml file will now allow any version of the pack that is compatible with the specified version (^version) in the following cases:
- When no version is specified (codeql pack add codeql/cpp-all).
- When the version is specified as latest (codeql pack add codeql/cpp-all@latest).
- When a single version is specified (codeql pack add codeql/cpp-all@1.0.0).
The ^version dependency allows any version of that pack with no breaking changes since version. For example, ^1.2.3 would allow versions 1.2.3, 1.2.5, and 1.4.0, but not 2.0.0, because changing the major version number to 2 indicates a breaking change.

Using ^version ensures that the added pack is not needlessly constrained to an exact version by default.
Upper-case variable names are no longer accepted by the QL compiler.

Such variable names have produced a deprecation warning since release 2.9.2 (released 2022-05-16), so QL code that compiles without warnings with a recent release of the CLI should still work.

New features

codeql database analyze and related commands now export file coverage information by default. GHAS customers using CodeQL in third-party CI systems will now see file coverage information on the tool status page without needing to modify their CI workflows.

Deprecations

The possibility to omit override annotations on class member predicates that override a base class predicate has been deprecated. This is to avoid confusion with shadowing behaviour in the presence of final member predicates.

class Foo extends Base {
  final predicate foo() { ... }

  predicate bar() { ... }

  predicate baz() { ... }
}

class Bar extends Foo {
  // This method shadows Foo::foo.
  predicate foo() { ... }

  // This used to override Foo::bar with a warning, is now deprecated.
  predicate bar() { ... }

  // This correctly overrides Foo::baz
  override predicate baz() { ... }
}

Release 2.12.7 (2023-04-18)

Bugs fixed

Fixed a bug in codeql database upload-results where the subcommand would fail with "A fatal error occurred: Invalid SARIF.", reporting an InvalidDefinitionException. This issue occurred when the SARIF file contained certain kinds of diagnostic information.

Release 2.12.6 (2023-04-04)

Known issues

We recommend that customers using the CodeQL CLI in a third party CI system do not upgrade to this release, due to an issue with codeql github upload-results. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1.

This issue occurs when uploading certain kinds of diagnostic information and causes the subcommand to fail with "A fatal error occurred: Invalid SARIF.", reporting an InvalidDefinitionException.

Customers who wish to use CodeQL 2.12.6 or 2.13.0 can work around the problem by passing --no-sarif-include-diagnostics to any invocations of codeql database analyze or codeql database interpret-results.

New features

Several experimental subcommands have been added in support of the new code scanning tool status page. These include codeql database add-diagnostic, codeql database export-diagnostics, and the codeql diagnostic add and codeql diagnostic export plumbing subcommands.

Bugs fixed

Fixed a bug in codeql database analyze and related commands where the --max-paths option was not respected correctly when multiple alerts with the same primary code location were grouped together. (This grouping is the default behavior unless the --no-group-alerts option is passed.) This bug caused some SARIF files produced by CodeQL to exceed the limits on the number of paths (threadFlows) accepted by code scanning, leading to errors when uploading results.

Release 2.12.5 (2023-03-21)

New features

The codeql pack install command now accepts a --additional-packs option. This option takes a list of directories to search for locally available packs when resolving which packs to install. Any pack that is found locally through --additional-packs will override any other version of a pack found in the package registry. Locally resolved packs are not added to the lock file.

Because the use of --additional-packs when running codeql pack install makes running queries dependent on the local state of the machine initially invoking codeql pack install, a warning is emitted if any pack is found outside of the package registry. This warning can be suppressed by using the --no-strict-mode option.

Bugs fixed

Fix a bug in codeql query run where queries whose path contain colons cannot be run.

Release 2.12.4 (2023-03-09)

Breaking changes

The default value of the --mode switch to codeql pack install has changed. The default is now --mode minimal-update. Previously, it was use-lock.

New features

The per-pack compilation cache has been replaced with a global compilation cache found within ~/.codeql.
codeql pack install now uses a new algorithm to determine which versions of the pack's dependencies to use, based on the PubGrub algorithm. The new algorithm is able to find a solution for many cases that the previous algorithm would fail to solve. When the new algorithm is unable to find a valid solution, it generates a detailed error message explaining why there is no valid solution.
Added a new command, codeql pack upgrade. This command is similar to codeql pack install, except that it ignores any existing lock file, installs the latest compatible version of each dependency, and writes a new lock file. This is equivalent to codeql pack install --mode update. Note that the --mode switch to codeql pack install is now deprecated.
Added a new command, codeql pack ci. This command is similar to codeql pack install, except if the existing lock file is missing, or if it conflicts with the version constraints in the qlpack.yml file, the command generates an error. This is equivalent to codeql pack install --mode verify. Note that the --mode switch to codeql pack install is now deprecated.

Deprecations

The --freeze switch for codeql pack create, codeql pack bundle, and codeql pack publish is now deprecated and ignored, as there is no longer a cache within a pack.
The --mode update switch to codeql pack resolve-dependencies is now deprecated. Instead, use the new --mode upgrade switch, which has identical behavior.
The --mode switch to codeql pack install is now deprecated.
- Instead of --mode update, use codeql pack upgrade.
- Instead of --mode verify, use codeql pack ci.

Release 2.12.3 (2023-02-23)

New features

The CodeQL compiler now produces better error messages when it is unable to find a QL library that the query being evaluated depends on.

Bugs fixed

Fixed a bug where the CLI would refuse to complete database creation if the OS reports less than about 1.5 GB of physical memory. Now an attempt will be made even on low-memory systems (but it might still run out of memory unless there's swap space available).

Release 2.12.2 (2023-02-07)

Bugs fixed

Fixed a QL evaluator bug introduced in release 2.12.1 which could in certain rare cases lead to wrong analysis results.
Fixed handling of -Xclang <arg> arguments passed to the clang compiler which could cause missing extractions for C++ code bases.
Fixed a bug where the --overwrite option was failing for database clusters.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.6.

Release 2.12.1 (2023-01-23)

New features

Added a new command-line flag --expect-discarded-cache, which gives a hint to the evaluator that the evaluation cache will be discarded after analysis completes. This allows it to avoid some unnecessary writes to the cache, for predicates that aren't needed by the query/suite being evaluated.

Release 2.12.0 (2023-01-10)

Breaking changes

The --[no-]count-lines option to codeql database create and related commands that was deprecated in 2.11.1 has been removed. Users of this option should instead pass --[no-]calculate-baseline.

New features

Query packs created by codeql pack create, codeql pack bundle, and codeql pack release now contain precompiled queries in a new format that aims to be compatible with future (and, to a certain extent, past) releases of the CodeQL CLI. Previously the precompiled queries were in a format specific to each CLI release, and all other releases would need to re-compile queries.

Published packs contain precompiled queries in files with a .qlx extension located next to each query's .ql source file. In case of differences between the .ql and .qlx files, the .qlx file takes priority when evaluating queries from the command line, so if you need to modify a published pack, be sure to delete the .qlx files first.

A new --precompile flag to codeql query compile can be used to construct *.qlx file explicitly, but in all usual cases it should be enough to rely on codeql pack create doing the right thing.
The codeql database init command now accepts a PAT that allows you to download queries from external, private repositories when using the --codescanning-config <config-file> option. For example, you can specify the following queries block in the config file, which will checkout the main branch of the codeql-test/my-private-repository repository and evaluate any queries found in that repository:
```
queries:
  - codeql-test/my-private-repository@main
```
If the repository is private, you can add a --external-repository-token-stdin option and supply a PAT with appropriate permissions via standard input. For more information on queries and external repositories in Code Scanning, see Using queries in QL packs.
The baseline information produced by codeql database init and codeql database create now accounts for paths and paths-ignore configuration.
In the VS Code extension, recursive calls will be marked with inlay hints. These can be disabled with the global inlay hints setting (editor.inlayHints.enabled). If you just want to disable them for codeql the settings can be scoped to just codeql files (language id is ql). See Language Specific Editor Settings in the VS Code documentation for more information.
The CLI now gives a more helpful error message when asked to run queries on a database that has not been finalized.

Bugs fixed

Fixed a bug where the codeql pack install command would fail if a CodeQL configuration file is used and the --additional-packs option is specified.

Release 2.11.6 (2022-12-13)

Breaking changes

Java and Kotlin analyses in this release of the CLI and all earlier releases are incompatible with Kotlin 1.7.30 and later. To prevent code scanning alerts being spuriously dismissed, Java and Kotlin analyses will now fail when using Kotlin 1.7.30 or later.

If you are unable to use Kotlin 1.7.29 or earlier, you can disable Kotlin support by setting CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN to true in the environment.

Bugs fixed

Fixed a bug where it was not possible to run queries in CodeQL query packs for C# that use the legacy libraryPathDependencies property in their qlpack.yml file. The associated error message complained about undefined extensional predicates.

Release 2.11.5 (2022-12-07)

Bugs Fixed

Fixed a bug that could cause log summary generation to fail in vscode.

Release 2.11.4 (2022-11-24)

New features

Kotlin support is now in beta. This means that Java analyses will also include Kotlin code by default. Kotlin support can be disabled by setting CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN to true in the environment.

Potentially breaking changes

CodeQL 2.11.1 to 2.11.3 contained a bug in indirect build tracing on Windows when using codeql database init with the --trace-process-level flag. In these versions, when --trace-process-level was set to a value greater than zero, (or left at the default value of 1), CodeQL attempted to inject its build tracer at a higher level in the process tree than the requested process level. This could lead to errors of the form "No source code found" or "Process tree ended before reaching required level". From 2.11.4 onwards, the CodeQL build tracer is injected at the requested process level.

Deprecations

The --[no-]fast-compilation option to codeql test run is now deprecated.

Release 2.11.3 (2022-11-11)

Breaking changes

The codeql pack ls --format json deep plumbing command now returns only the name and version properties for each found pack.

Potentially breaking changes

codeql pack download, codeql pack install, and codeql pack add will ignore CodeQL packs with pre-release versions, unless the --allow-prerelease option is passed to the command. This brings these commands into alignment with codeql pack publish that will avoid publishing CodeQL packs with pre-release versions unless the --allow-prerelease option is specified. Pre-release versions have the following format: X.Y.Z-qualifier where X, Y, and Z are respectively the major, minor, and patch number. qualifier is the pre-release version. For more information about pre-releases, see the Semantic Versioning specification.

Deprecations

The --[no-]fast-compilation option to codeql query compile is now deprecated.

New features

codeql resolve files and codeql database index-files have a new --find-any option, which finds at most one match.

Miscellaneous

The build of Apache Commons Text that is bundled with the CodeQL CLI has been updated to version 1.10.0. While previous releases shipped with version 1.6 of the library, no part of the CodeQL CLI references the StringSubstitutor class that the recently disclosed CVE-2022-42889 vulnerability applies to. We therefore do not believe that running previous releases of CodeQL exposes users to this vulnerability.
The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.5.

Release 2.11.2 (2022-10-25)

Breaking changes

Bundling and publishing a CodeQL pack will no longer include nested CodeQL packs. If you want to include a nested pack in your published pack, then you must explicitly include it using the include property in the top-level qlpack.yml file.

For example, if your package structure looks like this:
```
qlpack.yml
nested-pack
   ∟ qlpack.yml
     query.ql
```
then the contents of nested-pack will not be included by default within the published package. To include nested-pack, add an entry like this to the top level qlpack.yml file:
```
include:
  - nested-pack/**
```

Bugs fixed

Using the --codescanning-config=<file> option in codeql database init will now correctly process the paths and pathsIgnore properties of the configuration file in a way that is identical to the behavior of the codeql-action. Previously, paths or pathsIgnore entries that end in /** or start with / were incorrectly rejected by the CLI.
Fixed a bug where the --compilation-cache option to codeql pack publish and codeql pack create was being ignored when creating a query pack. Now, the indicated cache is used when pre-compiling the queries in it.
Fixed a bug that would make the "Show DIL" command in the VSCode extension display nothing.

Other changes

Emit a detailed warning if package resolution fails, the legacy --search-path option is provided, and there is at least one referenced pack that does not use legacy package resolution. In this case, --additional-packs should be used to extend the search to additional directories, instead of --search-path.

Release 2.11.1 (2022-10-11)

Breaking changes

Pack installation using the CodeQL Packaging beta will now fail if a compatible version cannot be found. This replaces the previous behavior where codeql pack download and related commands would instead install the latest version of the pack in this situation.

Deprecations

The --[no-]count-lines option to codeql database create and related commands is now deprecated and will be removed in a future release of the CodeQL CLI (earliest 2.12.0). It is replaced by --[no-]calculate-baseline to reflect the additional baseline information that is now captured as of this release.

New features

Subcommands that compile QL accept a new --no-release-compatibility option. It does nothing for now, but in the future it will be used to control a trade-off between query performance and compatibility with older/newer releases of the QL evaluator.
codeql database analyze and related commands now support absolute paths containing the @ or : characters when specifying which queries to run. To reference a query file, directory, or suite whose path contains a literal @ or :, prefix the query specifier with path:, for example:
```
    codeql database analyze --format=sarif-latest --output=results <db> path:C:/Users/ci/workspace@2/security/query.ql
```

Bugs fixed

It is no longer an error to call codeql pack create <path> with a <path> option pointing to a file name. The CLI will walk up the directory tree and run the command in the first directory containing the qlpack.yml or codeql-pack.yml file.
Fixed a concurrency error observed when using codeql database import or codeql database finalize with multiple threads and multiple additional databases on a C++ codebase.

Release 2.11.0 (2022-09-28)

Deprecation

The CodeQL CLI now uses Python 3 to extract both Python 2 and Python 3 databases. Correspondingly, support for using Python 2 to extract Python databases is now deprecated. Starting with version 2.11.3, you will need to install Python 3 to extract Python databases.

Miscellaneous

The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.4.

Release 2.10.5 (2022-09-13)

New features

You can now define which registries should be used for downloading and publishing CodeQL packs on a per-workspace basis by creating a codeql-workspace.yml file and adding a registries block. For more infomation, see About CodeQL Workspaces.

Release 2.10.4 (2022-08-31)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
This release does not include any user-facing changes.

Release 2.10.3 (2022-08-15)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

New features

When called with --start-tracing, the codeql database init command now accepts extractor options for the indirect tracing environment via --extractor-option. Users should continue to specify extractor options for direct tracing environments by passing them to codeql database trace-command invocations.

Other changes

The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.4.

Release 2.10.2 (2022-08-02)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

Breaking change

The option --compiler-spec to codeql database create (and codeql database trace-command) no longer works. It is replaced by --extra-tracing-config, which accepts a tracer configuration file in the new, Lua-based tracer configuration format instead. See tools/tracer/base.lua for the precise API available. If you need help help porting your existing compiler specification files, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering.

Potentially breaking changes

Versions of the CodeQL extension for Visual Studio Code released before February 2021 may not work correctly with this CLI, in particular if database upgrades are necessary. We recommend keeping your VS Code extension up-to-date.

Deprecation

The experimental codeql resolve ml-models command has been deprecated. Advanced users calling this command should use the new codeql resolve extensions command instead.

New features

The codeql github upload-results command now supports a --merge option. If this option is provided, the command will accept the paths to multiple SARIF files, and will merge those files before uploading them as a single analysis. This option is recommended only for backwards compatibility with old analyses produced by the CodeQL Runner, which combined the results for multiple languages into a single analysis.

Release 2.10.1 (2022-07-19)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

New features

Improved error message from codeql database analyze when a query is missing @id or @kind query metadata.

Release 2.10.0 (2022-06-27)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

Breaking changes

The --format=stats option of codeql generate log-summary has been renamed to --format=overall. It now produces a richer JSON object that, in addition to the previous statistics about the run (which can be found in the stats property) also records the most expensive predicates in the evaluation run.

Potentially breaking changes

The codeql resolve ml-model command now requires one or more query specifications as command line arguments in order to determine the set of starting packs from which to initiate the resolution process. The command will locate all ML models in any qlpack that is a transitive dependency of any of the starting packs. Also, the output of the command has been expanded to include for each model the containing package's name, version, and path.
The buildMetadata inside of compiled CodeQL packs no longer contains a creationTime property. This was removed in order to ensure that the content of a CodeQL pack is identical when it is re-compiled.
The codeql pack download command, when used with the --dir option, now downloads requested packs in directories corresponding to their version numbers. Previously, codeql pack download --dir ./somewhere codeql/java-queries@0.1.2 would download the pack into the ./somewhere/codeql/java-queries directory. Now, it will download the pack into the ./somewhere/codeql/java-queries/0.1.2 directory. This allows you to download multiple versions of the same pack using a single command.

New features

You can now include diagnostic messages in the summary produced by the --print-diagnostics-summary option of the codeql database interpret-results and codeql database analyze commands by running these commands at high verbosity levels.

Bugs fixed

Fixed a bug where codeql pack download, when used with the --dir option, would not download a pack that is in the global package cache.
Fixed a bug where some versions of a CodeQL package could not be downloaded if there are more than 100 versions of this package in the package registry.
Fixed a bug where the --also-match option for codeql resolve files and codeql database index-files does not work with relative paths.
Fixed a bug that caused codeql query decompile to ignore the --output option when producing bytecode output (--kind=bytecode), writing only to stdout.

Release 2.9.4 (2022-06-20)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

New features

Users of CodeQL Packaging Beta can now optionally authenticate to Container registries on GitHub Enterprise Server (GHES) versions 3.6 and later using standard input instead of the CODEQL_REGISTRIES_AUTH environment variable. To authenticate via standard input, pass --registries-auth-stdin. The value you provide will override the value of the CODEQL_REGISTRIES_AUTH environment variable.

Release 2.9.3 (2022-05-31)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

New features

Users can now use CodeQL Packaging Beta to publish and download CodeQL packs on GitHub Enterprise Server (GHES) versions 3.6 and later.

To authenticate to a package registry on GHES 3.6+, first create a ~/.codeql/qlconfig.yml file. For example, the following file specifies that all CodeQL packages should be uploaded to the GHES instance with the hostname GHE_HOSTNAME:
```
registries:
- packages: '*'
  url: https://containers.GHE_HOSTNAME/v2/
```
You can now download public packages from GHES using codeql pack download.

To publish any package or download private packages, authenticate to GHES by specifying registry/token pairs in the CODEQL_REGISTRIES_AUTH environment variable. You can authenticate using either a GitHub Apps token or a personal access token. For example, https://containers.GHEHOSTNAME1/v2/=TOKEN1,https://containers.GHEHOSTNAME2/v2/=TOKEN2 will authenticate the CLI to the GHEHOSTNAME1 and GHEHOSTNAME2 GHES instances.

Bugs Fixed

Fixed a bug where precompiled CodeQL packages in the CodeQL bundle were being recompiled if they were in a read-only directory.
Fixed a bug where new versions of the VS Code extension wouldn't run two queries in parallel against one database.

Release 2.9.2 (2022-05-16)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

Features removed

The table printed by codeql database analyze to summarize the results of metric queries that were part of the analysis now reports a single row per metric name independently of the verbosity level of the command. Previously, at higher verbosity levels, this table would contain multiple rows for metric names with multiple values.

New features

The tables produced by codeql database analyze summarizing the results of any diagnostic and metric queries that were run now exclude the results of queries tagged telemetry.
Uploading SARIF results using the codeql github upload-results command now has a timeout of 5 minutes.
Downloading CodeQL packs using the codeql pack download, codeql pack install and related commands now have a timeout of 5 minutes and will retry 3 times before failing. Similar behavior has been added to the codeql pack publish command.
The codeql generate log-summary command will now print progress updates to stderr.

Bugs fixed

Fixed a bug that could make it unpredictable whether the QL compiler reports problems about query metadata tags, and thereby make codeql test run fail spuriously in some cases.

Release 2.9.1 (2022-05-05)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

Release 2.9.0 (2022-04-26)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.

New features

codeql database create now supports the --[no-]-count-lines option, which was previously only available with codeql database init.
codeql resolve files and codeql database index-files has a new --also-match option, which allows users to specify glob patterns that are applied in conjunction with the existing --include option.

New language features

This release introduces experimental support for parameterized QL modules. This language feature is still subject to change and should not be used in production yet.

Bugs fixed

Fixed a bug that would prevent resolution of a query suite in a published CodeQL query pack that has a reference to the pack itself.
Fixed inaccurate documentation of what the --include-extension option to codeql resolve files and codeql database index-files does. The actual behavior is unchanged.

Release 2.8.5 (2022-04-07)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
There are no user-facing changes in this release.

Release 2.8.4 (2022-03-29)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.

Bugs fixed

Fixed an error where running out of memory during query evaluation would cause codeql to exit with status 34 instead of the 99 that is documented for this condition.
Fixed a bug in our handling of Clang's header maps, which caused missing files for Xcode-based projects on macOS (e.g. WebKit).

Release 2.8.3 (2022-03-14)

This release of CodeQL (and all future ones) will not include the CodeQL runner, which is now deprecated. For more information, and instructions on how to migrate to using the CodeQL CLI, see CodeQL runner deprecation.
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.

New features

Executable binaries for Windows are now digitally signed by a GitHub certificate.

Other changes

The evaluator logs produced by --evaluator-log now default to the maximum verbosity level and will therefore contain more information (and, accordingly, grow larger). The verbosity level can still be configured with --evaluator-log-level. In particular, --evaluator-log-level=1 will restore the previous default behavior.

Release 2.8.2 (2022-02-28)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.

Breaking change

The support for the output formats SARIF v1.0.0 and SARIF v2.0.0 (Committee Specification Draft 1) that were deprecated in 2.7.1 has been removed. If you need this functionality, please file a public issue against https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub Support and request an escalation to engineering.

New Features

The CodeQL CLI is now compatible with Windows 11 and Windows Server 2022, including building databases for compiled languages.

Release 2.8.1 (2022-02-15)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.

New Features

Commands that find or run queries now allow you to refer to queries within a named CodeQL pack. For example:

# Analyze a database using all queries in the experimental/Security folder within the codeql/cpp-queries
# CodeQL query pack.
codeql database analyze --format=sarif-latest --output=results <db> \
    codeql/cpp-queries:experimental/Security

# Analyse using only the RedundantNullCheckParam.ql query in the codeql/cpp-queries CodeQL query pack.
codeql database analyze --format=sarif-latest --output=results <db> \
    'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql'

# Analyse using the cpp-security-and-quality.qls query suite in the codeql/cpp-queries CodeQL query pack.
codeql database analyze --format=sarif-latest --output=results <db> \
    'codeql/cpp-queries:codeql-suites/cpp-security-and-quality.qls'

# Analyse using the cpp-security-and-quality.qls query suite from a version of the codeql/cpp-queries pack
# that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen).
# All valid semver ranges are allowed. See https://docs.npmjs.com/cli/v6/using-npm/semver#ranges
codeql database analyze --format=sarif-latest --output=results <db> \
    'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls'

The complete way to specify a set of queries is in the form scope/name@range:path, where:

scope/name is the qualified name of a CodeQL pack.
range is a semver range.
path is a file system path

If a scope/name is specified, the range and path are optional. A missing range implies the latest version of the specified pack. A missing path implies the default query suite of the specified pack.

The path can be one of a *.ql query file, a directory containing one or more queries, or a .qls query suite file. If there is no pack name specified, then a path must be provided, and will be interpreted relative to the current working directory of the current process.

If a scope/name and path are specified, then the path cannot be absolute. It is considered relative to the root of the CodeQL pack.

The relevant commands are:
- codeql database analyze
- codeql database run-queries
- codeql execute queries
- codeql resolve queries

Bugs fixed

Fixed a bug that would sometimes lead to query evaluation on M1-based Macs to crash with Did not preallocate enough memory error.

Release 2.8.0 (2022-02-04)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.

Breaking change

The CodeQL Action versions up to and including version 1.0.22 are not compatible with the CodeQL CLI 2.8.0 and later. The CLI will emit an error if it detects that it is being used by an incompatible version of the codeql-action.

New features

A new extractor option has been added to the Java extractor. The flag --extractor-option exclude='<glob>' allows specifying a glob that describes which paths need to be excluded from extraction but still need to be compiled. This is useful when some files are necessary for a successful build but are uninteresting for analysis.

See also: https://codeql.github.com/docs/codeql-cli/extractor-options/
Summary metrics can now associate messages with their results, for instance to report the name and number of uses of a particular API endpoint within a repository. To associate messages with summary metrics, define a query with @kind metric and @tags summary metadata and use either the location, message, value or the message, value results pattern.

Bug fixed

Fixed a bug where codeql resolve upgrades ignores the --target-dbscheme option.

Release 2.7.6 (2022-01-24)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Bug fixed

A bug where creation of a CodeQL database could sometimes fail with a NegativeArraySizeException has now been fixed.

New feature

The CLI and evaluator contain a number of new features in support of internal machine learning experiments. This includes an experimental resolve ml-models subcommand and new mlModels metadata in pack definition files. As these new features are not yet ready for general use, they should be ignored by external CodeQL users.

Release 2.7.5 (2022-01-17)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Deprecation

The CodeQL Action versions up to and including version 1.0.22 are now deprecated for use with CodeQL CLI 2.7.5 and later. The CLI will emit a warning if it detects that it is being used by a deprecated version of the codeql-action. This warning will become a fatal error with version 2.8.0 of the CLI.

New feature

The codeql github upload-results command will now print the API response body in JSON format if a --format=json flag is given. Otherwise the command will print the URL of the SARIF upload. This URL can be used to get status information for the upload.

See also: https://docs.github.com/en/rest/reference/code-scanning

Documentation fixes

The documentation for the --trace-process-level flag of codeql database init (which is used with indirect build tracing on Windows) was erroneous.

The help text previously claimed that --trace-process-level=1 would inject CodeQL's build tracer into the calling process. This is actually what --trace-process-level=0 achieves. The help text has now been corrected to match the actual (unchanged) behavior.

Also, some log messages incorrectly stated which process CodeQL was injected into. These have also been corrected.

Other changes

For commands that run queries, the --timeout option now controls the maximal time it may take to evaluate a "layer" of a query rather than a "stage". There are usually many "layers" in each "stage", but it is usually a single one of the layers in a stage that uses most of the time, so there is no need to reduce existing timeout values as a result of this change.

Release 2.7.4

This release was skipped.

Release 2.7.3 (2021-12-06)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Potentially breaking changes

The experimental command-line option --ml-model-path that was introduced to support internal experiments has been removed.

Bugs fixed

Editing support (content assist, code navigation, etc.) in files under the .github directory will now work properly. This is because files under the .github directory will now be indexed and processed by the CodeQL language server. Other hidden directories that start with . will remain un-indexed. This affects the vscode-codeql extension and any other IDE extension that uses the CodeQL language server.
Fixed authentication with GitHub package registries via the GITHUB_TOKEN environment variable and the --github-auth-stdin flag when downloading and publishing packs.
Fixed an incompatibility with glibc version 2.34 on Linux, where build tracing failed with an error message.
Fixed a bug where codeql generate log-summary could sometimes fail with a JsonMappingException.

New features

The CodeQL CLI for Mac OS now ships with a native Java virtual machine for M1 Macs, and this will be used by default where applicable to run the CodeQL engine, thus improving performance. Rosetta 2 is still required as not all components of the CodeQL CLI are natively compiled.
Commands that execute queries will now exit with status code 34 if certain errors that prevent the evaluation of one or more individual queries are detected. Previously some of these errors would crash the evaluator and exit with status code 100.

(This is currently used for "external predicate not found" errors).

Release 2.7.2 (2021-11-22)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Potentially breaking changes

The Java extractor now defaults to extracting all XML documents under 10MB in size, a change from the previous default of only extracting documents with particular well-known names (e.g. pom.xml). However, if the source tree contains more than 50MB of XML in total, it prints a warning and falls back to the old default behaviour. Set the environment variable LGTM_INDEX_XML_MODE to byname to get the old default behaviour, or all to extract all documents under 10MB regardless of total size.
The experimental command-line option --native-library-path that was introduced to support internal experiments has been removed.
The beta codeql pack publish command will now prevent accidental publishing of packages with pre-release version qualifiers. Prerelease versions are those that include a - after the major, minor, and patch versions such as 1.2.3-dev. To avoid this change, use the --allow-prerelease option.

Bugs fixed

Fixed an issue when using the --evaluator-log option where a NullPointerException could sometimes occur non-deterministically.
Fixed bugs observed when using indirect build tracing using a CodeQL distribution unpacked to a path containing spaces or on Arch Linux.

New features

CodeQL databases now contain metadata about how and when they were created. This can be found in the creationMetadata field of the codeql-database.yml file within the CodeQL database directory. More information may be added to this field in future releases.

Release 2.7.1 (2021-11-15)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Potentially breaking changes

Previously, codeql test run would fall back to looking for an accompanying queries.xml file if it found a qlpack.yml that did not declare an extractor to use when extracting a test database. This has been removed because the internal use case that neccessitated the fallback are now removed. If you suddenly encounter errors that complain of missing extractor declarations, check whether you had a queries.xml you were inadvertently relying on.
When queries are specified by naming a directory to scan for *.ql files, subdirectories named .codeql will now be ignored. The new QL packaging support uses subdirectories with this name of various scratch and caching purposes, so they may contain *.ql files that are not intended to be directly user-visible.
When copying dependencies for CodeQL packages into a query pack bundle, *.ql files in these dependencies will now be included inside of the query pack's .codeql directory.
The tables printed by codeql database analyze to summarize the results of diagnostic and metric queries that were part of the analysis have a new format and contains less (but hopefully more pertinent) information. We recommend against attempting to parse this human-readable output programmatically. Instead, use the runs[].tool.driver.invocations[].toolExecutionNotifications property in the SARIF output.
The experimental plumbing command codeql pack packlist has a new format for its JSON results. Previously, the results were a list of paths. Now, the results are an object with a single property paths that contains the list of paths.

The internal qlpacks directory of the CodeQL bundle available on the CodeQL Action releases page has a new structure. This directory is internal to the CLI and can change without notice in future releases.

The currently-shipped qlpacks directory mirrors the structure of CodeQL package caches and looks like this:

qlpacks
  - codeql
    - {lang}-all
      - {version}
        - qlpack contents
    - {lang}-examples
      - {version}
        - qlpack contents
    - {lang}-queries
      - {version}
        - qlpack contents
    - {lang}-upgrades
      - {version}
        - qlpack contents
    - ... and so on for all languages

Deprecations

The output formats SARIF v1.0.0 and SARIF v2.0.0 (Committee Specification Draft 1) have been deprecated. They will be removed in a later version (earliest 2.8.0). If you need this functionality, please file a public issue against https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub Support and request an escalation to engineering.
The qlpack: instruction in query suite definitions has been deprecated due to uncertainty about whether it is intended to include all the *.ql files in the named pack, or only the pack's "default query suite". The behavior of the instruction is determined by whether the named pack declares any default query suite, but this means that a pack starting to declare such a suite may break the behavior of existing query suites that reference the pack from outside.

We recommend replacing qlpack: by one of
```
- queries: '.' # import all *.ql files
  from: some/pack-name
  version: 1.2.3 # optional
```
or
```
- import: path/to/actual/suite.ql # just that suite
  from: some/pack-name
  version: 1.2.3 # optional
```
A warning will now be printed when a qlpack: instruction resolves to a default suite, because that is the case where the effect may not be what the query suite author intended.

Bugs fixed

Fixed a bug where the paths and paths-ignore properties of a Code Scanning config file specified using --codescanning-config were being interpreted the wrong way around.
Fixed a bug where queries specified using the --codescanning-config option could not be run after an explicit call to codeql database finalize.
Fixed a bug where -J options would erroneously be recognized even after -- on the command line.
When running codeql database analyze and codeql database interpret-results without the --sarif-group-rules-by-pack flag, the SARIF output did not include baseline lines-of-code counts. This is now fixed.
Fixed a bug where expansion of query suites would sometimes fail if a query suite in a compiled query pack referenced that pack itself explicitly.

New language features

Set literal expressions can now optionally contain a trailing comma after the last element.

New features

Beta support for database creation on Apple Silicon has been added. It depends on the following requirements:
- Rosetta 2 needs to be installed
- Developer tools need to be installed. CodeQL requires the lipo, codesign, and install_name_tool tools to be present.
- Build systems invoking csh may experience intermittent crashes.
codeql database analyze can now include query-specific help texts for alerts in the SARIF output (for SARIF v2.1.0 or later). The help text must be located in an .md file next to (and with the same basename as) the .ql file for each query. Since this can significantly increase SARIF file size, the feature is not enabled by default; give a --sarif-add-query-help option to enable it.
The query metadata validator now knows about queries that produce alert scores, so these queries no longer need to be run with a --no-metadata-verification flag.
codeql database create and codeql-finalize have a new flag --skip-empty that will cause a language with no extracted source code to be ignored with a warning instead of treated like a fatal error. This can be useful with --db-cluster where not all of the languages may exist in the source tree. It will not be possible to run queries against the skipped database.
codeql resolve extractor and codeql resolve languages now support an extended output format --format=betterjson wich includes information about each extractor's language-specific options.
This release introduces rudimentary support for parallelizing database creation by importing unfinished databases (or database clusters) into another unfinished database (or cluster) under creation. This is implemented by the new flag --additional-dbs for codeql database finalize, or the new plumbing command codeql database import.
codeql database create, codeql database index-files, and codeql database trace-command support a unified syntax for passing language-specific options to the extractor with the new --extractor-option and --extractor-options-file options. (The extractors do not make use of this yet, though).

Release 2.7.0 (2021-10-27)

The extractor for Ruby is now included. CodeQL analysis for Ruby is currently in beta. During the beta, analysis of Ruby will not be as comprehensive as CodeQL analysis of other languages. The source code of the extractor and the queries can be found in the github/codeql repository.
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Bugs fixed

Fixed a bug where indirect tracing would sometimes not manage to observe build processes if certain environment variables were unset during the build.

Release 2.6.3 (2021-10-06)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Potentially breaking changes

The option --compiler-spec accepted by some subcommands of codeql database is deprecated. It will be removed in a later version (earliest 2.7.0). If you need this option, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering.
By default, databases created using the CodeQL CLI will now have their underlying datasets finalized, meaning that no further data can be subsequently imported into them. This change should not affect most users.
The codeql resolve qlref command will now throw an error when the target is ambiguous. The qlref resolution rules are now as follows:
1. If the target of a qlref is in the same qlpack, then that target is always returned.
2. If multiple targets of the qlref are found in dependent packs, this is an error.
Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities.

Bugs fixed

Linux/MacOS: When tracing a build that involves an execvp/execvpe (Linux-only)/posix_spawnp syscall where PATH was not set in the environment, CodeQL sometimes would break the build. Now, CodeQL uses the correct, platform-specific fallback for PATH instead.
Linux/MacOS: When tracing a build that involves an execvpe (Linux-only)/posix_spawnp syscall, the PATH lookup of the executable wrongly took place in the environment provided via envp, instead of the environment of the process calling execvpe/posix_spawnp. Now, the correct environment is used for the PATH lookup.
A bug where query compilation would sometimes fail with a StackOverflowError when compiling a query that uses instanceof has now been fixed.

New features

The codeql query compile command now accepts a --keep-going or -k option, which indicates that the compiler should continue compiling queries even if one of the queries has a compile error in it.
CLI commands now run default queries if none are specified. If no queries are specified, the codeql database analyze, codeql database run-queries, and codeql database interpret-results commands will now run the default suite for the language being analyzed.
codeql pack publish now copies the published package to the local package cache. In addition to publishing to a remote repository, the codeql pack publish command will also copy the published package to the local package cache.

Release 2.6.2 (2021-09-21)

CodeQL CLI 2.6.2 includes the same functionality as the CodeQL runner, which is being deprecated. For more information, see CodeQL runner deprecation.
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Bugs fixed

A bug where codeql generate log-summary would sometimes crash with a JsonMappingException has been fixed.

New features

The CodeQL CLI now counts the lines of code found under --source-root when codeql database init or codeql database create is called. This information can be viewed later by either the new codeql database print-baseline command or the new --print-baseline-loc argument to codeql database interpret-results.
qlpack.yml files now support an additional field include in which glob patterns of additional files that should be included (or excluded) when creating a given CodeQL pack can be specified.
QL packs created by the experimental codeql pack create command will now include some information about the build in a new buildMetadata field of their qlpack.yml file.
codeql database create now supports the same flags as codeql database init for automatically recognizing the languages present in checkouts of GitHub repositories:
- --github-url accepts the URL of a custom GitHub instance (previously only github.com was supported).
- --github-auth-stdin allows a personal access token to be provided through standard input (previously only the GITHUB_TOKEN environment variable was supported).

Notable documentation changes

Documentation has been added detailing how to use the "indirect build tracing" feature, which is enabled by using the --begin-tracing flag provided by codeql database init. The new documentation can be found here. This feature was temporarily described as "sandwiched tracing" in the 2.6.0 release notes.

Release 2.6.1 (2021-09-07)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Potentially breaking changes

The codeql resolve qlref command will now throw an error when the target is ambiguous.

The qlref resolution rules are now as follows:
1. If the target of a qlref is in the same qlpack, then that target is always returned.
2. If multiple targets of the qlref are found in dependent packs, this is an error.
Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities.
The qlpack directive in query suites has its semantics changed. Previously, this directive would return all queries in the qlpack. Now, the directive returns only those queries matched by the defaultSuite directive in the query pack. Here is an example:

Consider a qlpack.yml like the following:
```
name: codeql/my-qlpack
version: 0.0.1
defaultSuite:
  queries: standard
```
And the directory structure is the following:
```
qlpack.yml
standard/
  a.ql
experimental/
  b.ql
```
A query suite suite.qls like this:
```
- qlpack: codeql/my-qlpack
```
Previously, would return all the queries in all subdirectories (i.e, standard/a.ql and experimental/b.ql). Now, it only returns standard/a.ql, since that is the only query matched by its default suite.

If you want to have the same behavior as before, you must update your query suites to use the queries directive with a from attribute, like this:
```
- queries: .
  from: codeql/my-qlpack
```

New features

Commands that evaluate CodeQL queries now support an additional option --evaluator-log=path/to/log.json that will result in the evaluator producing a structured log (in JSON format) of events that occurred during evaluation in order to aid debugging of query performance. The format of these logs will be subject to change with no notice as we make modifications to the evaluator.

There is also a new CLI command codeql generate log-summary that will produce a summary of the predicates that were evaluated from these event logs. We will aim to keep this summary format more stable, although it is also subject to change. Unless you have a good reason to use the event logs directly, it is strongly recommended you use this command to produce summary logs and use these instead.

For further information on these new logs and additional options to configure their format and verbosity, please refer to the CLI documentation.

New language features

QL classes can now be non-extending subtypes via the instanceof keyword, allowing for a form of private subtyping that is not visible externally. Methods of the supertype are accessible from within a non-extending subtype class through extended semantics of the super keyword.
```
class Foo instanceof int {
  Foo() { this in [1 .. 10] }
  string toString() { result = "foo" + super.toString() }
}
```

Release 2.6.0 (2021-08-24)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.

Bugs fixed

The physicalLocation.artifactLocation.uri fields in SARIF output are now properly encoded as specified by RFC 3986.
The --include-extension option to the codeql database index-files command no longer includes directories that are named with the provided extension. For example, if the option --include-extension=.rb is provided, then a directory named foo.rb/ will be excluded from the indexing.

New features

A new codeql database unbundle subcommand performs the reverse of codeql database bundle and extracts a CodeQL database from an archive.
The CLI now understands per-codebase configuration files in the format already supported by the CodeQL Action. The configuration file must be given in a --codescanning-config option to codeql database create or codeql database init. For some languages, this configuration can contain pathname filters that control which parts of the codebase is analysed; the configuration file is the only way this functionality is exposed. The configuration file can also control which queries are run, including custom queries from repositories that must first be downloaded. To actually use those queries, run codeql database analyze without any query-selection arguments.
The CLI now supports the "sandwiched tracing" feature that has previously only been offered through the separate CodeQL Runner. This feature is intended for use with CI systems that cannot be configured to wrap build actions with codeql database trace-command. Instead the CI system must be able to set custom environment variables for each build action; the required environment variables are output by codeql database init when given a --begin-tracing argument.

On Windows, codeql database init --begin-tracing will also inject build-tracing code into the calling process or an ancestor; there are additional options to control this.
This version contains beta support for a new packaging and publishing system for third-party QL queries and libraries. It comprises the following new commands:
- codeql pack init: Creates an empty CodeQL pack from a template.
- codeql pack add: Adds a dependency to a CodeQL pack.
- codeql pack install: Installs all pack dependencies specified in the qlpack.yml file.
- codeql pack download: Downloads one or more pack dependencies into the global package cache.
- codeql pack publish: Publishes a package to the GitHub Container Registry.
- (Plumbing) codeql pack bundle: Builds a .zip file for a CodeQL query or library pack from sources. Used by codeql pack publish.
- (Plumbing) codeql pack create: Creates a compiled CodeQL query or library pack from sources. Used by codeql pack bundle.
- (Plumbing) codeql pack packlist: Lists all files in a local CodeQL pack that will be included in the pack's bundle. Used by codeql pack create.
- (Plumbing) codeql pack resolve-dependencies: Resolves all transitive dependencies of a local CodeQL pack. Used by codeql pack install.

Release 2.5.9 (2021-08-09)

This release corresponds to release 1.28.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.6.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

Release 2.5.8 (2021-07-26)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

The QL compiler now verifies that @security-severity query metadata is numeric. You can disable this verification by passing the --no-metadata-verification flag.

New features

The database index-files and database trace-command CLI commands now support --threads and --ram options, which are passed to extractors as suggestions.
The database finalize CLI command now supports the --ram option, which controls memory usage for finalization.
The database create CLI command now supports the --ram option, which controls memory usage for database creation. - The generate query-help CLI command now support rendering query help in SARIF format.

Release 2.5.7 (2021-07-02)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

New features

codeql database create and codeql database init can now automatically recognise the languages present in checkouts of GitHub repositories by making an API call to the GitHub server. This requires a PAT token to either be set in the GITHUB_TOKEN environment variable, or passed by stdin with the --github-auth-stdin argument.
Operations that make outgoing HTTP calls (that is, codeql github upload-results and the language-detection feature described above) now support the use of HTTP proxies. To use a proxy, specify an $https_proxy environment variable for HTTPS requests or a $http_proxy environment variable for HTTP requests. If the $no_proxy variable is also set, these variables will be ignored and requests will be made without a proxy.

New language features

The QL language now has a new method toUnicode on the int type. This method converts Unicode codepoint to a one-character string. For example, 65.toUnicode() = "A", 128512.toUnicode() results in a smiley, and any(int i | i.toUnicode() = "A") = 65.

Release 2.5.6 (2021-06-22)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Features added

codeql database create (and the plumbing commands it comprises) now supports creating databases for a source tree with several languages while tracing a single build. This is enabled by a new --db-cluster option. Once created, the multiple databases must be analyzed one by one.
codeql database create and codeql database init now accept an --overwrite argument which will lead existing CodeQL databases to be overwritten.
codeql database analyze now supports "diagnostic" queries (tagged @kind diagnostic), which are intended to report information about the analysis process itself rather than problems with the analyzed code. The results of these queries will be summarized in a table printed to the terminal when codeql database analyze finishes.

They are also included in the analysis results in SARIF output formats as notification objects so they can be displayed by subsequent tooling such as the Code Scanning user interface.
- For SARIF v2.1.0, a reporting descriptor object for each diagnostic query is output to output to runs[].tool.driver.notifications, or runs[].tool.extensions[].notifications if running with --sarif-group-rules-by-pack. A rule object for each diagnostic query is output to runs[].resources[].rules for SARIF v2, or to runs[].rules for SARIF v1.
- Results of diagnostic queries are exported to the runs[].invocations[].toolExecutionNotifications property in SARIF v2.1.0, the runs[].invocations[].toolNotifications property in SARIF v2, and the runs[].toolNotifications property in SARIF v1.
SARIF v2.1.0 output will now also contain version information for query packs in runs[].tool.extensions[].semanticVersion, if the Git commit the queries come from is known.
codeql github upload-results has a --checkout-path option which will attempt to automatically configure upload target parameters. When this is given, the --commit option will be taken from the HEAD of the checkout Git repository, and if there is precisely one remote configured in the local repository, the --repository and --github-url options will also be automatically configured.
The CodeQL C++ extractor includes beta support for C++20. This is only available when building codebases with GCC on Linux. C++20 modules are not supported.

Release 2.5.5 (2021-05-17)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

When scanning the disk for QL packs and extractors, directories of the form .../SOMETHING/SOMETHING.testproj (where the two SOMETHING are identical) will now be ignored. Names of this form are used by codeql test run for ephemeral test databases, which can sometimes contain files that confuse QL compilations.

Features added

Query writers can now optionally use @severity in place of @problem.severity in the metadata for alert queries. SARIF consumers should continue to consume this severity information using the rule.defaultConfiguration.level property for SARIF v2.1.0, and corresponding properties for other versions of SARIF. They should not depend on the value stored in the rule.properties property bag, since this will contain either @problem.severity or @severity based on exactly what was written in the query metadata.
When exporting analysis results to SARIF v2.1.0, results and metric results now contain a reporting descriptor reference object that specifies the rule that produced them. For metric results, this new property replaces the metric property.
codeql database analyze now outputs a table that summarizes the results of metric queries that were part of the analysis. This can be suppressed by passing the --no-print-metrics-summary flag.

Bugs fixed

When using the --sarif-group-rules-by-pack flag to place the SARIF rule object for each query underneath its corresponding query pack in runs[].tool.extensions, the rule property of result objects can now be used to look up the rule within the rules property of the appropriate query pack in runs[].tool.extensions. Previously, rule lookup for result objects in the SARIF output was not well-defined when the --sarif-group-rules-by-pack flag was passed.

Release 2.5.4 (2021-05-03)

This release is identical to release 2.5.3, except that codeql database analyze no longer produces a generated automationDetails.id field when the --sarif-category is not explicitly provided. Previously, the --sarif-category was autogenerated if not present.
Code Scanning users should upgrade to this version and avoid 2.5.3.

Release 2.5.3 (2021-04-30)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Next release: Features added

When tracing a C/C++ build, the C compiler entries in compiler-settings must now specify order compiler,extractor. The default configuration already does this, so no change is necessary if using the default configuration.
codeql database analyze and codeql database interpret-results now report the results of summary metric queries in the <run>.properties.metricResults property of the SARIF output. Summary metric queries describe metrics about the code analyzed by CodeQL. They are identified by the query metadata @kind metric and @tag summary. For example, see the lines of code summary metric query for C++.
codeql database analyze and codeql database interpret-results now calculate an automation ID and add it to the resulting SARIF. In SARIF v2.1.0, this field is runs[].automationDetails.id. In SARIF v2, this field is runs[].automationLogicalId. In SARIF v1, this field is runs[].automationId. By default, this automation ID will be derived from the database language and the operating system of the machine that performed the run. It can be set explicitly using a new --sarif-category option.
In query metadata, @kind alert and @kind path-alert are now recognized as (more accurate) synonyms of @kind problem and @kind path-problem, respectively.
Diagnostic queries are now permitted by the metadata verifier. They are identified by @kind diagnostic metadata. Currently the result patterns of diagnostic queries are not verified. This will change in a future CLI release.

Bugs fixed

Ensure the correct URL is generated during codeql github upload-results for GitHub Enterprise Server.

Release 2.5.2 (2021-04-21)

This release is identical to release 2.5.1, except that an internal incompatibility with the CodeQL action (and the codeql-runner that some customers use for CI integrations) has been fixed.

The fix does not affect any use cases where the CLI is downloaded from github/codeql-cli-binaries, so if you're seeing this release there, there's no need to upgrade from 2.5.1.

Release 2.5.1 (2021-04-19)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

The QL compiler will now reject queries where the query metadata (if present) at the top of the .ql file is inconsistent with the output format of the query. This check can be disabled by giving the --no-metadata-verification flag. (The flag already existed but has not had any effect until now.)

Bugs fixed

Environment variables required for Java extraction are now propagated by the tracer. This may resolve issues with tracing and extraction in the context of certain build systems such as Bazel.
A number of --check-CONDITION options to codeql database finalize and codeql dataset import designed to look for consistency errors in the intermediate "TRAP" output from extractors erroneously did nothing. They will now actually print warnings if errors are found. The warnings become fatal errors if the new --fail-on-trap-errors option is also given.

Features added

codeql resolve qlref is a new command that takes in a .qlref file for a CodeQL test case and returns the path of the .ql file it references.
codeql database analyze and codeql database interpret-results have a new --sarif-group-rules-by-pack option which will place the SARIF rule object for each query underneath its corresponding query pack in runs[].tool.extensions.
codeql database finalize and codeql dataset import have a new --fail-on-trap-errors option that will make database creation fail if extractors produce ill-formatted "TRAP" data for inclusion into a database. This is not enabled by default because some of the existing extractors have minor output bugs that cause the check to fail.
codeql database finalize and codeql dataset import have a new --check-undefined-labels option that enables stricter consistency checks on the "TRAP" output from extractors.

QL language improvements

super may now be used unqualified, e.g. super.predicateName(), when the declaring class has multiple super types, as long as the call itself is unambiguous.

Release 2.5.0 (2021-03-26)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

By default, codeql test now performs additional compiler checks when extracting test code written in Java. Existing Java tests that previously passed may therefore fail due to this change, if they do not compile using the javac compiler. To allow time to migrate existing tests, the new behavior can be disabled by setting the environment variable CODEQL_EXTRACTOR_JAVA_FLOW_CHECKS=false.

Features added

Log files that contain output from build processes will now prefix it with [build-stdout] and [build-stderr] instead of [build] and [build-err]. In particular the latter sometimes caused confusion.

QL language improvements

The QL language now recognizes new pragma[only_bind_into](...) and pragma[only_bind_out](...) annotations on expressions. Advanced users may use these annotations to provide hints to the compiler to influence binding behavior and thus indirectly performance.

Release 2.4.6 (2021-03-19)

This release corresponds to release 1.27.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.5.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

Fixed a bug in codeql test run that causes tests to fail messily if the freshly-extracted test database needed to be upgraded in order to be compatible with the QL source under test. This would happen more often at the end of a release cycle, after updates to the QL repository had happened.
codeql github upload-results should now work correctly against GitHub Enterprise Server instances that are configured with a path prefix.

Release 2.4.5 (2021-03-08)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.
The C/C++ extractor can now parse more Microsoft language extensions when in C++14 and C++17 mode.
codeql database analyze now reports the name and version of each QL pack used by the analysis. You can find this information in the SARIF output. In particular, the runs[0].tool.extensions property contains an object for each QL pack used by the analysis. Each object contains the name and semanticVersion of the corresponding QL pack, if such information is available.
codeql github upload-results is a new command that uploads a SARIF file generated by CodeQL to GitHub's Code Scanning.

Release 2.4.4 (2021-02-12)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.

Potentially breaking changes

The name property in qlpack.yml must now meet the following requirements:
- Only lowercase ASCII letters, ASCII digits, and hyphens (-) are allowed.
- A hyphen is not allowed as the first or last character of the name.
- The name must be at least one character long, and no longer than 128 characters.

New features

Alert and path queries can now give a score to each alert they produce. You can incorporate alert scores in an alert or path query by first adding the @scored property to the query metadata. You can then introduce a new numeric column at the end of the select statement structure to represent the score of each alert. Alert scores are exposed in the SARIF output of commands like codeql database analyze as the score property in the property bags of result objects.

Bugs fixed

The default value of the --working-dir options for the index-files and trace-command subcommands of codeql database has been fixed to match the documentation; previously, it would erroneously use the process' current working directory rather than the database source root.
codeql test run will not crash if database extraction in a test directory fails. Instead only the tests in that directory will be marked as failing, and tests in other directories will continue executing.

Release 2.4.3 (2021-01-29)

Fixes several bugs introduced in 2.4.2, related to searching the disk for QL packs:

In many cases the search would scan through more of the file system than it should. Often the only effect of this was that the scan would take longer (sometimes significantly longer) but in some corner cases it could lead to packs being found that shouldn't be found, which could lead to compilation failure if different versions of the same pack exist on disk.
The search would terminate a fatal error if it met a directory without read permission.
A provide entry in .codeqlmanifest.json that ended with * would erroneously not match a .codeqlmanifest.json in a subdirectory.

As a consequence of the latter fix, the semantics of .codeqlmanifest.json files has changed slightly: Directory names that start with a dot used to not be matched by the pattern elements * and **, whereas now even dotted directories match such a pattern element. The previous behavior was never documented, and only very few users have .codeqlmanifest.json files of their own in the first place, so this change is expected to have minimal practical effect.

Release 2.4.2 (2021-01-22)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.

Release 2.4.1 (2020-12-19)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.

Features added

codeql query format now checks all files rather than stopping after the first failure when the --check-only option is given.
codeql resolve database will produce a languages key giving the language the database was created for. This can be useful in IDEs to help describe the database and suggest default actions or queries. For databases created by earlier versions, the result will be a best-effort guess.
codeql database interpret-results can now produce Graphviz .dot files from queries with @kind graph.

Features removed

codeql test run had some special compatibility support for running unit tests for the "code duplication" extractor features of certain discontinued Semmle products. Those tests have since been removed from the public QL repository, so the compatibility support for them has been removed. This should not affect any external users (since the extractor feature in question was never supported by codeql database create anyway), but if you run codeql test run against the unit tests belonging to an old checkout of the repository, you may now see some failures among Metrics tests.

Release 2.3.4 (2020-12-15)

This release corresponds to release 1.26.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.4.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we recommend that you upgrade to CLI releases numbered 2.4.x or later.

Release 2.4.0 (2020-11-25)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.
Much of the work done by codeql database upgrade now happens implicitly (and reversibly) as part of ordinary query evaluation. This should make the need to explicitly run codeql database upgrade much less common. However there are still some corner cases that will require it, particularly for very old databases.
codeql test run with a --threads argument will now compile test queries in parallel even if they belong to the same single test directory. This can speed up localized testing considerably.

Release 2.3.3 (2020-11-06)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.
Fixed bug in codeql test run where the --format and --failing-exitcode options would not work reliably when --ram was also given
The $CODEQL_JAVA_HOME environment variable will now be passed to extractors such that extractors implemented in Java can be affected too. Beware that this variable will override the JVM that executes the main codeql process. It should not normally be set explicitly.

Release 2.3.2 (2020-10-27)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

Release 2.3.1 (2020-10-15)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

Features added

codeql database create now accepts a --working-dir option, which allows the working directory for extractor scripts to differ from the source root. This is useful in some specialized integration situations.
codeql database create will now pass a --compiler-spec option on to codeql database trace-command. This allows adapting the build tracing process when unusual compiler toolchains are used.
codeql database init accepts an --allow-missing-source-root option, which is useful in some specialized integration situations.

Release 2.3.0 (2020-09-28)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

Potentially breaking changes

The Java extractor no longer supports builds running on a Java 6 JRE. The minimum supported version is Java 7.
The interpretation of binding set annotations in QL has changed subtly. In rare cases, existing QL code that contains explicit binding set annotations on overriding class predicates may now be rejected with errors of the form "... is not bound to a value". You can fix this by adding explicit binding sets to the overridden predicate, or to the abstract class itself in the case of the characteristic predicate. For more information about binding sets, see Annotations in the QL language reference.

QL language improvements

You can now use binding sets on class bodies. This lets you explicitly annotate dynamically dispatched characteristic predicates.

New features

Query authors can use the new subcommand codeql generate query-help to validate query help files and render the files as Markdown. For more information, see Testing query help files.
The new subcommand codeql bqrs hash computes a stable hash of a BQRS file.
codeql query decompile now accepts a --kind flag. This allows advanced users to choose which intermediate representation to show for a compiled QL query. --kind dil shows the Datalog representation while --kind ra shows the relational algebra representation used by the evaluator.

Release 2.2.6 (2020-09-11)

This release corresponds to release 1.25.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.3.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available.

Release 2.2.5 (2020-08-21)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
Updated license terms with a rewritten description of what is and is not allowed. No substantive changes are intended, but the new text is hopefully easier to understand.

New features

The CLI can now execute queries that use QL's external predicate feature. All subcommands that execute queries have a new --external option to specify the value set for those predicates.
A new codeql bqrs diff command can be used to compute the difference between two binary query result sets.
codeql test run has some new options to improve support for testing of extractors:
- --check-databases which will run codeql dataset check on every test database produced during a run.
- --consistency-queries which will run a set of additional queries over all the test databases produced during a run.
- --show-extractor-output

Release 2.2.4 (2020-06-29)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.

Bugs fixed

QL packs found through the --search-path option, or in a sibling directory to the unpacked CLI would erroneously take precedence over the content of the workspace when using the CodeQL extension for Visual Studio Code. This is now fixed such that the workspace takes priority.
Two command-line options that control the amount of disk space that the QL evaluator will try to keep free of disk cache are now called --min-disk-free and --min-disk-free-pct. Previously they were called --max-disk-free instead, which made no sense. The old names are still recognized such as not to break existing scripts, but are now undocumented and deprecated.

Release 2.2.3 (2020-06-15)

CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new version number because the v2.2.2 folder on the download site originally contained the 2.2.0 binaries instead of the correct 2.2.2 ones.

If you have downloaded release 2.2.2, and codeql --version correctly identifies itself as being that version, you don't need to upgrade to 2.2.3.

Release 2.2.2 (2020-06-12)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.

Improvements

Query evaluations that time out due to a --timeout option are no longer silently discarded. Instead codeql will terminate with exit code 33. Commands that evaluate multiple queries will produce as much output as they can even if one of the queries times out.

Release 2.2.1

There is no CodeQL CLI version 2.2.1. This version number was used internally to work around restrictions in the CodeQL for VS Code extension.

Release 2.2.0 (2020-05-29)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
Starting with this release, the CodeQL CLI can be downloaded either as a single codeql.zip file containing the CLI for all supported platforms, or as a codeql-PLATFORM.zip that contains the files for just one platform. The single-platform zips are faster to download.

QL language improvement

QL now supports the definition of new types as type unions. This feature currently allows unions of branches from an already existing algebraic data type and unions of database types.

Release 2.1.4 (2020-05-26)

This release corresponds to release 1.24.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.2.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available.

Features added

A new codeql query format command exposes the QL autoformatter for use on the command line.

Bugs fixed

-J command-line options that contain spaces now ought to work on Windows. They still do not work reliably on Linux or MacOS, though.

Release 2.1.3 (2020-05-13)

Bugs fixed

Fixes a bug in codeql execute cli-server (a helper used by the VS Code extension) which would sometimes cause query compilation to fail until the extension was restarted.
Fixes a bug in codeql database upgrade which could lead to performance losses if the upgraded database was subsequently used with LGTM or the legacy Semmle Core product.
Fixes a bug in the QL evaluator that would sometimes lead to crashes for queries that use the new unique aggregate added in release 2.1.0.
The value of the --compilation-cache-size option is now correctly interpreted as a number of megabytes rather than a number of bytes.

Release 2.1.2 (2020-05-06)

Updated license terms to allow CI use with GitHub Actions for open-source software.

Potentially breaking changes

In query suite definitions, filter instructions that filter on the query path pseudo-tag will now always see the relative path to the query expressed with / as a directory separator, independently of the platform. Previously they erroneously used the platform's directory separator, meaning that query suites developed on Windows would not work correctly on Unix systems (and vice versa) if they used query path. Existing suite definitions developed on Windows may need to be updated to match the new behavior.

Features added

A new codeql test accept subcommand helps automate updating the expected output for unit tests after a desired change in query behavior. This can also be done by the new --learn option for codeql test run.

Bugs fixed

codeql database create will now report an explicit error if given a --command argument that specifies an empty string. Previously this would be accepted initially, leading to confusing failures later.

Release 2.1.1 (2020-04-20)

The bundled extractors are updated to match the versions currently used on LGTM.com.

Features added

codeql resolve queries accepts a --format=bylanguage option. This is used to help automated workflows determine which languages to create databases for, from the queries that are available to run.
It is now possible to attempt to execute .ql files that are not in a QL pack. This is used by a few specialized internal workflows. However, standalone queries cannot import any of the dependencies that you would usually declare in a qlpack.yml file, so will not be useful in most cases.

Release 2.1.0 (2020-03-27)

The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.23) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.23 instance, you need to create them with release 2.0.1. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Potentially breaking changes

If you pass a directory name as a command-line argument to codeql test run, it will now consider all .ql or .qlref files found under that directory to be test queries, even if they have no accompanying .expected file. Tests that lack an .expected file will fail, but will generate an .actual file that you can rename to .expected if you want to use the results.

The goal of this change is to support existing workflows of experienced CodeQL users, and also to provide clear error indications if an .expected file is accidentally lost, renamed, or misspelled.

However, if you invoke codeql test run on a directory tree that contains both tests and non-test queries, you will now encounter errors if any of the .ql files can't be processed as test queries. If you're affected by this change, you can suppress these errors by:
- Adding a tests property to this QL pack to define specify which directories contain only test queries and associated test code. For more information, see About QL packs.
- Running codeql test run with a new --strict-test-discovery option.
In the longer term, we recommend that you reorganize the queries so that test queries are stored in a directory tree that's separate from actual queries.
codeql database create and codeql database finalize will no longer recognize a --no-duplicate-code option. This option has never had any effect, and its positive variant --duplicate-code previously led to a fatal error.

Features added

A new XML extractor is included. It is not intended to be used as a stand-alone extractor, but rather to augment the data produced by other extractors. In particular, the C# and Java extractors invoke it during database creation to include information relevant to the analysis of those languages, much like LGTM.com does.
Two new plumbing commands codeql database index-files and codeql resolve files have been added for support of invoking the XML extractor support. These commands are generally only of interest for extractor authors.
Two new plumbing commands have been added to codeql dataset. The measure subcommand can be used to collect size information from a dataset, and the check subcommand can scan a dataset for database inconsistencies. These commands are useful when developing a new CodeQL extractor.
The QL evaluator contains a number of features in support of an internal experiment with using machine-learning techniques to identify functions in unknown codebases as sources or sinks of taint. This includes new command-line options --ml-model-path and --native-library-path to several subcommands. As the new features are not yet ready for general use, these new options should be ignored by external CodeQL users.

Bugs fixed

Fixes a bug that could result in empty databases for C/C++. Previously, extraction would mistakenly be skipped for source files compiled with the Clang compiler, if the -fintegrated-cc1 option was specified.
codeql database create and codeql database init will now, as they have always been documented, refuse to create a database whose parent directory doesn't already exist.
codeql test run will no longer leave .actual files from previous runs in the file system after a test passes.

QL language improvements

QL now supports set literals, and the QL extractor can identify them with the SetLiteral class. For more information, see Set literal expressions in the QL language reference.
QL now supports a uniqueness aggregate. This can express constraints that there is precisely one value. The syntax is taken from previous aggregates such as min and max.
```
unique(int x | x = 4 or x = 2 * 2 | x)
```

Release 2.0.6 (2020-03-16)

Bugs fixed

Fixes a problem preventing codeql database create from working with Python 3 on macOS.
Fixes a problem preventing codeql database create from finding locally installed Python packages.

Release 2.0.5 (2020-03-13)

The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Features added

codeql test run has a new --slice option that can be used to parallelize tests over more machines.

Release 2.0.4 (2020-02-21)

The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Features added

Subcommands that execute queries (such as codeql database analyze) now have a --timeout option that can be used to set a timeout to automatically cancel query evaluations that appear to diverge.
A new plumbing command codeql query decompile can display the DIL intermediate representations that is included in the output of codeql query compile --dump-qlo --include-dil-in-qlo. This is useful mainly for certain internal workflows; the information produced is the same as what codeql query compile --dump-dil already outputs.

Bugs fixed

The --debug and --tuple-counting options to codeql test run erroneously had no effect. Now they ought to work.

Release 2.0.3 (2020-02-12)

Bugs fixed

Fixes a bug where codeql test run would fail with the message CatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it when running tests against the master branch of the CodeQL libraries for certain languages.
Otherwise identical to release 2.0.2.

Release 2.0.2 (2020-02-05)

The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.
The parent and sibling directories of the unpacked CLI are no longer searched recursively for QL packs. QL packs will only be found if there's a qlpack.yml or .codeqlmanifest.json directly in a parent or sibling directory. This should eliminate the very long disk-scanning delays experienced by users who unpacked earlier versions of the CLI in their home directory.
Parent and sibling directories of the unpacked CLI will now be searched for QL packs as a last resort, even if you give an explicit --search-path option. This means, for example, that you can define a search path in the per-user configuration file without it depending on where the CLI is unpacked. In particular, the setting can now be meaningfully used by users who let the CodeQL for VS Code extension manage the downloading and unpacking of the CLI.

Security updates

The codeql database create command and its relatives will no longer attempt to find extractors located in the parent and sibling directories of the unpacked CLI. This closes a security risk for users who unpacked the CodeQL CLI in their home directory. This could've resulted in arbitrary code execution if the user unpacked a file archive containing a malicious extractor anywhere in the home directory. Extractors will now only be found within the unpacked CLI itself, or in directories explicitly listed in the --search-path. It is expected that users will only point --search-path to locations they trust at least as much as the CLI download itself.

Features added

This release supports executing query regression tests using the codeql test command. For further information, see Testing custom queries.
The error message if you try executing a query against a database that needs to be upgraded (which can happen routinely if you're using a fresh master checkout of the CodeQL libraries with the bundled extractors) will now explicitly suggest a codeql database update command to run. The database is not automatically upgraded, as this may make it irreversibly incompatible with older versions of the CodeQL libraries. This allows users who want to compare behavior of different versions of the libraries against the same database to make a copy before they upgrade it.

Release 2.0.1 (2019-12-17)

Corresponds to LGTM Enterprise release 1.23.
The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the extractor versions used in LGTM Enterprise.
No other changes to the core CLI.

Release 2.0.0 (2019-11-14)

First public release.

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

CodeQL CLI changelog

Release 2.20.0 (2024-12-09)

Known issues

New features

Miscellaneous

Release 2.19.4 (2024-12-02)

Improvements

Bugs fixed

Release 2.19.3 (2024-11-07)

Bugs fixed

Miscellaneous

Release 2.19.2 (2024-10-21)

Potentially breaking changes

Bugs fixed

Miscellaneous

Release 2.19.1 (2024-10-04)

New Features

Release 2.19.0 (2024-09-18)

Improvements

Miscellaneous

Release 2.18.4 (2024-09-12)

New Features

Release 2.18.3 (2024-08-28)

Release 2.18.2 (2024-08-13)

Deprecations

Miscellaneous

Release 2.18.1 (2024-07-25)

Security Updates

New features

Improvements

Release 2.18.0 (2024-07-11)

Breaking changes

Regressions

Improvements

Bugs fixed

Release 2.17.6 (2024-06-27)

New features

Improvements

Release 2.17.5 (2024-06-12)

Breaking changes

Improvements

Miscellaneous

Release 2.17.4 (2024-06-03)

New features

Release 2.17.3 (2024-05-17)

Improvements

Release 2.17.2 (2024-05-07)

Known issues

Improvements

Release 2.17.1 (2024-04-24)

Deprecations

Improvements

Miscellaneous

Release 2.17.0 (2024-04-04)

Deprecations

Improvements

Miscellaneous

Release 2.16.6 (2024-03-26)

Bugs fixed

Release 2.16.5 (2024-03-21)

New features

Release 2.16.4 (2024-03-11)

Potentially breaking changes

Improvements

Bugs fixed

Release 2.16.3 (2024-02-22)

Security patches

New Features

Bugs fixed

Release 2.16.2 (2024-02-12)

Release 2.16.1 (2024-01-25)

Improvements

QL language improvements

Release 2.16.0 (2024-01-16)