CHANGELOG.md

Changelog for Mass Driver

The format is based on Keep a Changelog. The project uses semantic versioning (see semver).

[Unreleased]

v0.18.0 - 2023-11-21

Added

• New SingleFileEditor, derived from PatchDriver, for editing single files.
  • Parameter target_file, will be fed its text content to process_file().
  • Use via from mass_driver.drivers.bricks import SingleFileEditor.
• New GlobFileEditor, similarly for editing multiple files from glob pattern.
  • Use via from mass_driver.drivers.bricks import GlobFileEditor.

v0.17.1 - 2023-11-13

Added

• Update pyGithub to 2.1.1, now throttles Github API to avoid ratelimits

Fixed

• Catch and log import errors during activity loading, previously silent crashes

v0.17.0 - 2023-11-12

Added

• Completing an activity (source, migration, forge) now summarizes results
  • Breakdown of repo count per outcome type, then sorted list of repos per type
• FASTER: Optional, experimental multi-threaded per-repo processsing! Handling of clone, scan, patching is done as individual thread per repo, with N=8 pooled threads.
  • Early data shows a x6 improvement in performance, as cloning one repo doesn't block others anymore.
  • Enable via new experimental flag run --parallel, defaulting to False

Changed

• BREAKING: RenamedRepo to SourcedRepo in mass_driver.models.repository
  • This exposes better the idea of "a Repo as it was Source-d", in contrast to ClonedRepo "a Repo after it was cloned".
  • Also avoids clashes with git.Repo object from gitpython dependency.
• Replaced all print() calls to logging module
• Loggers used are mostly nested:
  • from root (default)
  • to run (or other file-activity-based)
  • to run.repo.<repo-id> for logs for a specific repo's processing
  • to subloggers like run.repo.<repo-id>.driver.<driver-plugin-name>
• PatchDriver now has a logger obj for such customized logging: Repo-processing sets Patchdriver.logger named run.repo.<repo-id>.driver.<driver-plugin-name>
  • ACTION: Please replace any print with self.logger.info!

v0.16.4 - 2023-11-04

Fixed

• Attempting to run a Forge activity with git_push_first=True, without migration or scan activity, no longer causes exit without processing any repos. Clone step invoked properly mean remote-clone URLs are now supported, converted to local filepaths internally.
• make docs now works again with sphinx-autodoc2: Pinned astroid dep to 2.15.8 (< 3.0.0) to avoid the regression caused by unpinned astroid.

v0.16.3 - 2023-09-11

Fixed

• ShellDriver no longer crashes due to irrelevant dataclass import
• Updated pyGithub, fixes "missing cryptography" error
• Error messages for bad config file for Sources no longer insist wrongly about "Forge config error". Now detecting the validation error's model properly.
• Failing to load a scanner that was selected in config now throws ImportError on the first plugin-load failure, instead of silently skipping the scanner.

Changed

• Updated to python-template v1.3.0 (from 1.1.0)

v0.16.2 - 2023-07-25

Fixed

• github-app forge no longer crashes due to type confusion on a param.

v0.16.1 - 2023-07-15

Fixed

• Formatting of the % of PR per status back to 2 digits precision not 11.

v0.16.0 - 2023-07-13

Breaking interface of Forge to facilitate new view-pr feature.

Added

• New view-pr feature for bulk reviewing the status of PRs that already exist.

Changed

• BREAKING: New Forge.get_pr_status(), required from derived classes, returning a string status, used as key to group PRs together for summary purposes.
• BREAKING: New Forge.pr_statuses property, required from derived classes, returning a list of all possible string statuses of Forge.get_pr_status(), sorted from most complete (e.g. merged) to least complete (e.g. not merged, has merge-conflicts).

Removed

• BREAKING: Removed Forge.get_pr(), which had unclear return type anyway.

v0.15.0 - 2023-07-05

Changed

Major break of interface: Rework of the cloning system, merges migration/scan codepaths, enabling use of Source-discovered information in PatchDriver.run.

• BREAKING: PatchDriver.run() passes ClonedRepo obj, not pathlib.Path.
  • Any use of repo in your PatchDriver.run() should use repo.cloned_path.
  • See ClonedRepo docs, contains information derived from Source, such as patch_data field, arbitrary source-issued information dict.
• BREAKING: tests.fixtures.massdrive() now returns 3-item-tuple, not 2.
  • Returned tuple: PatchResult, ForgeResult, ScanResult (or None)
  • Any tests using fixtures.massdrive should now set mig, forge, scan =...
  • Swap fixtures.massdrive_scan with fixtures.massdrive accepting 2 junk arg
• BREAKING: mass-driver scan CLI removed, now part of mass-driver run. Activity flow for run command is now:
  • Source discovery phase (if any, or from CLI), generating Repo list
  • Main phase, iterating over each Repo, first to clone them =ClonedRepo list
  • Inside main phase, scan (if any), generating ScanResult
  • Inside main phase, migrate (if any), generating MigrationResult
  • After main phase, interactively pause for review if requested
  • Forge activity, iterating over each repo again, creating ForgeResult
• BREAKING: models.source module renamed to models.repository.

Added

• Scan+Migration+Forge can now ALL happen in one run command:
  • Clones one repo, then scanning it, then migrating it, then next repo
  • Can thus do all of Source -> [Clone] -> Scan -> Migrate -> Forge
• New csv-filelist Source for importing repos in CSV file format
• New tests.fixture.massdrive_runlocal() func to enable source testing

Fixed

• Secret tokens for Github plugins no longer leak on config dump (--json-outfile flag), by replacing str with pydantic.SecretStr.
  • Docs updated to warn downstream devs about this risk.
• Pin pydantic to 1.*, as breaking version 2.0 was just released.

v0.14.0 - 2023-06-08

Added

• New source feature for discovering what repos to patch/scan.
  • Sources are plugins with discover() method, returning Repos by ID.
  • Alternative sources subcommand to list and detail them
  • New TOML file entry [mass-driver.source], with subkey source_name used to select which source plugin to enable.
  • Simple sources provided:
    • repo-list for in-activity-file repository list
    • repo-filelist to point to a separate file listing repos
    • template-filelist to expand a template against a file listing repos
  • github-search and github-app-search Sources for Github Repository search
  • CLI args --repo-path and --repo-filelist still available, overriding any source, so that massdriver.source is only required if lacking CLI args
• CI (pytest, pre-commit) set up via Github Actions: PR #1

v0.13.2 - 2023-06-03

Added

• New file_ownership parameter for stamper, defaulting to 0664.

Fixed

• Exit codes harmonized:
  • 0 for success
  • 1 for failures during the main function
  • 2 for argument parsing errors
• stamper driver now creates any missing parent folder to the target
• Remove test depending on git clone from Github: Faster, offline tests now

v0.13.1 - 2023-04-17

Fixed

• scan command now uses --json-outfile as expected

v0.13.0 - 2023-04-17

Added

• New scan feature for scanning repos with arbitrary python functions. See new "Scanning" docs:
  • Scanners are plugins declared under mass-driver.scanners, linking to functions like my_scanner(repo: Path) -> dict[str, Any]
  • Alternative scanners command to list out detected, available scanners
  • New TOML file entry [mass-driver.scan], with subkey scanner_names used to select which scanner plugins to enable.
  • Simple scanners root-files and dockerfile-from provided for reference
  • New fixture massdriver_scan and massdriver_scan_check for testing scanners
• New optional CLI parameter --json-outfile for run and scan, to save the activity outcome to JSON files for analysis

Changed

• Test fixture massdrive_check_file now returns unchecked result and reference blobs for equality assertion (assert result == reference) to be done by the caller. This enables plugins like pytest-clarity to show colorful diff. Users of massdrive_check_file need to change (on pain of lack of test assertion):

- massdrive_check_file(workdir)
+ result, reference = massdrive_check_file(workdir)
+ assert result == reference, "Massdriver result should match reference"

v0.12.0 - 2023-04-05

Added

• Auto-detect repo's base branch for Forge: parameter base_branch now optional, defaulting to repo's default branch

v0.11.0 - 2023-04-02

Added

• New github-app forge plugin for creating PRs on Github when running mass-driver as a Github App
• New Forge params:
  • forge_config dict, for Forge-specific non-sensitive config to keep in config file, complementing envvars. Similar to driver_config for Migration.
  • interactive_pause_every int, for blocking the Forge, pausing for confirmation interactively every few PRs generated. Disabled by default, set to 1 to block every PR, or 5 every 5...

v0.10.0 - 2023-04-02

Removed

• Unused migration_name field of Migration now removed

Added

• Forge subclasses can now grab config via envvars prefixed FORGE_. Observe that Forge now derives from pydantic.BaseSettings, see BaseSettings docs.
• New, simpler testing fixture massdrive_check_file for PatchDriver that affect single files

v0.9.0 - 2023-03-12

Added

• New optional Migration params: commit_author_name + commit_author_email, used to override the git commit author.

v0.8.0 - 2023-02-24

Added

• New file type Activity combines Migration and Forge
• New Forge named dummy for testing purposes
• New git_push_first boolean param in Forge to disable git pushing.
• New ActivityOutcome to capture the full result of a migration/forge sequence

Changed

• Replace commands run-migration + run-forge by new run, using the Activity file type with optionals.
• Internals refactored: all Pydantic objects now under mass_driver.models (PatchDriver, Forge, Activity)

Removed

• Options --really-commit-changes and --dry-run

v0.7.0 - 2023-02-22

Added

• New ROADMAP.md to clarify The Plan.

Changed

• Upgrade minimum Python version to 3.11 to use tomllib
• Replace tomlkit with stdlib tomllib

Fixed

• Migrations, once loaded from TOML, are now proper dict again
• PatchDriver instance now independent across repos

v0.6.1 - 2023-02-06

Added

• Upgrade pre-commit dependencies + poetry in Dockerfile
• Update pyproject.toml for release to Pypi

v0.6.0 - 2023-02-04

Added

• Forges now discovered via setuptools
• New "run-forge" subcommand for creating PRs for on-disk branches
• Implemented GithubForge for PR creation on Github
• New "Stamper" PatchDriver for "stamping" new files on repos

Fixed

• Incorrect Repo creation causing silent cloning errors on some devices

v0.5.0 - 2023-01-24

Added

• New plugin system for including third-party PatchDrivers, based on the setuptools "entrypoints" system.
• CLI Subcommands for inspecting drivers, running migrations...
• PatchDriver test harness, for ease of plugin development.
• Project objectives (gherkin features in features/) now part of docs

Changed

• PatchDriver now inherits Pydantic BaseModel (allows serialization)
• PatchDriver func prototype: run(repo: Path) -> PatchResult

Removed

• Complex PatchDrivers (the ones requiring package dependencies) removed, moved to separate "plugins" repo.

v0.4.0 - 2022-11-16

Added

• New Jsonpatch Driver, for applying an RFC6902 JSON Patch
• New PreCommit Driver, for running pre-commit autoupdate
• New ShellDriver for running arbitrary shell commands
• New mass-driver(1) man page output to make docs

v0.3.1 - 2022-11-11

Changed

• PatchDriver API simplified, now using single func for detect + patch: called via PatchDriver.run(repo: Path, dry_run: bool).
• CLI simplified when exposing PatchDriver.run(), now uses either --dry-run (the default setup) or --really-commit-changes for committing (not pushing)

v0.3.0 - 2022-11-11

Added

• Poetry Driver stub for major version of packages in pyproject.toml. Showcases JSON Pointers (RFC6901) for structured fields modification.

v0.2.0 - 2022-11-11

Added

• Basic clone/detect/patch/commit workflow, with basic Github support

v0.1.0 - 2022-11-10

Added

• New python module mass_driver, exposed as shell command mass-driver