Merge pull request #5 from brooklyn-data/SR/v1.2

v1.2.0

.editorconfig

@@ -8,3 +8,6 @@ end_of_line = lf
 charset = utf-8
 trim_trailing_whitespace = true
 insert_final_newline = true
+
+[*.yml]
+indent_size = 2

.github/workflows/ci_tests.yml

@@ -0,0 +1,23 @@
+name: CI tests
+
+on:
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  build_package:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: actions/setup-python@v2
+        with:
+          python-version: '3.9'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+
+      - name: Build package
+        run: python -m build

.github/workflows/publish_release_to_pypi.yml

@@ -0,0 +1,29 @@
+name: Publish release to PyPI
+
+on:
+  release:
+    types:
+      - published
+
+jobs:
+  publish_package:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: actions/setup-python@v2
+        with:
+          python-version: '3.9'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish package
+        uses: pypa/[email protected]
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}

.gitignore

@@ -1,4 +1,5 @@
 __pycache__/
+dist/
 *.egg-info/
 .venv/
 venv/

CHANGELOG.md

@@ -5,22 +5,43 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased](https://github.com/brooklyn-data/dbtenv/compare/v1.1.1...HEAD)
+
+## [Unreleased](https://github.com/brooklyn-data/dbtenv/compare/v1.2.0...HEAD)
+
+### Added
+
+### Changed
+
+### Fixed
+
+
+## [1.2.0](https://github.com/brooklyn-data/dbtenv/compare/v1.1.1...v1.2.0) - 2021-11-29
 
 ### Added
+- New `--quiet` argument to not output any nonessential information as dbtenv runs.
+- Allow location of dbt version-specific Python virtual environments to be configured with `DBTENV_VENVS_DIRECTORY` and `DBTENV_VENVS_PREFIX` environment variables.
+- Publish dbtenv package to PyPI.
 
 ### Changed
+- If no specific dbt version has been selected then default to using the max installed version (if any) or the max installable version (preferring stable versions).
+- If no compatible dbt version can be found for a dbt project and its installed packages then ignore dbt version requirements from installed packages in case they're simply out of date.
+- When installing with pip, upgrade pip to avoid problems with packages that might require newer pip features.
+- When installing with Homebrew, automatically add the dbt Homebrew tap if necessary.
+- Switch from distutils to setuptools.
+
 
 ## [1.1.1](https://github.com/brooklyn-data/dbtenv/compare/v1.1.0...v1.1.1) - 2021-07-15
 
 ### Fixed
 - Fix error when `~/.dbt/versions` directory doesn't exist yet.
 
+
 ## [1.1.0](https://github.com/brooklyn-data/dbtenv/compare/v1.0.0...v1.1.0) - 2021-07-14
 
 ### Added
 - Support installation of dbt versions >= 0.20.0 in a Python 3.9 environment.
 
+
 ## [1.0.0](https://github.com/brooklyn-data/dbtenv/releases/tag/v1.0.0) - 2021-04-16
 
 ### Added

README.md

@@ -2,22 +2,11 @@
 
 dbtenv lets you easily install and run multiple versions of [dbt](https://docs.getdbt.com/docs/introduction) using [pip](https://pip.pypa.io/) with [Python virtual environments](https://docs.python.org/3/library/venv.html), or optionally using [Homebrew](https://brew.sh/) on Mac or Linux.
 
-### Table of contents
-- **[Installation](#installation)**
-- **[How it works](#how-it-works)**
-  - [Using pip and/or Homebrew](#using-pip-andor-homebrew)
-  - [Installing dbt versions](#installing-dbt-versions)
-  - [Switching between dbt versions](#switching-between-dbt-versions)
-  - [Running dbt versions](#running-dbt-versions)
-  - [Running dbt with dbtenv more seamlessly](#running-dbt-with-dbtenv-more-seamlessly)
-  - [Uninstalling dbt versions](#uninstalling-dbt-versions)
-- **[Development](#development)**
-  - [Development setup](#development-setup)
-
 
 ## Installation
 
-Run `pip3 install git+https://github.com/brooklyn-data/dbtenv#egg=dbtenv`.
+1. Install [pipx](https://pypa.github.io/pipx/) if you haven't already.
+2. Run `pipx install dbtenv`.
 
 
 ## How it works
@@ -35,6 +24,7 @@ You can run `dbtenv versions` to list the versions of dbt available to install,
 If you don't want to have to run `dbtenv install <version>` manually, you can set a `DBTENV_AUTO_INSTALL=true` environment variable so that as you run commands like `dbtenv version` or `dbtenv execute` any dbt version specified that isn't already installed will be installed automatically.
 
 Some tips when dbtenv is using pip:
+- You can customize where the dbt version-specific Python virtual environments are created by setting `DBTENV_VENVS_DIRECTORY` and `DBTENV_VENVS_PREFIX` environment variables.
 - You can have dbtenv only install Python packages that were actually available on the date the dbt version was released by setting a `DBTENV_SIMULATE_RELEASE_DATE=true` environment variable, or specifying `--simulate-release-date` when running `dbtenv install`.
   This can help if newer versions of dbt's dependencies are causing installation problems.
 - By default dbtenv uses whichever Python version it was installed with to install dbt, but that can be changed by setting a `DBTENV_PYTHON` environment variable to the path of a different Python executable, or specifying `--python <path>` when running `dbtenv install`.
@@ -49,6 +39,8 @@ dbtenv determines which dbt version to use by trying to read it from the followi
    - If the dbt version requirements specify a range of versions rather than an exact version, then dbtenv will try to read a preferred dbt version from the sources below and will use that version if it's compatible with the requirements.
 5. The first `.dbt_version` file found by searching the dbt project's parent directories.
 6. The `~/.dbt/version` file.
+7. The max installed dbt version (preferring stable versions).
+8. The max installable dbt version (preferring stable versions).
 
 You can:
 - Run `dbtenv version` to show which dbt version dbtenv determines dynamically based on the current environment.
@@ -81,7 +73,7 @@ Some examples:
 Note that after adding such a `dbt` alias/function to your shell profile you'll need to reload the profile to activate it (e.g. by running `. ~/.bash_profile` in bash, or `. $PROFILE` in PowerShell).
 
 ### Uninstalling dbt versions
-You can run `dbtenv versions --installed` to list the versions of dbt that dbtenv has installed under `~/.dbt/versions` and/or with Homebrew, and then run `dbtenv uninstall <version>` to uninstall a version.
+You can run `dbtenv versions --installed` to list the versions of dbt that dbtenv has installed in Python virtual environments and/or with Homebrew, and then run `dbtenv uninstall <version>` to uninstall a version.
 
 
 ## Development

dbtenv/__init__.py

@@ -14,9 +14,10 @@
 from typing import Any, List, Optional
 
 
-__version__ = '1.1.1'
+__version__ = '1.2.0'
 
-VENVS_DIRECTORY = os.path.normpath('~/.dbt/versions')
+DEFAULT_VENVS_DIRECTORY = os.path.normpath('~/.dbt/versions')
+DEFAULT_VENVS_PREFIX    = ''
 
 GLOBAL_VERSION_FILE = os.path.normpath('~/.dbt/version')
 LOCAL_VERSION_FILE  = '.dbt_version'
@@ -26,7 +27,10 @@
 DEBUG_VAR                 = 'DBTENV_DEBUG'
 DEFAULT_INSTALLER_VAR     = 'DBTENV_DEFAULT_INSTALLER'
 PYTHON_VAR                = 'DBTENV_PYTHON'
+QUIET_VAR                 = 'DBTENV_QUIET'
 SIMULATE_RELEASE_DATE_VAR = 'DBTENV_SIMULATE_RELEASE_DATE'
+VENVS_DIRECTORY_VAR       = 'DBTENV_VENVS_DIRECTORY'
+VENVS_PREFIX_VAR          = 'DBTENV_VENVS_PREFIX'
 
 
 def string_is_true(value: str) -> bool:
@@ -71,9 +75,13 @@ def __str__(self) -> str:
 
 
 class Version(distutils.version.LooseVersion):
-    def __init__(self, version: str, source: Optional[str] = None) -> None:
+    def __init__(self, version: str, source: Optional[str] = None, source_description: Optional[str] = None) -> None:
         self.pypi_version = self.homebrew_version = self.raw_version = version.strip()
         self.source = source
+        if source and not source_description:
+            self.source_description = f"set by {source}"
+        else:
+            self.source_description = source_description
 
         version_match = re.match(r'(?P<version>\d+\.\d+\.\d+)(-?(?P<prerelease>[a-z].*))?', self.raw_version)
         self.is_semantic = version_match is not None
@@ -138,7 +146,9 @@ def __init__(self) -> None:
         self.project_file = self.find_file_along_working_path('dbt_project.yml')
         self.project_directory = os.path.dirname(self.project_file) if self.project_file else None
 
-        self.venvs_directory     = os.path.expanduser(VENVS_DIRECTORY)
+        self.venvs_directory = os.path.expanduser(self.env_vars.get(VENVS_DIRECTORY_VAR) or DEFAULT_VENVS_DIRECTORY)
+        self.venvs_prefix = self.env_vars.get(VENVS_PREFIX_VAR) or DEFAULT_VENVS_PREFIX
+
         self.global_version_file = os.path.expanduser(GLOBAL_VERSION_FILE)
 
         self.homebrew_installed = False
@@ -166,7 +176,32 @@ def debug(self) -> bool:
     @debug.setter
     def debug(self, value: bool) -> None:
         self._debug = value
-        LOGGER.setLevel(logging.DEBUG if self._debug else logging.INFO)
+        self.update_logging_level()
+
+    _quiet: Optional[bool] = None
+
+    @property
+    def quiet(self) -> bool:
+        if self._quiet is None:
+            if QUIET_VAR in self.env_vars:
+                self._quiet = string_is_true(self.env_vars[QUIET_VAR])
+            else:
+                self._quiet = False
+
+        return self._quiet
+
+    @quiet.setter
+    def quiet(self, value: bool) -> None:
+        self._quiet = value
+        self.update_logging_level()
+
+    def update_logging_level(self) -> None:
+        if self.debug:
+            LOGGER.setLevel(logging.DEBUG)
+        elif self.quiet:
+            LOGGER.setLevel(logging.ERROR)
+        else:
+            LOGGER.setLevel(logging.INFO)
 
     _default_installer: Optional[Installer] = None
 

dbtenv/execute.py

@@ -57,7 +57,7 @@ def execute(self, args: Args) -> None:
             version = args.dbt_version
         else:
             version = dbtenv.version.get_version(self.env)
-            logger.info(f"Using dbt {version} (set by {version.source}).")
+            logger.info(f"Using dbt {version} ({version.source_description}).")
 
         dbt = dbtenv.which.try_get_dbt(self.env, version)
         if not dbt:

dbtenv/homebrew.py

@@ -27,7 +27,20 @@ def get_dbt_version_keg_directory(env: Environment, version: Version) -> str:
     return os.path.join(env.homebrew_prefix_directory, 'opt', get_dbt_version_formula(version))
 
 
+def ensure_homebrew_dbt_tap() -> None:
+    # While it would be simplest to just always run `brew tap dbt-labs/dbt`, we need to first check for
+    # an existing dbt tap because the "dbt-labs" GitHub organization used to be named "fishtown-analytics"
+    # so people could have a "fishtown-analytics/dbt" tap, and having multiple dbt taps causes errors.
+    tap_list_result = subprocess.run(['brew', 'tap'], stdout=subprocess.PIPE, text=True)
+    if not re.search(r'\b(fishtown-analytics|dbt-labs)/dbt\b', tap_list_result.stdout):
+        logger.info('Adding the dbt Homebrew tap.')
+        tap_dbt_result = subprocess.run(['brew', 'tap', 'dbt-labs/dbt'])
+        if tap_dbt_result.returncode != 0:
+            raise DbtenvError("Failed to add the dbt Homebrew tap.")
+
+
 def get_homebrew_dbt_versions() -> List[Version]:
+    ensure_homebrew_dbt_tap()
     brew_args = ['info', '--json', 'dbt']
     logger.debug(f"Running `brew` with arguments {brew_args}.")
     brew_result = subprocess.run(['brew', *brew_args], stdout=subprocess.PIPE)
@@ -59,6 +72,8 @@ def install(self, force: bool = False) -> None:
                 self.uninstall(force=True)
             else:
                 raise DbtenvError(f"dbt {self.version.homebrew_version} is already installed with Homebrew.")
+        else:
+            ensure_homebrew_dbt_tap()
 
         logger.info(f"Installing dbt {self.version.homebrew_version} with Homebrew.")
         brew_args = ['install', get_dbt_version_formula(self.version)]

dbtenv/install.py

@@ -70,7 +70,7 @@ def execute(self, args: Args) -> None:
             version = args.dbt_version
         else:
             version = dbtenv.version.get_version(self.env)
-            logger.info(f"Using dbt {version} (set by {version.source}).")
+            logger.info(f"Using dbt {version} ({version.source_description}).")
 
         if self.env.primary_installer == Installer.PIP:
             pip_dbt = dbtenv.pip.PipDbt(self.env, version)

dbtenv/main.py

@@ -30,7 +30,8 @@ def build_root_args_parser(env: Environment) -> argparse.ArgumentParser:
         description=f"""
             Lets you easily install and run multiple versions of dbt using pip with Python virtual environments,
             or optionally using Homebrew on Mac or Linux.
-            Any dbt version-specific Python virtual environments are created under `{dbtenv.VENVS_DIRECTORY}`.
+            Any dbt version-specific Python virtual environments are created under `{dbtenv.DEFAULT_VENVS_DIRECTORY}` by default,
+            but that can be configured using {dbtenv.VENVS_DIRECTORY_VAR} and {dbtenv.VENVS_PREFIX_VAR} environment variables.
             The dbt version to use can be configured in your shell using a {dbtenv.DBT_VERSION_VAR} environment variable,
             in dbt projects using the `require-dbt-version` configuration, locally within specific directories using
             `{dbtenv.LOCAL_VERSION_FILE}` files, or globally in a `{dbtenv.GLOBAL_VERSION_FILE}` file.
@@ -56,8 +57,18 @@ def build_common_args_parser(env: Environment, dest_prefix: str = '') -> argpars
         const=True,
         help=f"""
             Output debug information as dbtenv runs.
-            The default is to not output debug information, but that can be overridden by setting a {dbtenv.DEBUG_VAR}
-            environment variable.
+            This can also be enabled by setting a {dbtenv.DEBUG_VAR} environment variable.
+        """
+    )
+    common_args_parser.add_argument(
+        '--quiet',
+        dest=f'{dest_prefix}quiet',
+        action='store_const',
+        const=True,
+        help=f"""
+            Don't output any nonessential information as dbtenv runs.
+            This can also be enabled by setting a {dbtenv.QUIET_VAR} environment variable.
+            Note that if outputting debug information has been enabled this setting will have no effect.
         """
     )
     if env.homebrew_installed:
@@ -134,6 +145,10 @@ def main(args: List[str] = None) -> None:
 
         logger.debug(f"Parsed arguments = {parsed_args}")
 
+        quiet = parsed_args.quiet or parsed_args.get('subcommand_quiet')
+        if quiet:
+            env.quiet = quiet
+
         installer = parsed_args.get('installer') or parsed_args.get('subcommand_installer')
         if installer:
             env.installer = installer