Code

Release version 1.5.0#318

Code

Merged

indy-independence merged 195 commits intomasterSUNET/cnaas-nms:masterfrom developSUNET/cnaas-nms:developCopy head branch name to clipboardSep 25, 2023

+1,938-304

Commits

Commits on Jan 15, 2023

• dev branch is now v1.5.x

  indy-independence

  committed
  9355066

Commits on Jan 16, 2023

• rebase with black formating

  indy-independence

  committed
  a8e1af3

• Pass different args to napalm_config_X depending on what task function, so we can get job_id into the confirmed version

  indy-independence

  committed
  ef09806

• Begin work on commit confirmed with 3 modes, 0=no confirm, 1=per device confirm, 2=next job id with confirm (platform dependent?)

  indy-independence

  committed
  01b7304

• break out parts of sync_device into separate function. add job function confirm_devices to run as second job after syncto, to confirm all devices. works with Arista if EOS-driver is changed to not …

  Show description for 0da30be
  indy-independence

  committed
  0da30be

• let nornir handle connection open/close when doing commit_confirm_mode 2 instead of closing connection immediately. warn if syncing more than 50 devices, connection timeouts might cause issues?

  indy-independence

  committed
  784c5c8

Commits on Jan 18, 2023

• allow overriding of settings for testing purpose. add tests for syncto commitconfirm mode 1

  indy-independence

  committed
  9fbcd7a

• fix scheduler state check for integrationtests

  indy-independence

  committed
  d1a2e8f

• fix scheduler state check for integrationtests

  indy-independence

  committed
  e2b5f90

• mark equipment instead of integrationtest for github actions

  indy-independence

  committed
  1105fbd

• run syncto with mode 0, 1 and 2. check captured logging data for correct mode

  indy-independence

  committed
  f091889

Commits on Jan 19, 2023

• Ensure SQL schema is updated before test run

  Show description for f1f0669
  lunkwill42

  committed
  f1f0669

• Merge pull request #288 from Uninett/sikt.test.alembic-upgrade

  Show description for b225b32
  indy-independence

  authored
  b225b32

• try to get code coverage for the confirm_devices function

  indy-independence

  committed
  3a9dec2

Commits on Jan 23, 2023

• longer timeout since sync doesn't finish on kvm vm

  indy-independence

  committed
  6b3d362

• resync=True to get device to new config with settings_override even if already synced

  indy-independence

  committed
  63c3f69

• reattempt acquire joblock, sometimes unittest can fail otherwise

  indy-independence

  committed
  1eec6bd

• syncto tests: sleep in between getting jobstatus. wait for next_job to finish for commit confirm mode 2.

  indy-independence

  committed
  a32d84f

Commits on Jan 24, 2023

• more verbose logging for pytest. test super-long timeout for config test

  indy-independence

  committed
  b7af7eb

Commits on Jan 25, 2023

• cleanup after scheduler fixture. print jobstatus from test if not == finished

  indy-independence

  committed
  295e15e

• make sure to refresh session objects between polls

  indy-independence

  committed
  bf1ecde

• add scheduler fixture to global conftest for pytest, and make both syncto and test scheduler use that one with scope=session to only have one scheduler. rework test_scheduler to pytest format.

  indy-independence

  committed
  a3a5fc3

Commits on Jan 26, 2023

• scope should be on decorator, not on function

  indy-independence

  committed
  31c44ff

Commits on Jan 27, 2023

• fix next_job_res typing, log bad status for confirm job as well

  indy-independence

  committed
  981a5a9

• support commit mode 2 for junos

  indy-independence

  committed
  b3721b4

• make sure confirm job gets scheduled_by string

  indy-independence

  committed
  8e6d969

• allow calling syncto with confirm_mode_override to override setting specified in yaml

  indy-independence

  committed
  1eb880d

• add API parameter for confirm_mode

  indy-independence

  committed
  108ddeb

• docs for confirm_mode in syncto API

  indy-independence

  committed
  3f3e67a

• docs for commit_confirm_mode and commit_confirmed_timeout settings

  indy-independence

  committed
  a20567f

• default api.yml settings for commit_confirmed

  indy-independence

  committed
  9c0c577

• fix argument name confirm_mode_override

  indy-independence

  committed
  7ba1715

• log commit confirmed complete

  indy-independence

  committed
  195c7d8

Commits on Jan 30, 2023

• break out post sync update confighash to separate function, and make sure confirm_devices job also runs it

  indy-independence

  committed
  aece43a

• Merge branch 'develop' of github.com:SUNET/cnaas-nms into develop

  indy-independence

  committed
  651af66

• rebase with black formating

  indy-independence

  committed
  dfa216c

• Pass different args to napalm_config_X depending on what task function, so we can get job_id into the confirmed version

  indy-independence

  committed
  56f9197

• Begin work on commit confirmed with 3 modes, 0=no confirm, 1=per device confirm, 2=next job id with confirm (platform dependent?)

  indy-independence

  committed
  bf986fe

• break out parts of sync_device into separate function. add job function confirm_devices to run as second job after syncto, to confirm all devices. works with Arista if EOS-driver is changed to not …

  Show description for a3f3ddd
  indy-independence

  committed
  a3f3ddd

• let nornir handle connection open/close when doing commit_confirm_mode 2 instead of closing connection immediately. warn if syncing more than 50 devices, connection timeouts might cause issues?

  indy-independence

  committed
  2d4a7d0

• allow overriding of settings for testing purpose. add tests for syncto commitconfirm mode 1

  indy-independence

  committed
  a73c7d4

• fix scheduler state check for integrationtests

  indy-independence

  committed
  0392ff2

• fix scheduler state check for integrationtests

  indy-independence

  committed
  a07ebeb

• mark equipment instead of integrationtest for github actions

  indy-independence

  committed
  8442b02

• run syncto with mode 0, 1 and 2. check captured logging data for correct mode

  indy-independence

  committed
  7a435b4

• try to get code coverage for the confirm_devices function

  indy-independence

  committed
  30ff29c

• longer timeout since sync doesn't finish on kvm vm

  indy-independence

  committed
  b0e2c52

• resync=True to get device to new config with settings_override even if already synced

  indy-independence

  committed
  23763f2

• reattempt acquire joblock, sometimes unittest can fail otherwise

  indy-independence

  committed
  8553f4d

• syncto tests: sleep in between getting jobstatus. wait for next_job to finish for commit confirm mode 2.

  indy-independence

  committed
  92c2d47

• more verbose logging for pytest. test super-long timeout for config test

  indy-independence

  committed
  22c2929

• cleanup after scheduler fixture. print jobstatus from test if not == finished

  indy-independence

  committed
  e787551

• make sure to refresh session objects between polls

  indy-independence

  committed
  63babf2

• add scheduler fixture to global conftest for pytest, and make both syncto and test scheduler use that one with scope=session to only have one scheduler. rework test_scheduler to pytest format.

  indy-independence

  committed
  bb5d212

• scope should be on decorator, not on function

  indy-independence

  committed
  824d0dd

• fix next_job_res typing, log bad status for confirm job as well

  indy-independence

  committed
  b448ebe

• support commit mode 2 for junos

  indy-independence

  committed
  789d99b

• make sure confirm job gets scheduled_by string

  indy-independence

  committed
  6850501

• allow calling syncto with confirm_mode_override to override setting specified in yaml

  indy-independence

  committed
  d52ac38

• add API parameter for confirm_mode

  indy-independence

  committed
  afb722b

• docs for confirm_mode in syncto API

  indy-independence

  committed
  3d136e9

• docs for commit_confirm_mode and commit_confirmed_timeout settings

  indy-independence

  committed
  ed20ea7

• default api.yml settings for commit_confirmed

  indy-independence

  committed
  773f5f7

• fix argument name confirm_mode_override

  indy-independence

  committed
  f436fbf

• log commit confirmed complete

  indy-independence

  committed
  d6d2369

• break out post sync update confighash to separate function, and make sure confirm_devices job also runs it

  indy-independence

  committed
  0a99f4f

• Merge branch 'feature.confirm_commit' of github.com:SUNET/cnaas-nms into feature.confirm_commit

  indy-independence

  committed
  76b9fdd

Commits on Feb 1, 2023

• set thread data with jobid so logging for confirm task gets jobid prepended

  indy-independence

  committed
  a3085a5

• don't schedule confirm job if failed_hosts is 1 or more devices in commitmode2, instead rollback all devices

  indy-independence

  committed
  3a415dd

Commits on Feb 3, 2023

• make confirm_commit task update finished_devices for job

  indy-independence

  committed
  6ad4cc3

• add confirm_devices to progress functions to make sure finished_devices are updated

  indy-independence

  committed
  3c7ba36

Commits on Feb 8, 2023

• Merge pull request #287 from SUNET/feature.confirm_commit

  Show description for fcaa76a
  indy-independence

  authored
  fcaa76a

• allow json dumping of pathlib Path objects to string

  indy-independence

  committed
  3302ae4

• move swagger docs to /api/doc/ so everything is under /api path, which makes it easier when using a reverse-proxy in front etc like cnaas-front

  indy-independence

  committed
  ddbf01d

• add /settings/server path to get api server settings

  indy-independence

  committed
  7090d1d

• docs example for /settings/server

  indy-independence

  committed
  0b391a1

• convert api test_settings to pytest

  indy-independence

  committed
  ee67486

Commits on Feb 22, 2023

• expand interface ranges in get verify_peer_device so init works on interface range ports

  indy-independence

  committed
  124e0a8

• allow setting interface data untagged_vlan to null

  indy-independence

  committed
  eee258f

• don't return duplicates of neighbors in get_neighbors()

  indy-independence

  committed
  cfe4c9a

• Merge pull request #290 from SUNET/bugfix.set_untagged_vlan_null

  Show description for a4b9c31
  indy-independence

  authored
  a4b9c31

• make sure we compare to set in unittest

  indy-independence

  committed
  d8ed9e6

Commits on Feb 28, 2023

• try allowing capital letters in hostname and domain for settings

  indy-independence

  committed
  edd9f7f

Commits on Mar 1, 2023

• Merge pull request #291 from SUNET/bugfix.no_generate_duplicate_underlay_peers

  Show description for 5497065
  indy-independence

  authored
  5497065

• allow group names with - in name

  indy-independence

  committed
  1c9251c

Commits on Mar 2, 2023

• Support branch refspecs also for dhcpd image

  Show description for adde878
  lunkwill42

  committed
  adde878

Commits on Mar 17, 2023

• Merge pull request #294 from Uninett/sikt.feature.dhcpd-branches

  Show description for b45bdd4
  indy-independence

  authored
  b45bdd4

Commits on Mar 21, 2023

• include ssh-client in api container, useful for git clone via ssh

  indy-independence

  committed
  cdd0f3b

Commits on Mar 22, 2023

• try allowing capital letters in hostname and domain for settings

  indy-independence

  committed
  99875db

• allow group names with - in name

  indy-independence

  committed
  c668833

• security fixes for werkzeug

  indy-independence

  committed
  b660267

• update flask-restx to work with security fixed werkzeug

  indy-independence

  committed
  d930ff4

• Add changelog for 1.4.1

  indy-independence

  committed
  62f01d5

• Merge pull request #293 from SUNET/feature.hostname_capital_letters

  Show description for 8cbc79e
  indy-independence

  authored
  8cbc79e

• Merge pull request #289 from SUNET/feature.get_apisettings

  Show description for 97d0c0a
  indy-independence

  authored
  97d0c0a

• make dev docker-compose only bind to the br-cnaas bridge ip

  indy-independence

  committed
  5a780f5

Commits on Mar 29, 2023

• Revert "make dev docker-compose only bind to the br-cnaas bridge ip"

  Show description for 5bdc08c
  indy-independence

  committed
  5bdc08c

Commits on Mar 30, 2023

• this command must have been left uncommented after some cleanup in 6740188 I guess

  indy-independence

  committed
  8cbd098

Commits on Mar 31, 2023

• Add ipv6_gw field to mgmtdomain

  lunkwill42

  committed
  b91164c

• Accept ipv6_gw argument in mgmtdomain APIs

  Show description for d1a545d
  lunkwill42

  committed
  d1a545d

• Accept IPv6 addresses on all Device IP attrs

  lunkwill42

  committed
  ce1d4c3

• Add IPv6 to Mgmtdomain.find_free_mgmt_ip

  Show description for 4d88567
  lunkwill42

  committed
  4d88567

• Add new mgmtdomain_primary_ip_version api setting

  Show description for d2f59b3
  lunkwill42

  committed
  d2f59b3

• Add methods for dual-stack Mgmtdomain operations

  lunkwill42

  committed
  7d3a000

• Add secondary_management_ip to Device

  Show description for fd8c026
  lunkwill42

  committed
  fd8c026

• Assign secondary mgmt IPs in access device init

  Show description for 1a69c5a
  lunkwill42

  committed
  1a69c5a

• Add extra variables for secondary mgmt addr

  Show description for fada077
  lunkwill42

  committed
  fada077

• Allow both IPv4 and IPv6 mgmt ip vars

  lunkwill42

  committed
  6c3ccd5

• Include vars for secondary addrs on ACCESS devices

  Show description for ade0aab
  lunkwill42

  committed
  ade0aab

• Accept secondary_management_ip in device PUT

  Show description for 5e323f6
  lunkwill42

  committed
  5e323f6

Commits on Apr 4, 2023

• Merge pull request #266 from Uninett/feature.ipv6_mgmt_domains

  Show description for 35f1e82
  indy-independence

  authored
  35f1e82

Commits on Apr 6, 2023

• update redis for CVE-2023-28859

  indy-independence

  committed
  6af8535

• script to manually generate ESI IDs etc for custom ports

  indy-independence

  committed
  8a7ef84

Commits on Apr 11, 2023

• Merge pull request #298 from SUNET/feature.tool_generate_esi

  Show description for 1711362
  indy-independence

  authored
  1711362

Commits on Apr 26, 2023

• Fix bug on access->access ZTP where initcheck would fail when configtypes is downlink but data is empty/null

  indy-independence

  committed
  622125e

• Merge pull request #299 from SUNET/bugfix.redundant_option_notset

  Show description for 39b93ee
  indy-independence

  authored
  39b93ee

Commits on May 8, 2023

• Fix error "Error in repository: HEAD is a detached" when refreshing settings repo

  Show description for 668e0a1
  indy-independence

  committed
  668e0a1

• handle more error cases like when changing to/from other branch with syntax errors

  indy-independence

  committed
  d4c637e

• Merge pull request #300 from SUNET/bugfix.refresh_repo_detached_head

  Show description for 676793a
  indy-independence

  authored
  676793a

• Add missing docs for job API abort action

  indy-independence

  committed
  94f1609

• try to get github dependabot to not hang on this file

  indy-independence

  committed
  d40d911

Commits on May 9, 2023

• on PR wolkflow action use pr number to reference branch instead of branch name, since branch name only works for PRs from same repo and not from forks etc

  indy-independence

  committed
  86b6058

• Bump flask from 2.1.2 to 2.2.5

  Show description for e008fd2
  dependabot[bot]

  authored
  e008fd2

• Merge pull request #302 from SUNET/dependabot/pip/flask-2.2.5

  Show description for 5436b5a
  indy-independence

  authored
  5436b5a

• Bump redis from 4.3.6 to 4.4.4

  Show description for df6a0af
  dependabot[bot]

  authored
  df6a0af

• Merge pull request #301 from SUNET/dependabot/pip/redis-4.4.4

  Show description for 66ca745
  indy-independence

  authored
  66ca745

Commits on May 15, 2023

• When doing init of a new device, negihbor/uplink device sync statuses are checked, but if they have local changes saved the devices were not marked as unsynced in database

  indy-independence

  committed
  53666e8

Commits on May 22, 2023

• Merge pull request #303 from SUNET/bugfix.init_unsync_neighbor_save

  Show description for 119b58a
  indy-independence

  authored
  119b58a

Commits on Jun 13, 2023

• allow settings in base_system to have groups auto filtering

  indy-independence

  committed
  38f1aff

• Add support for users settings, with some vendor specific fields for password hashes and permissions. Add support for some campus settings dot1x_multi_host and poe_reboot_maintain

  indy-independence

  committed
  710ab1e

Commits on Jun 27, 2023

• Merge pull request #304 from SUNET/feature.settings_campus_spring2023

  Show description for 9be8560
  indy-independence

  authored
  9be8560

• routingpolicy and prefixset models

  indy-independence

  committed
  4d10c7d

Commits on Jun 29, 2023

• docs for prefix_sets and routing_policies

  indy-independence

  committed
  6976a5d

Commits on Jul 4, 2023

• Merge pull request #305 from SUNET/feature.routingpolicy_settings

  Show description for a43bb19
  indy-independence

  authored
  a43bb19

Commits on Aug 11, 2023

• Bump gitpython from 3.1.30 to 3.1.32

  Show description for 9b04fe8
  dependabot[bot]

  authored
  9b04fe8

Commits on Aug 18, 2023

• Merge pull request #307 from SUNET/dependabot/pip/gitpython-3.1.32

  Show description for d755dfe
  indy-independence

  authored
  d755dfe

Commits on Aug 29, 2023

• add dataclasses and methods for writing and reading sync history events from redis

  indy-independence

  committed
  2f00568

• use redis hget/hset on specific keys instead of entire hash to avoid thread issues hopefully

  indy-independence

  committed
  8c661a0

• make sure add_sync_event doesn't cause exceptions for anyone calling it, it's not critical for operations

  indy-independence

  committed
  f2a0c22

• create sync status events when devices are marked unsync. documentation for events

  indy-independence

  committed
  a5c5c47

• fix arguments when calling set_devtype_syncstatus without platform

  indy-independence

  committed
  a200101

• Bugfix, make refresh repo git-diff only check file changes in current branch

  indy-independence

  committed
  72abe61

• Add API call to query synchistory events for a single device or all devices

  indy-independence

  committed
  787662d

• API call to manually add synchistory event. clear synchistory event if device PUT API is called with synchronized=true

  indy-independence

  committed
  ea3181e

• Add more unittests and docs for synchistory events

  indy-independence

  committed
  c0982ab

• fix add_sync_event argument order

  indy-independence

  committed
  0255e1a

• don't add unsync event when dryrun discovers diff on already unsync device

  indy-independence

  committed
  3038205

• send websocket message for sync events

  indy-independence

  committed
  7fe374d

• join sync room if sync key exists with value 'all'

  indy-independence

  committed
  f3ed2b6

• send hostname and only new syncevent data

  indy-independence

  committed
  6d8bb5a

• Make device unsynchronized and remove sync events when device is moved to unmanaged state

  indy-independence

  committed
  412d158

• Add was_unmanaged syncevent if device is moved from unmanaged to managed

  indy-independence

  committed
  d01376b

• Make was_unmanaged syncevent store the 'by' user

  indy-independence

  committed
  e78002b

• use correct commit mode override values inside push_sync_device task

  indy-independence

  committed
  1dba7dc

• fix confirm_mode option for syncto API

  indy-independence

  committed
  96b0a67

• Make factory default device also mark neighbors as unsync, and rollback if unsuccessful in deleting from database. Do logging with job_id

  indy-independence

  committed
  97d29e4

• return job_id as separate field when submitting device delet job

  indy-independence

  committed
  4bb5825

• return http status 500 for device delete exceptions

  indy-independence

  committed
  6ab80bf

• remove sync events when deleting a device

  indy-independence

  committed
  5a606ae

• Try fix for "Too many packets in payload" from engineio

  Show description for 02e4dc2
  indy-independence

  committed
  02e4dc2

• Add device_update websocket events for action CREATED and DELETED

  indy-independence

  committed
  92e980b

• change to maintained fork

  indy-independence

  committed
  e079bc1

Commits on Aug 31, 2023

• Merge pull request #306 from SUNET/feature.sync_history

  Show description for 314b233
  indy-independence

  authored
  314b233

• Add API to query full running config from a device, or running config for a specific interface

  indy-independence

  committed
  a926a90

Commits on Sep 6, 2023

• Bump gitpython from 3.1.32 to 3.1.34

  Show description for f584c2f
  dependabot[bot]

  authored
  f584c2f

Commits on Sep 8, 2023

• If commit confirmed mode 2 is used and all devices succeed with push,

  Show description for ffe826b
  indy-independence

  committed
  ffe826b

• use get facts since ping is about pinging another external device

  indy-independence

  committed
  66f245a

• Log error device not reachable after commit

  indy-independence

  committed
  d2a56e9

• task.run fails with exception, so catch that instead of checking fail status

  indy-independence

  committed
  1c6c736

• Configurable time to wait between comitting configuration and checking

  Show description for 97367a9
  indy-independence

  committed
  97367a9

• Make sure commit confirm 2 checks for failed hosts before hosts with empty diff,

  Show description for 897cb2f
  indy-independence

  committed
  897cb2f

• Fix for missing log events, before events added between two xread

  Show description for 1c6f069
  indy-independence

  committed
  1c6f069

Commits on Sep 11, 2023

• workaround for scheduled_by arg not passed to job wrapped functions

  indy-independence

  committed
  39c98ad

• better error handling of napalm_configure fails

  indy-independence

  committed
  769276b

• wait to release device lock if one or more devices failed configuration with commit confirmed mode 1

  indy-independence

  committed
  6343563

• Fix the order of printing of log message to release job lock

  indy-independence

  committed
  33cacd7

Commits on Sep 18, 2023

• Support junos on get_running_config_interface, and remove first whitespace level

  indy-independence

  committed
  652cfca

• Merge pull request #308 from SUNET/feature.get_running_config

  Show description for 38ae417
  indy-independence

  authored
  38ae417

• Merge pull request #309 from SUNET/dependabot/pip/gitpython-3.1.34

  Show description for 3a7812b
  indy-independence

  authored
  3a7812b

• Bump gitpython from 3.1.34 to 3.1.35

  Show description for f4aa4bc
  dependabot[bot]

  authored
  f4aa4bc

• Add tags support for dist interfaces

  indy-independence

  committed
  b59c59a

• Fix unattached syncevents after init of new device

  indy-independence

  committed
  0964fcb

• Option to filter output from generate only api call

  indy-independence

  committed
  550e368

• Option to filter output from generate only api call

  indy-independence

  committed
  b0fd8b2

Commits on Sep 19, 2023

• Merge pull request #313 from SUNET/dependabot/pip/gitpython-3.1.35

  Show description for 17c888c
  indy-independence

  authored
  17c888c

Commits on Sep 20, 2023

• Use field masks from flask_restx marshal to do "partial object fetching" using X-Fields header instead of custom query params, hopefully a bit more standard and more flexible

  Show description for fe3be7e
  indy-independence

  committed
  fe3be7e

• Merge pull request #314 from SUNET/feature.filter_generate_output

  Show description for c855fcf
  indy-independence

  authored
  c855fcf

• Merge pull request #317 from SUNET/bugfix.confirmed_2_failed_after_push_ok

  Show description for af4a55e
  indy-independence

  authored
  af4a55e

• Changelog v1.5

  indy-independence

  committed
  a03d808

• version 1.5b1

  indy-independence

  committed
  6c9626b

Commits on Sep 21, 2023

• Some newer 4.30 image with compression stuff takes even longer to activate, change timeout 60s to 120s

  indy-independence

  committed
  a7ff2bc

• fix for using override confirm mode 0 evaluating to false/none

  indy-independence

  committed
  2df4cf5

Commits on Sep 22, 2023

• Add support for IP settings in interfaces.yml

  indy-independence

  committed
  a50a3f8

Commits on Sep 25, 2023

• Version 1.5.0

  indy-independence

  committed
  dd461cd

• Merge branch 'master' into develop

  indy-independence

  authored
  ea46777