Catalogs 'from-puppetdb' contain differences to catalogs from puppetserver

[alexjfisher]

Hi!
I'm just starting out with the tool and have been trying to use the --from-puppetdb option. (Should this option be documented here?)

octocatalog-diff  -n node1.example.com \
                  --from-puppetdb \
                  -t production \
                  --to-puppet-master puppet.example.com

In the above example, the node is already in the production environment and running puppet. I would therefore expect there to be no differences. Unfortunately, I get quite a few. I think I could filter them out with some ignore options, but I was planning to use the tool in a CI setup (seeing which nodes will be affected by a change), so I don't think this would be practical.

On my servers I'm using CentOS 7 with...
puppetserver-2.7.2
puppetdb-4.4.0 (with postgresql 9.6)
puppet-agent-1.10.4

I'm running octocatalog-diff 1.4.0 on ruby 2.4.1 via rvm on ubuntu.

Many thanks,
Alex

  Apache::Listen[443] =>
   parameters =>
     alias =>
      - ["443"]
   title =>
    - "443"
    + 443
*******************************************
  File[/etc/httpd/conf.d/ssl.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.d/ssl.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/access_compat.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/access_compat.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/actions.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/actions.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/alias.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/alias.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/alias.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/alias.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/auth_basic.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/auth_basic.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/auth_digest.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/auth_digest.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_anon.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_anon.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_core.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_core.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_dbm.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_dbm.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_file.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_file.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authnz_ldap.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authnz_ldap.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/authnz_ldap.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authnz_ldap.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_core.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_core.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_dbm.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_dbm.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_groupfile.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_groupfile.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_host.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_host.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_owner.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_owner.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_user.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_user.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/autoindex.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/autoindex.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/autoindex.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/autoindex.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/cache.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/cache.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/cgi.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/cgi.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/dav.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dav.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/dav_fs.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dav_fs.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/dav_fs.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dav_fs.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/deflate.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/deflate.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/deflate.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/deflate.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/dir.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dir.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/dir.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dir.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/env.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/env.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/expires.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/expires.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/ext_filter.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ext_filter.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/filter.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/filter.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/include.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/include.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/ldap.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ldap.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/ldap.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ldap.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/log_config.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/log_config.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/logio.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/logio.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime_magic.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime_magic.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime_magic.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime_magic.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/negotiation.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/negotiation.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/negotiation.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/negotiation.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/proxy.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/proxy.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/proxy.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/proxy.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/proxy_http.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/proxy_http.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/rewrite.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/rewrite.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/setenvif.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/setenvif.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/setenvif.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/setenvif.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/socache_shmcb.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/socache_shmcb.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/speling.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/speling.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/ssl.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ssl.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/substitute.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/substitute.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/suexec.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/suexec.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/systemd.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/systemd.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/unixd.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/unixd.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/usertrack.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/usertrack.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/version.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/version.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/vhost_alias.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/vhost_alias.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/wsgi.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/wsgi.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/wsgi.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/wsgi.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/xsendfile.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/xsendfile.load"]
*******************************************
  File[/etc/pulp/server.conf] =>
   parameters =>
     content =>
      @@ -35,5 +35,5 @@
       #                    the ca_cert parameter must point to a file of CA certificates used to
       #                    validate the connection.
      -# ca_path:           The ca_certs file contains a set of concatenated “certification authority”
      +# ca_path:           The ca_certs file contains a set of concatenated �certification authority�
       #                    certificates, which are used to validate certificates passed from the other end
       #                    of the connection.
*******************************************
  File[/etc/pulp/vhosts80] =>
   parameters =>
     alias =>
      - ["/etc/pulp/vhosts80"]
*******************************************
  Service[mongodb] =>
   parameters =>
     alias =>
      - ["mongod"]
*******************************************
  Service[ntp] =>
   parameters =>
     alias =>
      - ["ntpd"]
*******************************************
  Service[squid3_service] =>
   parameters =>
     alias =>
      - ["squid"]
*******************************************
  Stage[main] =>
   parameters =>
     alias =>
      - ["main"]
*******************************************

[alexjfisher]

Perhaps getting the from catalog from puppetdb can't ever work out that well. :(

https://docs.puppet.com/puppetdb/4.4/api/wire_format/catalog_format_v9.html

PuppetDB receives catalogs from Puppet masters in the following wire format. This format is subtly different from the internal format used by Puppet, so catalogs are converted by the PuppetDB catalog terminus before they are sent.

[christopherwood]

I think I ran into something similar this while attempting to diff a from-catalog right from puppetdb 4.4 with a to-catalog downloaded from puppetdb 5.1 and living in a file under /tmp.

octocatalog-diff --from-puppetdb --to-catalog /tmp/5_myhost.json --hostname myhost.me.com

BUG: catalog has no data::resources or ::resources array. Please report this.

I notice that in both versions of the puppetdb catalog "resources" is a hash (resources[data] is an array), but in the version used by the puppet agent for the same server "resources" is an array.