Use JSON output and input where available #74

paleolimbot · 2021-12-27T01:25:03Z

No description provided.

… on linux

paleolimbot · 2022-01-10T12:21:16Z

@florisvdh @nyalldawson here's a PR of the improvements from my end! It still needs some test added but it's here for you to try if you'd like.

The main thing is that I'm running into problems using JSON as output on Linux...the --json flag somehow makes it impossible for qgisprocess to find PROJ (I sent Nyall an email about it for when he's back from vacation!). I don't know if that's a problem limited to the Ubuntu GitHub Actions CI but may Floris can give this a test. You probably have to do options(qgisprocess.use_json_input = TRUE, qgisprocess.use_json_output = TRUE) and then you should get:

library(qgisprocess)
#> Using 'qgis_process' at '/Applications/QGIS-Dev.app/Contents/MacOS/bin/qgis_process'.
#> QGIS version: 3.23.0-Master
#> Configuration loaded from '~/Library/Caches/R-qgisprocess/cache-0.0.0.9000.rds'
#> Run `qgis_configure()` for details.
#> Using JSON for input serialization
#> Using JSON for output serialization

Another problem I'm running into is that when using the JSON output, the ordering of the parameters and outputs is different than in the help file (which is the same as in the dialog box). This matters because the parameters and outputs are often ordered as "most important first" (but I can also just scrape the help text for the ordering like I was doing before).

Example to play with:

library(qgisprocess)
#> Using 'qgis_process' at '/Applications/QGIS-Dev.app/Contents/MacOS/bin/qgis_process'.
#> QGIS version: 3.23.0-Master
#> Configuration loaded from '~/Library/Caches/R-qgisprocess/cache-0.0.0.9000.rds'
#> Run `qgis_configure()` for details.
#> Using JSON for input serialization
#> Using JSON for output serialization

curl::curl_download(
  "https://github.com/qgis/QGIS/raw/master/resources/data/world_map.gpkg",
  "world_map.gpkg"
)

json_input <- '
{
  "area_units": "m2",
  "distance_units": "meters",
  "ellipsoid": "EPSG:7030",
  "inputs": {
    "AGGREGATES": [
      {
        "aggregate": "first_value",
        "delimiter": ",",
        "input": "\\"admin\\"",
        "length": 36,
        "name": "admin",
        "precision": 0,
        "type": 10
      },
      {
        "aggregate": "concatenate",
        "delimiter": ",",
        "input": "\\"name\\"",
        "length": 3000,
        "name": "name",
        "precision": 0,
        "type": 10
      }
    ],
    "GROUP_BY": "\\"admin\\"",
    "INPUT": "world_map.gpkg|layername=states_provinces",
    "OUTPUT": "ogr:dbname=\'output_states.gpkg\' table=\\"output\\" (geom)"
  }
}
'

(result <- qgis_run_algorithm("native:aggregate", .raw_json_input = json_input))
#> JSON input ----
#> {
#>   "area_units": "m2",
#>   "distance_units": "meters",
#>   "ellipsoid": "EPSG:7030",
#>   "inputs": {
#>     "AGGREGATES": [
#>       {
#>         "aggregate": "first_value",
#>         "delimiter": ",",
#>         "input": "\"admin\"",
#>         "length": 36,
#>         "name": "admin",
#>         "precision": 0,
#>         "type": 10
#>       },
#>       {
#>         "aggregate": "concatenate",
#>         "delimiter": ",",
#>         "input": "\"name\"",
#>         "length": 3000,
#>         "name": "name",
#>         "precision": 0,
#>         "type": 10
#>       }
#>     ],
#>     "GROUP_BY": "\"admin\"",
#>     "INPUT": "world_map.gpkg|layername=states_provinces",
#>     "OUTPUT": "ogr:dbname='output_states.gpkg' table=\"output\" (geom)"
#>   }
#> }
#> 
#> Running /Applications/QGIS-Dev.app/Contents/MacOS/bin/qgis_process --json run \
#>   'native:aggregate' -
#> <Result of `qgis_run_algorithm("native:aggregate", ...)`>
#> List of 1
#>  $ OUTPUT: 'qgis_outputVector' chr "output_states.gpkg|layername=output"

qgis_run_algorithm(
  "native:aggregate",
  AGGREGATES = list(
    list(
      aggregate = "first_value",
      delimiter = ",",
      input = '"admin"',
      length = 36,
      name = "admin",
      precision = 0,
      type = 10
    ),
    list(
      aggregate = "concatenate",
      delimiter = ",",
      input = '"name"',
      length = 3000,
      name = "name",
      precision = 0,
      type = 10
    )
  ),
  INPUT = "world_map.gpkg|layername=states_provinces",
  GROUP_BY = "admin"
)
#> Using `OUTPUT = qgis_tmp_vector()`
#> JSON input ----
#> {
#>   "inputs": {
#>     "INPUT": "world_map.gpkg|layername=states_provinces",
#>     "GROUP_BY": "admin",
#>     "AGGREGATES": [
#>       {
#>         "aggregate": "first_value",
#>         "delimiter": ",",
#>         "input": "\"admin\"",
#>         "length": 36,
#>         "name": "admin",
#>         "precision": 0,
#>         "type": 10
#>       },
#>       {
#>         "aggregate": "concatenate",
#>         "delimiter": ",",
#>         "input": "\"name\"",
#>         "length": 3000,
#>         "name": "name",
#>         "precision": 0,
#>         "type": 10
#>       }
#>     ],
#>     "OUTPUT": "/var/folders/gt/l87wjg8s7312zs9s7c1fgs900000gn/T//RtmpYyUhAb/file49cf1f189760/file49cf16447612.gpkg"
#>   }
#> }
#> 
#> Running /Applications/QGIS-Dev.app/Contents/MacOS/bin/qgis_process --json run \
#>   'native:aggregate' -
#> <Result of `qgis_run_algorithm("native:aggregate", ...)`>
#> List of 1
#>  $ OUTPUT: 'qgis_outputVector' chr "/var/folders/gt/l87wjg8s7312zs9s7c1fgs900000gn/T//RtmpYyUhAb/file49cf1f189760/file49cf16447612.gpkg"

R/qgis-run-algorithm.R

florisvdh · 2022-01-14T16:49:11Z

This seems to work well here @paleolimbot, a few things to note but essentially works 🎉!!

First of all, I got no problems with setting the options and then loading qgisprocess (not sure if I get you right there):

options(qgisprocess.path = "/home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process",
        qgisprocess.use_json_input = TRUE,
        qgisprocess.use_json_output = TRUE)
library(qgisprocess)
#> Using 'qgis_process' at '/home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process'.
#> QGIS version: 3.23.0-Master
#> Configuration loaded from '~/.cache/R-qgisprocess/cache-0.0.0.9000.rds'
#> Run `qgis_configure()` for details.
#> Using JSON for input serialization
#> Using JSON for output serialization

^{Created on 2022-01-14 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Linux Mint 20
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language nl_BE:nl
#>  collate  nl_BE.UTF-8
#>  ctype    nl_BE.UTF-8
#>  tz       Europe/Brussels
#>  date     2022-01-14
#>  pandoc   2.14.0.3 @ /usr/lib/rstudio/bin/pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version     date (UTC) lib source
#>  cli           3.1.0       2021-10-27 [1] CRAN (R 4.1.2)
#>  digest        0.6.29      2021-12-01 [1] CRAN (R 4.1.2)
#>  evaluate      0.14        2019-05-28 [1] CRAN (R 4.1.0)
#>  fastmap       1.1.0       2021-01-25 [1] CRAN (R 4.1.0)
#>  fs            1.5.2       2021-12-08 [1] CRAN (R 4.1.2)
#>  glue          1.5.1       2021-11-30 [1] CRAN (R 4.1.2)
#>  highr         0.9         2021-04-16 [1] CRAN (R 4.1.0)
#>  htmltools     0.5.2       2021-08-25 [1] CRAN (R 4.1.1)
#>  knitr         1.36        2021-09-29 [1] CRAN (R 4.1.1)
#>  magrittr      2.0.1       2020-11-17 [1] CRAN (R 4.1.0)
#>  qgisprocess * 0.0.0.9000  2022-01-14 [1] Github (paleolimbot/qgisprocess@8f421b4)
#>  rappdirs      0.3.3       2021-01-31 [1] CRAN (R 4.1.0)
#>  reprex        2.0.1       2021-08-05 [1] CRAN (R 4.1.1)
#>  rlang         0.99.0.9009 2021-11-18 [1] local
#>  rmarkdown     2.11        2021-09-14 [1] CRAN (R 4.1.1)
#>  rstudioapi    0.13        2020-11-12 [1] CRAN (R 4.1.0)
#>  sessioninfo   1.2.2       2021-12-06 [1] CRAN (R 4.1.2)
#>  stringi       1.7.6       2021-11-29 [1] CRAN (R 4.1.2)
#>  stringr       1.4.0       2019-02-10 [1] CRAN (R 4.1.0)
#>  vctrs         0.3.8       2021-04-29 [1] CRAN (R 4.1.0)
#>  withr         2.4.3       2021-11-30 [1] CRAN (R 4.1.2)
#>  xfun          0.29        2021-12-14 [1] CRAN (R 4.1.2)
#>  yaml          2.2.1       2020-02-01 [1] CRAN (R 4.1.0)
#> 
#>  [1] /home/floris/lib/R/library
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Below reprex applies your code plus a few extras:

Code and output (click to expand)

options(qgisprocess.path = "/home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process")

library(qgisprocess)
#> Using 'qgis_process' at '/home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process'.
#> QGIS version: 3.23.0-Master
#> Configuration loaded from '~/.cache/R-qgisprocess/cache-0.0.0.9000.rds'
#> Run `qgis_configure()` for details.
library(sf)
#> Linking to GEOS 3.9.1, GDAL 3.3.2, PROJ 7.2.1

curl::curl_download(
  "https://github.com/qgis/QGIS/raw/master/resources/data/world_map.gpkg",
  "world_map.gpkg"
)

### Providing algo arguments as JSON input (invokes JSON output)

json_input <- '
{
  "area_units": "m2",
  "distance_units": "meters",
  "ellipsoid": "EPSG:7030",
  "inputs": {
    "AGGREGATES": [
      {
        "aggregate": "first_value",
        "delimiter": ",",
        "input": "\\"admin\\"",
        "length": 36,
        "name": "admin",
        "precision": 0,
        "type": 10
      },
      {
        "aggregate": "concatenate",
        "delimiter": ",",
        "input": "\\"name\\"",
        "length": 3000,
        "name": "name",
        "precision": 0,
        "type": 10
      }
    ],
    "GROUP_BY": "\\"admin\\"",
    "INPUT": "world_map.gpkg|layername=states_provinces",
    "OUTPUT": "ogr:dbname=\'output_states.gpkg\' table=\\"output\\" (geom)"
  }
}
'

(result <- qgis_run_algorithm("native:aggregate", .raw_json_input = json_input))
#> JSON input ----
#> {
#>   "area_units": "m2",
#>   "distance_units": "meters",
#>   "ellipsoid": "EPSG:7030",
#>   "inputs": {
#>     "AGGREGATES": [
#>       {
#>         "aggregate": "first_value",
#>         "delimiter": ",",
#>         "input": "\"admin\"",
#>         "length": 36,
#>         "name": "admin",
#>         "precision": 0,
#>         "type": 10
#>       },
#>       {
#>         "aggregate": "concatenate",
#>         "delimiter": ",",
#>         "input": "\"name\"",
#>         "length": 3000,
#>         "name": "name",
#>         "precision": 0,
#>         "type": 10
#>       }
#>     ],
#>     "GROUP_BY": "\"admin\"",
#>     "INPUT": "world_map.gpkg|layername=states_provinces",
#>     "OUTPUT": "ogr:dbname='output_states.gpkg' table=\"output\" (geom)"
#>   }
#> }
#> 
#> Running \
#>   /home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process \
#>   --json run 'native:aggregate' -
#> qt5ct: using qt5ct plugin
#> <Result of `qgis_run_algorithm("native:aggregate", ...)`>
#> List of 1
#>  $ OUTPUT: 'qgis_outputVector' chr "output_states.gpkg|layername=output"

st_as_sf(result)  # does not yet support the QGIS syntax of referring a layer
#> Error: Cannot open "output_states.gpkg|layername=output"; The file doesn't seem to exist.

result_sf <- st_read("output_states.gpkg", "output")
#> Reading layer `output' from data source 
#>   `/tmp/RtmpF6G0Xd/reprex-f2a566d283c6-used-aidi/output_states.gpkg' 
#>   using driver `GPKG'
#> Simple feature collection with 251 features and 2 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: -179.9 ymin: -89.9 xmax: 179.9 ymax: 83.6341
#> Geodetic CRS:  WGS 84
plot(result_sf["admin"])

### Providing algo arguments as function arguments, but enforcing JSON output (--json)

options(qgisprocess.use_json_output = TRUE)
qgis_use_json_output()
#> [1] TRUE

qgis_run_algorithm(
  "native:aggregate",
  AGGREGATES = list(
    list(
      aggregate = "first_value",
      delimiter = ",",
      input = '"admin"',
      length = 36,
      name = "admin",
      precision = 0,
      type = 10
    ),
    list(
      aggregate = "concatenate",
      delimiter = ",",
      input = '"name"',
      length = 3000,
      name = "name",
      precision = 0,
      type = 10
    )
  ),
  INPUT = "world_map.gpkg|layername=states_provinces",
  GROUP_BY = "admin"
)
#> Error: Don't know how to convert object of type 'list' to QGIS type 'aggregates'

# It seems that the AGGREGATES argument (list) is not accepted here.
# Adding options(qgisprocess.use_json_input = TRUE) did not solve this.

qgis_arguments("native:aggregate")
#> # A tibble: 4 × 6
#>   name   description   qgis_type default_value available_values acceptable_valu…
#>   <chr>  <chr>         <chr>     <list>        <list>           <list>          
#> 1 INPUT  Input layer   source    <NULL>        <NULL>           <chr [1]>       
#> 2 GROUP… Group by exp… expressi… <chr [1]>     <NULL>           <chr [1]>       
#> 3 AGGRE… Aggregates    aggregat… <NULL>        <NULL>           <list [0]>      
#> 4 OUTPUT Aggregated    sink      <NULL>        <NULL>           <chr [1]>

# Hence trying with another algorithm - this runs with `--json` and it works.

result2 <-
  qgis_run_algorithm(
    "native:extractbyattribute",
    FIELD = "NAME",
    INPUT = "world_map.gpkg|layername=countries",
    OPERATOR = 0,
    VALUE = "Norway"
  )
#> Using `OUTPUT = qgis_tmp_vector()`
#> Using `FAIL_OUTPUT = qgis_tmp_vector()`
#> Running \
#>   /home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process \
#>   --json run 'native:extractbyattribute' \
#>   '--INPUT=world_map.gpkg|layername=countries' '--FIELD=NAME' '--OPERATOR=0' \
#>   '--VALUE=Norway' \
#>   '--OUTPUT=/tmp/RtmpeMFzVJ/filef54f4d11c8ad/filef54f3b7fd15b.gpkg' \
#>   '--FAIL_OUTPUT=/tmp/RtmpeMFzVJ/filef54f4d11c8ad/filef54f72dd9f83.gpkg'
#> qt5ct: using qt5ct plugin

# st_as_sf(result2)  # issue #76

(read_sf(result2$OUTPUT))
#> Simple feature collection with 1 feature and 6 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: -9.117421 ymin: -54.45566 xmax: 33.64039 ymax: 80.76948
#> Geodetic CRS:  WGS 84
#> # A tibble: 1 × 7
#>   iso_a2 NAME   FIPS_10_ ISO_A3 WB_A2 WB_A3                                 geom
#>   <chr>  <chr>  <chr>    <chr>  <chr> <chr>                   <MULTIPOLYGON [°]>
#> 1 NO     Norway NO       NOR    NO    NOR   (((3.457286 -54.39007, 3.486664 -54…

^{Created on 2022-01-14 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Linux Mint 20
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language nl_BE:nl
#>  collate  nl_BE.UTF-8
#>  ctype    nl_BE.UTF-8
#>  tz       Europe/Brussels
#>  date     2022-01-14
#>  pandoc   2.14.0.3 @ /usr/lib/rstudio/bin/pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version     date (UTC) lib source
#>  assertthat    0.2.1       2019-03-21 [1] CRAN (R 4.1.0)
#>  class         7.3-19      2021-05-03 [4] CRAN (R 4.1.0)
#>  classInt      0.4-3       2020-04-07 [1] CRAN (R 4.1.0)
#>  cli           3.1.0       2021-10-27 [1] CRAN (R 4.1.2)
#>  crayon        1.4.2       2021-10-29 [1] CRAN (R 4.1.2)
#>  curl          4.3.2       2021-06-23 [1] CRAN (R 4.1.1)
#>  DBI           1.1.1       2021-01-15 [1] CRAN (R 4.1.0)
#>  digest        0.6.29      2021-12-01 [1] CRAN (R 4.1.2)
#>  dplyr         1.0.7       2021-06-18 [1] CRAN (R 4.1.1)
#>  e1071         1.7-9       2021-09-16 [1] CRAN (R 4.1.1)
#>  ellipsis      0.3.2       2021-04-29 [1] CRAN (R 4.1.0)
#>  evaluate      0.14        2019-05-28 [1] CRAN (R 4.1.0)
#>  fansi         0.5.0       2021-05-25 [1] CRAN (R 4.1.0)
#>  fastmap       1.1.0       2021-01-25 [1] CRAN (R 4.1.0)
#>  fs            1.5.2       2021-12-08 [1] CRAN (R 4.1.2)
#>  generics      0.1.1       2021-10-25 [1] CRAN (R 4.1.2)
#>  glue          1.5.1       2021-11-30 [1] CRAN (R 4.1.2)
#>  highr         0.9         2021-04-16 [1] CRAN (R 4.1.0)
#>  htmltools     0.5.2       2021-08-25 [1] CRAN (R 4.1.1)
#>  httr          1.4.2       2020-07-20 [1] CRAN (R 4.1.0)
#>  jsonlite      1.7.2       2020-12-09 [1] CRAN (R 4.1.0)
#>  KernSmooth    2.23-20     2021-05-03 [4] CRAN (R 4.1.0)
#>  knitr         1.36        2021-09-29 [1] CRAN (R 4.1.1)
#>  lifecycle     1.0.1       2021-09-24 [1] CRAN (R 4.1.1)
#>  magrittr      2.0.1       2020-11-17 [1] CRAN (R 4.1.0)
#>  mime          0.12        2021-09-28 [1] CRAN (R 4.1.1)
#>  pillar        1.6.4       2021-10-18 [1] CRAN (R 4.1.2)
#>  pkgconfig     2.0.3       2019-09-22 [1] CRAN (R 4.1.0)
#>  processx      3.5.2       2021-04-30 [1] CRAN (R 4.1.0)
#>  proxy         0.4-26      2021-06-07 [1] CRAN (R 4.1.0)
#>  ps            1.6.0       2021-02-28 [1] CRAN (R 4.1.0)
#>  purrr         0.3.4       2020-04-17 [1] CRAN (R 4.1.0)
#>  qgisprocess * 0.0.0.9000  2022-01-14 [1] Github (paleolimbot/qgisprocess@8f421b4)
#>  R6            2.5.1       2021-08-19 [1] CRAN (R 4.1.1)
#>  rappdirs      0.3.3       2021-01-31 [1] CRAN (R 4.1.0)
#>  Rcpp          1.0.7       2021-07-07 [1] CRAN (R 4.1.1)
#>  reprex        2.0.1       2021-08-05 [1] CRAN (R 4.1.1)
#>  rlang         0.99.0.9009 2021-11-18 [1] local
#>  rmarkdown     2.11        2021-09-14 [1] CRAN (R 4.1.1)
#>  rstudioapi    0.13        2020-11-12 [1] CRAN (R 4.1.0)
#>  sessioninfo   1.2.2       2021-12-06 [1] CRAN (R 4.1.2)
#>  sf          * 1.0-4       2021-11-14 [1] CRAN (R 4.1.2)
#>  stringi       1.7.6       2021-11-29 [1] CRAN (R 4.1.2)
#>  stringr       1.4.0       2019-02-10 [1] CRAN (R 4.1.0)
#>  tibble        3.1.6       2021-11-07 [1] CRAN (R 4.1.2)
#>  tidyselect    1.1.1       2021-04-30 [1] CRAN (R 4.1.0)
#>  units         0.7-2       2021-06-08 [1] CRAN (R 4.1.1)
#>  utf8          1.2.2       2021-07-24 [1] CRAN (R 4.1.1)
#>  vctrs         0.3.8       2021-04-29 [1] CRAN (R 4.1.0)
#>  withr         2.4.3       2021-11-30 [1] CRAN (R 4.1.2)
#>  xfun          0.29        2021-12-14 [1] CRAN (R 4.1.2)
#>  xml2          1.3.3       2021-11-30 [1] CRAN (R 4.1.2)
#>  yaml          2.2.1       2020-02-01 [1] CRAN (R 4.1.0)
#> 
#>  [1] /home/floris/lib/R/library
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

I couldn't get the AGGREGATES list argument to work within R when not providing as JSON directly. While debugging, this argument gets stuck in below code since there is no as_qgis_argument.list method - so I don't get how it works with your code above 🙃.

https://github.com/paleolimbot/qgisprocess/blob/8f421b43b05df40538e57b64829ca1ca27cb942e/R/qgis-arguments.R#L82-L85

However, with another algorithm the --json option runs fine in Linux, so this works both with function arguments (dots in qgis_run_algorithm()) and with JSON input.

florisvdh · 2022-01-14T18:48:04Z

@paleolimbot let me know if you'd like to have further things tested. I didn't encounter the problem with --json, but from the commits it seems you also provided workarounds, so perhaps I should rerun with another (earlier?) package version.

In the GH Actions there is https://github.com/paleolimbot/qgisprocess/runs/4645887045?check_suite_focus=true#step:11:310, where indeed proj.db is not found. Could it be caused by the software versions? My above test uses the ubuntugis-unstable (PPA) version of geospatial libraries (though not yet updated to the recently available update), as also recommended in https://geocompr.github.io/post/2020/installing-r-spatial-ubuntu/, cf. below.

$ apt-cache search libproj[0-9]
libproj15 - Cartographic projection library
libproj19 - Cartographic projection library
libproj22 - Cartographic projection library
$
$ apt policy libproj15 libproj19 libproj22
libproj15:
  Installed: (none)
  Candidate: 6.3.1-1
  Version table:
     6.3.1-1 500
        500 http://ftp.belnet.be/ubuntu focal/universe amd64 Packages
libproj19:
  Installed: 7.2.1-1~focal0
  Candidate: 7.2.1-1~focal0
  Version table:
 *** 7.2.1-1~focal0 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status
libproj22:
  Installed: (none)
  Candidate: 8.2.0-1~focal0
  Version table:
     8.2.0-1~focal0 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu focal/main amd64 Packages
$ 
$ apt policy qgis
qgis:
  Installed: 1:3.22.2+32focal-ubuntugis
  Candidate: 1:3.22.2+32focal-ubuntugis
  Version table:
 *** 1:3.22.2+32focal-ubuntugis 500
        500 https://qgis.org/ubuntugis focal/main amd64 Packages
        100 /var/lib/dpkg/status
     3.16.14+dfsg-1~focal0 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu focal/main amd64 Packages
     3.10.4+dfsg-1ubuntu2 500
        500 http://ftp.belnet.be/ubuntu focal/universe amd64 Packages

Co-authored-by: Floris Vanderhaeghe <[email protected]>

florisvdh · 2022-01-17T14:00:20Z

Now the standard approach with a list argument works too! 👍

--json is used by default both with and without JSON input.

Reprex using ± latest QGIS master, GEOS 3.10.1, GDAL 3.4.0, PROJ 8.2.0 (click to expand)

options(qgisprocess.path = "/home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process")

library(qgisprocess)
#> Using 'qgis_process' at '/home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process'.
#> QGIS version: 3.23.0-Master
#> Configuration loaded from '~/.cache/R-qgisprocess/cache-0.0.0.9000.rds'
#> Run `qgis_configure()` for details.
#> Using JSON for input serialization
#> Using JSON for output serialization
library(sf)
#> Linking to GEOS 3.10.1, GDAL 3.4.0, PROJ 8.2.0; sf_use_s2() is TRUE

cat(qgis_run("--version")$stdout)
#> QGIS 3.23.0-Master 'Master' (8ef270b2)
#> QGIS code revision 8ef270b2
#> Qt version 5.12.8
#> Python version 3.8.10
#> GDAL/OGR version 3.4.0
#> PROJ version 8.2.0
#> EPSG Registry database version v10.038 (2021-10-21)
#> GEOS version 3.10.1-CAPI-1.16.0
#> SQLite version 3.31.1
#> OS Linux Mint 20

qgis_use_json_output()
#> [1] TRUE

curl::curl_download(
  "https://github.com/qgis/QGIS/raw/master/resources/data/world_map.gpkg",
  "world_map.gpkg"
)

### Providing algorithm arguments as JSON input
####################################################################

json_input <- '
{
  "area_units": "m2",
  "distance_units": "meters",
  "ellipsoid": "EPSG:7030",
  "inputs": {
    "AGGREGATES": [
      {
        "aggregate": "first_value",
        "delimiter": ",",
        "input": "\\"admin\\"",
        "length": 36,
        "name": "admin",
        "precision": 0,
        "type": 10
      },
      {
        "aggregate": "concatenate",
        "delimiter": ",",
        "input": "\\"name\\"",
        "length": 3000,
        "name": "name",
        "precision": 0,
        "type": 10
      }
    ],
    "GROUP_BY": "\\"admin\\"",
    "INPUT": "world_map.gpkg|layername=states_provinces",
    "OUTPUT": "ogr:dbname=\'output_states.gpkg\' table=\\"output\\" (geom)"
  }
}
'

(result <- qgis_run_algorithm("native:aggregate", .raw_json_input = json_input))
#> JSON input ----
#> {
#>   "area_units": "m2",
#>   "distance_units": "meters",
#>   "ellipsoid": "EPSG:7030",
#>   "inputs": {
#>     "AGGREGATES": [
#>       {
#>         "aggregate": "first_value",
#>         "delimiter": ",",
#>         "input": "\"admin\"",
#>         "length": 36,
#>         "name": "admin",
#>         "precision": 0,
#>         "type": 10
#>       },
#>       {
#>         "aggregate": "concatenate",
#>         "delimiter": ",",
#>         "input": "\"name\"",
#>         "length": 3000,
#>         "name": "name",
#>         "precision": 0,
#>         "type": 10
#>       }
#>     ],
#>     "GROUP_BY": "\"admin\"",
#>     "INPUT": "world_map.gpkg|layername=states_provinces",
#>     "OUTPUT": "ogr:dbname='output_states.gpkg' table=\"output\" (geom)"
#>   }
#> }
#> 
#> Running \
#>   /home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process \
#>   --json run 'native:aggregate' -
#> qt5ct: using qt5ct plugin
#> <Result of `qgis_run_algorithm("native:aggregate", ...)`>
#> List of 1
#>  $ OUTPUT: 'qgis_outputVector' chr "output_states.gpkg|layername=output"

st_as_sf(result)  # does not yet support the QGIS syntax of referring a layer
#> Error: Cannot open "output_states.gpkg|layername=output"; The file doesn't seem to exist.

result_sf <- st_read("output_states.gpkg", "output")
#> Reading layer `output' from data source 
#>   `/tmp/RtmpRLcXfg/reprex-2bf811b20af2e-smoky-moose/output_states.gpkg' 
#>   using driver `GPKG'
#> Simple feature collection with 251 features and 2 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: -179.9 ymin: -89.9 xmax: 179.9 ymax: 83.6341
#> Geodetic CRS:  WGS 84
plot(result_sf["admin"])

### Providing algorithm arguments as function arguments
####################################################################

(result2 <-
  qgis_run_algorithm(
    "native:aggregate",
    AGGREGATES = list(
      list(
        aggregate = "first_value",
        delimiter = ",",
        input = '"admin"',
        length = 36,
        name = "admin",
        precision = 0,
        type = 10
      ),
      list(
        aggregate = "concatenate",
        delimiter = ",",
        input = '"name"',
        length = 3000,
        name = "name",
        precision = 0,
        type = 10
      )
    ),
    INPUT = "world_map.gpkg|layername=states_provinces",
    GROUP_BY = "admin"
  ))
#> Using `OUTPUT = qgis_tmp_vector()`
#> JSON input ----
#> {
#>   "inputs": {
#>     "INPUT": "world_map.gpkg|layername=states_provinces",
#>     "GROUP_BY": "admin",
#>     "AGGREGATES": [
#>       {
#>         "aggregate": "first_value",
#>         "delimiter": ",",
#>         "input": "\"admin\"",
#>         "length": 36,
#>         "name": "admin",
#>         "precision": 0,
#>         "type": 10
#>       },
#>       {
#>         "aggregate": "concatenate",
#>         "delimiter": ",",
#>         "input": "\"name\"",
#>         "length": 3000,
#>         "name": "name",
#>         "precision": 0,
#>         "type": 10
#>       }
#>     ],
#>     "OUTPUT": "/tmp/RtmpyZZoXo/file2c0a532c7889f/file2c0a578781be6.gpkg"
#>   }
#> }
#> 
#> Running \
#>   /home/floris/git_repositories2/QGIS/build-master/output/bin/qgis_process \
#>   --json run 'native:aggregate' -
#> qt5ct: using qt5ct plugin
#> <Result of `qgis_run_algorithm("native:aggregate", ...)`>
#> List of 1
#>  $ OUTPUT: 'qgis_outputVector' chr "/tmp/RtmpyZZoXo/file2c0a532c7889f/file2c0a578781be6.gpkg"

st_as_sf(result2)
#> Simple feature collection with 251 features and 2 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: -179.9 ymin: -89.9 xmax: 179.9 ymax: 83.6341
#> Geodetic CRS:  WGS 84
#> # A tibble: 251 × 3
#>    admin       name                                                         geom
#>    <chr>       <chr>                                          <MULTIPOLYGON [°]>
#>  1 Norway      Troms,Nordland,Nord-Trøn… (((3.457286 -54.39007, 3.486664 -54.40…
#>  2 Sweden      Västerbotten,Jämtland,Vä… (((14.23471 55.8627, 14.20851 55.82128…
#>  3 Germany     Schleswig-Holstein,Niede… (((6.499686 49.71221, 6.485526 49.7337…
#>  4 Netherlands Groningen,Zeeland,Noord-… (((-62.99714 17.52562, -62.94506 17.48…
#>  5 Russia      Murmansk,Sakhalin,Nenets… (((33.52516 44.62885, 33.51482 44.6491…
#>  6 Cambodia    Kaôh Kong,Môndól Kiri,Pr… (((103.7434 10.48086, 103.7639 10.4945…
#>  7 Croatia     Zadarska,Medimurska,Vara… (((15.19679 43.88638, 15.19451 43.8726…
#>  8 Myanmar     Tanintharyi,Rakhine,Mon,… (((93.39479 14.07636, 93.37485 14.0646…
#>  9 Vietnam     Quảng Ninh,Gia Lai,Long … (((105.3151 8.812237, 105.2588 8.76046…
#> 10 Canada      British Columbia,Nunavut… (((-127.2406 50.02171, -127.2723 49.97…
#> # … with 241 more rows

^{Created on 2022-01-17 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Linux Mint 20
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language nl_BE:nl
#>  collate  nl_BE.UTF-8
#>  ctype    nl_BE.UTF-8
#>  tz       Europe/Brussels
#>  date     2022-01-17
#>  pandoc   2.14.0.3 @ /usr/lib/rstudio/bin/pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version     date (UTC) lib source
#>  assertthat    0.2.1       2019-03-21 [1] CRAN (R 4.1.0)
#>  class         7.3-20      2022-01-13 [4] CRAN (R 4.1.2)
#>  classInt      0.4-3       2020-04-07 [1] CRAN (R 4.1.0)
#>  cli           3.1.0       2021-10-27 [1] CRAN (R 4.1.2)
#>  crayon        1.4.2       2021-10-29 [1] CRAN (R 4.1.2)
#>  curl          4.3.2       2021-06-23 [1] CRAN (R 4.1.1)
#>  DBI           1.1.2       2021-12-20 [1] CRAN (R 4.1.2)
#>  digest        0.6.29      2021-12-01 [1] CRAN (R 4.1.2)
#>  dplyr         1.0.7       2021-06-18 [1] CRAN (R 4.1.1)
#>  e1071         1.7-9       2021-09-16 [1] CRAN (R 4.1.1)
#>  ellipsis      0.3.2       2021-04-29 [1] CRAN (R 4.1.0)
#>  evaluate      0.14        2019-05-28 [1] CRAN (R 4.1.0)
#>  fansi         1.0.2       2022-01-14 [1] CRAN (R 4.1.2)
#>  fastmap       1.1.0       2021-01-25 [1] CRAN (R 4.1.0)
#>  fs            1.5.2       2021-12-08 [1] CRAN (R 4.1.2)
#>  generics      0.1.1       2021-10-25 [1] CRAN (R 4.1.2)
#>  glue          1.6.0       2021-12-17 [1] CRAN (R 4.1.2)
#>  highr         0.9         2021-04-16 [1] CRAN (R 4.1.0)
#>  htmltools     0.5.2       2021-08-25 [1] CRAN (R 4.1.1)
#>  httr          1.4.2       2020-07-20 [1] CRAN (R 4.1.0)
#>  jsonlite      1.7.3       2022-01-17 [1] CRAN (R 4.1.2)
#>  KernSmooth    2.23-20     2021-05-03 [4] CRAN (R 4.1.0)
#>  knitr         1.37        2021-12-16 [1] CRAN (R 4.1.2)
#>  lifecycle     1.0.1       2021-09-24 [1] CRAN (R 4.1.1)
#>  magrittr      2.0.1       2020-11-17 [1] CRAN (R 4.1.0)
#>  mime          0.12        2021-09-28 [1] CRAN (R 4.1.1)
#>  pillar        1.6.4       2021-10-18 [1] CRAN (R 4.1.2)
#>  pkgconfig     2.0.3       2019-09-22 [1] CRAN (R 4.1.0)
#>  processx      3.5.2       2021-04-30 [1] CRAN (R 4.1.0)
#>  proxy         0.4-26      2021-06-07 [1] CRAN (R 4.1.0)
#>  ps            1.6.0       2021-02-28 [1] CRAN (R 4.1.0)
#>  purrr         0.3.4       2020-04-17 [1] CRAN (R 4.1.0)
#>  qgisprocess * 0.0.0.9000  2022-01-17 [1] Github (paleolimbot/qgisprocess@3357d16)
#>  R6            2.5.1       2021-08-19 [1] CRAN (R 4.1.1)
#>  rappdirs      0.3.3       2021-01-31 [1] CRAN (R 4.1.0)
#>  Rcpp          1.0.8       2022-01-13 [1] CRAN (R 4.1.2)
#>  reprex        2.0.1       2021-08-05 [1] CRAN (R 4.1.1)
#>  rlang         0.99.0.9009 2021-11-18 [1] local
#>  rmarkdown     2.11        2021-09-14 [1] CRAN (R 4.1.1)
#>  rstudioapi    0.13        2020-11-12 [1] CRAN (R 4.1.0)
#>  sessioninfo   1.2.2       2021-12-06 [1] CRAN (R 4.1.2)
#>  sf          * 1.0-5       2021-12-17 [1] CRAN (R 4.1.2)
#>  stringi       1.7.6       2021-11-29 [1] CRAN (R 4.1.2)
#>  stringr       1.4.0       2019-02-10 [1] CRAN (R 4.1.0)
#>  tibble        3.1.6       2021-11-07 [1] CRAN (R 4.1.2)
#>  tidyselect    1.1.1       2021-04-30 [1] CRAN (R 4.1.0)
#>  units         0.7-2       2021-06-08 [1] CRAN (R 4.1.1)
#>  utf8          1.2.2       2021-07-24 [1] CRAN (R 4.1.1)
#>  vctrs         0.3.8       2021-04-29 [1] CRAN (R 4.1.0)
#>  withr         2.4.3       2021-11-30 [1] CRAN (R 4.1.2)
#>  xfun          0.29        2021-12-14 [1] CRAN (R 4.1.2)
#>  xml2          1.3.3       2021-11-30 [1] CRAN (R 4.1.2)
#>  yaml          2.2.1       2020-02-01 [1] CRAN (R 4.1.0)
#> 
#>  [1] /home/floris/lib/R/library
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

nyalldawson · 2022-01-17T22:56:10Z

@paleolimbot

I'm back on board now, and trying to pick up the latest status with this work. Where are we sitting at the moment?

paleolimbot · 2022-01-18T01:14:26Z

We're sitting at 'awesome'! The JSON input is working beautifully and the JSON output contains some nice information that we can use to make better guesses about how to serialize R objects.

I'm pretty sure the PROJ thing is me setting environment variables wrong, although it is odd that this failed with --json and not otherwise. I wonder if it would have failed either way if I tried to do anything that required the PROJ database (I'm not sure I do in the tests).

@florisvdh thank you for all the testing! I'll make sure to fix the st_as_sf() issue in this PR once I get the Ubuntu check working properly.

nyalldawson · 2022-01-18T01:17:23Z

Great, thanks for the summary @paleolimbot !!

Just let me know if there's anything I can assist with here.

(BTW... looking at this awesomeness makes me start to wonder about a potential "render layer/map using qgis" type tool... !!)

paleolimbot · 2022-01-18T01:23:06Z

The last time I tried that it segfaulted! (see #68)

paleolimbot added 19 commits December 26, 2021 20:30

revert workaround for processx and use latest version

1e2ab11

use json output

3376b09

na not null for string

be5821a

use jsonlite

55b1fb3

assume utf-8 output, even on Windows

1e1aef2

see if --json is causing the failures on ubuntu

c923cef

only use utf-8 encoding when using json output, don't use json output…

f14f963

… on linux

use JSON input!

eff8012

use qgis input

6575643

make json input/output an option and print it

81a2046

add workaround for PROJ library as an option

af3e45f

use JSON output for qgis_algorithms()

5012ff4

cache algorithm and path info

3d1dfa5

better cache communication/organization

e9f926d

run document, use JSON output instead of parsing help

86c7127

fix ordering of parameters

ce85add

cache help output

6938256

allow raw JSON as input

bd228f3

fix function test

8f421b4

florisvdh reviewed Jan 14, 2022

View reviewed changes

R/qgis-run-algorithm.R Outdated Show resolved Hide resolved

florisvdh mentioned this pull request Jan 14, 2022

st_as_sf() method for qgis_result may miss the OUTPUT element #76

Closed

paleolimbot and others added 6 commits January 14, 2022 21:43

Update R/qgis-run-algorithm.R

cd8c53b

Co-authored-by: Floris Vanderhaeghe <[email protected]>

add qgis_argument for list()

e5252f8

turn on JSON output even for linux

bb1a10a

maybe output PROJ info

1d79b64

try again to get version output on github runner

49a2cde

maybe more info

f696aca

check that json output will work before setting default to TRUE

3357d16

paleolimbot added 2 commits January 17, 2022 15:41

redocument

9645f23

maybe some more output to help with --json

6cf8a59

paleolimbot added 5 commits January 17, 2022 20:16

fix tests for updated terra

636c713

try again to get some diagnostic output for Ubuntu runner

e01d722

more tries at diagnostic output

92f51dc

fix terra test to pass with multiple terra versions

0f0a145

remove diagnostics

8180e79

run pkgdown on ubuntu-20.04

0ebdec4

paleolimbot merged commit 4a79540 into master Jan 18, 2022

paleolimbot deleted the use-json branch January 18, 2022 11:54

florisvdh mentioned this pull request Jan 31, 2022

Support needed in st_as_sf() for '<dsn>|layername=<name>' output as obtained from qgis_process #84

Closed

florisvdh mentioned this pull request Jun 13, 2022

Is there a workaround for the qgis type 'aggregates'? #56

Closed

florisvdh mentioned this pull request Aug 12, 2022

Ability to use QGIS expressions to create new attributes of an sf class object #104

Closed

florisvdh mentioned this pull request Mar 29, 2023

Revising the namespace #133

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use JSON output and input where available #74

Use JSON output and input where available #74

paleolimbot commented Dec 27, 2021

paleolimbot commented Jan 10, 2022

florisvdh commented Jan 14, 2022

florisvdh commented Jan 14, 2022

florisvdh commented Jan 17, 2022

nyalldawson commented Jan 17, 2022

paleolimbot commented Jan 18, 2022

nyalldawson commented Jan 18, 2022

paleolimbot commented Jan 18, 2022

Use JSON output and input where available #74

Use JSON output and input where available #74

Conversation

paleolimbot commented Dec 27, 2021

paleolimbot commented Jan 10, 2022

florisvdh commented Jan 14, 2022

florisvdh commented Jan 14, 2022

florisvdh commented Jan 17, 2022

nyalldawson commented Jan 17, 2022

paleolimbot commented Jan 18, 2022

nyalldawson commented Jan 18, 2022

paleolimbot commented Jan 18, 2022