Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a workaround for the qgis type 'aggregates'? #56

Closed
florisvdh opened this issue Aug 4, 2021 · 8 comments
Closed

Is there a workaround for the qgis type 'aggregates'? #56

florisvdh opened this issue Aug 4, 2021 · 8 comments

Comments

@florisvdh
Copy link
Member

I'm not so experienced with QGIS and qgisprocess, so the below problem may be a demonstration of just that...

According to #13 work is still needed to support the 'aggregates' argument type; it mentions rlang::quo() so I wondered whether something can be done in below code to make it work already. Purpose below is the same as doing sf::st_union(object), I'm only interested in the single aggregated object, not in attributes. It works when done interactively using the QGIS GUI.

> sg_union_qgis <- 
+     qgis_run_algorithm("native:aggregate",
+                        INPUT = file.path(datapath, 
+                                          "sg_repairedgeom.gpkg"))
Argument `GROUP_BY` is unspecified (using QGIS default value).
Argument `AGGREGATES` is unspecified (using QGIS default value).
Using `OUTPUT = qgis_tmp_vector()`
Running qgis_process run 'native:aggregate' \
  '--INPUT=/pathprefix/sg_repairedgeom.gpkg' \
  '--OUTPUT=/tmp/RtmpcrzaRP/filebef483308cbd0/filebef481edb5fa4.gpkg'
qt5ct: using qt5ct plugin
Problem with SAGA installation: SAGA was not found or is not correctly installed

----------------
Inputs
----------------

INPUT:	/pathprefix/sg_repairedgeom.gpkg
OUTPUT:	/tmp/RtmpcrzaRP/filebef483308cbd0/filebef481edb5fa4.gpkg

ERROR: The following mandatory parameters were not specified

	AGGREGATES:	Aggregates
> sg_union_qgis <- 
+     qgis_run_algorithm("native:aggregate",
+                        INPUT = file.path(datapath, 
+                                          "sg_repairedgeom.gpkg"),
+                        AGGREGATES = NULL)
Argument `GROUP_BY` is unspecified (using QGIS default value).
Using `OUTPUT = qgis_tmp_vector()`
Running qgis_process run 'native:aggregate' \
  '--INPUT=/pathprefix/sg_repairedgeom.gpkg' \
  '--OUTPUT=/tmp/RtmpcrzaRP/filebef483308cbd0/filebef485bf98088.gpkg'
qt5ct: using qt5ct plugin
Problem with SAGA installation: SAGA was not found or is not correctly installed

----------------
Inputs
----------------

INPUT:	/pathprefix/sg_repairedgeom.gpkg
OUTPUT:	/tmp/RtmpcrzaRP/filebef483308cbd0/filebef485bf98088.gpkg

ERROR: The following mandatory parameters were not specified

	AGGREGATES:	Aggregates
> sg_union_qgis <- 
+     qgis_run_algorithm("native:aggregate",
+                        INPUT = file.path(datapath, 
+                                          "sg_repairedgeom.gpkg"),
+                        AGGREGATES = list())
Argument `GROUP_BY` is unspecified (using QGIS default value).
Using `OUTPUT = qgis_tmp_vector()`
Error: Error : Don't know how to convert object of type 'list' to QGIS type 'aggregates'

Processing parameters from the QGIS GUI log:

Algorithm 'Aggregate' starting…
Input parameters:
{ 'AGGREGATES' : [], 'GROUP_BY' : 'NULL', 'INPUT' : '/pathprefix/sg_repairedgeom.gpkg|layername=sg_repairedgeom', 'OUTPUT' : 'TEMPORARY_OUTPUT' }
Session info
─ Session info ─────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 4.1.0 (2021-05-18)
 os       Linux Mint 20               
 system   x86_64, linux-gnu           
 ui       RStudio                     
 language nl_BE:nl                    
 collate  nl_BE.UTF-8                 
 ctype    nl_BE.UTF-8                 
 tz       Europe/Brussels             
 date     2021-08-04                  

─ Packages ─────────────────────────────────────────────────────────────────────────────
 package     * version    date       lib source                                  
 assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.1.0)                          
 base64enc     0.1-3      2015-07-28 [1] CRAN (R 4.1.0)                          
 bookdown      0.22       2021-04-22 [1] CRAN (R 4.1.0)                          
 cachem        1.0.5      2021-05-15 [1] CRAN (R 4.1.0)                          
 callr         3.7.0      2021-04-20 [1] CRAN (R 4.1.0)                          
 class         7.3-19     2021-05-03 [4] CRAN (R 4.1.0)                          
 classInt      0.4-3      2020-04-07 [1] CRAN (R 4.1.0)                          
 cli           2.5.0      2021-04-26 [1] CRAN (R 4.1.0)                          
 codetools     0.2-18     2020-11-04 [4] CRAN (R 4.1.0)                          
 colorspace    2.0-1      2021-05-04 [1] CRAN (R 4.1.0)                          
 crayon        1.4.1      2021-02-08 [1] CRAN (R 4.1.0)                          
 crosstalk     1.1.1      2021-01-12 [1] CRAN (R 4.1.0)                          
 DBI           1.1.1      2021-01-15 [1] CRAN (R 4.1.0)                          
 desc          1.3.0      2021-03-05 [1] CRAN (R 4.1.0)                          
 devtools      2.4.2      2021-06-07 [1] CRAN (R 4.1.0)                          
 digest        0.6.27     2020-10-24 [1] CRAN (R 4.1.0)                          
 dplyr       * 1.0.6      2021-05-05 [1] CRAN (R 4.1.0)                          
 e1071         1.7-7      2021-05-23 [1] CRAN (R 4.1.0)                          
 ellipsis      0.3.2      2021-04-29 [1] CRAN (R 4.1.0)                          
 evaluate      0.14       2019-05-28 [1] CRAN (R 4.1.0)                          
 fansi         0.5.0      2021-05-25 [1] CRAN (R 4.1.0)                          
 fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.1.0)                          
 forcats       0.5.1      2021-01-27 [1] CRAN (R 4.1.0)                          
 fs            1.5.0      2020-07-31 [1] CRAN (R 4.1.0)                          
 generics      0.1.0      2020-10-31 [1] CRAN (R 4.1.0)                          
 ggplot2     * 3.3.3      2020-12-30 [1] CRAN (R 4.1.0)                          
 git2r         0.28.0     2021-01-10 [1] CRAN (R 4.1.0)                          
 git2rdata     0.3.1      2021-01-21 [1] CRAN (R 4.1.0)                          
 glue          1.4.2      2020-08-27 [1] CRAN (R 4.1.0)                          
 googledrive * 1.0.1      2020-05-05 [1] CRAN (R 4.1.0)                          
 gtable        0.3.0      2019-03-25 [1] CRAN (R 4.1.0)                          
 htmltools     0.5.1.1    2021-01-22 [1] CRAN (R 4.1.0)                          
 htmlwidgets   1.5.3      2020-12-10 [1] CRAN (R 4.1.0)                          
 KernSmooth    2.23-20    2021-05-03 [4] CRAN (R 4.1.0)                          
 knitr       * 1.33       2021-04-24 [1] CRAN (R 4.1.0)                          
 lattice       0.20-44    2021-05-02 [4] CRAN (R 4.1.0)                          
 leafem        0.1.6      2021-05-24 [1] CRAN (R 4.1.0)                          
 leaflet       2.0.4.1    2021-01-07 [1] CRAN (R 4.1.0)                          
 lifecycle     1.0.0      2021-02-15 [1] CRAN (R 4.1.0)                          
 link2GI       0.4-5      2020-12-07 [1] CRAN (R 4.1.0)                          
 lwgeom        0.2-6      2021-04-02 [1] CRAN (R 4.1.0)                          
 magrittr      2.0.1      2020-11-17 [1] CRAN (R 4.1.0)                          
 mapview     * 2.10.0     2021-06-05 [1] CRAN (R 4.1.0)                          
 memoise       2.0.0      2021-01-26 [1] CRAN (R 4.1.0)                          
 munsell       0.5.0      2018-06-12 [1] CRAN (R 4.1.0)                          
 n2khab      * 0.5.0      2021-05-12 [1] local                                   
 pillar        1.6.1      2021-05-16 [1] CRAN (R 4.1.0)                          
 pkgbuild      1.2.0      2020-12-15 [1] CRAN (R 4.1.0)                          
 pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.1.0)                          
 pkgload       1.2.1      2021-04-06 [1] CRAN (R 4.1.0)                          
 plyr          1.8.6      2020-03-03 [1] CRAN (R 4.1.0)                          
 png           0.1-7      2013-12-03 [1] CRAN (R 4.1.0)                          
 prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.1.0)                          
 processx      3.5.2      2021-04-30 [1] CRAN (R 4.1.0)                          
 proxy         0.4-26     2021-06-07 [1] CRAN (R 4.1.0)                          
 ps            1.6.0      2021-02-28 [1] CRAN (R 4.1.0)                          
 purrr       * 0.3.4      2020-04-17 [1] CRAN (R 4.1.0)                          
 qgisprocess * 0.0.0.9000 2021-08-03 [1] Github (paleolimbot/qgisprocess@2b61d6f)
 R6            2.5.0      2020-10-28 [1] CRAN (R 4.1.0)                          
 raster        3.4-10     2021-05-03 [1] CRAN (R 4.1.0)                          
 Rcpp          1.0.6      2021-01-15 [1] CRAN (R 4.1.0)                          
 remotes       2.4.0      2021-06-02 [1] CRAN (R 4.1.0)                          
 rgrass7     * 0.2-5      2021-01-29 [1] CRAN (R 4.1.0)                          
 rlang         0.4.11     2021-04-30 [1] CRAN (R 4.1.0)                          
 rmarkdown     2.8        2021-05-07 [1] CRAN (R 4.1.0)                          
 roxygen2      7.1.1      2020-06-27 [1] CRAN (R 4.1.0)                          
 rprojroot   * 2.0.2      2020-11-15 [1] CRAN (R 4.1.0)                          
 rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.1.0)                          
 satellite     1.0.2      2019-12-09 [1] CRAN (R 4.1.0)                          
 scales        1.1.1      2020-05-11 [1] CRAN (R 4.1.0)                          
 sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.1.0)                          
 sf          * 0.9-8      2021-03-17 [1] CRAN (R 4.1.0)                          
 sp            1.4-5      2021-01-10 [1] CRAN (R 4.1.0)                          
 stringi       1.6.2      2021-05-17 [1] CRAN (R 4.1.0)                          
 stringr     * 1.4.0      2019-02-10 [1] CRAN (R 4.1.0)                          
 testthat      3.0.2      2021-02-14 [1] CRAN (R 4.1.0)                          
 tibble        3.1.2      2021-05-16 [1] CRAN (R 4.1.0)                          
 tidyr         1.1.3      2021-03-03 [1] CRAN (R 4.1.0)                          
 tidyselect    1.1.1      2021-04-30 [1] CRAN (R 4.1.0)                          
 units       * 0.7-1      2021-03-16 [1] CRAN (R 4.1.0)                          
 usethis       2.0.1      2021-02-10 [1] CRAN (R 4.1.0)                          
 utf8          1.2.1      2021-03-12 [1] CRAN (R 4.1.0)                          
 vctrs         0.3.8      2021-04-29 [1] CRAN (R 4.1.0)                          
 webshot       0.5.2      2019-11-22 [1] CRAN (R 4.1.0)                          
 withr         2.4.2      2021-04-18 [1] CRAN (R 4.1.0)                          
 xfun          0.23       2021-05-15 [1] CRAN (R 4.1.0)                          
 XML         * 3.99-0.6   2021-03-16 [1] CRAN (R 4.1.0)                          
 xml2          1.3.2      2020-04-23 [1] CRAN (R 4.1.0)                          
 yaml          2.2.1      2020-02-01 [1] CRAN (R 4.1.0)                          

[1] /home/floris/lib/R/library
[2] /usr/local/lib/R/site-library
[3] /usr/lib/R/site-library
[4] /usr/lib/R/library
@florisvdh florisvdh changed the title I there a workaround for the qgis type 'aggregates'? Is there a workaround for the qgis type 'aggregates'? Aug 4, 2021
@paleolimbot
Copy link
Collaborator

My guess is that it probably won't work until we use JSON to specify the algorithm. The aggregates argument is its own argument type ("aggregates") which I'm guessing means that it is a Python object. If you can figure out the proper JSON specification for the aggregates type it will help me implement it when I get there! (Probably in about two months).

A workaround might be to make a Workflow (model? pipeline? I forget the QGIS terminology) which you can run using qgis_run() (I think you can use a model file as the first argument to qgis_process).

@JanCaha
Copy link
Collaborator

JanCaha commented Aug 4, 2021

It might be possible. The AGGREGATES is just a field of dictionaries (in Python terms).

Each record looks like this and specifies one output column:
{'aggregate': 'sum','delimiter': ',','input': '\"fid\"','length': 0,'name': 'fid','precision': 0,'type': 4}

  • name is the name of the resulting data column
  • input is the name of the input data column
  • aggregate is the specification of the aggregating function - an enum
  • delimiter is used in aggregate function - i am not sure, but i guess it will be mostly ","
  • type specifies type of the resulting data colum - an enum
  • length and precision are used to specify resulting data column

@florisvdh
Copy link
Member Author

That matches the GUI:

Schermafdruk van 2021-08-04 15-49-06

I guess since qgis_run_algorithm() constructs the qgis_process command with arguments and options, a working shell syntaxis would need to be known. I'm still confused what's needed in this case (it may need to be asked at QGIS), the simplest case being no GROUP_BY and no attributes requested. From the shell:

shell experiments
$ qgis_process help native:aggregate # not all output shown

----------------
Arguments
----------------

INPUT: Input layer
	Argument type:	source
	Acceptable values:
		- Path to a vector layer
GROUP_BY: Group by expression (NULL to group all features)
	Argument type:	expression
	Acceptable values:
		- A valid QGIS expression string, e.g "road_name" = 'MAIN RD'
AGGREGATES: Aggregates
	Argument type:	aggregates
OUTPUT: Aggregated
	Argument type:	sink
	Acceptable values:
		- Path for new vector layer
$ qgis_process run native:aggregate -- INPUT=sg_repairedgeom.gpkg AGGREGATES={} OUTPUT=try.gpkg
qt5ct: using qt5ct plugin
	- 'VirtualXPath'	[XML Path Language - XPath]
	- 'VirtualXPath'	[XML Path Language - XPath]
Problem with SAGA installation: SAGA was not found or is not correctly installed

----------------
Inputs
----------------

AGGREGATES:	{}
INPUT:	sg_repairedgeom.gpkg
OUTPUT:	try.gpkg

	- 'VirtualXPath'	[XML Path Language - XPath]
ERROR:	An error was encountered while checking parameter values
	Incorrect parameter value for AGGREGATES
$ qgis_process run native:aggregate -- INPUT=sg_repairedgeom.gpkg AGGREGATES={'aggregate':'sum','delimiter':',','input':'\"fid\"','length':0,'name':'fid','precision':0,'type':4} OUTPUT=try.gpkg
qt5ct: using qt5ct plugin
	- 'VirtualXPath'	[XML Path Language - XPath]
	- 'VirtualXPath'	[XML Path Language - XPath]
Problem with SAGA installation: SAGA was not found or is not correctly installed

----------------
Inputs
----------------

AGGREGATES:	
INPUT:	sg_repairedgeom.gpkg
OUTPUT:	try.gpkg

	- 'VirtualXPath'	[XML Path Language - XPath]
ERROR:	An error was encountered while checking parameter values
	Incorrect parameter value for AGGREGATES
$ qgis_process run native:aggregate -- INPUT=sg_repairedgeom.gpkg AGGREGATES={aggregate:sum,delimiter:',',input:fid,length:0,name:fid,precision:0,type:4} OUTPUT=try.gpkg
qt5ct: using qt5ct plugin
	- 'VirtualXPath'	[XML Path Language - XPath]
	- 'VirtualXPath'	[XML Path Language - XPath]
Problem with SAGA installation: SAGA was not found or is not correctly installed

----------------
Inputs
----------------

AGGREGATES:	
INPUT:	sg_repairedgeom.gpkg
OUTPUT:	try.gpkg

	- 'VirtualXPath'	[XML Path Language - XPath]
ERROR:	An error was encountered while checking parameter values
	Incorrect parameter value for AGGREGATES
$ qgis_process run native:aggregate -- INPUT=sg_repairedgeom.gpkg AGGREGATES="{'aggregate': 'sum','delimiter': ',','input': '\"fid\"','length': 0,'name': 'fid','precision': 0,'type': 4}" OUTPUT=try.gpkg
qt5ct: using qt5ct plugin
	- 'VirtualXPath'	[XML Path Language - XPath]
	- 'VirtualXPath'	[XML Path Language - XPath]
Problem with SAGA installation: SAGA was not found or is not correctly installed

----------------
Inputs
----------------

AGGREGATES:	{'aggregate': 'sum','delimiter': ',','input': '"fid"','length': 0,'name': 'fid','precision': 0,'type': 4}
INPUT:	sg_repairedgeom.gpkg
OUTPUT:	try.gpkg

	- 'VirtualXPath'	[XML Path Language - XPath]
ERROR:	An error was encountered while checking parameter values
	Incorrect parameter value for AGGREGATES

@florisvdh
Copy link
Member Author

@paleolimbot "the JSON interface for the command-line runner" (from #40 (comment)): is it described somewhere? Didn't see that in https://docs.qgis.org/3.16/en/docs/user_manual/processing/standalone.html. The --json option of qgis_process is said to apply to the output (not the input):

Options:
	--json		Output results as JSON objects

@paleolimbot
Copy link
Collaborator

You're right...this applies only to output. I was remembering qgis/QGIS#39324 (comment) but it looks like that was never implemented (unless I'm missing something from my reading of the source code).

@nyalldawson Did I interpret that correctly (that you can't use JSON via stdin)? JSON via stdin would also help with passing WKT2 as a command-line argument in Windows because of some complexities with shell parsing when running .bat files as an external process.

@florisvdh
Copy link
Member Author

@paleolimbot thanks for the reference. Also, it appears that, from qgis_process help native:aggregate, no 'Acceptable values' are shown for AGGREGATES, while it is generally the case for other arguments. Another, related aspect is that there seems no distinction between required and optional arguments, and what is the default value of the latter.

nyalldawson added a commit to nyalldawson/QGIS that referenced this issue Dec 15, 2021
specified as a JSON object passed via stdin to qgis_process

This provides a mechanism to support complex input parameters
for algorithms, and a way for qgis_process to gain support
for parameter types which are themselves specified as a dictionary
type object.

To indicate that parameters will be specified via stdin then
the qgis_process command must follow the format

    qgis_process run algid -

(with a trailing - in place of the usual arguments list).

The JSON object must contain an "inputs" key, which is a map
of the input parameter values.

E.g.

    echo "{"inputs": {\"INPUT\": \"my_shape.shp\", DISTANCE: 5}}" | qgis_process run native:buffer -

Specifying input parameters via stdin implies automatically
the --json output format for results.

One big motivation behind this enhancement is to provide a way for
the qgisprocess R libraries to support parameter types such as
aggregates.

Refs r-spatial/qgisprocess#56
Refs r-spatial/qgisprocess#44

Sponsored by the Research Institute for Nature and Forest, Flemish Govt
@nyalldawson
Copy link

@paleolimbot qgis/QGIS#46497 should provide a solution to this! Are you able to take a look and confirm that the enhancement work for you to help address this issue?

nyalldawson added a commit to nyalldawson/QGIS that referenced this issue Dec 15, 2021
specified as a JSON object passed via stdin to qgis_process

This provides a mechanism to support complex input parameters
for algorithms, and a way for qgis_process to gain support
for parameter types which are themselves specified as a dictionary
type object.

To indicate that parameters will be specified via stdin then
the qgis_process command must follow the format

    qgis_process run algid -

(with a trailing - in place of the usual arguments list).

The JSON object must contain an "inputs" key, which is a map
of the input parameter values.

E.g.

    echo "{"inputs": {\"INPUT\": \"my_shape.shp\", DISTANCE: 5}}" | qgis_process run native:buffer -

Specifying input parameters via stdin implies automatically
the --json output format for results.

One big motivation behind this enhancement is to provide a way for
the qgisprocess R libraries to support parameter types such as
aggregates.

Refs r-spatial/qgisprocess#56
Refs r-spatial/qgisprocess#44

Sponsored by the Research Institute for Nature and Forest, Flemish Govt
nyalldawson added a commit to nyalldawson/QGIS that referenced this issue Dec 16, 2021
specified as a JSON object passed via stdin to qgis_process

This provides a mechanism to support complex input parameters
for algorithms, and a way for qgis_process to gain support
for parameter types which are themselves specified as a dictionary
type object.

To indicate that parameters will be specified via stdin then
the qgis_process command must follow the format

    qgis_process run algid -

(with a trailing - in place of the usual arguments list).

The JSON object must contain an "inputs" key, which is a map
of the input parameter values.

E.g.

    echo "{"inputs": {\"INPUT\": \"my_shape.shp\", DISTANCE: 5}}" | qgis_process run native:buffer -

Specifying input parameters via stdin implies automatically
the --json output format for results.

One big motivation behind this enhancement is to provide a way for
the qgisprocess R libraries to support parameter types such as
aggregates.

Refs r-spatial/qgisprocess#56
Refs r-spatial/qgisprocess#44

Sponsored by the Research Institute for Nature and Forest, Flemish Govt
nyalldawson added a commit to qgis/QGIS that referenced this issue Dec 20, 2021
specified as a JSON object passed via stdin to qgis_process

This provides a mechanism to support complex input parameters
for algorithms, and a way for qgis_process to gain support
for parameter types which are themselves specified as a dictionary
type object.

To indicate that parameters will be specified via stdin then
the qgis_process command must follow the format

    qgis_process run algid -

(with a trailing - in place of the usual arguments list).

The JSON object must contain an "inputs" key, which is a map
of the input parameter values.

E.g.

    echo "{"inputs": {\"INPUT\": \"my_shape.shp\", DISTANCE: 5}}" | qgis_process run native:buffer -

Specifying input parameters via stdin implies automatically
the --json output format for results.

One big motivation behind this enhancement is to provide a way for
the qgisprocess R libraries to support parameter types such as
aggregates.

Refs r-spatial/qgisprocess#56
Refs r-spatial/qgisprocess#44

Sponsored by the Research Institute for Nature and Forest, Flemish Govt
@florisvdh
Copy link
Member Author

This issue was solved a whole time ago - solution demonstrated with reprex in #74 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants