support for non latin1/utf8 encodings in geoserver

[drh-stanford]

example: ty249zp2774 has a chinese encoding but geoserver shows inspection results as utf8/latin1

see

gis-robot-suite/robots/gisDelivery/load-vector.rb

Lines 82 to 87 in 9b6179d

	# first try decoding with UTF-8 and if that fails use LATIN1
	begin
	run_shp2pgsql(projection, 'UTF-8', shpfn, schema, druid, sqlfn, errfn)
	rescue RuntimeError => e
	run_shp2pgsql(projection, 'LATIN1', shpfn, schema, druid, sqlfn, errfn)
	end

[drh-stanford]

For example:

https://earthworks.stanford.edu/catalog/stanford-ty249zp2774

inspection shows wrong character encoding...

[drh-stanford]

Another example is yq395kh3847

[drh-stanford]

some shapefiles come with .cpg files that will have the character encoding. the data appear corrupted in the PostGIS database so the shp2pgsql is where we need to set the encoding. We currently try UTF8 and then LATIN1.

[drh-stanford]

FYI - The yq395kh3847 example is encoded in the BIG5 encoding. If you load it into QGIS with BIG5, then the characters show up.

[drh-stanford]

using shp2pgsql -W BIG5 fixes this particular layer -- the problem is that we don't know the character encoding, in general. the .dbf file is just binary data and you have to try different encodings until it works (visual inspection).