SQalchemy error during installation

[m-sinclair]

<class 'sqlalchemy.exc.InvalidRequestError'>: Table 'datarequests' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

[aitormagan]

Please, can you specify the versions (CKAN and PostgreSQL) you are using. I have installed this extension in several instances and it's the first time I see this error.

[m-sinclair]

CKAN 2.41 and psql (PostgreSQL) 9.1.19 - I have the pages extension installed as well and I think there may be a conflict.

[aitormagan]

Hi again,

Sorry for my delay, but I did not found time to take care of your issue until now. I have created a completely new CKAN instance and installed ckanext-datarequests and ckanext-pages and the instance is working smoothly so I am afraid I cannot reproduce the error.

Here you have the versions I have installed:

• Ubuntu 12.04
• CKAN 2.4.1
• PostgreSQL 9.1.19
• SQLAlchemy 0.9.6

Are you using the same versions?

You can check the instance I've created at: http://130.206.126.145/

[aitormagan]

Hi,

Since I have not heard from you in a week, I will close this issue. Please, if you are still not able to install this extension, do not hesitate to reopen this issue again.

BR
Aitor

[LukeTully]

@aitormagan I'd like to reopen this issue with the following information and reproducibility steps.

CKAN 2.6.1
psql (9.6.2, server 9.3.16)

The reason you may have been seeing a smooth install is that datarequests works fine on install install and server startup, but then produces this exact error once navigated to another page or after the server has been restarted.

[aitormagan]

Hi @LukeTully,

I will reopen the issue without any inconvenience. :)

In regard to your issue, I can tell you that the extension is properly working in many CKAN instances. For example, the instance is deployed at https://data.lab.fiware.org/ and we don't have the problem you have mention. As you can see, you can navigate though the pages and no exception is risen. Additionally, this server has been restarted so many times and this error did not appear.

The extension has not been tested with the CKAN version you have posted. Maybe the extension si not working properly with that version. Have you tried to install the extension in a older CKAN version? I can check compatibility with that version, but I am afraid it will take me some time to do so.

[LukeTully]

Thanks for the quick response and for re-opening the issue. I'll see if I can isolate the issue by installing it with a known to be working version. Would you have a version that is definitely known to be working? Are there any other commonly reported conflicts I might have missed?

It seems like quite a few extensions have undocumented incompatibilities with 2.6.* and so I've been going through the closed issues, commit logs, and forks, to see if they can be of any help before opening new issues myself.

[aitormagan]

@LukeTully, the instance I mentioned before is using a CKAN 2.6 with Postgres 9.1. Anyway, I will try to install the versions you mentioned before in order to reproduce the error and try to fix it. It will take me some time, but I will try to do it as soon as possible. If you find a solution to the problem you can submit a Pull Request too. We will be glad to accept it!! :)

[LukeTully]

@aitormagan Thanks for the effort. I'll give it a go once I figure out this archiver issue I'm also running into. I've started looking through the code, starting with db.py, but it's a little still a little unclear to me so far and the errors are cryptic. I'll paste the whole relevant portion of the error log below as a text file. Maybe you can point me in the right direction.

ckan-3-10-2017.txt

[aitormagan]

Hi @LukeTully,

I am afraid I am not able to reproduce the issue. I have started an EC2 instance with Ubuntu 14.04 and installed PostgreSQL 9.3 and CKAN 2.6.1 and the datarequests extension is working smoothly. I have installed the extension using the pip command and tried to restart Apache2 some times but it is still working :(. Maybe the issue is related with the operating system you are using.

Here you have a complete list of all the versions I have used:

• Ubuntu Server 14.04
• CKAN 2.6.1
• Postgres Server 9.3.16
• datarequests 0.3.2

pip list: (ensure you are using the same SQLAlchemy version)

appdirs (1.4.0)
Babel (2.3.4)
Beaker (1.8.0)
bleach (1.4.3)
ckan (2.6.1, /usr/lib/ckan/default/src/ckan)
ckanext-datarequests (0.3.2)
decorator (4.0.6)
fanstatic (0.12)
Flask (0.10.1)
FormEncode (1.3.0)
funcsigs (1.0.2)
html5lib (0.9999999)
itsdangerous (0.24)
Jinja2 (2.8)
Mako (1.0.4)
Markdown (2.6.6)
MarkupSafe (0.23)
nose (1.3.7)
ofs (0.4.2)
ordereddict (1.1)
packaging (16.8)
Pairtree (0.7.1-T)
passlib (1.6.5)
Paste (1.7.5.1)
PasteDeploy (1.5.2)
PasteScript (2.0.2)
pbr (0.11.1)
pip (9.0.1)
psycopg2 (2.4.5)
Pygments (2.1.3)
Pylons (0.9.7)
pyparsing (2.1.10)
pysolr (3.5.0)
python-dateutil (1.5)
pytz (2016.4)
pyutilib.component.core (4.6.4)
repoze.lru (0.6)
repoze.who (2.3)
repoze.who-friendlyform (1.0.8)
requests (2.10.0)
Routes (1.13)
setuptools (34.2.0)
simplejson (3.8.2)
six (1.10.0)
SQLAlchemy (0.9.6)
sqlalchemy-migrate (0.10.0)
sqlparse (0.1.19)
Tempita (0.5.2)
tzlocal (1.2.2)
unicodecsv (0.14.1)
vdm (0.13)
WebError (0.13.1)
WebHelpers (1.3)
WebOb (1.0.8)
WebTest (1.4.3)
Werkzeug (0.11.10)
wheel (0.29.0)
zope.interface (4.2.0)

[LukeTully]

Thanks @aitormagan for looking into that. That gives me a lot of information to work with and I'll document the solution here when I resolve the issue. I have had some success in isolating problems with other extensions so far. ckanext-scheming seems to be conflicting with other extensions and not reporting errors. Not sure if that's what the problem is, but making progress.

Any chance you have some previous experience with obscure problems happening in other extensions that manifest in weird ways?

[aitormagan]

I would be fantastic if you find a solution to the issue so other users can take advantage of it. It is not the first time I see some incompatibilities between CKAN extensions so maybe that is the problem :). Let me know with your progress. P.S.: thank you so much for investigating the issue and using the extension!

[LukeTully]

@aitormagan So I have a small update on this issue. No solution yet, but I have isolated the problem, at least in my situation, to a conflict between datarequests and featuredviews.

[aitormagan]

@LukeTully, so if you disable "featuredviews" does the "datarequests" extension work again? I have checked the "featuredviews" extension and I do not understand the problem since this extension does not create any table called "datarequests". I will try to install that extension on my instance so I can see if I can reproduce the problem :)

[aitormagan]

@LukeTully Thanks you for the reporting!!! I have installed the featuredviews extension and datarequests stopped working with the same error you have reported. I have discovered the error is caused by the way the datarequests extension handles the connection with the database. I have to look into that with more time, but I have found a workaround to your problem so you can run both extensions. Here you have the steps:

0. Uninstall the ckanext-datarequests extension (pip uninstall ckanext-datarequests)
1. Download the ckanext-datarequests extension manually (git clone ...)
2. Replace the db.py file with the content of this gist: https://gist.github.com/aitormagan/32615387707c62350f2c2d1b543e6f11
3. Install the datarequests extension again. To do so:
   3.1. Activate the ckan virtenv
   3.2. Run python setup.py install

I hope this helps! :)

[LukeTully]

Hey @aitormagan. Sorry for the late reply. I could have answered with a simple yes to that first comment. Thanks for the patch, I appreciate the effort to look into that and know it can be a lot of investment handling issues and support. I'll try applying it. Hopefully with some more exposure to both python and CKAN I'll be able to more thoroughly diagnose things like this and issue a PR instead of just an issue.