-
Notifications
You must be signed in to change notification settings - Fork 1
Dev Meeting Notes
- Meeting times: Tuesday / Thursday 12:00 London time
- Meeting URL: http://bit.ly/ckandevmeeting (or if that doesn't work: https://plus.google.com/hangouts/_/event/cb1vmkss426lkqicj94a3amvd1c?authuser=0&hl=en-GB )
- Meeting pad: http://pad.okfn.org/p/ckan-meeting
Everyone is welcome, just add an item to the pad if you want something raised.
Chair: Vitor Notes: Nigel
- #1714 - dread to review
- #1792 - not blocked, but still WIP, vitor talk to adria
- #1701 - already assigned to joet. dateutil assumes mm/dd/yyyy if ambigous. It may bite someone in the future.
- #1858 - assigned to nigel
- #1857 - good for contribution
- #1856 - vitor to take a look
- #1852 - currently the defaultview is hard-coded and needs changing. vitor and adria working on it.
- #1849 - vitor to ping johnmartin/sam
General questions:
- Datapusher/ckan-service-provider issues/pull requests (nigelb) - agreed to discuss at next dev meeting
- full-text search on non-textual fields (vitor) - can't filter for time/year. Only text. vitor will talk to adria.
(no meeting, everyone off at ckancon. kindly and wardi chatted about what's been happening the past few months)
Chair joe Notes ian
- #1837 - adria
- #1836 - adria
- #1792 - depends on resource views, mess of branches to discuss in berlin
- #1830 - joe already assigned, not blocked review please (pretty please?)
- #1838 - vitor to work on, to talk in general questions
General questions:
- allow full text search on specific columns (#1838), related to #1792
- make sure datastore user is different than main ckan db user in docs
- (dread) Tag Extras https://github.com/ckan/ideas-and-roadmap/issues/68 - look at using json filter index
- (dread) DataPusher notification triggered on before_commit - suggest change to after commit
Chair adria Notes ian
- #1828 env variables for config settings, wip seanh
- #1792 wip resource views, to revisit when vitor is at meeting
- #1836 exception logging middleware, can we set inherit=True as a default to avoid future interface change breakage? add a prettier error message? adria to investigate
- #1732 tests written by submitter - wardi to review
- #1826 sphinx test - nigel
- #1591 ongoing translation discussions - adria
- #1830 multiple datastore values - assigned to joe
- #1835 docker-test-environment, wardi
- #1834 ?
- #1833 wardi assigned
- #1822 IPackageController docs incomplete, let's remove the deprecated methods, nigel assigned
- #1829 ongoing lessc discussion, lesspyc doesn't work for our less files; let's keep using the ppa
General questions:
* CKAN command? (Sean added this but can't make the dev meeting today)
Really needed to make running commands in the docker container easier.
Also needed so we can template both the dev/prod and test config files.
How to implement it?
(See Sean's thoughts at the bottom)
-
Any topic that the tech team wants to raise at CKANCon [ckon!] with the community? https://docs.google.com/a/okfn.org/document/d/12Oj5n1POukUW6Kl7oFRIAXYAAGivExNtOEylD7Dvo5g/edit Sean's thoughts:
Rewrite cli.py with docopt. Don't want to duplicate any of the UI code that defines the commands, sub-commands, options and docstrings. Delete all this and re-write it with docopt. But the code that actually carries out the commands - just keep it as-is but have it called by our new docopt stuff. Most of the non-CKAN paster commands aren't needed, but we will still need paster serve and paster post at least. For these, re-implement their UIs and docs in docopt, duplicating the UI stuff, just wrap them and call paster. paster make-config to be completely replaced by a new ckan make-config that just renders a Jinja2 template, accepting settings from env vars, and can be used to render both the dev/prod config file and the test one. The ckan executable to be installed as a script or console command or whatever in setup.py, so it ends up in the bin dir of the virtualenv. As soon as you activate a virtualenv, you get that CKAN instance's ckan command on your PATH. For the package install, we can make it symlink the ckan_default virtualenv's ckan executable onto the system-wide path so you don't even need to activate the virtualenv. For the source install we can add a command to make the same symlink. So by default you'll get the default CKAN instance's ckan command, but if you have multiple instances installed and activate one of their virtualenvs then you will get that ckan command. For docker we'll use CMD or ENTRY_POINT in the Dockerfile so you can just do
docker run serve
or maybedocker run ckan serve
without typing the full path. -
ckanext-scheming is alive #1795
- a vanilla schema: https://github.com/open-data/ckanext-scheming/blob/master/ckanext/scheming/ckan_dataset.json
- setup_template_variables (c. ew) ? add package_type (group_type) to each call?
- use dataset_type
- how should I allow registering new converters/validators?
- new interface to allow users to register converters/validators that get_converter/get_validator will return
Chair ian Notes ?
- #1814 - nigelb
- #1591 - amercader already speaking to them
- #1826 - joet
- #1697 - package patch - discussion continued. wardi arguing with himself. Preferring simpler method wip
- #1827 - is a fix for 1720 that is already assigned to nigel
- #1824 - related to 1826? joet
- #1799 - amercader
- #1819 - do we want to allow patch releases of requirements to be automatically updated? Disucssion has a while back, where requirements were pinned
- #1530 - passlib - amercader will try to do a 2nd review
- #1830 - vitor
- #1829 - Do we want to move away from nodejs and use a python implementation? - nigelb
- #1822 - Already marked as good for contribution - possible backwards incompatilibity issue if they are relying on dataset reprensentation in api v2
- #1821 - wardi
- #1820 - joet - probably broken for many other unicode characters?
- #1816 - amercader
General questions:
* CKAN command? (Sean added this but can't make the dev meeting today)
Really needed to make running commands in the docker container easier.
Also needed so we can template both the dev/prod and test config files.
How to implement it?
Sean's thoughts:
Rewrite cli.py with docopt. Don't want to duplicate any of the UI code that defines the commands, sub-commands, options and docstrings.
Delete all this and re-write it with docopt. But the code that actually carries out the commands - just keep it as-is but have it called by our
new docopt stuff.
Most of the non-CKAN paster commands aren't needed, but we will still need paster serve and paster post at least. For these, re-implement
their UIs and docs in docopt, duplicating the UI stuff, just wrap them and call paster.
paster make-config to be completely replaced by a new ckan make-config that just renders a Jinja2 template, accepting settings from env vars,
and can be used to render both the dev/prod config file and the test one.
The ckan executable to be installed as a script or console command or whatever in setup.py, so it ends up in the bin dir of the virtualenv.
As soon as you activate a virtualenv, you get that CKAN instance's ckan command on your PATH.
For the package install, we can make it symlink the ckan_default virtualenv's ckan executable onto the system-wide path so you don't even need to activate the virtualenv.
For the source install we can add a command to make the same symlink.
So by default you'll get the default CKAN instance's ckan command, but if you have multiple instances installed and activate one of their virtualenvs then you will get that ckan command.
For docker we'll use CMD or ENTRY_POINT in the Dockerfile so you can just do `docker run serve` or maybe `docker run ckan serve` without typing the full path.
Chair: Adria Notes: ian
- #1817 - upgrade sqlalchemy, requires some vdm changes in sqlalchemy-0.9-support branch, dread assigned
- #1777 - add api example to docs, better in a source file with a test, sean takes it
- #1734 - sean to merge
- #1658 - looking for second reviewer, adria busy for next couple weeks.. anyone?
- #1725 - joe merged
- #1732 - requester to add tests
- #1814, #1813 - docker-related nick assigned
- #1879 - package_patch: to discuss in general questions
- #1799 - sentry middleware, nigel to review; discussion around entry point required for sentry and other similar middleware, current middleware plugin isn't perfect for this case
- #1412 - users regenerate api keys, ongoing
- docker was merged!
- #1798 - sean assigned
- #1797 -
- #1819, #1816: requests + sqlalchemy version upgrades; nor more requirement upgrades add comments to #1816
- #1818 recource_show auth done wrong, sean to respond
- #1815, vitor is working on distinct with datastore
- (lots of docker issues), #1809 suggests removing old packaging scripts
- #1800 issues migrating from old docker to new docker config, issue for dev list?
- #1796, celery? ckanext-queue? we need background tasks for a few things. Celery+redis? redis by itself? queues are hard; let's go shopping. Ask open spending about their celery experience https://github.com/okfn/ckanext-queue
extras to discuss:
- #1768 organization_list all_fields misses out extras and users
- #1796 Changing group name can time out
General Questions
- rufus wants commit access to be able to close old PRs himself (raised last meeting)
- wardi: package_patch and json-patch APIs https://github.com/ckan/ckan/pull/1679
Chair: ian Notes: joe
- #1765
- #1659 - still discussing tests
- #1793 - Cannot determine why all cookies starting with ckan exists, behaviour seems bad. dread to review
- #1755 - dockerfile :-) ping nigel to see if has time soon to review.
- #1786 - joet to merge
- #1782 - going to be moved to wiki as it seems it will be more useful, issue will be closed, sean will comment
- #1725 - to merge subject to minor change re read_url
- #1251
- #1492
- #1796 - assigned to dread for now, discussion to continue
- #1794 - leave with vitor for now, joet thought we required this for ckan-pakistan project so it might exist already.
- extensions.ckan.org
General Questions
- retiring issues- we should cull old issues
- rufus wants commit access to be able to close old PRs himself (discuss at next meeting)
- #1659 - Almost there, model for new functional tests
- #1755 - Docker: docker links for postgres / solr
- #1776 - joe assigned
- #1786 - nick to look at test failure, joe to review
- #1783 - ckan hardware requirements: easy to get out of date, really depends on the number of datasets, other factors, joe to come back to adam
- #1789: ross to look at it
- #1788: expected 404 on deleted thing
- #1785: should be possible to only connect to the db / solr when needed,
- #1784 amercader to work on it at some point
General Questions
- Release 2.2.1 we talked with nigelb about splitting the work, but if someone wants to help here's a list of stuff that needs to be backported. By helping I mean reviewing and writing a comment "this looks good to be backported" or "this might cause issues https://github.com/ckan/ckan/issues?direction=desc&labels=Backport+pending%2CBackport+2.2.1&page=2&sort=created&state=closed https://github.com/ckan/ckan/wiki/Release-branches-tips-and-tricks
Chair: joe Notes: wardi (for a bit at least)
- #1686 nigel already assigned
- #1734 assuming nigel is still working on it
- #1755 nigel to review, possibly others
- #1725 joe assigned, nigel to look as well
- #1412 joe assigned
- #1777 nigel assigned
- #1781 unicode in site title issue - will ask for pr, sean assigned
- #1780 many ways to get user - wardi assigned
- #1779 resources disappearing (we should fix our API for updating resources) - nigel assigned
- #1776 datastore empty array failure - assigned to vitor
- #1775 owner_org_validator can't be overridden by auth - general agreement that the issue should be fixed, sean assigned and will mentor submitter towards a fix
General questions
- wardi: removing 2+year deprecated package form feature https://github.com/ckan/ckan/commit/faa8b296, related to #1659, generally agreed
- wardi: adding package type to template for scheming extension. harvester extension needs the same thing; adria says pass it to all the package templates
- nigel: ckan on SSL https://github.com/ckan/ckan/issues/1773 need to move an icon
Chair: amercader Notes: nigelb
- #1745: Agreed that style (d) works best, though we don't mind style (b). Adria to look.
- #1757: Assigned to sean.
- #1766: Assigned to nigel.
- #1765: Assigned to Ian. We would stick the unvalidated package dict as validated.
- #1664: Waiting for tests
- #1658: Already assigned to Ian
- #/1659: Assigned to vitor.
- #1761, #1760, #1759, #1758 - Assigned to sean.
- #1768: We'll wait for dread to explain.
- #1767: Assigned to vitor.
- #1764: Related to #1765.
- #1763: User-error. Adria will get back to user.
General questions
- Road to 2.3 (amercader)
What should go into the next stable release (without thinking about time)
- Resource views (#1251) - vitor working on this
- Upgrade requirements
- SQLAlchemy 0.7 > 0.8 - we can't go further because of vdm.
- rest: http://requires.io/
- Turn off OpenID (#1661) - joe
- Docker (#1755) - nick
- Ubuntu 14.04 (#1651) - nigel
- #1530] use passlib for passwords / apache log - joe
- Nuke Legacy templates +∞ need to create an issue-> (some) new functional tests -ian
- call_action - open, no issue yet.
- after_/before_action - Needs a bit of discussion
- responsive(?) - nigel to follow up with sam.
- How to avoid a plugin to be loaded? (vitor) - vitor to create an issue explaining the use case
- Validators that convert data as well (vitor)- vitor to create an issue explaining the use case
Chair: adria Notes: ian
- #1734 main.debug.css - can we compile these less files with python? node is difficult to install for some users. wardi: compile debug at the same as prod css as part of release?
- #1659 - vitor assigned again
- #1753 no new_ names, adria: +1, toby: rename new_authz and let things importing it break.
- #1752 #1751 example extensions should have the same directory layout as real extensions
- #1750 vendor versions not minified (?) assigned to vitor
- #1749 good for contribution
- #1748 web site problem
- #1747 option to allow html in some markdown fields, toby to make PR
- #1713 to backport fix to 2.2
General questions
- Nigel: responsive branch http://responsive.staging.ckanhosted.com/ responsive changes backported to 2.2, may not be maintained but required for some users https://github.com/ckan/ckan/tree/responsive-2.2 and https://github.com/mintcanary/ckan/tree/responsive present at next community meeting?
- Nigel: Post to ckan-announce about critical datapusher bug / new package release. kindly's fix is on master, but not 2.2.
- Nigel: ckanext-searchhistory https://github.com/ckan/ckanext-searchhistory In action: http://data.satapps.org/ present at next community meeting?
- ian: coding standards import change #1745 rename helpers to template_helpers? to open separate issue
- Chair: Dread
- Notes: Adrià
- https://github.com/ckan/ckan/pull/1744 joe
- https://github.com/ckan/ckan/pull/1694 nigel to merge
- https://github.com/ckan/ckan/issues/1737 Adrià to look for the old code on archiver that use to / does that and add a comment
General questions
sanitize urls / content
- https://github.com/reddit/reddit/blob/master/r2/r2/lib/utils/utils.py#L267
- https://github.com/okfn/ckanext-iati/commit/5dada32e0d37a18819e63d9c114a0affdf658c19
Chair: Ian Notes: dread
- https://github.com/ckan/ckan/pull/1740 assigned to joe
- https://github.com/ckan/ckan/pull/1736 Making user more explicit for the factory func. Might need more discussion. Ian agrees with it but maybe Adrià should give opinion. Assigned to Adrià. Ian not happy with site user concept - 'creation if not already created' causes unpredictable errors.
- https://github.com/ckan/ckan/pull/1739 assigned to dread
- https://github.com/ckan/ckan/issues/1741 Don't think the vuln is exploitable without edit permissions. No deliberate DOS events so far. Discussion about python 3, moving off pylons. Ian says hard bit to be joint python 2.7/3.3 compatible is the strings. Point him at move from pylons info. Assigned to wardi.
- https://github.com/ckan/ckan/issues/1737 could be difficult -needs a background service to do it - an extension or something else. wardi will point him to roadmap.
Chair: Ian
Notes: Adrià
No new issues
General Questions:
-
sanitize resource urls to solve https://github.com/okfn/ckanext-iati/issues/44
- Amercader to investigate libraries to sanitize urls
-
removing revisions - help may be coming
- Make revisioning optional Includes changing queries to not query Revision tables but the actual table
- Implement history in activites
- Move revision info to activities
- Completely remove them
-
storing extras in a single row
- https://github.com/GSA/ckanext-geodatagov/blob/master/ckanext/geodatagov/plugins.py#L254
- https://github.com/GSA/ckanext-geodatagov/blob/master/ckanext/geodatagov/plugins.py#L363
- amercader: first link merges them in a single extra, second expands them for showing
--- nigel away, deferred
- Nigel: responsive branch http://responsive.staging.ckanhosted.com/
- Nigel: Post to ckan-announce about datapusher bug / new package release
- Nigel: ckanext-searchhistory https://github.com/ckan/ckanext-searchhistory
Chair: Sean
Notes: Ian
- #
- #1724 - baseimage docker concerns from sean, vitor has been using docker for ~8 months with no problems
- #1731 - resource extras in solr - wardi assigned
- #1730 - vitor assigned
General Questions:
- navl_validate is in toolkit, adria has another use for it
- sean: plugin toolkit refactor, all agreed with removing class that replaces module
- adriaL check_access_core for plugins overriding access checking to call original check
- sean: docker discussion
--- nigel away, deferred
- Nigel: responsive branch
- Nigel: Post to ckan-announce about datapusher bug / new package release
- Nigel: ckanext-searchhistory
Chair: Adria
Notes: Ian
- #
- #1503 assigned to dread
- #1686 docs for recommended use of templates - nigel
- #1694 more docs, also nigel
- #1727 to add a doc page about deprecated features
- #
- #1726 docs improvement, good for contribution?
- #1725 suggestion for IDatastore interface, how should we allow datastore to be extended?
General Questions:
-
Adria: Census is great - could i get responses shared
- Rufus would say this should be kept to only e.g. tech team or steering group - people may have submitted thinking stuff was confidential ...
-
Rufus: CKAN extension registry - https://github.com/ckan/ideas-and-roadmap/issues/22
- ckan.org ...
-
Rufus: CKAN Service provider Directory - https://docs.google.com/a/okfn.org/forms/d/1ljyAam2fYtO2i9JLM5ZDjE8clqidAHMXq91kVglRFBE/edit?usp=drive_web
- Idea: from the http://ckan.org/2014/03/11/improve-how-the-ckan-community-works-your-suggestions-wanted/ survey
-
Rufus: recent chats with data.gouv.fr - have created ckan-lite in flask, but have committed to coming back. encourage more engagement & invite to tech team.
-
Nick: Dockerfile
- primary motivation is making installs for end users easier, developer benefits secondary
- adria: against master, use apache / mod_wsgi (consistency for support, docs) - both agreed
- sean: needs docs, same default directory (for consistency) - agreed
- vitor: let's get our tests using docker, separate work for later
--- nigel away, deferred
- Nigel: responsive branch
- Nigel: Post to ckan-announce about datapusher bug / new package release
- Nigel: ckanext-searchhistory
Chair: Adria
Notes: Ian
- #1707 - show a resource in any dataset, adria to help fix controller tests
- #1652 - nigel to look at
- #1724 - to discuss on thursday
- #1721 - avoids an extra get request on signup - nigel to review
- #1658 - passlib, nick and a number of us to view and comment, ian assigned
- #1701 - solr indexing current date on empty fields - joe
- #1723 - joe assigned
- #1711 - david raznick's opinion of some changes sought, will merge first anyway
- no new issues! (applause)
General Questions:
-
Rufus Pollock: review of draft post about technical team - https://docs.google.com/a/okfn.org/document/d/1ecqTGtX9FQPNgzlFB9my5tqK27xZ7rEC3D-hu-hyqq8/edit
- AGREED: look today, add corrections by 10am tomorrow and then RP to post
-
Rufus Pollock: CKAN Community Hangout 28th May - http://ckan.org/2014/05/19/ckan-community-hangout-may-2014/
- @Ian: will you be able to make it - yes
-
ckan-news coming up soon
- Newsletter scratchpad - any thoughts - https://docs.google.com/a/okfn.org/document/d/15giHeBLFWPEwJuDS6mAwwn9y3xw_sscCruZBi0sN0Hk/edit
-
ckan assoc roles - wip - https://docs.google.com/a/okfn.org/document/d/1vAxwTAVCEW0WMzD2UOOrIGDAKKYCOnQCQzPOYUj_4YI/edit
-
ckan-discuss migration to google groups
-
Heads up - http://csvconf.com/
-
DKAN - relationship with CKAN association (Maybe this does not need to go the Wiki?)
-
Not blessed, not part of CKAN association
-
Agree with them how we talk about each other
-
Would like to see API compatability
-
Happy to collaborate on components
-
-- rest punted until Thursday
- Census is great - could i get responses shared
- Nigel: responsive branch
- Nigel: Post to ckan-announce about datapusher bug / new package release
- Nigel: ckanext-searchhistory
- Chair: amercader
- Notes:
- #1711 pr is ready, original intention seems to be that you cannot update packages in organization update. pr fixes this (and the broken test)
General Questions:
- dread - reports show&tell - (http://data.gov.uk/data/report) make into extension? Yes!
- wardi - context antipatterns - to put into ideas issue and get feedback from kindly, before assigning some CKAN Assoc time to it.
- joet - registering users saves username/pass to apache log files.
- nigelb - responsive 2.2 (https://github.com/ckan/ckan/tree/responsive-2.2 )
Chair: Nigel
Notes: Adrià
- #1711 wardi: package memebership to orgs is defined both in member table and owner_org field, probably to ensure uniqueness. Also group_dict_save does a commit half way which looks really bad (https://github.com/ckan/ckan/blob/master/ckan/lib/dictization/model_save.py#L426) wardi to investigate
- #1668 nigel
- #1659 needs some tests
- #1665 Add datasets to any group regardless of membership. Do we need a config option or users should use custom auth functions (nigelb says that won't be enough
- #1715 Assigned to Ross
- #1713 Ask user to test on master and see if it's still fails
- #1712 Assigned to vitor
- #1710 Ask user for a patch
- #1709 How do we manage multiple extras in multiple resources? Assigned to adrià
- #1708 Ask user for a patch, assigned to joe
- #1706 leave as is
- #1705 remove /dataset/search endpoint. Ask more details about user extension that might be causing the issue
General Questions:
-
amercader: https://github.com/ckan/ckan/issues/1714 https://github.com/ckan/ckan/blob/master/ckan/logic/action/get.py#L2053 on CLI commands, session not handled as well as a web response. Try to change commit_and_remove on get_site_user by commit. Joe to look into cli db initialization field
-
wardi: organization_update packages again - adria says no more special behaviours for group/org_update for users field Investigate if _patch actions can be helpful here
- Chair
- amercader
- Notes
- nigelb
- #1665: nigel already reviewing.
- #1679: new type of action. should probably be in its own file. lets you provide a json patch to update a package. allows us to do atomic updates or updates that don't clobber each other. Ian wants feedback and someone else to take the issue :) json patch is list of operations. there's move, test, etc. So you can test your patch if a certain parameter fails. You can make it fail if it isn't what you wanted it to be. You can update a resource object or add a resource. If you want to update a whole resource you can do that. David asks if we would use revision id, Ian clarifies that is not how it's done. We're using a json patch library. It needs to be wrapped in things. The four lines it is right now is not complete. This is not atomic. If we want it to be better, we can build the patch into package_update. All of the business logic will need to be explicit, like about resources. The complexity will be in package_patch. We will need to consider what happens when someone sends an empty list of extras. We don't want to break anything. package_update should stay as is. It could potentially provide atomic updates to our user. There's more work, like it's own module and validation. Ian will stay assigned. Generally agreed it'll be really cool. The sooner we integrate, the better.
- [#1696](https://github.com/ckan/ckan/1696) Groups authorization bug. Nigel to continue working on.
- #1251 ongoing
- #1697 Ian to look again
- #1677 Vitor already reviewing
- #1512 Vitor will review. Need to check about localization.
- #1688 Adria to look
- #1617 assigned already
- #1605 assigned already
- #1659 need to talk about tests
- #1704 Need to take a look at library support. Allowing HTML is scary. Assigned to Nigel.
- #1701 If you're building a filter with dates, you'll get a lot of datasets with index date. We could also have 2 fields, one for string. And one for indexed. But the date field in solr will still be a proper date field. Adria already has a patch. We can discuss more.
- #1700 Vitor and Nigel to figure out. * ubuntu 14.04 support - to be decided about packages. We don't support ckan 1.8 unless there's a massive security bug.
General Questions:
- (amercader) Feedback on CKAN surveys. Add stuff, remove? http://pad.okfn.org/p/ckan-surveys
- (ian) functional tests with webtest: We need a way to test controllers. The suggestion is to start using webtest again, because in the scenario we don't care about js. Unless there's opposition, we'll use webtest with BeautifulSoup. Let's write one example with a pull request to discuss next week. Nigel to do this. David Read already has an example: https://github.com/ckan/ckan/blob/master/ckan/new_tests/controllers/test_util.py
- (nigel) Template tracebacks. Nigel wrote pull request to catch all exceptions so templates don't blow up. We should do a wrapper or something. Adria thinks helper should catch all expected exceptions. Catch the exception but re-raise it if debug mode is on. Otherwise make sure it sends an email. Ian wants the emails sent out. And about failures that may be related to us catching the exceptions. Agreed that catching all exceptions is a bad thing.
- (vitor) Logic about querying models outside of the models themselves: There were few places where we're querying models, like cli.py. Should we refactor that into models themselves. Adria agrees that this is a good idea. If the method from the model returned a query, we could add more filters. The might raise DetachedInstanceError. Vitor will bring up on ckan-dev about a more concrete example.
- (ian) vdm removal + sql for clearing old revisions (support from statcan). The options Ian had were clearing all revisions or dumping the packages into a clean database. Re-creating datasets might change UUIDs. David has a script that delete revisions (It's incredibly hard to do). Ian will try to convince new people to work on removing vdm.
- Clearing harvest sources (purging all its datasets) https://github.com/ckan/ckanext-harvest/blob/master/ckanext/harvest/logic/action/update.py#L110
- (ian) rename new_tests->tests and tests->legacy_tests: +1
- Chair
- joet
- Notes
- vitorbaptista
- #1665 - take care with how this changes the organization permissions as well (nigelb)
- #1679 - everyone agrees that it's a cool feature, but needs more discussion. we will revisit, please add your comments to the PR. (wardi)
- #1697 - needs more convincing. vitor will add a good test name that's over the line length limit to prove the point (vitorbaptista)
- #1659 - 2-stage dataset create is done, just missing (frontend) tests (vitorbaptista)
General Questions:
- nigelb: Do we land UI changes without tests or do we write old tests rather than no tests?
- we wondered what' s the issue with webtest. Nigel suggested doing integration tests with webtest through the controllers, and capybara (or something else) for pages that need javascript.
- Ian would like to unit test controllers. Nigel disagrees, saying that controllers are an integration point, so we shouldn't test them in isolation. Using webtest for controllers is fine. And we can use something for integration tests with javascript later.
- All in all, we felt webtest was good, but we would like to hear seanh/amercader opinions on this. Nigel'll mail ckan-dev talking about it.
- vitorbaptista: pull requests with commit messages not following our guidelines
- ask the person sending the PR to read http://docs.ckan.org/en/latest/contributing/commit-messages.html
- Chair
- amercader
- Notes
- nigelb
- #1692 assigned to vitor
- re: clean_action pull requst, it breaks calling dataset_show instead of package_show
- #1696 nigel to take a look. reproducable.
- #1694 sean will fix.
- #1598 If you hit the back button in the third page, the data in the third page is lost. The two state data creation might help. The state gets messed up by the package controller. There's no easy solution. We could use hidden fields without breaking customization.
General Questions:
- amercader: Proposal, dev polls. To know more about the ecosystem, help decide roadmap and recruit help
- Deployment: What OS do you use, Source install or package, How do you find the install instructions? Solr/ Postgres same machine, What OS would you like to see supported, Would you be willing to help test new releases on your distro? - generally agreed this is a good idea. amercader will setup a demo by next week.
- Chair
- wardi
- Notes
- nigelb
- #1692 sean assigned
- #1617 discussion continued, sean to merge soon
- #1685 joe
- #1684 joe
- #1683 wardi
- #1682 wardi
- #1677 vitor
- #1679 nigelb
- #1680 vitor
- #291 nigelb
- #1673 wardi
- #1503 sean to add new text
- #1671 nigelb
- #1693 sean to reply
- #1691 Documentation talks about old version of UI. Assigned to vitor.
- #1690 Activity stream can be disabled, but not revisions. Sean to close.
- #1688 wardi was trying get support for idea of removing draft stage. marked as good for contribution.
- #1687 context should not be in templates because of how it tightly coupled. Use get_action helper instead. Jinja has scoping problems when you use set. In other frameworks you would use a dictionary. c is messy and hard to track. Generally agreed that not using c is a good thing.
- #1686 But calling action functions from template not liked generally.
- #1681 It's kinda hard writing an expression to catch all domains and all email possibilities.
- #1678 wardi fixed it.
- #1676 assigned to adria
- #1675 general instructions/fixing https issues. talk to other people who have deployed ckan.
General Questions:
- [ckanext-realtime plugin by Justas Azna (reederz)] ( https://github.com/alexandrainst/ckanext-realtime ) Jutas joined us to talk about it. It observes datastores in real-time. It can makes websockets available to observe datastore. It's an internship project. There's documentation and tests. Once you setup the datastore, you need to make an API call which inserts a trigger. Uses postgres to listen to real time changes. There's a datastore listener script which listens for this, which sends to ckan to redis which sends to websocket server. There's a twisted eventloop between redis and websocket. Just calling the API makes it observable. It has some limitations. Currently, there can only be one websocket connection because it needs authentication and one API key can only get one connection. Justus is going to change it. It wouldn't be an additional load. May need to add multiple websocket servers.
- vitor: convert issues to pull requests - use "hub pull-request -i " (https://github.com/github/hub). Easy to convert issue to pull request. P
- Chair
- seanh
- Notes
- Ian
- #1667 break long title words - nigel to review
- #1666 fix for serious errors to data pusher - need a decision on front end tests to add a test, joe to review
- #1503 two-licenses issue - continued discussion on ambiguity, discusson to continue on pull request
- #1664 package_create/update auth fix - adria to review, sean to comment on how to add test
- #1251 resource views - continued work, adria to review
- #1663 remove repeated string - vitor assigned
- #1617 DetachedInstanceError - when new session is created old one is closed automatically, leading to a DetachedInstanceError later on, nigel has a fix but further explanation is requested. deep ckan knowledge required
- #1656 paster db clean fix - seanh assigned
- #1412 regenerate api keys - needs more tests joe reviewing, currently only usable from the front end. multiple api keys? possibly in the future
- #1668 deleted user bug - nigel assigned
- #1665 group auth for logged in user - possibly not complicated to implement, need to investigate why feature was removed in the first place - vitor volunteered
- #1662 can't deprecate with maintain.deprecated - likely decorator broken on object or controller method, possibly not designed for use around actions - joe to take a look
General Questions:
- joet: turn off openid by default? - to proceed
- joet: passlib is ok? https://github.com/ckan/ckan/pull/1658 agreed a great idea, work to continue and multiple reviewers to be requested when ready
- nigel: email hash https://github.com/ckan/ckan/pull/1657 helper function vs. special context vs. sysadmin access to user_list action, nigel to work on removing from api while still supporting gravatar in the user list controller
- seanh: DetachedInstanceError questions, discussion of controller, logic, model layers. discussed adding to description to contribting docs along with guidance about how to avoid replicating old code. session and model objects in context? are they singletons? No: session is per-request. model is passed because model.session might create a new session or return the existing one. model is also possibly good for mocking in tests.
- Chair
- Adria
- Notes
- Ian
- #1656 joe to add some notes to docstring
- #1412 joe to look at it
- #1617 ready for review, assigned to sean
- #1655 nigel assigned
- #1654 wardi assigned
- #1652 nigel assigned
- #1651 ubuntu 14.04 LTS support, ongoing
General Questions:
- nigelb: Documentation or UI for revisions. https://github.com/ckan/ideas-and-roadmap/issues/41 proposal to merge the revision information into the activities. First, revision/diff page can be resurrected https://github.com/ckan/ideas-and-roadmap/issues/33 https://github.com/ckan/ckan/pull/1590 based on activity and resource data that already exists in the database
- amercader: coveralls comments: can we turn them off? coveralls creates a lot of noise in emails, particlarly for small changes. can we set a threshold for the % change that creates an notification? yes https://coveralls.io/docs/notifications can we get notifications if it drops below 80%? vitor to investigate
- wardi: workflow plugins based on activities model: https://github.com/ckan/ideas-and-roadmap/issues/45 the future! we just need funding (wardi) JSON Patch is a thing and has a python implementation http://jsonpatch.com/ https://tools.ietf.org/html/rfc6902
- vitor: pull requests that lower our test coverage implement a policy on acceptance of patches that lower coverage? vitorbaptista is going to write some guidelines into the contributing docs and core devs will be stricter with their own PRs
- Chair
- Adria
- Notes
- Nigel
- #1251: sean is looking
- #1641: still WIP.
- #1550: nigel still working
- #1539: adria got volunteered at the last meeting
- #1649, #1648, #1647: All of them are correct. Someone needs to review and fix it. We'll ask Ross to take a look and review on Tuesday again if he doesn't want to.
- #1646: nigel
- #1643: It may potentially need to be removed. A type of homepage might show facets, need digging. This also assumes that there's only one type of package. It could be used in a different type of home page. We could change it so that you can request facets instead of having them declared in your controller.
General Questions:
-
CLA Feedback.
- Background: As we started the CKAN Association and moved to encouraging more contributions - we needed to get the legal status of CKAN core code on better footing. We (mostly Ira) explored both licence agreements and copyright assignments. Legally we are better protected with copyright assignment (presenting a united front to any who may want to challenge / sue), but data.gov.uk and others presented a strong preference for licencing agreement instead. We also reviewed all the various options that other open source projects and foundations use (usually a mix of the two). Ira talked lots to the JQuery Director who both have copyright assignement (for core devs) and licence agreements for others. After talking to the OKF lawyer, we decided to re-use their agreement (see here: http://contribute.jquery.org/CLA/) which they gave us permission for, with a few small tweaks our lawyer suggested. This meant that we take on less risk, using an agreement that already exists and has been used ‘in the wild’ and hence has set a precedent for how any legal challenges and complications are handled and how well this stands up in court etc.
- From Meeting: We all agree that we need but the proposed one has issues raised on the repo. The issues raised are not resolved yet. The issues would prefer to be closed when the person who opened it is happy with the response, but we shouldn't bikeshed and decide on a process. Rufus has responded and the dev team will chase it. Most people who have contributed have been employed by Open Knowledge, so historical contributors aren't that big of a problem. Going forward, we need to get it setup for new people contributing. How does this affect extensions? How are things on github.com/ckan considered core and do we make github.com/ckan similarly licensed and marked as core? Extensions need to be published anyway under AGPL. Someone that has been given the CKAN produce re-licensed would have difficulty with extensions. Anyone who created an issue on the cla needs to reply about issue, so we can keep the discussion moving. nigel to arrange meeting to discuss remaining issues in 2 weeks.
- Office hours. Rufus suggested and the people who want to come in are not peple who want to discuss things. Related issue people in the US who want to make it to the meetings, but the meetings are difficult for the US. From a user POV, this is great, but dev POV, this may end up being dedicated support time. The name is important and "office hours" might convey the wrong meaning. The discussions that we wanted part of it sounded good. Rufus is willing to step up and run a community hangout but needs someone from the tech team to help out. People should propose what they want to talk about. Ira suggested that this needs to be about QA, roadmap, etc. We generally think this is a good idea.
- (adam) Persistent Hangout? Adam to setup a persistent hangout. - https://plus.google.com/hangouts/_/okfn.org/ckan-dev-call
- (amercader) Supporting Ubuntu 14.04. Should we support 12.04 and 14.04 at the same time? It also switches to Python 3. Adria will create a new issue and we can start tracking this in there. Do we drop support for 12.04? We can't drop support easily. We could perhaps setup travis for 14.04 and also think about packages.
- (ira) Frequency of server or internal errors on demo; is this across all of ckan? who looks at logs? why is this so flaky? Demo is flaky for various reasons. Nigel to talk to Ira. David suggested setting up sentry.
- (amercader) 2.2.1 needs to happen. There are enough bugs to justify this. If you think something needs to be backported, tag it as such.
- (ira) possible for previews for what the resource file looks on the dataset page. Like a thumbnail of it. Potentially a great extension. If there's already a graph, a thumbnail of it will be really engaging. Potentially do it from a datapusher sort of thing in an extension. Ira to go and get lots of money to get it done!
- (wardi) moar cla discussion (only if there is time). Ian talked to devs at open contracting. They had seen CKAN but thought it was way too complicated and used Django instead. There wasn't a lot of compelling reasons. This adds danger to CKAN as a project. That'll give competitors the advantage. The copyright assignment bits are the dangerous bits. Was the lawyer's concern about copyright assignment? Copyright is not a very good open source citizen thing to do. The lawyer said this is the most legally defensible position. It is what others are using. It's not balanced, OKF has more rights than everyone else. Copyright assignment vs licencing. In a CLA, you give license, not give copyright. The difference is whether you actually assign copyright.
If there's a chance of someone going to pay to relicense CKAN, it's enough money to rewrite the whole thing. Ira is fairly happy as long as we have some clarity. Ira will look at DCO and see if it meets the things that Ira was worried about. Recommended reading: https://github.com/ckan/cla/issues/2. It's important to capture the questions and motivations of the CLA.
- Chair
- Nigel
- Notes
- Everyone
- #1641 resource_create should break when called without URL (vitorbapista)
- #1644 - add xlsx tsv to datapusher defaults (joet)
- #1617 - detached instance error (seanh)
- #1646 - nigelb will reply
- #1645 - should be part of a larger discussion about the api, is it consistent with package_show, user_list, etc. The count of the number of packages should be consistent as well, and not clash with another field in another api call - wardi
- #1643 - seanh
General Questions:
- (amercader) Meeting notes to wiki: Tricky, as no easy way to export to Markdown. Tried this but not great, as issues #xxx are translated as headings. Perhaps Etherpad allows export to Markdown? - (generally agreed)
https://docs.google.com/document/d/1MRNmhqBthZILBF8oG1NkwmyofRfOXr3NRJA1CCuHVD4/pub