Skip to content

Dev Meeting Notes

Jump to bottom Edit New page
Ian Ward edited this page Jun 30, 2014 · 32 revisions

Everyone is welcome, just add an item to the pad if you want something raised.

Meeting 2014-06-26

Chair: ian Notes: joe

Pull Requests (47 open)

  • #1765
  • #1659 - still discussing tests
  • #1793 - Cannot determine why all cookies starting with ckan exists, behaviour seems bad. dread to review
  • #1755 - dockerfile :-) ping nigel to see if has time soon to review.
  • #1786 - joet to merge
  • #1782 - going to be moved to wiki as it seems it will be more useful, issue will be closed, sean will comment
  • #1725 - to merge subject to minor change re read_url
  • #1251
  • #1492

Issues (250 open)

  • #1796 - assigned to dread for now, discussion to continue

  • #1794 - leave with vitor for now, joet thought we required this for ckan-pakistan project so it might exist already.

  • extensions.ckan.org

General Questions

  • retiring issues- we should cull old issues
  • rufus wants commit access to be able to close old PRs himself (discuss at next meeting)

Meeting 2014-06-24

Pull Requests (51 open)

  • #1659 - Almost there, model for new functional tests
  • #1755 - Docker: docker links for postgres / solr
  • #1776 - joe assigned
  • #1786 - nick to look at test failure, joe to review
  • #1783 - ckan hardware requirements: easy to get out of date, really depends on the number of datasets, other factors, joe to come back to adam

Issues (x open)

  • #1789: ross to look at it
  • #1788: expected 404 on deleted thing
  • #1785: should be possible to only connect to the db / solr when needed,
  • #1784 amercader to work on it at some point

General Questions

Meeting 2014-06-19

Chair: joe Notes: wardi (for a bit at least)

Pull Requests (48 open)

  • #1686 nigel already assigned
  • #1734 assuming nigel is still working on it
  • #1755 nigel to review, possibly others
  • #1725 joe assigned, nigel to look as well
  • #1412 joe assigned
  • #1777 nigel assigned

Issues (244 open)

  • #1781 unicode in site title issue - will ask for pr, sean assigned
  • #1780 many ways to get user - wardi assigned
  • #1779 resources disappearing (we should fix our API for updating resources) - nigel assigned
  • #1776 datastore empty array failure - assigned to vitor
  • #1775 owner_org_validator can't be overridden by auth - general agreement that the issue should be fixed, sean assigned and will mentor submitter towards a fix

General questions

Meeting 2014-06-12

Chair: amercader Notes: nigelb

Pull Requests (55 open)

  • #1745: Agreed that style (d) works best, though we don't mind style (b). Adria to look.
  • #1757: Assigned to sean.
  • #1766: Assigned to nigel.
  • #1765: Assigned to Ian. We would stick the unvalidated package dict as validated.
  • #1664: Waiting for tests
  • #1658: Already assigned to Ian
  • #/1659: Assigned to vitor.
  • #1761, #1760, #1759, #1758 - Assigned to sean.

Issues (247 open)

  • 1769: Nigel

  • #1768: We'll wait for dread to explain.
  • #1767: Assigned to vitor.
  • #1764: Related to #1765.
  • #1763: User-error. Adria will get back to user.

General questions

  • Road to 2.3 (amercader) What should go into the next stable release (without thinking about time)
    • Resource views (#1251) - vitor working on this
    • Upgrade requirements
    • Turn off OpenID (#1661) - joe
    • Docker (#1755) - nick
    • Ubuntu 14.04 (#1651) - nigel
    • #1530] use passlib for passwords / apache log - joe
    • Nuke Legacy templates +∞ need to create an issue-> (some) new functional tests -ian
    • call_action - open, no issue yet.
    • after_/before_action - Needs a bit of discussion
    • responsive(?) - nigel to follow up with sam.
    • How to avoid a plugin to be loaded? (vitor) - vitor to create an issue explaining the use case
    • Validators that convert data as well (vitor)- vitor to create an issue explaining the use case

Meeting 2014-06-10

Chair: adria Notes: ian

Pull Requests (49 open)

  • #1734 main.debug.css - can we compile these less files with python? node is difficult to install for some users. wardi: compile debug at the same as prod css as part of release?
  • #1659 - vitor assigned again

Issues (239 open)

  • #1753 no new_ names, adria: +1, toby: rename new_authz and let things importing it break.
  • #1752 #1751 example extensions should have the same directory layout as real extensions
  • #1750 vendor versions not minified (?) assigned to vitor
  • #1749 good for contribution
  • #1748 web site problem
  • #1747 option to allow html in some markdown fields, toby to make PR
  • #1713 to backport fix to 2.2

General questions

Meeting 2014-06-05

  • Chair: Dread
  • Notes: Adrià

Pull Requests (50 open)

Issues (236 open)

General questions

sanitize urls / content

Meeting 2014-06-03

Chair: Ian Notes: dread

Pull Requests (49 open)

Issues (235 open)

  • https://github.com/ckan/ckan/issues/1741 Don't think the vuln is exploitable without edit permissions. No deliberate DOS events so far. Discussion about python 3, moving off pylons. Ian says hard bit to be joint python 2.7/3.3 compatible is the strings. Point him at move from pylons info. Assigned to wardi.
  • https://github.com/ckan/ckan/issues/1737 could be difficult -needs a background service to do it - an extension or something else. wardi will point him to roadmap.

Meeting 2014-05-29

Chair: Ian

Notes: Adrià

Pull Requests (46 open)

  • #1734 sean will look into it
  • #1727 nigel to answer back or merge it if he is happy

Issues (229 open)

No new issues

General Questions:

--- nigel away, deferred

Meeting 2014-05-27

Chair: Sean

Notes: Ian

Pull Requests (47 open)

  • #
  • #1724 - baseimage docker concerns from sean, vitor has been using docker for ~8 months with no problems
  • #1731 - resource extras in solr - wardi assigned
  • #1730 - vitor assigned

Issues (231 open)

  • #

  • 1733 - adria: nigel said this would be easy for the new filestore, nigel assigned

  • 1729 - need to backport fix from resource-views branch

General Questions:

  • navl_validate is in toolkit, adria has another use for it
  • sean: plugin toolkit refactor, all agreed with removing class that replaces module
  • adriaL check_access_core for plugins overriding access checking to call original check
  • sean: docker discussion

--- nigel away, deferred

  • Nigel: responsive branch
  • Nigel: Post to ckan-announce about datapusher bug / new package release
  • Nigel: ckanext-searchhistory

Meeting 2014-05-22

Chair: Adria

Notes: Ian

Pull Requests (46 open)

  • #
  • #1503 assigned to dread
  • #1686 docs for recommended use of templates - nigel
  • #1694 more docs, also nigel
  • #1727 to add a doc page about deprecated features

Issues (228 open)

General Questions:

--- nigel away, deferred

  • Nigel: responsive branch
  • Nigel: Post to ckan-announce about datapusher bug / new package release
  • Nigel: ckanext-searchhistory

Meeting 2014-05-20

Chair: Adria

Notes: Ian

Pull Requests (47 open)

  • #1707 - show a resource in any dataset, adria to help fix controller tests
  • #1652 - nigel to look at
  • #1724 - to discuss on thursday
  • #1721 - avoids an extra get request on signup - nigel to review
  • #1658 - passlib, nick and a number of us to view and comment, ian assigned
  • #1701 - solr indexing current date on empty fields - joe
  • #1723 - joe assigned
  • #1711 - david raznick's opinion of some changes sought, will merge first anyway

Issues (231 open)

  • no new issues! (applause)

General Questions:

-- rest punted until Thursday

  • Census is great - could i get responses shared
  • Nigel: responsive branch
  • Nigel: Post to ckan-announce about datapusher bug / new package release
  • Nigel: ckanext-searchhistory

Meeting 2014-05-15

  • Chair: amercader
  • Notes:

Pull Requests (47 open)

  • #1711 pr is ready, original intention seems to be that you cannot update packages in organization update. pr fixes this (and the broken test)

Issues (233 open)

General Questions:

Meeting 2014-05-13

Chair: Nigel

Notes: Adrià

Pull Requests (46 open)

Issues (230 open)

  • #1715 Assigned to Ross
  • #1713 Ask user to test on master and see if it's still fails
  • #1712 Assigned to vitor
  • #1710 Ask user for a patch
  • #1709 How do we manage multiple extras in multiple resources? Assigned to adrià
  • #1708 Ask user for a patch, assigned to joe
  • #1706 leave as is
  • #1705 remove /dataset/search endpoint. Ask more details about user extension that might be causing the issue

General Questions:

Meeting 2014-05-08

Chair
amercader
Notes
nigelb

Pull Requests (45 open)

  • #1665: nigel already reviewing.
  • #1679: new type of action. should probably be in its own file. lets you provide a json patch to update a package. allows us to do atomic updates or updates that don't clobber each other. Ian wants feedback and someone else to take the issue :) json patch is list of operations. there's move, test, etc. So you can test your patch if a certain parameter fails. You can make it fail if it isn't what you wanted it to be. You can update a resource object or add a resource. If you want to update a whole resource you can do that. David asks if we would use revision id, Ian clarifies that is not how it's done. We're using a json patch library. It needs to be wrapped in things. The four lines it is right now is not complete. This is not atomic. If we want it to be better, we can build the patch into package_update. All of the business logic will need to be explicit, like about resources. The complexity will be in package_patch. We will need to consider what happens when someone sends an empty list of extras. We don't want to break anything. package_update should stay as is. It could potentially provide atomic updates to our user. There's more work, like it's own module and validation. Ian will stay assigned. Generally agreed it'll be really cool. The sooner we integrate, the better.
  • [#1696](https://github.com/ckan/ckan/1696) Groups authorization bug. Nigel to continue working on.
  • #1251 ongoing
  • #1697 Ian to look again
  • #1677 Vitor already reviewing
  • #1512 Vitor will review. Need to check about localization.
  • #1688 Adria to look
  • #1617 assigned already
  • #1605 assigned already
  • #1659 need to talk about tests

Issues (221 open)

  • #1704 Need to take a look at library support. Allowing HTML is scary. Assigned to Nigel.
  • #1701 If you're building a filter with dates, you'll get a lot of datasets with index date. We could also have 2 fields, one for string. And one for indexed. But the date field in solr will still be a proper date field. Adria already has a patch. We can discuss more.
  • #1700 Vitor and Nigel to figure out. * ubuntu 14.04 support - to be decided about packages. We don't support ckan 1.8 unless there's a massive security bug.

General Questions:

  • (amercader) Feedback on CKAN surveys. Add stuff, remove? http://pad.okfn.org/p/ckan-surveys
  • (ian) functional tests with webtest: We need a way to test controllers. The suggestion is to start using webtest again, because in the scenario we don't care about js. Unless there's opposition, we'll use webtest with BeautifulSoup. Let's write one example with a pull request to discuss next week. Nigel to do this. David Read already has an example: https://github.com/ckan/ckan/blob/master/ckan/new_tests/controllers/test_util.py
  • (nigel) Template tracebacks. Nigel wrote pull request to catch all exceptions so templates don't blow up. We should do a wrapper or something. Adria thinks helper should catch all expected exceptions. Catch the exception but re-raise it if debug mode is on. Otherwise make sure it sends an email. Ian wants the emails sent out. And about failures that may be related to us catching the exceptions. Agreed that catching all exceptions is a bad thing.
  • (vitor) Logic about querying models outside of the models themselves: There were few places where we're querying models, like cli.py. Should we refactor that into models themselves. Adria agrees that this is a good idea. If the method from the model returned a query, we could add more filters. The might raise DetachedInstanceError. Vitor will bring up on ckan-dev about a more concrete example.
  • (ian) vdm removal + sql for clearing old revisions (support from statcan). The options Ian had were clearing all revisions or dumping the packages into a clean database. Re-creating datasets might change UUIDs. David has a script that delete revisions (It's incredibly hard to do). Ian will try to convince new people to work on removing vdm.
  • Clearing harvest sources (purging all its datasets) https://github.com/ckan/ckanext-harvest/blob/master/ckanext/harvest/logic/action/update.py#L110
  • (ian) rename new_tests->tests and tests->legacy_tests: +1

Meeting 2014-05-06

Chair
joet
Notes
vitorbaptista

Pull Requests (44 open)

  • #1665 - take care with how this changes the organization permissions as well (nigelb)
  • #1679 - everyone agrees that it's a cool feature, but needs more discussion. we will revisit, please add your comments to the PR. (wardi)
  • #1697 - needs more convincing. vitor will add a good test name that's over the line length limit to prove the point (vitorbaptista)
  • #1659 - 2-stage dataset create is done, just missing (frontend) tests (vitorbaptista)

Issues (221 open)

General Questions:

  • nigelb: Do we land UI changes without tests or do we write old tests rather than no tests?
    • we wondered what' s the issue with webtest. Nigel suggested doing integration tests with webtest through the controllers, and capybara (or something else) for pages that need javascript.
    • Ian would like to unit test controllers. Nigel disagrees, saying that controllers are an integration point, so we shouldn't test them in isolation. Using webtest for controllers is fine. And we can use something for integration tests with javascript later.
    • All in all, we felt webtest was good, but we would like to hear seanh/amercader opinions on this. Nigel'll mail ckan-dev talking about it.
  • vitorbaptista: pull requests with commit messages not following our guidelines

Meeting 2014-05-01

Chair
amercader
Notes
nigelb

Pull Requests (48 open)

  • #1692 assigned to vitor
  • re: clean_action pull requst, it breaks calling dataset_show instead of package_show

Issues (230 open)

  • #1696 nigel to take a look. reproducable.
  • #1694 sean will fix.
  • #1598 If you hit the back button in the third page, the data in the third page is lost. The two state data creation might help. The state gets messed up by the package controller. There's no easy solution. We could use hidden fields without breaking customization.

General Questions:

  • amercader: Proposal, dev polls. To know more about the ecosystem, help decide roadmap and recruit help
    • Deployment: What OS do you use, Source install or package, How do you find the install instructions? Solr/ Postgres same machine, What OS would you like to see supported, Would you be willing to help test new releases on your distro? - generally agreed this is a good idea. amercader will setup a demo by next week.

Meeting 2014-04-29

Chair
wardi
Notes
nigelb

Pull Requests (47 open)

Issues (231 open)

  • #1693 sean to reply
  • #1691 Documentation talks about old version of UI. Assigned to vitor.
  • #1690 Activity stream can be disabled, but not revisions. Sean to close.
  • #1688 wardi was trying get support for idea of removing draft stage. marked as good for contribution.
  • #1687 context should not be in templates because of how it tightly coupled. Use get_action helper instead. Jinja has scoping problems when you use set. In other frameworks you would use a dictionary. c is messy and hard to track. Generally agreed that not using c is a good thing.
  • #1686 But calling action functions from template not liked generally.
  • #1681 It's kinda hard writing an expression to catch all domains and all email possibilities.
  • #1678 wardi fixed it.
  • #1676 assigned to adria
  • #1675 general instructions/fixing https issues. talk to other people who have deployed ckan.

General Questions:

  • [ckanext-realtime plugin by Justas Azna (reederz)] ( https://github.com/alexandrainst/ckanext-realtime ) Jutas joined us to talk about it. It observes datastores in real-time. It can makes websockets available to observe datastore. It's an internship project. There's documentation and tests. Once you setup the datastore, you need to make an API call which inserts a trigger. Uses postgres to listen to real time changes. There's a datastore listener script which listens for this, which sends to ckan to redis which sends to websocket server. There's a twisted eventloop between redis and websocket. Just calling the API makes it observable. It has some limitations. Currently, there can only be one websocket connection because it needs authentication and one API key can only get one connection. Justus is going to change it. It wouldn't be an additional load. May need to add multiple websocket servers.
  • vitor: convert issues to pull requests - use "hub pull-request -i " (https://github.com/github/hub). Easy to convert issue to pull request. P

Meeting 2014-04-24

Chair
seanh
Notes
Ian

Pull Requests (39 open)

  • #1667 break long title words - nigel to review
  • #1666 fix for serious errors to data pusher - need a decision on front end tests to add a test, joe to review
  • #1503 two-licenses issue - continued discussion on ambiguity, discusson to continue on pull request
  • #1664 package_create/update auth fix - adria to review, sean to comment on how to add test
  • #1251 resource views - continued work, adria to review
  • #1663 remove repeated string - vitor assigned
  • #1617 DetachedInstanceError - when new session is created old one is closed automatically, leading to a DetachedInstanceError later on, nigel has a fix but further explanation is requested. deep ckan knowledge required
  • #1656 paster db clean fix - seanh assigned
  • #1412 regenerate api keys - needs more tests joe reviewing, currently only usable from the front end. multiple api keys? possibly in the future

Issues (212 open)

  • #1668 deleted user bug - nigel assigned
  • #1665 group auth for logged in user - possibly not complicated to implement, need to investigate why feature was removed in the first place - vitor volunteered
  • #1662 can't deprecate with maintain.deprecated - likely decorator broken on object or controller method, possibly not designed for use around actions - joe to take a look

General Questions:

  • joet: turn off openid by default? - to proceed
  • joet: passlib is ok? https://github.com/ckan/ckan/pull/1658 agreed a great idea, work to continue and multiple reviewers to be requested when ready
  • nigel: email hash https://github.com/ckan/ckan/pull/1657 helper function vs. special context vs. sysadmin access to user_list action, nigel to work on removing from api while still supporting gravatar in the user list controller
  • seanh: DetachedInstanceError questions, discussion of controller, logic, model layers. discussed adding to description to contribting docs along with guidance about how to avoid replicating old code. session and model objects in context? are they singletons? No: session is per-request. model is passed because model.session might create a new session or return the existing one. model is also possibly good for mocking in tests.

Meeting 2014-04-22

Chair
Adria
Notes
Ian

Pull Requests (33 open)

  • #1656 joe to add some notes to docstring
  • #1412 joe to look at it
  • #1617 ready for review, assigned to sean
  • #1655 nigel assigned
  • #1654 wardi assigned
  • #1652 nigel assigned

Issues (207 open)

  • #1651 ubuntu 14.04 LTS support, ongoing

General Questions:

Meeting 2014-04-17

Chair
Adria
Notes
Nigel

Pull Requests (30 open)

  • #1251: sean is looking
  • #1641: still WIP.
  • #1550: nigel still working
  • #1539: adria got volunteered at the last meeting

Issues (205 open)

  • #1649, #1648, #1647: All of them are correct. Someone needs to review and fix it. We'll ask Ross to take a look and review on Tuesday again if he doesn't want to.
  • #1646: nigel
  • #1643: It may potentially need to be removed. A type of homepage might show facets, need digging. This also assumes that there's only one type of package. It could be used in a different type of home page. We could change it so that you can request facets instead of having them declared in your controller.

General Questions:

  • CLA Feedback.
    • Background: As we started the CKAN Association and moved to encouraging more contributions - we needed to get the legal status of CKAN core code on better footing. We (mostly Ira) explored both licence agreements and copyright assignments. Legally we are better protected with copyright assignment (presenting a united front to any who may want to challenge / sue), but data.gov.uk and others presented a strong preference for licencing agreement instead. We also reviewed all the various options that other open source projects and foundations use (usually a mix of the two). Ira talked lots to the JQuery Director who both have copyright assignement (for core devs) and licence agreements for others. After talking to the OKF lawyer, we decided to re-use their agreement (see here: http://contribute.jquery.org/CLA/) which they gave us permission for, with a few small tweaks our lawyer suggested. This meant that we take on less risk, using an agreement that already exists and has been used ‘in the wild’ and hence has set a precedent for how any legal challenges and complications are handled and how well this stands up in court etc.
    • From Meeting: We all agree that we need but the proposed one has issues raised on the repo. The issues raised are not resolved yet. The issues would prefer to be closed when the person who opened it is happy with the response, but we shouldn't bikeshed and decide on a process. Rufus has responded and the dev team will chase it. Most people who have contributed have been employed by Open Knowledge, so historical contributors aren't that big of a problem. Going forward, we need to get it setup for new people contributing. How does this affect extensions? How are things on github.com/ckan considered core and do we make github.com/ckan similarly licensed and marked as core? Extensions need to be published anyway under AGPL. Someone that has been given the CKAN produce re-licensed would have difficulty with extensions. Anyone who created an issue on the cla needs to reply about issue, so we can keep the discussion moving. nigel to arrange meeting to discuss remaining issues in 2 weeks.
  • Office hours. Rufus suggested and the people who want to come in are not peple who want to discuss things. Related issue people in the US who want to make it to the meetings, but the meetings are difficult for the US. From a user POV, this is great, but dev POV, this may end up being dedicated support time. The name is important and "office hours" might convey the wrong meaning. The discussions that we wanted part of it sounded good. Rufus is willing to step up and run a community hangout but needs someone from the tech team to help out. People should propose what they want to talk about. Ira suggested that this needs to be about QA, roadmap, etc. We generally think this is a good idea.
  • (adam) Persistent Hangout? Adam to setup a persistent hangout. - https://plus.google.com/hangouts/_/okfn.org/ckan-dev-call
  • (amercader) Supporting Ubuntu 14.04. Should we support 12.04 and 14.04 at the same time? It also switches to Python 3. Adria will create a new issue and we can start tracking this in there. Do we drop support for 12.04? We can't drop support easily. We could perhaps setup travis for 14.04 and also think about packages.
  • (ira) Frequency of server or internal errors on demo; is this across all of ckan? who looks at logs? why is this so flaky? Demo is flaky for various reasons. Nigel to talk to Ira. David suggested setting up sentry.
  • (amercader) 2.2.1 needs to happen. There are enough bugs to justify this. If you think something needs to be backported, tag it as such.
  • (ira) possible for previews for what the resource file looks on the dataset page. Like a thumbnail of it. Potentially a great extension. If there's already a graph, a thumbnail of it will be really engaging. Potentially do it from a datapusher sort of thing in an extension. Ira to go and get lots of money to get it done!
  • (wardi) moar cla discussion (only if there is time). Ian talked to devs at open contracting. They had seen CKAN but thought it was way too complicated and used Django instead. There wasn't a lot of compelling reasons. This adds danger to CKAN as a project. That'll give competitors the advantage. The copyright assignment bits are the dangerous bits. Was the lawyer's concern about copyright assignment? Copyright is not a very good open source citizen thing to do. The lawyer said this is the most legally defensible position. It is what others are using. It's not balanced, OKF has more rights than everyone else. Copyright assignment vs licencing. In a CLA, you give license, not give copyright. The difference is whether you actually assign copyright.

If there's a chance of someone going to pay to relicense CKAN, it's enough money to rewrite the whole thing. Ira is fairly happy as long as we have some clarity. Ira will look at DCO and see if it meets the things that Ira was worried about. Recommended reading: https://github.com/ckan/cla/issues/2. It's important to capture the questions and motivations of the CLA.

Meeting 2014-04-15

Chair
Nigel
Notes
Everyone

Pull Requests (32 open)

  • #1641 resource_create should break when called without URL (vitorbapista)
  • #1644 - add xlsx tsv to datapusher defaults (joet)
  • #1617 - detached instance error (seanh)

Issues (205 open)

  • #1646 - nigelb will reply
  • #1645 - should be part of a larger discussion about the api, is it consistent with package_show, user_list, etc. The count of the number of packages should be consistent as well, and not clash with another field in another api call - wardi
  • #1643 - seanh

General Questions:

  • (amercader) Meeting notes to wiki: Tricky, as no easy way to export to Markdown. Tried this but not great, as issues #xxx are translated as headings. Perhaps Etherpad allows export to Markdown? - (generally agreed)

Older meeting notes

https://docs.google.com/document/d/1MRNmhqBthZILBF8oG1NkwmyofRfOXr3NRJA1CCuHVD4/pub

Add a custom footer
Add a custom sidebar
Clone this wiki locally