Skip to content

Weekly Meeting 2016 08 04

Jump to bottom Edit New page
Tim Pepper edited this page Aug 4, 2016 · 5 revisions

Agenda

##Minutes

#ciao-project Meeting

Meeting started by sameo at 16:01:48 UTC. The full logs are available at ciao-project/2016/ciao-project.2016-08-04-16.01.log.html .

Meeting summary

  • Opens (sameo, 16:05:41)

    • ACTION: obedmr to present ansible/docker during the f2f week. (sameo, 16:08:40)
    • ACTION: obedmr to add some github wiki pages documenting the design and usage of ansible deployment for ciao. (sameo, 16:09:41)
    • ACTION: leoswaldo to improve swagger on its own fork and try to push upstream (sameo, 16:26:09)
    • ACTION: ciao to use leoswaldo's swagger fork until PRs get merged. (sameo, 16:26:41)
    • LINK: https://github.com/01org/ciao/issues/334 (obedmr, 16:29:09)
    • mcastelino, obedmr and fuentess will work together on single VM for travis (sameo, 16:29:43)
    • LINK: https://github.com/01org/ciao/pull/425 almost looks good to me but will force us into changing the configuration file format. Since it's not a sprint 2 issue, should we postpone merging it ? (sameo, 16:31:20)
    • ACTION: sameo to push PR #425 and fix all possible config files out there. (sameo, 16:35:07)
    • ACTION: sameo drop an email to the mailing list when merged (tcpepper, 16:35:25)
    • ACTION: mrkz to continue working on PR #425 and check for duplicated entry (sameo, 16:40:50)
    • ACTION: mrkz to add the subnet slices comparison routine to the networking package. (sameo, 16:41:07)

  • bugs (sameo, 16:42:22)

  • P1 bugs (sameo, 16:42:37)

  • prior week action items (tcpepper, 17:07:03)

  • anything else? (tcpepper, 17:07:56)

Meeting ended at 17:09:29 UTC.

Action Items

  • obedmr to present ansible/docker during the f2f week.
  • obedmr to add some github wiki pages documenting the design and usage of ansible deployment for ciao.
  • leoswaldo to improve swagger on its own fork and try to push upstream
  • ciao to use leoswaldo's swagger fork until PRs get merged.
  • sameo to push PR #425 and fix all possible config files out there.
  • sameo drop an email to the mailing list when merged
  • mrkz to continue working on PR #425 and check for duplicated entry
  • mrkz to add the subnet slices comparison routine to the networking package.
  • any failing travis test, please copy/paste the failure details into Issue 434
  • tcpepper to notify mailing list about short term ask for reporting travis failures.
  • tcpepper to try running travis on a 2 weeks old ciao

Action Items, by person

  • leoswaldo
    • leoswaldo to improve swagger on its own fork and try to push upstream
    • ciao to use leoswaldo's swagger fork until PRs get merged.
  • mrkz
    • mrkz to continue working on PR #425 and check for duplicated entry
    • mrkz to add the subnet slices comparison routine to the networking package.
  • obedmr
    • obedmr to present ansible/docker during the f2f week.
    • obedmr to add some github wiki pages documenting the design and usage of ansible deployment for ciao.
  • sameo
    • sameo to push PR #425 and fix all possible config files out there.
    • sameo drop an email to the mailing list when merged
  • tcpepper
    • tcpepper to notify mailing list about short term ask for reporting travis failures.
    • tcpepper to try running travis on a 2 weeks old ciao
  • UNASSIGNED
    • any failing travis test, please copy/paste the failure details into Issue 434

People Present (lines said)

  • tcpepper (76)
  • sameo (62)
  • leoswaldo (23)
  • mcastelino (13)
  • mrkz (12)
  • obedmr (8)
  • ciaomtgbot (3)
  • albertom (3)
  • _erick0zcr (2)
  • fuentess (1)
  • ciaoslackbridge (1)
  • carlosag_ (1)

Generated by MeetBot_ 0.1.4

.. _MeetBot: http://wiki.debian.org/MeetBot

###Full IRC Log

16:01:48 <sameo> #startmeeting
16:01:48 <ciaomtgbot> Meeting started Thu Aug  4 16:01:48 2016 UTC.  The chair is sameo. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:48 <ciaomtgbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:01:55 <mrkz> o/
16:01:57 <sameo> tcpepper: sure.
16:01:58 <leoswaldo> o/
16:02:01 <sameo> o/
16:02:02 <tcpepper> o/
16:02:55 <mcastelino> o
16:03:32 <_erick0zcr> o/
16:03:37 <albertom> o/
16:04:38 <carlosag_> o/
16:04:53 <obedmr> o/
16:05:41 <sameo> #topic Opens
16:05:48 <sameo> Let's start with opens...
16:06:02 <leoswaldo> I have one, regarding docs, let me know when I can share
16:06:13 * tcpepper has one: ansible and docker with ciao...we should have obedmr present to us the design/implementation/usage
16:06:50 <obedmr> tcpepper: yes, we're planning it for next week
16:07:49 <obedmr> albertom: will be presenting the Ansible stuff, and I will be presenting the docker thing
16:07:51 <mcastelino> another open... can we have status of moving obedmr single machine tests to travis
16:07:54 <sameo> tcpepper: Are you thinking about a more open forum for that ?
16:08:03 <tcpepper> obedmr: we have a lot of people on vacation next week. perhaps it should go to the agenda for during the f2f
16:08:16 <obedmr> tcpepper: a ok, sounds good to me
16:08:35 <tcpepper> sameo: public documentation of it is good yes.  but more what I want is to get design/implementation/usage documentation asap so I can actually use it.
16:08:40 <sameo> #action obedmr to present ansible/docker during the f2f week.
16:08:52 <tcpepper> but at this point I wont be using it for a few weeks (ie: vacation and the f2f get in the way)
16:08:57 <sameo> tcpepper: Agreed.
16:09:41 <sameo> #action obedmr to add some github wiki pages documenting the design and usage of ansible deployment for ciao.
16:09:50 <sameo> leoswaldo: Please share
16:10:00 <leoswaldo> this week, I looked into swagger to automate the documentation of APIs, I made a basic small test, and found that swagger is very powerful however it generates sections we don't need, and for what I saw there is no way to tell swagger to not generate them, it generates them empty, as "table of contents" with a list of all api calls, which later are defined, and a specific section for details of EACH call, even if you only specify ge
16:10:00 <leoswaldo> neric data
16:10:00 <leoswaldo> swagger can generate markdown and other formats, but does not look very nice, the results of the small POC I made can be found in http://paste.openstack.org/show/549254/
16:10:02 <leoswaldo> you can see there are empty sections, which I don't like how they look, neither the format
16:10:04 <leoswaldo> so till today, for what I can see we have three options,
16:10:06 <leoswaldo> 1 - use swagger and made a script that modifies the result, which I don't like the idea of making a "patcher"
16:10:08 <leoswaldo> 2 - there are other solutions which automate the API documentation, we could invest more time on that
16:10:10 <leoswaldo> or
16:10:12 <leoswaldo> 3 - make a script that does what we exactly need
16:10:14 <leoswaldo> from my opinion I would go for option 3, I know it may be to re-invent the wheel, but we would have solution resolves what we need
16:10:16 <leoswaldo> any thoughts ?
16:11:56 <tcpepper> is there something we can do with godoc to get cleaner results?
16:13:20 <leoswaldo> I read godoc is a bit more about flying through the code, rather than generating tables/sections
16:14:04 <sameo> leoswaldo: It does generate sections for you, not sure about tables.
16:14:17 <tcpepper> hmm.  I suppose.  maybe I'm too close to the code, but I treat godoc like api documentation
16:15:10 <tcpepper> but as you note https://godoc.org/github.com/01org/ciao/ssntp for example is more function names/args/usage in a code way versus what one might normally see for an API doc like http://developer.openstack.org/api-ref-guides/bk-api-ref.pdf
16:15:55 <sameo> tcpepper: Yes, godoc doesn't really work for a rest API documentation.
16:16:21 <sameo> leoswaldo: Whic other options did you look at ?
16:17:21 <leoswaldo> there is another option like test2go , but its documentation under unit testing
16:18:37 <sameo> leoswaldo: Which input would any potential tool take ?
16:19:01 <tcpepper> is there potential for option 1b:  enhance swagger internally (ie: not external post-processor script), propose changes upstream
16:19:06 <leoswaldo> somthing like  https://github.com/yvasiyarov/swagger/wiki/Declarative-Comments-Format
16:20:34 <leoswaldo> we could send PR regarding modifications for it. but for what I've seen in the issues of it. the maintainers are not take care of the tool that frequently
16:21:09 <tcpepper> well, it's on github and there's a "fork it on github" button
16:21:22 <tcpepper> could fork, modify, PR to the upstream, but use our fork until they maintain theirs
16:22:04 <leoswaldo> that would be another option
16:22:36 <sameo> leoswaldo: I agree with tcpepper
16:23:11 <tcpepper> given swagger's close, and your input would follow the https://github.com/yvasiyarov/swagger/wiki/Declarative-Comments-Format ...feels like a reasonable approach to contribute to them
16:23:12 <ciaoslackbridge> <> Pssst! I didn’t unfurl <https://github.com/yvasiyarov/swagger/wiki/Declarative-Comments-Format> because it was already shared in this channel quite recently (within the last hour) and I didn’t want to clutter things up.
16:23:17 <tcpepper> even if they're slow to pick up changes
16:23:39 <leoswaldo> ok, so I'll take a look in that option
16:23:40 <tcpepper> maybe even some other github fork of swagger has something incrementally close to what you want
16:23:57 <tcpepper> there's 75 forks on github
16:24:28 <leoswaldo> ok, so seems we agree for  the fork
16:24:43 <mrkz> leoswaldo: I could help you drop some lines too if needed
16:25:06 <leoswaldo> sure, thanks
16:26:09 <sameo> #action leoswaldo to improve swagger on its own fork and try to push upstream
16:26:41 <sameo> #action ciao to use leoswaldo's swagger fork until PRs get merged.
16:27:37 <sameo> mcastelino: You had an open as well ?
16:27:47 <leoswaldo> thanks team
16:28:05 <mcastelino> obedmr and I will work with fuentess to close on that
16:28:14 <obedmr> yes
16:28:50 <obedmr> we'll include BAT testing into Travis with Single VM functionality testing script
16:29:09 <obedmr> #link https://github.com/01org/ciao/issues/334
16:29:43 <sameo> #info mcastelino, obedmr and fuentess will work together on single VM for travis
16:30:04 <sameo> I also have an open if there are no other ones.
16:30:18 <mrkz> sameo: please go ahead
16:31:20 <sameo> https://github.com/01org/ciao/pull/425 almost looks good to me but will force us into changing the configuration file format. Since it's not a sprint 2 issue, should we postpone merging it ?
16:31:37 <mrkz> I'
16:31:57 <mrkz> m ok with not taking it atm, I could rebase with no problem :)
16:32:15 <tcpepper> I vote for make the breaking change asap and move forward
16:32:41 <tcpepper> imho: we're early, unstable now.  easier to make these changes now versus later.
16:32:45 <sameo> tcpepper: I'm all for that, but I don't want to disrupt sprint2 progress.
16:32:48 <mrkz> only issue I have there is the duplicate function which I'm pondering pushing it into testutils package, but would prefer feedback here :)
16:33:08 <tcpepper> sameo: the disruption would be editing six config files maybe in the world?
16:33:21 <sameo> tcpepper: Pretty much.
16:34:34 <tcpepper> mcastelino: what do you think?
16:34:44 <sameo> tcpepper: I'll push once we clarify a few things on the PR. And will fix our cluster config files. And the single VM one, I guess.
16:34:44 <mcastelino> we should do it now...
16:35:07 <sameo> #action sameo to push PR #425 and fix all possible config files out there.
16:35:18 <mcastelino> we have several auto config issues we need to fix anyway in our test infrastructure that we need to address anyway
16:35:25 <tcpepper> #action sameo drop an email to the mailing list when merged
16:35:32 <sameo> tcpepper: Sure thing.
16:35:52 <sameo> mcastelino: There was a pending question for you on PR #425, if you have a minute.
16:36:23 <mcastelino> yes addressed it
16:36:34 <mcastelino> the order does not matter. you do not need to sort or even check overlaps
16:36:46 <mcastelino> I will handle it properly internally
16:36:59 <mcastelino> all they need to be is valid subnets
16:37:22 <sameo> Ok, so to compare 2 slices we need to sort them first.
16:37:54 <sameo> mrkz: I don't think duplication is a major issue here.
16:38:33 <sameo> mrkz: If anything it could be added somewhere in the libsnet package.
16:38:49 <sameo> It really compares subnets slices.
16:38:58 <mrkz> sameo: however should I take care of cleaning duplications? (e.g: subnet is specified twice)
16:39:42 <mcastelino> mrkz: I will handle it properly internally.. even if there are duplicates... but if you want to check a priori it does not hurt
16:39:56 <sameo> mrkz: You could, yes.
16:40:26 <mrkz> ok then, I'll do, thanks for the feedback :D
16:40:50 <sameo> #action mrkz to continue working on PR #425 and check for duplicated entry
16:41:07 <sameo> #action mrkz to add the subnet slices comparison routine to the networking package.
16:41:18 <sameo> No more opens ?
16:41:31 <mrkz> not from me :)
16:41:57 <sameo> Let's move on then.
16:42:22 <sameo> #topic bugs
16:42:37 <sameo> #topic P1 bugs
16:42:55 <sameo> https://github.com/01org/ciao/issues/434
16:43:02 <tcpepper> I know this is an ambiguous bug, but...something's happened
16:43:13 <tcpepper> I'm not sure if travis changed or us
16:43:24 <tcpepper> we've certainly got some controller races, latent since april/may
16:43:24 <sameo> tcpepper: Are the failures consistent ?
16:43:26 <mcastelino> tcpepper: does this show up on tip
16:43:31 <mcastelino> or both go versions
16:43:35 <tcpepper> but in the past week or so we're getting a looooot more failures
16:43:53 <tcpepper> it's 50:50 usually whether it's 1.6 or tip go job that fails
16:44:05 <tcpepper> and almost 100 0f you restart the job, it will pass
16:44:14 <tcpepper> the specific unit test that fails is seemingly random
16:45:04 <tcpepper> sometimes it will be 1, sometimes 10, sometimes in controller, sometimes launcher, etc.
16:45:40 <sameo> tcpepper: have we run the race detector on controller recently ?
16:45:51 <tcpepper> i have some yes
16:46:21 <tcpepper> #link https://github.com/01org/ciao/issues/83
16:46:57 <sameo> Ouch.
16:47:13 <tcpepper> controller compute_test.go and controller_test.go definitely have races
16:47:27 <tcpepper> I _think_ the way we do pagination in the datastore is racey too
16:47:51 <mcastelino> tcpepper: I also saw some errors about corrupted block device... have you seen those?
16:48:03 <sameo> pagination in the datastore ? where ?
16:48:14 <tcpepper> the pagination thing might explain the "sometimes start 2 instances only starts 1" bug
16:48:51 <tcpepper> i suppose it's pagination in the cli and webui, relative to the datastore?  I've not dug into how this is implemented
16:49:15 <sameo> we do pagination in the compute.go implementation.
16:49:42 <sameo> tcpepper: do you suspect there's a race there ?
16:50:09 <tcpepper> I do know the datastore's "instances" doesn't have protection against concurrent writes
16:50:35 <tcpepper> my worry is that if it shifts around due to a race, that compute.go implementation could be referencing non-existant stuff
16:50:57 <tcpepper> mcastelino: I'm not sure if I've seen the corrupted block device one
16:51:14 <tcpepper> I'm also realizing that travis when you re-run, doesn't appear to keep a log of the prior run?
16:51:23 <tcpepper> so...
16:51:29 <sameo> tcpepper: I'll look at it, but this is an aspect that I tried to take into account when implementing it on compute.go.
16:51:44 <tcpepper> imho we need to start a concerted data collection effort and race finding hunt
16:51:59 <tcpepper> as we aim to implement a major feature in the coming weeks with storage, we need base stability
16:52:48 <sameo> Sure. Maybe this could be a workstream during the f2f ?
16:52:53 <tcpepper> yeah
16:53:00 <tcpepper> my other worry here is the travis side
16:53:10 <tcpepper> we know they change stuff under us
16:53:14 <sameo> #info add a race hunt stream to the f2f.
16:53:16 <tcpepper> which is good...it breaks our assumptions
16:53:32 <sameo> yes, we should be robust against that.
16:53:44 <tcpepper> there may be something though we're pulling in implicitly from the travis side that is actually broken
16:54:51 <tcpepper> so my short term ask:
16:54:52 <sameo> tcpepper: Would it make sense to build a branch with all controller tests disabled and see if we get a better travis stability ? To at least isolate a component ?
16:55:10 <tcpepper> #action any failing travis test, please copy/paste the failure details into Issue 434
16:55:37 <sameo> tcpepper: Can you send an email to the list about that ?
16:55:43 <tcpepper> will do
16:56:01 <tcpepper> as for the branch, the failures in travis were frequently not in controller unit tests
16:56:08 <sameo> fair enough.
16:56:14 <tcpepper> which is what made me wonder about the travis side
16:56:23 <tcpepper> we may have some random broken ubuntu thing under us
16:56:26 <tcpepper> which is ... hard to find
16:56:40 <sameo> #action tcpepper to notify mailing list about short term ask for reporting travis failures.
16:57:28 <sameo> tcpepper: travis running on an LTS Ubuntu, and a pretty old one, I would be surprised if something random broke under us.
16:58:08 <mrkz> shall we continue with the bug scrub?
16:58:14 <tcpepper> another thought...we've re-vendored a few times in the past weeks right?
16:58:33 <sameo> tcpepper: yes, mostly gophercloud afaik.
16:59:06 <sameo> tcpepper: we could also git bisect ciao as a whole on a separate branch.
16:59:29 <sameo> or at least revert to a 2 weeks old tag and see if travis fails so much.
16:59:35 <tcpepper> yeah that's an idea
16:59:54 <sameo> Mind assigning it to you tcpepper ?
17:00:26 <tcpepper> sure
17:00:33 <tcpepper> done and email sent to list also
17:00:49 <sameo> #action tcpepper to try running travis on a 2 weeks old ciao
17:00:59 <sameo> I have to run to another meeting I'm afraid...
17:01:20 <mrkz> sameo: you could assing other chairs for the meeting with #chair <nick>
17:01:31 <tcpepper> I can finish things off
17:01:37 <sameo> #chair tcpepper
17:01:37 <ciaomtgbot> Current chairs: sameo tcpepper
17:01:48 <sameo> tcpepper: Thanks.
17:02:02 <tcpepper> we spent a lot of time on opens and this P1, let's try to wrap up:
17:02:12 <tcpepper> the list of P2's is mostly unchanged
17:02:15 <tcpepper> but we have one new one:
17:02:23 <tcpepper> #link https://github.com/01org/ciao/issues/433
17:03:00 <tcpepper> I set this to P2 and arguably might be higher.  Looks like it lets a tenant DoS the keystone server via ciao-cli
17:03:26 <tcpepper> one ciao-cli is able to chew up about one cpu on the keystone server
17:03:51 <tcpepper> it's a misconfiguration of course on the client side, but the ciao-cli code should gracefully handle it
17:04:29 <tcpepper> albertom, _erick0zcr: do either of you have the bandwidth to look at it?
17:04:34 <albertom> yes
17:04:39 <albertom> just add steps by step to reproduce
17:04:48 <_erick0zcr> sure
17:04:56 <tcpepper> thx
17:05:52 <tcpepper> ok, then the prior week's todo list
17:06:02 <tcpepper> I've got 4 in the agenda at
17:06:06 <tcpepper> #link https://github.com/01org/ciao/wiki/Weekly-Meeting-2016-08-04
17:06:22 <tcpepper> based on what I saw I filled status for 1,3,4, but the 2nd one...
17:06:31 <tcpepper> fuentess: can you comment on test plan for vol attach/detach
17:07:03 <tcpepper> #topic prior week action items
17:07:28 <fuentess> tcpepper: havent had the chance to work on that, I will create an issue and document the test plan there
17:07:40 <tcpepper> excellent thx!
17:07:56 <tcpepper> #topic anything else?
17:08:13 <mrkz> not from my side
17:09:17 <leoswaldo> none from my side
17:09:22 <obedmr> none from my side
Add a custom footer

Meetings

Development

Architecture

Usage

HTTPS API

Clone this wiki locally