Split out metriks into multiple gems to reduce dependencies

[eric]

In an attempt to try to increase the usage of metriks and make it easier to integrate, @josephruscio has suggested that the reporters be split into separate gems (or maybe their own gems).

This would mean the core metriks gem would just contain:

• the metric building blocks (Timer, Meter, etc)
• the registry
• a reporter base class? (if we ever have such a thing)

Reporters

We would need to decide if it made sense to have a single metriks-reporter gem that contains all of the reporters, or if it would make sense to have separate gems for each reporter. The benefit of separate gems is the reduction in dependencies that would be required, which seems like a nice gain.

Naming and upgrading

My first thought is to call the core gem metriks and have the reporters in their own gems. The downside of this is that anyone who is upgrading metriks will have their code break, as it will no longer contain any of the reporters. I'm not sure what the best way to solve this is a smooth upgrade path, so I'm open to suggestions here.

CC: @lmarburger, @josephruscio, @nextmat

[lmarburger]

Great idea. On the surface I think it makes sense to have individual gems for each reporter. Then using service-specific gems like librato-metrics to aid in reporting would be fair game because they'd only be installed when actually needed.

What about including the Logger reporter in core metriks? Maybe that becomes the base logger?

My thought on upgrading would be to deprecate the current set of reporters when upgrading 1.0 (or 0.10) and remove them completely in 1.1 (or 0.11). That leaves the 0.9 line free for bug fixes. It'd be great to upgrade from 0.9 and have metriks continue to work as expected except with warnings. On the flip side, I can't say I would ever see these deprecation messages. I only use the Librato reporter on production so I could imagine upgrading to 1.0 and then 1.1 and only noticing when things broke hard.

[josephruscio]

@lmarburger @eric I feel pretty strongly that for broadest adoption there should be a minimal set of functionality in the core metriks gem, meaning only:

• the metric building blocks (Timer, Meter, etc)
• the registry

I think the metriks-reporter gem could contain the base reporter and probably the stdout logger at a minimum. The only open question is whether it would also include the other specific reporter implementations, or if other gems would have those. /cc @nextmat

[eric]

I'm leaning towards the option of ripping out the reporters for the next release and having a big <h1> "READ THIS IF YOU'RE BROKEN".

...though I'm wondering if it would cause too many people to have code that breaks at startup.

These sorts of decisions are hard.

[josephruscio]

@eric the issue is more that people upgrading unawares will lose the reporting piece, so the broken part would be that the metrics stop flowing to their eventual destination? There shouldn't be any exceptions in their running code and as long as you rev the major version number, correctly packaged apps shouldn't be affected?

Assuming the above is the case, semantic versioning is supposed to enable breaking changes like this. I don't see a way around it unless you make a new gem like metriks-core, which seems undesirable for other reasons.

[eric]

The problem I see is if we remove the reporters from the metriks gem, you'll get an exception when you try to instantiate the reporter because the class won't be found.

If you had this in your rails initializers (as I do), bad things could happen.

[jnunemaker]

I would definitely split them out. I'd have all the core stuff in metriks and then have a gem for each reporter (metrics-reporter-liberator, metriks-reporter-log, etc.). I do this with most of my gems that are based on adapters (ie: adapter, adapter-mongo, adapter-redis, flipper, flipper-mongo).

There is no reason to install the redis gem for adapter-mongo and vice versa. Same holds true. Just my two cents. I wouldn't have more than 1 gem for the core parts though.

[nextmat]

The other option would be to take the extracted portion and name that something new. Make that a dependency and allow metriks to continue to be a full-stack option without breakage for existing users.

Either way my preference would be that the gem that includes the metric types and local storage not include anything having to do with reporters or logging. I think that mixes the concerns and makes the implementation less broadly useful since adoption then requires the user or integrator to agree with your choice/style of reporting/logging, etc. In my mind there is a pretty strict separation of concerns between local aggregation and reporting/transfer to another system/store.

If there are concerns about being able to instrument this core gem I'd propose an abstract set of hooks that if unimplemented do nothing but allow gem users to sign up for instrumentation events, etc.

[nextmat]

FWIW I would also steer clear of the -core naming scheme if you are aiming for the broadest possible adoption since the name then infers that users have to use it with other related libraries. If the goal is to provide a simple implementation of the core that folks can build other things on I think a simple name (whether that is metriks or something else) that isn't associated with a network of related gems is a better options. -core just makes me thing of merb and that I'm about to install a million gems.

Seems like you could make metriks the core thing and give a new name to the full-stack option with plug-in adapters or vice versa but I'd make them feel discrete from each other, branding-wise.

[eric]

Here's my first attempt at extracting a reporter gem: https://github.com/eric/metriks-graphite

[jnunemaker]

@eric I think that works.

[josephruscio]

@eric looks good. I think this demonstrates that there's not a ton of value in embedding some "base" reporting capabilities in the core metriks gem versus (per @nextmat's and my earlier comments) the potential negatives.

[eric]

@josephruscio: I'm not sure if I quite agree in the longer term — there are a lot of things that ended up being repeated (and a lot of niceties in the librato-rails project related to forking, etc) that would be relevant for any kind of reporter.

That said, until that stuff is standardized, I don't think there's a lot of value in having a base class just to have one.

[bernd]

@eric I just saw your tweet wrt Monitorama and this ticket. I'm at Monitorama and would like to help with that.

What's your plan regarding all the repositories? Do you want to create a metriks github organization where all metriks related repositores (metriks, metriksd, metriks-reporter-graphite, ...) live? I think that would be nice because it's easier to find the related libraries.

[eric]

I hadn't made any concrete decisions about things like github orgs, etc. I was waiting until I actually had a chance to do the legwork on the code first, but I could see it being a helpful thing to have.

[bernd]

I started some of the legwork based on your metriks-graphite repository.

http://dl.dropbox.com/u/12671828/metriks-legwork.zip