Thoughts on regulations-site

[ascott1]

Currently, regulations-site is a CFPB branded Django app. It's the result of a process where we were rapidly testing ideas and re-designing on the fly. This means that there are shards of code left where they shouldn't be and overall the app feels a bit messy.

As 18F expands regulations-site there is a need for it to be re-branded for various agencies. The current solution for this is to fork a version of reg-site. This works ok, but means that it's hard to contribute back to the original project.

Proposal

github.com/eregs/regulations-site could house a new and stripped down version of regulation site. This would contain minimal templates, styles, and JavaScript to produce a working regulations-site when paired with regulations-core. The output would look something like a wireframe.

Based on this frame of a site, individual organizations would be able to create themes that would contain template, style, and JS overrides (or complete replacements). This would allow organizations to style their own versions of eRegs however they'd like while also being able to use the base site.

Open questions

• Is Django the right format for this? Could it be a static site generator? Something else?
• Once of the nice things about the current setup is that eRegs is both a single page application as well as a Django app that works without JavaScript. Is this something we would like to preserve? If so, how else might we best accomplish this?

[cmc333333]

I'm in favor of the outcome -- a wireframe version of regulations-site, but against a rewrite. I think it'd be a much better use of our time to move to this goal incrementally, starting with the CFPB-specific style sheets and slicing them off piece by piece.

Right now we don't have good technology to get to either approach -- for ATF, we have a script which effectively

1. copies all of the relevant styles into a directory
2. replaces any that need ATF-specific modifications
3. builds the CSS

This is pretty hacky -- none of the Less files "know" they are being overridden, if that makes sense. The orchestration all happens in Python land. If we had a better strategy for this, I think we could start picking off CFPB specific bits and be on our way to a generic solution.

Is Django the right format for this? Could it be a static site generator? Something else?

I think this is an orthogonal question. Whether or not we switch to a static site generator or something else doesn't really affect our frontend build system much. For the moment, django's template overrides work out just fine.

Once of the nice things about the current setup is that eRegs is both a single page application as well as a Django app that works without JavaScript. Is this something we would like to preserve? If so, how else might we best accomplish this?

I think we absolutely want to preserve this, but again, it seems orthogonal towards the goal of creating a unified base which can be themed.

[ascott1]

Here's a summary of how I think this might work (moved from a chat conversation):

• a base set of CSS and JS exists as an npm package and is used in the build of the default reg-site
• a theme could choose to import those base styles and extend them or alternatively not use them at all

That’s just one idea, but would allow the developer the choice of how they’d like to work with those styles and avoid the potential!important pitfalls

[grapesmoker]

To me a rewrite makes sense for a number of reasons.

First, while I don't think it matters a great deal whether we use Django as a backend or not, the current way that we're using it is probably suboptimal. There's a ton of rendering of partials happening on the backend, in addition to a great deal of app logic that modifies the data extracted from the db. I think the correct way of doing this is to use the backend only for storage and render on the front-end; that's why we have things like Backbone, is so we can implement this sort of separation of concerns. Additionally, if we do it this way it should be easier to encapsulate the differences in appearance between different nodes in the tree; right now, the logic for doing this is very tightly coupled to the tree labels.

Second, our API should ideally just be rolled into the main site project. Right now, in order to deploy eRegs, you need to deploy three separate components and god help you if you need t ochange anything about the logic of any one of them because then you'll need to make sure your changes don't affect anything further down the line. Just adding an extra field is a real pain, and it shouldn't be. As I've said before, I think a JSON-based db like Mongo would serve us much better here than MySQL.

[cmc333333]

If we had infinite resources, a rewrite would make sense, but I don't see a good business case here. What does a full rewrite get us that an incremental upgrade doesn't? I think we can make all of the improvements you are suggesting without pausing progress for an extended period of time.

I don't understand the desire to render everything on the front end. We're largely displaying text, after all, why not continue to use progressive enhancement? This approach is what's allowed the app to work seamlessly on older browsers, any page to be bookmarked without special logic, and every page to be crawl-able by search engines. I'd argue it's also what will make creating a static version of the site feasible a feat which would be much more difficult if all of the logic lived in Backbone. I think the pain around Backbone actually comes from the other direction, the assumption that Backbone should "drive" the app rather than relying on the backend for the proper data.

I definitely agree that we should rework our templates. Right now, most are very aware of their context rather than being introspective -- we have lots of special case templates which are selected by the view code rather than having general purpose templates. I think Python should only provide the tree structure and the template should determine how to render it.

Regarding separate components, I think your work on defining a schema will help a lot. If we had a single schema, we could also create a shared library which would prevent a lot of duplication and synchronization issues. On the separation between -core and -site specifically, the original hope was that they'd evolve independently. Given that we only have one client for the API, though, that never really panned out :)

On MongoDB, -core is already set up to allow different data storage systems. This currently allows you to switch between Elastic Search and Django's ORM. That said, we're currently just serializing the JSON blob and serving it up by key; I'm not sure what new goals doing this would Mongo would accomplish.

[grapesmoker]

So I think I probably didn't do a good job of explaining what I'm after
here. I'm going to try again with a concrete example.

Today, Adam and I had a chat about styles in regulations. One thing that
came up when I was looking at an interp that I thought was rendered
incorrectly is that apparently the logic of the rendering is essentially
based on the structure of the label. In other words, "1030-2-a-Interp-1"
does not display equivalently to "1030-2-Interp-a-1." Now, from one
standpoint that's the right thing to do, because those represent
semantically different structures within the regulation. But on the other
hand, this is the kind of thing that renders the whole process fragile; if
the parser makes some mistake about labeling during parsing, even if the
nested structure of the tree is the same, you have misrendered text.

This is a pretty minor example and trivially fixed (especially with the new
pipeline, it took me about 15 seconds after understanding what the problem
was to rerun the XML parser and push to demo!), but there are lots of
things like this, where the way that the text is rendered to the user is
dependent on some complicated logic deep within regsite and its template
structure. And each time you need to change something, you have to be
cognizant of modifying the logic in multiple places or you'll end up with
the wrong display.

I think this is a serious problem for expanding the user base for eRegs.
Right now, if you have a problem, the only way to fix that problem is to do
a deep dive into three different pieces of code (parser, core, site), make
sure there are no inconsistencies between them, and then make sure your
template logic didn't confuse one thing for another right now. This is a
consequence of visual representation being very tightly coupled to the
overall semantic representation. Of course you want some coupling because
obviously the representation of a node should depend on its place within
the tree hierarchy, but it should be controllable in a straightforward
fashion. For example, I should be able to write something like <title display-style="paragraphHeader"> vs. <title display-style
"subParagraphHeader"> and have them render differently. Right now, I can't
do that without adding a lot of complicated logic in the templates. Most of
the time, the templates do the right thing, but when they don't, it's never
as simple as just changing an XML attribute somewhere, and it should be
that simple. I'm not necessarily saying that everything must be rendered
on the front-end (although it grates my soul that "because IE8" is still
something we need to think about), but I would like to see the rendering
system simplified.

With regard to a static site, I actually think it would be a huge
improvement! Static would be great; all you really need (in theory) is an
XSLT that eats the XML and spits out static HTML. It would be great and
also totally customizable in just the way I'm suggesting above. So I'm
totally on board with efforts to make this happen. The downside of this is
of course that you don't get the search and API that comes with having a
real storage system, but on the other hand, if you have an emergency at the
level of "I NEED TO STAND UP EREGS TODAY," a static site would go a long
way to making that happen.

As for storage, I think it's a secondary issue to the app complexity. I do
think that Mongo's document-focused store is spiritually closer to what
we're doing than a relational db. It's also more flexible, in that if you
change something about the structure of the node like adding a field, you
don't then have to root around in the db to make sure you didn't break
constraints. It's a minor issue in the grand scheme of things, but I think
it would be nice. It might well not be worth the effort needed to make it
happen.

I hope this provides a better illustration of the kind of issues I think
are facing future eRegs development. I'm certainly not opposed to
incremental improvements, but is "refactor the whole templating system"
really an incremental improvement? Like, if that's what we need to call it,
I'm not hung up on the term; it just seems like a pretty serious
undertaking that isn't just moving a few pieces around. It might end up
amounting to an effective rewrite in the end anyway.

On Tue, Dec 1, 2015 at 12:27 PM, CM Lubinski [email protected]
wrote:

If we had infinite resources, a rewrite would make sense, but I don't see
a good business case here. What does a full rewrite get us that an
incremental upgrade doesn't? I think we can make all of the improvements
you are suggesting without pausing progress for an extended period of time.

I don't understand the desire to render everything on the front end. We're
largely displaying text, after all, why not continue to use progressive
enhancement? This approach is what's allowed the app to work seamlessly on
older browsers, any page to be bookmarked without special logic, and every
page to be crawl-able by search engines. I'd argue it's also what will make
creating a static version of the site feasible a feat which would be
much more difficult if all of the logic lived in Backbone. I think the
pain around Backbone actually comes from the other direction, the
assumption that Backbone should "drive" the app rather than relying on the
backend for the proper data.

I definitely agree that we should rework our templates. Right now, most
are very aware of their context rather than being introspective -- we
have lots of special case templates which are selected by the view code
rather than having general purpose templates. I think Python should only
provide the tree structure and the template should determine how to render
it.

Regarding separate components, I think your work on defining a schema will
help a lot. If we had a single schema, we could also create a shared
library which would prevent a lot of duplication and synchronization
issues. On the separation between -core and -site specifically, the
original hope was that they'd evolve independently. Given that we only have
one client for the API, though, that never really panned out :)

On MongoDB, -core is already set up to allow different data storage
systems. This currently allows you to switch between Elastic Search and
Django's ORM. That said, we're currently just serializing the JSON blob and
serving it up by key; I'm not sure what new goals doing this would Mongo
would accomplish.

—
Reply to this email directly or view it on GitHub
https://github.com/eregs/eRegulations/issues/8#issuecomment-161040275.

http://www.google.com/profiles/grapesmoker

[theresaanna]

I have some related thoughts, and have put them in a gist, and you can feel free to comment there, here, wherever seems appropriate.

From Slack:
"I wrote up the picture I have in my brain of platform/plugin things. it needs merging/improving by your input, @ascott1, @cmc333333 ’s and whoever else wants to talk practical details.
@jbarnicle, @jehlers and I came to a place where it made sense, on the theming stuff, to work independently on pieces and then come together to merge everyone’s work. I realize that we started out that way, too, @ascott1 - collaborating on markup and such. also, if no one else gets it into writing, I wanted to make sure I do, so that there’s something to start with"

Here it is:
https://gist.github.com/theresaanna/34f8006f929c80f6327c

[ascott1]

I'm trying to go through the gist and leave comments where appropriate. I have a lot of thoughts for how this might be approached from a practical standpoint and it seems that we're more or less on the same page. 👍