Evaluate replacing the AppManifest with package.json

[brianbaker]

It has been mentioned a few times across various threads to switch the AppManifest to use package.json. I think the best time to make such a switch would be for v2. I'm opening this thread to track pro/con discussion surrounding switching to package.json.

https://github.com/npm/npm/blob/master/doc/files/package.json.md
http://wiki.commonjs.org/wiki/Packages/1.1
http://package.json.nodejitsu.com/

[montlebalm]

I think package.json is a red herring. A config object of some sort is helpful, but an actual package.json is an inappropriate alternative to AppManifest. Reasons:

• package.json files are intended to be static. They describe installation, not instance configuration.
• Because they're server configs, most of the available fields don't apply to client-side apps.
  • e.g., "devDependencies", "scripts", "preferGlobal", "bin", "analyze", "engines", etc
• It looks like there are actual specs for the file that I believe most apps would fail to satisfy. We could choose not to follow the spec, but then why choose package.json?

Instead, I think we should simply rectify the shortcomings of the existing AppManifest. It simply fulfills a wholly different purpose than package.json, however, I'm open to being wrong.

Here are a few things I think AppManifest is lacking today:

• Versioning: this ties into the AppConfig as well, but it would be nice to have some structure in place to support versioning.
• Dependency aliasing: allow apps to specify their scripts with a friendly name (e.g., "jquery-1.10.2") and provide a fallback url. This would help containers prevent duplicate script loading.

scripts: {
  "jquery-1.10.2": "http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"
}

• Name & Description: I think it might be helpful to have the desired name and description come back in the Manifest in case they're dynamic. Currently, the container has to hard code those, which could be undesirable in some situations.

[ilinkuo]

package.json files are intended to be static

See my comment as to why that's actually a desirable characteristic.

most of the available fields don't apply to client-side apps

Agreed, Most don't apply. But I think if you sample a lot of npm projects, you'll find that most package.json are minimal and don't contain a lot of those values either.

actual specs for the file that I believe most apps would fail to satisfy

I'm no expert on package.json, but I thought the only required elements were name and version?

For me, the immediate benefit of switching over to package.json is divorcing the dynamic parts of the AppManifest, enabling it to be cached and to be managed by AMD.

Another immediate benefit is the ability to plug into the npm ecosystem. A private npm registry can be created via npm-registry and it's immediately searchable. As a developer, I would actually prefer this interface over F2's registry.

It would take a little research and work, but I think F2 could leverage npm's version compatibility check instead of trying to write its own.

A longer term benefit would be a better leveraging of gruntjs and package.json as an sdk to set up a local F2 dev environment.

[markhealey]

A private npm registry can be created via npm-registry and it's immediately searchable. As a developer, I would actually prefer this interface over F2's registry.

Very interesting idea, @ilinkuo. We spent about a month exploring the F2 registry using CouchDB in early 2013. We ultimately decided against it because we couldn't add what we deemed appropriate relationships and authentication to the JSON documents for secure cross-organization reuse. Private registries cloned from the master using Couch is a trivial task so it might be worth exploring that again if F2 ends up shifting to a package.json format for the AppManifest.

For reference, here are the fields captured in the current (beta) Registry on a per-app basis. There are definitely gaps in the information captured today vs. fields defined in package.json.

[markhealey]

I stumbled on this earlier when looking around for package.json-style formats for client-side apps. It is an interesting approach worth considering.

http://financial-times.github.io/ft-origami/docs/syntax/origamijson/

[brianbaker]

I added another link to npm's documentation of package.json in the original comment.

Our present usage of "scripts" would conflict with how it is intended to be used in package.json which is more for server-side functionality. Also, in looking towards v2, I've thought about allowing apps to specify additional paths, but at least with CommonJS 'paths' is a reserved word. I've also seen it suggested that you should use underscores or other characters for custom fields to prevent any conflict in the future.

[markhealey]

Are we talking about the literal use of package.json or a format based on that standard?

[brianbaker]

I believe most of the past comments were wanting the literal use of package.json. At this point, I lean towards a format based on the standard (in other words, JSON, like it is today). Maybe that just means we stop calling it the "AppManifest" and instead call it manifest.json. I think the next point of discussion would be whether to put into the spec whether the manifest should be static or whether it can be dynamic like it is today.

[ilinkuo]

@markhealey
The origami link is very interesting. It's quite ambitious as they cover web service standards too, as well as provide a build service. Some of their practices are ones that can be adopted by F2. I don't think too much of their origami manifest format, however.

There are a number of other projects good for looking at -- bower, yeoman, ender, volo, jam -- that have package.json extensions (I think, not sure).

[markhealey]

@ilinkuo I agree some of the Origami practices or methodologies would be useful if adopted by F2, in particular the web service standards as we begin to discuss F2 "Data Apps".

The Bower package.json looks very similar to the NPM package.

(I'm teasing this a little...)

What would be the advantages of a static F2 AppManifest over a dynamic one? Let's enumerate them.

[ilinkuo]

1. A static manifest can be cached by javascript using AMD. The second request of the app will load much faster.

2. A static manifest can be cached on the server side.

3. A static manifest can be cached by the browser, provided that the server is configured to properly handle ETags, last-modified headers.

4. A static manifest is server technology agnostic, It can be deployed on Apache instead of an app server, on any OS.

5. A static manifest can be distributed on a CDN

6. It's much easier to compute a SHA-1of a static manifest. The Container can use the SHA-1 to check whether an App's resources have changed.

7. A static manifest can be versioned, diffed, and source-controlled.

8. If an App is built with a static manifest and the only dynamic part served by a hosted webservice api such as Apigee or Parse, then the entire app may be source-controlled and versioned, and may be hosted on any server or distributed as a zip.

   ==== UPDATE =====

9. A dynamic manifest server that also services the javascript and the data services is a single point of failure for all of MOD's F2 Apps. A static manifest allows the app to be easily distributed among multiple servers or even organizations, while sharing the same data services (which could be backed by Apigee/Parse).

[montlebalm]

Here are my assumptions around what the package would contain:

• AppId, Name, Version
• Script/Style dependency urls
• HTML (either full payload or placeholder)

@ilinkuo I believe points 1, 2, 3, and 5 are about caching, which is a good thing. If the server can store the package then it won't need to do the initial request for the Manifest.

For 6, I'm not sure how much you'd gain here. Presumably the contents of the Script/Style dependencies won't be in the package, so hashing it won't tell you if the app logic has changed. You could send the entire app to the container developer, but you could probably do that today without changing the F2 spec.

A static package seems to favor apps that are initially rendered as HTML shells to be filled in by AJAX calls. Today, a lot of apps deliver a fully rendered module in the Manifest.

One little problem is with static file servers that generate a url at runtime (e.g., ASP.NET's System.Web.Optimization). That might make it difficult to provide a url for a static package.

@brianbaker brought up that many of the F2 containers are hosted by companies that have a slow promotion process, making it difficult to incorporate updates. For them it's helpful that the app developer can push changes outside of their cycle.

If we like static packages, let's support them in addition to dynamic ones. I don't think we want the framework to be too opinionated when it comes to app development.

[ilinkuo]

For 6, I'm not sure how much you'd gain here. Presumably the contents of the Script/Style dependencies won't be in the package, so hashing it won't tell you if the app logic has changed. You could send the entire app to the container developer, but you could probably do that today without changing the F2 spec.

I didn't go into in detail but I'd also go a level deeper -- either compute a single SHA-1 for all the resources directly pointed to by the manifest, or compute an individual SHA-1 for each of the direct resources. I'd stop there, however, and not go a step further. The first computation just let's me know something has changed, while the second computation let's me know which resource changed.

You could send the entire app to the container developer

You don't need to do that. A grunt task can fetch and compute hashes, given just the manifest, for instance.

A static package seems to favor apps that are initially rendered as HTML shells to be filled in by AJAX calls. Today, a lot of apps deliver a fully rendered module in the Manifest.

That's correct. A change to a static manifest is a clear preference in F2 app architecture. Today, an app can be written with a dynamic manifest or a static manifest. That flexibility is, in my opinion, a negative. It allows the developer to take a shortcut that makes the application less flexible in the long run. The dynamic manifest approach makes the manifest server mix presentation and data. The HTML + AJAX data calls clearly separates the presentation and data, and is compatible with a lot of the js mvc approaches out there like Angular, Backbone, etc. This architecture makes it possible for the simpler F2 Apps to deploy an app without having to maintain a server -- a static manifest served by an Apigee instance for the data persistence is all that is needed. The front-end is the View and Controller, while the backend is the Model. The dynamic manifest is a knot of entanglement between MVC that has real and negative consequences. If you think about the list of benefits that I listed, they come about because making the manifest static decouples concerns. It's the same reason why MVC is preferable to writing everything in a single ASP page -- the decoupling of concerns makes the app more flexible.

The advantages of this architecture are its performance and scaling and deployment flexibility. This kind of architecture is now familiar to many developers (not true 5 years ago) because of the JS frameworks like Angular and Backbone. So, to a developer who is familiar with this architecture, it's easy to rewrite a dynamic manifest F2 App to a static manifest F2 App. And to a developer who is a proponent of an MVC architecture, he/she can see how the dynamic manifest is taking a step back to the early days of ASP when everything was mixed together.

So why take away the option of a dynamic manifest? Because if F2 doesn't take away the option, I would guess <3% of the apps are going to be written this way, and the ecosystem of Apps is far less performant. In our vendor interactions, TDA can unilaterally impose a requirement that manifests be s static, but there's a lot of effort there if F2 doesn't go that way...

but you could probably do that today without changing the F2 spec.

... and that's really the problem. Technically, a lot of the benefits I've listed in my comment can be implemented today without changing the spec. So it can be argued "why not leave it in for flexibility?" My answer to that is that if F2 doesn't point the way to a better architecture by removing the bad architecture option, then no one is going to go down the better path.

One little problem is with static file servers that generate a url at runtime (e.g., ASP.NET's System.Web.Optimization). That might make it difficult to provide a url for a static package.

I'm not familiar with System.Web.Optimization though I did look it up. It doesn't look to me it's significantly different from what MOD is doing today. I gather that MOD's manifests today are dynamic because the manifest must be including all the Sys.Web.Opt generated urls?

I don't see that as an issue, though. The dynamic manifest server ought to be internal, for development only. When MOD is ready to push a development version to staging or prod, simply take the response from the dev server and publish that manifest to staging.

many of the F2 containers are hosted by companies that have a slow promotion process, making it difficult to incorporate updates. For them it's helpful that the app developer can push changes outside of their cycle.

TDA is one of those companies and we do like the fact that MOD can push changes outside our cumbersome release cycle. What we don't like is that the App can push updates without us being aware of it. For us, app updates must be accompanied by a version number change in the manifest. I don't think these two concerns are necessarily exclusive, and so we'd like to have our cake and eat it too, and it would be nice if the F2 Spec shares this point of view as well.

If we like static packages, let's support them in addition to dynamic ones. I don't think we want the framework to be too opinionated when it comes to app development.

If you really, really, really must accommodate both, then the above emphasis is wrong, in my opinion. It should be that in 2.0 static manifests are the preferred way and that dynamic manifests are supported for backwards compatibility only, but not recommended.

[brianbaker]

With a static manifest, what happens with batched requests from v1? Are those no longer supported?

In the case of a static manifest, each app would need to make an http request to fetch data. With a batched request, all apps in the request can share the same data. This is really important primarily for real-time quotes since many companies have to pay per quote. If I have 4 apps making separate requests then that will mean 4 quotes. If I can batch those and request only 1 quote that is shared across the apps, then I've saved the company some money.

If we had to have a static manifest, we would need secure app messaging and data apps in order to get around the quote counting issue. Here's a hypothetical where an 'optionchain' app has a dependency on the 'quotemanager' data app.

define('com_example_optionchain', ['F2', 'com_example_quotemanager'], function(F2, QuoteManager) {
    // ask for a quote - this results in an http request
    var quote =  QuoteManager.retrieve('msft');

    // get the rest of the option chain data - another http request
   $.ajax({ ... });
});

We saved 1 http request by having a static manifest but gained 2 or more http requests, per app, to get the data.

[brianbaker]

@montlebalm I think the System.Web.Optimization (Bundles) argument or really any type of combining might be a moot point depending on the AMD thread

[ilinkuo]

@montlebalm I've read the batched request feature but never really understood it. From my recollection of 1.2.1 code, the batched request just applied to manifest loading but didn't offer any of the other capabilities you'd mentioned. Is this all being done on the server side, because I can't recall seeing anything like that in the 1.2.1 code?

How does the batched request feature allow subsequent quote requests to be shared? There's nothing in the client F2 framework that's doing this as far as I know.

I think in your comment there are are two problems

• Minimize number of quote requests because client pays for it.
• The initial manifest requires multiple data calls

The first problem -- saving of quote requests -- can be solved on client or server side. I'm guessing that the existing solution is a server-side solution where the server is able to piggy back multiple quotes into a single http request to a third party server that charges based on number of requests?

But this kind of solution is possible on the client side as well. It just requires that the app does not directly request a quote via an http request, but rather makes requests through a client side proxy that makes the actual http request. That client side proxy can choose to delay quote requests, wait for other requests and combine them, and then cache the results for some specified time (300ms?). The F2 App should expect to interact with this proxy in an asynchronous manner, using Javascript (promise api)[https://github.com/promises-aplus/promises-spec/blob/master/implementations.md], implemented by jQuery, Dojo, and many other popular libraries.

The second problem is that of a data mashup -- you need multiple sources of data in order to populate the app. In this case, the manifest server is making calls to the multiple data sources and combining them to send back in the manifest for the current F2 spec. Well, if the multiple calls to fetch data on the client side is really a concern, then you can simply factor out your existing manifest server code so that the client makes a single call to the manifest server but the manifest server makes the underlying n data fetches and sends it back to the client as json. As a side benefit, you can delete the code on the manifest server that mashes up the different data sources as that will be done by the client side.

[brianbaker]

How does the batched request feature allow subsequent quote requests to be shared? There's nothing in the client F2 framework that's doing this as far as I know.

Correct, there is nothing in the client framework because this is a server-side solution. It's a single http request to the server for many apps, all of the data handling logic for those apps can be run at the same time and share resources.

But this kind of solution is possible on the client side as well.

Yes, I believe my example above was demonstrating the client-side proxy (see the QuoteManager).

If we wanted to go the route of allowing for batched data requests (similar to what we have in v1, but this time just for data), then maybe that's where the "1.5 fetch dynamic html and data" from the AMD thread comes into play. It would be a fully-supported and promoted way to cut down on the number of http requests. I know our network guys would be concerned knowing that we're going from a single http request (data + html) to many (data mashup).

[markhealey]

I think F2 needs to be able to support batch requesting apps, many F2 adopters are using this functionality for performance gains (reducing HTTP requests). I suppose it is possible F2 could support both static manifests and dynamic ones, only favoring (or allowing) the latter when batching is enabled. I say that yet I feel strongly that some form of version control is a critical feature missing from the current F2 AppManifest.

One of the key features in F2 today, in my opinion, is the fact that it is flexible in what you do on the server. F2 has no requirement or preference for a server side stack and we have built libs with little effort in both .NET, Java and Node.js to do everything F2 needs on the client. This flexibility allows solutions to be developed on the server that solve the multiple data calls issue or whatever else might be required by a NetOps or InfoSec team.

The dynamic manifest approach makes the manifest server mix presentation and data.

Finally, mixing data and presentation (html) in the AppManifest was very important at the outset because it allowed developers to write multi-purpose apps ultimately making the integrated Container much faster. Take an interactive chart, for example. I've seen apps built where the chart image is included in the AppManifest's html and the data property included the full timeseries in JSON. I suspect this data/presentation combination is still important for F2 adopters.

[ilinkuo]

I think F2 needs to be able to support batch requesting apps,

I'll start a new issue where I propose something which is workable and supports batch.

I've seen apps built where the chart image is included in the AppManifest's html

If it's not confidential, can you explain how? The only way I can think of to do this is to return svg.

[zackferraro]

You can also use an img (or a div with a background-image) with the byte array as a data URI. It would remove an additional XHR request for the image.

[ilinkuo]

@zackferraro Ah thanks. I didn't think of that.

[markhealey]

If it's not confidential, can you explain how? The only way I can think of to do this is to return svg.

Not confidential at all. MOD has the capability of generating images on-the-fly and handing back a URL for the image src attribute. It would look like this where both the image PNG is referenced in html and the end of day/close prices are available in data:

F2_jsonpCallback_com_companyname_appname({
    "scripts":[
        "http://www.domain.com/js/appclass.js"
    ],   
    "styles":[
        "http://www.domain.com/css/app.css"
    ],   
    "apps":[{
            "data":{ 
                "close": {
                  [1392751532169, 24.92],
                  [1392751551296, 24.87]
                  ...
                }
            },
            "html":"<div class=\"sunrise\">Hello world. <img src=\"/path/to/chart/image.png\"></div>",
            "status":"good"
    }]
})