Symlink file assets (3.0)

[btaens]

Hey,
This is an amazing plugin, it has been incredibly useful for asset management for me.
My work process is the following: I pull all my vendor files (plugins, libraries, etc...) through composer, and the front-end stuff through bower into a publically inaccassible folder.

After that, I compile my files through asset_compress into a single css and javascript file, hence cutting down on the number of requests, and managing exactly what is publically accessible to our users.

Because of this, it would be extremely useful to be able to simply specify asset files (images, directories) to symlink, or if that is unavailable, copy to a public directory. With this, it would be possible to precisely manage all assets through asset_compress.ini.

I was thinking something akin to this:

[symlink]
source[] = ROOT/vendor/assets/select2/select2.png
dest[] = WEBROOT/img/select2.png
source[] = ROOT/vendor/assets/bootstrap/fonts
dest[] = WEBROOT/img/fonts/bootstrap

[markstory]

Sounds like an interesting idea, would the assets to move/symlink be tied to specific build targets?

[btaens]

Might make sense, perhaps even have the possibility to replace references in stylesheets at compile time?

That might require some advanced parsing though, really just having the possibility to simply symlink would be awesome.
I'm also thinking the symlinking would have to happen before the compiling of javascripts and stylesheets, that way it would also be possible to symlink vendor sass and less files for example to a specific place, and then import them to our own sass or less flow. I'd do this with Twitter Bootstrap for example, thus having the benefit of mixins while still easily keeping it up-to-date.

[markstory]

AssetCompress already passes search paths into less/sass. As long as your vendor files are on the plugin's search path you should already be able to use them.

I can see how symlinking/copying assets could be useful, but wouldn't that also require rewriting paths?

[btaens]

Changing css references would be very useful, but simple symlinking is already really good. Rewriting paths might get tricky. There are a lot of ways to reference assets. Though yes, if that was somehow possible, that would solve a whole lot of asset management.

While yes, sass and less files can be compiled directly into the composite file with asset_compress, the problem with that is that you can't use the vendor mixins and variables in your own sass and less files. For example, I want a custom built element to have the same colors and look as the standard Twitter Bootstrap elements, I can't use it's vendor specific variables, like, say, $panel-default-heading-bg.
Granted, this can be solved in a multitude of ways (@importing it directly from the vendor directory, for example), for workflow reasons it would just make sense to symlink files before the compiling of js and css files, thus leaving options open.

[btaens]

Actually, I do see a CSSUrlRewrite pull request from back in 2013. Did that not work out? What happened?

[markstory]

It didn't work out from what I remember, but that was a long time ago.

[btaens]

Alright, I've been thinking about this CSS URL rewriting business, looking around the code.
Theoretically, it could be a seperate Filter. The user would simply provide a target directory for the assets, and apply the filter to a file composition. The filter would then check for all relative urls in the css files, symlink or copy them to the target directory, and change said urls accordingly (perhaps the user could also provide a web accessible route, or a relative route it would prepend behind them, like ../assets/, or /assets/.

The only problem is, for this to work, the filter would need to see the original paths of the original css files. So it would either have to be a high priority filter, that gets the list of css files and compiles the first, concatenated input to the other filters (this would allow things like, if 2 css files have assets with the same names, but seperate files, it would know and symlink them as 2 seperate files),
The other option is it's a normal filter, but filters can access the input file list paths (like global settings ala $this->_settings["paths"], but something like $this->_files, which would give it the list of input files for this composition). This however runs the risk of 2 assets with the same relative path getting mixed up, or one of them getting lost (as we don't know where the strings we replace came from).

Do you think it would be possible for filters to at least see the input file list? Is there something that might make this undesirable?

[btaens]

There's also the possible problem of absolute paths being provided in a css file (like /img/logo.png), as we don't know what it constitutes as the root folder.

[markstory]

Root paths get extra tricky when you are dealing with applications in sub-directories as well. It sounds like URL rewriting is going to be pretty tricky, would we be able to get most of the benefit by sticking to a manifest of inputs and output files?

[btaens]

Yeah, that's the other option.

If we're working with a manifest though, the question again is, do we just ask for a file path, and then try to replace in the css based on that, or do we also ask for a path in the css to replace? Asking only for file paths might again cause conflict.

A file manifest might also get pretty lengthy. My app right now has about 200 assets. Perhaps a css mining shell? One that updates the manifest based on found assets, and asks for user input at the appropriate time?

[btaens]

A simple solution might just be to require relative paths. Absolute ones are bad practice anyway, and vendor libraries don't much do it. So maybe we just don't touch absolute paths?

[berarma]

I think being able to rewrite relative paths for images in CSS vendor files would solve the most common scenario where this would be useful, vendor libraries. Image directories could be symlinked instead of files, every vendor image directory symlinked to a unique name in the webroot (cahe_img/XXX or img/cache/XXX) directory would be easier maybe.

We would need some way to declare that a set of CSS files should get their URLs rewritten to that image cache directory. We just need to specify the vendor directory we want to symlink and the set of CSS files to apply it to.

Another solution I was thinking about is URL rewriting in the web server but it wouldn't work with the built-in server and URLs might clash.

[berarma]

Symlinking isn't the hard work here for me, I manually create the links and commit them to git. I can live with that without problems. The ugly part is having to copy and commit the vendor's library CSS to my repository. If there was a filter that could rewrite image URLs in CSS I'd go with that.

By now I'm overriding CSS styles with my own to change URLs. It's good enough unless there's a lot to change.

[markstory]

In my projects, I either use absolute URLs images - as most of the things I work on don't live in a subdirectory. For local development I create virtualhosts like site.localhost so all the paths work all the time.

[btaens]

Could we assume the provided paths are the root path for a file in question? That would basically solve it, although you'd have to provide a path to each vendor. Technically that's how it's supposed to be done anyway.

So for example:

[css]
paths[] = ROOT/vendor/assets/select2/
[main.css]
files[] = css/select2.css

That way, if select2.css has an asset like url('/img/loader.png'), we'd look for the file in ROOT/vendor/assets/select2/img/loader.png.

[berarma]

References like url('../img/loader.png') would fail. Most libraries use relative URLs.

All these solutions are almost good enough, but not fully. The problem stems from AssetCompress being based on individual files while vendor libraries are a cohesive collection of files. We are taking a CSS file out of its context thus breaking references.

There should be some way to let AssetCompress know about contexts. Vendor libraries' assets could be cloned in WEBROOT then all references in CSS files for that context redirected to the cloned directory.

Let's not just think in image URLs, there could be also CSS imports, though much less common.

All this talk makes me think if it would be easier to install vendor assets directly in WEBROOT and load CSS/JS without AssetCompress.

JM2C.

[btaens]

Why would it fail? We can differentiate absolute and relative assets. Relative assets are the simplest problem to solve. Given that they are relative, you just have to append the path to the css file's location.

CSS imports already have their separate, working filter.

It's also not impossible to solve. Symfony's Assetic already has it down, we just have to accept a standard.
I plan to write this filter on the weekend, which is why I'm asking how it should be done.

Cloning entire libs into webroot is a pretty bad idea. The idea is control over what you put out into a public library. One faulty update on a vendor side could jeopordize security. Bower has solutions for this, but I think CakePHP should have a comprehensive, flexible solution, so we can do interesting things inside the framework.

[berarma]

Bad assumption on my part on your proposed solution. This could be a task not belonging in AssetCompress, maybe Composer, a Cake shell or some deployment tool, but I'm interested to see where you take it to.