Buildpack support for Symfony 5 demo app

[fg-j]

I haven't been able to get the PHP buildpack to build the Symfony 5 demo app. Expected behaviour is that the buildpack should be able to build this app, as it's apparently a canonical Symfony app and users want support for the framework (see #3)

Repro steps:

1. Clone the symfony 5 demo app
2. Update dependencies in the lock file composer update --no-install
3. Add a buildpack.yml containing:

   ---
   composer:
     vendor_directory: "vendor"
   
   php:
     webdirectory: "public"
   

4. Build the app

   pack build symfony-demo -b gcr.io/paketo-buildpacks/php --env APP_ENV=prod
   

Result:
Build fails with output

pack build symfony-demo -b gcr.io/paketo-buildpacks/php --env APP_ENV=prod
base: Pulling from paketobuildpacks/builder
Digest: sha256:95015c909b3031bb618123866447f2c440d5e56e9dca3ab4fe7cb432b88941a3
Status: Image is up to date for paketobuildpacks/builder:base
base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:155ee4b4389fa88b8b9bd5444601417218b3da44e491b1aad8c8b6c2365e7909
Status: Image is up to date for paketobuildpacks/run:base-cnb
latest: Pulling from paketo-buildpacks/php
Digest: sha256:9e3d9c0c192773d559fbf232667fb2c82425240d9f36c7fbd1354a6c991b433f
Status: Image is up to date for gcr.io/paketo-buildpacks/php:latest
0.9.3: Pulling from buildpacksio/lifecycle
Digest: sha256:bc253af2edf1577717618cb3a95f0f16bb18fc9e804efbcc1b85f657d931a757
Status: Image is up to date for buildpacksio/lifecycle:0.9.3
===> DETECTING
[detector] 3 of 5 buildpacks participating
paketo-buildpacks/php-dist     0.0.208
paketo-buildpacks/php-composer 0.0.127
paketo-buildpacks/php-web      0.0.133
===> ANALYZING
[analyzer] Previous image with name "symfony-demo" not found
===> RESTORING
===> BUILDING
[builder] Paketo PHP Buildpack 0.0.208
  Resolving PHP version
    Candidate version sources (in priority order):
      composer.lock    -> "^7.2.9"
      default-versions -> "7.2.*"

[builder]     Selected PHP version (using composer.lock): 7.4.9
[builder]
  Executing build process
[builder]     Installing PHP 7.4.9
[builder]       Completed in 4.168s

  Configuring environment
[builder]     MIBDIRS           -> "/layers/paketo-buildpacks_php-dist/php/mibs"
    PATH              -> "/layers/paketo-buildpacks_php-dist/php/sbin:$PATH"
    PHP_API           -> "20190902"
    PHP_EXTENSION_DIR -> "/layers/paketo-buildpacks_php-dist/php/lib/php/extensions/no-debug-non-zts-20190902"
    PHP_HOME          -> "/layers/paketo-buildpacks_php-dist/php"

[builder]
Paketo PHP Composer Buildpack 0.0.127
   1.10.10: Contributing to layer
[builder]     Downloading from https://buildpacks.cloudfoundry.org/dependencies/composer/composer_1.10.10_linux_noarch_any-stack_8f16aa77.phar
[builder]     Verifying checksum
[builder]     Expanding to /layers/paketo-buildpacks_php-composer/composer
[builder]   PHP Composer Cache e57a3520f2fcb109f78c013afc32e8c9178ebda4c9b9e14a706c666172c8902a: Contributing to layer
[builder] php: error while loading shared libraries: libxml2.so.2: cannot open shared object file: No such file or directory
[builder] exit status 127
[builder] ERROR: failed to build: exit status 106
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 145

[arjun024]

If you're using the base bionic builder, you have to have add a few mixins because php-dist buildpack requires it. See https://github.com/paketo-buildpacks/php-dist/blob/v0.0.209/buildpack.toml#L102-L103

Alternatively, you can use the full-builder. The following build succeeded for me:

pack build symfony-demo -b gcr.io/paketo-buildpacks/php:0.0.11 --env APP_ENV=prod --builder paketobuildpacks/builder:0.1.31-full

[fg-j]

Thanks for that tip. Now I've gotten the app to build but something's going wrong when I try to run it and access the homepage:
Repro steps:

1. Clone the symfony 5 demo app
2. Update dependencies in the lock file composer update --no-install
3. Add a buildpack.yml containing:

   ---
   composer:
     vendor_directory: "vendor"
   
   php:
     webdirectory: "public"
   

4. Build the app

   pack build symfony-demo -b gcr.io/paketo-buildpacks/php --env APP_ENV=prod --builder paketobuildpacks/builder:full
   

5. Run the app container

   docker run --interactive --tty --env PORT=8080 --publish 8080:8080 --env APP_ENV=prod symfony-demo
   

6. Curl the app:

   curl 0.0.0.0:8080
   

Output from server:

docker run --interactive --tty --env PORT=8080 --publish 8080:8080 --env APP_ENV=prod symfony-demo
[Fri Nov 13 17:17:54 2020] PHP 7.4.9 Development Server (http://0.0.0.0:8080) started
[Fri Nov 13 17:18:00 2020] 172.17.0.1:49926 Accepted
[Fri Nov 13 17:18:00 2020] 172.17.0.1:49926 [500]: GET /
[Fri Nov 13 17:18:00 2020] 2020-11-13T17:18:00+00:00 [critical] Uncaught Error: Class 'Symfony\Bundle\DebugBundle\DebugBundle' not found

And here's what the page looks like when I go to access it:

[fg-j]

The stack trace reads:

Symfony\Component\ErrorHandler\Error\ClassNotFoundError: Attempted to load
class "DebugBundle" from namespace "Symfony\Bundle\DebugBundle".  Did you
forget a "use" statement for another namespace?

  at
  /layers/paketo-buildpacks_php-composer/php-composer-packages/vendor/symfony/framework-bundle/Kernel/MicroKernelTrait.php:74
  at App\Kernel->registerBundles()
  (/layers/paketo-buildpacks_php-composer/php-composer-packages/vendor/symfony/http-kernel/Kernel.php:371)
  at Symfony\Component\HttpKernel\Kernel->initializeBundles()
  (/layers/paketo-buildpacks_php-composer/php-composer-packages/vendor/symfony/http-kernel/Kernel.php:128)
  at Symfony\Component\HttpKernel\Kernel->boot()
  (/layers/paketo-buildpacks_php-composer/php-composer-packages/vendor/symfony/http-kernel/Kernel.php:191)
  at Symfony\Component\HttpKernel\Kernel->handle(object(Request))
  (/workspace/public/index.php:28)

[arjun024]

Thanks for the detailed report. I'm not able to do composer update --no-install locally - keep running into sh: 1: symfony-cmd: not found.
Then I try composer install --no-dev --optimize-autoloader and it fails with the same error you described above:

!!  Symfony\Component\ErrorHandler\Error\ClassNotFoundError {#70
!!    #message: """
!!      Attempted to load class "DebugBundle" from namespace "Symfony\Bundle\DebugBundle".\n
!!      Did you forget a "use" statement for another namespace?
!!      """
!!    #code: 0
!!    #file: "./vendor/symfony/framework-bundle/Kernel/MicroKernelTrait.php"
!!    #line: 74
!!    trace: {
!!      ./vendor/symfony/framework-bundle/Kernel/MicroKernelTrait.php:74 { …}
!!      ./vendor/symfony/http-kernel/Kernel.php:371 { …}
!!      ./vendor/symfony/http-kernel/Kernel.php:128 { …}
!!      ./vendor/symfony/framework-bundle/Console/Application.php:168 { …}
!!      ./vendor/symfony/framework-bundle/Console/Application.php:74 { …}
!!      ./vendor/symfony/console/Application.php:142 { …}
!!      ./bin/console:43 {
!!        › $application = new Application($kernel);
!!        › $application->run($input);
!!        ›
!!        arguments: {
!!          $input: Symfony\Component\Console\Input\ArgvInput {#3 …}
!!        }
!!      }
!!    }
!!  }
!!  2020-11-13T21:52:54+00:00 [critical] Uncaught Error: Class 'Symfony\Bundle\DebugBundle\DebugBundle' not found
!!

Do you have an app after you have successfully completed steps 1,2 and 3?

[fg-j]

Try this fork of the demo app. It has the updated composer.lock and the buildpack.yml added.

[arjun024]

Thanks. We'll investigate if this is a problem with the buildpack or the app

[fg-j]

Hey @arjun024 – did you end up learning anything interesting about this problem?

[till]

@fg-j did you manage to build the app?

[fg-j]

No, this still isn't working for me. But it might be a question of configuration/lack of documentation, since the PHP buildpack has been significantly overhauled since I initially filed my issue. @paketo-buildpacks/php-maintainers, can you take another look at Symfony support in the buildpack?

[till]

It wasn't working for us either out of the box, I'll pull together what we had to do later in the day.

[sophiewigmore]

Hey @till @fg-j I'll try to take a look at this in the next few days as well

[till]

@fg-j @sophiewigmore we published a working example here: https://github.com/hostwithquantum/runway-example-php

Not ideal yet, it works, but IMHO too many changes required for a smooth (user) experience.

The readme outlines everything that's necessary, especially the patching of paths. Don't want to rule out that we didn't fully grok it yet either.

[sophiewigmore]

@till thanks for putting so much detail into to the runway-example-php repo, I totally see that there are too many extra changes needed to make it super usable. Did you ever get the app to run correctly? I tested out your sample app but see an error when I try to query the app "PHP message: PHP Fatal error: Uncaught Symfony\Component\Dotenv\Exception\PathException: Unable to read the "/layers/paketo-buildpacks_composer-install/composer-packages/.env" environment file. in /layers/paketo-buildpacks_composer-install/composer-packages/vendor/symfony/dotenv/Dotenv.php:557

Going through the changes you outlined, setting the PHP extensions, creating the project.toml configuration, and adding custom NGINX configuration are all parts of using the PHP buildpack I would expect. It's unfortunate that so much configuration is needed to make your app work, but the PHP buildpack just has so many different use cases to support, it's difficult to achieve that behaviour without some user-set configuration. Hopefully those steps were at least well-documented enough to get that set up without too much pain?

The other changes you outlined are definitely less than ideal:

• The modifications you had to make to the composer.json to handle symlinks by providing the full path including /workspace, I would not have expected to be necessary with the latest release of the PHP buildpack. We just made some changes that hopefully should've addressed a similar issue. What error did you see without that change? And what version of the buildpack did you use?

• The removal of the post-install @auto-scripts line is an issue I ran into myself when testing out the app that Frankie provided as well. I'm not super familiar with the problem though. Could you elaborate a bit about what the script was doing / the failures you saw? This might be something we should support in the buildpack.

[till]

Thanks for the detailed response. :)

The error I got is the error in this issue. 👆 It fails to load classes from other paths than ./vendor.

I understand the part about webservers, etc. though I'd argue the majority of PHP needs a webserver to run and CLI is not unheard of but in most cases it's also part of an application which needs a webserver.

As for the paths, I think the conceptual layers are a problem here.

Everything should be in one layer for the code to run.

A composer install puts files in 99.99999999% of all cases into the application in a vendor dir. That would address the other issues as well.

Do you know how to adjust it to flatten the layers into one?

Otherwise it could probably use include_path magic though that is relatively expensive and this is why dependencies are local these days (vs. global like pear).

What other use cases is the bp adressing right now? I am curious as to how to solve this issue and not break anything else.

[sophiewigmore]

@till sorry for the delayed response here. I really appreciate the feedback, and also all of the insights - I am not a PHP expert.

Can you clarify the line and CLI is not unheard of? What is CLI referring to in that sentence? The buildpacks default to using the PHP builtin-server if Nginx or HTTPD are not selected. In your experience with PHP, would Nginx or HTTPD be a better default?

I totally agree that the layers are causing an issue. I think we missed this because we are testing against a pretty vanilla set of applications. We could benefit from adding some more complicated examples, at least to the composer-install integration suite, since that seems to be the buildpack that has the most issues. We could also benefit from being more intentional with where we put the outputs of the various composer commands that get run.

The entire vendor directory gets symlinked to the composer-install layer so I'm not entirely sure what it's missing. I need to do a bit of investigation

The top-level PHP use cases are outlined here: https://github.com/paketo-buildpacks/rfcs/blob/main/text/php/0001-restructure.md#buildpacks. Like I mentioned before, the composer-install buildpack is responsible for everything related to package installation, and the use cases are best outlined by the test apps we have: https://github.com/paketo-buildpacks/composer-install/tree/main/integration/testdata.

[till]

@sophiewigmore thanks for taking the time.

I'll try to answer your questions one by one, but let me know if I should clarify anything. We can also Zoom (or so) some time next week. Sorry, this is a rather long answer. I hope it's an answer though.

Can you clarify the line and CLI is not unheard of? What is CLI referring to in that sentence?

CLI - command line interface. I wanted to say that the majority of PHP applications (and frameworks) require a web server:

• Symfony
• Laraval
• Slim
• Code Igniter
• ...

People use the above to build applications that run in the browser, or provide some kind of "web-based" API. I have to admit that I am also not (100%) up to date as to what else people use these days. So if anyone reads this and I am missing their favorite framework, apologies.

More specific applications/frameworks:

• Wordpress
• Magento/OXID/Shopware/osC
• Drupal
• ...

What I meant by "CLI is not unheard of" is that people build CLI applications/tools with PHP, but usually these are part of an application which also requires a web server. So defaulting to a web server is good.

So as an example, the Symfony framework itself is used to build an application for the browser, but it also includes a CLI tool (through a Symfony/Console component) that you can use to e.g. clear a cache, generate assets, etc.. Of course that CLI tool is extensible as well. This is very similar to rails or django.

The buildpacks default to using the PHP builtin-server if Nginx or HTTPD are not selected.

The PHP built-in web server is not a great choice, but then again, I am not entirely sure what the buildpack targets here. If the objective is to get something running for local development, then the built-in web server is okay, but I feel like buildpacks are too involved for that.

I think, this is one of the key differences of PHP to languages like Ruby, Python or NodeJS. All these languages have libraries to provide web server parts. It's not absolutely necessary to front them with nginx/Apache.

In your experience with PHP, would Nginx or HTTPD be a better default?

As for defaulting, (personal preferences aside) I think most of these applications include instructions how to setup URL rewriting with .htaccess, so then Apache (HTTPD) is probably a better choice.

I totally agree that the layers are causing an issue. I think we missed this because we are testing against a pretty vanilla set of applications. We could benefit from adding some more complicated examples, at least to the composer-install integration suite, since that seems to be the buildpack that has the most issues.

Yeah, great idea to improve the test suite. I think it's safe to assume that composer is everywhere these days. It's similar to bundler in Ruby, I don't think people write anything without using it. Or pip for Python.

We could also benefit from being more intentional with where we put the outputs of the various composer commands that get run.

Yeah, I am just starting to learn more about this, and I was a bit stretched this week as well.

So regarding what happens (for example) during/after install, I think in our example, we should have adjusted BP_COMPOSER_INSTALL_OPTIONS to add --no-scripts instead of removing the following from composer.json:

--- a/composer.json
+++ b/composer.json
@@ -87,7 +87,6 @@
             "assets:install %PUBLIC_DIR%": "symfony-cmd"
         },
         "post-install-cmd": [
-            "@auto-scripts"
         ],
         "post-update-cmd": [
             "@auto-scripts"

That would fix one of the problems with Symfony setup.

For more context: the post-install-cmd is a bit of a tricky one.

I think for convenience ("easy demo app setup") it may be (often) (ab)used to setup a database schema, but also generate assets (css, images, javascript). While generating assets (and baking them into the image) is a good idea, setting up a database schema is definitely far away during a build process.

The entire vendor directory gets symlinked to the composer-install layer so I'm not entirely sure what it's missing. I need to do a bit of investigation

Yeah, not sure either. This may be somewhere related to how Symfony's flex plugin handles paths. Though it seemed like Drupal, etc. suffered from similar problems. See the root-dir in my example, also haven't investigated yet as to why exactly this is necessary.

But maybe to add some context:

(I don't want to bore anyone with this, and apologies if any/all of that is already known.)

Before we had composer, developers either copied code around or used the pear installer in PHP.

The pear installer was most often used "globally", so an entire server/VM had the same packages (php libraries) available. (For the record, it was also possible to change that and maintain a local installation, but that was error prone and rare). pear heavily relied on the proper setup of the include_path to be able to load PHP libs (not extensions):

<?php
require_once 'Foo/Bar.php';

By default the include_path of a PHP installation today is still something like, .:/usr/lib/pear/whatever, so for example, /usr/lib/pear/whatever/Foo/Bar.php can be included (or required) without the full path.

If you want a similar behaviour for other paths, you can add them like so: include_path=.:/usr/lib/pear/whatever:/workspace/lib. Because of i/o penalties (all three paths are now searched for a require/include), PHP tried to optimise this with autoloading. Which was cache-able and what not.

Then enter composer, most of PHP development (radically) switched to including all libraries needed, within "the scope" of an application. Instead of globally installing libraries with pear, the composer.json/lock files are used by composer install to create a tree of dependencies in a local ./vendor folder.

The path/name ./vendor folder can be customised, and developers can hook into the process, but that's semantics.

There is a really long list of things that composer improves for developers, but that is probably not relevant right now. My take away were these two things.

A developer can run any amount of applications on the same server/VM:

• /workspace/app1
• /workspace/app2

... and they can use different dependencies because they are included:

• /workspace/app1/vendor
• /workspace/app2/vendor

As in, let's say, I rely on different versions of the same library in each of these applications. Because each is installed into a different folder, they don't collide. I also no longer have to ask someone with root to install something for me, and so on.

Looking at the include_path above, the first directory in the path is . (separator is :) which is the current directory. So in order to use dependencies installed through composer, a developer does the following in their application bootstrap/front controller:

<?php
require 'vendor/autoload.php`;

Which brings me to the second thing: composer generates an autoloader which generates a class-map to include/load classes on demand. So developers today only require 'vendor/autoload.php'; and all dependencies are loaded when used in code. No more include_path magic or custom autoloaders required.

You can also test this with a composer dump-autoload, that should resolve everything that is somehow a part of ./vendor. Even if ./vendor itself is a symlink. This is also automatically part of a post composer install (or composer update). I am not sure what is missing here. Or why the frameworks don't like it (#253, #366).

The top-level PHP use cases are outlined here: https://github.com/paketo-buildpacks/rfcs/blob/main/text/php/0001-restructure.md#buildpacks.

I would probably prioritize composer a bit as it seems to be core to everything that people do.

Like I mentioned before, the composer-install buildpack is responsible for everything related to package installation, and the use cases are best outlined by the test apps we have: https://github.com/paketo-buildpacks/composer-install/tree/main/integration/testdata.

Thanks for that link. And again, sorry for posting here. I am guessing this should have all been in that repo. But yeah, those use-cases should do...

Though I don't fully grok the difference between:
https://github.com/paketo-buildpacks/composer-install/tree/main/integration/testdata/default_app
https://github.com/paketo-buildpacks/composer-install/tree/main/integration/testdata/default_app_global

Is the default_app_global executing something that is installed outside of the local composer.json?

If that is the (use-)case, another approach to this these days is to add php-cs-fixer as a devDependency (composer require --dev ...) and then the php-cs-fixer in installed in vendor and ultimately symlinked into ./vendor/bin (you could use composer exec php-cs-fixer to run it, or define a script in composer.json to invoke it).

Just looking at the README, I think BP_COMPOSER_INSTALL_GLOBAL seems unnecessary in a buildpack context. At least I can't think of a reason why I would need to use that.

And instead I would extend the buildpack to support more granular control over composer run-script or composer exec. But maybe that's a separate discussion?

[sophiewigmore]

I appreciate the effort you put into this thread, it's got some great context. I think it'll come in handy as we iterate on the PHP buildpacks. It really helps having a user's input on what makes sense/ what's confusing. I should also point out that Paketo Slack is a great place to chat with people as well!

Let me see if I can summarize where we're at:

• Most PHP apps leverage frameworks and web servers together. Outside of just building an app, the frameworks also have CLIs. In this space, there is probably room to create new buildpacks or add new functionality down the line?

Actionable things:

• The built-in server isn’t probably the best default server choice in your opinion (I agree). HTTPD could be a better choice. I'd probably want an RFC about changing the default written if this is something we felt strongly about changing.
• We could and should make some improvements to the composer-install test suite and implementation. Included in this is:
  • investigating failures to build Symfony/other framework-related apps and symlinking. THat will hopefully resolve this specific original issue.
  • Understand why --no-scripts is needed to work with Symfony

Your question regarding the global composer app is extremely valid. I think the point was just to show in the related integration test, that if the BP_COMPOSER_INSTALL_GLOBAL env. var is set, then the buildpack will run a different composer command and should still make the installed packages available to the app in the image. I do agree that in a buildpack context it doesn't help much to have global installation abilities because of the way packages are handled in layers. I also don't think it hurts at all to have as a feature, so I'm fine with it for now. All of the features of the buildpacks were ported over from some older PHP buildpacks we had, and we didn't want to break support for any features of the old buildpacks. That's kind of a different issue.

I'd definitely like to hear a bit more about what you had in mind for composer run-script and composer exec (maybe in a separate issue on the composer-install buildpack repo?)