Extending the HSL: the role/future of hsl-experimental, vendor extensions, etc

[fredemmott]

The HSL will shortly be built-in to HHVM and the typechecker; this requires defining what actual is (and isn't) part of "the HSL". I am intending for this to be:

• the HSL is entirely built-in. If it's not built-in, it's not the HSL
• the HH\Lib\ namespace is reserved for the HSL

We're also having increasingly frequent discussions about how to extend the HSL; there are a few common kinds of proposed extensions:

1. vendor-specific extensions/sub-libraries/sub-namespaces
2. existing libraries/namespaces we are considering merging into the HSL
3. additional functions for existing libraries/namespaces
4. future: additional methods for existing classes
5. new from-scratch libraries/namespaces

hsl-experimental has historically been used for (2)

My proposal

Overall:

• clear namespace-level separation of these concepts, except for 5, where a marker should be in the method name
• expectation should be that libraries either get merged to the HSL or removed from hsl-experimental

Situations 1-5

Posted as standalone comments below to ease replies.

Everything should be removed from hsl-experimental (eventually)

Another standalone comment :)

Why distinct namespaces again?

Previously, everything in hsl-experimental was in the HH\Lib\Experimental namespace; this meant that whenever something was moved from hsl-experimental to the HSL, all callers would be broken.

With the HSL becoming built in, this reverses: using the same namespace for the HSL and experimental becomes more problematic: it will pin any version of hsl-experimental much more tightly to a specific HHVM version.

Using a different namespace avoids this problem by effectively allowing duplicate definitions, just in different namespaces.

[fredemmott]

My proposal for scenario 1 - Vendor-specific extensions/sub-libraries/sub-namespaces

These should not be in hsl-experimental.

HH\Lib\ is reserved for the HSL; vendor-specific extensions are not part of the HSL, so SHOULD NOT be in sub-namespaces.

• Bad: HH\Lib\Vec\mycorp\foo(); + use namespace HH\Lib\Vec;
• Good: MyCorp\Lib\VecMC\foo(); + use namespace HH\Lib\Vec; use namespace MyCorp\Lib\VecMC;

In practical terms, we are unlikely to enforce this, but we will not consider adding new sub-namespaces in HH\Lib\ to be a breaking change - which could conflict with your vendor definition.

Existing vendor lib maintainers should consider this, but are not required to move.

[fredemmott]

My proposal for scenario 2 - existing libraries we are considering merging

hsl-experimental is the right place for these - however, the namespace should be changed to keep HH\Lib\ reserved for built-ins; perhaps HH\Staging, reserving Experimental for case (5), new libraries

[alexeyt]

I don't know that there is a need for a separate HH\Staging yet? Perhaps if we have things that have to be LTS (as opposed to the 6 weeks you mention below) and we want to mark that by separating them from HH\Experimental. If we do go down this route, I think it would be good to still have Lib in the namespace name (e.g. HH\LibStaging or HH\StagingLib or perhaps even HH\Lib\Staging) to make the relationship to HH\Lib more obvious.

[fredemmott]

so, this is going to come up relatively soon for subprocesses (but we're not there yet). Based on HSL IO merging, I don't think a separate namespace or repository is useful.

Changes should be made in the original namespace and repository (or a fork) until we're actually happy with it.

The repostiory should not contain other 'in progress' HSL APIs, just the one

Then pull request/diff review is sufficient

[fredemmott]

My proposal for scenario 3 - additional functions

This should be evaluated on a case-by-case basis; for example, OS\'s functions are relatively formulaic - in most cases, it will probably be appropriate to directly add to the HSL.

Otherwise, they should be in this library, but in appropriate subnamespaces like (2) - for example, a new Vec\foo() should be in HH\Staging\VecS - a suffix is useful to avoid the need for as clauses in use statements, encouraging consistency.

[fredemmott]

My proposal for scenario 4 - new methods for existing classes

Similarly case-by-case. If we're not confident, they will still need adding directly to the original class. It would probably be best to establish a naming convention.

[fredemmott]

My proposal for scenario 5 - new, from-scratch libraries

In general, I do not believe these should be in hsl-experimental: my preferred flow for adding to the HSL is to develop as a standalone library as much as possible, to

• gain community feedback
• encourage competition
• test, grow, and improve from real-world use. Libraries made without real-world use tend to have usability problems.

If there is a clear 'best winner', that would be a candidate for merging; if there are multiple libraries, perhaps making one solution the 'standard' one is not the best approach

Exceptions

If a new feature is heavily coupled to runtime or typechecker implementation changes or features, this is an appropriate repository - however, steps should be taken to minimize the disadvantages, such as:

• seeking feedback on the concept, design, etc of the library
• comparing to other libraries in the same problem space - both in Hack, and in other languages
• seeking adoption as soon as possible to get practical, hands-on feedback

These should be added to HH\Experimental

[azjezz]

I don't think this is really practical: this means that the entire stack back to your <<__EntryPoint>> must be marked experimental.

yes, if using experimental features is part of the public API, then __Experimental should fall-through to the entry point, unless it's an implementation details.

however, i think hide_experimental_implementation_detail=true is wrong, and should be an attribute as well.

example of using experimental feature as an implementation detail:

namespace Log {
  interface Logger {
   public function log(string $message): Awaitable<void>;
  }

  final class LoggerFactory {
    public static function createLogger(): Logger { return new Internal\HslIOLogger(); }
  }
}

namespace Log\Internal {
  final class HslIOLogger {
    <<__ExperimentalImplementationDetail>>
    public async function log(string $message): Awaitable<vodi> {
      await IO\output_handle()->writeAll($message);
    }
  }
}

namespace MyApplication {
  // MyApplication doesn't need to know about experimental features being used.
  <<__EntryPoint>>
  async function main(): Awaitable<void> {
   $logger = \Log\LoggerFactory::createLogger();
   await $logger->log('hello');
  }
}

[fredemmott]

if using experimental features is part of the public API

I'll add a few things to my goals :)

• all non-experimental features will be built in; you will no longer depend on hhvm/hsl
• any usage of hsl-experimental/ will be by-definition experimental - and all accessible APIs in this library will be experimental
• we should aim to move stuff out of hsl-experimental much more quickly in the future - either to HSL, or 'this belongs in an external library' etc

If hsl-experimental uses distinct namespaces, I don't see the value of a top-level project-level opt-in:

• people will just add it without consideration if a dependency uses it
• using namespace HH\Staging\FooS or using namespace HH\Experimental\FooE is pretty explicit already, and easy to create CI rules about

[fredemmott]

I don't see the value of a top-level project-level opt-in

... also, I'd like hsl-experimental to be as un-special as possible. Who should use this attribute? Just hsl-experimental? any 0.x library? I don't think there's a good line, and it doesn't add much value

[azjezz]

I'd like hsl-experimental to be as un-special as possible. Who should use this attribute? Just hsl-experimental? any 0.x library?

I agree, and i think anyone should be able to, it doesn't have to be an 0.x library, it can be a fully developed library, but it added a new functionality that is still considered experimental.

e.g: symfony contains a new component in 5.3 rate-limiter which is experimental ( using @experimental annotation https://github.com/symfony/symfony/blob/5.x/src/Symfony/Component/RateLimiter/RateLimiterFactory.php#L28 )

[fredemmott]

I think we'd really need per-directory configs for that

For example, I'd like to ban <<__Experimental>> or [experimental] in src/, but not care about it in vendor/

[fredemmott]

My proposal: everything should be removed from hsl-experimental (eventually)

• if something is merged to the HSL, it should be removed once that happens
• if something is not going to be merged, it should be removed

The simple question is how much notice to give when we decide a sub-library should be removed; a bit of a strawman, but I'd like to go for 6 weeks: it matches the main HHVM release cycle, and this is an experimental library - the LTS cycle doesn't feel appropriate.

The more involved question is what to put in the HSL, and what to drop.

What belongs in the HSL?

Historically, this has been based on need, usage, and "is this the way we want things to be done in Hack?". By this metric, HSL IO should be added, and Str\Utf8 + Str\Grapheme should be removed.

I think we should weaken the last constraint: we should allow intermediate steps to be merged into the HSL, if:

• there is not a viable alternative other than PHP builtins
• we do want the HSL to address this problem eventually, even if not quite in this way
• we believe that the work will make it easier to automatically migrate to our hypothetical 'correct' solution in the future
• it improves type safety (weak factor)

By this definition, we should merge the entirety of HSL-experimental into the HSL, including Str\Utf8 and Str\Grapheme; using Str\Utf8 as an example:

• we do want the HSL to address this, but in a slightly different way: we want Str\ to be UTF8-focused, and the existing ascii functions to be somewhere else
• adding Str\Utf8 now will make that work easier in the future: rewriting Str\Utf8\foo() to Str\foo() is easier than safely rewriting \mb_foo() to Str\foo()

[azjezz]

What belongs in the HSL?

this is the approach I'm taking with psl ):

• for every function/extension in PHP:
  • [1] does HSL/PSl contain an alternative that can be used with maximum 2 lines of code?
    • yes
      • shouldn't be in HSL/PSL, you probably won't miss it
    • no
      • [2] is this going to be opinionated? is there multiple alternatives out there?
        • yes
          • shouldn't be in HSL/PSL, the community already made it, people can pick and choose
        • no
          • [3] is there a library/package that solves this?
            • yes
              • [4] is it will maintained? can we be sure that people will use it?
                • yes
                  • shoudln't be in HSL/PSL
                • no
                  • [5.1] try to work with the maintainer, and see if that improves the situation, then go back to 4, if the answer is still no, then 6.
            • no
              • [6] is this problem better solved in the user-land ( http-router, csv reader, query builder )?
                • yes
                  • deliver it as a separate package, not part of HSL/PSL
                • no
                  • should be in HSL/PSL

[jjergus]

For purely practical reasons, I'd like to keep as many of these as possible in a single repository (e.g. there's a lot of overhead in managing multiple repositories; it's likely that things will move between these categories in the future which should be much simpler in a shared repository...).

It doesn't have to be called hsl-experimental (one suggestion was "flib" which roughly stands for "Facebook libraries" and matches some internal usage), your suggestions for the different categories and namespaces seem reasonable.

[fredemmott]

I think we disagree more broadly on (5), but whatever we end up deciding there, yep, whatever we decide is appropriate should all be in this repository, just in different namespaces