Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Plugins system support #143

Open
damon-kwok opened this issue Jul 16, 2020 · 11 comments
Open

Add Plugins system support #143

damon-kwok opened this issue Jul 16, 2020 · 11 comments
Labels
enhancement New feature or request needs discussion Needs to be discussed further

Comments

@damon-kwok
Copy link

damon-kwok commented Jul 16, 2020 •
edited
Loading

Summary

The plug-in system is similar to a set of callbacks that allow Corral to trigger and run custom tasks as it processes commands.

Motivation

  • Provide expansion capability for Corral.
  • Users can contribute Corral without changing corral code.

Detailed design

  • The plug-in system is designed to support cross-platform
  • All plug-ins must support cross-platform, which needs to be checked automatically before merging PR.

Your new plug-in would look like:

| corral
├── plugins
│   ├── example
│       ├── INSTALL     # plugin installer
│       ├── COMMAND     # (optional) run user command
│       ├── on_create   # (optional)
│       ├── on_fetch    # (optional)
│       ├── on_update   # (optional)
│       ├── on_build    # (optional)
│       ├── on_run      # (optional)
│       ├── on_clean    # (optional)
│       └── on_deploy   # (optional)

For windows:

MSYS2 needs to be built for windows
@damon-kwok
Copy link
Author

damon-kwok commented Jul 16, 2020 •
edited
Loading

Example 1: Write a statistic plugin to counts code lines at build time

| corral
├── plugins
│   └── vcpkg
│       ├── INSTALL
│       └── on_build

INSTALL:

curl -L "https://statistic.sh" | bash

on_build:

statistic *.pony ${project_root}

use statistic:

corral run -- ponyc

output:

pony        11567
markdown    115
sh          76

@damon-kwok
Copy link
Author

damon-kwok commented Jul 16, 2020 •
edited
Loading

Example 2: Write a vcpkg plugin to handle cross-platform native deps #142

| corral
├── plugins
│   └── vcpkg
│       ├── INSTALL
│       └── COMMAND

INSTALL:

  github_deps  microsoft/vcpkg $CORRAL_REPO/vcpkg
  cd $CORRAL_REPO/vcpkg
  chmod +x ./bootstrap-vcpkg.sh
  if [ "$OS_TYPE" = "cygwin" ] || [ "$OS_TYPE" = "msys2" ] || [ "$OS_TYPE" = "winnt" ]; then
    ./bootstrap-vcpkg.bat
  else
    ./bootstrap-vcpkg.sh
  fi
  vcpkg_setenv

COMMAND:

#!/usr/bin/env sh

vcpkg $@

Use vcpkg plugin

corral.json: (NOTE, Event: #144 )

{
  "plugins": ["vcpkg"],
  "deps": [
    {
      "locator": "github.com/ponylang/valbytes.git",
      "version": "0.5.0"
    }
  ],
  "info": {}, 
  "event": [      
    {
      "on_fetch": "vcpkg install openssl",
    }
  ]
}

use openssl in pony:

use "lib:crypto"
use "lib:ssl"

Test

corral fetch

output:

Computing installation plan...
The following packages will be built and installed:
    openssl[core]:x64-linux
  * openssl-unix[core]:x64-linux
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-linux...
Starting package 1/2: openssl-unix:x64-linux
Building package openssl-unix[core]:x64-linux...
Could not locate cached archive: /home/damon/.cache/vcpkg/archives/a2/a2158e014fef4ae1cf65b8c0b9dff1aabb9e494f.zip
-- Downloading https://www.openssl.org/source/openssl-1.1.1g.tar.gz...
-- Extracting source /home/damon/.ome_local/repo/vcpkg/downloads/openssl-1.1.1g.tar.gz
-- Using source at /home/damon/.ome_local/repo/vcpkg/buildtrees/openssl-unix/src/1.1.1g-bb2a9a5801.clean
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/aes.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1_mac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asn1t.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/async.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/asyncerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bio.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bioerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/blowfish.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bn.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/bnerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/buffer.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/buffererr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/camellia.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cast.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cmac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cms.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cmserr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/comp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/comperr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/conf.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/conf_api.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/conferr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/crypto.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cryptoerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ct.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/cterr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/des.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dh.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dherr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dsa.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dsaerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/dtls1.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/e_os2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ebcdic.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ec.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ecdh.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ecdsa.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ecerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/engine.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/engineerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/evp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/evperr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/hmac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/idea.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/kdf.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/kdferr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/lhash.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/md2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/md4.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/md5.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/mdc2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/modes.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/obj_mac.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/objects.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/objectserr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ocsp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ocsperr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/opensslconf.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/opensslv.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ossl_typ.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pem.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pem2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pemerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs12.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs12err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs7.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/pkcs7err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rand.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rand_drbg.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/randerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rc2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rc4.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rc5.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ripemd.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rsa.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/rsaerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/safestack.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/seed.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/sha.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/srp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/srtp.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ssl.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ssl2.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ssl3.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/sslerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/stack.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/store.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/storeerr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/symhacks.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/tls1.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ts.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/tserr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/txt_db.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/ui.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/uierr.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/whrlpool.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509_vfy.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509v3.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/include/openssl/x509v3err.h
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl-unix_x64-linux/share/openssl-unix/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/damon/.cache/vcpkg/archives/a2/a2158e014fef4ae1cf65b8c0b9dff1aabb9e494f.zip
Building package openssl-unix[core]:x64-linux... done
Installing package openssl-unix[core]:x64-linux...
Installing package openssl-unix[core]:x64-linux... done
Elapsed time for package openssl-unix:x64-linux: 3.329 min
Starting package 2/2: openssl:x64-linux
Building package openssl[core]:x64-linux...
Could not locate cached archive: /home/damon/.cache/vcpkg/archives/b7/b78320e8f13a52578c025cb1bc4b3173cd4c3054.zip
-- Installing: /home/damon/.ome_local/repo/vcpkg/packages/openssl_x64-linux/share/openssl//usage
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/damon/.cache/vcpkg/archives/b7/b78320e8f13a52578c025cb1bc4b3173cd4c3054.zip
Building package openssl[core]:x64-linux... done
Installing package openssl[core]:x64-linux...
Installing package openssl[core]:x64-linux... done
Elapsed time for package openssl:x64-linux: 18.05 ms

Total elapsed time: 3.336 min

The package openssl is compatible with built-in CMake targets:

    find_package(OpenSSL REQUIRED)
    target_link_libraries(main PRIVATE OpenSSL::SSL OpenSSL::Crypto)

Package openssl:x64-linux is already installed.

@SeanTAllen
Copy link
Member

I dont see from reading this how it provides any value other than "can write things in a language other than Pony" but that opens up a mess of questions about dependencies for the other code. So to use a plugin, I might have to have bash installed. How does that work on Windows? Does the plugin need to also have a powershell version and a batch version?

At the moment, I don't see how if plugins are part of the corral repo that this is desirable given all the questions that would be raised versus writing the command in Pony.

@damon-kwok
Copy link
Author

Yes, bash is a required dependency. We can use the idea of OPAM( ocaml's package manager), which will automatically download msys2 on the windows platform. We can also use git-bash.

@SeanTAllen
Copy link
Member

A plugin system that isn't in Pony also bypasses the capabilities based directory security that using only Pony provides. Commands written in Pony using the existing corral system to add commands can be limited to working on a subset of directories. By including non-pony code from ambient auth languages, that goes out the window.

@SeanTAllen
Copy link
Member

Yes, bash is a required dependency. We can use the idea of OPAM( ocaml's package manager), which will automatically download msys2 on the windows platform. We can also use git-bash.

This is a huge requirement and one I'm not willing to take on lightly.

@SeanTAllen SeanTAllen added enhancement New feature or request needs discussion Needs to be discussed further labels Jul 16, 2020
@damon-kwok
Copy link
Author

The size of msys2 compressed package is about 50mb

@SeanTAllen
Copy link
Member

The size of msys2 compressed package is about 50mb

The size doesn't matter. We are asking people to take on an additional currently unneeded dependency for what benefit?

@damon-kwok
Copy link
Author

It allows users to use the common commands of unix to do anything.

@SeanTAllen
Copy link
Member

It allows users to use the common commands of unix to do anything.

What problem that exists now that people need solved does this address?

@damon-kwok
Copy link
Author

I want to add a plug-in system to corral. The plug-in system can help people achieve:

  • Handle native dependencies.
  • Custom automated tasks
  • ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request needs discussion Needs to be discussed further
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SeanTAllen @damon-kwok