[token-cli] upgrade to clap-v3

[samkim-crypto]

Problem

There has been useful updates to solana-clap-v3-utils in the monorepo, but the token-cli still uses clap-v2. For example:

• [token-cli] Some token-cli commands panics when --nonce specified #4097 should be resolved by switching to clap-v3-utils with version 1.16.0
• Support for encryption keys needed for token-2022 is only available in solana-clap-v3-utils

Proposed changes

Upgrade token-cli to use clap v3 and solana-clap-v3-utils in a sequence of steps suggested in the clap change logs.

Migrating
From clap v2

1. Add CLI tests, -h and --help output at a minimum (recommendation: trycmd for snapshot testing)
2. Update your dependency
   i. If you use no-default-features: add the std feature
3. Resolve compiler errors
4. Resolve behavior changes
   i. Refactor your App creation to a function and add a test similar to the one below, resolving any of its assertions
   ii. Look over the "subtle changes" under BREAKING CHANGES
   iii. If using builder: test your application under various circumstances to see if ArgMatches asserts regarding AllowInvalidUtf8.
5. At your leisure: resolve deprecation notices

Status:

• [token-cli] upgrade to clap-v3 #4506 implements steps 1-4 above
• Step 5 can be implemented in follow-up PRs that are split in digestible chunks

[samkim-crypto]

From clap change logs v3.0.0:

Subtle changes (i.e. compiler won't catch):

• AppSettings::UnifiedHelpMessage is now default behaviour
  • {flags} and {unified} will assert if present in App::help_template
  • See Default help grouping is confusing and unwanted clap-rs/clap#2807
  • COMPLETED: this is a cosmetic change and is acceptable behavior (description of UnifiedHelpMessage)
• AppSettings::EnableColoredHelp is now the default behavior but can be opted-out with AppSettings::DisableColoredHelp (User confusion over help messages not being colored, despite color being enabled clap-rs/clap#2806)
  • COMPLETED: this is a cosmetic change and is acceptable behavior)
• App::override_usage no longer implies a leading \t, allowing multi lined usages
  • COMPLETED: App::override_usage is not used in the token-cli
• Arg::require_equals no longer implies ArgSettings::ForbidEmptyValues (Multiple set to false fails clap-rs/clap#2233)
  • COMPLETED: Arg::require_equals is not used in the token-cli
• Arg::require_delimiter no longer implies ArgSettings::TakesValue and ArgSettings::UseValueDelimiter (Multiple set to false fails clap-rs/clap#2233)
  • COMPLETED: Arg::require_delimiter is not used in the token-cli
• Arg::env, Arg::env_os, Arg::last, Arg::require_equals, Arg::allow_hyphen_values, Arg::hide_possible_values, Arg::hide_default_value, Arg::hide_env_values, Arg::case_insensitive and Arg::multiple_values no longer imply ArgSettings::TakesValue (Multiple set to false fails clap-rs/clap#2233)
  • COMPLETED: None of Arg::* above are used in the token-cli
• ArgMatches::is_present no longer checks subcommand names
  • COMPLETED: ArgMathces::is_present is never used on subcommand names in the token-cli
• Some env variable values are now considered false for flags, not just "not-present" (Change behavior of flag arguments specified as env vars clap-rs/clap#2539)
• Changed ...s meaning in usage parser. Before, it always meant multiple which is still true for --option [val].... Now [name]... --option [val] results in ArgSettings::MultipleOccurrences.
  • COMPLETED: ... is not used in the token-cli
• Usage exit code changed from 1 to 2 (Use precise exit code on usage error clap-rs/clap#1327)
• Reject --foo=bar when takes_value(false) (Arguments with takes_value(false) still accept "compact"-style arguments clap-rs/clap#1543)
  • COMPLETED: this is a bug fix and is acceptable behavior
• No longer accept an arbitrary number of - for long arguments (-----long)
  • COMPLETED: acceptable behavior

[samkim-crypto]

From clap change logs v3.0.0:

Easier to catch changes:

• When using no-default-features, you now have to specify the std feature (reserved for future work)
• Gated env support behind env feature flag
  • Impacts Arg::env, Arg::env_os, Arg::hide_env_values, ArgSettings::HideEnvValues
  • See Add env feature gate clap-rs/clap#2694
• Gated crate information behind cargo feature flag
  • Impacts crate_name!, crate_version!, crate_authors!, crate_description!, app_from_crate!
• AppSettings::StrictUtf8 is now default behaviour and asserts if AppSettings::AllowInvalidUtf8ForExternalSubcommands and ArgSettings::AllowInvalidUtf8 and ArgMatches::value_of_os aren't used together
  • AppSettings::AllowInvalidUtf8 has been removed
  • use cases for value_of/values_of clap-rs/clap#751
• Arg::short and Arg::value_delimiter now take a char instead of a &str
  • COMPLTED in [token-cli] upgrade to clap-v3 #4506 commit: d93af20
• ArgMatches panics on unknown arguments
• Removed VersionlessSubcommands, making it the default (see Please default to DisableVersionForSubcommands clap-rs/clap#2812)
• Completion generation has been split out into clap_complete.
• Removed ArgSettings::EmptyValues in favor of ArgSettings::ForbidEmptyValues
• Validator signatures have been loosed:
  • Arg::validator now takes first argument as Fn(&str) -> Result<O, E: ToString> instead of Fn(String) -> Result<(), String>
  • Arg::validator_os now takes first argument as Fn(&OsStr) -> Result<O, OsString> instead of Fn(&OsStr) -> Result<(), OsString>
  • COMPLETED in [token-cli] upgrade to clap-v3 #4506 commit: 3171490
• Arg::value_name now sets, rather than appends (see value_name appends, rather than overrides clap-rs/clap#2634)
• Upgrade yaml-rust from 0.3 to 0.4
• Replaced ArgGroup::from(BTreeMap) to ArgGroup::from(yaml)
• Replaced ArgMatches::usage with App::generate_usage
• Replaced Arg::settings with Arg::setting(Setting1 | Setting2)
• App and Arg now need only one lifetime
  • COMPLETED in [token-cli] upgrade to clap-v3 #4506 commits: 0062096, c296c2e
• Removed deprecated App::with_defaults, replaced with app_from_crate
• Removed deprecated AppSettings::PropagateGlobalValuesDown (now the default)
• Some App functions, like App::write_help now take &mut self instead of &self
• Error::message is now private, use Error::to_string
• Arg::default_value_if, Arg::default_value_if_os, Arg::default_value_ifs, Arg::default_value_ifs_os now takes the default value parameter as an option (unset default value if some other Arg is present (or has a specific value) clap-rs/clap#1406)
• Changed App::print_help & App::print_long_help to now return std::io::Result
• Changed App::write_help & App::write_long_help to now return std::io::Result
• Changed Arg::index, Arg::number_of_values, Arg::min_values, Arg::max_values to taking usize instead of u64
  • COMPLETED in [token-cli] upgrade to clap-v3 #4506 commit: 3273d86
• Changed Error::info to type Vec<String> instead of Option<Vec<String>>
• Changed ArgMatches::subcommand to now return Option<(&str, &ArgMatches)>
  • COMPLETED in [token-cli] upgrade to clap-v3 #4506 commit: 6f851be
• Renamed ErrorKind::MissingArgumentOrSubcommand to ErrorKind::DisplayHelpOnMissingArgumentOrSubcommand
• Renamed ErrorKind::HelpDisplayed to ErrorKind::DisplayHelp
• Renamed ErrorKind::VersionDisplayed to ErrorKind::DisplayVersion
• Added #[non_exhaustive] to clap::{ValueHint, ErrorKind, AppSettings, ArgSettings} (fix!: Allow adding new enum variants clap-rs/clap#3167)

[samkim-crypto]

From clap change logs v3.1.0:

Compatibility:
Changes in behavior of note that are not guaranteed to be compatible across releases:

• (help) help subcommand shows long help like --help, rather than short help (-h), deprecated clap::AppSettings::UseLongFormatForHelpSubcommand (Add example of deserializing the SlotHash sysvar while remaining with… #3440)
• (help) Pacman-style subcommands are now ordered the same as usage errors (spl-token: error while loading shared libraries: libssl.so.1.1 #3470)
• (help) Pacman-style subcommands use standard alternate syntax in usage (spl-token: error while loading shared libraries: libssl.so.1.1 #3470)

Deprecations:

• clap::Command is now preferred over clap::App (Remove duplciate output of 'Stake Accounts' header #3089 in spl-token js: Unpack methods are no exposed to clients #3472)
  • clap::command! is now preferred over clap::app_from_crate (Remove duplciate output of 'Stake Accounts' header #3089 in build(deps-dev): bump @solana/web3.js from 1.36.0 to 1.53.0 in /token-lending/js #3474)
  • clap::CommandFactory::command is now preferred over clap::IntoApp::into_app (Remove duplciate output of 'Stake Accounts' header #3089 in build(deps-dev): bump @types/node from 16.11.21 to 18.7.3 in /token/js #3473)
• (help) help subcommand shows long help like --help, rather than short help (-h), deprecated clap::AppSettings::UseLongFormatForHelpSubcommand (Add example of deserializing the SlotHash sysvar while remaining with… #3440)
• (error) Deprecate clap::AppSettings::WaitOnError, leaving it to the user to implement
• (validation) clap::Command::subcommand_required(true).arg_required_else_help(true) is now preferred over clap::AppSettings::SubcommandRequiredElseHelp (token-2022: add tests for withdraw of confidential withheld tokens from accounts #3280)
• (builder) clap::AppSettings are nearly all deprecated and replaced with builder methods and getters (Fixup tests to use rent-exempt accounts #2717)
• (builder) clap::ArgSettings is deprecated and replaced with builder methods and getters (Fixup tests to use rent-exempt accounts #2717)
• (builder) clap::Arg::id and clap::ArgGroup::id are now preferred over clap::Arg::name and clap::ArgGroup::name ([stake-pool] instruction to add metadata for pool token #3335)
• (help) clap::Command::next_help_heading is now preferred over clap::Command::help_heading (build(deps-dev): bump @types/eslint from 7.2.10 to 7.2.12 in /token-lending/js #1807, spl-token-cli: Simplify owner resolution #1553)
• (error) clap::error::ErrorKind is now preferred over clap::ErrorKind (stake-pool-py: Fix tests for Solana 1.10.33 #3395)
• (error) clap::Error::kind() is now preferred over clap::Error::kind
• (error) clap::Error::context() is now preferred over clap::Error::info (Bump Rust version to 1.57.0 #2628)

[samkim-crypto]

From clap change logs v3.2.0:

Compatibility
MSRV is now 1.56.0 (#3732)

Behavior

• Defaults no longer satisfy required and its variants (build(deps-dev): bump @types/node from 18.7.10 to 18.11.9 in /token/js #3793)
• When misusing ArgMatches::value_of and friends, debug asserts were turned into panics

Moving (old location deprecated)

• clap::{PossibleValue, ValueHint} to clap::builder::{PossibleValue, ValueHint}
• clap::{Indices, OsValues, ValueSource, Values} to clap::parser::{Indices, OsValues, ValueSource, Values}
• clap::ArgEnum to clap::ValueEnum (token-2022: Amount Casting - Imprecision/Overflow - HAL-05 #3799)

Replaced

• Arg::allow_invalid_utf8 with Arg::value_parser(value_parser!(PathBuf)) (build(deps-dev): bump @types/node from 16.11.14 to 18.11.5 in /token-lending/js #3753)
• Arg::validator / Arg::validator_os with Arg::value_parser (build(deps-dev): bump @types/node from 16.11.14 to 18.11.5 in /token-lending/js #3753)
• Arg::validator_regex with users providing their own builder::TypedValueParser (revert CpiGuard reversion #3756)
• Arg::forbid_empty_values with builder::NonEmptyStringValueParser / builder::PathBufValueParser (build(deps-dev): bump @types/node from 16.11.14 to 18.11.5 in /token-lending/js #3753)
• Arg::possible_values with Arg::value_parser([...]), builder::PossibleValuesParser, or builder::EnumValueParser (build(deps-dev): bump @types/node from 16.11.14 to 18.11.5 in /token-lending/js #3753)
• Arg::max_occurrences with arg.action(ArgAction::Count).value_parser(value_parser!(u8).range(..N)) for flags (API request #3797)
• Arg::multiple_occurrences with ArgAction::Append or ArgAction::Count though positionals will need Arg::multiple_values (governance: unify final proposal state as Completed for proposal with no transaction inserted #3772, API request #3797)
• Command::args_override_self with ArgAction::Set (build(deps): bump prismjs from 1.20.0 to 1.25.0 in /docs #2627, API request #3797)
• AppSettings::NoAutoVersion with ArgAction or Command::disable_version_flag (token-cli: Specify that default-account-state takes a value #3800)
• AppSettings::NoHelpVersion with ArgAction or Command::disable_help_flag / Command::disable_help_subcommand (token-cli: Specify that default-account-state takes a value #3800)
• ArgMatches::{value_of, value_of_os, value_of_os_lossy, value_of_t} with ArgMatches::{get_one,remove_one} (build(deps-dev): bump @types/node from 16.11.14 to 18.11.5 in /token-lending/js #3753)
• ArgMatches::{values_of, values_of_os, values_of_os_lossy, values_of_t} with ArgMatches::{get_many,remove_many} (build(deps-dev): bump @types/node from 16.11.14 to 18.11.5 in /token-lending/js #3753)
• ArgMatches::is_valid_arg with ArgMatches::{try_get_one,try_get_many} (build(deps-dev): bump @types/node from 16.11.14 to 18.11.5 in /token-lending/js #3753)
• ArgMatches::occurrences_of with ArgMatches::value_source or ArgAction::Count (API request #3797)
• ArgMatches::is_present with ArgMatches::contains_id or ArgAction::SetTrue (API request #3797)
• ArgAction::StoreValue with ArgAction::Set or ArgAction::Append (API request #3797)
• ArgAction::IncOccurrences with ArgAction::SetTrue or ArgAction::Count (API request #3797)
• (derive) #[clap(parse(...))] replaced with: (Concurrent Merkle Tree: Improve Docs #3589, build(deps-dev): bump @types/node from 16.11.14 to 18.11.9 in /token-lending/js #3794)
  • For default parsers (no parse attribute), deprecation warnings can be silenced by opting into the new behavior by adding either #[clap(action)] or #[clap(value_parser)] (ie requesting the default behavior for these attributes). Alternatively, the unstable-v4 feature changes the default away from parse to action/value_parser.
  • For #[clap(parse(from_flag))] replaced with #[clap(action = ArgAction::SetTrue)] (build(deps-dev): bump @types/node from 16.11.14 to 18.11.9 in /token-lending/js #3794)
  • For #[clap(parse(from_occurrences))] replaced with #[clap(action = ArgAction::Count)] though the field's type must be u8 (build(deps-dev): bump @types/node from 16.11.14 to 18.11.9 in /token-lending/js #3794)
  • For #[clap(parse(from_os_str)] for PathBuf, replace it with #[clap(value_parser)] (as mentioned earlier this will call value_parser!(PathBuf) which will auto-select the right ValueParser automatically).
  • For #[clap(parse(try_from_str = ...)], replace it with #[clap(value_parser = ...)]
  • For most other cases, a type implementing TypedValueParser will be needed and specify it with #[clap(value_parser = ...)]