Express Checkout buttons fail to to create an order if required custom field is present

[ecairol]

Describe the bug

If the Checkout page contains a required custom field, the HTTP Request to ?wc-ajax=wcpay_create_order will return a failed response, indicating that the custom field is required (even if it was populated).

This happens for Google Pay and Apple Pay buttons.

To Reproduce

1. Create a custom field using one of these hooks: woocommerce_checkout_fields, woocommerce_shipping_fields or woocommerce_billing_fields (find an example below *)
2. Add any product to your cart and go to the Checkout page
3. Make sure to populate all the required fields, including the custom field from step one.
4. Click on the Google Pay button (or Apple Pay if you have that configured)

• Example of a custom required field

function add_dummy_field( $fields ) {
	$fields['dummy_text'] = array(
		'label'       => __( 'Dummy required field', 'novos' ),
		'placeholder' => _x( 'Dummy required field', 'placeholder', 'novos' ),
		'required'    => true,
		'class'       => array( 'form-row-first' ),
		'clear'       => true,
	);
	return $fields;
}
add_filter( 'woocommerce_checkout_fields', 'add_dummy_field' );

Actual behavior

The HTTP Request to /?wc-ajax=wcpay_create_order returns: {"result":"failure","messages":"<ul class=\"woocommerce-error\" role=\"alert\">\n\t\t\t<li data-id=\"dummy_text\">\n\t\t\t<strong>Billing Dummy required field<\/strong> is a required field.\t\t<\/li>\n\t<\/ul>\n","refresh":false,"reload":false}

Screenshots

Expected behavior

The HTTP Request to create the order goes thru successfully.

Desktop (please complete the following information):

• OS: macOS
• Browser Chrome
• Version 113.0.0

Smartphone (please complete the following information):

• Device: N/A
• OS: N/A
• Browser N/A
• Version N/A

Additional context

[zmaglica]

This issue impacts express checkout, so assigning to team Heisenberg (based on team responsibilities) @bborman22 Assigning as part of Gamma Triage process PcreKM-yM-p2

[bborman22]

Just wanted to follow up @zmaglica that I think this should actually go to Fusion. Heisenberg team responsibilities is for the WooPay Express Checkout button, while Fusion covers Additional Payment Methods (including Apple Pay / Google Pay). Let me know if that makes sense or if I'm mistaken on that.

[Brianmitchtay]

Seeing this come up in 6766888-zen

[bborman22]

Sorry I hadn't followed up on this, but since we got another report of this, I just wanted to confirm my previous assumption that this would be better handled by @Automattic/fusion? We don't have much experience or exposure to Apple Pay and Google Pay and wouldn't want to jump into anything without gaining the additional context (for example long term plans) around this request.

[beaulebens]

Is this specific to WooPayments, or does it potentially impact other (all) payment gateways using express payment options?

[c-shultz]

After shifting team responsibilities, this is clearly now owned by Buyer Experience (and therefore Heisenberg). I'm adding this to the maintenance queue as a high priority.

[frosso]

To provide additional context, it appears that the problem can still manifest itself even when the "company" field is designated as "required" through the customizer. In other words, when we address this issue by incorporating custom fields, we must also ensure that we address it for other fields that can be marked as mandatory - this issue is not just related to custom fields.

[frosso]

Is this specific to WooPayments, or does it potentially impact other (all) payment gateways using express payment options?

@beaulebens I conducted a brief investigation, and it appears that the behavior may vary based on how each payment gateway is implemented.

For instance, with Apple Pay and Google Pay, they seize control of the checkout process by employing a 'modal' interface (although it's technically not a modal, the concept is similar, as it overlays the WooCommerce checkout). Unfortunately, this 'modal' experience doesn't permit the addition of custom fields.

I also conducted a cursory examination of WC Stripe, and the issue persists when using Apple Pay or Google Pay with that particular plugin.

This issue doesn't seem to present itself when Stripe Link is used, instead.

[c-shultz]

they seize control of the checkout process by employing a 'modal' interface (although it's technically not a modal, the concept is similar, as it overlays the WooCommerce checkout). Unfortunately, this 'modal' experience doesn't permit the addition of custom fields.

Given @frosso's findings, we're going going to need to take a wider look at how WooPayments, Stripe (and likely other gateways) handle compatibility with express checkout methods that take over the checkout flow. By design, these payment methods hijack/streamline the checkout flow, so they inherently conflict with extensions that introduce new required fields in the checkout form.

I would say our quickest fix here will be to better educate merchants when known conflicting extensions are installed (e.g. "Checkout Field Editor is active on your site and may introduce required fields that conflict with Google Pay"). For sites using the Cart and Checkout Blocks (defaulting on in WooCommerce 8.3), these incompatibilities are highlighted in the site editor (see: pdFofs-1qY-p2). Aside from merchant education, a fix here would have to modify the checkout flow in a way that ensures shoppers have entered this required data before starting an express checkout flow.

I'm marking this as blocked pending product/design feedback @pierorocca @nikkivias

I'll also note that Rubik is re-working custom fields to be introduced in a more block-friendly way (see: pdFofs-1tU-p2). This may not solve anything for most express checkout methods (the notable exception would be WooPay, our first-party express checkout method, since we have full control of that experience and can integrate it with these new APIs).

[pierorocca]

Great detail. I do have an open question. AFAIK we're using the legacy elements implementation for PRBs. Does a Stripe PE or Express Checkout integration provide more customization flexibility that's not possible with legacy elements?

[beaulebens]

I want to loop @manospsyx in on this one, because he recently raised the idea that we need a consistent API across all Express Checkout methods, presumably implemented in Core (in the context of being able to disable them all in certain environments).

[manospsyx]

we're going going to need to take a wider look at how WooPayments, Stripe (and likely other gateways) handle compatibility with express checkout methods that take over the checkout flow. By design, these payment methods hijack/streamline the checkout flow, so they inherently conflict with extensions that introduce new required fields in the checkout form.

we need a consistent API across all Express Checkout methods, presumably implemented in Core (in the context of being able to disable them all in certain environments)

It's a very common headache for Woo plugin developers: Depending on the context (product/cart/checkout), a product type or feature (or a required form field in a product/checkout context) may need to suppress Express Checkout functionality in a payment gateway-agnostic manner. Not having such an API in Core means that

• plugin developers are forced implement a gateway-dependent solution in every context, which doesn't scale :(
• gateway developers have no way to make express checkout functionality conditionally available in an environment-agnostic way.

Lack of an API in WooCommerce Core leads to the usual difficulty to determine which side should be expected to act when a conflict like this is identified.

[manospsyx]

we need a consistent API across all Express Checkout methods, presumably implemented in Core

For instance, with Apple Pay and Google Pay, they seize control of the checkout process by employing a 'modal' interface (although it's technically not a modal, the concept is similar, as it overlays the WooCommerce checkout). Unfortunately, this 'modal' experience doesn't permit the addition of custom fields.

I'm not sure if it's the right place or time to talk about a solution or details -- lack of this API in Core also makes it impossible to validate product/checkout fields and the cart state before a gateway goes ahead to process a payment.

A complete solution in Woo Core would probably consist of:

• An API to retrieve / validate product and cart data, responsible for green-lighting express payment code in different contexts and passing error codes/details in a standardized manner. Could the Store API play this part already? Usually product form data is ignored, and cart contents not validated by express payment button implementations I've worked with.
• An API (filter/hook/extend-able) to control the visibility of express payment buttons in product/cart/checkout blocks (possibly decoupled from the one above).

PS: In the past, I found this payment button implementation to be the most robust one and the best candidate for inclusion in a "Core API" (product context). The code was equally well-mannered in the cart/checkout. Check out how it relays form field validation to Woo Core -- behind the scenes, and if I remember correctly, it "simulates" an add-to-cart operation and validates cart contents. Imagine having a shared foundation like this in Core for all express checkout buttons to use.

[pierorocca]

Appreciate the holistic look at the problem and potential solution. Is this a near time architectural change or a longer term vision? Want to make sure we address the immediate need while seriously contemplating the long term.

[manospsyx]

Opened an issue here for reference.

@pierorocca if this problem statement/definition is correct, then I think that we could definitely approach the "express payment buttons visibility" problem as a near-time change.

We could, for example, make some backwards-compatible additions in the abstract payment gateway class to introduce a standard way for gateways to:

• Declare support for express checkout functionality / buttons in product/cart/checkout context.
• Let users enable/disable express payment buttons in the supported context(s), via the Woo Settings API, which is used to populate these options:

I think that the main challenge here is not so much the engineering effort involved in providing a common API and UI options for controlling payment buttons visibility, but rather the effort to educate and encourage partners and developers to adopt/utilize this quickly. From a backwards compatibility perspective, I think it would be a relatively painless change for most.

we could definitely approach the "express payment buttons visibility" problem as a near-time change

I can't say the same for the ideas I shared around a "validation API" for express payment buttons: That would require more study and effort to implement in Core, and of course some work to educate developers and encourage adoption in the payment extensions of our first-tier partners.

[frosso]

Seeing some discussion in #5018 as well

[pierorocca]

Thanks for the excellent writeup in woocommerce/woocommerce#41242 @manospsyx! Super helpful. Love your thought process.

As I started writing a response and thinking through this, custom fields aside, this is less about express checkout buttons and more related to core Add to cart, Proceed to checkout, and Place order buttons and checkout extensibility and interoperability. Express buttons are effectively representations of the core buttons and follow their behavior for visibility and state. More express checkout (and APMs, LPMs. etc) acceptance is a better aspirational goal than suppression and less acceptance. Bear with me...

For example today on the product page:

• Product variations that haven't met the conditions - all buttons are visible and enabled, will throw an alert dialog on click
• no product inventory, no backorder - all buttons are unavailable/hidden
• no inventory, backorder allowed - all buttons are visible and enabled
• pre-order - all buttons visible and enabled

What use cases exist where the Add to cart and Express Button visibility would be different from each other, setting aside downstream checkout incompatibility?

When the product is purchasable and an express checkout is initiated, an extension like mix & match, product bundles, and others may lead to misleading cart summaries and a poor shopper UX. Is the right investment to at scale, suppress express checkouts (which convert higher and are preferred by shoppers) or to invest in better interoperability and more express checkouts? On WooPay we've invested in the latter, in a custom manner out of necessity.

a product may not be purchasable by the current user (e.g. memberships) or under the current conditions (e.g. inventory)

For the inventory use case and the membership example, I don't see a difference between the core buttons and the express buttons in terms of visibility. If a conditional applies to the core button, it would apply equally to the express button. Same question, is there a use case where the core and express buttons would have different behavior from each other, setting aside checkout incompatibility?

If not, then for me this is a question about how do 3PDs and express buttons (also 3rd party to core) hook into core consistently, predictably, and in an interoperable way rather than how do 3PDs manage other 3PDs' express checkout buttons?

Thoughts? What are my blind spots?

---

Custom Fields
Having looked at the Checkout Field Editor data that Raluca had pulled earlier in the year, there was a small set of use cases that were truly bespoke. Most of the custom data I noted was created to fix core issues with localization and missing use cases like tipping, gifting and gift messages, and tax IDs. For those use cases it's preferred to add what's commonly missing than wrap more complexity around it. I'll keep being a broken record on that position.

For legitimate bespoke needs, I'd love to see an approach that allows fields to be added and accessible to core and any extension. WooPay, GPay, Apple Pay for example should be able to access the data, not get suppressed as a checkout option because data access isn't available.

It's a very common headache for Woo plugin developers: Depending on the context (product/cart/checkout), a product type or feature (or a required form field in a product/checkout context) may need to suppress Express Checkout functionality in a payment gateway-agnostic manner. Not having such an API in Core means that

Being on rotation week has me on a heightened level of sensitivity to a merchant's, not a developer's, plight of dealing with the complexity of setting up a store. I could see a merchant setting up Apple Pay and Google Pay and installs an extension that suppresses express checkouts and then wonders why it's not working. Having to tell a merchant to run conflict testing or something doesn't work because of A through Z incompatibilities doesn't feel great.

Apologies for the long brain dump here. Hopefully I'm not off on a tangent.

[manospsyx]

What use cases exist where the Add to cart and Express Button visibility would be different from each other, setting aside downstream checkout incompatibility?

I think it's very reasonable to guide express payment button developers around placement. Right now we don't even provide that guidance. Note however, that:

1. There is no way to enforce this.
2. Assuming they (developers) adhere to our placement guidelines, visibility (or "state", e.g. greyed-out) is not enough to control form submission: A tech savvy user can easily simulate a click by writing some JS in the browser, or manually un-hide and make those buttons click-able in the Inspector. This points back to the whole idea of requiring an API to handle both payment button visibility/state and validation.
3. On the subject of visibility, the assumption that a common container will always exist to have Woo Core automatically control the visibility of payment buttons doesn't hold, esp when using "classic themes". Blocks (e.g. add-to-cart, proceed to checkout, place order) generally make it easier to enforce/encourage placement and control visibility at the same time, but blocks can't fix (2).
4. To add another angle to the idea to have Core control visibility/state -- some payment solution providers / partners may object to the idea of allowing Core and/or plugin code to control visibility/state for their payment buttons, and insist that their payment buttons always be rendered/visible in a prominent/functional state, consistent site-wide (e.g. same placement in all product pages) for obvious reasons. That is, of course, unless the merchant has decided that they don't want to expose express payment buttons in a certain context, at all. It would be unrealistic of them to demand that (i) their merchants (and/or developers) have no way to suppress payment buttons universally, or that (ii) no validation is done on form submission at all (as that leads to incorrect orders being submitted, e.g. with missing required fields). Both require an API in Core.

[manospsyx]

On a related note --

In the distant future, when/if all of the WooCommerce front-end is React-driven, then I suspect that what you describe here would be more enforce-able; e.g. consider how the block-based checkout is fully able to aggregate (and control the visibility/function of) express payment buttons. In fact, the block-based checkout may already be able to validate those payment requests via the Store API before relaying payment to the payment gateway (not sure about this).

[pierorocca]

A tech savvy user can easily simulate a click by writing some JS in the browser, or manually un-hide and make those buttons click-able in the Inspector. This points back to the whole idea of requiring an API to handle both payment button visibility/state and validation.

This makes a lot of sense to have the platform be the arbiter/orchestrator rather than leaving it up to each party to come up with their own implementation, own guidelines.

[pierorocca]

@bborman22 for this error condition, do you have a sense of what error is displayed to the shopper today, if any? Knowing current state would help determine the design and copy required to help steer the shopper to a different payment method. Many thanks.

[bborman22]

@pierorocca I'm not sure what is displayed, but will get a demo of this in the next couple days to get a better sense of where it's at right now.

[asumaran]

I'd like to leave a comment with the latest findings regarding this issue. I attempted to reproduce the bug using the following code:

add_filter( 'woocommerce_checkout_fields' , 'custom_override_checkout_fields' );

// Our hooked in function - $fields is passed via the filter!
function custom_override_checkout_fields( $fields ) {
     $fields['billing']['billing_myfield'] = array(
        'label'       => __( 'My field', 'woocommerce' ),
        'placeholder' => _x( 'My field', 'placeholder', 'woocommerce' ),
        'required'    => true,
        'class'       => array( 'form-row-wide' ),
        'clear'       => true
     );

     return $fields;
}

I can confirm that all instances where ECE is available fail with the error message: ‘Billing My field is a required field.’

However, considering that the Tokenized Cart implementation for ECE will soon become the default, I tested it as well. In this case, the checkout process with ECE worked without issues on the Product Page, as well as on both Block-based and shortcode-based Checkout and Cart pages.

While this is a positive outcome, it may not be ideal if the field is indeed required for the merchant.

[pierorocca]

While this is a positive outcome, it may not be ideal if the field is indeed required for the merchant.

Thanks @asumaran. I had to read the bug title twice. Double checking as the original requestor indicated WITH the custom field present on the form, the order fails. Is that field populated and passed in your testing or it's passing despite the field data not being submitted?

[asumaran]

I forgot to mention that the code above adds the additional field only to the shortcode-based Checkout. Therefore, the field is only required on that page. In the block-based Checkout, there is no validation for this field as it's not present in the form.

Is that field populated and passed in your testing or it's passing despite the field data not being submitted?

@pierorocca with Tokenized Carts it's passing despite the additional custom field not being submitted. Tokenized ECE and the Checkout block use the Store API which seems to ignore these fields. That’s why Tokenized ECE works correctly on all pages, including the shortcode-based Checkout, as it relies on the Store API, which ignores the requirement for the additional field.

[pierorocca]

Thanks @asumaran for the extra context. What happens if the additional field in Blocks is added using the Additional Fields API? Does the Store API then care about the required field?

[asumaran]

@pierorocca Based on this documentation page, I was able to add a custom field to the block-based checkout form using the following code:

 add_action(
	'woocommerce_init',
	function() {
		woocommerce_register_additional_checkout_field(
			array(
				'id'            => 'namespace/my-field',
				'label'         => 'My Field',
				'optionalLabel' => 'My Field (optional)',
				'location'      => 'address',
				'required'      => true,
			),
		);
	}
);

The field is validated, and the block-based checkout form fails to submit if no data is provided for the additional field. Tokenized ECE also fails, returning the following response:

{
    "code": "woocommerce_rest_checkout_missing_required_field",
    "message": "There was a problem with the provided billing address: My Field is required",
    "data": {
        "status": 400
    }
}

It seems that the Store API is now validating the presence of the field.

However, just to note, the checkout process works fine in the current ECE implementation