Hacky Support for /as4 endpoint due to platform dependent line endings

[piotr-dajlido]

Problem Description

There are some issues with sending direct requests to the default /as4 endpoint. Requests with a Content-Type: multipart/related are parsed using a copy of the org.apache.tomcat.util.http.fileupload.MultipartStream class, which is utilized by AS4IncomingHandler.class. The MultipartStream class adheres to RFC 1867, which specifies that multipart part boundaries are delimited by double CRLF line endings. This causes a StreamClosedUnexpectedly error when the payload is sent with LF (Linux) or CR (Mac) line endings.

Additionally, there is no guarantee that some RequestFilters might not be rewriting the request payload's InputStream to convert CRLF to LF, which depends on the framework being used. In my case, disabling security and turning off all default Spring filters did not resolve the issue.

Hacky Solution

To address this issue, I created the following class:

public class CarriageReturnInputStream extends InputStream {
    private final InputStream originalStream;
    private final ByteArrayOutputStream bufferStream;
    private int bufferPos;
    private boolean previousWasCarriageReturn;

    public CarriageReturnInputStream(InputStream originalStream) {
        this.originalStream = originalStream;
        this.bufferStream = new ByteArrayOutputStream();
        this.bufferPos = 0;
        this.previousWasCarriageReturn = false;
    }

    @Override
    public int read() throws IOException {
        if (bufferPos < bufferStream.size()) {
            return bufferStream.toByteArray()[bufferPos++] & 0xFF;
        }

        int byteRead = originalStream.read();
        if (byteRead == -1) {
            return -1;
        }

        bufferStream.reset();
        bufferPos = 0;

        if (byteRead == '\n' && !previousWasCarriageReturn) {
            bufferStream.write('\r');
            bufferStream.write('\n');
        } else {
            bufferStream.write(byteRead);
        }

        previousWasCarriageReturn = (byteRead == '\r');
        return bufferStream.toByteArray()[bufferPos++] & 0xFF;
    }
}

This class wraps the InputStream used in AS4IncomingHandler during multipart/related deserialization (around line 213 of AS4IncomingHandler.class):

...
try (final InputStream aRequestIS = new CarriageReturnInputStream(
        AS4DumpManager.getIncomingDumpAwareInputStream(aRealIncomingDumper,
                                                       aPayloadIS,
                                                       aMessageMetadata,
                                                       aHttpHeaders,
                                                       aDumpOSHolder))) {
    ...
}

Moreover requests from Postman (Windows) work fine, problem starts when triggered from for e.g. Linux container

[phax]

@piotr-dajlido well, phase4 is designed independent of any specific application server, Spring framework and the like. Therefore, you need to make sure, that you don't add any specific application server filtering on that servlet - it contains it all.

My assumption is, that you added the /as4 servlet statically into you web.xml (or the like) which then gets picked up Tomcat which applies the default filters defined in your web.xml. That's the reason, why the /as4 servlet in phase4-peppol-standalone gets added dynamically: https://github.com/phax/phase4-peppol-standalone/blob/main/src/main/java/com/helger/phase4/peppolstandalone/servlet/ServletConfig.java#L101
this seem to prevent Tomcat from interfering with any Servlet filters.
See also the code a few lines above that, which explicitly disables multipart handling on the servlet layer:

 // Multipart is handled specifically inside
      settings ().setMultipartEnabled (false);

[piotr-dajlido]

@phax I am currently using this implementation: https://github.com/phax/phase4-peppol-standalone/blob/main/src/main/java/com/helger/phase4/peppolstandalone/servlet/ServletConfig.java#L101

The setting settings().setMultipartEnabled(false); doesn't make any difference. I have disabled filters, but the problem is much broader than Phase4. I'm highlighting that many of these low-level implementations, which are beyond the scope of any framework, could be problematic in the future.

Right now, I'm struggling with the application.properties file:

mock.sender.address=${MOCK_OUTGOING_ADDRESS:outgoing}
mock.destination.address=${MOCK_DESTINATION_ADDRESS:http://localhost:8080/incoming}
phase4.endpoint.address=${MOCK_INCOMING_ADDRESS:http://localhost:8080/incoming}
phase4.dump.path=${PHASE4_DUMP_PATH:generated/}

The parser provided in the library doesn't resolve ENV variables, which is a pretty standard practice nowadays. I need this to enable elastic configuration for containers deployed via Helm charts. Instead, it interprets MOCK_OUTGOING_ADDRESS:outgoing literally, which causes issues since MOCK_OUTGOING_ADDRESS: is an unexpected part.

@phax Is there a way to provide a custom implementation of the properties resolver? I would like to use the @Value("some.property") annotation for the entire configuration.

@phax I assue that extending AbstractAS4CryptoFactory might do the trick here

[phax]

@piotr-dajlido thanks for your comment, lets see:

• well, the settings().setMultipartEnabled(false); really doesn't make a difference - it only affects multipart/form-data content types
• The issue is, that AS4 messgaes can have different layouts - they can be simple SOAP messages or they can adhere to the SOAP with attachments specification, which maps to the MIME format. Both messages are sent to the same servlet, and it must be decided on the fly, what parser to use. The differentiation is handled there:

  phase4/phase4-lib/src/main/java/com/helger/phase4/servlet/AS4IncomingHandler.java

  Line 204 in a17024d

  if (aPlainContentType.equals (AS4RequestHandler.MT_MULTIPART_RELATED))

  As such, you should really not add any kind of filtering/mods around the input streams.
• Regarding the configuration: as I am not building on Spring, I created my own configuration backend system. The decription can be found at https://github.com/phax/ph-commons/wiki/ph-config
  • Currently it doesn't support the provision of "fallback values" like generated/ in ${PHASE4_DUMP_PATH:generated/}
  • The resolution of environment variables works without any issue. Just don't add any prefix or suffix to the var name (see above).
  • If you mix it with Spring application.properties processing, please find the smallest common denominator, or put the phase4 specific properties in a file call phase4.properties which has precedence over application.properties. However, I must assume you know your framework well enough to know how to :)
• And of course you are free to provide your own implementations of IAS4CryptoFactory - you don't have to rely on the one using the properties from the configuration file. See e.g. AS4CryptoFactoryInMemoryKeyStore where you can provide a ready-made KeyStore object (from any source) and use that instead

hth

[piotr-dajlido]

My Solution to property resolution.

@Component
public class AS4CryptoProperties extends AbstractAS4CryptoFactory {

 @Autowired
    public AS4CryptoProperties(Environment env) {
        CRYPTO_PROVIDER = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.CRYPTO_PROVIDER);
        KEYSTORE_TYPE = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.KEYSTORE_TYPE);
        KEYSTORE_FILE = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.KEYSTORE_FILE);
        KEYSTORE_PASSWORD = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.KEYSTORE_PASSWORD);
        KEY_ALIAS = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.KEY_ALIAS);
        KEY_PASSWORD = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.KEY_PASSWORD);
        LOAD_CACERTS = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.LOAD_CACERTS);
        TRUSTSTORE_FILE = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_FILE);
        TRUSTSTORE_PASSWORD = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_PASSWORD);

        /*
         * The provider used to load truststores. By default, it’s the same as the keystore provider.
         * Set to an empty value to force use of the JRE’s default provider.
         */
        TRUSTSTORE_PROVIDER = Optional.fromNullable(env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_PROVIDER)).orNull();
        TRUSTSTORE_TYPE = env.getProperty(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_TYPE);

        m_aKeyStore = KeyStoreHelper.loadKeyStore(Objects.requireNonNull(EKeyStoreType.getFromIDCaseInsensitiveOrDefault(KEYSTORE_TYPE, DEFAULT_KEYSTORE_TYPE)),
                        KEYSTORE_FILE,
                        KEYSTORE_PASSWORD)
                .getKeyStore();

        m_aTrustStore = KeyStoreHelper.loadKeyStore(Objects.requireNonNull(EKeyStoreType.getFromIDCaseInsensitiveOrDefault(TRUSTSTORE_TYPE, DEFAULT_TRUSTSTORE_TYPE)),
                        TRUSTSTORE_FILE,
                        TRUSTSTORE_PASSWORD)
                .getKeyStore();

        properties = new Properties();
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.CRYPTO_PROVIDER, CRYPTO_PROVIDER);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.KEY_ALIAS, KEY_ALIAS);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.KEY_PASSWORD, KEY_PASSWORD);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.KEYSTORE_FILE, KEYSTORE_FILE);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.KEYSTORE_PASSWORD, KEYSTORE_PASSWORD);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.KEYSTORE_TYPE, KEYSTORE_TYPE);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.LOAD_CACERTS, LOAD_CACERTS);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_FILE, TRUSTSTORE_FILE);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_PASSWORD, TRUSTSTORE_PASSWORD);
        properties.put(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_TYPE, TRUSTSTORE_TYPE);

        if (Objects.nonNull(TRUSTSTORE_PROVIDER)) {
            properties.put(com.helger.phase4.crypto.AS4CryptoProperties.TRUSTSTORE_PROVIDER, TRUSTSTORE_PROVIDER);
        }

        CryptoType cryptoType = new CryptoType(CryptoType.TYPE.ALIAS);
        cryptoType.setAlias(KEY_ALIAS);
        try {
            certificate = Arrays.stream(getCrypto().getX509Certificates(cryptoType)).findFirst().orElseThrow(() -> new RuntimeException("Couldn't get X509 certificate based on provided KEY_ALIAS and KEYSTORE"));
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

[piotr-dajlido]

@phax could you please explain to me what is WebScopeManager responsible for?
I have found out that when it is created from SpringServletContext it's actually cleaning inputStream from httpRequest.

@Slf4j
@Configuration
public class AS4SpringServletConfig {

    public AS4SpringServletConfig(ServletContext servletContext) {
        if (!WebScopeManager.isGlobalScopePresent()) {
            WebScopeManager.onGlobalBegin(servletContext);
            _initGlobalSettings(servletContext);
            _initAS4();
        }

    }

Tested twice, with same request payload

1 Request:

2 Request

@phax I'am extreamly close to achive integration with Spring Web. I have already managed to get Actuator to list /as4 endpoint, but WebScopeManager appears very unclear to me 🤕

This is how I picture use of Phase4

Which acctualy makes me think that PModeManager, MPCManager or DuplicateManager should be an optional and extendible part of request piping. Can't justify worring about requests scopes 😄 Maybe creation of such flow diagram for Wiki would shed some light on library workflow, I personally would find it very usefull.

[phax]

I come from the C++ world and like consistent object lifecycles. That's why I implemented the "scope" principle to allow objects to have a clear initialization and shutdown paradigm. The WebScopeManager supports for 3 different types of scopes: a) RequestScope - for each HttpServletRequest; b) SessionScope - for objects within a single HTTP session and c) GlobalScope - for objects that survive globally.
With each scope manager it is possible to create objects as "singletons" so that they are lazily created and gracefully shutdown, when the outer scope ends.

Therefore PModeManager etc. are nothing more then global singletons. But implemented in a framework-unspecific way. phase4 works basically on an InputStream and fills an OutputStream. That's why this scoping mechanism must also be generic.

And yes, I agree that such diagrams would make sense, but it's a matter of time. The Wiki also gracefully accepts PRs :)