INVALID_HANDLE_VALUE is 64bit, GetConsoleMode() always returns failure

[dlmiles]

https://learn.microsoft.com/en-us/windows/win32/winprog/windows-data-types confirms HANDLE is of type typedef PVOID HANDLE; so 64bit on 64bit windows, and a long in Java.

INVALID_HANDLE_VALUE this is of type HANDLE

https://github.com/jline/jline3/blob/master/native/src/main/native/kernel32.c#L765
This looks to be using a Java int to store a windows HANDLE type, that could be 64bits on Win64. Code using this may appear to work due to int to long signed promotion of size occurring.

If this is intentional maybe a comment would help prevent it being looked at again.

The other values such as STD_OUTPUT_HANDLE are not actually system handle, they are variables that happen to have HANDLE in their name of type DWORD, so all is ok there.

---

This was spotted while reviewing to debug why I am unable to verify org.jline as working on 64bit windows.

From what I can see GetConsoleMode() always returns a failure indication. Not sure if that is from Windows or JNI wrapper. It would be nice if this particular failure also recorded the GetLastError() in the diagnostic logging. Handle 0x10123 Error 0x80000004 as it seems intrinsic to it detecting and working.

This results with DumbTerminal type=Unsupported.

When I use https://github.com/fusesource/jansi latest release (by just switching dependency away from org.jline:jansi:3.26.1) that is working. AnsiConsole.out().type==Redirected.

[dlmiles]

[dlmiles]

Screenshot of my guess at the problematic area. The 3 handles in system are all FALSE. So nothing can be selected, so I always get DumbTerminal.

[gnodet]

Could you explain how you use jansi, what the behaviour with the previous jansi, and the current behaviour ?
In your use case, are you redirecting the input/output/error streams ?

[gnodet]

https://learn.microsoft.com/en-us/windows/win32/winprog/windows-data-types confirms HANDLE is of type typedef PVOID HANDLE; so 64bit on 64bit windows, and a long in Java.

INVALID_HANDLE_VALUE this is of type HANDLE

https://github.com/jline/jline3/blob/master/native/src/main/native/kernel32.c#L765 This looks to be using a Java int to store a windows HANDLE type, that could be 64bits on Win64. Code using this may appear to work due to int to long signed promotion of size occurring.

If this is intentional maybe a comment would help prevent it being looked at again.

The other values such as STD_OUTPUT_HANDLE are not actually system handle, they are variables that happen to have HANDLE in their name of type DWORD, so all is ok there.

Good catch. That should be fixed.

From what I can see GetConsoleMode() always returns a failure indication. Not sure if that is from Windows or JNI wrapper. It would be nice if this particular failure also recorded the GetLastError() in the diagnostic logging. Handle 0x10123 Error 0x80000004 as it seems intrinsic to it detecting and working.

Are you referring to the org.jline.terminal.impl.jni.win.NativeWinSysTerminal#isWindowsSystemStream() here ?

This results with DumbTerminal type=Unsupported.

When I use https://github.com/fusesource/jansi latest release (by just switching dependency away from org.jline:jansi:3.26.1) that is working. AnsiConsole.out().type==Redirected.

Can you run java -jar [path-to]/jansi-3.26.1.jar and paste the output ?

[dlmiles]

Could you explain how you use jansi, what the behaviour with the previous jansi, and the current behaviour ? In your use case, are you redirecting the input/output/error streams ?

I am a new user of both jansi and when researching it org.jline as found as I am looking for a cross-platform (main desktops) CLI solution that meets modern requirements that supports native.

My example starter program looks like (I am to be using picocli.ansi=true support in due course):

//import org.fusesource.jansi.AnsiConsole
import org.jline.jansi.AnsiConsole

    fun main(args: Array<String>) {
        Logger.getGlobal().level = Level.ALL
        Logger.getGlobal().addHandler(ConsoleHandler())

        val xlogger = Logger.getLogger("org.jline") // .utils.Log")
        xlogger.level = Level.ALL

        //System.setProperty("org.jline.terminal.provider", "jansi")  // not needed
        //System.setProperty("jansi", "true")  // not needed
        kotlin.runCatching { AnsiConsole.systemInstall() }

        println("${AnsiConsole.isInstalled()}")
        // println("${AnsiConsole.getTerminal()}")  // only supported in org.jline only seen DumbTerminal here
        println("${AnsiConsole.out().type}")
    }

I have been using a number of terminal scenarios:

• Windows Terminal (cmd.exe / developer prompt)
• IntelliJ gradle run
• cmd.exe gradle run
• Different JVMs, mainly GraalVM in both standard and native-image mode.
• Testing under GraalVM native-image EXEs

When working (only with jansi:jansi) all have consistently reported type==Redirected. I have no idea what that term means on windows and can only assume it is the default for windows. But I see colour output, so mark that as working.

When using org.jline everything looks good in the detection process (correct org.jline:jang all-in-one-jar-deps) is finding the jni and exec providers. It polls each of them. But the polling process appears to reject what is found between the breakpoint shown in image and the current line of execution (the blue highlight line). This appears to be because GetConsoleMode always returns 0 and false. It is unclear if a Kernel32 DDL call was made to the kernel, or if the JNI error occurred internally, as there is a failure path there (maybe that should thrown error?)

--

Feedback (as a new user) should you be interested.

The project could do with a clear example that is focused on more basic features, such as ANSI output, curses like output, other output only matters, which I found jansi documentation covers better. Including details of the correct JAR dependency combinations to use for output only use cases and considerations in the choice of options there may be.
Maybe this is just a copy-and-paste of existing info from jansi website into existing wiki that is currently highly focussed on the advanced features without explaining the how to get started with more basic scenarios.

It could also do with a more straightforward breakdown of every output JAR, covering its purpose, the project has a few too many JAR output artifacts to not explain this. The topics the info per JAR might cover are:

• when to use it
• required platform-support/JDK/native-image support
• if it should be included along side other JARs or not,
• which JARs should be used directly in dependencies in client projects,
• which JARs are internal deps and will be added transitively.
• clear info on which byte-code compatibility is exists
  Some of this info is half explained for a few of the JARs but most of the JARs are unknown.

A dependency graph if all JARs were included, maybe confirming how things are organised, visible crossing out any that should not be included as well. With a note as to why, for example org.jline:jansi with org.jine:native, would have org.jline:jansi crossed out in a big list of all deps.

[dlmiles]

Are you referring to the org.jline.terminal.impl.jni.win.NativeWinSysTerminal#isWindowsSystemStream() here ?

Yes, that is the site where GetConsoleMode is being called and always returning from isWindowsSystemStream with FALSE.

Can you run java -jar [path-to]/jansi-3.26.1.jar and paste the output ?

Maybe that is a good thing to place into a main heading in the Wiki "Diagnostic" with example of use, expected output for each main platform. Maybe also a code snippet to invoke output within a more complex client application, to run that diagnostics within the context of a larger application ?

Then also maybe another page for "Configuration" with a breakdown of the various System Properties that affect behaviour and other similar things. At the moment I run with debugger and read source and then guess that the property function is doing and its allowed range of values.

C:\Users\Name\ProjectDir\project-name>java -jar "C:\Users\Name\.gradle\caches\modules-2\files-2.1\org.jline\jansi\3.26.1\9f133fa77ba456422f47b7c0b57e26a5e196ca63\jansi-3.26.1.jar"
System properties
=================
os.name =         Windows 10
OSTYPE =          null
MSYSTEM =         null
PWD =             null
ConEmuPID =       null
WSL_DISTRO_NAME = null
WSL_INTEROP =     null

OSUtils
=================
IS_WINDOWS = true
IS_CYGWIN =  false
IS_MSYSTEM = false
IS_WSL =     false
IS_WSL1 =    false
IS_WSL2 =    false
IS_CONEMU =  false
IS_OSX =     false

FFM Support
=================
FFM support not available: java.io.IOException: Unable to find terminal provider ffm

JnaSupport
=================
JNA support not available: java.io.IOException: Unable to find terminal provider jna

Jansi2Support
=================
Jansi 2 support not available: java.io.IOException: Unable to find terminal provider jansi

JniSupport
=================
StdIn stream =    true
StdOut stream =   true
StdErr stream =   true
Unable to check stream names: java.lang.UnsatisfiedLinkError: 'void org.jline.nativ.CLibrary.init()'
Terminal size: Size[cols=173, rows=48]
The terminal seems to work: terminal org.jline.terminal.impl.jni.win.NativeWinSysTerminal

Exec Support
=================
StdIn stream =    false
StdOut stream =   false
StdErr stream =   false
StdIn stream name =     null
StdOut stream name =    null
StdErr stream name =    null
Not supported by provider
Jansi null

jansi.providers=

os.name= Windows 10, os.version= 10.0, os.arch= amd64
file.encoding= UTF-8
sun.stdout.encoding= null, sun.stderr.encoding= null
stdout.encoding= cp850, stderr.encoding= cp850
java.version= 21.0.3, java.vendor= Oracle Corporation, java.home= C:\Devel\deps\graalvm-jdk-21.0.3+7.1
Console: java.io.ProxyingConsole@6833ce2c

jansi.graceful=
jansi.mode=
jansi.out.mode=
jansi.err.mode=
jansi.colors=
jansi.out.colors=
jansi.err.colors=
jansi.noreset= false
org.jline.jansi.Ansi.disable= false

IS_WINDOWS: true
IS_CONEMU: false
IS_CYGWIN: false
IS_MSYSTEM: false


Resulting Jansi modes for stout/stderr streams:
  - System.out: AnsiPrintStream{type=Native, colors=Colors16, mode=Default, resetAtUninstall=true}
  - System.err: AnsiPrintStream{type=Native, colors=Colors16, mode=Default, resetAtUninstall=true}
Processor types description:
  - Native: Supports ansi sequences natively
  - Unsupported: Ansi sequences are stripped out
  - VirtualTerminal: Supported through windows virtual terminal
  - Emulation: Emulated through using windows API console commands
  - Redirected: The stream is redirected to a file or a pipe
Colors support description:
  - Colors16: 16 colors
  - Colors256: 256 colors
  - TrueColor: 24-bit colors
Modes description:
  - Strip: Strip all ansi sequences
  - Default: Print ansi sequences if the stream is a terminal
  - Force: Always print ansi sequences, even if the stream is redirected

test on System.out: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
            bright: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
              bold: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
             faint: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
        bold+faint: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
        256 colors:





         truecolor: /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\
test on System.err: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
            bright: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
              bold: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
             faint: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
        bold+faint: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE DEFAULT
        256 colors:





         truecolor: /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
                    /\/\/\/\/\/\/\/\

                      ┌──┐┌─────┐ ┌─────┐ ┌──────┬──┐
                      │██├┘█████└┬┘█████└┬┘██████│▐▌│
                 ┌──┐ │██│██▄▄▄██│██┌─┐██│██▄▄▄▄ │▄▄│
                 │▒▒└─┘▒█│▒█┌─┐▒█│▒█│ │▒█│ ▀▀▀▀▒█│▒█│
                 └┐▓▓▓▓▓┌┤▓▓│ │▓▓│▓▓│ │▓▓│▀▓▓▓▓▓▀│▓▓│
                  └─────┘└──┘ └──┴──┘ └──┴───────┴──┘

The output looks good to me. This was Windows Terminal. Also works directly from cmd.exe and jdk-21.

My specific use case wants to see it working under IDE like shell environments, so the shell presented.
While I understand a lot about Linux ttys I don't about Windows and have never worked with their low level API and the way they are implemented. I shall look to see if the gradle run use can be improved by looking at the default gradle exec config setup, but the user experience looks just like a regular terminal.

[gnodet]

I've raised a first PR #1015 to fix the INVALID_HANDLE_VALUE as a long.

[gnodet]

I'm getting back to your issue, but I'm not sure to see the exact problems. Did you at least succeeded in running your simple JLine app in a basic environment ?

The fact that Jansi returns Redirected and that JLine returns a dumb terminal may not be an easy problem to solve in your use cases. I'd really advise to also try the same scenarii on linux or macOS.

When launching with gradle, you may hit problems with grade and the terminal. Have a look at https://stackoverflow.com/questions/13172137/console-application-with-java-and-gradle

[dlmiles]

Thanks you for your responses, good to see the INVALID_HANDLE_VALUE fix.

Give me a week or so to properly experiment with jbang and jline (using output only APIs), across Win64/linux64 and multiple JVMs and options on Native (GraalVM, Kotlin/Native - if I get chance on this). I am still in the stages of learning limitations and caveats with various options, while trying to look for a unified seamless API use.