Getting the "call ash_fetch, error:can't read "public(cpucount)": no such element in array" with Postgres

[Fadaig]

Dear friends,

I'm running HammerDB connected to Postgres14 on RHEL 7.9.
When starting the Metrics, getting the following error "call ash_fetch, error:can't read "public(cpucount)": no such element in array
"
Kindly, suggest what might be missing here.

Thank you,
Fadai.

[sm-shaw]

OK, firstly just to check that pg_sentinel and pg_stat_statements are installed correctly. You should see a message such as follows if it is:

Waiting for Connection to PostgreSQL for Database Metrics...
Metrics Connecting to host:localhost port:5432
Metrics Connected
Starting Metrics, read 59 rows from pg_active_session_history

If something is not installed correctly it should find pg_active_session_history empty.

Waiting for Connection to PostgreSQL for Database Metrics...
Metrics Connecting to host:localhost port:5432
Metrics Connected
Metrics Error: No rows found in pg_active_session_history,run a workload to populate metrics
Metrics Disconnect from PostgreSQL...
Metrics Closed

After this, then create a test program such as follows:

proc pg_HowManyProcessorsLinux {} {
        global S cpu_model
        set fin [open /proc/cpuinfo r]
        set data [read $fin]; list
        close $fin
        set ::S(cpus) [llength [regexp -all -inline processor $data]]
        set ::S(cpus,list) {}
        for {set i 0} {$i < $::S(cpus)} {incr i} {
            lappend ::S(cpus,list) $i
        }
        foreach line [ split $data "\n" ] {
            if {[string match {model name*} $line]} {
                regexp {(.*):\ (.*)} $line all header cpu_model
                break
            }
        }
    }

global S
pg_HowManyProcessorsLinux
set public(cpucount) $S(cpus)
puts $public(cpucount)

And run it as follows:

$ tclsh test.tcl
4

Do you get a number such as above?
If not what CPU are you using and what do you see in /proc/cpuinfo on the processor lines?
If something is not parsing correctly, we can change the discussion into an issue.

[Fadaig]

Hello Steve,

Thank you for the prompt response.

Yes, the extensions installed correctly. I can see the following output, once I click the Start Metrics

Waiting for Connection to PostgreSQL for Database Metrics... Metrics Connecting to host: linuxserver port:5432 Metrics Connected Starting Metrics, read 83 rows from pg_active_session_history call ash_fetch, error:can't read "public(cpucount)": no such element in array

However, when I try to run the program via tclsh I get the following error:
"./tclsh: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by ../lib/libtcl8.6.so)"

I have a lower version of GLIBC which is 2.17 which does not seem to be ok for this scenario.

Also, I have also double-checked the documentation and it turns out that for the Metrics to work properly the agent must be running on the server, right (the pg_HowManyProcessorsLinux function is there in the agent code)?

[sm-shaw]

No, you only need the agent to show the graphical CPU metrics. This is part of the PostgreSQL metrics and is different.
This should be the system installed tclsh for the test so tclsh and not ./tclsh

$ tclsh test.tcl
4

or if you prefer run in HammerDB. It should show you the CPU count.

$ ./hammerdbcli
HammerDB CLI v4.9
Copyright (C) 2003-2023 Steve Shaw
Type "help" for a list of commands
Initialized new Jobs on-disk database /tmp/hammer.DB
hammerdb>source test.tcl
4

hammerdb>

The important part to troubleshoot is what is in /proc/cpuinfo?

[Fadaig]

Hi Steve,

Here is the output of the command:

[root@localhost HammerDB-4.9]# tclsh ./bin/test.tcl 2

and this

[root@localhost HammerDB-4.9]# ./hammerdbcli HammerDB CLI v4.9 Copyright (C) 2003-2023 Steve Shaw Type "help" for a list of commands Initialized Jobs on-disk database /tmp/hammer.DB using existing tables (45,056 KB) hammerdb>source ./bin/test.tcl 2

and here is the content of /proc/cpuinfo:
`[root@localhost HammerDB-4.9]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 154
model name : 12th Gen Intel(R) Core(TM) i7-1255U
stepping : 4
microcode : 0xffffffff
cpu MHz : 2611.200
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes rdrand hypervisor lahf_lm abm 3dnowprefetch ibrs_enhanced fsgsbase bmi1 bmi2 invpcid rdseed clflushopt md_clear flush_l1d arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass swapgs retbleed eibrs_pbrsb
bogomips : 5222.40
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 154
model name : 12th Gen Intel(R) Core(TM) i7-1255U
stepping : 4
microcode : 0xffffffff
cpu MHz : 2611.200
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes rdrand hypervisor lahf_lm abm 3dnowprefetch ibrs_enhanced fsgsbase bmi1 bmi2 invpcid rdseed clflushopt md_clear flush_l1d arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass swapgs retbleed eibrs_pbrsb
bogomips : 5222.40
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:`

Interestingly, it cannot parse the portion of the agent code which retrieves the number of CPU but when running alone the code works just fine.

[Fadaig]

Apologies, for the unformatted output. Used the code tag which didn't help.

[sm-shaw]

Yes, so even if I create a file with your cpuinfo (many thanks for that) it works OK.
At the moment I can't reproduce the problem which is of course the first step.
For further troubleshooting if you run the agent and then connect with the metrics (use another database such as MySQL without the extended metrics) does it show the CPU data?

As before, this is not exactly the same area of code but it uses the same function so can help to see if the problem does lie in parsing cpuinfo

$ ./agent
Initializing HammerDB Metric Agent 4.9
HammerDB Metric Agent active @ id 18318 hostname CRANE (Ctrl-C to Exit)
Received a new display request from host 127.0.0.1
New display accepted @ 18597 CRANE
Prepared CPU metrics for Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
Sending CPU metrics to 18597 CRANE ...connection established

[Fadaig]

Unfortunately, when I run the agent on the server it throws an error:
[root@centos agent]# ./agent .././bin/tclsh8.6: /lib64/libm.so.6: version GLIBC_2.29' not found (required by .././lib/libtcl8.6.so)`.

Whether this might be related hard to say. CentOS 7.9 is shipped with glibc 2.17, and updating it to 2.29 resulted in a failure because of so many dependencies.

I've been struggling with this for a while but no success yet.

Now, trying with CentOS 7.9. Getting the same issue:
Waiting for Connection to PostgreSQL for Database Metrics... Metrics Connecting to host:192.168.100.7 port:5432 Metrics Connected Starting Metrics, read 50 rows from pg_active_session_history call ash_fetch, error:can't read "public(cpucount)": no such element in array

My server is a Virtual Box guest. If there is any chance of import it and debugging it I can share the export file (.ova or .ovf) in DropBox or similar.

I was also wondering whether it can be any permission issue or similar !?

[sm-shaw]

The first thing to fix will be the glibc error. If you are running on virtual box it would be best trying on Red Hat 8.0 for which there is a specific release that supports that glibc version:
Release 4.9 for Red Hat Enterprise Linux 8 (glibc 2.28) 64-bit compressed tar file

[Fadaig]

Would it be possible to change the code (temporarily for debugging purposes) in such a way that first it fetches other environment variables and then cpucount just to see whether all fail or only something in the pg_HowManyProcessorsLinux fails?

Since running the agent on the server fails, I run it locally and it succeeded:

Starting Metrics, read 30 rows from pg_active_session_history call ash_fetch, error:can't read "public(cpucount)": no such element in array Metric receive port open @ 28048 on laptop1 Connecting to HammerDB Agent @ laptop1:12452 Testing Agent Connectivity...OK Metrics Connected

[Fadaig]

Ok. Downloading RHEL 8. Will check and update here in few hours.
Meanwhile, may I ask how frequently the Metrics chart is updated while running a load?

[Fadaig]

Hi Steve,

I installed now CentOS8. First, the Metrics failed because the guest couldn't ping the host. Fixed it. Now works fine. But the the cpucount still cannot be fetched. Here is the output:

[root@centos8 ~]# cat /etc/redhat-release CentOS Stream release 8

[root@centos8 agent]# ./agent Initializing HammerDB Metric Agent 4.9 HammerDB Metric Agent active @ id 36995 hostname centos8 (Ctrl-C to Exit) Received a new display request from host 192.168.100.8 New display accepted @ 10276 laptop11 Prepared CPU metrics for 12th Gen Intel(R) Core(TM) i7-1255U Sending CPU metrics to 10276 laptop11 ...connection established

When starting the Metrics the error persists:
Starting Metrics, read 123 rows from pg_active_session_history call ash_fetch, error:can't read "public(cpucount)": no such element in array Metric receive port open @ 10276 on laptop11 Connecting to HammerDB Agent @ centos8:36995 Testing Agent Connectivity...OK Metrics Connected

Also, on the same server:
[root@centos8 home]# tclsh test.tcl 2

How can we debug it further?

[sm-shaw]

OK got it, after some troubleshooting and running PostgreSQL on RHEL8 on VBox I can reproduce. So a local client works fine and a Linux client works fine, but the crucial piece of information here is that if a client is running on Windows and connecting to a server running on Linux then you get the error:

Waiting for Connection to PostgreSQL for Database Metrics...
Metrics Connecting to host:razorbill port:5432
Metrics Connected
Starting Metrics, read 855 rows from pg_active_session_history
call ash_fetch, error:can't read "public(cpucount)": no such element in array

So I am guessing that your HammerDB client is running on Windows and connecting to your Linux server running on VBox on the same host and there is a mismatch in finding the CPU count.
This is a difference in that PostgreSQL doesn't allow you to query the CPU count from the database like Oracle does so instead that is why it is querying the client. There is this comment in the code:

#mon_execute cpucount
#cpucount cannot be retrieved by PostgreSQL. Cpucount is limited to running in the client.

So to fix you can modify the source file pgmet.tcl in src/postgresql and modify the start of proc pg_HowManyProcessorsWindows from this:

 proc pg_HowManyProcessorsWindows {} {
        global S cpu_model
        set cpu_model [lindex [twapi::get_processor_info 0 -processorname] 1]

To this:

 proc pg_HowManyProcessorsWindows {} {
        global S cpu_model
	if [catch {package require twapi} ] { 
        set ::S(cpus) 1
	return
	}

In my testing this resolves the error call ash_fetch, error:can't read "public(cpucount)": no such element in array.
I wall create a separate issue and PR to add the change.

[sm-shaw]

PR #638 has been created to fix the issue.