Work Around Solution Found: pl/r does not run on Windows 10 64 bit and *new* hardware #26
Comments
|
Unfortunately I don't know enough about debugging on windows to even point I do have an idea as to what is causing this, but I have no way to confirm Can you also provide me with output of select version() ; Dave Cramer On 3 September 2016 at 19:02, AndreMikulec [email protected] wrote:
|
postgres=# select version();
version
-------------------------------------------------------------
PostgreSQL 9.5.2, compiled by Visual C++ build 1800, 64-bit
(1 row) |
|
Here are some links on 'cache lookups for relations' that I came up with. |
|
Just eye-balling the code, Perhaps the call path is going through here. /*
* haveModulesTable(Oid) -- Check if table plr_modules exists in the namespace
* designated by the OID input argument.
*/
static bool
haveModulesTable(Oid nspOid)
{
StringInfo sql = makeStringInfo();
char *sql_format = "SELECT NULL "
"FROM pg_catalog.pg_class "
"WHERE "
"relname = 'plr_modules' AND "
"relnamespace = %u";
int spiRc;
appendStringInfo(sql, sql_format, nspOid);
spiRc = SPI_exec(sql->data, 1);
if (spiRc != SPI_OK_SELECT)
/* internal error */
elog(ERROR, "haveModulesTable: select from pg_class failed");
return SPI_processed == 1;
}Search on "SPI_exec" yields . . . Search on "SPI_execute" yields . . . Next, _SPI_execute_plan is called and this function is about 1300 lines of code. |
|
I'm aware what relcache is. The possible mismatch I am referring to may be Ironically in one of the slides it suggests that you may want to use a unix Dave Cramer On 4 September 2016 at 16:29, AndreMikulec [email protected] wrote:
|
|
Dave Cramer, Moreover, what I really need/want/should do is (1) 'isolate' the problem, (2) report it, and (3) possibly come up with a recommendation. When this happens, sometimes, at least one other person may look at the same problem ( and possibly come up with a 'better' solution ). That is why earlier I asked 'debug this problem?' An example is here: quantile produces decreasing output In this particular github issue, first 'debugging'drawback is the my C/C+ static/dll debugging skill is almost non-existent. The second drawback is that the PostgreSQL codebase is very large and I do not know where to start. So, that 'very large' problem is why I asked about ''debug this problem?' Do you know anyone or can you point me in any direction, to someone who is very good doing PostgreSQL C/C+ static/dll debugging and/or may have more ideas on where to start looking? |
|
Andrej, The problem isn't finding someone who is good at debugging postgres,the Debugging it is not a huge task once you
Then you need to fire up your debugger to debug postgres. In another window Put a breakpoint on plr_call_handler which is the function exposed by plr Unfortunately I don't have the skills but if you can figure out the other Dave Cramer On 5 September 2016 at 08:42, AndreMikulec [email protected] wrote:
|
|
Dave, (1) Using gdb, (and Code::Blocks), I am 50 levels deep in debugging the plr.c/postgresql. (2) In plr.c, of the function haveModulesTable That did not help. I still get the same error. (3). This is a question. postgres=# SELECT * FROM pg_catalog.pg_class WHERE relname = 'plr_modules';" supposed to return zero records? postgres=# SELECT * FROM pg_catalog.pg_class WHERE relname = 'plr_modules';
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode |
---------+--------------+---------+-----------+----------+-------+-------------+
(0 rows)postgres=# create extension plr;
CREATE EXTENSION
postgres=# SELECT NULL FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200;
?column?
----------
(0 rows)
postgres=# SELECT * FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200;
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode | reltablespace | relpa
---------+--------------+---------+-----------+----------+-------+-------------+---------------+------
(0 rows)
postgres=# SELECT * FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200;
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode | reltablespace | relpa
---------+--------------+---------+-----------+----------+-------+-------------+---------------+------
(0 rows)
postgres=# SELECT * FROM pg_catalog.pg_class WHERE relname = 'plr_modules';
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode |
---------+--------------+---------+-----------+----------+-------+-------------+
(0 rows)
postgres=# SELECT * FROM pg_catalog.pg_class WHERE relnamespace = 2200;
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode |
------------------+--------------+---------+-----------+----------+-------+-------------+
plr_environ_type | 2200 | 16447 | 0 | 10 | 0 | 16445 |
r_typename | 2200 | 16451 | 0 | 10 | 0 | 16449 |
r_version_type | 2200 | 16456 | 0 | 10 | 0 | 16454 |
(3 rows) |
|
Hi Andre, Dave Cramer On 16 September 2016 at 12:38, AndreMikulec [email protected]
off the top of my head, no, it should show something what does \d show ?
|
|
Seems to just show my user tables postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+---------+-------+----------
public | ct | table | postgres
public | mytable | table | postgres
(2 rows)postgres=# select count(*) from pg_catalog.pg_class;
count
-------
323
(1 row)postgres=# select * from pg_catalog.pg_class where relname like 'r\_%' order by 1;
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode | reltablespace |
----------------+--------------+---------+-----------+----------+-------+-------------+---------------+
r_typename | 2200 | 16588 | 0 | 10 | 0 | 16586 | 0 |
r_version_type | 2200 | 16593 | 0 | 10 | 0 | 16591 | 0 |
(2 rows)
postgres=# select * from pg_catalog.pg_class where relname like 'plr\_%' order by 1;
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode | reltablespace |
------------------+--------------+---------+-----------+----------+-------+-------------+---------------+
plr_environ_type | 2200 | 16584 | 0 | 10 | 0 | 16582 | 0 |
(1 row)Maybe, for some reason, CREATE EXTENSION, is not creating those functions? From INSTALL.txt I have all of those files The pgAdmin III, schema public functions shows this SELECT pr.oid, pr.xmin, pr.*, pg_get_function_result(pr.oid) AS typname, typns.nspname AS typnsp, lanname, proargnames, pg_get_expr(proargdefaults, 'pg_catalog.pg_class'::regclass) AS proargdefaultvals, pronargdefaults, proconfig, pg_get_userbyid(proowner) as funcowner, description,
(SELECT array_agg(label) FROM pg_seclabels sl1 WHERE sl1.objoid=pr.oid) AS labels,
(SELECT array_agg(provider) FROM pg_seclabels sl2 WHERE sl2.objoid=pr.oid) AS providers
FROM pg_proc pr
JOIN pg_type typ ON typ.oid=prorettype
JOIN pg_namespace typns ON typns.oid=typ.typnamespace
JOIN pg_language lng ON lng.oid=prolang
LEFT OUTER JOIN pg_description des ON (des.objoid=pr.oid AND des.classoid='pg_proc'::regclass)
WHERE proisagg = FALSE AND pronamespace = 2200::oid
AND typname NOT IN ('trigger', 'event_trigger')
ORDER BY proname;
oid | xmin | proname | pronamespace |
-------+------+---------------------+--------------+
16578 | 566 | install_rcmd | 2200 |
16590 | 566 | load_r_typenames | 2200 |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
16581 | 566 | plr_array_accum | 2200 |
16580 | 566 | plr_array_push | 2200 |
16574 | 566 | plr_call_handler | 2200 |
16585 | 566 | plr_environ | 2200 |
16598 | 566 | plr_get_raw | 2200 |
16597 | 566 | plr_set_display | 2200 |
16595 | 566 | plr_set_rhome | 2200 |
16579 | 566 | plr_singleton_array | 2200 |
16596 | 566 | plr_unset_rhome | 2200 |
16576 | 566 | plr_version | 2200 |
16589 | 566 | r_typenames | 2200 |
| | | |
| | | |
| | | |
| | | |
| | | |
16594 | 566 | r_version | 2200 |
| | | |
| | | |
16577 | 566 | reload_plr_modules | 2200 |
(15 rows)
Checking and looking for 'plr_modules'. I have not found it in here. plr.c talks about 'table plr_modules (if it exists)' /*
* plr_load_modules() - Load procedures from
* table plr_modules (if it exists)
*
* The caller is responsible to ensure SPI has already been connected
* DO NOT make this static --- it has to be callable by reload_plr_modules()
*/
void
plr_load_modules(void)
{
int spi_rc;
char *cmd;
int i;
int fno;
MemoryContext oldcontext;
char *modulesSql;
/* switch to SPI memory context */
SWITCHTO_PLR_SPI_CONTEXT(oldcontext);
/*
* Check if table plr_modules exists
*/
if (!haveModulesTable(plr_nspOid))
{
/* clean up if SPI was used, and regardless restore caller's context */
CLEANUP_PLR_SPI_CONTEXT(oldcontext);
return;
}
/* plr_modules table exists -- get SQL code extract table's contents */
modulesSql = getModulesSql(plr_nspOid);
/* Read all the row's from it in the order of modseq */
spi_rc = SPI_exec(modulesSql, 0);
/* modulesSql no longer needed -- cleanup */
pfree(modulesSql);
if (spi_rc != SPI_OK_SELECT)
/* internal error */
elog(ERROR, "plr_init_load_modules: select from plr_modules failed");
/* If there's nothing, no modules exist */
if (SPI_processed == 0)
{
SPI_freetuptable(SPI_tuptable);
/* clean up if SPI was used, and regardless restore caller's context */
CLEANUP_PLR_SPI_CONTEXT(oldcontext);
return;
}
/*
* There is at least on module to load. Get the
* source from the modsrc load it in the R interpreter
*/
fno = SPI_fnumber(SPI_tuptable->tupdesc, "modsrc");
for (i = 0; i < SPI_processed; i++)
{
cmd = SPI_getvalue(SPI_tuptable->vals[i],
SPI_tuptable->tupdesc, fno);
if (cmd != NULL)
{
load_r_cmd(cmd);
pfree(cmd);
}
}
SPI_freetuptable(SPI_tuptable);
/* clean up if SPI was used, and regardless restore caller's context */
CLEANUP_PLR_SPI_CONTEXT(oldcontext);
}In these files, no 'CREATE table' exists .... These files are exactly the same This is the same in the 10devel (SEP_11_2016) ( and plr ) C:\postgres-master_0ab9c56_debug\contrib\plr\sql\plr.sql CREATE TABLE plr_modules (
modseq int4,
modsrc text
);Comparing plr-8.3.0.16-pg9.4-win64/plr Again, These files are exactly the same ( AND I did get pl/r in PostgreSQL 9.4 on Windows to work ) From SQL in sql/plr.sql, -- check version
SELECT plr_version();
-- make typenames available in the global namespace
select load_r_typenames();
CREATE TABLE plr_modules (
modseq int4,
modsrc text
);
INSERT INTO plr_modules VALUES (0, 'pg.test.module.load <-function(msg) {print(msg)}');
select reload_plr_modules();
--
-- plr_modules test
--
create or replace function pg_test_module_load(text) returns text as 'pg.test.module.load(arg1)' language 'plr';
select pg_test_module_load('hello world');So, now I have data. postgres=# SELECT NULL FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200;
?column?
----------
(1 row)
postgres=# SELECT * FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200;
relname | relnamespace | reltype | reloftype | relowner | relam | relfilenode | reltablespace | relpages | reltuples | relallvisible | reltoastrelid | relhasindex | relisshared | relpersistence | relkind | relnatts | relchecks | relhasoids | relhaspk
ey | relhasrules | relhastriggers | relhassubclass | relrowsecurity | relforcerowsecurity | relispopulated | relreplident | relfrozenxid | relminmxid | relacl | reloptions
-------------+--------------+---------+-----------+----------+-------+-------------+---------------+----------+-----------+---------------+---------------+-------------+-------------+----------------+---------+----------+-----------+------------+---------
---+-------------+----------------+----------------+----------------+---------------------+----------------+--------------+--------------+------------+--------+------------
plr_modules | 2200 | 16602 | 0 | 10 | 0 | 16600 | 0 | 0 | 0 | 0 | 16603 | f | f | p | r | 2 | 0 | f | f
| f | f | f | f | f | t | d | 569 | 1 | |
(1 row)
postgres=# SELECT modseq, modsrc FROM public.plr_modules ORDER BY modseq;
modseq | modsrc
--------+--------------------------------------------------
0 | pg.test.module.load <-function(msg) {print(msg)}
(1 row)I get a 'different' error. postgres=# select load_r_typenames();
ERROR: could not open file "base/12418/16600": No such file or directory
CONTEXT: SQL statement "SELECT modseq, modsrc FROM public.plr_modules ORDER BY modseq"
postgres=#Eventually, this error this returns postgres=# select load_r_typenames();
ERROR: could not open file "base/12418/1259": No such file or directory
CONTEXT: SQL statement "SELECT NULL FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200"
postgres=#NOTE: SPI_execute in tablefunct.c works fine. Then, from ... psql (10devel)
Type "help" for help.
postgres=# select version();
version
----------------------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 10devel on x86_64-w64-mingw32, compiled by x86_64-w64-mingw32-gcc.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 6.2.0, 64-bit
(1 row)
C:\postgres-master_0ab9c56_debug\contrib\tablefunc>chcp 1252 > nul && "%PGSQL%\bin\psql.exe" -p 5434
psql (10devel)
Type "help" for help.
postgres=# \ir sql/tablefunc.sql
CREATE EXTENSION tablefunc;
CREATE EXTENSION
...
avg
-----
250
(1 row)
...
SELECT * FROM crosstab#('SELECT rowid, attribute, val FROM ...')
...
row_name | category_1 | category_2
----------+------------+------------
test1 | val2 | val2
test2 | val6 | val6
| val10 | val10
(3 rows)
...
row_name | category_1 | category_2 | category_3
----------+------------+------------+------------
test1 | val2 | val2 | val3
test2 | val6 | val6 | val7
| val10 | val10 | val11
(3 rows)
...
DROP TABLE
CREATE TABLE
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
keyid | parent_keyid | level | branch
-------+--------------+-------+----------
11 | | 0 | 11
10 | 11 | 1 | 11-10
111 | 11 | 1 | 11-111
1 | 111 | 2 | 11-111-1
(4 rows)Currently, I am using gdb and Code::Block to follow as you recommended. Again, the code is getting so complicated after 50 stack levels deep, I am having some difficulty following it anymore. I am trying to find some some sort of automated 'stepping' mechanism or script. I need to to know exactly where ( what function ) the the crash/error occurs. Does anyone know how to do that? O.K. I think I found out how to do that. I have some more ... postgres=# SELECT pg_backend_pid();
pg_backend_pid
----------------
3380
(1 row)Here ... postgres=# select r_version();Continuing... At c:\postgres-master_0ab9c56_debug\src\backend\utils\error\elog.c:414 Of #1 0x6c06f5 mdopen(reln=reln@entry=0x184b58, forknum=forknum@entry=MAIN_FORKNUM, behavior=behavior@entry=9) (md.c:607)
``
At ( path = )
C:\postgres-master_0ab9c56_debug\src\backend\storage\smgr\md.c:585
the Code::Block watch shows "fd = -1"
If fd means 'file descriptor?" ....
```C
path = relpath(reln->smgr_rnode, forknum);
fd = PathNameOpenFile(path, O_RDWR | PG_BINARY, 0600);
if (fd < 0) ## (Code::Blocks watches 'fd == -1' SO IA M GOING IN HERE) ##
{
## (DON'T WANT TO GO IN HERE) ##If 'fd means file descriptor' and my fd == -1, then that is not good. C:\postgres-master_0ab9c56_debug\src\backend\storage\smgr\md.c:69
#include <fcntl.h>An explanation is here ... At C:\postgres-master_0ab9c56_debug\src\backend\storage\file\fd.c:1228
/*
* open a file in an arbitrary directory
*
* NB: if the passed pathname is relative (which it usually is),
* it will be interpreted relative to the process' working directory
* (which should always be $PGDATA when this code is running).
*/
File
PathNameOpenFile(FileName fileName, int fileFlags, int fileMode)My environment variable is correct I want to see my permissions. Then I rerun. The exact same answers(Type, Username, Permissions,Inheritance) was obtained for subfolders Here is the point of view from msys, if it matters . . . I want to see the values of the fileFlags of O_RDWR and PG_BINARY. PathNameOpenFile(FileName fileName, int fileFlags, int fileMode)
fd = PathNameOpenFile(path, O_RDWR | PG_BINARY, 0600);Code::Blocks thread search finds PG_BINARY here. Code::Blocks thread search can not find the O_RDWR definition. So, I still can not find the defintion of O_RDWR. I guess: it seems like it should be in fd.c(or a header) #ifndef WIN32
#include <sys/mman.h>
#endifHere is something about it. I can not find O_RDWR anywhere. I am tired. I am going to sleep. Two questions.
|
|
On 27 September 2016 at 18:09, AndreMikulec [email protected]
Hi Andre I applaud your tenacity. What I would do is put a breakpoint in ereport and Thanks, Dave Cramer |
|
O.K. ereport ends up being a macro ( so I can not break on it) Here is what I have. ( There does exist an ereport further below. ) Breakpoints are at and I run select r_version();Here is what happens. * 1 Thread 3644.0x10d8 PathNameOpenFile (fileName=fileName@entry=0xf147d68 "base/12418/2682", fileFlags=fileFlags@entry=32770, fileMode=fileMode@entry=384) at fd.c:1229
>>>>>>cb_gdb:
SUCCEED ... CONTINUE ...
* 1 Thread 3644.0x10d8 PathNameOpenFile (fileName=fileName@entry=0xf147d68 "base/12418/2612", fileFlags=fileFlags@entry=32770, fileMode=fileMode@entry=384) at fd.c:1229
>>>>>>cb_gdb:
SUCCEED ... CONTINUE ...
PLR
* 1 Thread 3644.0x10d8 plr_call_handler (fcinfo=0xf145f80) at plr.c:199
>>>>>>cb_gdb:
PLR... CONTINUE ...
> bt
#0 plr_call_handler (fcinfo=0xf145f80) at plr.c:199
#1 0x0000000000598027 in ExecMakeFunctionResult (fcache=fcache@entry=0xf145f10, econtext=econtext@entry=0xf145ce8, isNull=isNull@entry=0xf1468c0 "", isDone=isDone@entry=0xf146a00) at execQual.c:1840
#2 0x0000000000598264 in ExecEvalFunc (fcache=0xf145f10, econtext=0xf145ce8, isNull=0xf1468c0 "", isDone=0xf146a00) at execQual.c:2437
#3 0x000000000059b4ad in ExecTargetList (isDone=0xf145bd0, itemIsDone=0xf146a00, isnull=0xf1468c0 "", values=0x1, econtext=0x0, tupdesc=<optimized out>, targetlist=0xf145ce8) at execQual.c:5486
#4 ExecProject (projInfo=<optimized out>, isDone=0xf145bd0, isDone@entry=0x387ec1c) at execQual.c:5710
#5 0x00000000005b18c1 in ExecResult (node=node@entry=0xf145bd0) at nodeResult.c:155
#6 0x0000000000593ff1 in ExecProcNode (node=node@entry=0xf145bd0) at execProcnode.c:392
#7 0x000000000058ffde in ExecutePlan (dest=0xf143dd8, direction=<optimized out>, numberTuples=0, sendTuples=1 '\001', operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0xf145bd0, estate=0xf145ab8) at execMain.c:1567
#8 standard_ExecutorRun (queryDesc=0x1cd948, direction=<optimized out>, count=0) at execMain.c:338
#9 0x000000000059009a in ExecutorRun (queryDesc=queryDesc@entry=0x1cd948, direction=direction@entry=ForwardScanDirection, count=count@entry=0) at execMain.c:286
#10 0x00000000006c0a03 in PortalRunSelect (portal=0x1dc348, portal@entry=0xf143dd8, forward=forward@entry=1 '\001', count=0, count@entry=252984792, dest=dest@entry=0xe2ba8) at pquery.c:948
#11 0x00000000006c20f9 in PortalRun (portal=0xf143dd8, portal@entry=0x1dc348, count=252984792, count@entry=2147483647, isTopLevel=isTopLevel@entry=0 '\000', dest=0xe2ba8, dest@entry=0xf143dd8, altdest=<optimized out>, altdest@entry=0xf143dd8, completionTag=0x387f180 "", completionTag@entry=0x387f1b8 "¸ãi") at pquery.c:789
#12 0x00000000006be813 in exec_simple_query (query_string=0x7cd03f <string_compare+17> "HƒÄ ]ÃUH‰åHƒì H‰ÊH‹\réT*") at postgres.c:1094
#13 PostgresMain (argc=<optimized out>, argv=argv@entry=0x1344d8, dbname=<optimized out>, username=<optimized out>) at postgres.c:4070
#14 0x0000000000653d3a in BackendRun (port=0x387fc30) at postmaster.c:4260
#15 SubPostmasterMain (argc=argc@entry=3, argv=argv@entry=0x10ffa0) at postmaster.c:4750
#16 0x00000000005cf6b8 in main (argc=3, argv=0x10ffa0) at main.c:205
> bt
#0 PathNameOpenFile (fileName=fileName@entry=0xf14f040 "base/12418/2663", fileFlags=fileFlags@entry=32770, fileMode=fileMode@entry=384) at fd.c:1229
#1 0x00000000006b75fb in mdopen (reln=reln@entry=0x18fdc8, forknum=forknum@entry=MAIN_FORKNUM, behavior=behavior@entry=9) at md.c:587
#2 0x00000000006b87e3 in _mdfd_getseg (reln=0x18fdc8, forknum=MAIN_FORKNUM, blkno=blkno@entry=0, skipFsync=skipFsync@entry=0 '\000', behavior=behavior@entry=9) at md.c:1783
#3 0x00000000006b8bc8 in mdread (reln=<optimized out>, forknum=<optimized out>, blocknum=0, buffer=0x6a42380 "") at md.c:741
#4 0x00000000006ba05a in smgrread (reln=0xf14f040, reln@entry=0x0, forknum=32770, forknum@entry=FSM_FORKNUM, blocknum=384, blocknum@entry=252787568, buffer=0x387c170 "Oð\024\017", buffer@entry=0x6a42380 "") at smgr.c:628
#5 0x000000000068d22e in ReadBuffer_common (smgr=0x0, relpersistence=<optimized out>, forkNum=FSM_FORKNUM, forkNum@entry=MAIN_FORKNUM, blockNum=252787568, blockNum@entry=0, mode=mode@entry=RBM_NORMAL, strategy=strategy@entry=0x0, hit=0x387c5bf "", hit@entry=0x387c5df "") at bufmgr.c:890
#6 0x000000000068dccb in ReadBufferExtended (reln=0xf113b70, forkNum=forkNum@entry=MAIN_FORKNUM, blockNum=0, mode=mode@entry=RBM_NORMAL, strategy=strategy@entry=0x0) at bufmgr.c:664
#7 0x000000000068dd98 in ReadBuffer (reln=<optimized out>, blockNum=<optimized out>) at bufmgr.c:596
#8 0x000000000045e5cf in _bt_getbuf (rel=rel@entry=0xf113b70, blkno=blkno@entry=0, access=access@entry=1) at nbtpage.c:576
#9 0x000000000045eac5 in _bt_getroot (rel=rel@entry=0xf113b70, access=1, access@entry=254581064) at nbtpage.c:158
#10 0x00000000004666e4 in _bt_search (rel=rel@entry=0xf113b70, keysz=1, keysz@entry=2, scankey=0xf141b68, scankey@entry=0x387d1e0, nextkey=nextkey@entry=0 '\000', bufP=0x387db8c, bufP@entry=0x387dbec, access=<optimized out>, access@entry=1, snapshot=<optimized out>) at nbtsearch.c:99
#11 0x0000000000467bf4 in _bt_first (scan=<optimized out>, scan@entry=0xf14ef28, dir=<optimized out>, dir@entry=ForwardScanDirection) at nbtsearch.c:983
#12 0x0000000000462f5d in btgettuple (scan=0xf14ef28, dir=ForwardScanDirection) at nbtree.c:321
#13 0x0000000000458d3e in index_getnext_tid (scan=scan@entry=0xf14ef28, direction=direction@entry=ForwardScanDirection) at indexam.c:415
#14 0x000000000045906d in index_getnext (scan=0xf14ef28, direction=direction@entry=ForwardScanDirection) at indexam.c:553
#15 0x0000000000457e42 in systable_getnext (sysscan=sysscan@entry=0xf14eed0) at genam.c:416
#16 0x00000000007ac61f in SearchCatCache (cache=0x159d38, v1=<optimized out>, v2=<optimized out>, v3=v3@entry=0, v4=0, v4@entry=1) at catcache.c:1248
#17 0x00000000007b9ec8 in SearchSysCache (cacheId=cacheId@entry=44, key1=<optimized out>, key2=<optimized out>, key3=key3@entry=0, key4=0, key4@entry=92863616) at syscache.c:982
#18 0x00000000007b9fa5 in GetSysCacheOid (cacheId=cacheId@entry=44, key1=<optimized out>, key2=<optimized out>, key3=key3@entry=0, key4=key4@entry=0) at syscache.c:1060
#19 0x00000000007bc2cb in get_relname_relid (relname=<optimized out>, relnamespace=<optimized out>) at lsyscache.c:1653
#20 0x00000000004bb000 in RangeVarGetRelidExtended (relation=relation@entry=0xf14e300, lockmode=lockmode@entry=1, missing_ok=missing_ok@entry=1 '\001', nowait=nowait@entry=0 '\000', callback=callback@entry=0x0, callback_arg=callback_arg@entry=0x0) at namespace.c:321
#21 0x00000000004432da in relation_openrv_extended (relation=relation@entry=0xf14e300, lockmode=lockmode@entry=1, missing_ok=missing_ok@entry=1 '\001') at heapam.c:1242
#22 0x000000000044352c in heap_openrv_extended (relation=relation@entry=0xf14e300, lockmode=lockmode@entry=1, missing_ok=missing_ok@entry=1 '\001') at heapam.c:1348
#23 0x000000000050eef4 in parserOpenTable (pstate=pstate@entry=0xf14eb88, relation=relation@entry=0xf14e300, lockmode=1) at parse_relation.c:1131
#24 0x00000000005104f2 in addRangeTableEntry (pstate=pstate@entry=0xf14eb88, relation=relation@entry=0xf14e300, alias=0x0, inh=<optimized out>, inFromCl=inFromCl@entry=1 '\001') at parse_relation.c:1196
#25 0x00000000004fa617 in transformTableEntry (r=0xf14e300, pstate=0xf14eb88) at parse_clause.c:439
#26 transformFromClauseItem (pstate=pstate@entry=0xf14eb88, n=0xf14e300, top_rte=0x8, top_rte@entry=0x387e370, top_rti=0xf14dca8, top_rti@entry=0x387e36c, namespace=namespace@entry=0x387e378) at parse_clause.c:872
#27 0x00000000004fbe93 in transformFromClause (pstate=pstate@entry=0xf14eb88, frmList=<optimized out>) at parse_clause.c:130
#28 0x00000000004dd2a9 in transformSelectStmt (stmt=0x0, pstate=0xf14eb88) at analyze.c:1156
#29 transformStmt (pstate=pstate@entry=0xf14eb88, parseTree=0x0, parseTree@entry=0xf14e898) at analyze.c:270
#30 0x00000000004def9a in transformTopLevelStmt (pstate=pstate@entry=0xf14eb88, parseTree=parseTree@entry=0xf14e898) at analyze.c:215
#31 0x00000000004df01c in parse_analyze (parseTree=parseTree@entry=0xf14e898, sourceText=sourceText@entry=0xf14bb20 "SELECT NULL FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200", paramTypes=paramTypes@entry=0x0, numParams=numParams@entry=0) at analyze.c:108
#32 0x00000000006bd49f in pg_analyze_and_rewrite (parsetree=0xf14e898, query_string=0xf14bb20 "SELECT NULL FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200", paramTypes=0x0, numParams=0) at postgres.c:658
#33 0x00000000005be25d in _SPI_execute_plan (plan=0x898, plan@entry=0x387e770, paramLI=0x13af18, paramLI@entry=0x0, snapshot=0x387e840, snapshot@entry=0x0, crosscheck_snapshot=0x6538354f <plr_load_modules+82>, crosscheck_snapshot@entry=0x0, ) at spi.c:2077
#34 0x00000000005bea0b in SPI_execute (src=0xf14bb20 "SELECT NULL FROM pg_catalog.pg_class WHERE relname = 'plr_modules' AND relnamespace = 2200", read_only=read_only@entry=0 '\000', tcount=tcount@entry=1) at spi.c:383
#35 0x00000000005bea45 in SPI_exec (src=<optimized out>, tcount=tcount@entry=1) at spi.c:395
#36 0x000000006538354f in haveModulesTable (nspOid=2200) at plr.c:1945
#37 plr_load_modules () at plr.c:500
#38 0x0000000065384148 in plr_init_all (funcid=17045) at plr.c:577
#39 plr_call_handler (fcinfo=0xf145f80) at plr.c:211
#40 0x0000000000598027 in ExecMakeFunctionResult (fcache=fcache@entry=0xf145f10, econtext=econtext@entry=0xf145ce8, isNull=isNull@entry=0xf1468c0 "", isDone=isDone@entry=0xf146a00) at execQual.c:1840
#41 0x0000000000598264 in ExecEvalFunc (fcache=0xf145f10, econtext=0xf145ce8, isNull=0xf1468c0 "", isDone=0xf146a00) at execQual.c:2437
#42 0x000000000059b4ad in ExecTargetList (isDone=0xf145bd0, itemIsDone=0xf146a00, isnull=0xf1468c0 "", values=0x1, econtext=0x0, tupdesc=<optimized out>, targetlist=0xf145ce8) at execQual.c:5486
#43 ExecProject (projInfo=<optimized out>, isDone=0xf145bd0, isDone@entry=0x387ec1c) at execQual.c:5710
#44 0x00000000005b18c1 in ExecResult (node=node@entry=0xf145bd0) at nodeResult.c:155
#45 0x0000000000593ff1 in ExecProcNode (node=node@entry=0xf145bd0) at execProcnode.c:392
#46 0x000000000058ffde in ExecutePlan (dest=0xf143dd8, direction=<optimized out>, numberTuples=0, sendTuples=1 '\001', operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0xf145bd0, estate=0xf145ab8) at execMain.c:1567
#47 standard_ExecutorRun (queryDesc=0x1cd948, direction=<optimized out>, count=0) at execMain.c:338
#48 0x000000000059009a in ExecutorRun (queryDesc=queryDesc@entry=0x1cd948, direction=direction@entry=ForwardScanDirection, count=count@entry=0) at execMain.c:286
#49 0x00000000006c0a03 in PortalRunSelect (portal=0x1dc348, portal@entry=0xf143dd8, forward=forward@entry=1 '\001', count=0, count@entry=252984792, dest=dest@entry=0xe2ba8) at pquery.c:948
#50 0x00000000006c20f9 in PortalRun (portal=0xf143dd8, portal@entry=0x1dc348, count=252984792, count@entry=2147483647, isTopLevel=isTopLevel@entry=0 '\000', dest=0xe2ba8, dest@entry=0xf143dd8, altdest=<optimized out>, altdest@entry=0xf143dd8, completionTag=0x387f180 "", completionTag@entry=0x387f1b8 "¸ãi") at pquery.c:789
#51 0x00000000006be813 in exec_simple_query (query_string=0x7cd03f <string_compare+17> "HƒÄ ]ÃUH‰åHƒì H‰ÊH‹\réT*") at postgres.c:1094
#52 PostgresMain (argc=<optimized out>, argv=argv@entry=0x1344d8, dbname=<optimized out>, username=<optimized out>) at postgres.c:4070
#53 0x0000000000653d3a in BackendRun (port=0x387fc30) at postmaster.c:4260
#54 SubPostmasterMain (argc=argc@entry=3, argv=argv@entry=0x10ffa0) at postmaster.c:4750
#55 0x00000000005cf6b8 in main (argc=3, argv=0x10ffa0) at main.c:205
#0 PathNameOpenFile
... DO NOT CONTINTUE ... NEXT ... NEXT ... STEP INTO INTO BasicOpenFile
> bt
#0 BasicOpenFile (fileName=fileName@entry=0xf14f040 "base/12418/2663", fileFlags=fileFlags@entry=32770, fileMode=fileMode@entry=384) at fd.c:902
#1 0x0000000000690b2b in PathNameOpenFile (fileName=fileName@entry=0xf14f040 "base/12418/2663", fileFlags=fileFlags@entry=32770, fileMode=fileMode@entry=384) at fd.c:1252Here, in BasicOpenFile, it tries to open the file twice. int
BasicOpenFile(FileName fileName, int fileFlags, int fileMode)
{
int fd;
tryAgain:
fd = open(fileName, fileFlags, fileMode); *** NET: WAS TRIED TWICE ***
if (fd >= 0)
return fd; /* success! */
if (errno == EMFILE || errno == ENFILE) *** 1ST ROUND IS TRUE *** *** 2ND ROUND IS FALSE ***
{
int save_errno = errno;
ereport(LOG, *** 1ST ROUND EXECUTED ONCE ***
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
errmsg("out of file descriptors: %m; release and retry")));
errno = 0;
if (ReleaseLruFile())
goto tryAgain; *** END OF FIRST ROUND ...JUMP TO 'tryAgain:' ***
errno = save_errno;
}
return -1; /* failure */ *** 2ND ROUND FAIL AND RETURN -1 ***
}EMFILE is slightly explaned here The comment is read. Linux is explained here. It seem here that in 2013 Linux had over 1000 file descriptors I am on Windows, so what number do I have? TargetUser@TARGETMACH /c/Users/Public
$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 2046
cpu time (seconds, -t) unlimited
max user processes (-u) 63
virtual memory (kbytes, -v) 2097152So, Here is mine. TargetUser@TARGETMACH /c/Users/Public
$ ulimit --help
sh: ulimit: --: invalid option
ulimit: usage: ulimit [-SHacdfilmnpqstuvx] [limit]Can I use ulimit? This guy here says that on Windows using unlimit to try to 'set' will not work. He ( 14 years ago: 2004? ) says do this. This other guy writes this in 2015. Specifically, he says. Hopefully, this is something to try. I am tired. I am going to bed again. |
|
Andre, Good work!
This has nothing to do with the stack. http://stackoverflow.com/questions/35690916/windows-git-bash-ulimit-too-many-open-files Suggests that ulimit can be used on Windows. Dave |
|
The science(so far) behind the problem is the following. When postgreSQL (on windows) boots up ( and tries to open files) For example, Maybe 50 or so of them are generated. These is DOS error 2 DOS Error Codes This error is also know as ERROR_FILE_NOT_FOUND These are called in the function in here. src/port/win32error.c void _dosmaperr(unsigned long e)Execution goes through this line. mapped win32 error code %lu to %d"So these errors are printed as These have been results of the Window function CreateFile() Also, create extension generates a file open misses (error 2) whenever, I try This happens on every contrib extension, I have tried Later, just after I type "select r_version(); The function returns INVALID_HANDLE_VALUE So, the output string is different. I have tried two hacking attempts at a solution. Try Number 1 I tried to eliminate any kind of OS buffering. So, I changes this C:\postgres-master_0ab9c56_debug\src\port pgwin32_open(const char *fileName, int fileFlags,...)
FILE_ATTRIBUTE_NORMAL |
((fileFlags & O_RANDOM) ? FILE_FLAG_RANDOM_ACCESS : 0) |
((fileFlags & O_SEQUENTIAL) ? FILE_FLAG_SEQUENTIAL_SCAN : 0) |
((fileFlags & _O_SHORT_LIVED) ? FILE_ATTRIBUTE_TEMPORARY : 0) |
((fileFlags & O_TEMPORARY) ? FILE_FLAG_DELETE_ON_CLOSE : 0) |
((fileFlags & O_DIRECT) ? FILE_FLAG_NO_BUFFERING : 0) |
((fileFlags & O_DSYNC) ? FILE_FLAG_WRITE_THROUGH : 0),
NULL)) == INVALID_HANDLE_VALUE)to this FILE_ATTRIBUTE_NORMAL |
((fileFlags & O_RANDOM) ? FILE_FLAG_RANDOM_ACCESS | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING : 0 ) |
((fileFlags & O_SEQUENTIAL) ? FILE_FLAG_SEQUENTIAL_SCAN | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING : 0 ) |
((fileFlags & _O_SHORT_LIVED) ? FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING : 0 ) |
((fileFlags & O_TEMPORARY) ? FILE_FLAG_DELETE_ON_CLOSE | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING : 0 ) |
((fileFlags & O_DIRECT) ? FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING : 0 ) |
((fileFlags & O_DSYNC) ? FILE_FLAG_WRITE_THROUGH | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING : 0 ),
NULL)) == INVALID_HANDLE_VALUE)This ended up having no effect. Try Number 2 I tried to have CreateFile 'try more often.' So, I replaced src/backend/storage/file BasicOpenFile(FileName fileName, int fileFlags, int fileMode
int
BasicOpenFile(FileName fileName, int fileFlags, int fileMode)
{
int fd;
tryAgain:
fd = open(fileName, fileFlags, fileMode);
if (fd >= 0)
return fd; /* success! */
if (errno == EMFILE || errno == ENFILE)
{
int save_errno = errno;
ereport(LOG,
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
errmsg("out of file descriptors: %m; release and retry")));
errno = 0;
if (ReleaseLruFile())
goto tryAgain;
errno = save_errno;
}
return -1; /* failure */
}with this code here Below, BasicOpenFile(FileName fileName, int fileFlags, int fileMode)
{
int fd;
tryAgain:
fd = open(fileName, fileFlags, fileMode);
if (fd >= 0)
return fd; /* success! */
if (errno == EMFILE || errno == ENFILE )
{
int save_errno = errno;
ereport(LOG,
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
errmsg("out of file descriptors: %m; release and retry")));
errno = 0;
if (ReleaseLruFile())
goto tryAgain;
errno = save_errno;
}
if (errno == ENOENT || errno == ENOENT )
{
int save_errno = errno;
ereport(LOG,
(errno,
errmsg("BasicOpenFile errno ==ENOENT || ENOENT: %m")));
errno = 0;
if (ReleaseLruFile())
goto tryAgain;
errno = save_errno;
}
return -1; /* failure */
}That did not help either. Tt only made everything run 'super super slow.' Currently, I have not found a 'direct' solution Since, the error occurred just after submitting the SQL that src/contrib/plr I replaced this ... with this ... static bool
haveModulesTable(Oid nspOid)
{
SPI_processed = 0;
return SPI_processed == 1;
}So, haveModulesTable always returns FALSE and 'modules related SQL' is elminated. postgres=# select r_version();
r_version
-------------------------------------------------
(platform,x86_64-w64-mingw32)
(arch,x86_64)
(os,mingw32)
(system,"x86_64, mingw32")
(status,"")
(major,3)
(minor,3.1)
(year,2016)
(month,06)
(day,21)
("svn rev",70800)
(language,R)
(version.string,"R version 3.3.1 (2016-06-21)")
(nickname,"Bug in Your Hair")
(14 rows)However, this does not free me completely from those errors. postgres=# drop extension plr;
ERROR: could not open file "base/12418/2673": No such file or directory
postgres=#
postgres=# drop extension plr;
ERROR: cannot drop extension plr because other objects depend on it
DETAIL: function r_max(integer,integer) depends on language plr
HINT: Use DROP ... CASCADE to drop the dependent objects too.
postgres=# drop extension plr cascade;
NOTICE: drop cascades to function r_max(integer,integer)
ERROR: could not open file "global/1232": No such file or directoryExiting out of psql and re-opening a new psql and My question is the following. What does MemoryContextSwitchTo do? I found these plr_caller_context = CurrentMemoryContext;
MemoryContextSwitchTo(plr_caller_context);Also I found these. SWITCHTO_PLR_SPI_CONTEXT(oldcontext)
CLEANUP_PLR_SPI_CONTEXT(oldcontext);Those last macros are defined in plr.h #define SWITCHTO_PLR_SPI_CONTEXT(the_caller_context) \
the_caller_context = MemoryContextSwitchTo(plr_SPI_context)
#define CLEANUP_PLR_SPI_CONTEXT(the_caller_context) \
MemoryContextSwitchTo(the_caller_context)MemoryContextSwitchTo is here MemoryContextSwitchTo(MemoryContext context)
{
MemoryContext old = CurrentMemoryContext;
CurrentMemoryContext = context;
return old;
}MemoryContextData is here /*
* Type MemoryContextData is declared in nodes/memnodes.h. Most users
* of memory allocation should just treat it as an abstract type, so we
* do not provide the struct contents here.
*/
typedef struct MemoryContextData *MemoryContext;
/*
typedef struct MemoryContextData
{
NodeTag type; /* identifies exact kind of context */
/* these two fields are placed here to minimize alignment wastage: */
bool isReset; /* T = no space alloced since last reset */
bool allowInCritSection; /* allow palloc in critical section */
MemoryContextMethods *methods; /* virtual function table */
MemoryContext parent; /* NULL if no parent (toplevel context) */
MemoryContext firstchild; /* head of linked list of children */
MemoryContext prevchild; /* previous child of same parent */
MemoryContext nextchild; /* next child of same parent */
char *name; /* context name (just for debugging) */
MemoryContextCallback *reset_cbs; /* list of reset/delete callbacks */
} MemoryContextData;When a memory context is turned 'on' what is happening? Thanks, |
PostgreSQL has code to manage it's own memory it's like a poor man's All memory you allocate in a context can be freed by freeing the context.
So the question you have to be asking yourself here is why does this only I cannot replicate this problem, nor does anyone else have the problem. If you take a bare machine, and install windows, R, Postgres does this Dave Cramer |
|
I found another bare machine. I will try the original pl/r ( without my hack ) on that machine. I will post back and explain what happens. |
|
Dave Cramer, I now again have a working PostgreSQL 9.6.1 pl/r 08.03.00.17 ( https://github.com/postgres-plr/plr ) to R 3.3.2 on my old 11 years old Windows 7 computer. This pl/r is without ( unlike before ) any modifications (omissisions of code) to/from plr.c I did only the following. Used 9.6.1 code exactly. Compiled/made pl/r in in an external directory that is different form the source. Specifically, in this external directory, I did the following
I also built non-debug everything. Specifically, I ran ./config (non-debug) and 'make' on the PostgreSQL source code and pl/r code. What is different from before, is that that I can no longer Before in Postgre 10devel of early September and PostgreSQL 9.5.x I could I do not know if this 'workingness' will keep up. postgres=# create extension plr;
CREATE EXTENSION
postgres=# select r_version();
r_version
-------------------------------------------------
(platform,x86_64-w64-mingw32)
(arch,x86_64)
(os,mingw32)
(system,"x86_64, mingw32")
(status,"")
(major,3)
(minor,3.2)
(year,2016)
(month,10)
(day,31)
("svn rev",71607)
(language,R)
(version.string,"R version 3.3.2 (2016-10-31)")
(nickname,"Sincere Pumpkin Patch")
(14 rows)postgres=# select plr_version();
plr_version
-------------
08.03.00.17 # https://github.com/postgres-plr/plr
(1 row)postgres=# select reload_plr_modules();
reload_plr_modules
--------------------
OK
(1 row)postgres=# select version();
version
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 9.6.1_CFLAGS_default_REL9_6_1_0ab9c56 on x86_64-w64-mingw32, compiled by x86_64-w64-mingw32-gcc.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 6.2.0, 64-bit
(1 row) |
|
Dave and Joe, From my 100+ hours of hard trial and error, I have came to this following conclusion. The case seem that sometimes a valid path exists, but Windows 'sometimes' forgets how to read the valid path. This problem seem to be isolated at 64bit Windows operating systems. 32bit Windows operating systems do not seem to have this problem. This does not make any sense but this a fact. For some strange bad luck, the unique way that plr is written ( but written well! ) with both C and SQL seems to irritate 64bit Windows leading it to 'forgetting how to read a valid path.' When I run into this problem, to make a solution, this is what I do. I copy all of my PostgreSQL files to I copy all of my R files to I change my PostgreSQL startup script PostgreSQL.9.6._.bat. Specifically,I change my R location to After than, I startup PostgreSQL. Everything works wonderful. I have done this solution three times so far( once on 9.5 and twice on 9.6). So, in a nutshell, sometimes 64bit Windows forgets how to read a valid path. So, when this happens, I just 'try another path, that is a 'very short' and a 'very simple' path. I do this for both PostgreSQL and R ( but I think that the R path is the one that really needs be 'short and simple') Example. The end. Please, close this ticket. |
AndreMikulec
changed the title
AndreMikulec
changed the title
|
Hi Andre, You don't set the variables in the system environment ? |
|
Dave, I do set my system variables in a 'startup' .bat script similar to ( and originally based on ) how garethflowers ( Gareth Flowers https://sourceforge.net/u/garethflowers/profile/ http://gareth.flowers) |
|
Update: The new pattern seems that I can only get 64-bit pl/r to work on Windows (10) computers that are running using an i7 processor. I have two machines that match this pattern. If any other processor ( i.e./e.g. i5 processor in my 3rd computer of the four) ) is tried than pl/r returns that old error. postgres=# select r_max(5,3);
ERROR: could not open file "base/12373/2663": No such file or directory
LINE 1: SELECT NULL FROM pg_catalog.pg_class WHERE relname 'plr_modules' AND relnamespace = 2200 |
I tried
Win64 plr.dll for Postgres 9.5.x, R-3.3.0 (plr-8.3.0.16)
on
with
on a new-ish Windows 10 64 bit laptop that is only 18 months old.
I am getting this error again.
So, my 'old hardware' guess is wrong.
pl/r runs fine on Windows 7 64 bit
What is the easiest way ( or an easy way ) to try to debug this problem?
The text was updated successfully, but these errors were encountered: