-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathCHANGES
700 lines (530 loc) · 21.8 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
=========== 4.X.X
* PBSQuery bugfix, parsing of node status line with multiple EVENT: settings
was wrong
Author: Bas van der Vlies
=========== 4.3.4
* PBSQuery bugfix. getjob() can now handle short jobname, eg:
* p.getjob('6073545')
Author: Willem Vermin (SARA) & Bas van der Vlies
* added an example: pbs_jobmonitor shows the output of top
-u user on the node
- one cycle of top
- user: the user the job belongs to
renamed as pbs_joblogin logs in to the node as the user
who invokes this script (os.getenv('USER'))
Author: Willem Vermin (SARA)
=========== 4.3.3
* New generated files for pbs_wrap.c and pbs.py to support python 3.X versions
Reported by: Steve Traylen <steve dot traylen add cern dot ch>
Author: Bas van der Vlies
* Fixed AdvancedParser when using 01-12 range the zero was not appended.
Reported by: Ramon Bastiaans
Author: Dennis Stam
* examples/sara_nodes.py: Catch the PBSQuery error if we can not make a
connection with the batch server and exit the program.
Author: Bas van der Vlies
* Remove the debian dependency on AdvancedParser. We now have
our own PBSAdvanceParser so it does not conflict with other SARA
packages.
Reported by: Ramon Bastiaans (SARA)
Fixed by: Bas van der Vlies
* Make pbs_torque only dependend on libtorque instead of torque and adjusted
the maintainer of the package, closes #30
Reported by: Guillermo Marcus <guillermo dot marcus add@gmail dot com>
Fixed by: Bas van der Vlies
=========== 4.3.0
* new_rack_pbsmon.py: For backward compatibility added -a/--all option.
Has the same functionality as -s/--summary
Author: Bas van der Vlies
* Added some license info to debian package and include GPL and LGPL 3.0
files.
Suggested by: W. Trevor King <wking at drexel dot edu>
* PBSQuery.py: Added support for new feature in node status field:
* message=EVENT:<key=value:>+
* message=ERROR <text>
Added the following syntax for node:
* node['error']
* node['event']
eg: node: n1
state = free
np = 8
properties = ib,switch1,highmem
ntype = cluster
jobs = 0/567403.sara.nl, 1/567403.sara.nl
status = ..,loadave=0.00,message=EVENT:sample.time=1288864220.003:cputotals.user=0:iconnect.pktout=0,netload=3487600394,state=free,...
example code:
p = PBSQuery()
node = p.getnode(n1)
print node.event['sample.time']
print node['event']['sample.time']
Requested by: Roy Dragseth <roy dot dragseth add uit dotno>
Author: Bas van der Vlies
=========== 4.2.0
* Updated LICENSE file to LGPLV3. Thanks to Justin Bronder
(jsbronder add gentoo dot org) for the information and
explanation.
* examples/pbsmon.py. Patched so it also works for torque 2.5
Author: Stijn De Weirdt stijn dot deweirdt at ugent dot be
Applied by: Bas van der Vlies
* setup.py.in changed determine which torque version is installed
so we also support snapshots.
Author: Mark Roberts ( mark at gingergeeks dot co dot uk)
Applied by: Bas van der Vlies
* PBSQuery:
- get_nodes function for jobs now works for old and new datastructure and
gives correct results if unique=True
Reported by + idea for patch: Shunjie Lau (eijnuhs add gmail dot com)
Author: Bas van der Vlies
* Added AdvanceParser module. new_rack_pbsmon.py and sara_nodes.py make use of
this module
Author: Dennis Stam (Dennis dot Stam at sara dot nl)
* examples/new_rack_pbsmon.py:
- Better specification for node expression
- added a new option -l/--list to list the state of a nodes and which job
is running on it, eg:
new_rack_pbsmon.py - l gb-r[1-3]n[3-6] gb-r4[n1-20]
Authors: Dennis Stam & Bas van der Vlies
Applied by: Bas van der Vlies
* examples/sara_nodes.py:
An utility to set/clear notes for multiple nodes, see ..
Author: Dennis Stam (Dennis dot Stam at sara dot nl)
* setup.py.in:
Added support for runtime-libraries when torque is installed in a non-default
location, closes #22
Author: Mattias (slabanja at chalmers dot se)
Applied by: Bas van der Vlies
* Changed build system for debian
Authors: Dennis Stam & Bas van der Vlies
=========== 4.1.2
* examples/new_rack_pbsmon.py now uses the new PBSQuery data
structure.
Author: Bas van der Vlies
* PBSQuery:
- fixed and error in get_jobs function for nodes. Did not
support new data structure and uniq did not work.
Authors: Shunjie Lau <eijnuhs at gmail dot com> and Bas van der Vlies
=========== 4.1.0
* Delete function pbs.version() replaced by
* pbs.version
This is a string variable
* pbs.version_info
Is a tuple variable. So we can use it in scripts
This analogue with sys module, closes #20
Requested by: Dennis Stam ( Dennis dot Stam add sara dot nl )
Author: Bas van der Vlies
* More robust make/configure/setup.py
Reported by:
- Saidi, Yacine <saidi add linmpi dot mpg dot de>
- Shunjie Lau <eijnuhs add gmail dot com>
Author: Bas van der Vlies
* Only use the pbs_wrap.c and pbs.py 2.4 version for torque version 2.4.7 and higher.
Reported by: Saidi, Yacine <saidi add linmpi dot mpg dot de>
Author: Bas van der Vlies
=========== 4.0.2
* Quick fix so we do not get any undefined symbols. Forgot to
add the torque library when we compile the package.
Reported by: Shunjie Lau ( eijnuhs add gmail dot com)
Author: Bas van der Vlies
=========== 4.0.1
* Fixed a bug in setting the right library path for compiling.
Reported by: Chad Vizino ( vizino add psc dot edu )
Author: Bas van der Vlies
=========== 4.0.0
* pbs.py added some new functions for 2.4 version:
- pbs_fbserver(void);
- pbs_get_server_list(void);
- pbs_sigjobasync(int connect, char *job_id, char *signal, char *extend);
- pbs_alterjob_async(int connect, char *job_id, struct attrl *ATTRL, char *extend);
- pbs_checkpointjob(int connect, char *job_id, char *extend);
- log_ext(int,char *,char *,int);
- log_init(char *, char *);
- log_remove_old(char *,unsigned long);
Note: version 2.1.X is still supported
* autotools environment
- Added support for finding pbs-config (borrowed for mpiexec source)
- Detecting which version of torque is installed
- only support installations that have pbs-config installed.
- Removed old code for installations that do not a have pbs-config
installed.
* PBSQuery.py
- There was a bug in the new data interface (Resource_List for jobs)
Reported & Patch by: Mark Roberts ( mark at gingergeeks dot co dot uk)
Applied by with some minor changes: Bas van der Vlies
- The new data structure is the default. You can switch back to the old
one with:
p = PBSQuery()
p.old_data_structure()
Author: Bas van der Vlies
* examples/ha_server.py
An example how to use the High Availabilty torque functions
Author: Bas van der Vlies & Mark Roberts ( mark at gingergeeks dot co dot uk )
=========== 3.6.0
* pbs, new generated pbs_wrap.c, pbs.py fixes a bug in pbs_runjob()
Reported by: Dennis Stam
Fixed by: Bas van der Vlies
=========== 3.5.0
* PBSQuery
The class functions of node, job and queue support old and new data
structure.
Changed the behaviour of the new data stucture, We can use it as
dictionary and as class attribute, this is equivalent, eg:
- print node['np'] and print node.np
for a node we parse the 'status' line and split on '=' char, You now can
use these statements, eg
- print node.status.arch (node['status'].arch or node['status']['arch'])
- print node.status.nsession
for a job we parse the 'Variable_List' line and split on '=' char, You now can
use the statements, eg:
- print job.Variable_List.PBS_O_WORKDIR
- print job.Variable_List.PBS_O_HOME
For more info see examples/new_interface.py
Author: Bas van der Vlies
* new_rack_pbsmon.py
Rewrite to new data structure and automatically determine how many nodes
and racks cluster has and skip printing of empty racks (default), use -w/--wide
for old behaviour.
Author: Bas van der Vlies
=========== 3.2.0
- PBSQuery:
New data structure. In the old structure it is a dictionary
with a value and the value is a string. This is changed
that dictionary values are now of type list or dictionary depends
on the value of the keyword, eg for a node:
- np = 2:
* node['np'] = [ '2' ]
- properties = cores2, mem4gb, parallel
* node['properties'] = [ 'cores2', 'mem4gb', 'parallel' ]
- status = arch=x86_64,sessions=22599,,size=70627864kb, ...
* node['status']['arch'] = [ 'x86_64' ]
* node['status']['sessions'] = [ '222599' ]
* ...
The data structure is activated by the function:
* new_data_structure()
In a future release it will be come the default.
example:
p = PBSQuery()
p.new_data_structure()
nodes = p.getnodes()
print nodes.np, nodes['np']
Author: Bas van der Vlies
- PBSQuery
For old and new data structure we now can use another syntax:
* node['np'] and node.np are equivalent
This new syntax works for all keywords.
Author: Bas van der Vlies
- Added iter object for job, node, queue and server objects, eg:
node = p.getnode('gb-r1n1')
print node.name
for attrib in node:
print '%\t%s = %s' %(attrib, node[attrib])
Author: Bas van der Vlies
- fixed an error in getnode, getqueue and getjob, return empty
dictionary if not found
Author: Bas van der Vlies
- New build system for rpm packages, make -f Makefile.rpm
Author: Michel Jouvin <jouvin add lal dot in2p3 dot fr>
Applied: Bas van der Vlies
=========== 3.0.1
- PBSQuery
* Removed a testing code line
Reported By: sam @ vpac dot org
Fixed by: Bas van der Vlies
=========== 3.0.0
- PBSQuery
* Cleaned up some old lines of code which didn't do anything.
* Changed the returnvalue of getqueue, getnode and getjob. Now
they return an instance of the object, instead of a dict.
This is API change so we bumped the version to 3.0.0
* Now the getnode-function accepts the short name as well as
the long name as an argument.
Author: Sil Westerveld <[email protected]>
- pbs.py, _pbs.so, PBSQuery.py
* Fixed a memory leak when used in daemon mode
Author: Bas van der Vlies <[email protected]>
=========== 2.9.8
- pbs_python.spec file patch added libdir and python defines so it
will build for RHEL5.1/CentOS 5.1
by : Michael Sternberg <sternberg add anl dot gov>
applied by: Bas van der Vlies
- Updated the header files for pbs_python. So it can use some new
defines variables/defines, eg:
* ATTR_NODE_status
* ATTR_NODE_note
* ...
Added by: Dennis Stam <[email protected]>, Bas van der Vlies
- Two new functions to log.h:
log_roll(int size);
log_size(void);
Added by: Bas van der Vlies
- We now have proper destructors (free memory) in SWIG for attrl and
attropl attributes.
Added by: Bas van der Vlies
- PBSQuery
* Added getnodes_with_property() function. This will get all nodes with
a certain property, eg:
- get all nodes with property 'cores8' and return all attributes:
getnodes_with_property('cores8')
- get all nodes with property 'cores2' and only return the
attributes 'state' and 'properties':
attrl = [ 'state', 'properties' ]
getnodes_with_property('cores2', attrl)
Only for torque version > 2.1.0
Added by: Bas van der Vlies
* Added to class node the function
- get_jobs(self, unique=None)
Returns a list of the currently running job-id('s) on the node.
Added by: Sil Westerveld <[email protected]>
* Added get_server_name() function.
This will return the PBS-server's name.
Added by: Sil Westerveld <[email protected]>
* Added to class _PBSobject the function
- uniq(self, list)
Filters out unique items of a list.
Added by: Sil Westerveld <[email protected]>
* Added to class job the function
- get_nodes(self, unique=None)
Returns a list of the nodes which run the job.
Added by: Sil Westerveld <[email protected]>
=========== Version 2.9.4
- Function pbs.pbs_geterrmsg does not exits any more is
replaced by pbs.error():
errno, text = pbs.error()
print errno, text
Reported by: pk at q-leap dot com
Fixed by : Bas van der Vlies
- PBSQuery.py
* Added get_version() for server class, returns the version of
the batch server
Added by : Bas van der Vlies
* Added get_nodes() for job class, returns a list of nodes on
which the job is run
Added by : Bas van der Vlies
* PBSQuery.py rewrite make use of UserDict module.
Suggested by: Ramon Bastiaans
Fixed by : Bas van der Vlies
=========== Version 2.9.2
- The function pbs_rescquery() did not work. Fixed it
- The function avail(), did not work fixed it
- The functions pbs_rescreserve and pbs_rescrelease did not work
fixed it
- Fixed a bug in new_rack_pbsmon.py. Counting of serial
nodes was wrong and state down has topmost priority
=========== Version 2.9.0
- Added support for new torque build system 2.1.0
- Added new_rack_pbsmon.py
Authors: Walter de Jong & Bas van der Vlies
=========== Version 2.8.2
Configure has a new option:
--with-pbsdir=PATH
If set it will use PATH specified as argument to
find the pbs/torque libraries. If unset it will use
some useful defaults to find the libaries.
Suggested by: Troy Baer
PBSQuery.py:
- If we can not make a connection with the server
then raise an exception and let the programmer
decides what to do.
pbs_python.spec:
- It can now handle mode bits
Author: Martin Pels from SARA
=========== Version 2.8.0
setup.py:
- Added /usr/lib to PBS_LIB_DIR
pbs.py:
- Added new function "pbs.error()". This function checks if
an error has occured with a pbs function, eg:
task_id = pbs.pbs_submit(c, attrl, "A.tsk", 'NULL', 'NULL')
error_number, err_txt = pbs.error()
if error_number:
print error_number, err_txt
prints the following message if script A.tsk does not exists:
15042 (qsub) cannot access script file
pbs_python.spec:
- to make rpm packages from the source package, initial version
Author: Martin Pels from SARA
README:
- Now requires python 2.1 and higher
- Explain how to build DEBIAN package
=========== Version 2.7.10
PBSQuery.py:
Fixed an error. You always got all resources from eg nodes
even when you supplied an attribute list where you only
requested the 'state' of the node.
Updated the inline python documentation
=========== Version 2.7.9
Fixed an error the previous version was in DEBUG mode. So
you get an lot undesired output on the screen.
debian package now also generated a package with the write
version info.
=========== Version 2.7.8
Setup.py:
Can also handle ROCKS installations + the version
info is now the same as pbs.py
Thanks to: Roy Dragseth
=========== Version 2.7.7
PBSQuery module:
It can now be used in daemon programs. The pbs_server closes
the connection after an certain amount of time. This is fixed,
before every query there is an new connection and if the query
is finished the connection is closed.
=========== Version 2.7.6
Now setup.py can also handle old Oscar Installations.
Thanks To: Robin Humble
Fixed an print statement fix in new_interface.py:
Thanks To: Robin Humble
Made some improvements for PBSQuery module:
- Updated the documentation
- Added the has_key() function to all
PBSObjects. So that the behaviour is similar to
an dictionary
Suggested By: Ramon Bastiaans
=========== Version 2.7.5
Added PBSQuery module. This module requires pbs.py and it a simple module
for querying the pbs server. The documentation is in the module. Use for
example ipython to read it.
Written By: Roy Dragseth
Bas van der Vlies
eg: ( see also examples/new_interface.py)
from PBSQuery import PBSQuery
p = PBSQuery()
jobs = p.getjobs()
for name, job in jobs.items()
print job
=========== Version 2.7.4
New versions fixes an bug in function pbs.pbs_statnode().
Reported by: Keith Poirier
Configure support:
Contributed By: Yaroslav Halchenko
Debian package support:
Contributed By: Yaroslav Halchenko
Added Support for Debuging the interface, You must edit pbs_wrap.c
and search for SARA_DEBUG:
#define SARA_DEBUG 1
Implemented by: Ramon Bastiaans
examples/pbsmon.py:
- Fixed an error when regex fails to determine node number,
no status was displayed
- It will now display and 'j' if the node is free for the batch
system and a job runs on the node (SMP-systems).
Changed by: Bas van der Vlies
examples/rack_pbsmon.py:
- An pbsmon that display node info per rack
- Edit the rack_pbsmon.py to adjust the values. It requires
that hostname contains rack and node id's,
eg: gb-r<number>n<number>
Contributed by: Walter de Jong
=========== Version 2.7.3
The name SPBS is changed to the new name TORQUE (Tera-scale Open-source
Resource and QUEing manager).
The interface support OPENPBS and TORQUE.
=========== Version 2.7.2
Some minor changes to include files. Now the Scalable PBS keywords
are also supported ( server attribute names: node_ping_rate and
node_check_rate). This does not interfere with the openpbs software.
pbsmon.py could not handle an one node cluster. Fixed it.
=========== Version 2.7
Forgot to wrap the pbs_statfree() function. So we could not free
allocated memory from functions that return 'struct batch_status *'
like pbs_statjob().
pbsmon.py can now handle 2 and 3 digit hostnames. Patch supplied by
Daniel Olson
=========== Version 2.6.1
Setup.py now checks if all openpbs libraries are installed to compile the
package.
=========== Version 2.6
Fixed a bug in the pbs python module. Forgot to map the pbs_statjob()
function. This bug prevented to pass 'struct attrl' variables to this
function. Thanks to Evelyn Shiu for reporting this bug.
=========== Version 2.5
Added the OpenPbs logging functions (log.h and liblog.a). There
is an example in the examples directory: logpbs.py
Fixed a bug in examples/resmom_info.py used the default port for
pbs_resmon if getservbyname fails.
Fixed a bug in resmom code. If the user has no permissions to query
the pbs_mom daemon a empty string is returned. Now we check for
the empty string. So we do not get a python exception.
Fixed a bug in setup.py forgot the exit statement when we did not
find the pbs libraries.
=========== Version 2.3
Bugfix in the pbsnodes-a.py. Forgot to import the sys module.
Removed some obsolete code.
As for now we make use of the distutils module. So the package
is automatically compiled and installed in proper directories, usage:
python setup.py install
This requires that the package 'distutils' must be installed. As for
version 1.6 and higher it is in the distribution.
Added a function that prints the pbs python interface version. So we
can check if we have the right interface version:
import pbs
print pbs.version()
=========== Version 2.2
In this release we can query the pbs_mom daemon with the aid of
resource management functions:
pbs.openrm()
pbs.closerm()
pbs.downrm()
pbs.configrm()
pbs.addreq()
pbs.allreq()
pbs.flushreq()
pbs.activereq()
pbs.fullresp()
pbs.getreq()
There is also a python function:
pbs.get_mom_values(id, [list]):
id - connection id of pbs.openrm(node-name, port-number)
list - Is a optional paramter. A list of resource keywords
If 'list' is not supplied then this function will get the values
for the standard resource keywords, eg uname, loadave, .. + 'arch'
depended keywords. The arch feature is only implemented for linux,
but it can easily be ported to other oses.
if 'list' is supplied then we only fetch values for the keywords
that are in the list.
The function returns a dictonary. The keys are the resource keywords.
See 'resmom_info.py' for a example of this new feature.
=========== Version 2.0
The previous version was a simple interface above the PBS C API LIB. This
interface has gone a major changes. The most noticeable change is that
the functions accept/return Python lists instead of C-lists. So you can now
use standard Python syntax for manipulating Python lists. If you are familar
with the Python syntax then you will appreciate this new interface.
NOTE:
This interface is NOT compatible with the old one
Here are some guidelines to convert your code to the new one:
The constructors for the struct has gone a major change:
old code:
temp = pbs.new_attrl();
attrl_p = pbs.attrlPtr(temp)
attrl_p.name = 'state'
new_code:
attr_l = pbs.new_attrl(1) // Creates a list of attrl structs length 1
attr_l[0].name = 'state'
The pbs_stat functions returns Python lists instead of C-lists. There is
NO next field anymore:
old_code:
temp = pbs.new_attrl();
attrl_p = pbs.attrlPtr(temp)
attrl_p.this = 'NULL'
nodes = pbs.pbs_statnode(con, "", attrl_p, "NULL")
while nodes.this != 'NULL':
print nodes.name
node_attrl = nodes.attribs
while node_attrl.this != 'NULL':
print '\t', node_attrl.name, '=', node_attrl.value
node_attrl = node_attrl.next
nodes = nodes.next
new_code:
nodes = pbs.pbs_statnode(con, "", "NULL", "NULL")
for node in nodes:
print node.name
for attrib in node.attribs:
print '\t', attrib.name, '=', attrib.value
Another advantage is you can use the print statement to show the
connect of attrl and attropl classes:
attr_l = pbs.new_attrl(2)
attr_l[0].name = 'bas'
attr_l[0].value = 'van der Vlies'
print attr_l[0]
>> (bas,,van der Vlies)
I hope these examples illustrate the changes. If you specify a wrong type
for a function then function wil raise a Python exception.