forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
3605 lines (2668 loc) · 131 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v7r0-pre35]
FIX: removed __init__ files from test directories to avoid importing
NEW: Change the schema of the Resources description in CS, use new Resources helper to
interpret this information
NEW: Base RSS resources representation on the new CS Resources schema
NEW: RSS Pilots Efficiency Policy
NEW: Added Licence statement available in all the commands through --license switch
*Core
FIX: multiple fixes to remove pylint complaints
CHANGE: Moved log coloring to utilities so it can be used consistently across DIRAC
FIX: Workflow package - fix wild wild card imports resulting in circular imports
NEW: MySQL - assign one MySQL connection per thread
FIX: LocalConfiguration - if short option is not set, do not print "-" in help message,
closes #1371
FIX: dirac-install - update creation of bashrc/cshrc to allow multiplatform installations
NEW: InstallTools - check inheritance of modules to be loaded to determine their types;
executors can now be handled by dirac-setup-site
FIX: LocalConfiguration - allow '-' character in long script options, e.g. "--long-script-option"
CHANGE: Removed wild card imports from DIRAC.__init__
NEW: dirac-install allows projects to define mirrors from where to download other projects
(DIRAC mirrors for instance)
FIX: Removed deprecation warning when importing MySQLdb package
FIX: BaseClient - take into account DISET decorator
CHANGE: MySQL - added okIfTableExists flag to the _createTables() method, in case of OK
returns a list of created tables
*Accounting
FIX: AccountingDB - align properly days with MySQL bucketing. Closes #1219
FIX: DataStoreClient - make a deep copy of the records' values
NEW: Accounting selectors will allow registered users to choose from all possible values
except from users and groups
FIX: AccountingDB - randomize the order type insertion in order to avoid type starvation
NEW: AccountingDB - add a monitoring record for each type in IN tables
*Framework
FIX: ProxyDB - prevent duplicate key errors on writing VOMSProxies to DB. Closes #1228
BUGFIX: NotificationDB - missing "," in SQL, closes #1373
FIX: dirac-proxy-get-uploaded-info.py - typo Value -> Message
*Configuration
NEW: Resources helper class to work with the new /Resources structure according to RFC #5
NEW: dirac-configuration-convert-resources-schema - command to convert old /Resources schema
to the new one
*Interfaces
CHANGE: Job.py - setPlatform renamed to setSubmitPools
*DMS
CHANGE: MigrationMonitoringAgent - removed obsoleted
NEW: FileCatalogClient - getFileUserMetadata to get both file and directory metadata for a given file
CHANGE: FileCatalogClient - list meta data sets
FIX: FileCatalog - extra check while new user registration for already existing entry
FIX: FileManagerBase - clean the FC_FileAncestors table when removing a file
BUGFIX: ReplicaManager - pass catalog argument in putAndRegister to registerFile call, closes #1369
FIX: SRM2Storage - in getCurrentStatus() use pythonCall to impose a timeout on top of
lcg_util.lcg_stmd call
NEW: IRODSStorageElement.py - first import
NEW: DFC - use ObjectLoader to instantiate catalog component plug-ins
NEW: DFC - createTables according to the in-class schema definitions
NEW: FileCatalogClientCLI - added -q (quite) option to the find command
*WMS
CHANGE: JobScheduling - is now extensible. Added unit test
NEW: SiteDirector - define which extensions pilot should install in the options, do not take from globals
FIX: SandboxStoreClient - change default from False to None ( to suite pylint )
NEW: JobAgent - send Accounting record for Jobs Rescheduled
CHANGE: JobDB, JobLoggingDB and PilotAgentsDB - use python table description
FIX: Fixed JobDB.JobJDLs BLOB columns ( they had defaults, but BLOB does not accept default values )
CHANGE: JobLoggingDB schema - StatusTimeOrder DOUBLE(12,3) instead of (11,3)
CHANGE: JobLoggingDB schema - a new field has been added, SeqNum (INTEGER), which is part of the
new PK (JobID, SeqNum). SeqNum is generated by a trigger at every insert and behave as a counter
within a given JobID
NEW: new Splitters framework
*Transformation
NEW: TaskManager - if a site is specified in the job definition, it is now taken into account
when submitting the task
NEW: Speeding up the getTransformationSummary call, using an UpdateTransformationCounters agent
FIX: Multiple fixes to please pylint
FIX: TransformationDB - removed bad index on TaskID
NEW: TransformationDB - added methods to DELETE rows from TransformationFIleTasks table,
minor changes in operation order to be ready for FK.
CHANGE: field definition for TaskID in TransformationFiles table (from
VARCHAR to INT, in order to be ready for FK definition: TaskID in
TransformationTasks and TransformationFileTasks is INT),
!!!!! needs update of the MySQL schema on already installed databases
CHANGE: TransformationDB - in DataFiles table removed LFN field from the Primary Key,
tt was already UNIQUE, Primary key is FileID only now.
*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'
*Resources
FIX: ComputingElement - properly check if the pilot proxy has VOMS before adding it to the
payload when updating it
NEW: StorageElement - enable Storage Element proxy configuration by protocol name
CHANGE: glexecComputingElement - updated to reflect new glexec evolution, added DIRAC_GLEXEC
to the environment when running under glexec
NEW: XXXStorage - added getCurrentStatus() method
NEW: SSHOSGComputingElement and condorgce script
*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs,
dirac-stager-monitor-requests, dirac-stager-show-stats
*SMS
NEW: StorageManagementDB - use python table description
[v6r12-pre19]
*Core
CHANGE: ProcessPool - do not stop working processes by default
NEW: ReturnValue - added returnSingleResult() utility
FIX: MySQL - correctly parse BooleanType
FIX: dirac-install - use python 2.7 by default
FIX: dirac-install-xxx commands - complement installation with the component setup
in runit
NEW: dirac-configure - added --SkipVOMSDownload switch, added --Output switch
to define output configuration file
CHANGE: ProcessPool - exit from the working process if a task execution timed out
NEW: ProcessMonitor - added evaluation of the memory consumed by a process and its children
NEW: InstallTools - added flag to require MySQL installation
FIX: InstallTools - correctly installing DBs extended (with sql to be sourced)
CHANGE: Os.py - the use of "which" is replaced by distutils.spawn.find_executable
NEW: Grid.py - ldapSA replaced by ldapSE, added getBdiiSE(CE)Info() methods
CHANGE: CFG.py - only lines starting with ^\s*# will be treated as comments
*Accounting
NEW: Allow to have more than one DB for accounting
CHANGE: Accounting - use TypeLoader to load plotters
*Framework
FIX: Logger - fix FileBackend implementation
*WMS
NEW: Refactored pilots ( dirac-pilot-2 ) to become modular following RFC #18,
added pilotCommands.py, SiteDirector modified accordingly
CHANGE: InputData(Executor) - use VO specific catalogs
NEW: JobWrapper, Watchdog - monitor memory consumption by the job ( in a Warning mode )
FIX: SandboxStoreHandler - treat the case of exception while cleaning sandboxes
CHANGE: JobCleaningAgent - the delays of job removals become CS parameters
BUGFIX: JobDB - %j placeholder not replaced after rescheduling
*Resources
NEW: StorageElement - added a cache of already created SE objects
CHANGE: SSHTorqueComputingElement - mv getCEStatus to remote script
*ResourceStatus
NEW: ResourceManagementClient/DB, DowntimeCommand - distinguish Disk and Tape storage
FIX: GODDBClient - downTimeXMLParsing() can now handle the "service type" parameter properly
CHANGE: dirac-rss-xxx commands use the printTable standard utility
FIX: dirac-dms-ftsdb-summary - bug fix for #2096
*DMS
NEW: DataManager - add masterCatalogOnly flag in the constructor
FIX: DataManager - fix to protect against non valid SE
CHANGE: FC.DirectoryLevelTree - use SELECT ... FOR UPDATE lock in makeDir()
FIX: FileCatalog - fixes in using file and replica status
CHANGE: DataManager - added a new argument to the constructor - vo
CHANGE: DataManager - removed removeCatalogFile() and dirac-dms-remove-catalog-file adjusted
CHANGE: Several components - field/parameter CheckSumType all changed to ChecksumType
*SMS
FIX: StorageManagementDB - small bugfix to avoid SQL errors
*RMS
NEW: Added 'since' and 'until' parameters for getting requests
NEW: Request - added optimize() method to merge similar operations when
first inserting the request
NEW: ReqClient, RequestDB - added getBulkRequest() interface. RequestExecutingAgent
can use it controlled by a special flag
FIX: Operation, Request - set LastUpdate time stamp when reaching final state
FIX: OperationHandlerBase - don't erase the original message when reaching the max attempts
FIX: removed some deprecated codes
FIX: RequestTask - always set useServerCerificate flag to tru in case of executing inside
an agent
*TMS
NEW: WorkflowTaskAgent is now multi-threaded
NEW: Better use of threads in Transformation Agents
CHANGE: TransformationDB - modified such that the body in a transformation can be updated
FIX: TransformationCleaningAgent - removed non-ASCII characters in a comment
[v6r11p14]
*WMS
CHANGE: JobDB - removed unused tables and methods
CHANGE: removed obsoleted tests
*DMS
FIX: FTSAgent - recover case when a target is not in FTSDB
CHANGE: FTSAgent(Job) - give possibility to specify a pin life time in CS
*RMS
FIX: Make RMS objects comply with Python Data Model by adding __nonzero__ methods
[v6r11p13]
*DMS
BUGFIX: SEManager - in SEManagerDB.__addSE() bad _getConnection call, closes #2062
[v6r11p12]
*Resources
CHANGE: ARCComputingElement - accomodate changes in the ARC job reported states
*Configuration
CHANGE: Resources - define a default FTS server in the CS (only for v6r11 and v6r12)
*DMS
FIX: FTSStrategy - allow to use a given channel more than once in a tree
FIX: FTSAgent - remove request from cache if not found
FIX: FTSAgent - recover deadlock situations when FTS Files had not been correctly
updated or were not in the DB
*RMS
FIX: RequestExecutingAgent - fix a race condition (cache was cleared after the request was put)
FIX: RequestValidator - check that the Operation handlers are defined when inserting a request
[v6r11p11]
*Core
FIX: TransportPool - fixed exception due to uninitialized variable
FIX: HTTPDISETSocket - readline() takes optional argument size ( = 0 )
*DMS
FIX: FTSAgent - check the type of the Operation object ( can be None ) and
some other protections
FIX: FTSClient - avoid duplicates in the file list
*RMS
FIX: ReqClient - modified log message
CHANGE: dirac-dms-fts-monitor - allow multiple comma separated LFNs in the arguments
[v6r11p10]
*RSS
FIX: DowntimeCommand, Test_RSS_Command_GOCDBStatusCommand - correctly interpreting list of downtimes
*RMS
FIX: ReplicateAndRegister - Create a RegisterReplica (not RegisterFile) if ReplicateAndRegister
fails to register
FIX: OperationHandlerBase - handle correctly Attempt counters when SEs are banned
FIX: ReplicateAndRegister - use FC checksum in case of mismatch request/PFN
FIX: FTSAgent - in case a file is Submitted but the FTSJob is unknown, resubmit
FIX: FTSAgent - log exceptions and put request to DB in case of exception
FIX: FTSAgent - handle FTS error "Unknown transfer state NOT_USED", due to same file
registered twice (to be fixed in RMS, not clear origin)
*WMS
FIX: JobStateUpdateHandler - status not updated while jobLogging is, due to time skew between
WN and DB service
FIX: JobStateUpdateHandler - stager callback not getting the correct status Staging
(retry for 10 seconds)
[v6r11p9]
*Core
NEW: AgentModule - set AGENT_WORKDIRECTORY env variable with the workDirectory
NEW: InstallTools - added methods for the new web portal installation
*DMS
FIX: ReplicateAndRegister - apply same error logic for DM replication as for FTS
*Resources:
FIX: SRM2Storage - fix log message level
FIX: SRM2Storage - avoid useless existence checks
*RMS
FIX: ForwardDISET - a temporary fix for a special LHCb case, to be removed asap
FIX: ReqClient - prettyPrint is even prettier
FIX: RequestTask - always use server certificates when executed within an agent
[v6r11p8]
*TMS
FIX: TransformationDB - fix default value within ON DUPLICATE KEY UPDATE mysql statement
[v6r11p7]
*Framework
BUGFIX: ProxyDB.py - bug in a MySQL table definition
*DMS
FIX: ReplicateAndRegister.py - FTS client is not instantiated in the c'tor as it
might not be used,
*WMS
FIX: JobWrapper - don't delete the sandbox tar file if upload fails
FIX: JobWrapper - fix in setting the failover request
*RMS
FIX: RequestDB - add protections when trying to get a non existing request
[v6r11p6]
*WMS
FIX: InpudDataResolution - fix the case when some files only have a local replica
FIX: DownloadInputData, InputDataByProtocol - fix the return structure of the
execute() method
*Resources
NEW: LocalComputingElement, CondorComputingElement
[v6r11p5]
FIX: Incorporated changes from v6r10p25 patch
*Framework
NEW: Added getUserProfileNames() interface
*WMS
NEW: WMSAdministrator - added getPilotStatistics() interface
BUGFIX: JobWrapperTemplate - use sendJobAccounting() instead of sendWMSAccounting()
FIX: JobCleaningAgent - skip if no jobs to remove
*DMS
BUGFIX: FileCatalogClientCLI - bug fix in the metaquery construction
*Resources
CHANGE: StorageElement - enable Storage Element proxy configuration by protocol name
*TMS
NEW: TransformationManager - add Scheduled to task state for monitoring
[v6r11p4]
*Framework
NEW: ProxyDB - added primary key to ProxyDB_Log table
CHANGE: ProxyManagerHandler - purge logs once in 6 hours
*DMS
FIX: DataManager - fix in the accounting report for deletion operation
CHANGE: FTSRequest - print FTS GUID when submitting request
FIX: dirac-dms-fts-monitor - fix for using the new FTS structure
FIX: DataLoggingDB - fix type of the StatusTimeOrder field
FIX: DataLoggingDB - take into account empty date argument in addFileRecord()
FIX: ReplicateAndRegister - use active replicas
FIX: FTS related modules - multiple fixes
*WMS
NEW: SiteDirector - pass the list of already registered pilots to the CE.available() query
FIX: JobCleaningAgent - do not attempt job removal if no eligible jobs
*Resources
FIX: LcgFileCatalogClient - if replica already exists while registration, reregister
NEW: CREAM, SSH, ComputingElement - consider only registered pilots to evaluate queue occupancy
[v6r11p3]
FIX: import gMonitor from it is original location
*Core
FIX: FC.Utilities - treat properly the LFN names starting with /grid ( /gridpp case )
*Configuration
FIX: LocalConfiguration - added exitCode optional argument to showHelp(), closes #1821
*WMS
FIX: StalledJobAgent - extra checks when failing Completed jobs, closes #1944
FIX: JobState - added protection against absent job in getStatus(), closes #1853
[v6r11p2]
*Core
FIX: dirac-install - skip expectedBytes check if Content-Length not returned by server
FIX: AgentModule - demote message "Cycle had an error:" to warning
*Accounting
FIX: BaseReporter - protect against division by zero
*DMS
CHANGE: FileCatalogClientCLI - quite "-q" option in find command
FIX: DataManager - bug fix in __initializeReplication()
FIX: DataManager - less verbose log message
FIX: DataManager - report the size of removed files only for successfully removed ones
FIX: File, FTSFile, FTSJob - SQL tables schema change: Size filed INTEGER -> BIGINT
*RMS
FIX: dirac-rms-reset-request, dirac-rms-show-request - fixes
FIX: ForwardDISET - execute with trusted host certificate
*Resources
FIX: SSHComputingElement - SSHOptions are parsed at the wrong place
NEW: ComputingElement - evaluate the number of available cores if relevant
*WMS
NEW: JobMonitoringHander - added export_getOwnerGroup() interface
*TMS
CHANGE: TransformationCleaningAgent - instantiation of clients moved in the initialize()
[v6r11p1]
*RMS
FIX: ReqClient - failures due to banned sites are considered to be recoverable
*DMS
BUGFIX: dirac-dms-replicate-and-register-request - minor bug fixes
*Resources
FIX: InProcessComputingElement - stop proxy renewal thread for a finished payload
[v6r11]
*Core
FIX: Client - fix in __getattr__() to provide dir() functionality
CHANGE: dirac-configure - use Registry helper to get VOMS servers information
BUGFIX: ObjectLoader - extensions must be looked up first for plug-ins
CHANGE: Misc.py - removed obsoleted
NEW: added returnSingleResult() generic utility by moving it from Resources/Utils module
*Configuration
CHANGE: Resources.getDIRACPlatform() returns a list of compatible DIRAC platforms
NEW: Resources.getDIRACPlatforms() used to access platforms from /Resources/Computing/OSCompatibility
section
NEW: Registry - added getVOs() and getVOMSServerInfo()
NEW: CE2CSAgent - added VO management
*Accounting
FIX: AccountingDB, Job - extra checks for invalid values
*WMS
NEW: WMS tags to allow jobs require special site/CE/queue properties
CHANGES: DownloadInputData, InputDataByProtocol, InputDataResolution - allows to get multiple
PFNs for the protocol resolution
NEW: JobDB, JobMonitoringHandler - added traceJobParameters(s)() methods
CHANGE: TaskQueueDirector - use ObjectLoader to load directors
CHANGE: dirac-pilot - use Python 2.7 by default, 2014-04-09 LCG bundles
*DMS
NEW: DataManager to replace ReplicaManager class ( simplification, streamlining )
FIX: InputDataByProtocol - fix the case where file is only on tape
FIX: FTSAgent - multiple fixes
BUGFIX: ReplicateAndRegister - do not ask SE with explicit SRM2 protocol
*Interfaces
CHANGE: Dirac - instantiate SandboxStoreClient and WMSClient when needed, not in the constructor
CHANGE: Job - removed setSystemConfig() method
NEW: Job.py - added setTag() interface
*Resources
CHANGE: StorageElement - changes to avoid usage PFNs
FIX: XROOTStorage, SRM2Storage - changes in PFN construction
NEW: PoolComputingElement - a CE allowing to manage multi-core slots
FIX: SSHTorqueComputingElement - specify the SSHUser user for querying running/waiting jobs
*RSS
NEW: added commands dirac-rss-query-db and dirac-rss-query-dtcache
*RMS
CHANGE: ReqDB - added Foreign Keys to ReqDB tables
NEW: dirac-rms-reset-request command
FIX: RequestTask - always execute operations with owner proxy
*SMS
FIX: few minor fixes to avoid pylint warnings
[v6r10p25]
*DMS
CHANGE: FileCatalog - optimized file selection by metadata
[v6r10p24]
*DMS
FIX: FC.FileMetadata - optimized queries for list interception evaluation
[v6r10p23]
*Resoures
CHANGE: SSHComputingElement - allow SSH options to be passed from CS setup of SSH Computing Element
FIX: SSHComputingElement - use SharedArea path as $HOME by default
[v6r10p22]
*CS
CHANGE: Operations helper - if not given, determine the VO from the current proxy
*Resources
FIX: glexecComputingElement - allows Application Failed with Errors results to show through,
rather than be masked by false "glexec CE submission" errors
*DMS
CHANGE: ReplicaManager - in getReplicas() rebuild PFN if
<Operations>/DataManagement/UseCatalogPFN option is set to False ( True by default )
[v6r10p21]
*Configuration
FIX: CSGlobals - allow to specify extensions in xxxDIRAC form in the CS
*Interfaces
FIX: Job - removed self.reqParams
FIX: Job - setSubmitPools renamed to setSubmitPool, fixed parameter definition string
*WMS
FIX: JobMonitorigHandler, JobPolicy - allow JobMonitor property to access job information
[v6r10p20]
*DMS
FIX: FTSAgent/Client, ReplicateAndRegister - fixes to properly process failed
FTS request scheduling
[v6r10p19]
*DMS
FIX: FTSAgent - putRequest when leaving processRequest
FIX: ReplicaManager - bug in getReplicas() in dictionary creation
[v6r10p18]
*DMS
FIX: ReplicateAndRegister - dictionary items incorrectly called in ftsTransfer()
[v6r10p17]
*RMS
FIX: RequestDB.py - typo in a table name
NEW: ReqManagerHandler - added getDistinctValues() to allow selectors in the web page
*DMS
CHANGE: ReplicaManager - bulk PFN lookup in getReplicas()
[v6r10p16]
*Framework
NEW: PlottingClient - added curveGraph() function
*Transformation
FIX: TaskManagerAgentBase - add the missing Scheduled state
*WMS
FIX: TaskQueueDB - reduced number of lines in the matching parameters printout
*DMS
FIX: dirac-dms-show-se-status - exit on error in the service call, closes #1840
*Interface
FIX: API.Job - removed special interpretation of obsoleted JDLreqt type parameters
*Resources
FIX: SSHComputingElement - increased timeout in getJobStatusOnHost() ssh call, closes #1830
[v6r10p15]
*DMS
FIX: FTSAgent - added missing monitoring activity
FIX: FileCatalog - do not check directory permissions when creating / directory
*Resources
FIX: SSHTorqueComputingElement - removed obsoleted stuff
[v6r10p14]
*SMS
FIX: RequestPreparationAgent - typo fixed
[v6r10p13]
*SMS
FIX: RequestPreparationAgent - use ReplicaManager to get active replicas
*DMS
FIX: ReplicaManager - getReplicas returns all replicas ( in all statuses ) by default
CHANGE: FC/SecurityManager - give full ACL access to the catalog to groups with admin rights
*WMS
CHANGE: SiteDirector - changes to reduce the load on computing elements
FIX: JobWrapper - do not set Completed status for the case with failed application thread
[v6r10p12]
*WMS
CHANGE: Replace consistently everywhere SAM JobType by Test JobType
FIX: JobWrapper - the outputSandbox should be always uploaded (outsized, in failed job)
*DMS
FIX: RemoveFile - bugfix
FIX: ReplicateAndRegister - fixes in the checksum check, retry failed FTS transfer
with RM transfer
NEW: RegisterReplica request operation
*RMS
FIX: ReqClient - fix in the request state machine
FIX: Request - enhance digest string
NEW: dirac-dms-reset-request command
CHANGE: dirac-rms-show-request - allow selection of a request by job ID
*TS
FIX: TransformationDB - in getTransformationParameters() dropped "Submitted" counter
in the output
[v6r10p11]
*Core
FIX: X509Chain - cast life time to int before creating cert
*Accounting
FIX: DataStoreClient - self.__maxRecordsInABundle = 5000 instead of 1000
FIX: JobPolicy - allow access for JOB_MONITOR property
*RMS
FIX: ReqClient - fix the case when a job is Completed but in an unknown minor status
*Resources
BUGFIX: ProxyStorage - use checkArgumentFormat() instead of self.__checkArgumentFormatDict()
[v6r10p10]
*DMS
FIX: Several fixes to make FTS accounting working (FTSAgent/Job, ReplicaManager, File )
[v6r10p9]
*Core
BUGFIX: LineGraph - Ymin was set to a minimal plot value rather than 0.
*DMS
CHANGE: FTSJob(Agent) - get correct information for FTS accounting (registration)
[v6r10p8]
*Core
FIX: InstallTools - admin e-mail default location changed
*Framework
FIX: SystemAdministratorClientCLI - allow "set host localhost"
FIX: BundleDelivery - protect against empty bundle
*WMS
FIX: SiteDirector - Pass siteNames and ceList as None if any is accepted
FIX: WorkloadManagement.ConfigTemplate.SiteDorectory - set Site to Any by default
*DMS
FIX: FileCatalogCLI - ignore Datasets in ls command for backward compatibility
*Resources
FIX: SSH - some platforms use Password instead of password prompt
[v6r10p7]
*Core
FIX: dirac-install - execute dirac-fix-mysql-script and dirac-external-requirements after sourcing the environment
FIX: InstallTools - set basedir variable in fixMySQLScript()
FIX: InstallTools - define user [email protected] in installMySQL()
*Framework
BUGFIX: SystemAdministratorCLI - bug fixed in default() call signature
*DMS
FIX: FTSRequest - handle properly FTS server in the old system
FIX: ReplicaManager - check if file is in FC before removing
FIX: Request/RemovalTask - handle properly proxies for removing files
BUGFIX: DatasetManager - in the table description
[v6r10p6]
*Core
FIX: X509Certificate - reenabled fix in getDIRACGroup()
*Configuration
FIX: CSAPI - Group should be taken from the X509 chain and not the certificate
*RMS
CHANGE: ReqClient - if the job does not exist, do not try further finalization
[v6r10p5]
*Core
FIX: X509Certificate - reverted fix in getDIRACGroup()
[v6r10p4]
*Core
NEW: dirac-info - extra printout
CHANGE: PrettyPrint - extra options in printTable()
FIX: X509Certificate - bug fixed in getDIRACGroup()
*Framework
NEW: SystemAdministratorCLI - new showall command to show components across hosts
NEW: ProxyDB - allow to upload proxies without DIRAC group
*RMS
CHANGE: ReqClient - requests from failed jobs update job status to Failed
CHANGE: RequestTask - retry in the request finalize()
[v6r10p3]
*Configuration
CHANGE: Registry - allow to define a default group per user
*WMS
BUGFIX: JobReport - typo in generateForwardDISET()
[v6r10p2]
*TMS
CHANGE: Backward compatibility fixes when setting the Transformation files status
*DMS
BUGFIX: ReplicateAndRegister - bugfix when replicating to multiple destination by ReplicaManager
*WMS
BUGFIX: JobManager - bug fix when deleting no-existing jobs
[v6r10p1]
*RMS
FIX: ReqDB.Operations - Arguments field changed type from BLOB to MEDIUMBLOB
*DMS
FIX: FileCatalog - check for non-exiting directories in removeDirectory()
*TMS
FIX: TransformationDB - removed constraint that was making impossible to derive a production
[v6r10]
*Core
FIX: Several fixes on DB classes(AccountingDB, SystemLoggingDB, UserProfileDB, TransformationDB,
JobDB, PilotAgentsDB) after the new movement to the new MySQL implementation with a persistent
connection per running thread
NEW: SystemAdministratorCLI - better support for executing remote commands
FIX: DIRAC.__init__.py - avoid re-definition of platform variable
NEW: Graphs - added CurveGraph class to draw non-stacked lines with markers
NEW: Graphs - allow graphs with negative Y values
NEW: Graphs - allow to provide errors with the data and display them in the CurveGraph
FIX: InstallTools - fix for creation of the root@'host' user in MySQL
FIX: dirac-install - create links to permanent directories before module installation
CHANGE: InstallTools - use printTable() utility for table printing
CHANGE: move printTable() utility to Core.Utilities.PrettyPrint
NEW: added installation configuration examples
FIX: dirac-install - fixBuildPath() operates only on files in the directory
FIX: VOMSService - added X-VOMS-CSRF-GUARD to the html header to be compliant with EMI-3 servers
*CS
CHANGE: getVOMSVOForGroup() uses the VOMSName option of the VO definition
NEW: CE2CSAgent - added ARC CE information lookup
*Framework
FIX: SystemAdministratorIntegrator - use Host option to get the host address in addition to the section name, closes #1628
FIX: dirac-proxy-init - uses getVOMSVOForGroup() when adding VOMS extensions
*DMS
CHANGE: DFC - optimization and bug fixes of the bulk file addition
FIX: TransferAgent - protection against badly defined LFNs in collectFiles()
NEW: DFC - added getDirectoryReplicas() service method support similar to the LFC
CHANGE: DFC - added new option VisibleReplicaStatus which is used in replica getting commands
CHANGE: FileCatalogClientCLI client shows number of replicas in the 2nd column rather than
unimplemented number of links
CHANGE: DFC - optimizations for the bulk replica look-up
CHANGE: DFC updated scalability testing tool FC_Scaling_test.py
NEW: DFC - methods returning replicas provide also SE definitions instead of PFNs to construct PFNs on the client side
NEW: DFC - added getReplicasByMetadata() interface
CHANGE: DFC - optimized getDirectoryReplicas()
CHANGE: FileCatalogClient - treat the reduced output from various service queries restoring LFNs and PFNs on the fly
NEW: DFC - LFNPFNConvention flag can be None, Weak or Strong to facilitate compatibility with LFC data
CHANGE: FileCatalog - do not return PFNs, construct them on the client side
CHANGE: FileCatalog - simplified FC_Scaling_test.py script
NEW: FileCatalog/DatasetManager class to define and manipulate datasets corresponding to meta queries
NEW: FileCatalogHandler - new interface methods to expose DatasetManager functionality
NEW: FileCatalogClientCLI - new dataset family of commands
FIX: StorageFactory, ReplicaManager - resolve SE alias name recursively
FIX: FTSRequest, ReplicaManager, SRM2Storage - use current proxy owner as user name in accounting reports, closes #1602
BUGFIX: FileCatalogClientCLI - bug fix in do_ls, missing argument to addFile() call, closes #1658
NEW: FileCatalog - added new setMetadataBulk() interface, closes #1358
FIX: FileCatalog - initial argument check strips off leading lfn:, LFN:, /grid, closes #448
NEW: FileCatalog - added new setFileStatus() interface, closes #170, valid and visible file and replica statuses can be defined in respective options.
CHANGE: multiple new FTS system fixes
CHANGE: uniform argument checking with checkArgumentFormat() in multiple modules
CHANGE: FileCatalog - add Trash to the default replica valid statuses
CHANGE: ReplicaManager,FTSRequest,StorageElement - no use of PFN as returned by the FC except for file removal,
rather constructing it always on the fly
*SMS
CHANGE: PinRequestAgent, SENamespaceCatalogCheckAgent - removed
CHANGE: Use StorageManagerClient instead of StorageDB directly
*WMS
CHANGE: JobPolicy - optimization for bulk job verification
NEW: JobPolicy - added getControlledUsers() to get users which jobs can be accessed for
a given operation
CHANGE: JobMonitoringHandler - Avoid doing a selection of all Jobs, first count matching jobs
and then use "limit" to select only the required JobIDs.
NEW: JobMonitoringHandler - use JobPolicy to filter jobs in getJobSummaryWeb()
NEW: new Operations option /Services/JobMonitoring/GlobalJobsInfo ( True by default ) to
allow or not job info lookup by anybody, used in JobMonitoringHandler
BUGFIX: SiteDirector - take into account the target queue Platform
BUGFIX: JobDB - bug in __insertNewJDL()
CHANGE: dirac-admin-show-task-queues - enhanced output
CHANGE: JobLoggingDB.sql - use trigger to manage the new LoggingInfo structure
CHANGE: JobWrapper - trying several times to upload a request before declaring the job failed
FIX: JobScheduling executor - fix race condition that causes a job to remain in Staging
NEW: SiteDirector - do not touch sites for which there is no work available
NEW: SiteDirector - allow sites not in mask to take jobs with JobType Test
NEW: SiteDirector - allow 1 hour grace period for pilots in Unknown state before aborting them
CHANGE: Allow usage of non-plural form of the job requirement options ( PilotType, GridCE, BannedSite,
SubmitPool ), keep backward compatibility with a plural form
*RSS
FIX: DowntimeCommand - take the latest Downtime that fits
NEW: porting new Policies from integration
NEW: RSS SpaceToken command querying endpoints/tokens that exist
*Resources
NEW: added SSHOARComputingElement class
NEW: added XROOTStorage class
FIX: CREAMComputingElement - extra checks for validity of returned pilot references
*TS
CHANGE: TransformationClient(DB,Manager) - set file status for transformation as bulk operation
CHANGE: TransformationClient - applying state machine when changing transformation status
BUGFIX: TransformationClient(Handler) - few minor fixes
NEW: TransformationDB - backported __deleteTransformationFileTask(s) methods
CHANGE: TransformationDB(Client) - fixes to reestablish the FileCatalog interface
FIX: TransformationAgent - added MissingInFC to consider for Removal transformations
BUGFIX: TransformationAgent - in _getTransformationFiles() variable 'now' was not defined
FIX: TransformationDB.sql - DataFiles primary key is changed to (FileID) from (FileID,LFN)
CHANGE: TransformationDB(.sql) - schema changes suitable for InnoDB
FIX: TaskManager(AgentBase) - consider only submitted tasks for updating status
CHANGE: TransformationDB(.sql) - added index on LFN in DataFiles table
*RMS
NEW: Migrate to use the new Request Management by all the clients
CHANGE: RequestContainer - Retry failed transfers 10 times and avoid sub-requests to be set Done
when the files are failed
CHANGE: Use a unique name for storing the proxy as processes may use the same "random" name and
give conflicts
NEW: RequestClient(Handler) - add new method readRequest( requestname)
*Workflow
NEW: Porting the LHCb Workflow package to DIRAC to make the use of general purpose modules and
simplify construction of workflows
[v6r9p33]
*Accounting
BUGFIX: AccountingDB - wrong indentation
[v6r9p32]
*Accounting
FIX: AccountingDB - use old style grouping if the default grouping is altered, e.g. by Country
[v6r9p31]
*Accounting
CHANGE: AccountingDB - changes to speed up queries: use "values" in GROUP By clause;
drop duplicate indexes; reorder fields in the UniqueConstraint index of the
"bucket" tables
[v6r9p30]
*DMS
CHANGE: FileCatalogFactory - construct CatalogURL from CatalogType by default
*SMS
FIX: dirac-stager-stage-files - changed the order of the arguments
[v6r9p29]
*TS
FIX: TaskManager(AgentBase) - fix for considering only submitted tasks
[v6r9p28]
*TS
FIX: TransformationDB(ManagerHandler) - several portings from v6r10
[v6r9p27]
*SMS
FIX: StorageManagementDB - in removeUnlinkedReplicas() second look for CacheReplicas
for which there is no entry in StageRequests
[v6r9p26]
*Resources
CHANGE: CREAMComputigElement - Make sure that pilots submitted to CREAM get a
fresh proxy during their complete lifetime
*Framework
FIX: ProxyDB - process properly any SQLi with DNs/groups with 's in the name
[v6r9p25]
*TS
CHANGE: TransformationClient - changed default timeout values for service calls
FIX: TransformationClient - fixes for processing of derived transformations
[v6r9p24]
*TS
FIX: TransformationClient - in moveFilesToDerivedTransformation() set file status
to Moved-<prod>
[v6r9p23]
*Core
BUGFIX: InstallTools - improper configuration prevents a fresh new installation
*WMS
BUGFIX: PilotDirector - Operations Helper non-instantiated
[v6r9p22]
*WMS
FIX: PilotDirector - allow to properly define extensions to be installed by the
Pilot differently to those installed at the server
FIX: Watchdog - convert pid to string in ProcessMonitor
*TS
FIX: TransformationDB - splitting files in chunks
*DMS
NEW: dirac-dms-create-removal-request command
CHANGE: update dirac-dms-xxx commands to use the new RMS client,
strip lines when reading LFNs from a file
[v6r9p21]
*TS
FIX: Transformation(Client,DB,Manager) - restored FileCatalog compliant interface
FIX: TransformationDB - fix in __insertIntoExistingTransformationFiles()
[v6r9p20]
*Core
BUGFIX: ProxyUpload - an on the fly upload does not require a proxy to exist
*DMS
CHANGE: TransferAgent - use compareAdler() for checking checksum
FIX: FailoverTransfer - recording the sourceSE in case of failover transfer request
*WMS
FIX: ProcessMonitor - some fixes added, printout when <1 s of consumed CPU is found
*Transformation
BUGFIX: TransformationClient - fixed return value in moveFilesToDerivedTransformation()
*RMS
BUGFIX: CleanReqDBAgent - now() -> utcnow() in initialize()
*Resources
FIX: ARCComputingElement - fix the parsing of CE status if no jobs are available
[v6r9p19]
*DMS
FIX: FileCatalog/DirectoryMetadata - inherited metadata is used while selecting directories
in findDirIDsByMetadata()
[v6r9p18]
*DMS
FIX: FTSSubmitAgent, FTSRequest - fixes the staging mechanism in the FTS transfer submission
NEW: TransferDBMonitoringHandler - added getFilesForChannel(), resetFileChannelStatus()
[v6r9p17]
*Accounting
FIX: DataStoreClient - send accounting records in batches of 1000 records instead of 100
*DMS:
FIX: FailoverTransfer - catalog name from list to string
FIX: FTSSubmitAgent, FTSRequest - handle FTS3 as new protocol and fix bad submission time
FIX: FTSSubmitAgent, FTSRequest - do not submit FTS transfers for staging files
*WMS
FIX: TaskQueueDB - do not check enabled when TQs are requested from Directors
FIX: TaskQueueDB - check for Enabled in the TaskQueues when inserting jobs to print an alert
NEW: TaskQueueDB - each TQ can have at most 5k jobs, if beyond the limit create a new TQ
to prevent long matching times when there are way too many jobs in a single TQ
[v6r9p16]
*TS
BUGFIX: typos in TransformationCleaningAgent.py
*DMS
CHANGE: DownloadInputData - check the available disk space in the right input data directory
FIX: DownloadInputData - try to download only Cached replicas
[v6r9p15]
*Core
FIX: MySQL - do not decrease the retry counter after ping failure
*DMS
CHANGE: FC/DirectoryMetadata - Speed up findFilesByMetadataWeb when many files match
FIX: RemovalTask - fix error string when removing a non existing file (was incompatible
with the LHCb BK client).
*WMS
FIX: JobReport - minor fix ( removed unused imports )
FIX: JobMonitoring(JobStateUpdate)Handler - jobID argument can be either string, int or long