forked from GENI-NSF/geni-tools
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME-omni.txt
2779 lines (2364 loc) · 132 KB
/
README-omni.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
{{{
#!comment
N.B. This page is formatted for a Trac Wiki.
}}}
[[PageOutline]]
= The Omni GENI Client =
Omni is a GENI experimenter tool that communicates with GENI Aggregate
Managers (AMs) via the GENI AM API (the common API that GENI
aggregates support). The Omni client also communicates with
clearinghouses and slice authorities (sometimes referred to as control
frameworks) in order to create slices, delete slices, and
enumerate available GENI Aggregate Managers (AMs).
A Control Framework (CF) is a framework of resources that provides
users with GENI accounts (credentials) that they can use to
reserve resources in GENI AMs.
See INSTALL.txt or
[https://github.com/GENI-NSF/geni-tools/wiki/QuickStart the Installation instructions]
for details on installing Omni.
See README-omniconfigure.txt or
https://github.com/GENI-NSF/geni-tools/wiki/Omni-Configuration-Automatically for details about how to configure Omni.
For 'stitching' (experimenter defined custom topologies) or
multi-aggregate topologies, see README-stitching.txt.
The currently supported CFs (clearinghouses) are the GENI Portal,
ProtoGENI, SFA (!PlanetLab), and GCF. Omni works with any GENI AM API compliant AM.
These include InstaGENI and ExoGENI racks, ProtoGENI, !OpenFlow, SFA, AL2S, FOAM and GCF.
Omni performs the following functions:
* Talks to each CF in its native API
* Contacts AMs via the GENI AM API
For the latest Omni documentation, examples, and trouble shooting
tips, see the Omni Wiki: https://github.com/GENI-NSF/geni-tools/wiki/Omni
== Release Notes ==
New in v2.10:
* Continue anyway if no aggregate nickname cache can be loaded. (#822)
* Sliver info reporting and operations on AMs by nickname will likely fail.
* For connections to servers, use TLSv1 if possible (falling back to SSLv3
in the usual OpenSSL way), plus a restricted set of ciphers
(`HIGH:MEDIUM:!ADH:!SSLv2:!MD5:!RC4:@STRENGTH`) when using python2.7.
This avoids some security issues, and allows Omni on older clients
to connect to some updated servers. (#745)
* Use `False` instead of `'f'` for `SLICE_EXPIRED` and `PROJECT_EXPIRED` when
using Common Federation API clearinghouses. (#856)
* Thanks to Umar Toseef for the bug report.
* Do not assume `PROJECT_MEMBER_UID` is returned when listing project members,
but allow it. (#857)
* Thanks to Umar Toseef for the bug report.
* Calling `getslicecred` while specifying a `slicecredfile` that exists
no longer means just return that file. Instead, that file will be
ignored and, if you specify `-o`, replaced. (#868, #869)
* Moved canonical `agg_nick_cache` location to Github. (#814, #882)
* Use `urllib2.urlopen` instead of `urllib.urlretrieve` to avoid bad
interaction with M2Crypto. (#881)
New in v2.9:
* If `sliverstatus` fails in a way that indicates there are no local resources,
and the caller specified `--raise-error-on-v2-amapi-error`, still delete any
sliver info records at the clearinghouse (as necessary). (#778)
* If `deletesliver` or `delete` fail indicating there are no local resources,
delete any sliver info records at teh clearinghouse (as necessary). (#782)
* Add 11 aggregates to the nickname cache. (#783)
* Trim fractional seconds from user input datetimes before passing to servers. (#795)
* Avoids errors in ProtoGENI based code.
* Allow getting Ad RSpecs (calling `ListResources` not in slice context)
without a user credential. (#793)
* Implements AM API Change Proposal AD.
* If the return from `POA` is a `geni_credential`, or per sliver has a
`geni_credential`, then save that cred in a separate file. (#803)
* Also when saving output, JSON that has XML content shouldn't get
the header inserted within the XML content.
New in v2.8:
* Allow configuring how many times Omni retries on a busy error from
an AM or CH. Use `--maxBusyRetries`. Default remains 4. (#749)
* Support `Update()` and `Cancel()` from AM APIv4 in any v3+ implementation. Support
is only known at ProtoGENI, and is limited. (#589)
* New option `--noCacheFiles` completely disables reading or writing the !GetVersion and
Aggregate nickname cache files. (#772)
* New config that sets the current release number and a release message,
so Omni can alert you if a new release is available. (#698)
* Better control of Omni logging configuration. (#458)
* Allow a Python logging configuration dictionary, and configure
logging from that if possible.
* New option `--noLoggingConfiguration` completely disables
configuring Python loggers from Omni. A script might use this to
allow it to configure logging later in its own way.
* Fix error message on expired user cert. (#756)
* Remove ticket #722 workaround (bug fixed at ION AM). (#724)
* Mac installer: remove old aliases before adding new ones. (#556)
* Clean up `listresources` summary string and include sliver expiration if known. (#704)
* Add support for `--start-time` option to specify a `geni_start_time` option
for any aggregates that support such a value. (#660)
* Update copyrights to 2015 (#764)
* Add nicknames for !CloudLab and Apt. (#767)
* Avoid exception on empty aggregate in `-a` argument. (#771)
* Support python 2.7.9+ where we must request not verifying server certificates
for the SSL connection. Thanks to Ezra Kissel. (#776)
New in v2.7:
* Calls to `status` and `sliverstatus` will also call the CH
to try to sync up the CH records of slivers with truth
as reported by the AM. (#634)
* Make `useSliceMembers` True by default and deprecate the option. (#667)
* By default, Omni will create accounts and install SSH keys for members of your slice,
if you are using a CHAPI style framework / Clearinghouse
which allows defining slice members (as the GENI Clearinghouse does).
* This is not a change for Omni users who configured Omni using the `omni-configure` script,
which already forced that setting to true.
* The old `--useSliceMembers` option is deprecated and will be
removed in a future release.
* Added new option `--noSliceMembers` to over-ride that default and tell Omni
to ignore any slice members defined at the Clearinghouse.
* You may also set `useslicemembers=False` in the `omni` section of
your `omni_config` to over-ride the `useSliceMembers` default of True.
* Honor the `useslicemembers` and `ignoreconfigusers` options in the `omni_config` (#671)
* Fix `get_member_email` e.g. from `listprojectmembers` for speaks-for. (#676)
* Fix nickname cache updating when temp and home directories are on
different disks - use `shutil.move`. (#646)
* Look for fallback `agg_nick_cache` in correct location (#662)
* Use relative imports in `speaksfor_util` if possible. (#657)
* Fix URL to URN lookups to better handle names that differ by a prefix. (#683)
* Increase the sleep between busy SSL call retries from 15 to 20 seconds. (#697)
* Rename `addAliases.sh` to `addAliases.command` for Mac install. (#647)
* Add new section `omni_defaults` to the Omni config file. (#713)
* This should set system defaults. Where these overlap in meaning with other
omni_config or commandline options, let those take precedence.
* Allow these `omni_defaults` to be specified in the `agg_nick_cache`. These
are read before those in the per-user omni config. If a default is set
in the `agg_nick_cache`, that takes precedence over any value in
the user's `omni_config`: if a user should be able to over-ride the
default, use a different omni config setting (not in `omni_defaults`), or
use a command line option.
* Stitcher uses this for the SCS URL.
* Allow FOAM/AL2S AMs to submit sliver URNs that are the slice URN with an ID
appended.
* This works around known bug http://groups.geni.net/geni/ticket/1294. (#719)
* Work around malformed ION sliver URNs: (#722)
* Allow submitting URNs that use the slice authority as the sliver authority.
* If the sliver urn reported by sliverstatus is malformed, replace it
with the proper URN as the manifest returns, so sliver_info reporting
works without even deleting the existing entry.
* See http://groups.geni.net/geni/ticket/1292
* Quiet down some debug logs when printing SSH keys and when talking to CHAPI. (#727)
New in v2.6:
* New function `removeslicemember <slice> <username>`:
Remove the user with the given username from the named slice. (#515)
* Add functions `listprojects` to list your projects, and `listprojectmembers`
to list the members of a project and their role in the project and
email address. (#495)
* Include `addMemberToSliceAndSlivers` in Windows and Mac binaries (#585)
* Include `remote-execute` in Mac binaries (#601)
* Record FOAM reservations at the clearinghouse when using the
`chapi` framework, by using fake sliver URNs. (#574)
* `listslicemembers` honors the `-o` option to save results to a
file, and `--tostdout` to instead go to STDOUT. (#489)
* `listslivers` honors the `-o` option to save results to a file,
and `--tostdout` to instead go to STDOUT. (#488)
* `get_ch_version`, `listaggregates`, `listslices`, `listmyslices`,
`listkeys`, `listmykeys`, `listimages`, and `nicknames`
honor the `-o` option to save results to a file,
and `--tostdout` to instead to to STDOUT. (#371)
* `listkeys` return is a list of structs of ('`public_key`',
'`private_key`'), where `private_key` is omitted for most
frameworks and most cases where not available. (#600)
* Added `print_sliver_expirations` to print the expirations of your
slivers at requested aggregates. Also print sliver expirations from
`sliverstatus`, `listresources` and `createsliver` calls. (#465, #564, #571)
* Added new utilities in `handler_utils` to extract sliver
expiration from the manifest and sliverstatus.
* Clean up console log messages. (#623)
* Retry on AM busy message two more times, sleeping 15 seconds instead
of 10. (#624,#635)
* Restore printing of non-standard options used in command summary. (#625)
* Help specifies defaults for more options. (#626)
* Mac install clears old `omni.py` and similar aliases (#556)
* Fix `get_cert_keyid` to get the key id from the certificate (#573)
* `renewslice` properly warns if your new expiration is not what you
requested (#575)
* rspec_util utility takes optional logger (#612)
* Add support for talking to SA/MA that speak Federation API v2.
To use the v2 APIs, add to your `omni_config`: `speakv2=true`. (#613)
* Ensure manifest from `createsliver` is printed prettily.
`getPrettyRSpec` takes a flag on whether it does pretty
printing, default True. Do not do pretty printing on most
Ads, and some manifests. Uses less memory. (#610)
* Clean up error getting slice credential for unknown slice from
`chapi` clearinghouses. (#538)
* Clarify error messages in `delegateSliceCred`. (#619)
* Harden update of `agg_nick_cache` to avoid replacing a good cache
with one that was empty or incomplete on download. (#631)
* Document creating an alias for `addMemberToSliceAndSlivers`
in `INSTALL.txt`. (#632)
* Avoid error doing `listprojects` when user has none. (#637)
* More use of `os.path.join`, `os.sep`, `os.normpath` for Windows support (#639)
* Ensure SFA libraries look for the temp dir in `TMP` as well as `TEMPDIR`, and try to create
the directory if it doesn't exist. (#560)
New in v2.5.3:
* Can now parse omni-configure sections of omni_config. (#436)
New in v2.5.2:
* Update the OpenSSL version used in the Windows package to 1.0.1g,
avoiding the heartbleed vulnerability. (#594)
* Update various packages in Windows and Mac binaries to be
consistent versions. (#595)
New in v2.5:
Highlights:
* Released Windows and Mac OS X packages of the Omni experimenter
utilities. (Developer gcf components are not included.)
* Omni adds the ability to contact clearinghouses that speak the
Uniform Federation API using framework type `chapi`
* When using the new `chapi` framework allow a `--useSliceAggregates`
option to specify that the aggregate action should be taken at all
aggregates at which you have resources in this slice. (#507)
* Added new options to allow installing SSH keys of all slice members
when using the new `chapi` framework. (#491, #278)
* Refactored the source code to make it easier to import Omni in
other tools. Look in `src/gcf` for directories that were
previously directly under `src`. (#388)
* Added utilities for creating and processing 'Speaks For'
credentials (which Omni can pass along to aggregates and to Uniform
Federation API clearinghouses).
* Timeout Omni calls to servers after 6 minutes (controlled by `--ssltimeout`)
Details:
- Add a new framework type `chapi` for talking the Uniform Federation API
(http://groups.geni.net/geni/wiki/UniformClearinghouseAPI)
to compliant clearinghouses (e.g. GENI Clearinghouse). (#345, #440)
- See `omni_config.sample` for config options required
- To upgrade a `pgch` config for `ch.geni.net` to a `chapi` config:
- Change `type = pgch` to `type = chapi`
- Change `ch = https://ch.geni.net...` to:
{{{
ch=https://ch.geni.net:8444/CH
ma=https://ch.geni.net/MA
sa=https://ch.geni.net/SA
}}}
Included in this change:
- When creating or renewing or deleting slivers, tell the Slice Authority.
This allows the SA to know (non-authoritatively) where your slice
has resources. (#439)
- New function `listslivers <slice>`: lists the slivers reported to
the slice authority in the given slice, by aggregate (with
the sliver expirations). Note that this information is not
authoritative - contact the aggregates if you want to be sure
not to miss any reservations.
- New function `listslicemembers <slice>`: lists the members of
the given slice, with their email and registered SSH public
keys (if any) and role in the slice. (#421, #431, #278)
- New function `addslicemember <slice> <username> [optional: role]`:
Adds the user with the given username to the named slice,
with the given role (or `MEMBER` by default). Note this
does not change what SSH keys are installed on any existing
slivers. (#422,#513)
- Support `geni_extend_alap` with new `--alap` option, allowing you to
request that slivers be renewed as long as possible, if your
requested time is not permitted by local AM policy. (#415)
- When using the new `chapi` framework allow a `--useSliceAggregates`
option to specify that the aggregate action should be taken at all
aggregates at which you have resources in this slice. (#507)
- Any `-a` aggregates are extra.
- At other frameworks, this is ignored.
- This option is ignored for commands like `createsliver`,
`allocate`, `provision`, and `getversion`.
- Added new option `--noExtraCHCalls` to disable calls to the
clearinghouse to report slivers, query for slivers, or query
a list of aggregates; explicit CH calls and retrieving credentials
is not effected. (#514)
- Added new options controlling what SSH keys are installed on
compute resources. Default behavior is unchanged. These options
control what users are created and SSH keys installed in new
slivers from `createsliver` or `provision`, or when you update
the installed users and keys using
`performoperationalaction <slice> geni_update_users`.
If you supply the new option `--useSliceMembers` and your
clearinghouse supports listing slice members (i.e. the new `chapi`
type), then Omni will fetch the members of your slice from the
clearinghouse and their public SSH keys, and send those to the
aggregate to install on new compute resources. (#278, #441)
By default, Omni will ''also'' read the users and SSH keys
configured in your `omni_config` as usual, ''adding'' those users
and keys to the set downloaded from the clearinghouse, if any.
You can skip reading the `omni_config` keys by supplying the new option
`--ignoreConfigUsers`.
As before, `performoperationalaction` allows you to specify a file
containing options with the `--optionsfile` option. If that file
specifies the `geni_users` option, then that is the only set of
users and keys that is supplied with `performoperationalaction`.
However, if you do not supply the `geni_users` option from a file,
Omni uses the same logic as for `createsliver`, optionally
querying your clearinghouse for slice members, and by default
reading users and keys configured in your `omni_config`. (#491)
- If set, `GENI_FRAMEWORK` environment variable is the default for
the `--framework` option (#315)
- If set, `GENI_USERCRED` and `GENI_SLICECRED` environment variables
set the default path to your saved user and slice credentials (#434)
- Handle `~` in `usercredfile` and `slicecredfile` (#455)
- Support querying for other users' SSH keys where the CH supports it (#472)
- Allow nicknames or URLs in the aggregates list in `omni_config` (#476)
- Allow `PerformOperationalAction` on v2 AMs (#412)
- Renew Slice returns actual new expiration (checks the SA, not just
assuming success means you got what you asked for) (#428)
- Add a 360 second timeout on AM and CH calls. Option `--ssltimeout`
allows changing this. (#407)
- If Omni hangs talking to a server you believe is up, try
specifying `--ssltimeout 0` to disable the timeout. Some servers
cannot handle the timeout request. (See ticket #506)
- Note this timeout does not work on old versions of python2.6 due
to a known python bug: http://bugs.python.org/issue5103
- Speed up `listaggregates` in `pgch` framework (don't test AM API
compliance) (#482)
- Refactored the source code to make it easier to import Omni in
other tools. Look in `src/gcf` for directories that were
previously directly under `src`. (#388)
- Your Omni based tool no longer needs to include any of the top
level scripts that Omni/GCF includes, nor to provide the existing
Omni `main`.
- Most of the code in `omni.py` has now been moved to
`gcf/oscript.py`
- To update your script that uses omni as a library:
- Change `import omni` to `import gcf.oscript as omni`
- Avoid sending options to `getversion` if there are none, to support querying v1 AMs (#375)
- Fix passing speaksfor and other options to `createsliver`, `renewsliver` (#377)
- `renewslice` when given a slice credential replaces the saved
slice credential in place, rather than in a new filename. (#386)
- Create any directories needed in the path to the agg_nick_cache (#383)
- If using `--AggNickCacheName` and can't read/write to the specified
file, omni should fall back to reading `agg_nick_cache.base` (#384)
- Look up AM URN by URL in the defined aggregate nicknames (#404)
- Support named timezones when renewing, etc (#503)
- Eliminated a repetitive log message (#384, #385)
- Fix bug in APIv3 calling status with slivers with different expiration times (#408)
- Fit Omni result summaries in 80 character wide terminals (#409)
- `ForceUseAggNickCache` avoids fetching new cache even if the agg
nick cache is old (#391)
- SFA slice and user records changed: keys and slices moved (#429)
- Fix bug in handling errors in `listimages` and `deleteimage` (#437)
- Support unicode urns (#448)
- Return any error message from a CH on `getusercred` (#452)
- Return error on SA error in `listslices` (#456)
- Omni `cred_util.py` uses an omni logger (#460)
- URN testing requires 4 `+` separated pieces (#483)
- Log at debug when downloading the aggregate nickname cache fails (#485)
- `chapi` framework looks up the MA and SA at the clearinghouse,
though you can configure where they run. (#490)
- Warn when acting at all AMs in the clearinghouse - slow (#461)
- Speaks for option that Omni passes to aggregates has been renamed
`geni_speaking_for` (#466)
- Show the AM nickname instead of URL in output (#424, #504)
- Properly parse the verbose config option and let the commandline
`--verbosessl` over-ride it talking to clearinghouses. (#509)
- Ensure `geni_version` on credential structs is a string.
Fix bug in `get_cred_type` and correct for a chapi bug. (#516)
- Notice invalid slice and member names earlier and suppress ugly
tracebacks on most `chapi` framework errors. (#517)
- Support AM API draft proposal O1 and allow '.' and '_' in sliver
names, and do not complain or stop needlessly on illegal sliver
names. (#518)
- Catch parse errors when determining credential type (#521)
- Using `chapi` framework, expired slice expirations are printed (#523)
- When doing `renewsliver --alap`, if the real expiration is not in
the `output` slot, call `sliverstatus` to get it. (#527)
- Bail early from `createsliver` or `createimage` if the user
didn't specify exactly one aggregate. (#395)
- Update copyrights to 2014 (#463, #426)
- Handle non string rspec when printing RSpec (#445)
- Allow `--optionsfile` with `createimage`, `deleteimage`, and
`listimages`. (#532)
- Allow underscore in generated clean filenames (#533)
- Handle `createslice` errors at the GENI Clearinghouse that might
be due to having the wrong case, now the project and slice names
are case sensitive. (#535)
- Trim trailing newlines before installing SSH keys (#537)
- Explicitly import framework files in `oscript.py` to support
Windows and Mac binaries. (#542)
- Fix wording and licenses for Windows and Mac binaries (#541)
Older changes are listed in the CHANGES file.
== Handling Omni Output ==
Omni supports the `-o` option to have Omni save the output of Omni to
one or more files. See the [#RunningOmni documentation] for individual
commands for details.
Omni output is done through the python logging package, and
prints to STDERR by default. Logging levels, format, and output
destinations are configurable by either supplying a Pythong logging
configuration dictionary (to `oscript.call` or `oscript.initialize`),
or by supplying a custom Python logging
configuration file, using the `-l` option. Note that these settings
will apply to the entire Python process. For help creating a logging
config file, see
http://docs.python.org/library/logging.config.html#configuration-file-format
and see the sample `omni_log_conf_sample.conf`. Note that if present
in your configuration file, Omni will use the special variable
'optlevel' to set logging to INFO by default, DEBUG if you
specify the `--debug` option to Omni, INFO if you specify `--info`,
etc. If multiple log level options are supplied, Omni uses the most
verbose setting specified. Note that at WARN and ERROR levels, command
outputs are not printed: use the `-o` option to save command results
to files, or --tostdout to print results to STDOUT.
You may also completely disable Omni output, by specifying the option
`--noLoggingConfiguration`. Unless you use Omni as a library and your
tool configures Python logging, Omni will not write any output to
Python logging streams. For example, a tool might include
`--noLoggingConfiguration` when initializing the Omni library, and
then programmatically configure Python logging itself. Note that you
should generally configure some logging; many errors will cause Python
to stop Omni immediately if a log message is called for and no logging
configuration has been done. (You will see an error like:
'`No handlers could be found for logger "omni"`'.)
For further control of Omni output, use Omni as a library from your
own python script (see [#OmniasaLibrary below] for details).
For example, your script can modify the `-l` logging config file
option between Omni calls.
Alternatively, you can call the Omni function
`omni.applyLogConfig(<path to your log config file>)`. See the
documentation for `applyLogConfig` for details.
When using Omni as a [#OmniasaLibrary script] and you do `omni.call`
or `omni.applyLogConfig` to load a logging configuration from a file,
existing loggers are NOT disabled (which is the python logging
default). However, those existing loggers will not be modified with
the new logging settings, unless they are explicitly named in the
logging config file (they or their ancestor, where 'root' does not
count).
== Omni as a Library ==
Omni is really a thin wrapper around `src/gcf/oscript.py`, which can
be imported as a library, enabling programmatic
access to Omni functions. To use Omni as a library,
`import gcf.oscript as omni` and use the `omni.call` function.
Note that prior to v2.5, the import statement was `import omni` - be
sure you have updated your scripts when you upgrade.
{{{
text, returnStruct = omni.call( ['listmyslices', username], options )
}}}
The return from `omni.call` is a list of 2 items: a human readable string summarizing the result
(possibly an error message), and the result object (may be `None` on error). The result
object type varies by the underlying command called. See the docs in
the source code for individual methods.
Omni scripting allows a script to:
* Have its own private options
* Programmatically set other omni options (like inferring the "-a")
* Accept omni options (like "-f") in your script to pass along to Omni
* Parse the returns from Omni commands and use those values in subsequent Omni calls
* Control or suppress the logging in Omni (see the above section for details)
For examples, see `src/stitcher.py` or `examples/expirationofmyslices.py` and `examples/myscript.py` in the gcf distribution.
Or [https://github.com/GENI-NSF/geni-tools/wiki/Omni-Scripting-Example-Showing-Expiration Omni Scripting Expiration]
and
[https://github.com/GENI-NSF/geni-tools/wiki/Omni-Scripting-Example-With-Options Omni Scripting with Options]
on the gcf wiki.
'''NOTE''': Omni uses multiple command line options, and creates its
own option names internally. Be sure not to pick the same option names. See `gcf/oscript.py` and the
`getParser()` function, around line 781 for all the option names.
== Extending Omni ==
Extending Omni to support additional frameworks with their own
clearinghouse APIs requires adding a new Framework extension
class. Adding other experiment management or utility functions can be
done using Omni scripting, or by adding functions to
`src/gcf/omnilib/amhandler.py`.
== Omni workflow ==
For a fully worked simple example of using Omni, see
http://groups.geni.net/geni/wiki/HowToUseOmni
1. Get your user certificate and keys: Pick a Clearinghouse you want to
use (that is the control framework you will use). Get a user
certificate and key pair.
2. Configure Omni: Be sure the appropriate section of omni_config for
your framework (sfa/gcf/pg/chapi/pgch) has appropriate settings for
contacting that CF, and user credentials that are valid for that
CF. Make sure the `[omni]` section refers to your CF as the default.
If you ran `src/omni-configure.py` this should automatically be
configured.
3. Find available resources: Run `omni.py -o listresources`
a. When you do this, Omni will contact your designated
Clearinghouse, using your framework-specific user credentials.
b. The Clearinghouse will list the AMs it knows about.
c. Omni will then contact each of the AMs that the
Clearinghouse told it about, and use the GENI AM API to ask each
for its resources.
d. Omni will save the Advertisement RSpec from each aggregate into a separate
file (the `-o` option requested that). Files will be named
`rspec-<server>.xml` or `rspec-<server>.json` depending on the AM
API version you are using.
4. Describe the resources you want to request: Create a request Rspec
to specify which resources you want to reserve. (See
[http://groups.geni.net/geni/wiki/GENIExperimenter/RSpecs RSpec Documentation]
for more details.)
5. Create a Slice:
Run: `omni.py createslice MySlice`
6. Allocate your resources:
Given a slice, and your request rspec file, you are ready to
allocate resources by creating slivers at each of the AMs. Note
you must specify the URL or nickname of the aggregate where you
want to reserve resources using the `-a` option. Note also that
here we show a local file for the rspec, but you can supply a URL.
In AM API v1:
`omni.py -V 1 createsliver -a pg-utah1 MySlice request.rspec`
In AM API v2 (default) do:
`omni.py createsliver -a pg-utah MySlice request.rspec`
In AM API v3 this requires 3 steps:
Step 1:
`omni.py -V 3 allocate -a myV3AM MySlice request.rspec`
Reserve the resources. Optionally you may delete the reservation if
you didn't get what you wanted, or hold your reservation and try
another reservation elsewhere to match. Be sure to `renew` your
reservation if you want to hold it a while before you `provision` it.
Step 2:
`omni.py -V 3 provision -a myV3AM MySlice`
Start Instantiating the resources.
At this point, you likely want to call `status` (see below), to check
when your slivers have been fully provisioned.
Step 3:
`omni.py -V 3 performoperationalaction -a myV3AM MySlice geni_start`
Boot or otherwise make available the resources. The specific actions
available will be aggregate and sliver type specific. Consult the
Advertisement RSpec from this aggregate for more information.
At this point, you have resources and can do your experiment.
7. Determine the status of your sliver: Use the `sliverstatus`
command in AM API v1 and v2 (or `status` in AM API v3+) to
determine the status of your resources. Resources must typically
be configured, and possibly booted, before they can be used.
In AM API v1 & v2:
When `geni_status` is `ready`, your resources are ready for your
experiment to use. Note: If `geni_status` is `unknown`, then
your resources might be ready.
In AM API v1 run:
`omni.py -V 1 sliverstatus -a pg-utah1 MySlice`
In AM API v2 run:
`omni.py sliverstatus -a pg-utah MySlice`
In AM API v3+:
After calling `provision`, use `status` to poll the aggregate manager
and watch as the resources are configured and become ready for use.
When calling `status`, look for a `geni_operational_state` other than
`geni_pending_allocation`. The actual operational state that the
sliver will change to depends on the sliver and aggregate
type. Operational states are sliver type and aggregate specific, and
defined in the aggregate's advertisement RSpec. In many cases, the
aggregate indicates that the sliver is fully allocated with a
`geni_operational_state` value of `geni_notready`. Once the resources
are ready for use, you can typically call `performoperationalaction
geni_start` to start the resources (e.g. boot a machine). You can
then call `status` again to watch the action take effect. In many
cases, the operational state will change from `geni_notready` to
`geni_ready`.
Run:
`omni.py -V 3 status -a myV3AM MySlice`
8. Renew your slice and slivers: Both slices and slivers have
distinct expiration times. After a while you may want to Renew
your Sliver before it expires and is deleted.
AM API v1:
`omni.py -V 1 renewsliver -a pg-utah1 MySlice 20120531`
AM API v2:
`omni.py renewsliver -a pg-utah MySlice 20120531`
AM API V3:
`omni.py -V 3 renew -a myV3AM MySlice 20120531`
9. Do your experiment!
Compute resources typically use SSH to let you log in to the
machines. The SSH keys configured in your omni_config `users` section
plus those for all members of your slice (when using select `chapi`
style clearinghouses like the GENI Clearinghouse)
should be available for use. See the options `--noSliceMembers` and
`--ignoreConfigUsers` to change this behavior.
10. Delete slivers when you are done, freeing the resources for others:
AM API v1:
`omni.py -V 1 deletesliver -a pg-utah1 MySlice`
AM API v2:
`omni.py deletesliver -a pg-utah MySlice`
AM API v3:
`omni.py -V 3 delete -a myV3AM MySlice`
11. Optional commands:
Occasionally you may run `listmyslices` to remind yourself of your
outstanding slices. Then you can choose to delete or renew them as
needed. If you don't recall when your slice expires, use
`print_slice_expiration` to remind yourself. If you use a `chapi`
framework (clearinghouse), you can additionally query the
clearinghouse to be reminded of where you may have reservations, using `listslivers`
To List your slices : `omni.py listmyslices`
To Print slice expiration : `omni.py print_slice_expiration MySlice`
To list slivers where you made reservations using Omni: `omni.py listslivers MySlice`
== Running Omni ==
=== Supported options ===
Omni supports the following command-line options.
{{{
$ ~/gcf/src/omni.py -h
Usage:
GENI Omni Command Line Aggregate Manager Tool Version 2.10
Copyright (c) 2011-2015 Raytheon BBN Technologies
omni.py [options] [--project <proj_name>] <command and arguments>
Commands and their arguments are:
AM API functions:
getversion
listresources [In AM API V1 and V2 optional: slicename]
describe slicename [AM API V3 only]
createsliver <slicename> <rspec URL, filename, or nickname> [AM API V1&2 only]
allocate <slicename> <rspec URL, filename, or nickname> [AM API V3 only]
provision <slicename> [AM API V3 only]
performoperationalaction <slicename> <action> [AM API V3 only]
poa <slicename> <action>
[alias for 'performoperationalaction'; AM API V3 only]
sliverstatus <slicename> [AMAPI V1&2 only]
status <slicename> [AMAPI V3 only]
renewsliver <slicename> <new expiration time in UTC> [AM API V1&2 only]
renew <slicename> <new expiration time in UTC> [AM API V3 only]
deletesliver <slicename> [AM API V1&2 only]
delete <slicename> [AM API V3 only]
shutdown <slicename>
update <slicename> <rspec URL, filename, or nickname> [Some AM API V3 AMs only]
cancel <slicename> [Some AM API V3 AMs only]
Non AM API aggregate functions (supported by some aggregates):
createimage <slicename> <imagename> [optional: false (keep image private)] -u <sliver urn> [ProtoGENI/InstaGENI only]
snapshotimage <slicename> <imagename> [optional: false (keep image private)] -u <sliver urn> [ProtoGENI/InstaGENI only]
[alias for 'createimage']
deleteimage <imageurn> [optional: creatorurn] [ProtoGENI/InstaGENI only]
listimages [optional: creatorurn] [ProtoGENI/InstaGENI only]
Clearinghouse / Slice Authority functions:
get_ch_version
listaggregates
createslice <slicename>
getslicecred <slicename>
renewslice <slicename> <new expiration time in UTC>
deleteslice <slicename>
listslices [optional: username] [Alias for listmyslices]
listmyslices [optional: username]
listprojects [optional: username] [Alias for listmyprojects]
listmyprojects [optional: username]
listmykeys [optional: username] [Alias for listkeys]
listkeys [optional: username]
getusercred
print_slice_expiration <slicename>
listslivers <slicename>
listprojectmembers <projectname>
listslicemembers <slicename>
addslicemember <slicename> <username> [optional: role]
removeslicemember <slicename> <username>
Other functions:
nicknames
print_sliver_expirations <slicename>
See README-omni.txt for details.
And see the Omni website at https://github.com/GENI-NSF/geni-tools/wiki
Options:
--version show program's version number and exit
-h, --help show this help message and exit
Basic and Most Used Options:
-a AGGREGATE_URL, --aggregate=AGGREGATE_URL
Communicate with a specific aggregate
--available Only return available resources
-c FILE, --configfile=FILE
Config file name (aka `omni_config`)
-f FRAMEWORK, --framework=FRAMEWORK
Control framework to use for creation/deletion of
slices
-r PROJECT, --project=PROJECT
Name of project. (For use with pgch framework.)
--alap Request slivers be renewed as close to the requested
time as possible, instead of failing if the requested
time is not possible. Default is False.
-t RSPEC-TYPE RSPEC-VERSION, --rspectype=RSPEC-TYPE RSPEC-VERSION
RSpec type and version to return, default: '['GENI',
'3']'
-V API_VERSION, --api-version=API_VERSION
Specify version of AM API to use (default v2)
--useSliceAggregates
Perform the slice action at all aggregates the given
slice is known to use according to clearinghouse
records. Default is False.
AM API v3+:
Options used in AM API v3 or later
--best-effort Should AMs attempt to complete the operation on only
some slivers, if others fail
--cred=CRED_FILENAME
Send credential in given filename with any call that
takes a list of credentials
--end-time=GENI_END_TIME
Requested end time for any newly allocated or
provisioned slivers - may be ignored by the AM
--start-time=GENI_START_TIME
Requested start time for any allocated slivers
- NOW if not provided, could be for future reservations
--optionsfile=JSON_OPTIONS_FILENAME
Send all options defined in named JSON format file to
methods that take options
--speaksfor=USER_URN
Supply given URN as user we are speaking for in Speaks
For option
-u SLIVERS, --sliver-urn=SLIVERS
Sliver URN (not name) on which to act. Supply this
option multiple times for multiple slivers, or not at
all to apply to the entire slice
--cancelled Should Describe show sliver state of only
geni_provisioned slivers, ignoring any geni_updating
and geni_allocated slivers (default False)
Logging and Verboseness:
Control the amount of output to the screen and/or to a log
-q, --quiet Turn off verbose command summary for omni commandline
tool
-v, --verbose Turn on verbose command summary for omni commandline
tool
--debug Enable debugging output. If multiple loglevel are set
from commandline (e.g. --debug, --info) the more
verbose one will be preferred.
--info Set logging to INFO.If multiple loglevel are set from
commandline (e.g. --debug, --info) the more verbose
one will be preferred.
--warn Set log level to WARN. This won't print the command
outputs, e.g. manifest rspec, so use the -o or the
--outputfile options to save it to a file. If multiple
loglevel are set from commandline (e.g. --debug,
--info) the more verbose one will be preferred.
--error Set log level to ERROR. This won't print the command
outputs, e.g. manifest rspec, so use the -o or the
--outputfile options to save it to a file.If multiple
loglevel are set from commandline (e.g. --debug,
--info) the more verbose one will be preferred.
--verbosessl Turn on verbose SSL / XMLRPC logging
-l LOGCONFIG, --logconfig=LOGCONFIG
Python logging config file. Default: 'none'
--logoutput=LOGOUTPUT
Python logging output file [use %(logfilename)s in
logging config file]. Default: 'omni.log'
--tostdout Print results like rspecs to STDOUT instead of to log
stream
--noLoggingConfiguration
Do not configure python logging; for use by other
tools.
File Output:
Control name of output file and whether to output to a file
-o, --output Write output of many functions (getversion,
listresources, allocate, status, getslicecred,...) ,
to a file (Omni picks the name)
-p FILENAME_PREFIX, --prefix=FILENAME_PREFIX
Filename prefix when saving results (used with -o, not
--usercredfile, --slicecredfile, or --outputfile)
--outputfile=OUTPUT_FILENAME
Name of file to write output to (instead of Omni
picked name). '%a' will be replaced by servername,
'%s' by slicename if any. Implies -o. Note that for
multiple aggregates, without a '%a' in the name, only
the last aggregate output will remain in the file.
Will ignore -p.
--usercredfile=USER_CRED_FILENAME
Name of user credential file to read from if it
exists, or save to when running like '--usercredfile
myUserCred.xml -o getusercred'. Defaults to value of
'GENI_USERCRED' environment variable if defined.
--slicecredfile=SLICE_CRED_FILENAME
Name of slice credential file to read from if it
exists, or save to when running like '--slicecredfile
mySliceCred.xml -o getslicecred mySliceName'. Defaults
to value of 'GENI_SLICECRED' environment variable if
defined.
GetVersion Cache:
Control GetVersion Cache
--NoGetVersionCache
Disable using cached GetVersion results (forces
refresh of cache)
--ForceUseGetVersionCache
Require using the GetVersion cache if possible
(default false)
--GetVersionCacheAge=GETVERSIONCACHEAGE
Age in days of GetVersion cache info before refreshing
(default is 7)
--GetVersionCacheName=GETVERSIONCACHENAME
File where GetVersion info will be cached, default is
~/.gcf/get_version_cache.json
--noCacheFiles Disable both GetVersion and Aggregate Nickname cache
functionality completely; no files are downloaded,
saved, or loaded.
Aggregate Nickname Cache:
Control Aggregate Nickname Cache
--NoAggNickCache Disable using cached AggNick results and force refresh
of cache (default is False)
--ForceUseAggNickCache
Require using the AggNick cache if possible (default
False)
--AggNickCacheAge=AGGNICKCACHEAGE
Age in days of AggNick cache info before refreshing
(default is 1)
--AggNickCacheName=AGGNICKCACHENAME
File where AggNick info will be cached, default is
~/.gcf/agg_nick_cache
--AggNickDefinitiveLocation=AGGNICKDEFINITIVELOCATION
Website with latest agg_nick_cache, default is
https://raw.githubusercontent.com/GENI-NSF/geni-
tools/master/agg_nick_cache.base. To force Omni to
read this cache, delete your local AggNickCache or use
--NoAggNickCache.
For Developers / Advanced Users:
Features only needed by developers or advanced users
--useSliceMembers DEPRECATED - this option no longer has any effect. The
option is always true, unless you specify
--noSliceMembers.
--noSliceMembers Reverse of --useSliceMembers. Do NOT create accounts
or install slice members' SSH keys on reserved
resources in createsliver, provision or
performoperationalaction. Default is False. When
specified, only users from your omni_config are used
(unless --ignoreConfigUsers).
--ignoreConfigUsers
Ignore users and SSH keys listed in your omni_config
when installing SSH keys on resources in createsliver
or provision or performoperationalaction. Default is
false - your omni_config users are read and used.
--ssltimeout=SSLTIMEOUT
Seconds to wait before timing out AM and CH calls.
Default is 360 seconds.
--noExtraCHCalls Disable extra Clearinghouse calls like reporting
slivers. Default is False.
--devmode Run in developer mode: more verbose, less error
checking of inputs
--raise-error-on-v2-amapi-error
In AM API v2, if an AM returns a non-0 (failure)
result code, raise an AMAPIError. Default is False.
For use by scripts.
--maxBusyRetries=MAXBUSYRETRIES
Max times to retry AM or CH calls on getting a 'busy'
error. Default: 4
--no-compress Do not compress returned values
--abac Use ABAC authorization
--arbitrary-option Add an arbitrary option to ListResources (for testing
purposes)
--no-ssl do not use ssl
--no-tz Do not send timezone on RenewSliver
--orca-slice-id=ORCA_SLICE_ID
Use the given Orca slice id
}}}
==== Notes on Options ====
Most options are documented more fully with the commands where they
are relevant. Some notes on select options are provided here.
Commonly used options:
Omni provides many options. However, most users will only use a very
few of them.
- `-a`: specify the aggregate at which to operate
- `-r`: over-ride your default project setting
- `--alap`: ask aggregates to renew your reservations as long as possible
- `-o`: to save manifest RSpecs to a file
- `--useSliceAggregates`: renew or delete at all aggregates where you
used Omni to reserve resources
- `--optionsfile`: Specify custom options for commands
- `-u`: Specify a particular sliver to act on when using APIv3
- `--debug`: Turn on debug logging when resolving an Omni problem
- `--usercredfile`: Use a saved user credential (to save time
calling the clearinghouse re-retrieving it later)
- `--slicecredfile`: Use a saved slice credential (to save time
calling the clearinghouse re-retrieving it later)
Basic Options:
- The `-a` option may be an aggregate nickname or URL. Nicknames are
defined in your `omni_config` plus some standard nicknames are
defined globally and updated automatically by Omni. Run the command
`nicknames` for a list.
- `--available` is for use with `listresources` and `describe`
- `-f` specifies the name of the framework in your `omni_config`,
such as "my_genich". It is not required if you want to use the
framework specified by `default_cf` in your `omni_config`.
- If set, the `GENI_FRAMEWORK` environment variable will be the
default for the `--framework` option, over-riding any default from
your Omni config file.
- `-r` / `--project` is used by framework types `pgch` and `chapi` which
use a 'project' to group slices. It is only required if you want to
use project other than your configured `default_project` from your `omni_config`.
- `--alap` is supported only at select aggregates with `renew` and
`renewsliver`
- `-t` is typically not required. See `createsliver`
- `-V` is typically not required as AM API v2 is the default. Specify
`-V3` if you are calling an AM API V3 command
- `--useSliceAggregates` is supported only when using framework type
`chapi`, and then only with commands after your resources have been
reserved, such as `renewsliver`, `sliverstatus`, and
`deletesliver`. It relies on the clearinghouse's advisory list of aggregates where
you have used Omni to reserve resources - reservations made using
other tools may not be reflected.
AM API v3 Options:
- `--best-effort`: In AM API v3 operations are on multiple
slivers. This option, when supported by the aggregate, allows the AM
to let the operation succeed on slivers where possible and only fail
the operation on select slivers, rather than having to fail the whole
request. For example, you might use this with `renew` to renew your
reservation for 3 nodes, and allow the aggregate to fail to extend
your reservation on a 4th node.
- `--cred` allows you to specify an extra credential to any call that
takes a list of credentials.
- `--optionsfile` allows you to specify arbitrary options to be
passed to aggregate calls. For example, use this to specify options
for `performoperationalaction`.
File Output Options:
- `-o` specifies that many commands write their output to a
file. This includes RSpecs, sliver status, and slice and user
credentials. Not all commands support this. `-p` and `--outputfile`
work with `-o`.
- `--usercredfile` and `--slicecredfile`: Most aggregate operations
require a credential to prove your authorization. This credential
comes from your framework / clearinghouse. This credential typically
does not change frequently, so many users choose to save this
credential to a file using the commands `omni -o getusercred` and
`omni -o getslicecred`. Then by supplying the `--usercredfile` and
`--slicecredfile` options, Omni can uses the saved credential and
save the time required to re-fetch the credential from the
clearinghouse.
Advanced / Developer Options:
- `--ssltimeout`: Omni times out calls to servers, by default after
360 seconds (6 minutes). Use this option to change that timeout. If
commands to a server that you believe is up are failing, try
specifying a timeout of `0` to disable the timeout.
- `--noExtraCHCalls`: Omni makes multiple calls to the
clearinghouse, particularly when using framework type `chapi`. These
include reporting creation / renewal of slivers, querying for lists
of aggregates, etc. These functions are necessary to support some
Omni operations (such as provided by `--useSliceMembers` and
`--useSliceAggregates`). However, you can disable these calls with
this option.
- `--noSliceMembers`: Disable `useSliceMembers`, which is True by default.
By default, Omni tries to contact your clearinghouse to retrieve
any slice members, to install their SSH keys on any reserved
resources. This is only supported at some CHAPI-based
clearinghouses, specifically the GENI Clearinghouse. Note also that
`useslicemembers` in the `omni` section of your `omni_config` file
can be set to `false` to disable `useSliceMembers`.
- `--ignoreConfigUsers`: By default, `createsliver` and `provision`
tell the aggregate to create accounts including for users listed in the users