Description
This is the version of the GaiaDMP system that we have setup & will use NAM 2022.
What's Changed
- 20200103 zrq storage by @Zarquan in #40
- Notes on GitHub branch protection by @Zarquan in #64
- Notes and example of writing to HDFS File by @stvoutsin in #66
- 20200121 zrq ansible by @Zarquan in #70
- 20200425 zrq storage by @Zarquan in #125
- Added notes on fixing issue #108 by @stvoutsin in #122
- 20200617 zrq magnum terraform by @Zarquan in #124
- Notes on deploying Zeppelin on Kubernetes by @stvoutsin in #121
- Added some notes on experimenting with Hadoop on a K8s cluster by @stvoutsin in #120
- Added example of a Github CI workflow from zeppelinTo project by @stvoutsin in #113
- Notes on Disk space issue by @stvoutsin in #115
- Python tests & notes to address issue #41 by @stvoutsin in #114
- Notes and list of 3rd party Python libraries #104 by @stvoutsin in #119
- Notes on Setting up Spark on Kubernetes (with Magnum) on Openstack by @stvoutsin in #69
- Added notes on monitoring Spark applications via Python and the Spark UI by @stvoutsin in #117
- Notes on setting up and modifying Zeppelin accounts by @stvoutsin in #116
- Notes on experiment with Apache Arrow, and some notes with interesting links by @stvoutsin in #123
- 20200526 zrq planning by @Zarquan in #126
- 20200622 zrq git sync by @Zarquan in #128
- 20200622 zrq planning by @Zarquan in #146
- 20200623 zrq kubernetes by @Zarquan in #148
- 20200706 zrq drupal by @Zarquan in #149
- 20200718 zrq kubernetes by @Zarquan in #152
- 20200806 zrq spark s3 by @Zarquan in #156
- 20200824 zrq cephfs by @Zarquan in #177
- 20201012 zrq ceph mount by @Zarquan in #202
- Bump spring.version from 5.0.8.RELEASE to 5.2.9.RELEASE in /experiments/zrq/java/spark-tools by @dependabot in #178
- Stv issue 155 by @stvoutsin in #207
- Stv issue 104 by @stvoutsin in #208
- 20201102 zrq dependabot by @Zarquan in #210
- 20201019 zrq ceph gaia by @Zarquan in #213
- 20200917 zrq drupal by @Zarquan in #212
- 20200928 zrq ceph csi by @Zarquan in #214
- 20200908 zrq zeppelin by @Zarquan in #203
- 20201026 zrq helm by @Zarquan in #219
- Stv ansible deploy by @stvoutsin in #206
- Notes on issue #103 - Fine tuning Spark & Yarn by @stvoutsin in #118
- Stv ansible deploy by @stvoutsin in #220
- 20201112 zrq hadoop yarn by @Zarquan in #225
- 20201117 zrq hadoop yarn by @Zarquan in #228
- Stv zeppelin ansible by @stvoutsin in #237
- Stv issue 251 by @stvoutsin in #252
- 20201123 zrq user data by @Zarquan in #255
- 20201120 zrq kubernetes by @Zarquan in #230
- Stv zeppelin ansible by @stvoutsin in #268
- Stv issue 16 by @stvoutsin in #283
- 20201203 zrq drupal oauth by @Zarquan in #302
- 20201218 zrq container images by @Zarquan in #319
- 20210111 zrq catalog shares by @Zarquan in #322
- Debugging stalled service by @Zarquan in #326
- 20210118 zrq ssh keypair by @Zarquan in #328
- 20210125 zrq format notes by @Zarquan in #329
- 20210125 zrq kubernetes deploy by @Zarquan in #334
- Added notes on AXS Experiments by @stvoutsin in #357
- 20210127 zrq oauth by @Zarquan in #348
- 20210205 zrq error trap by @Zarquan in #350
- 20210205 zrq timeout by @Zarquan in #351
- 20210205 zrq deployname by @Zarquan in #352
- 20210205 zrq testing by @Zarquan in #353
- Delete experiments/stv/gdaf-openstack/config directory by @stvoutsin in #358
- Changes to increase notebook upload limit by @stvoutsin in #359
- 20210211 zrq gateway by @Zarquan in #360
- 20210211 zrq delete fix by @Zarquan in #361
- 20210211 zrq move vars by @Zarquan in #362
- 20210211 zrq fedora updates by @Zarquan in #363
- 20210211 zrq notes by @Zarquan in #364
- Notes on resource quotas and limits by @Zarquan in #383
- 20210219 live deployment by @Zarquan in #376
- 20210211 zrq volume mounts by @Zarquan in #368
- 20210218 zrq infraops by @Zarquan in #381
- Added notes on Spark tmp space investigation and some Benchmarking notes by @stvoutsin in #395
- 20210227 zrq housekeeping by @Zarquan in #389
- 20210308 zrq deployments by @Zarquan in #399
- Added notes on restart by @Zarquan in #396
- 20210308 live deployment by @Zarquan in #408
- 20210310 live deployment by @Zarquan in #407
- 20210228 zrq s3 echo by @Zarquan in #397
- Removed formatting tags from notes by @Zarquan in #414
- Edinburgh DataSync test by @Zarquan in #415
- Issue 419 cinder by @stvoutsin in #418
- Added wget to Fedora base install by @Zarquan in #427
- Add notes on latest live deploy on prod by @stvoutsin in #439
- Added details of partitioning test by @Zarquan in #440
- Added notes on most recent deploy by @stvoutsin in #441
- Update public ssh key for nch by @stvoutsin in #458
- Added notes on using Hadoop HDFS commands by @Zarquan in #424
- 20210422 zrq tiny-16-deploy by @Zarquan in #451
- 20210428 zrq spark conf by @Zarquan in #471
- Add Prometheus & Grafana monitoring to the Ansible deployment (Rebased) by @stvoutsin in #480
- 20210511 zrq hdfs transfers and data shares by @Zarquan in #482
- 20210518 zrq housekeeping by @Zarquan in #488
- 20210516 zrq postmortem by @Zarquan in #487
- Updates to fix issue #463 by @stvoutsin in #492
- 20210602 zrq resources by @Zarquan in #501
- Added Spark notes from Roger Mor by @Zarquan in #504
- Changes to Spark & Yarn configuration for performance optimization by @stvoutsin in #506
- 20210609 zrq resources by @Zarquan in #505
- Notes and spread-sheets for resource tests and new flavors by @Zarquan in #507
- Added new ssh key for dcr by @Zarquan in #508
- 20210620 zrq resources by @Zarquan in #516
- Configuration changes to the Ansible/Hadoop deploy for Dynamic Allocation & Multi user by @stvoutsin in #517
- Added notes on CU9 Meeting by @stvoutsin in #520
- 20210713 zrq crash debug by @Zarquan in #518
- Add notes on recent experiments with Dynamic allocation by @stvoutsin in #521
- 20210716 zrq test deploy by @Zarquan in #529
- Add notes on recent test of master branch by @stvoutsin in #530
- 20210727 zrq deploy test by @Zarquan in #536
- Issue live debugging by @stvoutsin in #537
- Removed task titles, fix for #547 by @Zarquan in #553
- 20210728 zrq live debug by @Zarquan in #538
- Added Dynamic Allocation configuration to Hadoop/Yarn deploy. (+ Idle Cache timeout) by @stvoutsin in #554
- Notes on Prometheus metrics and Spark performance by @Zarquan in #555
- Changing CephFS mounts to use kernel driver by @Zarquan in #556
- Notes on Prometheus metrics and Grafana dashboards by @Zarquan in #557
- Notes and config changes for cclake-large-06 deployment by @Zarquan in #558
- Notes on SSH tunnels and SOCKS proxies by @Zarquan in #559
- Notes on untangling the uber-branch by @Zarquan in #560
- Notes on diagnosing the cause of Zeppelin/Spark/Yarn failures by @Zarquan in #566
- 20210826 zrq socks proxy by @Zarquan in #568
- Notes on adding DuckDNS by @Zarquan in #572
- 20210702 zrq infraops by @Zarquan in #580
- 20210907 zrq config merge by @Zarquan in #576
- PR to solve issue: Missing dependency issue with recent deploy of aglais:master #587 by @stvoutsin in #588
- Updates to Ansible scripts to include an (optional) benchmark test by @stvoutsin in #583
- Upgrade Spark / Hadoop / Zeppelin (Issue #535) by @stvoutsin in #590
- 20211202 zrq pr testing by @Zarquan in #595
- 20211105 zrq iris 2022 by @Zarquan in #596
- Added gaiaxpy installation to Hadoop deploy (using Ceph share) by @stvoutsin in #611
- Create test share for storing data used by benchmarks by @stvoutsin in #597
- 20211011 zrq hdbscan config by @Zarquan in #609
- Store notebooks in Ceph share by @stvoutsin in #614
- Added a github workflow, that runs yamllint on our deployment scripts by @stvoutsin in #615
- 20211222 zrq cloud transfer by @Zarquan in #617
- Issue/yamllint fixes by @stvoutsin in #616
- 20220118 zrq arcus testing by @Zarquan in #620
- Remove notebook share and run test deploy by @stvoutsin in #628
- 20220131 zrq arcus deploy by @Zarquan in #635
- Re-enabling gaiaxpy by @stvoutsin in #626
- Added notebook to test the version of the python packages that get installed by @stvoutsin in #631
- Testing suite setup as a separate script / Separate configs for different test types by @stvoutsin in #639
- Added Koalas to GaiaDMP by @stvoutsin in #640
- 20220223 zrq deploy by @Zarquan in #641
- Added benchmark output validation by @stvoutsin in #645
- 20220309 zrq blue deploy by @Zarquan in #666
- Enable Apache arrow by @stvoutsin in #675
- GaiaDMPsetup library included by @stvoutsin in #677
- Feature/upgrade numpy by @stvoutsin in #682
- 20220329 zrq green deploy by @Zarquan in #683
- 20220405 zrq green deploy by @Zarquan in #684
- 20220331 zrq infra ops by @Zarquan in #685
- Create and enable a MySQL Shiro database for storing and authenticating Zeppelin Users by @stvoutsin in #687
- Add scripts for creating a new Zeppelin user by @stvoutsin in #702
- Fixing empty spaces in create-user-scripts yaml file by @stvoutsin in #703
- 20220517-zrq-user-accounts - all the changes in one big commit by @Zarquan in #712
- User impersonation by @stvoutsin in #714
- 20220519 zrq create users by @Zarquan in #723
- Changes to benchmarking suite (Passing user list in yml file) by @stvoutsin in #731
- Notes on concurrent testing by @Zarquan in #736
- 20220528 zrq refactoring by @Zarquan in #735
- Issue #750 / Fix Zeppelin interpreter setting page authorisation by @stvoutsin in #751
- Upgrade the version of aglais benchmarker by @stvoutsin in #752
- 20220601 zrq concurrent tests by @Zarquan in #785
- New deploy of gaiadmp by @stvoutsin in #803
- New deploy on green by @stvoutsin in #812
- Add Nginx proxy & Enable HTTPS (Lets Encrypt) on our Ansible deploy by @stvoutsin in #818
- Install required python libraries on all nodes by @stvoutsin in #828
- Add notes on new live deploy on blue by @stvoutsin in #837
New Contributors
- @dependabot made their first contribution in #178
Full Changelog: https://github.com/wfau/aglais/commits/v1.0.0
Contributors
Zarquan, stvoutsin, and dependabot