Setup a HTCondor Pool using the SIMPLE Grid Framework: User Guide
The SIMPLE Grid Project is a framework that reduces the task of setting up a Worlwide LHC Computing Grid (WLCG) to a few hundred lines of YAML! It is an acronym for Solution for Installation, Management and Provisioning of Lightweight Elements
The WLCG unites resources from over 169 sites spread across the world and the number is expected to grow in the coming years. However, setting up and configuring new sites to support WLCG workloads is still no straightforward task and often requires significant assistance from WLCG experts. A survey presented in CHEP 2016 revealed a strong wish among site admins for reduction of overheads through the use of prefab Docker containers or OpenStack VM images, along with the adoption of popular tools like Puppet for configuration. In 2017, the Lightweight Sites project a.k.a the SIMPLE Grid project was initiated to construct shared community repositories providing such building blocks.
The WLCG is a diverse environment. There are several flavors of compute elements, batch systems middleware packages etc. available. The sites and site admins too have their preferences towards infrastructure technologies for provisioning, configuration and management of nodes.
The mission of the SIMPLE Grid Project is to
- Encompass this diversity and deliver a consistent experience to site admins for setting up and configuring the various grid services supported by the SIMPLE Grid Framework.
- Abstract low level details related to configuration and operation of Grid Sites from the perspective of a Site Admin.
The SIMPLE Grid Project is completely open source and is fueled by the spirit of FOSS! There is something for everyone looking to contribute to the project's mission.
As mentioned earlier, the WLCG is diverse. The SIMPLE Grid Framework has a modular architecture enabling it to easily scale in terms of supported grid services and supported infrastructure technologies.
The team at CERN is currently focused to deliver one recipe for setting up a site using a subset of these grid services (CreamCE, Torque Batch System, Worker Nodes based on Torque Client) and infrastructure tools (Docker, Docker Swarm, Puppet YAIM). Support for more grid services(ARC Compute Elements, Slurm, HTCondor) and infrastructure tools (Ansible, SaltStack, Kubernetes, Apache Mesos etc.) can be added by experienced developers/ experts in parallel, allowing the framework to be more impactful in a lesser amount of time. Learn More!
Site admins are an integral part of the success of the project. As we are nearing the first release of the framework, we'd like to invite site admins to help test the framework and also experience some of the benefits of managing their grid site, the SIMPLE way! Learn More
The WLCG is one of the largest distributed computing infrastructures. It is spread over 170 research institutes and computing centers around the globe. Given the wide array of technologies involved, the project not only offers a great learning environment but also provides a platform to make impact to a truly massive and widespread computing infrastructures of today.
We are participating in the Google Summer of Code Program to help students sharpen their ITSM skillset and provide open source learning experience through the project. Their efforts in turn help expand the set of technologies supported by the project.
If you are a student looking for an interesting milestone in your open source journey, do get in touch using the channels specified in the next section!
No FOSS project is complete without FOSS Enthusiasts and a solid ITSM team to drive the community! To keep a community flourishing, the crucial role of various community managers, administrators and ITSM operators cannot be left out. If you see yourself as a driving force to the community, we welcome you to lead those efforts. Get in touch via the channels mentioned in the next section!
As part of GSoC 2019, Sneha Sinha, a final year student from IIIT-Delhi, India has improved the core python components of the framework. Her code is part of the v1.0.0 release of the framework. Check out is the final report of her contributions here.
The project is being presented at CHEP 2018, in Sofia, Bulgaria. Here is the link to the event and slides
As part of Google Summer of Code 2018, Tarang Mahapatra from the University of British Columbia, Vancouver is adding support for Ansible to configure grid services