Skip to content

Commit

Permalink
pool: drop RDB based CEPH support
Browse files Browse the repository at this point in the history
Motivation:
Turned out that CEHP support is not that popular. Moreover,
sites that have tried to run dcache+ceph decided to go for
CephFS (mounted as regular FS)  provided better performance,
scalability and erasure encoding.

Modification:
Drop RBD based CEPH file channel, corresponding documentation and
properties.

Result:
less code

Acked-by: Marina Sahakyan
Target: master
Require-book: yes
Require-notes: yes
  • Loading branch information
kofemann committed Oct 20, 2023
1 parent 2076857 commit 2f268fb
Show file tree
Hide file tree
Showing 10 changed files with 5 additions and 624 deletions.
47 changes: 0 additions & 47 deletions docs/TheBook/src/main/markdown/cookbook-pool.md
Original file line number Diff line number Diff line change
Expand Up @@ -457,53 +457,6 @@ This command does:

3. All new copies of the file will become `sticky`.

## Running pools with CEPH backends

dCache pools can be configured to store files on locally mounted file systems or use CEPH as a back-end. The property `pool.backend` is used to control which back-end should be used:

```ini
pool.backend = ceph
```

dCache uses CEPH's block devices interface, know as `RBD`, to store data. The dCache pools map one-to-one onto CEPH pools. The CEPH pool must be manually created and, if required, configured before dCache can use it.

```console-root
rados mkpool <pool-name>
```

By default, the CEPH pool name is expected to match the dCache pool name. This can be changed by using

```ini
pool.backend.ceph.pool-name = ceph-pool-to-use
```

dCache uses a locally configured ceph client to operate. The location to client configuration files is controlled by `pool.backend.ceph.config` property and defaults to _/etc/ceph/ceph.conf_.

For authentication, the property `pool.backend.ceph.cluster` is used to set the cluser name to use, and for a cluster name of "CLNAME", the corresponding file `/etc/ceph/ceph.client.CLNAME.keyring` is used as the key ring.

In order to support HSM with CEPH-backended pools, the HSM script interface provides URI-like syntax to pass file locations to the HSM script:

rbd://<ceph-pool>/<pnfsid>

for instance:

rbd://dcache-pool-A/00000051ADCB3BA14799844556CD3AF0A9DF

The HSM script is responsible to read, write and delete RBD image on GET, PUT and DELETE.

In order to improve the performance of the backend, tests point to the RBD caching configuration (in `/etc/ceph/ceph.conf`) as the most promisting starting point:

# Start out in write-through mode, and switch to write-back after the
# first flush request is received. Enabling this is a conservative but
# safe setting in case VMs running on rbd are too old to send flushes,
# like the virtio driver in Linux before 2.6.32.
# Type: Boolean
# Required: No
# (Default: true)
;rbd cache writethrough until flush = true
rbd cache writethrough until flush = false


## Keeping metadata on MongoDB

In order to speed up database operations for metadata, dCache pools (starting from version 3.2) can store their metadata on an external MongoDB instance. For production scenarios, a dedicated, performance-optimized and well-maintained MongoDB cluster is required.
Expand Down
10 changes: 0 additions & 10 deletions docs/TheBook/src/main/markdown/cookbook-writing-hsm-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,16 +44,6 @@ contain replicas. It is the responsibility of the nearline storage driver to
copy replicas from the pool to the nearline storage and to copy replicas back
from the nearline storage into the pool.

Traditionally, dCache pools have stored replicas in the local file system, and a
flush or stage request would refer to the replica in the local file system by
path. Starting with dCache 3.0, *replica store plugins* may provide alternative
backends such as CEPH and thus replicas in a pool may not be accessible through
the local file system. For this reason, dCache 3.0 and newer identify the
replica in the pool by URI. Unless an alternative replica store is used, such a
URI will always use the `file:` scheme. If compatibility with versions earlier
than 2.17 is desired, a driver should refrain from using the `getReplicaUri`
method.

Once flushed to a nearline storage, the replica is identified by a URI generated
by the driver. This URI should use the nearline storage type (typically
`enstore` or `osm`) as the scheme and the nearline storage instance name as the
Expand Down
1 change: 0 additions & 1 deletion docs/TheBook/src/main/markdown/cookbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ This part contains guides for specific tasks a system administrator might want t
- [Examples](cookbook-pool.md#examples)
- [Renaming a Pool](cookbook-pool.md#renaming-a-pool)
- [Pinning Files to a Pool](cookbook-pool.md#pinning-files-to-a-pool)
- [Running pool with CEPH backend](cookbook-pool.md#running-pools-with-ceph-backends)
- [Keeping metadata on MongoDB](cookbook-pool.md#keeping-metadata-on-mongodb)
- [Handling orphan movers](cookbook-pool.md#handling-orphan-movers)

Expand Down
3 changes: 0 additions & 3 deletions docs/TheBook/src/main/markdown/preface.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,6 @@ A dCache instance will generally consist of many storage (or "pool") nodes.
On those nodes, normal Linux filesystems (btrfs, ext4, XFS, ZFS) are used
to store data.

Alternatively, dCache pools can use storage space provided by a Ceph object
storage system.

In addition to those possibilities, dCache can use its hierarchical storage
management capabilities to transparently use storage systems with different characteristics (like tape libraries for lower-cost, but higher-latency
storage). Built-in mechanisms can be used to increase performance and balance loads,
Expand Down
4 changes: 0 additions & 4 deletions modules/dcache/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -301,10 +301,6 @@
<artifactId>jimfs</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.dcache</groupId>
<artifactId>rados4j</artifactId>
</dependency>

<!-- used by alarms, needs to be deployed -->
<dependency>
Expand Down
Loading

0 comments on commit 2f268fb

Please sign in to comment.