How to manually update the retention policy of an index (Quickwit 0.8)

[guilload]

There is no API to do so in Quickwit 0.8.

[guilload]

The API for updating the retention policy will be available in Quickwit 0.9. For versions 0.8 and below, you can manually update the retention policy by editing the metastore file (if using a file-backed metastore) or running a SQL query (if using PostgreSQL).

File-backed metastore

1. Stop the metastore instance.
2. Locate the metastore file for your index at <metastore URI>/<index ID>/metastore.json.
3. Open the file in a text editor and find the index.index_config.retention section.
4. Set the period to the desired retention duration.
5. If the retention object is missing or set to null, you can create it with the following structure:

"retention": {
  "period": "30 days",
  "schedule": "daily"
}

6. Save the file and restart the metastore instance to apply the changes.

PostgreSQL

Simply run this SQL query with your desired retention settings, and the janitor will apply the changes within the next 10 minutes.

UPDATE indexes
SET index_metadata_json = jsonb_set(
    index_metadata_json::jsonb,
    '{index_config,retention}',
    '{"period": "30 days", "schedule": "daily"}'::jsonb,
    true
)::text
WHERE indexes.index_id = '<index ID>'
;

[xrl]

Is it possible to set the retention of automatic indexes like otel-logs-v0_7 from the root config file?

[guilload]

No, not at the moment.

[anmolnagpal]

@guilload

I am using this config for backend metastore .

default_index_root_uri: azure://quickwit/indexes
gossip_listen_port: 7282
listen_address: 0.0.0.0
storage:
  azure:
    access_key: key
    account: qa-storage-acount

These are the pods running right now using helm chart.
I have stop the metstore pod and look //metastore.json in quickwit-indexer-0 but i am not able to find metastore.json file.

quickwit-control-plane-3f3r
quickwit-indexer-0                         
quickwit-janitor-3f343
quickwit-metastore-3f343
quickwit-searcher-0                   
quickwit-searcher-1                
quickwit-searcher-2

Please guide if I am not following correct path.

[fulmicoton]

with your configuration, this file is on the metastore node, in the qwdata dir.

You should consider put it on azure however. Right now, if your metastore node loses its disk for any reason, you will lose all of your metadata (and your data will not be accessible). You can set that by adding a metastore_uri to azure://quickwit/ in your config. Unfortunately you will use all of your data if you do so.

[anmolnagpal]

I look into the qwdata dir on my dev and prod env.
But dir is showing blank for both env.

I have no name!@quickwit--metastore-:/quickwit/qwdata$

even my node.yaml is correct which comes from configmap using config .

default_index_root_uri: azure://quickwit/indexes
gossip_listen_port: 7282
listen_address: 0.0.0.0
storage:
  azure:
    access_key: key
    account: qwsaaccount

[anmolnagpal]

@guilload do you get any chance to look into this. ?