Not duplicating meta data when we upload a derived dataset to SciCat.

[YooSunYoung]

Context

I was in a meeting with SWAT team about imaging workflows.
Data analysis software needs to search the derived dataset(reduced data) on SciCat based on the meta data of input datasets.
It's because data-analysis tools will need to use some physical characteristics or description about the sample, and those information will be in the metadata.

And I just talked to Max in SIMS team, and he said they can provide a simple query that can search derived dataset based on input-dataset-metadata.

Discussion Point

One of the suggested solutions is to just copy+paste input-dataset-metadata into derived dataset so that it's self-containing.
But we concluded that it should be avoided to copy the metadata, since metadata can be updated relatively easily, and then there is a risk that the metadata between input dataset and the derived dataset don't match.

I thought it's worth documenting this here since it might be relevant for all instruments.

[jl-wynen]

There is a subset of metadata that we can and should copy to the derived dataset; the sample in particular.
But this will depend on the workflow. So I don't think we can have a completely generic solution.

[YooSunYoung]

In which cases should we copy the metadata? I thought we will always have a link to the input-dataset.

[jl-wynen]

I'm talking mainly about the main fields of Dataset, not scientificMetadata. The former should always contain at least a link to the sample, instrument, and proposal. Otherwise, users have to use more complicated queries and follow chains of related datasets to find all information.

But of course, these fields depend on the workflow. E.g., we would want to preserve the sample under investigation but not the normalisation sample. Or, when combining data from multiple instruments, we can't set the instrument field.

[YooSunYoung]

I think we should still draw a line somewhere.
If any of the metadata is used in the data reduction, that should also fall into the derived dataset (in the body),
but if not, I think they shouldn't be duplicated, especially in the scientificMetadata.

I agree that it'll be more complicated for users to search them. That's why this discussion started.
Because it'll be easier if we just copy+paste the metadata into all derived dataset.
But then I realized it can go wrong subtly and easily, talked to Max about it and he also agreed.

So IMO, data-reduction shouldn't concern about the searching and duplicate the metadata
but the SciCat should handle them.

[jl-wynen]

So IMO, data-reduction shouldn't concern about the searching and duplicate the metadata
but the SciCat should handle them.

I generally agree. But the frontend is unfortunately a long way off supporting this properly.

Also think about the metadata in the file: We will still write the sample information into it (where possible). So, to make it findable, that metadata should also be in the SciCat dataset.

I agree that we cannot blindly copy everything. All I am saying is that we should duplicate some metadata. And that we have to make informed decisions about it when writing workflows.

[YooSunYoung]

Yes, Max also said it's not so simple from the frontend.
(I still think we should guide the users to find the input dataset first and then find the derived dataset based on them.
But I guess that's out of scope of this discussion.)

I would like to narrow this down to the data analysis steps, that will be maintained by SWAT team.
I think (if we agree) we should document somewhere that downstream data analysis SW shouldn't rely on the metadata of the derived dataset,
and always use the metadata of its parent dataset.
So that we don't have to waste time discussing about what should be where.