Simplified obs.setup with metadata

[anders-kiaer]

Background:

The current ERT observation format has some limitations:

• it is not compatible with adding context/metadata on observations
• it is not user friendly to set up for standard observation types like RFT (which requires consistent configuration in 4-5 different files today)
• for "summary" observations the current format is verbose and not user friendly to set up

New observation format (draft):

In early 2024, team members from Atlas, Sudo, Scout, Webviz and (what today is) Targa, designed a draft for new format. At a high level these were the suggested design principles for the new format:

• The top level observation input file is .yaml.
• To avoid this .yaml becoming very large/verbose, for relevant observation types it refers to .csv / .xlsx containing observation points.
  • This can also be extended to seismic observation type, where it refers to e.g. binary files (since the seismic observation data itself could be massive), compared to today's requirement where ERT requires all observation types coming as text files. This might also improve FMU run performance, since the forward models can dump to binary files instead of text files.

YAML schema sketch:

- label: Gas-oil-ratio observations
  type: timeseries
  error: 10%  # optional (% indicates relative, otherwise absolute)
  min_error: 20  # optional, always absolute
  max_error: 30  # optional, always absolute
  # Columns: vector, date, value, error, min_error, max_error
  # (last three columns optional for each row, overrides global error for the rows where they are presnet)
  observation: "./input/observations/summary_gor.csv"

# The user can include the same observation type multiple times,
# with the benefit of easily giving different error to e.g. gas-oil-ratio observations
# compared with e.g. pressure observations, and also have different names useful in post-run mismatch
# analysis to group time series observations in different relevant "buckets" .
- name: Bottom hole pressure observations
  type: timeseries
  error: 5%
  observation: "./input/observations/summary_bhp.csv"

# Example of another observation type, rft:
- label: RFT pressure observations
  type: rft  # we have a plugin that "knows about rft"
  plugin_arguments:
    zonemap: "./rms/output/zone/layer_zone_table.txt"
    trajectories: "./rms/output/trajectories.csv"
  error: 5 
  # Columns: well, date, measured_depth, zone, value, error, min_error, max_error (last three columns optional)
  observation: "./input/observations/rft_pressure.csv"
  metadata:
    # should the metadata below have default values? TBD with `fmu-dataio`
    columns:
      measured_depth:
         unit: m
         reference: msl
      value:
         unit: bar

# Example of another observation type, rft, in this case saturation:
- label: RFT water saturation observations
  type: rft  # we have a plugin that "knows about rft"
  plugin_arguments:
    zonemap: "<RUNPATH>/rms/output/zone/layer_zone_table.txt"
    trajectories: "./rms/output/trajectories.csv"
  error: 5 
  # TBD: Should we support saturation values for multiple phases in the csv file below?
  # Columns: well, date, measured_depth, zone, value, error, min_error, max_error (last three columns optional)
  observation: "./input/observations/rft_saturation.csv"
  metadata:
    columns:
      measured_depth:
         unit: m
         reference: msl
      value:
         unit: fraction

# Other observation types this can be extended to support:
- label: 4D gravity
  type: gravity
  ...

- label: 4D seismic
  type: seismic
  ...

- label: Breakthrough time
  type: breakthrough-time
  ...

For each observation type, ERT knows how to internally translate it down to:

• An array of values and errors (since this is what ERT needs to do AHM).
• A corresponding "simulated response extraction job".
  • Today ERT automatically extracts simulated .UNSMRY when a realization is finished, and implicitly know this should be compared with the list of (value, error) pairs from the SUMMARY_OBSERVATION. The proposal in the improved format is that this is extended to natively support also other observations.
    • In other words, the type timeseries would extract simulated responses similar to what ERT does today from .UNSMRY.
    • For e.g. the observation type rft, ERT would extract simulated responses similar to what the FORWARD_MODEL GENDATA_RFT does today.

This could either be maintained centrally in ERT, or maybe as a plugin system (similar to FORWARD_MODELS today).

[oddvarlia]

Suggestion for adding additional specifications to the observation config file:
For type: series add the keyword to this observation group:

• influence_range: [ range1, range2, angle] (This is then default for all observations of this group)
• The columns in the csv file can be extended to contain:
  well_name, vector, date, value, error, xpos, ypos, hwell_length, hwell_angle, min_error, max_error, influence_range1, influence_range2, influence_angle

The well_name can be useful, but maybe not necessary. There can be cases where influence_ranges are well dependent, not only observation type (BHP, WWCT,...) dependent. The additional columns define position and for horizontal wells also info about approximate length and lateral orientation, this is info that can be used to modify the influence ellipse around horizontal wells, but this may be unnecessary if we instead modify the ranges for observations from this well instead and not assume some recalculation is done. It is then up to the user to ensure that the influence ellipse is large enough for the horizontal wells.

About user friendliness: Manual specification of the csv files is probably not realistic. I think in practice that we will need some preparation scripts using input from original sources to create the input.