Dataset Metadata
================

Each ICAT dataset can have metadata associated with it, describing experimental and contextual information.
The definitions of these metadata elements are documented in the
`ICAT metadata definitions project <https://icat-esrf-definitions.readthedocs.io/en/latest/>`_.
Additionally, metadata specific to the experimental techniques and their ontology is described in the
`ESRF ontologies documentation <https://esrf-ontologies.readthedocs.io/en/latest/>`_.

BLISS
-----

For details, see the `BLISS documentation <https://bliss.gitlab-pages.esrf.fr/bliss/master/data/data_dataset_metadata.html>`_. 

In short the `SCAN_SAVING` object of the BLISS session has attributes `proposal`, `collection` and `dataset` to define
metadata uploaded to ICAT for every dataset within the scope of the proposal, collection or individual dataset respectively.

Setting and getting metadata can be done with the namespace attribute `metadata` or by using keys

.. code-block:: python

    DAU [10]: SCAN_SAVING.dataset.metadata.instrument.beam.incident_energy = 7.1
    DAU [11]: SCAN_SAVING.dataset["InstrumentBeam_incident_energy"] = 7.1
    DAU [12]: SCAN_SAVING.dataset.existing
    Out [12]: Namespace contains:
            .startDate                      = '2026-03-14T16:08:56.485562+01:00'
            .InstrumentBeam_incident_energy = 7.1
            .Sample_name                    = 'dautest'

Unknown metadata keys
+++++++++++++++++++++

Using an unknown key raises a `KeyError`

.. code-block:: python

    DAU [15]: SCAN_SAVING.dataset["InstrumentBeam_incident_energ"] = 7.1
    !!! === KeyError: "'InstrumentBeam_incident_energ' is not a valid ICAT field" === !!! ( for more details type cmd 'last_error(2)' )

Validation and type/unit coercion
+++++++++++++++++++++++++++++++++

Metadata values are validated, type-coerced and unit-coerced when closing a dataset.
Failures will be logged but do not raise exceptions

.. code-block:: python

    DAU [11]: newdataset()
    Dataset set to '0003'
    Data path: /data/id00/inhouse/id002603/id00/20260301/RAW_DATA/dautest/dautest_0003

    DAU [5]: SCAN_SAVING.dataset.metadata.instrument.beam.incident_energy = "wrong"
    DAU [8]: SCAN_SAVING.dataset["InstrumentBeam_incident_energy"] = "wrong"

    DAU [6]: SCAN_SAVING.dataset.existing
    Out [6]: Namespace contains:
            .startDate                      = '2026-03-14T16:05:33.333542+01:00'
            .InstrumentBeam_incident_energy = 'wrong'
            .Sample_name                    = 'dautest'

    DAU [10]: loopscan(10,0.1,diode1)
    Out [24]: Scan(number=1, name=loopscan, path=/data/id00/inhouse/id002603/id00/20260301/RAW_DATA/dautest/dautest_0003/dautest_0003.h5)

    DAU [11]: newdataset()
    WARNING: ValidationError: 1 validation error for IcatDatasetParameters
    instrument.beam.incident_energy
    Value error, could not convert string to float: 'wrong' [type=value_error, input_value='wrong', input_type=str]
        For further information visit https://errors.pydantic.dev/2.12/v/value_error

    Documentation on ICAT fields: https://icat-esrf-definitions.readthedocs.io/en/v2.5.0/icat.html
    Dataset set to '0004'
    Data path: /data/id00/inhouse/id002603/id00/20260301/RAW_DATA/dautest/dautest_0004

Metadata units
++++++++++++++

Values with units can be set like this

.. code-block:: python

    DAU [17]: SCAN_SAVING.dataset["InstrumentBeam_incident_energy"] = 7100,"eV"

The value shown in the Data Portal will be `7.1 keV`, since the units of this ICAT field are `keV`.

When the unit does not correspond to the dimensionality, you get a warning but again no exception is raised

.. code-block:: python

    DAU [23]: SCAN_SAVING.dataset["InstrumentBeam_incident_energy"] = 7100,"m"

    DAU [24]: loopscan(10,0.1,diode1)
    Out [24]: Scan(number=1, name=loopscan, path=/data/id00/inhouse/id002603/id00/20260301/RAW_DATA/dautest/dautest_0005/dautest_0005.h5)

    DAU [25]: newdataset()
    WARNING: DimensionalityError: Cannot convert from 'meter' ([length]) to 'kiloelectron_volt' ([mass] * [length] ** 2 / [time] ** 2)

    Documentation on ICAT fields: https://icat-esrf-definitions.readthedocs.io/en/v2.5.0/icat.html
    Dataset set to '0006'
    Data path: /data/id00/inhouse/id002603/id00/20260301/RAW_DATA/dautest/dautest_0006