Use VHC ******* Usage ===== Run ``vhc`` issuing the command:: vhc path [subdir] [extra [extra ...]] where ``path`` is the directory containing the data ``vhc`` should run on and where the output files are located. The drivers :ref:`shipped togeter with VHC ` interpret the second argument, ``subdir``, as the subdirectory of ``path`` from which the files must be collected. Any ``extra`` argument is ignored. As example consider the :doc:`following directory structure`. The following command:: vhc 20120301/virus0003000 collects all the FITS files in all the ``exp*/virus`` directories. Instead:: vhc 20120301/virus0003000 exp02 collects only the FITS files in the ``exp02/virus`` directory. Third party drivers might interpret ``subdir`` differently and make use of the ``extra`` arguments. For more info type ``vhc -h``:: usage: vhc [-h] [-V] [-c CONFIG] [-f FPLANE_FILE] path [subdir] [extra [extra ...]] Virus Health Check (VHC): check that Virus files contained into the given directory are fine positional arguments: path Path containing the files to explore subdir For the drivers shipped with VHC: the files will be collected only from the ``subdir`` subdirectory of ``path``. Third party drivers might interpret this argument as the first part of ``extra``. (default: None) extra Any extra arguments; ignored by the drivers shipped with VHC (default: None) optional arguments: -h, --help show this help message and exit -V, --version show program's version number and exit -c CONFIG, --config CONFIG Configuration file. If provided, overrides the search for it in the default positions. (default: None) Override options in the [fplane] section: -f FPLANE_FILE, --fplane-file FPLANE_FILE Override the fplane file name with the given option (default: None) The official configuration file can be found in the ``vhc_config`` svn repository. The minimum suggested svn revision is 58. See the documentation (http://www.mpe.mpg.de/~montefra/documentation/vhc/latest) for more information. .. _input: Input files =========== The main and only file ``vhc`` depends on for running is the configuration file described in :doc:`configuration`. When loading the configuration file, as described in :ref:`useit`, ``vhc`` checks the svn revision number from the ``revision`` section in the ``[svn]`` section. If it is not present, not in standard format (e.g. ``Revision: 42``, as produced by the `svn keyword substitution `_)), or has a smaller version than suggested, a warning is issued. Afterwards, if required, ``vhc`` checks that the ``path`` provided via the command line is a shot. The check is enabled if the ``shot_subdir`` of the ``[general]`` section is found and not empty. If a value is found, the code checks that it exists, as a directory or a file, below the input ``path``. If it's not found an exception is raised and ``vhc`` aborts. After loading the configuration file and setting up the logging system, ``vhc`` looks for two files in ``path``:: v_recipe.txt v_driver.txt Recipe file: ``v_recipe.txt`` ----------------------------- The recipe file describe the type of files that ``vhc`` is going to analyse. It consists of one single line, that is read and compared with the list of available recipes, loaded from the entry point group ``vhc.recipes`` (see :ref:`plug_recipe` for more information). VHC provides the following recipes:: flat twilight arc bias dark hetdex_dithers If the recipe file is not found, a new one is created. To find out the recipe to use , ``vhc`` looks for a file matching :data:`~libvhc.utils.generic_virus_match` and infers the corresponding name comparing it with the expected files names for the available recipes. If one file matches, the corresponding recipes is assumed, otherwise ``vhc`` will abort with a ``VHCRecipeError``. .. warning:: The recipe name is inferred on from the fist file found. Also the order at which plugins are loaded might not be always the same. If two recipes have the same file name templates, it would be better to explicitly create the ``v_recipe.txt`` file Driver file: ``v_driver.txt`` ----------------------------- The driver file ``v_driver.txt`` contains the names, each on one line, of the checks (drivers) to execute. To each name should correspond a plugin, advertised under the ``vhc.drivers`` entry point (see :ref:`plug_driver` for more information). If, during the execution, one of the drivers is not found, fails to load or to execute the error is logged and VHC continues. If the driver file is not found, a default set of drivers for the recipe at hand is used, unless a list of drivers is provided in the configuration file. After obtaining the list of drivers in one of the three above ways, a new driver file ``v_driver_{N}.txt`` is created. The counter ``N`` is increased every time ``vhc`` is run. The value of ``N`` is the same in the driver files and in the output files (see :ref:`output` for more information). Drivers from the configuration file ................................... Before looking for the driver files, ``vhc`` search the sections corresponding to the current recipe for the configuration options: * ``drivers``: a comma separated list of drivers to execute if the driver file is not found; if not present or empty, the default drivers are used; * ``override_driver_file``: if it is set to ``true`` or ``yes`` and ``drivers`` is given, then the drivers from the configuration are executed even if the ``v_driver.txt`` file is found. As example, see the following configuration extract: .. code-block:: ini [DEFAULT] override_driver_file = true [recipe1] drivers = driver1, driver2 [recipe2] drivers = [recipe3] drivers = driver1, driver3 override_driver_file = false This will set ``override_driver_file`` to ``true`` globally, except in ``[recipe3]``, where it gets overwritten. Then ``[recipe1]`` will always use the drivers ``driver1, driver2``, ``[recipe2]`` will use the driver file or the default drivers, while ``[recipe3]`` will use the drivers ``driver1, driver3`` unless the driver file is found. .. _vhc_drivers: Available drivers ................. With VHC we ship a number of drivers, listed the table below together to the recipes they are associated to by default. If the configuration option ``write_vhc_header_key`` in the ``[headerkeys]`` section is active, the builtin drivers also add/edit one header keyword in each of the fits files they analyse. The name of the keyword is given as last column of the table; the values that are assigned are either ``PASSED``, ``FAILED`` or ``NOTRUN``. Drivers in bold are implemented and the ones with the green tick mark are in the list of default drivers for the corresponding recipes. .. role:: green .. csv-table:: Builtin drivers :header: Driver, Recipe, Keyword :green:`✔` :func:`common:n_files `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCNFILE :green:`✔` :func:`common:n_pixels `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCNPIX :green:`✔` :func:`common:headerkeys `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCHEAD :green:`✔` :func:`common:check_overscan `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCOVSCN :green:`✔` :func:`common:exptime `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCEXPTM :green:`✔` :func:`common:dettemp `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCTEMP :green:`✔` :func:`common:bitstat `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCBITS :green:`✔` :func:`common:nullpix `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCPIXEL :green:`✔` :func:`common:repeat `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCREPT :green:`✔` :func:`common:saturation `, "arc, bias, dark, flat, twilight, hetdex_dithers", VHCSATUR :green:`✔` :func:`arc:find_peaks `, arc, VHCPEAKS :green:`✔` :func:`bias:compare `, "bias, dark", VHCBCMP :green:`✔` :func:`bias:flat `, "bias, dark", VHCBFLAT :green:`✔` :func:`flat:min_flux `, "flat, twilight", VHCMFLUX :green:`✔` :func:`flat:n_fibers `, "flat, twilight", VHCNFIB :green:`✔` :func:`flat:col_cte `, "flat, twilight", VHCCSCTE flat:check_throughput, "flat, twilight", VHCTHRUP flat:check_distortion, "flat, twilight", VHCDIST :green:`✔` :func:`hetdex_dithers:n_dithers `, hetdex_dithers , VHCNDITH :green:`✔` :func:`hetdex_dithers:sky_level `, hetdex_dithers , VHCSKYLV :green:`✔` :func:`hetdex_dither:col_cte `, hetdex_dither, VHCCSCTE :green:`✔` :func:`hetdex_dither:row_cte `, hetdex_dither, VHCRSCTE If the header keywords are written, the above drivers set also two extra keywords: * ``VHCCHECK``: this can be used by users and other codes to check whether ``VHC`` has been run on a file; * :data:`libvhc.utils.EXTRA_KEY`: save the version of VHC used for the drivers; if the ``write_vhc_config_revision`` option of the ``[headerkeys]`` configuration section is set to ``yes``/``true`` the revision number of the configuration file added after the VHC version. For more details see :doc:`custom_drivers`. .. _fplane_file: The fplane file --------------- A file essential for mapping IDs and for creating the :ref:`html_recap` is the fplane file. Its name is taken from the configuration file: .. code-block:: ini [fplane] fplane_file = ${general:fplane_dir}/fplane.txt The command line option ``-f/--fpane-file`` overrides the option above. The fplane files is used by most of the builtin drivers. Most of them uses it to convert from ``IFUSLOT`` id to ``IFUID`` and/or ``SPECID``. The ``common:n_files`` driver uses the fplane file to estimate the number of IFUs, and therefore files, expected. It is possible to exclude IFUs available to the drivers using the following configuration entry: .. code-block:: ini [fplane] # Comma separated list of IFUSLOT to ignore when running VHC # If empty or not present, no IFU is skipped skip_ifuslot = 000, 056, 066, 555 .. _output: Output ====== Each ``vhc`` run stores execution information and the results of the drivers in two text files and one html document. Log files --------- ``vhc`` logs into two files: * ``log_{N}.txt``: it contains all the log message starting from the :data:`libvhc.loggers.LOWER_LEVEL` level. The first three lines of each file contains the date, revision and author from the :doc:`configuration`. * ``v_results_{N}.txt``: log all the errors and critical messages; it contains all the failed drivers as errors messages and exceptions, possibly with tracebacks, as critical messages. * ``v_json_{N}.txt``: if the ``write_json`` option for the ``[general]`` configuration section is set to ``yes``/``true``, such file is also created and contains in each row a JSON representation of the messages present in the ``log_{N}.txt`` file. ``{N}`` is a counter that is incremented every time ``vhc`` is run and avoid overwriting or append to any existing files. .. _html_recap: The html recap file ------------------- For every ``vhc`` run, an html file recapping the drivers success or failure for every IFU is created. The html file name is ``{recipe}_recap_{N}.html``, where ``{recipe}`` is the name of the recipe at hand and ``{N}`` is the same counter as the log files. The upper part of the file recap the directory in which it has been run. Below it are five tabs: * Visualization: it shows the focal plane with the overall success/failure status for each IFU; * Error Logs: show the errors from the execution of VHC; * Full Logs: show the full logs. * Drivers: list of drivers run * Configuration version: the configuration files version After the html file is create it can be automatically opened in the default browser using the ``open_html`` configuration option: .. code-block:: ini [html] # automatically open the newly created html in the default browser open_html = yes Optionally, it is possible possible to have an extra copy of the html file saved in a different location. To do so uncomment and edit the ``copy_html_name`` configuration option. It is also possible to open automatically this file using the ``open_copy_html`` option: .. code-block:: ini [html] # (Optional) write the html recap to the given file. If this option is given and # not empty, check that the parent directory exists and then save the file in # "copy_html_name". Default: no copy made copy_html_name = /path/to/file.html # (Optional) automatically open the copied html in the default browser. Default: # False open_copy_html = no The ``copy_html_name`` can be useful in cases when one does not want to open a new browser window but to reload an existing one every time VHC is run. The visualization part of the html recap shows the focal plane as defined in :ref:`fplane_file`. If no further configuration is provided, the IFUSLOT ids listed in ``skip_ifuslot`` option are now shown in the html file. It is possible to provide a different list of IFUs to skip with the following option: .. code-block:: ini [fplane] # Comma separated list of IFUSLOT to skip in the HTML # If not present, it uses skip_ifuslot as the value, # If empty, no IFUSLOT is ignored skip_ifuslot_html = 555 # skip_ifuslot_html = ${skip_ifuslot} For the HTML it is also possible to mark some IFUs as ignored: this means that they are drawn like the others but with much lighter colors. This could be useful to have some IFUs for reference but marked as different from the others. A list of ignore IFUs can be given via the following configuration entry: .. code-block:: ini [fplane] # Comma separated list of IFUSLOT that are shown in the HTML but marked as # ignored. No test report is supposed to be sent to the IFUs marked here. # If an IFU is listed in ``skip_ifuslot_html`` and in ``ignore_ifuslot_html`` it # will not be shown: the former take priority on the latter. # If empty or not present, no IFU is marked as ignored. ignore_ifuslot_html = 000, 056, 066 ``vhc_plugins``: discover plugins ================================= To know which recipes, drivers and reference file parsers are installed on your system ``VHC`` installs the ``vhc_plugins`` executable. Running it lists all the known recipes, drivers and reference file parsers and reports when any of those cannot be properly loaded. For more information type ``vhc_plugins -h``:: usage: vhc_plugins [-h] [-V] [-d DESCRIPTION] [-w {all,recipes,drivers,parsers}] [-f] [-v] [-r] List and basic introspection of installed VHC plugins optional arguments: -h, --help show this help message and exit -V, --version show program's version number and exit -d DESCRIPTION, --description DESCRIPTION Add 'description' lines from the plugins doc-string (default: 0) -w {all,recipes,drivers,parsers}, --what {all,recipes,drivers,parsers} What to show (default: all) -f, --full-exception When the loading or the validation fails print the full exception instead of just the error message (default: False) -v, --validate Load the plugins through the default mechanism, which performs validation. The validation is done after printing the names and description (default: False) -r, --return-value If the validation is enabled, print the return value of the plugins. Available for the type 'recipes'. (default: False)