Use VHC¶

Usage¶

Run vhc issuing the command:

vhc path [subdir] [extra [extra ...]]

where path is the directory containing the data vhc should run on and where the output files are located.

The drivers shipped togeter with VHC interpret the second argument, subdir, as the subdirectory of path from which the files must be collected. Any extra argument is ignored. As example consider the following directory structure. The following command:

vhc 20120301/virus0003000

collects all the FITS files in all the exp*/virus directories. Instead:

vhc 20120301/virus0003000 exp02

collects only the FITS files in the exp02/virus directory.

Third party drivers might interpret subdir differently and make use of the extra arguments.

For more info type vhc -h:

usage: vhc [-h] [-V] [-c CONFIG] [-f FPLANE_FILE]
           path [subdir] [extra [extra ...]]

Virus Health Check (VHC): check that Virus files contained into the given
directory are fine

positional arguments:
path                  Path containing the files to explore
subdir                For the drivers shipped with VHC: the files will be
                      collected only from the ``subdir`` subdirectory of
                      ``path``. Third party drivers might interpret this
                      argument as the first part of ``extra``. (default:
                      None)
extra                 Any extra arguments; ignored by the drivers shipped
                      with VHC (default: None)

optional arguments:
-h, --help            show this help message and exit
-V, --version         show program's version number and exit
-c CONFIG, --config CONFIG
                      Configuration file. If provided, overrides the search
                      for it in the default positions. (default: None)

Override options in the [fplane] section:
-f FPLANE_FILE, --fplane-file FPLANE_FILE
                      Override the fplane file name with the given option
                      (default: None)

The official configuration file can be found in the ``vhc_config`` svn
repository. The minimum suggested svn revision is 58. See the documentation
(http://www.mpe.mpg.de/~montefra/documentation/vhc/latest) for more
information.

Input files¶

The main and only file vhc depends on for running is the configuration file described in VHC configuration.

When loading the configuration file, as described in Use it, vhc checks the svn revision number from the revision section in the [svn] section. If it is not present, not in standard format (e.g. Revision: 42, as produced by the svn keyword substitution)), or has a smaller version than suggested, a warning is issued.

Afterwards, if required, vhc checks that the path provided via the command line is a shot. The check is enabled if the shot_subdir of the [general] section is found and not empty. If a value is found, the code checks that it exists, as a directory or a file, below the input path. If it’s not found an exception is raised and vhc aborts.

After loading the configuration file and setting up the logging system, vhc looks for two files in path:

v_recipe.txt
v_driver.txt

Recipe file: `v_recipe.txt`¶

The recipe file describe the type of files that vhc is going to analyse. It consists of one single line, that is read and compared with the list of available recipes, loaded from the entry point group vhc.recipes (see Plug new recipes for more information).

VHC provides the following recipes:

flat
twilight
arc
bias
dark
hetdex_dithers

If the recipe file is not found, a new one is created. To find out the recipe to use , vhc looks for a file matching generic_virus_match and infers the corresponding name comparing it with the expected files names for the available recipes. If one file matches, the corresponding recipes is assumed, otherwise vhc will abort with a VHCRecipeError.

Warning

The recipe name is inferred on from the fist file found. Also the order at which plugins are loaded might not be always the same. If two recipes have the same file name templates, it would be better to explicitly create the v_recipe.txt file

Driver file: `v_driver.txt`¶

The driver file v_driver.txt contains the names, each on one line, of the checks (drivers) to execute. To each name should correspond a plugin, advertised under the vhc.drivers entry point (see Plug new drivers for more information). If, during the execution, one of the drivers is not found, fails to load or to execute the error is logged and VHC continues.

If the driver file is not found, a default set of drivers for the recipe at hand is used, unless a list of drivers is provided in the configuration file. After obtaining the list of drivers in one of the three above ways, a new driver file v_driver_{N}.txt is created. The counter N is increased every time vhc is run. The value of N is the same in the driver files and in the output files (see Output for more information).

Drivers from the configuration file¶

Before looking for the driver files, vhc search the sections corresponding to the current recipe for the configuration options:

drivers: a comma separated list of drivers to execute if the driver file is not found; if not present or empty, the default drivers are used;
override_driver_file: if it is set to true or yes and drivers is given, then the drivers from the configuration are executed even if the v_driver.txt file is found.

As example, see the following configuration extract:

[DEFAULT]
override_driver_file = true

[recipe1]
drivers = driver1, driver2

[recipe2]
drivers =

[recipe3]
drivers = driver1, driver3
override_driver_file = false

This will set override_driver_file to true globally, except in [recipe3], where it gets overwritten. Then [recipe1] will always use the drivers driver1, driver2, [recipe2] will use the driver file or the default drivers, while [recipe3] will use the drivers driver1, driver3 unless the driver file is found.

Available drivers¶

With VHC we ship a number of drivers, listed the table below together to the recipes they are associated to by default. If the configuration option write_vhc_header_key in the [headerkeys] section is active, the builtin drivers also add/edit one header keyword in each of the fits files they analyse. The name of the keyword is given as last column of the table; the values that are assigned are either PASSED, FAILED or NOTRUN. Drivers in bold are implemented and the ones with the green tick mark are in the list of default drivers for the corresponding recipes.

Builtin drivers¶
Driver	Recipe	Keyword
✔ `common:n_files`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCNFILE
✔ `common:n_pixels`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCNPIX
✔ `common:headerkeys`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCHEAD
✔ `common:check_overscan`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCOVSCN
✔ `common:exptime`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCEXPTM
✔ `common:dettemp`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCTEMP
✔ `common:bitstat`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCBITS
✔ `common:nullpix`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCPIXEL
✔ `common:repeat`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCREPT
✔ `common:saturation`	arc, bias, dark, flat, twilight, hetdex_dithers	VHCSATUR
✔ `arc:find_peaks`	arc	VHCPEAKS
✔ `bias:compare`	bias, dark	VHCBCMP
✔ `bias:flat`	bias, dark	VHCBFLAT
✔ `flat:min_flux`	flat, twilight	VHCMFLUX
✔ `flat:n_fibers`	flat, twilight	VHCNFIB
✔ `flat:col_cte`	flat, twilight	VHCCSCTE
flat:check_throughput	flat, twilight	VHCTHRUP
flat:check_distortion	flat, twilight	VHCDIST
✔ `hetdex_dithers:n_dithers`	hetdex_dithers	VHCNDITH
✔ `hetdex_dithers:sky_level`	hetdex_dithers	VHCSKYLV
✔ `hetdex_dither:col_cte`	hetdex_dither	VHCCSCTE
✔ `hetdex_dither:row_cte`	hetdex_dither	VHCRSCTE

If the header keywords are written, the above drivers set also two extra keywords:

VHCCHECK: this can be used by users and other codes to check whether VHC has been run on a file;
libvhc.utils.EXTRA_KEY: save the version of VHC used for the drivers; if the write_vhc_config_revision option of the [headerkeys] configuration section is set to yes/true the revision number of the configuration file added after the VHC version.

For more details see Create and use a drivers.

The fplane file¶

A file essential for mapping IDs and for creating the The html recap file is the fplane file. Its name is taken from the configuration file:

[fplane]
fplane_file = ${general:fplane_dir}/fplane.txt

The command line option -f/--fpane-file overrides the option above.

The fplane files is used by most of the builtin drivers. Most of them uses it to convert from IFUSLOT id to IFUID and/or SPECID. The common:n_files driver uses the fplane file to estimate the number of IFUs, and therefore files, expected. It is possible to exclude IFUs available to the drivers using the following configuration entry:

[fplane]
# Comma separated list of IFUSLOT to ignore when running VHC
# If empty or not present, no IFU is skipped
skip_ifuslot = 000, 056, 066, 555

Output¶

Each vhc run stores execution information and the results of the drivers in two text files and one html document.

Log files¶

vhc logs into two files:

log_{N}.txt: it contains all the log message starting from the libvhc.loggers.LOWER_LEVEL level. The first three lines of each file contains the date, revision and author from the VHC configuration.
v_results_{N}.txt: log all the errors and critical messages; it contains all the failed drivers as errors messages and exceptions, possibly with tracebacks, as critical messages.
v_json_{N}.txt: if the write_json option for the [general] configuration section is set to yes/true, such file is also created and contains in each row a JSON representation of the messages present in the log_{N}.txt file.

{N} is a counter that is incremented every time vhc is run and avoid overwriting or append to any existing files.

The html recap file¶

For every vhc run, an html file recapping the drivers success or failure for every IFU is created. The html file name is {recipe}_recap_{N}.html, where {recipe} is the name of the recipe at hand and {N} is the same counter as the log files.

The upper part of the file recap the directory in which it has been run. Below it are five tabs:

Visualization: it shows the focal plane with the overall success/failure status for each IFU;
Error Logs: show the errors from the execution of VHC;
Full Logs: show the full logs.
Drivers: list of drivers run
Configuration version: the configuration files version

After the html file is create it can be automatically opened in the default browser using the open_html configuration option:

[html]

# automatically open the newly created html in the default browser
open_html = yes

Optionally, it is possible possible to have an extra copy of the html file saved in a different location. To do so uncomment and edit the copy_html_name configuration option. It is also possible to open automatically this file using the open_copy_html option:

[html]

# (Optional) write the html recap to the given file. If this option is given and
# not empty, check that the parent directory exists and then save the file in
# "copy_html_name". Default: no copy made
copy_html_name = /path/to/file.html
# (Optional) automatically open the copied html in the default browser. Default:
# False
open_copy_html = no

The copy_html_name can be useful in cases when one does not want to open a new browser window but to reload an existing one every time VHC is run.

The visualization part of the html recap shows the focal plane as defined in The fplane file. If no further configuration is provided, the IFUSLOT ids listed in skip_ifuslot option are now shown in the html file. It is possible to provide a different list of IFUs to skip with the following option:

[fplane]
# Comma separated list of IFUSLOT to skip in the HTML
# If not present, it uses skip_ifuslot as the value,
# If empty, no IFUSLOT is ignored
skip_ifuslot_html = 555
# skip_ifuslot_html = ${skip_ifuslot}

For the HTML it is also possible to mark some IFUs as ignored: this means that they are drawn like the others but with much lighter colors. This could be useful to have some IFUs for reference but marked as different from the others. A list of ignore IFUs can be given via the following configuration entry:

[fplane]
# Comma separated list of IFUSLOT that are shown in the HTML but marked as
# ignored. No test report is supposed to be sent to the IFUs marked here.
# If an IFU is listed in ``skip_ifuslot_html`` and in ``ignore_ifuslot_html`` it
# will not be shown: the former take priority on the latter.
# If empty or not present, no IFU is marked as ignored.
ignore_ifuslot_html = 000, 056, 066

`vhc_plugins`: discover plugins¶

To know which recipes, drivers and reference file parsers are installed on your system VHC installs the vhc_plugins executable. Running it lists all the known recipes, drivers and reference file parsers and reports when any of those cannot be properly loaded. For more information type vhc_plugins -h:

usage: vhc_plugins [-h] [-V] [-d DESCRIPTION]
                [-w {all,recipes,drivers,parsers}] [-f] [-v] [-r]

List and basic introspection of installed VHC plugins

optional arguments:
-h, --help            show this help message and exit
-V, --version         show program's version number and exit
-d DESCRIPTION, --description DESCRIPTION
                        Add 'description' lines from the plugins doc-string
                        (default: 0)
-w {all,recipes,drivers,parsers}, --what {all,recipes,drivers,parsers}
                        What to show (default: all)
-f, --full-exception  When the loading or the validation fails print the
                        full exception instead of just the error message
                        (default: False)
-v, --validate        Load the plugins through the default mechanism, which
                        performs validation. The validation is done after
                        printing the names and description (default: False)
-r, --return-value    If the validation is enabled, print the return value
                        of the plugins. Available for the type 'recipes'.
                        (default: False)