Defining Single Station Observations and Observation Metadata¶

Session Definition Files¶

Module that contains all of the relevant class to build up a representation of a session definition file as defined in MCS0030v5. The hierarchy of classes is:

Project - class that holds all of the information about the project (including
the observer) and one or more sessions. Technically, a SD file has only one session but this approach allows for the generation of multiple SD files from a single Project object.

Observer - class that hold the observer’s name and numeric ID

Session - class that holds all of the observations associated with a particular
DP output.

Observations - class that hold information about a particular observation. It
includes a variety of attributes that are used to convert human- readable inputs to SDF data values. The observation class is further subclasses into:

TBW - class for TBW observations

TBN - class for TBN observations

DRX - class for general DRX observation, with sub-classes: * Solar - class for solar tracking * Jovian - class for Jovian tracking

Stepped - class for stepped observations

BeamStep - class that holds the information about a particular step in a Stepped
Observation

All of the classes, except for Stepped and BeamStep, are complete and functional. In addition, most class contain ‘validate’ attribute functions that can be used to determine if the project/session/observation are valid or not given the constraints of the DP system.

In addition to providing the means for creating session definition files from scratch, this module also includes a simple parser for SD files.

Changed in version 2.0.0: Added support for astropy.time.Time and astropy.coordinates.Angle instances

Changed in version 1.0.0: Added the get_observation_start_stop() function. Renamed parse_timeString() to parse_time() parse_time() can now accept dates/times as timezone-aware datetime instances Observations can now be initialized with durations as timedelta instances Observations can now be initialized with RA/dec/az/alt as ephem.hours and ephem.degrees instances

Session Structure¶

lsl.common.sdf (DP-based stations) and lsl.common.sdfADP (ADP-based stations) provide means to represent a set of observations as Python objects. For each lsl.common.sdf.Project, there is:

An observer (lsl.common.sdf.Observer)

The project office comments (lsl.common.sdf.ProjectOffice)

A single session that defines the SDF (lsl.common.sdf.Session)

The session contains one or more observerions (lsl.common.sdf.Observation). Each observing mode supported by the LWA is sub-classed (see below).

class lsl.common.sdf.Project(observer, name, id, sessions=None, comments=None, project_office=None)¶

Class to hold all the information about a specific session for a project/proposal.

Changed in version 1.2.1: Added a new writeto() method to directly write the SDF to a file.

append(newSession)¶: Add a new Session to the list of sessions.

render(session=0, verbose=False)¶: Create a session definition file that corresponds to the specified session. Returns the SD file’s contents as a string.

update()¶: Update the various sessions that are part of this project.

validate(verbose=False)¶: Examine all of the sessions and all of their observations to check for validity. If everything is valid, return True. Otherwise, return False.

writeto(filename, session=0, verbose=False, overwrite=False)¶: Create a session definition file that corresponds to the specified session and write it to the provided filename.

class lsl.common.sdf.Observer(name, id, first=None, last=None)¶: Class to hold information about an observer.

class lsl.common.sdf.ProjectOffice(project=None, sessions=None, observations=None)¶: Class to hold comments from the LWA object office. This class isn’t really needed to create SD files, but it is helpful for parsing SD files.

class lsl.common.sdf.Session(name, id, observations=None, data_return_method='DRSU', comments=None, station=<LWAStation id='VL', name='LWA1', lat=34:04:08.0, long=-107:37:42.1, elev=2133.6, pressure=0.0, horizon=0:00:00.0, antennas=[...], interface=<LSLInterface backend='lsl.common.dp', mcs='lsl.common.mcs', sdf='lsl.common.sdf', metabundle='lsl.common.metabundle', sdm='lsl.common.sdm'>>)¶

Class to hold all of the observations in a session.

append(newObservation)¶: Add a new Observation to the list of observations.

set_mib_record_interval(component, interval)¶

Set the record interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:

-1 = use the MCS default interval
0 = never record the MIB entries (the entries are still updated, however)

set_mib_update_interval(component, interval)¶

Set the update interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:

-1 = use the MCS default interval
0 = request no updates to the MIB entries

property spectrometer_channels¶: Number of spectrometer channesl to output, 0 is disables.

property spectrometer_integration¶: Number of FFT windows per spectrometer integration, 0 to disable.

property spectrometer_metatag¶: Spectrometer polarization selection.

update()¶: Update the various observations in the session.

validate(verbose=False)¶: Examine all of the observations associated with the session to check for validity. If everything is valid, return True. Otherwise, return False.

class lsl.common.sdf.Observation(name, target, start, duration, mode, ra, dec, frequency1, frequency2, filter, gain=- 1, max_snr=False, comments=None)¶

Class to hold the specifics of an observations. It currently handles TBW, TBN, TRK_RADEC, TRK_SOL, TRK_JOV, and Stepped

Changed in version 1.0.0: Added support for RA/dec values as ephem.hours/ephem.degrees instances

property dec¶: Target dec. (J2000).

property duration¶: Duration in seconds.

estimate_bytes()¶: Place holder for functions that return the estimate size of the data set being defined by the observation.

property fixed_body¶: Place holder for functions that return ephem.Body objects (or None) that define the pointing center of the observation.

property frequency1¶: Tuning 1 frequency in Hz.

property frequency2¶: Tuning 2 frequency in Hz.

get_beam_type()¶: Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

property ra¶: Target RA (J2000).

property start¶: Start time.

property target_visibility¶: Place holder for functions that return the fractional visibility of the target during the observation period.

update()¶: Update the computed parameters from the string values.

validate(verbose=False)¶: Place holder for functions that evaluate the observation and return True if it is valid, False otherwise.

Observing Modes¶

class lsl.common.sdf.TBW(name, target, start, samples, bits=12, comments=None)¶

Sub-class of Observation specifically for TBW observations. It features a reduced number of parameters needed to setup the observation and provides extra information about the number of data bits and the number of samples.

Note

TBW read-out times in ms are calculated using (samples/196000+1)*5000 per MCS

Required Arguments:

observation name
observation target
observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)
integer number of samples

Optional Keywords:

bits - number of data bits (4 or 12)
comments - comments about the observation

estimate_bytes()¶: Estimate the data volume for the specified type and duration of observations. For TBW:

bytes = samples / samplesPerFrame * 1224 bytes * 260 stands

validate(verbose=False)¶: Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.TBN(name, target, start, duration, frequency, filter, gain=- 1, comments=None)¶

Sub-class of Observation specifically for TBN observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:

observation name
observation target
observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
observation duration (HH:MM:SS.SSS string or timedelta instance)
observation frequency (Hz)
integer filter code

Optional Keywords:

comments - comments about the observation

estimate_bytes()¶: Estimate the data volume for the specified type and duration of observations. For TBN:

bytes = duration * sample_rate / 512 * 1048 bytes * 260 stands * 2 pols.

validate(verbose=False)¶

Evaluate the observation and return True if it is valid, False otherwise.

..note::: This version of sdf allows for TBN tuning between 5 and 93 MHz.

class lsl.common.sdf.DRX(name, target, start, duration, ra, dec, frequency1, frequency2, filter, gain=- 1, max_snr=False, comments=None)¶

Sub-class of Observation specifically for DRX-style observations.

Required Arguments:

observation name
observation target
observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
observation duration (HH:MM:SS.SSS string or timedelta instance)
observation RA in hours, J2000.0 or ephem.hours instance
observation Dec in degrees, J2000.0 or ephem.hours instance
observation tuning frequency 1 (Hz)
observation tuning frequency 1 (Hz)
integer filter code

Optional Keywords:

max_snr - specifies if maximum signal-to-noise beam forming is to be used
(default = False)
comments - comments about the observation

estimate_bytes()¶: Estimate the data volume for the specified type and duration of observations. For DRX:

bytes = duration * sample_rate / 4096 * 4128 bytes * 2 tunings * 2 pols.

property fixed_body¶: Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is an all-sky mode.

set_beamdipole_mode(stand, beam_gain=0.04, dipole_gain=1.0, pol='X')¶

Convert the current observation to a ‘beam-dipole mode’ observation with the specified stand. Setting the stand to zero will disable the ‘beam-dipole mode’ for this observation’.

Keywords:

beam_gain - BAM gain to use for each dipole in the beam
default: 0.04; range: 0.0 to 1.0
dipole_gain - BAM gain to use for the single dipole
default: 1.0; range: 0.0 to 1.0
pol - Polarization to record default: “X”

property target_visibility¶: Return the fractional visibility of the target during the observation period.

validate(verbose=False)¶: Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.Solar(name, target, start, duration, frequency1, frequency2, filter, gain=- 1, max_snr=False, comments=None)¶

Sub-class of DRX specifically for Solar DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:

observation name
observation target
observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
observation duration (HH:MM:SS.SSS string or timedelta instance)
observation tuning frequency 1 (Hz)
observation tuning frequency 1 (Hz)
integer filter code

Optional Keywords:

max_snr - specifies if maximum signal-to-noise beam forming is to be used
(default = False)
comments - comments about the observation

property fixed_body¶: Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Jovian(name, target, start, duration, frequency1, frequency2, filter, gain=- 1, max_snr=False, comments=None)¶

Sub-class of DRX specifically for Jovian DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:

observation name
observation target
observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
observation duration (HH:MM:SS.SSS string or timedelta instance)
observation tuning frequency 1 (Hz)
observation tuning frequency 1 (Hz)
integer filter code

Optional Keywords:

max_snr - specifies if maximum signal-to-noise beam forming is to be used
(default = False)
comments - comments about the observation

property fixed_body¶: Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Stepped(name, target, start, filter, steps=None, is_radec=True, gain=- 1, comments=None)¶

Sub-class of Observation for dealing with STEPPED-mode observations. It features a reduced number of parameters needed to setup the observation and added support for the individual steps.

Required Arguments:

observation name
observation target
observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
integer filter code

Optional Keywords:

steps - array of BeamStep objects that specify the different steps
comments - comments about the observation

append(newStep)¶: Add a new BeamStep step to the list of steps.

property duration¶: Parse the list of BeamStep objects to get the total observation duration as the number of seconds in that period.

estimate_bytes()¶: Estimate the data volume for the specified type and duration of observations. For DRX:

bytes = duration * sample_rate / 4096 * 4128 bytes * 2 tunings * 2 pols.

set_beamdipole_mode(stand, beam_gain=0.04, dipole_gain=1.0, pol='X')¶

Convert the current observation to a ‘beam-dipole mode’ observation with the specified stand. Setting the stand to zero will disable the ‘beam-dipole mode’ for this observation’.

Keywords:

beam_gain - BAM gain to use for each dipole in the beam
default: 0.04; range: 0.0 to 1.0
dipole_gain - BAM gain to use for the single dipole
default: 1.0; range: 0.0 to 1.0
pol - Polarization to record default: “X”

property target_visibility¶: Return the fractional visibility of the target during the observation period.

update()¶: Update the computed parameters from the string values.

validate(verbose=False)¶: Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.BeamStep(c1, c2, duration, frequency1, frequency2, is_radec=True, max_snr=False, spec_delays=None, spec_gains=None)¶

Class for holding all of the information (pointing center, tuning frequencies, etc.)associated with a particular step.

Required Keywords:

pointing coordinate 1 (RA [hours] or azimuth [degrees] or ephem.hours/ephem.degrees instance)
pointing coordinate 2 (dec or elevation/altitude [degrees] or ephem.degrees instance)
observation duration (HH:MM:SS.SSS string or timedelta instance)
observation tuning frequency 1 (Hz)
observation tuning frequency 1 (Hz)

Optional Keywords:

is_radec - whether the coordinates are in RA/Dec or Az/El pairs (default=RA/Dec)
max_snr - specifies if maximum signal-to-noise beam forming is to be used
(default = False)
spec_delays - 520 list of delays to apply for each antenna
spec_gains - 260 by 2 by 2 list of gains ([[XY, XY], [YX, YY]]) to apply for each antenna

Note

If spec_delays is specified, spec_gains must also be specified. Specifying both spec_delays and spec_gains overrides the max_snr keyword.

Changed in version 1.0.0: Added support for azimuth/altitude and RA/dec values as ephem.hours/ephem.degrees instances

property c1¶: Coordinate 1 - hours (J2000) if RA, degrees if azimuth.

property c2¶: Coordinate 2 - degrees (J2000) if dec., degrees if elevation.

property duration¶: Duration in seconds.

property fixed_body¶: Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

property frequency1¶: Tuning 1 frequency in Hz.

property frequency2¶: Tuning 2 frequency in Hz.

get_beam_type()¶: Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

update()¶: Update the settings.

validate(verbose=False)¶: Evaluate the step and return True if it is valid, False otherwise.

MCS Metadata Tarball Utilities¶

Module for working with an MCS meta-data tarball and extracting the useful bits out it and putting those bits into Python objects, e.g, lsl.common.stations.LWAStation and lsl.common.sdm.SDM.

lsl.common.metabundle.get_asp_configuration(tarname, which='beginning')¶: Given an MCS meta-data tarball, extract the ASP MIB contained in it and return a dictionary of values for the filter, AT1, AT2, and ATSplit. The ‘which’ keyword is used to specify whether or not the configuration returned is at the beginning (default) or end of the session.

New in version 0.6.5.

lsl.common.metabundle.get_asp_configuration_summary(tarname, which='beginning')¶: Similar to get_asp_configuration, but returns only a single value for each of the four ASP paramters: filter, AT, AT2, and ATSplit. The values are based off the mode of the parameter.

New in version 0.6.5.

lsl.common.metabundle.get_command_script(tarname)¶: Given an MCS meta-data tarball, extract the command script and parse it. The commands are returned as a list of dictionaries (one dictionary per command).

lsl.common.metabundle.get_observation_spec(tarname, obs_id=None)¶: Given an MCS meta-data tarball, extract one or more observation specification file (MCS0030, Section 6) and return a list of dictionaries corresponding to each OBS file. If the obs_id keyword is set to a list of observation numbers, only observations matching the numbers in obs_id are returned.

lsl.common.metabundle.get_sdf(tarname)¶: Given an MCS meta-data tarball, extract the session specification file, the session meta-data file, and all observation specification files to build up a SDF-representation of the session.

Note

This function returns a full lsl.common.sdf.Project instance with the session in question stored under project.sessions[0] and the observations under project.sessions[0].observations.

lsl.common.metabundle.get_sdm(tarname)¶

Given an MCS meta-data tarball, extract the information stored in the dynamic/sdm.dat file and return a lsl.common.sdm.SDM instance describing the dynamic condition of the station.

If a sdm.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.get_session_metadata(tarname)¶: Given an MCS meta-data tarball, extract the session meta-data file (MCS0030, Section 7) and return a dictionary of observations that contain dictionaries of the OP_TAG (tag), DRSU Barcode (drsu), OBS_OUTCOME (outcome), and the MSG (msg).

Changed in version 0.6.5: Update to the new _metadata.txt format

lsl.common.metabundle.get_session_spec(tarname)¶: Given an MCS meta-data tarball, extract the session specification file (MCS0030, Section 5) and return a dictionary of parameters.

lsl.common.metabundle.get_station(tarname, apply_sdm=True)¶

Given an MCS meta-data tarball, extract the information stored in the ssmif.dat file and return a lsl.common.stations.LWAStation object. Optionally, update the lsl.common.stations.Antenna instances associated whith the LWAStation object using the included SDM file.

If a ssmif.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.is_valid(tarname, verbose=False)¶: Given a filename, see if it is valid metadata tarball or not.

New in version 1.2.0.

lsl.common.metabundle.read_cs_file(filename)¶: Read in a command script file (MCS0030, currently undocumented) and return the data as a list of dictionaries.

lsl.common.metabundle.read_obs_file(filename)¶: Read in a observation specification file (MCS0030, Section 6) and return the data as a dictionary.

lsl.common.metabundle.read_ses_file(filename)¶: Read in a session specification file (MCS0030, Section 5) and return the data as a dictionary.

Supporting Functions¶

Conversion to/from MJD and MPM¶

These functions convert Python datetime instances to modified Julian Data (MJD) and milliseconds past midnight (MPM) pairs.

lsl.common.mcs.datetime_to_mjdmpm(dt)¶

Convert a UTC datetime instance to a MJD, MPM pair (returned as a two-element tuple).

Based off: http://paste.lisp.org/display/73536

Changed in version 2.0.1: Better support for time zone-aware datetime instances

New in version 0.5.2.

lsl.common.mcs.mjdmpm_to_datetime(mjd, mpm, tz=None)¶: Convert a MJD, MPM pair to a naive datetime instance. If tz is not None the value is converted to a time zone-aware instance in the specified time zone.

Changed in version 2.0.1: Added the tz keyword and fixed the documentation.

New in version 0.5.2.

Specifiying Delay and Gains for the Digital Processor¶

These functions are intended to help define observations that are run in Stepped mode with the beamforming method set to “SPEC_DELAYS_GAINS”.

lsl.common.mcs.delay_to_mcsd(delay)¶: Given a delay in ns, convert it to a course and fine portion and into the form expected by MCS in a custom beamforming SDF (little endian 16.12 unsigned integer).

New in version 0.6.3.

lsl.common.mcs.mcsd_to_delay(delay)¶: Given delay value from an OBS_BEAM_DELAY field in a custom beamforming SDF, return the delay in ns.

New in version 0.6.3.

lsl.common.mcs.gain_to_mcsg(gain)¶: Given a gain (between 0 and 1), convert it to a gain in the form expected by MCS in a custom beamforming SDF (little endian 16.1 signed integer).

lsl.common.mcs.mcsg_to_gain(gain)¶: Given a gain value from an OBS_BEAM_GAIN field in a custom beamforming SDF, return the decimal equivalent.

New in version 0.6.3.

Interpretting MCS Numeric Codes¶

These functions convert various MCS numeric codes found in the metatdata into strings.

lsl.common.mcs.status_to_string(code)¶: Convert a numerical MCS status code to a string.

lsl.common.mcs.summary_to_string(code)¶: Convert a numerical MCS overall status code to an explination.

lsl.common.mcs.sid_to_string(sid)¶: Convert a MCS subsystem ID code into a string.

lsl.common.mcs.cid_to_string(cid)¶: Convert a MCS command code into a string.

lsl.common.mcs.mode_to_string(mode)¶: Convert a MCS numeric observing mode into a string.