Defining Single Station Observations and Observation Metadata

Session Definition Files

Module that contains all of the relevant class to build up a representation of a session definition file as defined in MCS0030v5. The hierarchy of classes is:

  • Project - class that holds all of the information about the project (including

    the observer) and one or more sessions. Technically, a SD file has only one session but this approach allows for the generation of multiple SD files from a single Project object.

  • Observer - class that hold the observer’s name and numeric ID

  • Session - class that holds all of the observations associated with a particular

    DP output.

  • Observations - class that hold information about a particular observation. It

    includes a variety of attributes that are used to convert human- readable inputs to SDF data values. The observation class is further subclasses into:

    • TBT - class for triggered transient buffer observations

    • TBS - class for streaming transient buffer observations

    • DRX - class for general DRX observation, with sub-classes: * Solar - class for solar tracking * Jovian - class for Jovian tracking * Lunar - class for Lunar tracking

    • Stepped - class for stepped observations

  • BeamStep - class that holds the information about a particular step in a Stepped

    Observation

All of the classes, except for Stepped and BeamStep, are complete and functional. In addition, most class contain ‘validate’ attribute functions that can be used to determine if the project/session/observation are valid or not given the constraints of the DP system.

In addition to providing the means for creating session definition files from scratch, this module also includes a simple parser for SD files.

Changed in version 2.0.0: Added support for astropy.time.Time and astropy.coordinates.Angle instances

Changed in version 1.0.0: Added the get_observation_start_stop() function. Renamed parse_timeString() to parse_time() parse_time() can now accept dates/times as timezone-aware datetime instances Observations can now be initialized with durations as timedelta instances Observations can now be initialized with RA/dec/az/alt as ephem.hours and ephem.degrees instances

Session Structure

lsl.common.sdf provides means to represent a set of observations as Python objects. For each lsl.common.sdf.Project, there is:

  1. An observer (lsl.common.sdf.Observer)

  2. The project office comments (lsl.common.sdf.ProjectOffice)

  3. A single session that defines the SDF (lsl.common.sdf.Session)

The session contains one or more observerions (lsl.common.sdf.Observation). Each observing mode supported by the LWA is sub-classed (see below).

class lsl.common.sdf.Project(observer, name, id, sessions=None, comments=None, project_office=None)

Class to hold all the information about a specific session for a project/proposal.

Changed in version 1.2.1: Added a new writeto() method to directly write the SDF to a file.

append(newSession)

Add a new Session to the list of sessions.

render(session=0)

Create a session definition file that corresponds to the specified session. Returns the SD file’s contents as a string.

update()

Update the various sessions that are part of this project.

validate()

Examine all of the sessions and all of their observations to check for validity. If everything is valid, return True. Otherwise, return False.

writeto(filename, session=0, overwrite=False)

Create a session definition file that corresponds to the specified session and write it to the provided filename.

class lsl.common.sdf.Observer(name, id, first=None, last=None)

Class to hold information about an observer.

class lsl.common.sdf.ProjectOffice(project=None, sessions=None, observations=None)

Class to hold comments from the LWA object office. This class isn’t really needed to create SD files, but it is helpful for parsing SD files.

class lsl.common.sdf.Session(name, id, observations=None, data_return_method='DRSU', comments=None, station=<LWAStation id='VL', name='LWA1', lat=34:04:08.0, long=-107:37:42.1,     elev=2133.6, pressure=0.0, horizon=0:00:00.0, antennas=[...],     interface=<LSLInterface backend='lsl.common.ndp',     mcs='lsl.common.mcs', sdf='lsl.common.sdf',     metabundle='lsl.common.metabundle', sdm='lsl.common.sdm'>>)

Class to hold all of the observations in a session.

append(newObservation)

Add a new Observation to the list of observations.

set_mib_record_interval(component, interval)

Set the record interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:
  • -1 = use the MCS default interval

  • 0 = never record the MIB entries (the entries are still updated, however)

set_mib_update_interval(component, interval)

Set the update interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:
  • -1 = use the MCS default interval

  • 0 = request no updates to the MIB entries

property spectrometer_channels

Number of spectrometer channesl to output, 0 is disables.

property spectrometer_integration

Number of FFT windows per spectrometer integration, 0 to disable.

property spectrometer_metatag

Spectrometer polarization selection.

update()

Update the various observations in the session.

validate()

Examine all of the observations associated with the session to check for validity. If everything is valid, return True. Otherwise, return False.

class lsl.common.sdf.Observation(name, target, start, duration, mode, ra, dec, frequency1, frequency2, filter, gain=-1, high_dr=False, comments=None)

Class to hold the specifics of an observations. It currently handles TBT, TBS, TRK_RADEC, TRK_SOL, TRK_JOV, TRK_LUN and Stepped

Changed in version 1.0.0: Added support for RA/dec values as ephem.hours/ephem.degrees instances

property dec

Target dec. (J2000).

property duration

Duration in seconds.

estimate_bytes()

Place holder for functions that return the estimate size of the data set being defined by the observation.

property fixed_body

Place holder for functions that return ephem.Body objects (or None) that define the pointing center of the observation.

property frequency1

Tuning 1 frequency in Hz.

property frequency2

Tuning 2 frequency in Hz.

get_beam_type()

Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

property ra

Target RA (J2000).

property start

Start time.

property target_visibility

Place holder for functions that return the fractional visibility of the target during the observation period.

update()

Update the computed parameters from the string values.

validate()

Place holder for functions that evaluate the observation and return True if it is valid, False otherwise.

Observing Modes

class lsl.common.sdf.TBT(name, target, start, samples, comments=None)

Sub-class of Observation specifically for TBT observations. It features a reduced number of parameters needed to setup the observation and provides extra information about the number of data bits and the number of samples.

Note

TBT read-out times in ms are calculated using ((samples/196000 + 1)*150 + 5000) per MCS (tpss.c), where samples are in sampler-clock ticks, 196000 is ticks/ms, 150 is the inverse duty cycle, and 5000 ms is the buffer fill lag.

Required Arguments:
  • observation name

  • observation target

  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)

  • integer number of samples

Optional Keywords:
  • comments - comments about the observation

estimate_bytes()

Estimate the data volume for the specified type and duration of observations. For a full-band TBT dump:

bytes = (samples / ticksPerWindow) * (nchan / chanPerFrame)
  • (28 + chanPerFrame * nstand * npol)

where samples is in sampler-clock ticks, ticksPerWindow is the F-engine FFT size (8192), chanPerFrame is 16, and a full station has up to 3072 channels, 256 stands, and 2 pols.

validate()

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.TBS(name, target, start, duration, frequency, filter, comments=None)

Sub-class of Observation specifically for TBS observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:
  • observation name

  • observation target

  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)

  • observation duration (HH:MM:SS.SSS string or timedelta instance)

  • observation frequency (Hz)

  • integer filter code

Optional Keywords:
  • comments - comments about the observation

estimate_bytes()

Estimate the data volume for the specified type and duration of observations. For TBS:

bytes = duration * sample_rate / 512 * 1048 bytes * 256 stands * 2 pols.

validate()

Evaluate the observation and return True if it is valid, False otherwise.

Note

This version of sdf allows for TBS tuning between 5 and 93 MHz.

class lsl.common.sdf.DRX(name, target, start, duration, ra, dec, frequency1, frequency2, filter, gain=-1, high_dr=False, comments=None)

Sub-class of Observation specifically for DRX-style observations.

Required Arguments:
  • observation name

  • observation target

  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)

  • observation duration (HH:MM:SS.SSS string or timedelta instance)

  • observation RA in hours, J2000.0 or ephem.hours instance

  • observation Dec in degrees, J2000.0 or ephem.hours instance

  • observation tuning frequency 1 (Hz)

  • observation tuning frequency 1 (Hz)

  • integer filter code

Optional Keywords:
  • high_dr - specifies if a high dynamic range beam output is to be used

    (default = False)

  • comments - comments about the observation

estimate_bytes()

Estimate the data volume for the specified type and duration of observations. For DRX:

bytes = duration * sample_rate / 4096 * 4128 bytes * 2 tunings * 2 pols.

property fixed_body

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is an all-sky mode.

set_beamdipole_mode(stand, beam_gain=0.04, dipole_gain=1.0, pol='X')

Convert the current observation to a ‘beam-dipole mode’ observation with the specified stand. Setting the stand to zero will disable the ‘beam-dipole mode’ for this observation’.

Keywords:
  • beam_gain - BAM gain to use for each dipole in the beam

    default: 0.04; range: 0.0 to 1.0

  • dipole_gain - BAM gain to use for the single dipole

    default: 1.0; range: 0.0 to 1.0

  • pol - Polarization to record default: “X”

property target_visibility

Return the fractional visibility of the target during the observation period.

validate()

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.Solar(name, target, start, duration, frequency1, frequency2, filter, gain=-1, high_dr=False, comments=None)

Sub-class of DRX specifically for Solar DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:
  • observation name

  • observation target

  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)

  • observation duration (HH:MM:SS.SSS string or timedelta instance)

  • observation tuning frequency 1 (Hz)

  • observation tuning frequency 1 (Hz)

  • integer filter code

Optional Keywords:
  • high_dr - specifies if a high dynamic range beam output is to be used

    (default = False)

  • comments - comments about the observation

property fixed_body

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Jovian(name, target, start, duration, frequency1, frequency2, filter, gain=-1, high_dr=False, comments=None)

Sub-class of DRX specifically for Jovian DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:
  • observation name

  • observation target

  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)

  • observation duration (HH:MM:SS.SSS string or timedelta instance)

  • observation tuning frequency 1 (Hz)

  • observation tuning frequency 1 (Hz)

  • integer filter code

Optional Keywords:
  • high_dr - specifies if a high dynamic range beam output is to be used

    (default = False)

  • comments - comments about the observation

property fixed_body

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Stepped(name, target, start, filter, steps=None, is_radec=True, gain=-1, comments=None)

Sub-class of Observation for dealing with STEPPED-mode observations. It features a reduced number of parameters needed to setup the observation and added support for the individual steps.

Required Arguments:
  • observation name

  • observation target

  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)

  • integer filter code

Optional Keywords:
  • steps - array of BeamStep objects that specify the different steps

  • comments - comments about the observation

append(newStep)

Add a new BeamStep step to the list of steps.

property duration

Parse the list of BeamStep objects to get the total observation duration as the number of seconds in that period.

estimate_bytes()

Estimate the data volume for the specified type and duration of observations. For DRX:

bytes = duration * sample_rate / 4096 * 4128 bytes * 2 tunings * 2 pols.

set_beamdipole_mode(stand, beam_gain=0.04, dipole_gain=1.0, pol='X')

Convert the current observation to a ‘beam-dipole mode’ observation with the specified stand. Setting the stand to zero will disable the ‘beam-dipole mode’ for this observation’.

Keywords:
  • beam_gain - BAM gain to use for each dipole in the beam

    default: 0.04; range: 0.0 to 1.0

  • dipole_gain - BAM gain to use for the single dipole

    default: 1.0; range: 0.0 to 1.0

  • pol - Polarization to record default: “X”

property target_visibility

Return the fractional visibility of the target during the observation period.

update()

Update the computed parameters from the string values.

validate()

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.BeamStep(c1, c2, duration, frequency1, frequency2, is_radec=True, high_dr=False, spec_delays=None, spec_gains=None)

Class for holding all of the information (pointing center, tuning frequencies, etc.)associated with a particular step.

Required Keywords:
  • pointing coordinate 1 (RA [hours] or azimuth [degrees] or ephem.hours/ephem.degrees instance)

  • pointing coordinate 2 (dec or altitude [degrees] or ephem.degrees instance)

  • observation duration (HH:MM:SS.SSS string or timedelta instance)

  • observation tuning frequency 1 (Hz)

  • observation tuning frequency 1 (Hz)

Optional Keywords:
  • is_radec - whether the coordinates are in RA/Dec or Az/El pairs (default=RA/Dec)

  • high_dr - specifies if a high dynamic range beam output is to be used

    (default = False)

  • spec_delays - 520 list of delays to apply for each antenna

  • spec_gains - 260 by 2 by 2 list of gains ([[XY, XY], [YX, YY]]) to apply for each antenna

Note

If spec_delays is specified, spec_gains must also be specified. Specifying both spec_delays and spec_gains overrides the high_dr keyword.

Changed in version 1.0.0: Added support for azimuth/altitude and RA/dec values as ephem.hours/ephem.degrees instances

property c1

Coordinate 1 - hours (J2000) if RA, degrees if azimuth.

property c2

Coordinate 2 - degrees (J2000) if dec., degrees if altitude.

property duration

Duration in seconds.

property fixed_body

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

property frequency1

Tuning 1 frequency in Hz.

property frequency2

Tuning 2 frequency in Hz.

get_beam_type()

Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

update()

Update the settings.

validate()

Evaluate the step and return True if it is valid, False otherwise.

MCS Metadata Tarball Utilities

Module for working with an MCS meta-data tarball and extracting the useful bits out it and putting those bits into Python objects, e.g, lsl.common.stations.LWAStation and lsl.common.sdm.SDM.

lsl.common.metabundle.get_asp_configuration(tarname, which='beginning')

Given an MCS meta-data tarball, extract the ASP MIB contained in it and return a dictionary of values for the filter, AT1, AT2, and AT3. The ‘which’ keyword is used to specify whether or not the configuration returned is at the beginning (default) or end of the session.

New in version 0.6.5.

lsl.common.metabundle.get_asp_configuration_summary(tarname, which='beginning')

Similar to get_asp_configuration, but returns only a single value for each of the four ASP paramters: filter, AT, AT2, and AT3. The values are based off the mode of the parameter.

New in version 0.6.5.

lsl.common.metabundle.get_beamformer_min_delay(tarname)

Given an MCS meta-data tarball, extract the minimum beamformer delay in samples and return it. If no minimum delay can be found in the tarball, None is returned.

lsl.common.metabundle.get_command_script(tarname)

Given an MCS meta-data tarball, extract the command script and parse it. The commands are returned as a list of dictionaries (one dictionary per command).

lsl.common.metabundle.get_mcs_hostname(tarname)

Given an MCS meta-data tarball, extract the information stored in the mcs.host file and return it as a string.

If a mcs.host file cannot be found in the tarball, ‘unknown’ is returned.

lsl.common.metabundle.get_observation_spec(tarname, obs_id=None)

Given an MCS meta-data tarball, extract one or more observation specification file (MCS0030, Section 6) and return a list of dictionaries corresponding to each OBS file. If the obs_id keyword is set to a list of observation numbers, only observations matching the numbers in obs_id are returned.

lsl.common.metabundle.get_sdf(tarname)

Given an MCS meta-data tarball, extract the session specification file, the session meta-data file, and all observation specification files to build up a SDF-representation of the session.

Note

This function returns a full lsl.common.sdf.Project instance with the session in question stored under project.sessions[0] and the observations under project.sessions[0].observations.

lsl.common.metabundle.get_sdm(tarname)

Given an MCS meta-data tarball, extract the information stored in the dynamic/sdm.dat file and return a lsl.common.sdm.SDM instance describing the dynamic condition of the station.

If a sdm.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.get_session_metadata(tarname)

Given an MCS meta-data tarball, extract the session meta-data file (MCS0030, Section 7) and return a dictionary of observations that contain dictionaries of the OP_TAG (tag), DRSU Barcode (drsu), OBS_OUTCOME (outcome), and the MSG (msg).

Changed in version 0.6.5: Update to the new _metadata.txt format

lsl.common.metabundle.get_session_spec(tarname)

Given an MCS meta-data tarball, extract the session specification file (MCS0030, Section 5) and return a dictionary of parameters.

lsl.common.metabundle.get_station(tarname, apply_sdm=True)

Given an MCS meta-data tarball, extract the information stored in the ssmif.dat file and return a lsl.common.stations.LWAStation object. Optionally, update the lsl.common.stations.Antenna instances associated whith the LWAStation object using the included SDM file.

If a ssmif.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.is_valid(tarname)

Given a filename, see if it is valid metadata tarball or not.

New in version 1.2.0.

lsl.common.metabundle.read_cs_file(filename)

Read in a command script file (MCS0030, currently undocumented) and return the data as a list of dictionaries.

lsl.common.metabundle.read_obs_file(filename)

Read in a observation specification file (MCS0030, Section 6) and return the data as a dictionary.

lsl.common.metabundle.read_ses_file(filename)

Read in a session specification file (MCS0030, Section 5) and return the data as a dictionary.

MCS Supporting Functions

Module that contains common values found in the MCS Joint Release 5 header file src/exec/me.h and other functions useful for working with the MCS metadata. The header file values are:

  • ME_SSMIF_FORMAT_VERSION - SSMIF format version code

  • ME_MAX_NSTD - Maximum number of stands that can be described

  • ME_MAX_NFEE - Maximum number of FEEs that can be described

  • ME_MAX_FEEID_LENGTH - Number of characters in FEE ID name

  • ME_MAX_RACK - Maximum number of racks?

  • ME_MAX_PORT - Maximum number of ports?

  • ME_MAX_NRPD - Maxmimum number of RPD cables

  • ME_MAX_RPDID_LENGTH - Number of characters in the RPD ID name

  • ME_MAX_NSEP - Maximum number of SEP cable connections

  • ME_MAX_SEPID_LENGTH - Number of characters in the SEP ID name

  • ME_MAX_SEPCABL_LENGTH - Number of characters in the SEP cable ID name

  • ME_MAX_NARB - Maximum number of ARX boards

  • ME_MAX_NARBCH - Number of ARX channels per board

  • ME_MAX_ARBID_LENGTH - Number of characters in the ARX ID name

  • ME_MAX_NSNAP - Maximum number of SNAP boards

  • ME_MAX_NSNAPCH - Number of channels per SNAP board

  • ME_MAX_SNAPID_LENGTH - Number of characters in the SNAP board ID name

  • ME_MAX_NSERVER - Maximum number of servers

  • ME_MAX_SERVERID_LENGTH - Number of characters in the server ID name

  • ME_MAX_NDR - Maximum number of data recorders

  • ME_MAX_DRID_LENGTH - Number of characters in the DR ID name

  • ME_MAX_NDPOUT - Number of NDP data outputs

  • ME_MAX_NPWRPORT - Maximum number of power ports

  • ME_MAX_SSNAME_LENGTH - Number of characters in the power port ID names, for

    codes used for PWR_NAME

  • LWA_MAX_NSTD - Maximum number of stands for the LWA

  • MIB_REC_TYPE_BRANCH - eType for MIB branch entries

  • MIB_REC_TYPE_VALUE - etype for MIB value entries

  • MIB_INDEX_FIELD_LENGTH - Number of characters in a MIB index field

  • MIB_LABEL_FIELD_LENGTH - Number of characters in a MIB label field

  • MIB_VAL_FIELD_LENGTH - Number of characters in a MIB value field

  • SSMIF_STRUCT - String representing the C structure of the binary SSMIF

The other functions:
  • Parse the binary packed metadata,

class lsl.common.mcs.CommandID(value)

MCS Command IDs.

class lsl.common.mcs.MIB

Class to represent an entire MCS MIB.

New in version 0.6.5.

classmethod from_file(filename, init_filename=None)

Given the name of a GDBM database file, initialize the MIB. Optionally, use the name of the MCS MIB initialization file to help convert indicies to names.

keys(name=False)

Return a list of entry indicies (or names if the ‘name’ keyword is set to True) for the MIB.

parse_init_file(filename)

Given a MCS MIB initialization file, i.e., ASP_MIB_init.dat, create a dictionary that maps indicies to human-readable names that can be used to clarify a MIBEntry. Return this dictionary.

class lsl.common.mcs.MIBEntry

Class for accessing and storing MCS MIB information contained in a GDBM database.

New in version 0.6.5.

classmethod from_entry(value)

Given an MIB entry straight out of a GDBM database, populate the MIBEntry instance.

Note

The MIBEntry class does not currently support branch entries. Entries of this type will raise a ValueError.

Note

The MIBEntry class currently only support entries with indicies that start with a number. All other indicies will raise a ValueError.

class lsl.common.mcs.ObservingMode(value)

MCS Observing Modes.

class lsl.common.mcs.StatusCode(value)

MCS component status code.

class lsl.common.mcs.SubsystemID(value)

MCS subsystem IDs.

class lsl.common.mcs.SummaryCode(value)

MCS subsystem summary codes.

lsl.common.mcs.apply_pointing_correction(az, el, theta, phi, psi, lat=34.07, degrees=True)

Given a azimuth and elevation pair, and an axis to rotate about, perform the rotation.

http://en.wikipedia.org/wiki/Rotation_matrix#Rotation_matrix_from_axis_and_angle

lsl.common.mcs.datetime_to_mjdmpm(dt)

Convert a UTC datetime instance to a MJD, MPM pair (returned as a two-element tuple).

Based off: http://paste.lisp.org/display/73536

Changed in version 2.0.1: Better support for time zone-aware datetime instances

New in version 0.5.2.

lsl.common.mcs.delay_to_mcsd(delay)

Given a delay in ns, convert it to a course and fine portion and into the form expected by MCS in a custom beamforming SDF (little endian 16.12 unsigned integer).

New in version 0.6.3.

lsl.common.mcs.flat_to_multi(inputList, *shape)

Convert a 1-D list into a multi-dimension list of shape ‘shape’.

Changed in version 3.0.0: Switch from explicit ‘dim1’, ‘dim2’, etc. to a catch-all ‘shape’.

lsl.common.mcs.gain_to_mcsg(gain)

Given a gain (between 0 and 1), convert it to a gain in the form expected by MCS in a custom beamforming SDF (little endian 16.1 signed integer).

lsl.common.mcs.mcsd_to_delay(delay)

Given delay value from an OBS_BEAM_DELAY field in a custom beamforming SDF, return the delay in ns.

New in version 0.6.3.

lsl.common.mcs.mcsg_to_gain(gain)

Given a gain value from an OBS_BEAM_GAIN field in a custom beamforming SDF, return the decimal equivalent.

New in version 0.6.3.

lsl.common.mcs.mjdmpm_to_datetime(mjd, mpm, tz=None)

Convert a MJD, MPM pair to a timezone-aware datetime instance in UTC. If tz is not None the value is converted to the specified time zone.

Changed in version 2.0.1: Added the tz keyword and fixed the documentation.

New in version 0.5.2.

lsl.common.mcs.parse_c_struct(cStruct, char_mode='str', endianness='native', overrides=None)

Function to take a C structure declaration and build a ctypes.Structure out of it with the appropriate alignment, character interpretation*, and endianness (little, big, network, or native).

Note

ctypes converts character arrays to Python strings until the first null is

incountered. This behavior causes problems for multi-dimension arrays of null filled strings. By setting char_mode to ‘int’, all char types are retuned as bytes which can be converted to strings via chr().