Data Readers¶

LSL Developer Primitives¶

LWA Development Primitives - A set of utilities that should make developing new analysis software easier. These functions wrap the nitty gritty of the file reading and unpacking behind Python objects.

Data format objects included are:

TBWFile
TBNFile
DRXFile
DRSpecFile
TBFFile
CORFILE

Also included are the LWA1DataFile, LWASVDataFile, and LWADataFile functions that take a filename and try to determine the correct data format object to use.

Changed in version 1.2.0: Added support for LWA-SV ADP data

class lsl.reader.ldp.DRSpecFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶

Class to make it easy to interface with a DR Spectrometer file. Methods defined for this class are:

get_info - Get information about the file’s contents

get_remaining_frame_count - Get the number of frames remaining in the file

offset - Offset a specified number of seconds into the file

read_frame - Read and return a single lsl.reader.drspec.Frame instance

read - Read a chunk of data in and return it as a numpy array

offset(offset)¶: Offset a specified number of seconds in an open DR spectrometer file. This function returns the exact offset time.

read(duration, time_in_samples=False)¶

Given an open DR spectrometer file and an amount of data read in in seconds, read in the data and return a three-element tuple of the actual duration read in, the times at the beginning of each stream, and the data as numpy array.

The time tag is returned as seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance by default. However, the time tags can be returns as samples at lsl.common.dp.fS if the `time_in_samples’ keyword is set.

..note::: This function always returns a 3-D array with the first dimension indexing over data product, the second over time and the third over frequency channel.

read_frame()¶: Read and return a single lsl.reader.drspec.Frame instance.

class lsl.reader.ldp.DRXFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶

Class to make it easy to interface with a DRX file. Methods defined for this class are:

get_info - Get information about the file’s contents
get_remaining_frame_count - Get the number of frames remaining in the file
offset - Offset a specified number of seconds into the file
read_frame - Read and return a single lsl.reader.drx.Frame instance
read - Read a chunk of data in and return it as a numpy array
estimate_levels - Estimate the n-sigma level for the absolute value of the voltages

estimate_levels(nframe=100, sigma=5.0)¶

Estimate the n-sigma level for the absolute value of the voltages. Returns a list with indicies corresponding to:

Tuning 1, X pol.

Tuning 1, Y pol.

Tuning 2, X pol.

Tuning 2, Y pol.

..note::: The returned list always has four items, regardless of whether or not the input DRX file has one or two tunings.

offset(offset)¶: Offset a specified number of seconds in an open DRX file. This function returns the exact offset time.

read(duration, time_in_samples=False)¶

Given an open DRX file and an amount of data to read in in seconds, read in the data and return a three-element tuple of the actual duration read in, the time for the first sample, and the data as numpy array.

The time tag is returned as seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance by default. However, the time tags can be returns as samples at lsl.common.dp.fS if the `time_in_samples’ keyword is set.

..note::

This function always returns a 2-D array with the first dimension holding four elements. These elements contain, in order:

Tuning 1, polarization X

Tuning 1, polarization Y

Tuning 2, polarization X

Tuning 2, polarization Y

read_frame()¶: Read and return a single lsl.reader.drx.Frame instance.

lsl.reader.ldp.LWA1DataFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶: Wrapper around the various LWA1-related classes defined here that takes a file, determines the data type, and initializes and returns the appropriate LDP class.

lsl.reader.ldp.LWADataFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶: Wrapper around the various classes defined here that takes a file, determines the data type, and initializes and returns the appropriate LDP class.

lsl.reader.ldp.LWASVDataFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶: Wrapper around the various LWA-SV-related classes defined here that takes a file, determines the data type, and initializes and returns the appropriate LDP class.

class lsl.reader.ldp.TBFFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶

Class to make it easy to interface with a TBF file. Methods defined for this class are:

get_info - Get information about the file’s contents
get_remaining_frame_count - Get the number of frames remaining in the file
offset - Offset a specified number of seconds into the file
read_frame - Read and return a single lsl.reader.tbw.Frame instance
read - Read in the capture and return it as a numpy array

offset(offset)¶: Offset a specified number of seconds in an open TBF file. This function returns the exact offset time.

Note

The offset provided by this function is relatively crude due to the structure of TBF files.

Changed in version 1.2.4: Offsets are now relative to the current location in the file rather than to the start of the file

read(duration=None, time_in_samples=False)¶

Read and return the entire TBF capture. This function returns a three-element tuple with elements of:

the actual duration of data read in,

the time tag for the first sample, and

a 3-D Numpy array of data.

The time tag is returned as seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance by default. However, the time tags can be returns as samples at lsl.common.dp.fS if the `time_in_samples’ keyword is set.

The sorting order of the output data array is by digitizer number - 1.

read_frame()¶: Read and return a single lsl.reader.tbf.Frame instance.

class lsl.reader.ldp.TBNFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶

Class to make it easy to interface with a TBN file. Methods defined for this class are:

get_info - Get information about the file’s contents
get_remaining_frame_count - Get the number of frames remaining in the file
offset - Offset a specified number of seconds into the file
read_frame - Read and return a single lsl.reader.tbn.Frame instance
read - Read a chunk of data in and return it as a numpy array
estimate_levels - Estimate the n-sigma level for the absolute value of the voltages

estimate_levels(nframe=100, sigma=5.0)¶: Estimate the n-sigma level for the absolute value of the voltages. Returns a list with indicies that are the digitizer numbers minus one.

offset(offset)¶: Offset a specified number of seconds in an open TBN file. This function returns the exact offset time.

Note

The offset provided by this function is relatively crude due to the structure of TBN files.

Changed in version 1.2.4: Offsets are now relative to the current location in the file rather than to the start of the file

read(duration, time_in_samples=False)¶

Read in a chunk (in seconds) of TBN data. This function returns a three-element tuple with elements of:

the actual duration of data read in,

the time tag for the first sample, and

a 2-D Numpy array of data.

The time tag is returned as seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance by default. However, the time tags can be returns as samples at lsl.common.dp.fS if the `time_in_samples’ keyword is set.

The sorting order of the output data array is by digitizer number - 1.

read_frame()¶: Read and return a single lsl.reader.tbn.Frame instance.

class lsl.reader.ldp.TBWFile(filename=None, fh=None, ignore_timetag_errors=False, buffering=- 1)¶

Class to make it easy to interface with a TBW file. Methods defined for this class are:

get_info - Get information about the file’s contents
get_remaining_frame_count - Get the number of frames remaining in the file
read_frame - Read and return a single lsl.reader.tbw.Frame instance
read - Read in the capture and return it as a numpy array

read(duration=None, time_in_samples=False)¶

Read and return the entire TBW capture. This function returns a three-element tuple with elements of:

the actual duration of data read in,

the time tag for the first sample, and

a 2-D Numpy array of data.

The time tag is returned as seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance by default. However, the time tags can be returns as samples at lsl.common.dp.fS if the `time_in_samples’ keyword is set.

The sorting order of the output data array is by digitizer number - 1.

Note

Setting the ‘duration’ keyword has no effect on the read process because the entire capture is always read in.

read_frame()¶: Read and return a single lsl.reader.tbw.Frame instance.

Python module that contains various helpers for the lsl.reader module.

New in version 2.0.0.

class lsl.reader.utils.FilePositionSaver(fh)¶: Simple class to save the current location in a file and return to that position when we are done with it.

class lsl.reader.utils.SplitFileWrapper(filenames, sort=True, buffering=- 1)¶: Class to allow seamless access to a file that has been split into parts.

class lsl.reader.base.FrameTimestamp(si=0, sf=0.0)¶

Class to represent the UNIX timestamp of a data frame as an integer number of seconds and a fractional number of seconds.

property astropy¶: Timestamp as an astropy.time.Time instance.

property datetime¶: Timestamp as a naive datetime.datetime instance in UTC.

property dp_timetag¶: Timestamp as a DP timetag (ticks of a 196 MHz clock since UTC midnight on January 1, 1970).

classmethod from_dp_timetag(value, offset=0)¶: Create a new FrameTimestamp instance from a raw DP timetag with an optional offset.

classmethod from_mjd_mpm(mjd, mpm)¶: Create a new FrameTimestamp from a MJD/MPM (milliseconds past midnight) pair.

classmethod from_pulsar_mjd(mjd, mjd_frac, sec_frac)¶: Create a new FrameTimstamp from a three-element tuple of integer number of MJD days, fractional MJD day, and fractional seconds.

property jd¶: JD as a floating point value.

property mjd¶: MJD as a floating point value.

property pulsar_mjd¶: MJD as three-element tuple of integer number of MJD days, fractional MJD day, and fractional seconds.

property unix¶: UNIX timestamp as a floating point value.

property utc_datetime¶: Timestamp as a time zone-aware datetime instance in UTC.

Low-Level TBW – DP¶

Python module to reading in data from both 12-bit and 4-bit TBW files. This module defines the following classes for storing the TBW data found in a file:

Frame

object that contains all data associated with a particular TBW frame. The primary consituents of each frame are:

FrameHeader - the TBW frame header object and

FramePayload - the TBW frame data object.

Combined, these two objects contain all of the information found in the original TBW frame.

The functions defined in this module fall into two class:

convert a frame in a file to a Frame object and
describe the format of the data in the file.

For reading in data, use the read_frame function. It takes a python file- handle as an input and returns a fully-filled Frame object. read_frame is designed to work with both 4-bit and 12-bit observations.

For describing the format of data in the file, two function are provided:

get_data_bits: read in the first frame of an open file handle and return whether or not the data is 12 or 4-bit
get_frames_per_obs: read in the first several frames to see how many stands are found in the data.

Note

This function is a little flaky on TBW data sets that have less than a full complement or 12M (36M) samples.

Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.

class lsl.reader.tbw.Frame(header=None, payload=None, valid=True)¶

Class that stores the information contained within a single TBW frame. It’s properties are FrameHeader and FramePayload objects.

property data_bits¶: Convenience wrapper for the Frame.FrameHeader.data_bits property.

property id¶: Convenience wrapper for the Frame.FrameHeader.id property.

property is_tbw¶: Convenience wrapper for the Frame.FrameHeader.is_tbw property.

property time¶: Convenience wrapper for the Frame.FramePayload.time property.

class lsl.reader.tbw.FrameHeader(frame_count=None, second_count=None, tbw_id=None)¶

Class that stores the information found in the header of a TBW frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.

property data_bits¶: Function to parse the TBW ID field and return the size of number of bits that comprise the data. 12 is returned for 12-bit data, and 4 for 4-bit data.

property id¶: Function to parse the TBW ID field and return the stand number.

property is_tbw¶: Function to check if the data is really TBW and not TBN by examining the TBW ID field. Returns True if the data is TBW, false otherwise.

class lsl.reader.tbw.FramePayload(timetag=None, xy=None)¶

Class that stores the information found in the data section of a TBW frame. Both fields listed in the DP ICD version H are stored.

property time¶: Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance.

lsl.reader.tbw.get_data_bits(filehandle)¶: Find out the number of data bits used in the file be reading in the first frame.

lsl.reader.tbw.get_frames_per_obs(filehandle)¶: Find out how many frames are present per observation by examining the first frames for what would be 260 stands. This is done by reading two frames and then skipping the next 30,000.

Note

Post-IOC it is probably simpler to adopt a value of the number of frames per observation of 260 rather than try to find it from the file.

lsl.reader.tbw.read_frame(filehandle, verbose=False)¶: Function to read in a single TBW frame (header+data) and store the contents as a Frame object. This function wraps readerHeader and readData[(12)|4].

Low-Level TBF – ADP¶

New in version 1.2.

Python module to reading in data from TBF files. This module defines the following classes for storing the TBF data found in a file:

Frame

object that contains all data associated with a particular TBF frame. The primary consituents of each frame are:

FrameHeader - the TBF frame header object and

FramePayload - the TBF frame data object.

Combined, these two objects contain all of the information found in the original TBF frame.

The functions defined in this module fall into two class:

convert a frame in a file to a Frame object and
describe the format of the data in the file.

For reading in data, use the read_frame function. It takes a python file- handle as an input and returns a fully-filled Frame object.

New in version 1.2.0.

class lsl.reader.tbf.Frame(header=None, payload=None, valid=True)¶

Class that stores the information contained within a single TBF frame. It’s properties are FrameHeader and FramePayload objects.

property channel_freqs¶: Convenience wrapper for the Frame.FrameHeader.channel_freqs property.

property is_tbf¶: Convenience wrapper for the Frame.FrameHeader.is_tbf property.

property time¶: Convenience wrapper for the Frame.FramePayload.time property.

class lsl.reader.tbf.FrameHeader(adp_id=None, frame_count=None, second_count=None, first_chan=None)¶

Class that stores the information found in the header of a TBF frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.

property channel_freqs¶: Return a numpy.float32 array for the center frequencies, in Hz, of each channel in the data.

property is_tbf¶: Function to check if the data is really TBF. Returns True if the data is TBF, false otherwise.

class lsl.reader.tbf.FramePayload(timetag=None, fDomain=None)¶

Class that stores the information found in the data section of a TBF frame. Both fields listed in the DP ICD version H are stored.

property time¶: Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance.

lsl.reader.tbf.get_channel_count(filehandle)¶: Find out the total number of channels that are present by examining the first 1000 TBF records. Return the number of channels found.

lsl.reader.tbf.get_first_channel(filehandle, frequency=False)¶: Find and return the lowest frequency channel in a TBF file. If the frequency keyword is True the returned value is in Hz.

lsl.reader.tbf.get_first_frame_count(filehandle)¶: Find and return the lowest frame count encountered in a TBF file.

lsl.reader.tbf.get_frames_per_obs(filehandle)¶: Find out how many frames are present per time stamp by examining the first 1000 TBF records. Return the number of frames per observation.

lsl.reader.tbf.read_frame(filehandle, verbose=False)¶: Function to read in a single TBF frame (header+data) and store the contents as a Frame object.

Low-Level TBN – DP and ADP¶

Python module for reading data in from TBN files.This module defines the following classes for storing the TBN data found in a file:

Frame

object that contains all data associated with a particular TBN frame. The primary constituents of each frame are:

FrameHeader - the TBN frame header object and

FramePayload - the TBN frame data object.

Combined, these two objects contain all of the information found in the original TBN frame.

ObservingBlock object that stores a collection of Frames for all stands/polarizations for a particular time.

In addition to storing the data available in the frame, the Frame object also has attributes for holding information about the gain, central frequency, and filter code used for the observations.

The functions defined in this module fall into two class:

convert a frame in a file to a Frame object and
describe the format of the data in the file.

For reading in data, use the read_frame function. It takes a python file- handle as an input and returns a fully-filled Frame object. The readBlock function reads in a (user-defined) number of TBN frames and returns a ObservingBlock object.

For describing the format of data in the file, two function are provided:

get_sample_rate: read in the few frame of an open file handle and return the sampling rate of the data
get_frames_per_obs: read in the first several frames to see how many stands are found in the data.
..versionchanged:: 1.2.0: Dropped support for ObservingBlock since the lsl.reader.buffer modules does a better job.

Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.

Changed in version 0.5.0: Support for ECR 11 TBN header format change.

class lsl.reader.tbn.Frame(header=None, payload=None, valid=True)¶

Class that stores the information contained within a single TBN frame. It’s properties are FrameHeader and FramePayload objects.

Changed in version 0.5.0: Removed various attributes related to storing a central frequnecy and gain that aren’t needed with ECR 11.

property central_freq¶: Convenience wrapper for the Frame.FrameHeader.central_freq property.

property filter_code¶: Convenience wrapper for the Frame.FramePayload.filter_code property.

property gain¶: Convenience wrapper for the Frame.FrameHeader.gain property.

property id¶: Convenience wrapper for the Frame.FrameHeader.id property.

property is_tbn¶: Convenience wrapper for the Frame.FrameHeader.is_tbn property.

property sample_rate¶: Convenience wrapper for the Frame.FramePayload.sample_rate property.

property time¶: Convenience wrapper for the Frame.FramePayload.time property

class lsl.reader.tbn.FrameHeader(frame_count=None, tuning_word=None, tbn_id=None, gain=None)¶

Class that stores the information found in the header of a TBW frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.

Changed in version 0.5.0: Added various attributes to retrieve the central frequnecy and gain that are part of the ECR 11 changes.

property central_freq¶: Convert the tuning word to a frequency in Hz.

property filter_code¶: Function to convert the sample rate in Hz to a filter code.

property id¶: Function to parse the TBN ID field and return a tuple of the stand number and polarization.

property is_tbn¶: Function to check if the data is really TBN and not TBW by examining the TBN ID field. Returns True if the data is TBN, false otherwise.

class lsl.reader.tbn.FramePayload(timetag=None, iq=None)¶

Class that stores the information found in the data section of a TBN frame. Both fields listed in the DP ICD version H are stored.

Changed in version 0.5.0: Removed various attributes related to storing a central frequnecy and gain that aren’t needed with ECR 11.

property time¶: Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance.

lsl.reader.tbn.get_frames_per_obs(filehandle)¶

Find out how many frames are present per observation by examining the first 2,080 TBN frames. Return the number of frames per observations as a two- element tuple, one for each polarization.

So many TBN frames are read in order to try to compensate for the inter- leaving of the packets from the various DP1 boards during the recording.

Note

Post-IOC it is probably simpler to adopt a value of the number of frames per observation of 520 rather than try to find it from the file.

lsl.reader.tbn.get_sample_rate(filehandle, nframe=None, filter_code=False)¶: Find out what the sampling rate/filter code is from consecutive sets of observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to True.

lsl.reader.tbn.read_frame(filehandle, sample_rate=None, verbose=False)¶: Function to read in a single TBN frame (header+data) and store the contents as a Frame object.

Low-Level COR – ADP¶

New in version 1.2.

Python module to reading in data from COR files. This module defines the following classes for storing the COR data found in a file:

Frame

object that contains all data associated with a particular COR frame. The primary consituents of each frame are:

FrameHeader - the COR frame header object and

FramePayload - the COR frame data object.

Combined, these two objects contain all of the information found in the original COR frame.

The functions defined in this module fall into two class:

convert a frame in a file to a Frame object and
describe the format of the data in the file.

For reading in data, use the read_frame function. It takes a python file- handle as an input and returns a fully-filled Frame object.

Changed in version 1.2.1: Updated for the switch over to 72 channels, complex64 data, and no data weights

New in version 1.2.0.

class lsl.reader.cor.Frame(header=None, payload=None, valid=True)¶

Class that stores the information contained within a single COR frame. It’s properties are FrameHeader and FramePayload objects.

property channel_freqs¶: Convenience wrapper for the Frame.FrameHeader.channel_freqs property.

property gain¶: Convenience wrapper for the Frame.FrameHeader.gain property.

property id¶: Convenience wrapper for the Frame.FramePayload.id property.

property integration_time¶: Convenience wrapper for the Frame.FramePayload.integration_time property.

property is_cor¶: Convenience wrapper for the Frame.FrameHeader.is_cor property.

property time¶: Convenience wrapper for the Frame.FramePayload.time property.

class lsl.reader.cor.FrameHeader(adp_id=None, frame_count=None, second_count=None, first_chan=None, gain=None)¶

Class that stores the information found in the header of a COR frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.

property channel_freqs¶: Return a numpy.float32 array for the center frequencies, in Hz, of each channel in the data.

property is_cor¶: Function to check if the data is really COR. Returns True if the data is COR, false otherwise.

class lsl.reader.cor.FramePayload(timetag=None, navg=None, stand0=None, stand1=None, vis=None)¶

Class that stores the information found in the data section of a COR frame.

property id¶: Return a tuple of the two stands that contribute the this frame.

property integration_time¶: Return the integration time of the visibility in seconds.

property time¶: Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance.

lsl.reader.cor.get_baseline_count(filehandle)¶: Find out the total number of baselines that are present by examining the first several COR records. Return the number of baselines found.

lsl.reader.cor.get_channel_count(filehandle)¶: Find out the total number of channels that are present by examining the first several COR records. Return the number of channels found.

lsl.reader.cor.get_frames_per_obs(filehandle)¶: Find out how many frames are present per time stamp by examining the first several COR records. Return the number of frames per observation.

lsl.reader.cor.read_frame(filehandle, verbose=False)¶: Function to read in a single COR frame (header+data) and store the contents as a Frame object.

Low-Level DRX – DP and ADP¶

Python module to read in DRX data. This module defines the following classes for storing the DRX data found in a file:

Frame

object that contains all data associated with a particular DRX frame. The primary constituents of each frame are:

FrameHeader - the DRX frame header object and

FramePayload - the DRX frame data object.

Combined, these two objects contain all of the information found in the original DRX frame.

ObservingBlock object that stores a collection of Frames for all beams/tunings/ polarizations for a particular time.

The functions defined in this module fall into two class:

convert a frame in a file to a Frame object and
describe the format of the data in the file.

For reading in data, use the read_frame function. It takes a python file- handle as an input and returns a fully-filled Frame object. The readBlock function reads in a (user-defined) number of DRX frames and returns a ObservingBlock object.

For describing the format of data in the file, two function are provided:

get_beam_count: read in the first few frames of an open file handle and return how many beams are present in the file.
get_frames_per_obs: read in the first several frames to see how many frames (tunings/polarizations) are associated with each beam.
..versionchanged:: 1.2.0: Dropped support for ObservingBlock since the lsl.reader.buffer modules does a better job.

Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.

class lsl.reader.drx.Frame(header=None, payload=None, valid=True)¶

Class that stores the information contained within a single DRX frame. It’s properties are FrameHeader and FramePayload objects.

property central_freq¶: Convenience wrapper for the Frame.FramePayload.central_freq property.

property filter_code¶: Convenience wrapper for the Frame.FrameHeader.filter_code property.

property id¶: Convenience wrapper for the Frame.FrameHeader.id property.

property sample_rate¶: Convenience wrapper for the Frame.FrameHeader.sample_rate property.

property time¶: Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance.

class lsl.reader.drx.FrameHeader(frame_count=None, drx_id=None, second_count=None, decimation=None, time_offset=None)¶

Class that stores the information found in the header of a DRX frame. All six fields listed in the DP ICD version H are stored as well as the original binary header data.

property filter_code¶: Function to convert the sample rate in Hz to a filter code.

property id¶: Parse the DRX ID into a tuple containing the beam (1 through 4), tunning (1 and 2), and polarization (0 and 1).

property sample_rate¶: Return the sample rate of the data in samples/second.

class lsl.reader.drx.FramePayload(timetag=None, tuning_word=None, flags=None, iq=None)¶

Class that stores the information found in the data section of a DRX frame. All three fields listed in the DP ICD version H are stored.

property central_freq¶: Function to set the central frequency of the DRX data in Hz.

lsl.reader.drx.get_beam_count(filehandle)¶: Find out how many beams are present by examining the first 32 DRX records. Return the number of beams found.

lsl.reader.drx.get_frames_per_obs(filehandle)¶: Find out how many frames are present per beam by examining the first 32 DRX records. Return the number of frames per observations as a four- element tuple, one for each beam.

lsl.reader.drx.get_sample_rate(filehandle, nframes=None, filter_code=False)¶

Find out what the sampling rate/filter code is from a single observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to true.

This function is included to make easier to write code for TBN analysis and modify it for DRX data.

lsl.reader.drx.read_frame(filehandle, gain=None, verbose=False)¶: Function to read in a single DRX frame (header+data) and store the contents as a Frame object. This function wraps readerHeader and readData.

Low-Level DR Spectrometer – DP and ADP¶

New in version 0.5.

Python module to read in DR spectrometer data. This module defines the following classes for storing the spectra found in a file:

Frame

object that contains all data associated with a particular spectrometer frame. The primary constituents of each frame are:

FrameHeader - the spectrometer frame header object and

FramePayload - the spectral data object.

Combined, these two objects contain all of the information found in the original spectrometer frame.

The functions defined in this module fall into two class:

convert a frame in a file to a Frame object and
describe the format of the data in the file.

For reading in data, use the read_frame function. It takes a python file- handle as an input and returns a fully-filled Frame object For describing the format of data in the file, three function are provided:

get_sample_rate - get the sample rate in the file

getFRAME_SIZE - get the total (header + data) frame size

get_ffts_per_integration - get the number of FFT windows per integration

get_transform_size - get the FFT length

get_integration_time - get the integration time

Note

This reader works with the most current version of the DR spectrometer data format as specified in the version 1.7. To read data created with previous versions of the DR spectrometer, use LSL version 0.5.2.

Changed in version 1.0.1: Added in new functions to help better describe the contents of a DR spectrometer file.

class lsl.reader.drspec.Frame(header=None, payload=None, valid=True)¶

Class that stores the information contained within a single DR spectrometer/ DRX frame. It’s properties are FrameHeader and FramePayloadLinear/FramePayloadStokes objects.

Changed in version 0.6.0: By default the data contained with in a frame is normalized by the number of fills (header.fills parameter). For data products that are a function of more than one primary input, i.e., XY* or I, the minimum fill of X and Y are used for normalization.

property central_freq¶: Convenience wrapper for the Frame.FramePayload.central_freq property.

property data_products¶: Convenience wrapper for the Frame.FrameHeder.data_products property.

property ffts_per_integration¶: Conveinence wrapper for the Frame.FrameHeader.ffts_per_integration property.

New in version 1.0.1.

property filter_code¶: Convenience wrapper for the Frame.FrameHeader.filter_code property.

property id¶: Convenience wrapper for the Frame.FrameHeader.id property.

property integration_time¶: Return the integration time for data in seconds.

New in version 1.0.1.

property is_linear¶: Convenience wrapper for the Frame.FrameHeder.is_linear property.

property is_stokes¶: Convenience wrapper for the Frame.FrameHeder.is_stokes property.

property sample_rate¶: Convenience wrapper for the Frame.FrameHeader.sample_rate property.

property time¶: Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance.

property transform_size¶: Find out what the transform size is.

New in version 1.0.1.

class lsl.reader.drspec.FrameHeader(beam=0, format=0, decimation=None, time_offset=None, nints=None)¶

Class that stores the information found in the header of a DR spectrometer/DRX frame.

property data_products¶: Return a list of data products contained in the file.

New in version 0.6.0.

property ffts_per_integration¶: Return the number of FFT windows per integration.

New in version 1.0.1.

property filter_code¶: Function to convert the sample rate in Hz to a filter code.

property id¶: Return the beam the frame corresponds to.

property is_linear¶: Return whether or not the frame contains linear polarization products or not.

New in version 0.6.0.

property is_stokes¶: Return whether or not the frame contains Stokes polarization parameters or not.

New in version 0.6.0.

property sample_rate¶: Return the sample rate of the data in samples/second.

class lsl.reader.drspec.FramePayload(timetag=None, tuning_words=None, fills=None, errors=None, saturations=None)¶

Class that stores the information found in the data section of a DR spectrometer/ DRX frame.

Changed in version 0.5.3: Added the saturations field to keep up with saturation counts.

Changed in version 0.6.0: The attributes that store the data are not defined until a frame is read in order to account for the fact that spectrometer files can store either linear or Stokes data.

property central_freq¶: Function to set the central frequency of the DRX data in Hz.

lsl.reader.drspec.get_data_products(filehandle)¶: Find out the data products contained in the file by looking at a frame.

lsl.reader.drspec.get_ffts_per_integration(filehandle)¶: Find out what the number of FFT windows per integration is at the current file location.

New in version 1.0.1.

lsl.reader.drspec.get_frame_size(filehandle)¶: Find out what the frame size in a file is at the current file location. Returns the frame size in bytes.

lsl.reader.drspec.get_integration_time(filehandle)¶: Find out what the integration time is at the current file location.

lsl.reader.drspec.get_sample_rate(filehandle, nframes=None, filter_code=False)¶: Find out what the sampling rate/filter code is from a single observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to true.

lsl.reader.drspec.get_transform_size(filehandle)¶: Find out what the transform size in a file is at the current file location.

lsl.reader.drspec.is_linear(filehandle)¶: Find out if the file contains linear polarization products or not.

lsl.reader.drspec.is_stokes(filehandle)¶: Find out if the file contains Stokes parameters or not.

lsl.reader.drspec.read_frame(filehandle, gain=None, verbose=False)¶: Function to read in a single DR spectrometer/DRX frame (header+data) and store the contents as a Frame object.

Low-Level VIDF¶

Python module to read in VDIF data. This module defines the following classes for storing the VIDF data found in a file:

Frame

object that contains all data associated with a particular DRX frame. The primary constituents of each frame are:

FrameHeader - the VDIF frame header object and

FramePayload - the VDIF frame data object.

Combined, these two objects contain all of the information found in the original VDIF frame.

The functions defined in this module fall into two class:

convert a frame in a file to a Frame object and
describe the format of the data in the file.

For reading in data, use the read_frame function. It takes a python file- handle as an input and returns a fully-filled Frame object. The readBlock function reads in a (user-defined) number of VDIF frames and returns a ObservingBlock object.

For describing the format of data in the file, two function are provided:

get_thread_count: read in the first few frames of an open file handle and return how many threads are present in the file.

class lsl.reader.vdif.Frame(header=None, payload=None, valid=True)¶

Class that stores the information contained within a single VDIF frame. It’s properties are FrameHeader and FramePayload objects.

property central_freq¶: Convenience wrapper for the Frame.FrameHeader.central_freq property.

property extended_user_data¶: Convenience wrapper for the Frame.FrameHeader.extended_user_data property.

property id¶: Convenience wrapper for the Frame.FrameHeader.id property.

property sample_rate¶: Convenience wrapper for the Frame.FrameHeader.sample_rate property.

property time¶: Convenience wrapper for the Frame.FrameHeader.time property.

class lsl.reader.vdif.FrameHeader(is_invalid=0, is_legacy=0, seconds_from_epoch=0, ref_epoch=0, frame_in_second=0, version=1, nchan=0, frame_length=0, is_complex='C', bits_per_sample=0, thread_id=0, station_id=0, extended_data_1=None, extended_data_2=None, extended_data_3=None, extended_data_4=None, sample_rate=0.0, central_freq=0.0)¶

Class that stores the information found in the header of a VDIF frame. Most fields in the VDIF version 1.1.1 header are stored.

property extended_user_data¶: Parse the extended user data if it was included with the reader. The data is returned as a dictionary.

property id¶: Return a two-element tuple of the station ID and thread ID.

Note

The station ID is always returned as numeric.

property time¶: Function to convert the time tag to seconds since the UNIX epoch as a lsl.reader.base.FrameTimestamp instance.

class lsl.reader.vdif.FramePayload(data=None)¶: Class that stores the information found in the data section of a VDIF frame.

Note

Unlike the other readers in the lsl.reader module the data are stored as numpy.float32 values.

lsl.reader.vdif.get_frame_size(filehandle, nframes=None)¶: Find out what the frame size is in bytes from a single observation.

lsl.reader.vdif.get_frames_per_second(filehandle)¶: Find out the number of frames per second in a file by watching how the headers change. Returns the number of frames in a second.

lsl.reader.vdif.get_sample_rate(filehandle)¶: Find and return the sample rate in Hz by looking at how many frames there are per second and how many samples there are in a frame.

lsl.reader.vdif.get_thread_count(filehandle)¶: Find out how many thrads are present by examining the first 1024 records. Return the number of threads found.

lsl.reader.vdif.has_guppi_header(filehandle)¶: Determine if a VDIF file has a GUPPI header or not.

New in version 2.0.0.

lsl.reader.vdif.read_frame(filehandle, sample_rate=0.0, central_freq=0.0, verbose=False)¶: Function to read in a single VDIF frame (header+data) and store the contents as a Frame object. This function wraps the _readerHeader and _readData functions.

lsl.reader.vdif.read_guppi_header(filehandle)¶: Read in a GUPPI header at the start of a VDIF file from the VLA. The contents of the header are returned as a dictionary.

Reader Error Codes¶

Module that contains the error classes for the DRX, TBN, and TBW readers. These errors are currently meant to deal with file I/O problems.

exception lsl.reader.errors.BaseReaderError(strerror, errno='-1')¶: Base class for file I/O problem during numpy.fromfile calls and out-of- sync Mark5C headers.

exception lsl.reader.errors.EOFError¶: Extension to the base class for dealing with EOF errors. The error code is 1.

exception lsl.reader.errors.SyncError(type='Mark5C', location=None, sync1=None, sync2=None, sync3=None, sync4=None)¶: Extension to the base class for dealing with Mark 5C header sync word problems. If the sync word doesn’t match what is expected. The error code is 2.

lsl.reader.errors.list_error_codes(errno=None)¶: Function to provide a list of errors defined in this file. It alternatively takes an error code using the ‘errno’ keyword and returns its description.

Reader Ring Buffer¶

Buffer for dealing with out-of-order/missing frames.

Changed in version 0.5: Removed support for DRX FrameBuffers since they shouldn’t be needed.

Changed in version 0.6: Removed support for TBW FrameBuffers since they didn’t really work.

Changed in version 1.0.0: Added back in the DRX FrameBuffers since they might be useful. Dropped TBW support from within the FrameBufferBase class.

Changed in version 1.0.2: Added support for the LWA-SV ADP DRX8 mode

Changed in version 1.1.4: Dropped support for the LWA-SV ADP DRX8 mode

Changed in version 1.2.1: Added support for the LWA-SV TBF and COR modes

class lsl.reader.buffer.CORFrameBuffer(chans, nsegments=5, reorder=False)¶

Bases: lsl.reader.buffer.FrameBufferBase

A sub-type of FrameBufferBase specifically for dealing with COR frames. See lsl.reader.buffer.FrameBufferBase for a description of how the buffering is implemented.

Keywords:

chans: list of start channel numbers to expect data for
nsegments: number of ring segments to use for the buffer (default is 5)
reorder: whether or not to reorder frames returned by get() or flush() by start channel (default is False)

The number of segements in the ring can be converted to a buffer time in seconds:

Segments	Time
1	10
2	20
5	50

create_fill(key, frameParameters)¶: Create a ‘fill’ frame of zeros using an existing good packet as a template.

get_figure_of_merit(frame)¶: Figure of merit for sorting frames. For TBF this is: frame.payload.timetag

get_max_frames()¶: Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.

class lsl.reader.buffer.DRXFrameBuffer(beams=[], tunes=[1, 2], pols=[0, 1], nsegments=20, reorder=False)¶

Bases: lsl.reader.buffer.FrameBufferBase

A sub-type of FrameBufferBase specifically for dealing with DRX frames. See lsl.reader.buffer.FrameBufferBase for a description of how the buffering is implemented.

Keywords:

beams: list of beam to expect packets for
tunes: list of tunings to expect packets for
pols: list of polarizations to expect packets for
nsegments: number of ring segments to use for the buffer (default is 20)
reorder: whether or not to reorder frames returned by get() or flush() by stand/polarization (default is False)

The number of segements in the ring can be converted to a buffer time in seconds:

Segments	DRX Filter Code
Segments	1	2	3	4	5	6	7
10	0.16	0.08	0.04	0.02	0.01	0.004	0.002
20	0.33	0.16	0.08	0.04	0.02	0.008	0.004
30	0.49	0.25	0.12	0.06	0.03	0.013	0.006
40	0.66	0.33	0.16	0.08	0.03	0.017	0.008
50	0.82	0.41	0.20	0.10	0.04	0.021	0.010
100	1.64	0.82	0.41	0.20	0.08	0.042	0.021

create_fill(key, frameParameters)¶: Create a ‘fill’ frame of zeros using an existing good packet as a template.

get_figure_of_merit(frame)¶

Figure of merit for sorting frames. For DRX it is:: <frame timetag in ticks>

get_max_frames()¶: Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.

class lsl.reader.buffer.FrameBufferBase(mode='TBN', stands=[], beams=[], tunes=[], pols=[], chans=[], threads=[], nsegments=6, reorder=False)¶

Bases: object

Frame buffer for re-ordering TBN and DRX frames in time order. This class is filled with frames and a returns a frame list when the ‘nsegments’ starts filling. In that case, the oldest segment is returned.

The buffer also keeps track of what has already been read out so that tardy frames are just dropped. For buffers that are read out, missing frames are replaced with frames filled with zeros.

Note

Due to the nature of the buffer, it is possible that there will still be ‘nsegments’-1 segements in the buffer that are either full or partially full. This can be retrieved using the buffer’s ‘flush()’ function.

append(frames)¶: Append a new frame to the buffer with the appropriate time tag. True is returned if the frame was added to the buffer and False if the frame was dropped because it belongs to a buffer that has already been returned.

create_fill(key, frameParameters)¶

Create a ‘fill’ frame of zeros using an existing good packet as a template.

This will be overridden by sub-classes of FrameBufferBase.

property filled¶: Indicates whether or not the ring buffer is full or not.

New in version 1.2.4.

flush()¶: Generator to return all of the remaining frames in the buffer from buffers that are considered ‘full’. Afterwards, delete all buffers. This is useful for emptying the buffer after reading in all of the data.

Note

It is possible for this function to return list of packets that are mostly invalid.

get(key_to_return=None)¶: Return a list of frames that consitute a ‘full’ buffer. Afterwards, delete that buffer and mark it as closed so that any missing frames that are recieved late are dropped. If none of the buffers are ready to be dumped, None is returned.

get_figure_of_merit(frame)¶

Figure of merit for storing/sorting frames in the ring buffer.

This will be overridden by sub-classes of FrameBufferBase.

get_max_frames()¶

Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.

This will be overridden by sub-classes of FrameBufferBase.

is_empty()¶: Determine if there is anything in the buffer or not. Returns False if there is, True if there is not.

property overfilled¶: Indicates whether or not the ring buffer has too many segements or not.

New in version 1.2.4.

peek(require_filled=True)¶

Peek into the buffer to see what the next key to be retruned by a call to get() will be. Returns None if the buffer is not full and

require_filled is True.

New in version 1.2.4.

put(*args, **kwds)¶: Synonymous with ‘append’.

reset()¶: Emtpy the contents of the buffer and reset it to a clean state.

status()¶

Print out the status of the buffer. This contains information about:

The current buffer fill level
The numer of full and partial buffer dumps preformed
The number of missing frames that fill packets needed to be created for
The number of frames that arrived too late to be incorporated into one of the ring buffers

class lsl.reader.buffer.TBFFrameBuffer(chans, nsegments=25, reorder=False)¶

Bases: lsl.reader.buffer.FrameBufferBase

A sub-type of FrameBufferBase specifically for dealing with TBF frames. See lsl.reader.buffer.FrameBufferBase for a description of how the buffering is implemented.

Keywords:

chans: list of start channel numbers to expect data for
nsegments: number of ring segments to use for the buffer (default is 25)
reorder: whether or not to reorder frames returned by get() or flush() by start channel (default is False)

The number of segements in the ring can be converted to a buffer time in seconds:

Segments	Time
10	0.0004
25	0.001
50	0.002
100	0.004
200	0.008

create_fill(key, frameParameters)¶: Create a ‘fill’ frame of zeros using an existing good packet as a template.

get_figure_of_merit(frame)¶: Figure of merit for sorting frames. For TBF this is: frame.payload.timetag

get_max_frames()¶: Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.

class lsl.reader.buffer.TBNFrameBuffer(stands=[], pols=[0, 1], nsegments=20, reorder=False)¶

Bases: lsl.reader.buffer.FrameBufferBase

A sub-type of FrameBufferBase specifically for dealing with TBN frames. See lsl.reader.buffer.FrameBufferBase for a description of how the buffering is implemented.

Keywords:

stands: list of stand to expect packets for
pols: list of polarizations to expect packets for
nsegments: number of ring segments to use for the buffer (default is 20)
reorder: whether or not to reorder frames returned by get() or flush() by stand/polarization (default is False)

The number of segements in the ring can be converted to a buffer time in seconds:

Segments	TBN Filter Code
Segments	1	2	3	4	5	6	7
10	5.1	1.6	0.8	0.4	0.2	0.1	0.05
20	10.2	3.3	1.6	0.8	0.4	0.2	0.10
30	15.4	4.9	2.5	1.2	0.6	0.3	0.15
40	20.5	6.6	3.3	1.6	0.8	0.4	0.20
50	25.6	8.2	4.1	2.0	1.0	0.5	0.26
100	51.2	16.4	8.2	4.1	2.0	1.0	0.51

create_fill(key, frameParameters)¶: Create a ‘fill’ frame of zeros using an existing good packet as a template.

get_figure_of_merit(frame)¶

Figure of merit for sorting frames. For TBN it is:: <frame timetag in ticks>

get_max_frames()¶: Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.

class lsl.reader.buffer.VDIFFrameBuffer(threads=[0, 1], nsegments=10, reorder=False)¶

Bases: lsl.reader.buffer.FrameBufferBase

A sub-type of FrameBufferBase specifically for dealing with VDIF frames. See lsl.reader.buffer.FrameBufferBase for a description of how the buffering is implemented.

Keywords:

threads: list of thread IDs to expect data for
nsegments: number of ring segments to use for the buffer (default is 10)
reorder: whether or not to reorder frames returned by get() or flush() by stand/polarization (default is False)

create_fill(key, frameParameters)¶: Create a ‘fill’ frame of zeros using an existing good packet as a template.

get_figure_of_merit(frame)¶: Figure of merit for sorting frames. For VIDF this is: seconds_from_epoch * 1000000 + frame_in_second

get_max_frames()¶: Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.