CBF-SDP Emulator

This is an emulator for the Correlator Beamformer and its data sending capabilities. It reads data from a data source (typically a Measurement Set, but other sources will be added in the future, like correlator data dumps) and sends it over a network transport to a number of receivers, thus mimicking the CBF-SDP interface.

This is an extensible and configurable package that has been designed to support multiple communication protocols and provide a platform for testing consumers of CBF data payloads. It currently only implements the SPEAD transmission protocol, but other protocols can be added in the future.

Installation

This is a standard setuptools-based program so the usual installation methods should work, and all dependencies should be automatically pulled. The only caveat is that some of the dependencies are found in the CAR server instead of PyPI, so we need to point pip to the CAR in order to them:

# Go into the top-level directory of this repository
cd ska-cbf-sdp-emulator

# Using "pip" will automatically install any dependencies from PyPI
pip install --extra-index-url=https://artefact.skao.int/pypi-all/simple .

# Use pip in editable mode if you are actively changing the code
pip install --extra-index-url=https://artefact.skao.int/pypi-all/simple -e .

Note that this package can also be installed directly via pip from the CAR itself:

pip install --extra-index-url=https://artefact.skao.int/pypi-all/simple \
  cbf-sdp-emulator

Alternatively, if the latest development version is needed, it can also be installed from the upstream git repository itself:

pip install --extra-index-url=https://artefact.skao.int/pypi-all/simple \
  'git+https://gitlab.com/ska-telescope/ska-cbf-sdp-emulator.git#egg=cbf-sdp-emulator'

Sender

Configuration options

The following configuration categories/names are supported:

  • reader: these are configuration options applied when reading the input Measurement Set.
  • start_chan: the first channel for which data is read. Channels before this one are skipped. If start_chan is bigger than the actual number of channels in the input MS an error is raised.
  • num_chan: number of channels for which data is read. If num_chan + start_chan are bigger than the actual number of channels in the input MS then num_chan is adjusted.
  • num_timestamps: number of timestamps to be sent, defaults to 0 which is all of them.
  • num_repeats: defaults to 1 - number of times the number of timestamps are sent. This will send the same data over and over which is less realistic but imposes less stress on the file-system. TIME values increment with each repetition.
  • transmission: these are options that apply to the transmission method.
  • scan_id: the scan_id to use for all payloads in transmission.
  • method: the transmission method to use, defaults to spead2.
  • target_host: the host where data will be sent to.
  • target_port_start: the first port where data will be sent to.
  • endpoints: the endpoints where data will be sent to (see below). If present, target_host and target_port_start are ignored.
  • channels_per_stream: number of channels for which data will be sent in a single stream.
  • max_packet_size: the maximum size of packets to build, used by spead2.
  • buffer_size: the socket buffer size, used by spead2.
  • rate: the maximum send data rate, in bytes/s. Used by spead2, defaults to 1 GB/s.
  • time_interval:: the period of time to wait between sending data for successive data dumps. Positive values are used as-is. A value of 0 means to use the time differences in the TIME column of the Measurement Set. Negative values mean to don’t wait, sending data as fast as possible.
  • transport_protocol: the network transport protocol used by spead2. Supported values are udp and tcp, defaults to udp.

Endpoint specification

There are two ways to specify the target endpoints where data will be sent to. Note that in both cases the number of streams that are set up equals number_channels / transmission.channels_per_stream, where number_channels depends on reader.num_chan and the input Measurement Set.

  • Using transmission.target_host and transmission.target_port. When using these options then all streams will be sent to target_host, and successive streams will be sent to successive ports starting at target_port.
  • Using transmission.endpoints. This option is a single string of comma-separated endpoint specifications. Each endpoint takes the form of host:ports, where ports is either a single number, or a range like start-end. For example, 127.0.0.1:8000,127.0.0.1:8001 and 127.0.0.1:8000-8001 are equivalent.

If the list of endpoints to use is less than the number of streams an error is raised. If it’s larger, then the first endpoints are used, and the rest are silently ignored.

Running

An emu-send program should be available after installing the package. This program takes a Measurement Set and transmits it over the network using the preferred transmission method.

measurement_set

The measurement set to read data from

-e execution_block_id

An execution block id to monitor for scans

-c config

A configration file to read options from

-o option

Additional configuration options in the form of category.name=value

-q quiet

Additional parameter to silence info logging from standard output

Tango device wrapper

A Tango device wrapping the emulator sender is available under CBF-SDP Emulator TANGO Devices. The purpose of this Tango device is to be used as a simulation of the real CBF, making it possible to run a full end-to-end SKA system that exercises the visibility data flow.

In SDP

In the context of the SDP Integration the emulator is deployed as a Helm chart to exercise the visibility receive workflow.

API documentation

This section describes the main entry points for the emulator API. While most users will be using the emu-send program, the sender code can be embedded directly into arbitrary python programs, like in the case of the CBF-SDP Emulator TANGO Devices.

cbf_sdp.packetiser module

Primary send functions for ska-sdp-cbf-emulator

cbf_sdp.packetiser.packetise(config: configparser.ConfigParser, ms: Union[<sphinx.ext.autodoc.importer._MockObject object at 0x7fdb31ccce90>, pathlib.Path, str])[source]

Reads data off a Measurement Set and transmits it using the transmitter specified in the configuration.

Uses the vis_reader get data from the measurement set then gives it to the transmitter for packaging and transmission. This code is transmission protocol agnostic.

cbf_sdp.transmitters.spead2_transmitters module

Implementation for the SPEAD2 network transport

This module contains the logic to take ICD Payloads and transmit them using the SPEAD protocol.

class cbf_sdp.transmitters.spead2_transmitters.Spead2SenderPayload(num_baselines=None, num_channels=None)[source]

SPEAD2 payload following the CSP-SDP interface document

cbf_sdp.transmitters.spead2_transmitters.parse_endpoints(endpoints_spec)[source]

Parse endpoint specifications.

Each endpoint is a colon-separated host and port pair, and multiple endpoints are separated by commas. A port can be a single number or a range specified as “start-end”, both inclusive.

class cbf_sdp.transmitters.spead2_transmitters.transmitter(config)[source]

SPEAD2 transmitter

This class uses the spead2 library to transmit visibilities over multiple spead2 streams. Each visiblity set given to this class’ send method is broken down by channel range (depending on the configuration parameters), and each channel range is sent through a different stream.

close()[source]

Sends the end-of-stream message

prepare(num_baselines, num_channels)[source]

Create the sending SPEAD streams

send(scan_id: int, ts: int, ts_fraction: int, vis: <sphinx.ext.autodoc.importer._MockObject object at 0x7fdb31cb2150>)[source]

Send a visibility set through all SPEAD2 streams

Parameters:
  • int – the scan id
  • ts – the integer part of the visibilities’ timestamp
  • ts_fraction – the fractional part of the visibilities’ timestamp
  • vis – the visibilities