OTT testing: HTTP Live Streaming (HLS)

A Test Agent can measure user experience for adaptive video streaming using HTTP Live Streaming (HLS). On being told the URL of the video, the Test Agent will parse the manifest file and start downloading the video segments. The algorithm will adapt to current network conditions and select the highest possible quality (bit rate), while avoiding buffering.

OTT is an abbreviation for “Over The Top” and refers to the content being delivered on top of an ordinary Internet service, unlike IPTV, which runs as a separate service. HLS is one common OTT protocol, specified by Apple Inc. and supported by all Apple mobile devices as well as by the Safari browser.

The HLS protocol supports multiple qualities with different bit rates, called variants, for the same video, and the client can choose to download segments from the variant that best matches network and client performance.

This task works only with IPv4.

Prerequisites

To run HLS measurements you need to have at least one Test Agent installed. If you haven’t already done the installation, consult the installation guides found here.

In you test or monitor, add an OTT - HLS task and fill in the mandatory parameters below:

Parameters

See the common parameters page for the following:

• Parameters that are set on the test step level: Duration, Fail threshold, and Wait for ready.

• SLA thresholds for monitors: SLA Good and SLA Acceptable.

• Advanced settings common to all test tasks: Delayed start.

General

• Clients: Test Agent interfaces to use as clients.

• URL: URL of the video stream. This can be either a playlist linking to other playlists for the different variants, or a playlist containing the segments.

  The file extension is .m3u or .m3u8.

Thresholds for errored seconds (ES)

• Playback rate (Mbit/s): An errored second is triggered if the playback rate drops below this threshold.

  Min: 0 Mbit/s. No default.

• Download rate (Mbit/s): An errored second is triggered if the download rate drops below this threshold.

  Min: 0 Mbit/s. No default.

• Selected rate (Mbit/s): An errored second is triggered if the rate of the variant selected by the Test Agent is below this threshold.

  Min: 0 Mbit/s. No default.

• Buffer size (seconds): An errored second is triggered if the duration of the data segment buffered in the Test Agent drops below this threshold.

  Min: 0 s. No default.

Advanced

• Buffer size (seconds): Target duration of buffered data. When the duration of the buffered data falls below this value, new segments will be downloaded.

  Min: 0 s. Default: 60 s.

• Initial buffering (seconds): The duration of the initial buffered data required before the playback starts.

  Min: 0 s. Default: 10 s.

• Loop: If set to Yes, the video stream will loop when the end of the playlist is reached. If set to No, the playback will stop, and errored seconds will be triggered.

  The “No” option is relevant mainly for live video streams which you expect never to end, and for which you want to trigger an alarm if that happens. Default: Yes.

Result metrics

• Playback rate (Mbit/s): Actual data rate of the video stream.

• Download rate (Mbit/s): Download rate for the segments. The download is started when the amount of buffered data falls below the configured buffer size.

• Selected rate (Mbit/s): Data rate of the selected variant in the manifest file.

• (Min) Buffer (s): Actual buffer length in seconds.

• ES playback rate (%): Percentage of seconds during which the playback rate dropped below the Playback rate threshold.

• ES download rate (%): Percentage of seconds during which the download rate dropped below the Download rate threshold.

• ES selected rate (%): Percentage of seconds during which the selected rate dropped below the Selected rate threshold.

• ES buffer underrun (%): Percentage of seconds during which the buffered data dropped below the Buffer size threshold.

• ES buffering (%): Percentage of seconds during which the buffer became empty, so that rebuffering was needed.

• ES total: Aggregated errored second percentage, taking into account all types of error.

• SLA: Service level agreement fulfillment: equal to (100 – ES total) %.

Example

Below is an example of OTT testing.

The first graph shows the downloading of OTT data (red) and the amount of data stored in the buffer (gray).

The second graph shows the actual playback rate (green) compared to the playback rate currently selected by the client (blue).