Accurate Video S3 Storages

Introduction

This guide covers how to add an Amazon Simple Storage Service (S3) bucket to Accurate.Video, and how to ingest files from this storage as assets. The guide covers the following steps:

  • how to create a new AWS IAM user, which is used by Accurate.Video to get programmatic access through the S3 API. This is needed since the files located on the S3 bucket are not open to the public. By creating a new user it is also easy to revoke access, should it be needed.
  • how to add a bucket policy that gives access to the created user, and how to add the required CORS configuration to the bucket.
  • how to use the simple UI in Accurate Video to very quickly and easily connect the storage and ingest files.

Prerequisites

AWS access & knowledge

Adding storages to Accurate.Video is a task that is performed by an administrator or someone with access to the AWS infrastructure. Due to the nature of this task, some AWS knowledge is required. You'll need access to an AWS account where you can create users in IAM and buckets in S3.

Permissions for storages

Ensure your user has the sufficient permissions required for adding storages in Accurate.Video. Refer to the roles available in the guide Authentication - Roles and permissions configurations.

If you cannot see the storage tab, or the Add storage button is missing, your user lacks the required permissions.

Accurate.Video Storages

Storages are accessed through the Storages tab in the navigation bar at the top.

Screenshot from 2021-03-15 13-48-51.png

Clicking the Add storage button opens up a modal popup where you enter the required information about storage, shown below.

Screenshot from 2021-03-15 13-53-25.png

There are three options for supplying credentials when adding a storage:

  • Server credentials (default)
  • Custom credentials (using secret key/access key)
  • Anonymous access (read-only)

This guide covers using custom credentials only.

Add AWS user

The first step is to create a new user in AWS IAM (Identity and Access Management). The user will be used by the Accurate Video system to get access to the files located in the S3 bucket. By creating a new user, we can control and limit the access for this specific user, and it makes it easy to revoke or delete access in the future, should it be required.

Add user

Go to the AWS IAM page, and click on users. Click on “create user”. Give it a name that you will remember and enable the “programmatic access” checkbox.

Click to the next step of the user creation wizard.

Permissions

On the second screen, specific permissions can be given to the user. In this case, no permissions should be given to the user, so make sure the user is not added to any groups or that any permissions are set.

Permissions

By default, no permissions are added to the user. It should be OK to just click next to go to the third step of the user creation wizard.

Tags

Here you can add specific tags to the user if you wish. This is optional and is not required. A tag can be added to remember that this user was created for Accurate Video purposes.

Tags

Click next to proceed to the fourth step of the user creation wizard.

Review

Review that everything looks OK before proceeding to the final step. There should be a warning showing that the user has no permissions. Make sure that the user AWS access type is set to programmatic access - with an access key.

Review

Click on “create user” to proceed.

Copy access key & secret key

The user is now created!

Review

Take note and write down the following information:

  • User name
  • Access key ID
  • Secret access key

You will need the information above in subsequent steps, when the required bucket policy is added and when the storage is added to the Accurate Video interface.

Copy AWS user account ID

In order to proceed, there is one more type of information required, the User ARN of the user that was just created.

Copy user key

In order to find this information, go back to the start page of AWS IAM. Click on “users”, and select the user just created by clicking on the user name.

A screen similar to the one above will be shown, at the top of the screen you will see the User ARN. This full string will need to be copied and remembered. There is a small icon next to the string which conveniently copies the string. Click on this to copy the string.

In this example the value is arn:aws:iam::010652268016:user/av-s3-user

Create an S3 bucket

In the following steps, an S3 bucket is created and configured. You can either choose to create a new S3 bucket, or re-use an existing. In this example, a new bucket is created.

Create bucket

Navigate to the AWS S3 start page. From here there will be a button to create a new bucket. Give the bucket a name, and select the region where the bucket should be created. In doesn’t matter, but recommended is to keep the bucket as close to the end-users as possible, as all content will be served from this bucket.

For the rest of the options, simply follow through the bucket creation wizard and use all default values for options and permissions.

Add bucket policy

After the bucket has been created, the required permissions will need to be added. Click on the bucket to open the details page. Click “permissions” at the top, and then “bucket policy”. You should see an empty screen where you can add text.

Bucket policy

Copy and paste the following text into the bucket policy.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
      "AWS": "arn:aws:iam::010652268016:user/av-s3-user"
    },
    "Action": [
      "s3:GetObject",
      "s3:PutObject",
      "s3:ListBucket"
      ],
    "Resource": [
      "arn:aws:s3:::av-sales-demos",
      "arn:aws:s3:::av-sales-demos/*"
      ]
    }
  ]
}

Update the policy with correct "arn" values according to your configuration. Click save to update the bucket policy.

S3 CORS configuration

In order to access media files from a browser, a CORS policy must be added to the bucket.
The CORS policy is found at the bottom of the bucket Permission page.

s3-cors.png

Copy and paste the following text:

[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"GET",
"HEAD"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": [
"Content-Range"
],
"MaxAgeSeconds": 3000
}
]

Click save to set the CORS configuration.

Note 1: If you know the exact origin of the Accurate Video system, you can limit this instead of setting the AllowedOrigin to "*" (accept anything).

Note 2: If AllowedOrigins is not set to "*", then one of the allowed origins must be the literal string "null" (without quotation marks) in order for DASH and HLS playback to work. The reason for this is that all browsers set their origin to "null" when following redirects.

Here you can find more information about CORS and why this configuration is needed: https://en.wikipedia.org/wiki/Cross-origin_resource_sharing

Connect storage

All settings are now set in AWS. Now it’s time to add the storage to Accurate Video.

Connect storage

Go to the Accurate Video start page, click on “storages” at the top. You should see a screen similar to the above, empty without any storages. If you have already added storage, it will show up here.

Click the “add storage” button at the top.

Add storage

A modal with a form should pop up. Start by giving the storage a name, this is purely for identification and can be anything you want.

AddStorageNewModal.png

Input and output buckets

Recommended best practice is to differentiate between input and output buckets. Input buckets are storages where files are being read and ingested from, with only read-only permissions. Output buckets are dedicated buckets where artifacts generated by Accurate.Video is stored, such as thumbnails, waveform data, and metadata.

You have full control of how you map your buckets, and can even have multiple input and multiple output buckets. For input buckets, simply use the read and list permissions (list if you want to ingest through the UI). For output buckets, tags can be used to determine where output artifacts are stored.

Permissions

The permission section decides what AV can do with the storage and here you simply click the respective checkbox to activate certain permissions:

  • Read - Gives read access to storage. Required for input buckets.
  • Write - Gives write access to storage. Required for output buckets.
  • List - Gives list files access to storage. Required if ingesting from the UI to see files.
  • Delete - Gives delete file access to storage. Required for deleting assets with original files.

With the above permissions, you can control and easily create read-only storages where files are ingested from, and writable, output storages where files that are generated by Accurate.Video is stored, files such as transcoded proxy files, thumbnails, and waveforms.

Tags

Using tags, you can fine-tune where these files are stored. There are a number of pre-defined tags that can be used, refer to the guide on writable storage metadata fields for details.

Name

The next step is to enter the S3 bucket name, note that this represents the name in S3 that was used before. The region should also be set, this has to be the same region as in S3.

If you are unsure about what region codes, look at the following list:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions

Credentials

Click “use custom credentials” and enter the AWS Access key and AWS Secret key from before. The AWS access key is a string all in upper-case, and the AWS secret key is a slightly longer key mixed with both upper and lower characters.

Refresh interval

Optionally you can set a periodic refresh interval. This reflects how often the S3 bucket is queried and files within the bucket are indexed by Accurate.Video. Note that it will poll the S3 API according to this schedule, so you don’t want to set the interval too high. In case the storage is very rarely updated, the best option could be to disable it and doing manual refreshes in the UI when required.

Another option is to use Amazon SNS to automatically index new files as they appear in S3. Please refer to the Configuring SNS and S3 event notifications in Accurate.Video guide on how to set this up.

Ingest files

After connecting the storage, you should see a green text which says Connected, this is an indication that everything is ok. If it comes up as Broken instead, something is wrong and Accurate.Video failed to access the storage.

The next step is to start ingesting files, which can be done either through the UI or the API.

Refer to the Ingest assets using the UI guide for more details on how to ingest files through the UI.

You can also configure automatic ingest, see Auto Ingest using the UI for more details.

If you're interested in learning more about automatic ingest in general, see Automatic Ingest.

You can also connect storages and ingest files through the REST API, see REST API Examples for some examples.

AWS Elemental - Setup, Connect and Transcoding Setup Keycloak with AV