Auto ingest from S3 using event notifications and SNS

Introduction

This guide covers how to use the Amazon Simple Notification Service (SNS) and event notifications in Amazon Simple Storage Service (S3) to index and auto ingest files in Accurate.Video without having to rely on polling or through manual refreshes.

Files in storages are indexed by Accurate.Video and made available to be ingested. The index job can run as a result of a periodic refresh, which is configured directly on the storage itself, or by triggering a manual storage refresh in the UI. Both methods are clunky, polling introduces a lot of potential unused refreshes, and relying on manual refresh is slow and frustrating for users.

The solution to both of these problems, when using S3 storages, is through the use of an SNS topic and S3 event notifications. The end result is that every file uploaded to S3 will be indexed on-demand upon creation and no unnecessary refreshes are required. Let’s explore how to accomplish this.

Create SNS topic

The first step is to create a new SNS topic, from the Amazon SNS console. Set the type as standard, and give it a name. The rest of the options can be left in their default state.

Screenshot from 2021-03-12 16-44-14.png

Create subscription

Navigate to the topic you just created, and find the create subscription button for it.

Screenshot from 2021-03-12 16-45-48.png

Select the topic you just created and set the protocol to either HTTP or HTTPS depending on your setup. The endpoint should be the URL to the SNS REST endpoint, which is exposed at sns in the REST reference documentation. Note that depending on how the API is exposed in your configuration this can differ. In our case, the root REST API is exposed at /api, which means that the SNS endpoint is available at /api/sns

The rest of the settings can be left in their default state.

Screenshot from 2021-03-12 16-58-04.png

Wait until confirmed

Wait for a while, until the Status of the subscription changes to Confirmed. This could take a while, but not too long. If it’s not confirming automatically, you can manually trigger a Request confirmation on the subscription.

Screenshot from 2021-03-12 16-59-46.png

If you check the server logs, the adapter (ap-backend) will show the confirmed subscription:

2021-03-12 15:58:11,037 INFO [se.cod.ap.ada.pos.api.res.AwsSnsResourceImpl_Subclass] (executor-thread-2) >>Signature verification succeeded
2021-03-12 15:58:11,119 INFO [se.cod.ap.ada.pos.api.res.AwsSnsResourceImpl_Subclass] (executor-thread-2) Subscription confirmation return status: 200

Set access policy

The next step is to connect S3 event notifications, which will send messages to this SNS topic, which are fanned out and delivered to all subscriptions. In order to allow S3 to publish to SNS, we need to set an access policy on the topic itself.

Select your topic, and click access policy. Paste the text below, but modify the resource row to use the ARN of the topic you created.

Screenshot from 2021-03-12 17-30-17.png

{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "publish-from-s3",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com",
"AWS": "*"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:eu-west-1:381397495928:new-items-added",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:*"
}
}
}
]
}

S3 event notification

Navigate to the S3 console, and select the desired bucket that you wish to enable event notification for. Click on Properties, and scroll down to Event notifications.

Screenshot from 2021-03-12 17-18-01.png

Click on the create event notification button.

Screenshot from 2021-03-12 17-19-43.png

Create event notification

Create a new event notification by giving it a suitable name.

Screenshot from 2021-03-12 17-21-50.png

Make sure that for event types, you enable all object create events. You may also include more events here, deleted files for example, if desired.

Screenshot from 2021-03-12 17-22-43.png

For destination, make sure you select the SNS topic option. Select the SNS topic you created before, which should appear in the drop-down list.

Screenshot from 2021-03-12 17-23-38.png

Save your changes to create the event notification.

Screenshot from 2021-03-12 17-33-17.png

Connect storage

The next step is to connect an S3 bucket as storage to Accurate.Video. Refer to the guide below for a detailed walkthrough.

Accurate Video S3 Storages

Enable auto ingest

The last step is to make sure auto ingest is enabled on the storage in Accurate.Video. You'll need to set a periodic refresh for the auto ingest setting to be available, but you can set a high refresh delay to avoid polling the storage.

Enable auto ingest on storage

Upload a file

To test the SNS integration, the easiest is to upload a file to the S3 bucket. This will trigger an S3 event notification, which will message the SNS topic, which will in turn message the Accurate.Video SNS endpoint.

Once Accurate.Video receives the notification, the file will be indexed by a background process and shown in the ingest UI immediately. An auto ingest job should be automatically started, and show up in the jobs view.

Ingest job started

That’s it! Congratulations on configuring auto ingest with SNS successfully!

Accurate.Video Templates Explanation of time-coded metadata shown as end frame inclusive or exclusive