This guide covers how to use the Amazon Simple Notification Service (SNS) and event notifications in Amazon Simple Storage Service (S3) to index and auto ingest files in Accurate.Video without having to rely on polling or through manual refreshes.
Files in storages are indexed by Accurate.Video and made available to be ingested. The index job can run as a result of a periodic refresh, which is configured directly on the storage itself, or by triggering a manual storage refresh in the UI. Both methods are clunky, polling introduces a lot of potential unused refreshes, and relying on manual refresh is slow and frustrating for users.
The solution to both of these problems, when using S3 storages, is through the use of an SNS topic and S3 event notifications. The end result is that every file uploaded to S3 will be indexed on-demand upon creation and no unnecessary refreshes are required. Let’s explore how to accomplish this.
The first step is to create a new SNS topic, from the Amazon SNS console. Set the type as standard, and give it a name. The rest of the options can be left in their default state.
Navigate to the topic you just created, and find the create subscription button for it.
Select the topic you just created and set the protocol to either HTTP or HTTPS depending on your setup. The endpoint should be the URL to the SNS REST endpoint, which is exposed at sns in the REST reference documentation. Note that depending on how the API is exposed in your configuration this can differ. In our case, the root REST API is exposed at /api, which means that the SNS endpoint is available at /api/sns
The rest of the settings can be left in their default state.
Wait for a while, until the Status of the subscription changes to Confirmed. This could take a while, but not too long. If it’s not confirming automatically, you can manually trigger a Request confirmation on the subscription.
If you check the server logs, the adapter (ap-backend) will show the confirmed subscription:
2021-03-12 15:58:11,037 INFO [se.cod.ap.ada.pos.api.res.AwsSnsResourceImpl_Subclass] (executor-thread-2) >>Signature verification succeeded
2021-03-12 15:58:11,119 INFO [se.cod.ap.ada.pos.api.res.AwsSnsResourceImpl_Subclass] (executor-thread-2) Subscription confirmation return status: 200
The next step is to connect S3 event notifications, which will send messages to this SNS topic, which are fanned out and delivered to all subscriptions. In order to allow S3 to publish to SNS, we need to set an access policy on the topic itself.
Select your topic, and click access policy. Paste the text below, but modify the resource row to use the ARN of the topic you created.
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "publish-from-s3",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com",
"AWS": "*"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:eu-west-1:381397495928:new-items-added",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:*"
}
}
}
]
}
Navigate to the S3 console, and select the desired bucket that you wish to enable event notification for. Click on Properties, and scroll down to Event notifications.
Click on the create event notification button.
Create a new event notification by giving it a suitable name.
Make sure that for event types, you enable all object create events. You may also include more events here, deleted files for example, if desired.
For destination, make sure you select the SNS topic option. Select the SNS topic you created before, which should appear in the drop-down list.
Save your changes to create the event notification.
The next step is to connect an S3 bucket as storage to Accurate.Video. Refer to the guide below for a detailed walkthrough.
The last step is to make sure auto ingest is enabled on the storage in Accurate.Video. You'll need to set a periodic refresh for the auto ingest setting to be available, but you can set a high refresh delay to avoid polling the storage.
To test the SNS integration, the easiest is to upload a file to the S3 bucket. This will trigger an S3 event notification, which will message the SNS topic, which will in turn message the Accurate.Video SNS endpoint.
Once Accurate.Video receives the notification, the file will be indexed by a background process and shown in the ingest UI immediately. An auto ingest job should be automatically started, and show up in the jobs view.
That’s it! Congratulations on configuring auto ingest with SNS successfully!