aws-video-metadata-knowledge-graph-workshop

Video metadata extraction and knowledge graph

This repository contains a series of 4 jupyter notebooks demonstrating how AWS AI Services like Amazon Rekognition, Amazon Transcribe and Amazon Comprehend can help you extract valuable metadata from your video assets and store that information in a Graph database like Amazon Neptune for maximum query performance and flexibility. At the end of the workshop you'll typically be able to search for a specific label or entity and return a list of 1min video segments related to your search across your videos.

To extract metadata from a video, we'll use a the following AWS AI services:

• Amazon Rekognition to cut the video in scenes and detect label from the video itself
• Amazon Transcribe to convert audio into text
• Amazon Comprehend to extract entities and topics from the transcribed text via Topic Modelling and Named Entity recognition.

The metadata related to the video, segments, scenes, entities, labels will be stored in Amazon Neptune. Amazon Neptune is a fully managed low latency graph database service that will allow us to store metadata as nodes (aka vertices) and branches (aka edges) to represent relationships between the nodes. https://aws.amazon.com/neptune/

The diagram below summarises the workflow:

Topics addressed within the different notebooks:

Part 0: Create the environment (S3 bucket, IAM roles/polices, SNS topic, etc) and upload your sample video

Part 1: Use Amazon Rekognition to detect scenes and labels from your video

Part 2: Use Amazon Transcribe and Amazon Comprehend to respectively transcibe audio to text and extract metadata (topics, Named Entities) from transcripts.

Part 3: Store all the previously extracted metadata in Amazon Neptune and query the graph.

Part 4: Resources clean-up

Getting started

To run those notebooks you'll need to create a jupyter notebook instance in sagemaker.

In the AWS console, first make sure you're in the right region and search for Sagemaker. Write down the region as you'll later need to create your Amazon Neptune database in the same region.

In Amazon Sagemaker, click on "Notebook instances".

In the "Create notebook instance wizard", enter the following:

For Permissions, click on "create a new role".

Then specify an existing S3 bucket where you will later upload the .mp4 video sample to be used in the notebooks.

In Network, specify a VPC, subnet and a security group. I used the default ones on my end. Write down the VPC name as you'll need to deploy your Amazon Neptune DB in the same VPC later in notebook part0 and make sure your security group allows traffic between the two.