Utilizing Amazon EventBridge to trigger ECS tasks from S3 events by using AWS CDK
In this tutorial, I will explain how we can use Amazon EventBridge to trigger an ECS task from the events generated by S3; without using Lambda functions.
This approach will be useful for a system aligned toward event-driven architecture which desires fewer service maintenance (lambda functions)
As a precondition; I will assume there is an ECS task already created as it is beyond the scope of this tutorial.
Enable event notifications from S3
As the first step, we should enable event notifications from the S3 bucket of our interest.
If you have a bucket created already:
From Amazon Console, Go to S3 services, then search or select the bucket of your choice. Then select the properties tab from the header.
Scroll down to the Amazon Eventbridge section and click on Edit
Choose On and Save Changes.
By doing so, we will be sending all the notifications related to the S3 to Eventbridge. It is important to consider the pricing at this point depending on the usage of the S3 bucket.
If you are creating the bucket using the CDK:
class BucketStack extends Stack {
constructor(scope, id, props) {
super(scope, id, props);
const bucket = new s3.Bucket(this, 'MyBucket', {
...,
eventBridgeEnabled: true
});
}
}EventBridge Rule
Now we a have bucket that sends events to the Eventbridge, next step is to define a rule that accepts events from the bucket.
Let’s define a rule.
...
import * as events from 'aws-cdk-lib/aws-events';
class EventBridgeStack extends Stack {
constructor(scope, id, props) {
super(scope, id, props);
const rule = new events.Rule(this, 'EventBridgeRule', {});
}
}We have defined an empty rule. Let’s fill in the details.
Event pattern:
We need to listen to the S3 event, therefore we need to include an eventPattern in the rule.
...
const rule = new events.Rule(this, 'NewsToActivityEventBridgeRule', {
eventPattern: {
source: ['aws.s3'],
detailType: ['Object Created'],
detail: {
bucket: {
name: ['my-bucket'],
}
},
},
});- source: The source(s) of the event. It can be any AWS service, here it is S3. (ex: aws.ec2, aws.fargate, …)
- detailType: Types of the event(s) the rule will listen to. This rule will only listen to Object created. Here are all the S3-related events.
- detail: Specific filtration related to event type. Depending on the type of event, we can further filter by using this property. Below is an example of the various options available for S3 events.
detail: {
bucket: {
name: ['my-bucket'],
}
object: {
key: [
{ wildcard: "test/*.png" },
{ prefix: "test/" },
{ suffix: ".png" },
],
size: [{numeric :["<=", 1048576 ] }]
}
}Target:
The next piece of the puzzle is the target which is a ECS task in our example (This can be ECS on EC2 or Fargate).
We can either do
...
import * as targets from 'aws-cdk-lib/aws-events-targets';
const rule = new events.Rule(this, 'NewsToActivityEventBridgeRule', {
eventPattern: { ... },
targets: [
new targets.EcsTask({ ... })
]
});or
...
import * as targets from 'aws-cdk-lib/aws-events-targets';
const rule = new events.Rule(this, 'NewsToActivityEventBridgeRule', {
eventPattern: { ... }
});
rule.addTarget( new targets.EcsTask({ ... }) );To create an EcsTask, we must pass cluster and taskDefinition.
rule.addTarget( new targets.EcsTask({
cluster,
taskDefinition
}));If the cluster was not created in the same stack, we can use fromClusterArn or fromClusterAttributes methods to get the cluster.
Similarly, for taskDefintion, we can use fromTaskDefinitionArn or fromTaskDefinitionAttributes methods.
and that’s all! We now have a way to trigger ECS tasks when an object is created in S3 without a lambda function.
Wait!!
Our ECS does not know anything about the S3 event, not even the file that triggered the event. It is a common use case where we want the ECS task to process the newly created file. As the final step let’s see how we can achieve this.
Accessing the properties available in the Input event
events.EventField.fromPath('$.<field>...')We can use the above method to get the field value. Here are a few example event bodies. Say we want to get the S3 object name.
events.EventField.fromPath('$.detail.object.key')Let’s assume we are passing the created object/file name for the ECS task, we can do this by containerOverrides property in the ECS target. In the below example, we are passing the file name as the environment variable S3_FILE_PATH for the container.
new targets.EcsTask({
cluster,
taskDefinition,
containerOverrides: [
{
containerName: '<name given to the ECS task container>',
environment: [
{
name: 'S3_FILE_PATH',
value: events.EventField.fromPath('$.detail.object.key'),
},
],
},
],
}),Permissions:
The CDK role that is responsible for creation and updating stack must have the necessary permission to create/update/delete EventBridge and other associated services. Here you can find some common issues and how to troubleshoot them.
References:
