AWS, GO

Seamless Cross-Account Integration of AWS Lambda Functions with SNS

User Icon By Azam Akram,   Calendar Icon July 24, 2023
Cross Account AWS Lambda Functions with SNS

Amazon SNS (Simple Notification Service) fully manages messaging, enabling applications, services, and devices to send and receive notifications. It allows you to publish messages to a topic, which the subscribers of that topic can process, such as email, SMS, or other AWS services like Lambda. This makes it easy to set up communication between different parts of your system. Integrating cross account aws lambda functions with SNS empowers the creation of seamless communication via SNS topics, enhancing distributed event-driven architectures.

In this blog, I'll guide you step-by-step on integrating cross account AWS Lambda functions with SNS. We'll create Lambda functions in Go that publish and subscribe to events on an SNS topic. Additionally, I'll walk you through the CloudFormation stack setup, which includes the necessary resources like Lambda functions, the SNS topic, and IAM roles and policies.

Before We Proceed

I'll assume you're already familiar with creating and deploying AWS Lambda functions and SNS integration. But don't worry, if you need a refresher, I recommend reading following very informative articles,

Application Call Flow

To illustrate the process of creating cross account AWS Lambda functions, we will generate two individual functions: "calculation-service-lambda" and "calculation-requester-lambda", we deploy these functions in separate AWS accounts using a cloudformation template. In the first AWS account (let's call it AWS-Account-1), we set up an SNS topic called "calculation-service-sns-topic" along with the lambda function "calculation-service-lambda". Using second AWS account (let's name it AWS-Account-2), we create another lambda function named "calculation-requester-lambda". We provision both lambda functions to subscribe and publish different events to the SNS topic.

The call flow between both lambda function is as follows:

  1. Initially, we manually publish a "StartingEvent" to an SNS topic in AWS-Account-1. The "StartingEvent" contains an array of integers that need to be summed.
  2. The lambda function "calculation-requester-lambda" in AWS-Account-2 receives that event.
  3. The "calculation-requester-lambda" processes the received event by making certain modifications, such as changing the event name to "SumRequested" and adding a timestamp. It then publishes the "SumRequested" event to the SNS topic, located in AWS-Account-1.
  4. The "calculation-service-lambda" in AWS-Account-1 receives the "SumRequested" event, and calculates the sum of the integer array and returns the result in a "SumCompleted" event.
  5. Subsequently, the "calculation-requester-lambda" receives the "SumCompleted" event and logs the information on the console.

Calculation Service Lambda

Complete Source Code: calculation-service-lambda

The function, HandleRequest, is triggered by an SNS event named SumRequested. It proceeds to calculate the sum of the numbers provided in the event's payload and after calculating the sum, the function prepares a new event "SumCompleted" and publish on SNS topic.

func HandleRequest(ctx context.Context, snsEvent events.SNSEvent) error {
	fmt.Println("SNS event received:", snsEvent)

	for _, record := range snsEvent.Records {
		var event model.Event
		if err := json.Unmarshal([]byte(record.SNS.Message), &event); err != nil {
			return fmt.Errorf("error in unmarshaling JSON: %w", err)
		}

		fmt.Println("Unmarshalled Event:", event)

		if event.Name != "SumRequested" {
			// Ignore events other than SumRequested
			return nil
		}

		event.Name = "SumCompleted"
		event.Source = "Calculation Service"
		event.EventTime = time.Now().Format(time.RFC3339)

		sum := 0
		for _, num := range event.Payload.Numbers {
			sum += num
		}
		event.Payload.Sum = sum

		fmt.Println("Event to publish: ", event)

		if _, err := utils.PublishEvent(context.Background(), event); err != nil {
			return fmt.Errorf("error publishing event: %w", err)
		}
	}

	return nil
}

func main() {
	lambda.Start(HandleRequest)
}

Cloudformation template for Calculation-Service-Lambda

We deploy the lambda function calculation-service-lambda in AWS-Account-1 by following cloudformation template.

---
AWSTemplateFormatVersion: '2010-09-09'
Description: A cloudformation template to create a calculation service lambda functions and SNS topic
Parameters:
  pSnsTopicName:
    Type: String
  pOtherAccountID:
    Type: String
  pLambdaCodeBucket:
    Type: String
  pCalServiceCodeS3KeyPath:
    Type: String
Resources:
  CalculationSNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: !Ref pSnsTopicName
  TopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      Topics:
        - !Ref CalculationSNSTopic
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AllowSubscriptionFromOtherAccount
            Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${pOtherAccountID}:root"
            Action: 
            - SNS:Subscribe
            - SNS:Publish
            Resource: !Ref CalculationSNSTopic
  lfnLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
      Policies:
      - PolicyName: lambdaCloudWatchPolicy
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Resource: "*"
      - PolicyName: snsPublish
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - SNS:Publish
            Resource:
              Fn::Sub: arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${pSnsTopicName}
  lfnCalcService:
    Type: AWS::Lambda::Function
    DependsOn:
    - lfnLambdaRole
    Properties:
      Environment:
        Variables:
          SNS_TOPIC_ARN:
            Fn::Sub: arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${pSnsTopicName}
      Architectures:
      - x86_64
      Runtime: go1.x
      Handler: main
      Timeout: '120'
      Code:
        S3Bucket:
          Ref: pLambdaCodeBucket
        S3Key:
          Ref: pCalServiceCodeS3KeyPath
      Description: This is calculation service lambda function
      FunctionName: calculation-service-lambda
      Role:
        Fn::GetAtt:
        - lfnLambdaRole
        - Arn
  snsPermInvokeCalcService:
    Type: AWS::Lambda::Permission
    DependsOn:
    - lfnCalcService
    Properties:
      Action: lambda:InvokeFunction
      FunctionName:
        Fn::GetAtt:
        - lfnCalcService
        - Arn
      Principal: sns.amazonaws.com
      SourceArn:
        Fn::Sub: arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${pSnsTopicName}
  snsSubscriptionCalcService:
    Type: AWS::SNS::Subscription
    DependsOn:
    - lfnCalcService
    Properties:
      Endpoint:
        Fn::GetAtt:
        - lfnCalcService
        - Arn
      FilterPolicy:
        name:
        - SumRequested
      Protocol: lambda
      TopicArn:
        Fn::Sub: arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${pSnsTopicName}
Outputs:
  SNSTopicARN:
    Description: SNS Topic ARN
    Value: !Ref CalculationSNSTopic

Explanation

The template offers flexibility by using parameters to customize key attributes:

pSnsTopicName parameter sets the name of the SNS topic to be created, while pOtherAccountID allows cross account communication. pLambdaCodeBucket parameter is used to specify the S3 bucket containing the Lambda function code, and pCalServiceCodeS3KeyPath points to the path within the bucket.

The resources section consists of the creation of the SNS topic, IAM roles, and Lambda functions. TopicPolicy resource configures the IAM policy allowing the specified pOtherAccountID to subscribe and publish events to the SNS topic. The lfnLambdaRole resource defines an IAM role for the Lambda function, granting it permission to access CloudWatch logs and publish to SNS.

The lfnCalcService resource creates the Lambda function named "calculation-service-lambda" using Go runtime with a specified S3 bucket and code path. It is granted permission to invoke the Lambda function by the snsPermInvokeCalcService resource, which configures the required permissions for SNS to invoke the Lambda function. Finally, the snsSubscriptionCalcService resource creates a subscription for the Lambda function to the SNS topic, filtering events with the "SumRequested" name.

The CloudFormation outputs the ARN (Amazon Resource Name) of the newly created SNS topic. The calculation-service-lambda in other AWS account will use this topic ARN to subscribe and publish event.

We can now build the lambda function application, package it in zip format and upload to S3 bucket.

GOOS=linux go build main.go
build-lambda-zip.exe -o calculation-service-lambda.zip main
aws s3 cp calculation-service-lambda.zip s3://<<bucket-name>>/<<s3-prefix>>/calculation-service-lambda.zip

To deploy calculation-service-lambda in AWS-Account-1,

aws cloudformation deploy \
--template-file ./deploy/cloudformation.yml \
--stack-name calculation-service-lambda-stack \
--capabilities CAPABILITY_IAM \
--parameter-overrides \
    pSnsTopicName=calculation-service-sns-topic \
    pOtherAccountID={{AWS-Account-2}} \
    pLambdaCodeBucket=azam-test-s3-bucket \
    pCalServiceCodeS3KeyPath=lambda-external-sns-go/calculation-service-lambda.zip

This should create and deploy all required resources in AWS-Account-1 as,

The CloudFormation stack generates the SNS topic ARN as an output, which we will be using to link other lambda function in the AWS-Account-2.

Calculation Requester Lambda

Complete Source Code: calculation-requester-lambda

"calculation-requester-lambda" acts as our second Lambda function, getting activated when an event named "StartingEvent" is published on an SNS topic in an external AWS account. Upon receiving this event, the function prepares a new event called "SumRequested." It then calculates the sum of an integer array and includes it in event published to external SNS topic.

func HandleRequest(ctx context.Context, snsEvent events.SNSEvent) error {
	log.Println("SNS event received:", snsEvent)

	for _, record := range snsEvent.Records {
		var event model.Event
		err := json.Unmarshal([]byte(record.SNS.Message), &event)
		if err != nil {
			log.Println("Error unmarshaling JSON:", err)
			return err
		}
		switch event.Name {
		case "SumCompleted":
			if event.Source == "Calculator" {
				log.Println("Answer received:", event.Payload.Sum)
				return nil
			}
		case "StartingEvent":
			event.Name = "SumRequested"
			event.Source = "Calculation Requester"
			event.EventTime = time.Now().Format(time.RFC3339)

			log.Println("Event to publish: ", event)

			if _, err := utils.PublishEvent(context.Background(), event); err != nil {
				return fmt.Errorf("error publishing event: %w", err)
			}

			return nil

		default:
			log.Println("Unknown event, ignoring this..")
		}
	}

	return nil
}

func main() {
	lambda.Start(HandleRequest)
}

Cloudformation for Calculation-Requester-Lambda

Now we see the CloudFormation template to create a "Calculation Requester" stack that comprises AWS Lambda functions.

The template accepts different input parameters, such as pSnsTopicArn parameter specifies the ARN of the existing SNS topi, we noted it down as output of first cloudformation stack in previous section.

The template creates an IAM role, lfnLambdaRole, granting permissions to the Lambda function for creating CloudWatch logs and publishing to SNS. The lfnCalcRequester resource sets up the Lambda function named "calculation-requester-lambda" using the Go runtime.

The snsPermInvokeCalcRequester resource configures the necessary permissions for SNS to invoke the Lambda function, and the snsSubscriptionCalcRequester resource creates a subscription for the Lambda function to the SNS topic, filtering events with the "SumCompleted" and "StartingEvent" names.

---
AWSTemplateFormatVersion: '2010-09-09'
Description: A cloudformation template to create a calculation requester lambda functions and SNS topic
Parameters:
  pSnsTopicArn:
    Type: String
  pLambdaCodeBucket:
    Type: String
  pRequesterCodeS3KeyPath:
    Type: String
Resources:
  lfnLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
      Policies:
      - PolicyName: lambdaCloudWatchPolicy
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Resource: "*"
      - PolicyName: snsPublish
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - SNS:Publish
            Resource:
              Ref: pSnsTopicArn
  lfnCalcRequester:
    Type: AWS::Lambda::Function
    DependsOn:
    - lfnLambdaRole
    Properties:
      Environment:
        Variables:
          SNS_TOPIC_ARN:
            Ref: pSnsTopicArn
      Architectures:
      - x86_64
      Runtime: go1.x
      Handler: main
      Code:
        S3Bucket: !Ref pLambdaCodeBucket
        S3Key: !Ref pRequesterCodeS3KeyPath
      Description: This is calculation requester lambda function
      FunctionName: calculation-requester-lambda
      Role:
        Fn::GetAtt:
        - lfnLambdaRole
        - Arn
      Timeout: '120'
  snsPermInvokeCalcRequester:
    Type: AWS::Lambda::Permission
    DependsOn:
    - lfnCalcRequester
    Properties:
      Action: lambda:InvokeFunction
      FunctionName:
        Fn::GetAtt:
        - lfnCalcRequester
        - Arn
      Principal: sns.amazonaws.com
      SourceArn:
        Ref: pSnsTopicArn
  snsSubscriptionCalcRequester:
    Type: AWS::SNS::Subscription
    DependsOn:
    - lfnCalcRequester
    Properties:
      Endpoint:
        Fn::GetAtt:
        - lfnCalcRequester
        - Arn
      FilterPolicy:
        name:
        - SumCompleted
        - StartingEvent
      Protocol: lambda
      TopicArn:
        Ref: pSnsTopicArn

Follow instructions in the previous section to build and deploy the CloudFormation stack. On success, CloudFormation will create the required resources, including IAM roles and policies specified in the template.

We can see calculation-requester-lambda in AWS-Account-2 has successfully subscribed to SNS topic in AWS-Account-1.

Note: If you are unfamiliar with the process of building and deploying Lambda functions, you can refer to this comprehensive guide. It will provide step-by-step instructions on how to create and deploy Lambda functions.

Publishing event on SNS topic

Following code defines a set of utility functions to publish an SNS event using the AWS SDK for Go language. The PublishEvent function takes a context.Context and a model.Event as input, where the event represents a message to be sent to the SNS topic. The function first reads the AWS region and SNS topic ARN from environment variables, and then creates an AWS session and SNS client.

You can download the code from this git repository link.

func getAWSRegion() string {
	region := os.Getenv("AWS_REGION")
	fmt.Println("AWS region: ", region)
	return region
}

func getSNSTopicARN() string {
	topicArn := os.Getenv("SNS_TOPIC_ARN")
	fmt.Println("SNS Topic ARN: ", topicArn)
	return topicArn
}

func PublishEvent(_ context.Context, event model.Event) (msgId string, err error) {
	region := getAWSRegion()
	awsConfig := &aws.Config{
		Region: &region,
	}
	snsSession, err := session.NewSession(awsConfig)
	if err != nil {
		return "", err
	}
	snsClient := sns.New(snsSession)
	eventBytes, err := json.Marshal(event)
	if nil != err {
		return "", err
	}
	payload := string(eventBytes)
	snsInput := &sns.PublishInput{
		Message:  aws.String(payload),
		TopicArn: aws.String(getSNSTopicARN()),
		MessageAttributes: map[string]*sns.MessageAttributeValue{
			"name": {
				DataType:    aws.String("String"),
				StringValue: aws.String(event.Name),
			},
		},
	}
	snsMsg, err := snsClient.Publish(snsInput)
	if err != nil {
		return "", err
	}
	fmt.Println("Published event: ", snsMsg)
	return *snsMsg.MessageId, nil
}

Triggering Lambda functions

As mentioned previously we can trigger the call flow by manually publishing the following event on SNS topic created in AWS-Account-1.

Both Lambda functions publish and exchange events. The following is a snippet from the CloudWatch logs of the calculation-requester-lambda function.

And following is a snippet from the CloudWatch logs of the calculation-service-lambda function.

Conclusion

SNS events play a vital role in fostering seamless interactions and communication between various AWS services, particularly lambda functions. Acting as a powerful mechanism, SNS events enable different services to trigger and respond to events, creating a distributed and event-driven architecture. For example, a lambda function triggers a task by publishing an event on an SNS topic. While another function in a separate AWS account may processes the event and generates results. The capability of AWS to facilitate cross account AWS Lambda functions, working through SNS topics, empowers the creation of robust and scalable distributed systems.

Throughout this article, we went on a step-by-step journey to create cross account AWS Lambda functions, each residing in its dedicated AWS account. Using the Go programming language, we explored how to publish and subscribe to events on an SNS topic. CloudFormation streamlined the process, automatically provisioning essential resources like Lambda functions, SNS topics, IAM roles, and policies.