Searching ELB Access Logs With Lumberjack

Most people using AWS EC2 to host something use AWS Elastic Load Balancing for easy, cheap, and config-less load balancing. Since most of your traffic will pass through the ELB, it makes sense to enable access logs so you can refer to them when debugging requests to your services. AWS provides a simple setup for collecting access logs and storing them in S3, but there's basically no easy way to search those logs for specific information.

At Blue Matador, we created an integration on our Lumberjack product to make it easy to include access logs with the rest of the logs you have in Lumberjack, which makes them easily searchable. Our integration currently supports logs for the following AWS components:

  • Application Load Balancer (ELB) Logs
  • Classic Load Balancer (ELB) Logs
  • CloudFront RTMP Distribution Logs
  • CloudFront Web Distribution Logs
  • S3 Server Access Logs

Below I will describe how to set up ELB access logs in the AWS console in case you have never done it before, and then I will explain how to set up the AWS integration to get the logs in Lumberjack.


Configuring ELB Access Logs

This tutorial assumes you have already created either a Classic Load Balancer or Application Load Balancer and access to the AWS Web Console.

First, we will select the ELB that we want to configure logs for. Then click the "Configure Access Logs" button under the "Attributes" section of the detail view.

elb1.png

Check "Enable access logs," then select the Interval and S3 location to store logs in. I use a 5-minute interval so that the logs arrive more often, which is helpful when debugging traffic issues. The S3 location can be either an existing S3 bucket/prefix that has permissions for logging set up, or you can enter a new bucket and check the "Create this location for me" box to have all the permissions handled for you. When you are done click "Save".

elb2.png

If everything worked correctly, you should be able to head over to the S3 console and confirm that the test file was created by AWS. The location of the file will be <your bucket/prefix>/AWSLogs/<your account id>/ELBAccessLogTestFile.  If the ELB has received any requests since logging was enabled, then you should start seeing more files stored in S3 after 5 minutes or so, which contain the access logs we are interested in.

elb3 (1).png

For a more detailed explanation on enabling access logs, including how to set up the S3 permissions for existing buckets, visit the AWS Documentation.


Lumberjack's AWS Integration

Now that we have an ELB set up with access logs going to S3, let's configure Lumberjack to automatically read those logs and store them alongside our other logs. Our AWS Integration uses AWS Lambda to read, parse, and send the logs to our servers as soon as they show up in S3. The Lambda function runs in your AWS account, which means you are responsible for any of the fees incurred from running the function. Most implementations fall in the free tier for AWS Lambda though. Full pricing details for AWS Lambda can be found here.

First we should head to the Lumberjack Sources Page to see our existing Lumberjack sources. When there, just click the "AWS Watches" tab then the "Add AWS Watch" button.

elb4.png

Now you should see the "Add AWS Watch" form.  The following fields are needed to configure your account for the Lumberjack AWS integration and start reading the logs from S3.

Bucket: This is the name of the S3 bucket that logs are stored in. In our example above we used "bluematador-test-classic-elb-logs"

Prefix: This is the path in S3 that logs are stored at. This setting is optional, and leaving it blank use the entire bucket. This setting is necessary if you have multiple logs of different types going into the same S3 bucket. An example value could be "AWSLogs/111111111/myelb"

Format: The format tells Lumberjack how to parse the log entries found in S3. We have prebuilt formats for each of the currently supported AWS Services. For this example I wlil be using "Classic Load Balancer". Need to ingest logs for another sevice? Contact Us.

Tags: Tags are the key-value pairs that will allow you to search across multiple Lumberjack Sources. For this example I will set mine to "env:test service:aws elb:test-classic-elb"

After filling out the form, click "Next" to continue with the setup.

elb5.png

The final step to setting up your AWS Integration is to set up an IAM role that will be used to run the Lambda function and manage the integration.  Our integration is designed to minimize the amount of manual effort on your part. There is no manually uploading code to AWS Lambda, setting up complicated configuration, or worrying about updates -- we handle it for you!  In order to accomplish this, we need an IAM role in your account, that Blue Matador's account can use to configure AWS Lambda and S3.  We only ask for the exact permissions we need for the Lumberjack AWS integration to function, and never modify AWS resources outside of the scope of the integration. 

I prefer to use a single IAM role for all of Lumberjack AWS Watches, with a separate IAM policy for each watch. This makes it easy to audit the access level of our integration, and gives you the flexibility to use several S3 locations for your log files without granting broad permissions to Blue Matador. Head back over to the AWS console, go the IAM section, then Roles, then click "Create role".

elb6.png

On the Crete role screen, select "AWS Service" as the trusted entity, then "Lambda" as the service in the list below. Finally, click the Next button to head over to permissions.

elb7.png

Since we have a custom policy, click the "Create Policy" button, which should open a new window. Then Click "Select" under "Create Your Own Policy"

elb8.png

Fill out the Policy Name and Description fields, then copy the policy document from the Add AWS Watch screen.  Then click "Create Policy".

elb9.png

elb10.png

Return to the Create Role screen in AWS (which should have a section called "Attach permissions policies"), Refresh the policies list and select your newly created policy from the previous step.  Then click "Next: Review".

elb11.png

Now we enter arole name and description, and click "Create role".

elb12.png

Now the role is created with permissions to set up S3 bucket notifications, update Lambda, and read S3 objects in the S3 bucket you specified. The next thing we need to do is give access to the Blue Matador AWS account to assume this role so that you do not have to manually configure the Lambda function. To do this, select the newly created role in the role list, click the "Trust relationships" tab, and then click "Edit trust relationship".

elb13.png

Copy and paste the Trust Relationship JSON from the "Add AWS Watch" form and replace the entire existing JSON text. Then, click "Update Trust Policy".  Updating the trust relationship is only required once per IAM role you have configured with Lumberjack.  If you are using one role for all of your AWS Watches in Lumberjack, then you only have to do this step once!. Afterwards, you should see multiple lines in the "Trusted entities" area of the "Trust relationships" tab.

elb14.png

elb15.png

Now that the role is created with the correct policy and trust relationship, all we need to do is copy the Role ARN from the AWS console into the Add AWS Watch form and hit Submit.

elb16.png

And that's it! Now you can use the Lumberjack Log Explorer to search your AWS logs, and you can set up powerful Lumberjack Alerts that trigger on the contents of those logs.

PHIL SHOULD ADD SOME MARKETING FLUFF HERE

 

Picture of Keilan Jackson

Keilan Jackson

Author Bio

Keilan’s specialty is bringing development, quality assurance, and operations together for efficient delivery and maintenance of software. He believes that functional programming, typed JavaScript, and Docker make the world a better place. His free time is spent sampling Utah’s amazing micro brews and co-parenting cats Artemis and Queue with his boyfriend.

What is Blue Matador?

Blue Matador is the AI-powered DevOps monitoring platform that provides real-time, predictive alerts that help your team decrease downtime and increase customer confidence in your brand. Learn more

Our Monitoring Products

Watchdog is the free server monitor that sends you and your team proactive system vitals alerts, proactively notifying you of all the metrics you need to know to prevent downtime. Install for free


Lumberjack is the AI-powered centralized log management tool that proactively warns your DevOps team of impending server and app issues that affect uptime. Try free for 14 days