How to send gzipped requests with boto3

How to send gzipped requests
with boto3
Luciano Mammino ( )
@loige
AWS UserGroup Dublin
2021-07-01
loige.link/gzip-boto3
1

Get these slides!
loige 2

Let me introduce myself ﬁrst...
3

I'm Luciano ( 🍕🍝) 👋
3

Senior Architect @ fourTheorem (Dublin )
3

nodejsdp.link
Co-Author of Node.js Design Patterns 👉
3

nodejsdp.link
Co-Author of Node.js Design Patterns 👉
Connect with me:

(blog)
(twitter)
(twitch)
(github)
loige.co
@loige
loige
lmammino 3

We are business focused technologists that
deliver.
| |
Accelerated Serverless AI as a Service Platform Modernisation
We are hiring: do you want to ?
work with us
loige 4

loige
In the previous episodes...
SLIC WATCH:
SLIDES:
fth.link/slic-watch
fth.link/o11y-no-pain
5

loige
⚠ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland)
Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was
greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK ->
ALARM transition).
@wake_me_up_bot
6

loige
This guy was failing... a lot!
8

for event in payload['logEvents']:
# ...
cw_client.put_metric_data(...)
loige 10

for event in payload['logEvents']:
# ...
loige
Sending 1 metric per log line... 🙈
10

[
{
'MetricName': 'SomeMetric1',
'Dimensions': [
{
'Name': 'Dimension1Name',
'Value': 'Dimension1Value'
},
# Up to other 9 dimensions here ...
],
'Unit': 'Count',
'Values': [217, 220, 221], # Up to 150 items here ...
'Timestamp': 1624290910000
},
# Up to other 19 metric items here ...
]
loige 11

[
{
'MetricName': 'SomeMetric1',
'Dimensions': [
{
'Name': 'Dimension1Name',
'Value': 'Dimension1Value'
},
# Up to other 9 dimensions here ...
],
'Unit': 'Count',
'Values': [217, 220, 221], # Up to 150 items here ...
'Timestamp': 1624290910000
},
# Up to other 19 metric items here ...
]
loige
⚠Payload size
limit: 40 KB
11

loige
Maybe boto3 gzips the data
automatically! 😏
13

import boto3
import gzip
endpoint_url = "http://localhost:8000/"
cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False)
cw_client.put_metric_data(
MetricData = [
{
'MetricName': 'TEST_BOTO',
'Dimensions': [
{
'Name': 'APP_VERSION',
'Value': '1.0'
},
],
'Unit': 'None',
'Value': 17
},
],
Namespace='BotoTest'
)
loige 14

loige
... and there is no magic ﬂag like
GzipPayload=True! 😭
16

loige
Can we extend boto3 somehow? 🤔
17

loige
boto3 has an event system! 🤩
youtu.be/eM8uoGJO2AI
@thekyleknapp
18

import boto3
lambda_client = boto3.client('lambda')
# our event handler
def add_xtrace_header(request, **kwargs):
request.headers.add_header('x-trace-id', 'trace-trace')
# get the event system for the lambda_client
event_system = lambda_client.meta.events
# attach an event handler to the client for
event_system.register('before-sign.lambda.Invoke', add_xtrace_header)
# invoke a lambda function
lambda_client.invoke(FunctionName='my-function')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
loige 19

import boto3
# our event handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
loige 19

import boto3
# our event handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
# our event handler
import boto3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
loige 19

import boto3
# our event handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
# our event handler
import boto3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
loige 19

import boto3
# our event handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
# our event handler
import boto3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
loige 19

import boto3
# our event handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
# our event handler
import boto3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
loige 19

import boto3
# our event handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
# our event handler
import boto3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
1
2
3
4
# our event handler
5
6
7
8
9
10
11
12
13
14
15
16
import boto3
# our event handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
loige 19

loige
Event naming convention
<event-type>.<service-name>.<operation-name>
20

loige
provide-client-params
request-created
before-sign
before-send
response-received
20

loige
request-created
before-sign
before-send
response-received
s3
cloudwatch
lambda
...
20

loige
request-created
before-sign
before-send
response-received
s3
cloudwatch
lambda
...
ListBuckets
PutMetricData
Invoke
...
20

loige
You can be a 🌟too!

* (every event)
after-call.*.* (all responses)
after-call.lambda.* (all responses for lambda)
21

loige
OK, now we know enough to write
our own event handler for gzip
compression! 🤓
22

import boto3
import gzip
cw_client = boto3.client('cloudwatch')
event_system = cw_client.meta.events
# Gzip handler
def gzip_request_body(request, **kwargs):
request.headers.add_header('Content-Encoding', 'gzip')
gzipped_body = gzip.compress(request.body)
request.data = gzipped_body
# registers the custom handler
event_system.register('before-sign.cloudwatch.PutMetricData',
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
# Gzip handler
import boto3
1
import gzip
2
3
4
5
6
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
# Gzip handler
import boto3
1
import gzip
2
3
4
5
6
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
# Gzip handler
import boto3
1
import gzip
2
3
4
5
6
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
# Gzip handler
import boto3
1
import gzip
2
3
4
5
6
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
gzip_request_body)
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
# Gzip handler
import boto3
1
import gzip
2
3
4
5
6
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
gzip_request_body)
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
15
16
17 cw_client.put_metric_data(...)
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
loige 23

import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import gzip
import boto3
1
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
# Gzip handler
import boto3
1
import gzip
2
3
4
5
6
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
gzip_request_body)
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
15
16
17 cw_client.put_metric_data(...)
import boto3
1
import gzip
2
3
4
5
6
# Gzip handler
7
8
9
10
11
12
13
14
gzip_request_body)
15
16
17
import boto3
import gzip
# Gzip handler
gzip_request_body)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
loige 23

loige
Other potential use cases
Log data before/after it is sent to AWS
Add additional headers to AWS requests (tracing)
Additional input validation (your custom rules)
Payload enrichment (e.g. make sure certain tags exist)
25

loige
What if you don't use Python?
Customizing the AWS SDK for Go V2 Client Requests
Introducing Middleware Stack in Modular AWS SDK for
JavaScript
Handlers and Middleware in the AWS SDK for PHP Version 3
26

loige
What did we learn?
Always have alarms for your lambdas!
(have you checked already? 😉)
You can use Gzip compression to send big payloads to AWS
Boto3 is extensible through its event system!
SLIC Watch
27

loige
Do you think boto3 should make it easier to
send gzipped payloads?
👍UPVOTE this issue: github.com/boto/botocore/issues/2425
28

loige
Do you want more details?
loige.link/gzip-boto3-article
29

Cover picturea by on
Thanks doggie by Stefanie Shank
Moritz Mentges Unsplash
loige
nodejsdp.link
30

How to send gzipped requests with boto3

Related slideshows

More Related Content

What's hot

What's hot (20)

Similar to How to send gzipped requests with boto3

Similar to How to send gzipped requests with boto3 (20)

More from Luciano Mammino

More from Luciano Mammino (20)

Recently uploaded

Recently uploaded (20)

How to send gzipped requests with boto3