Circumventing AWS Lambda's Bundle Size Limit

As of this writing, the AWS Lambda's bundle size limit is 250MB (uncompressed).

This can be a problem if your function depends on large binaries.

The 250 MB limit can be overcome by fetching the binaries from an S3 bucket and caching them in the /tmp directory. Permissions in Lambda forbid writing to any other directory besides /tmp. The contents of /tmp are retained between executions if the interval between executions is < 10 minutes.

This tactic dramatically increases cold-start latency. However, if your lambda function is a worker processing jobs in a queue the increased cold start latency may be acceptable.

Moreover, since AWS Lambda reuses containers, in most cases you will not need to fetch the binary for each invocation. If you have control over the flow or rate of invocations, it may only be necessary to fetch the binary after publishing the function.

Here's an example. Let's say you wanted to run LaTeX in AWS Lambda to convert .tex files to PDFs. You might spin up an EC2 container using the same linux AMI as AWS Lambda, compile the binaries, compress them and put them in an S3 bucket. But since they're too large to upload alongside your Lambda function, you'd resort to the above approach.

const got = require('got')
const unzipper = require('unzipper')
const fs = require('fs')
const { promisify } = require('util')
const { exec } = require('child_process')
const chmod = promisify(fs.chmod)
const list = promisify(fs.readdir)
process.env.PATH += ':/tmp/texlive/2018/bin/x86_64-linux/'
process.env.HOME = '/tmp'
process.env.PERL5LIB = '/tmp/texlive/2018/tlpkg/TeXLive/'
function isInstalled () {
  return new Promise((resolve, reject) => {
    exec('pdflatex --version', (error, stdout, stderr) => {
      if (error) resolve(false)
      else resolve(true)
    })
  })
}
module.exports = async () => {
  const installed = await isInstalled()
  if (!installed) {
    const zip = unzipper.Extract({ path: '/tmp' })
    got.stream('https://s3.amazonaws.com/aws-lambda-binaries/texlive.zip').pipe(zip)
    await new Promise((resolve, reject) => {
      zip.on('close', resolve)
    })
    let files = await list('/tmp/texlive/2018/bin/x86_64-linux/')
    await Promise.all(
      files.map(file =>
        chmod(path.join('/tmp/texlive/2018/bin/x86_64-linux/', file), '0700')
      )
    )
  }
}