Reduce AWS Lambda Latencies with Keep-Alive in Python

It starts, as many stories do, with a question. On September 10th, AWS Serverless Hero Luc van Donkersgoed shared his observations on the relationship of reduced latency with increased request rate when using AWS Lambda. This is always an interesting conversation, and sure enough other AWS Heroes like myself are curious about some of the outlier behaviors, and what exactly is going into each request. AWS Data Hero Alex DeBrie, AWS Container Hero Vlad Ionescu both ask excellent questions about the setup and the behaviors, leading Luc to share what he’s seeing with regards to DNS lookups that don’t make sense to him.

After asking a couple of more questions of my own, I rolled up my sleeves and dug into the what, how, and why.

getting ready to read things and hit them with sticks

I dive in to all parts of the stack in use to try and understand why Luc’s code is seeing DNS lookups.
For example, if your function needs to call AWS S3 or a Twilio API, we usually provide the domain name, and have the code or library perform a request to a Domain Name System (DNS) server to return the current IP address, and then communicate using the IP address. This is a network call and can be expensive (in milliseconds) if it’s performed more frequently than the DNS response’s Time To Live (TTL) – kind of like an expiration date. The DNS lookup adds some more latency to your overall call, which is why many systems will cache DNS responses until the TTL is expired, and then make a new call. If you perform DNS lookups when not needed, that’s adding latency unnecessarily. Read the tweet thread for more!

I arrive at two possible solutions:

  1. If the Python code calls more than 10 AWS service endpoints, it will trigger a DNS lookup, as urllib3‘s PoolManager will only maintain 10 connections (set by botocore defaults) and will need to recycle if exceeded.
  2. Since we’re unlikely to be hitting the limit of 10, something else is at play.
    I found that the default behavior of boto3 is to not use Keep Alive, thus explaining why the occasional connection is reset, triggering a DNS lookup. (Read the tweet thread for the full discovery.)

Using Keep-Alive is nothing new, and was covered quite well by AWS Serverless Hero Yan Cui back in 2019 for Node. It’s even in the official AWS Documentation, citing Yan’s article for the proposed update. Thanks Yan!


There’s precious little literature on using Keep Alive for Python Lambdas that I could find, leading to issues like Luc’s and reports like this one, so I decided to dig a little further. Knowing now that the default for Keep Alive is off by default for users of the popular boto3 package to interact with AWS services, I wanted to explore what that looks like in practice.

I decided to pattern an app after Yan’s example – a function at receives an event body, and persists it to DynamoDB. All in all, not a too complex operation – we perform a single DNS lookup for the DynamoDB service endpoint, and then use the response IP address to connect over HTTP to put an object into the DynamoDB table.

After re-writing the same function in Python, I was able to test the same kind of behavior that Yan did, running a call to the function once per second, isolating any concurrency concerns, replicating Luc’s test. This should have the benefit of reusing the same Lambda context (no cold starts) and seeing that the latencies range from 7 to 20 milliseconds for the same operation:

filtered log view showing only the latency for put_item calls to DynamoDB for 30 seconds

So far, so good – pretty much the same. The overall values are lower than Yan’s original experiment, which I attribute to the entire Lambda ecosystem improving, but we can see there’s variance and we often enter double-digit latencies, when we know that the DynamoDB operation is likely to only take 6-7 milliseconds.

left side shows spiky responses; right side shows most responses are fast, with some slower outliers

As Yan showed in his approach adapted from Matt Levine’s talk snippets, he was able to reconstruct the AWS Config by rebuilding the lowest-level HTTP agent that the library relies on to make the calls, and thereby set the behavior for Keep Alive. This has since been obsoleted by the AWS Node.JS SDK adding an environment variable to enable the keep alive behavior, which is awesome! But what about Python? 🐍

In the recent release of botocore 1.27.84 we can modify the AWS Config passed into the client constructor:

# before:
import boto3
client = boto3.client("dynamodb")

# after:
import boto3
from botocore.config import Config
client = boto3.client("dynamodb", config=Config(tcp_keepalive=True))

With the new configuration in place, if you try this on AWS python3.9 execution runtime, you’ll get this error:
[ERROR] TypeError: Got unexpected keyword argument 'tcp_keepalive'

While the AWS Python runtime includes versions of boto3 and botocore, they do not yet support the new tcp_keepalive parameter – the runtime currently ships:
– boto3 1.20.32
– botocore 1.23.32

So we have to solve another way.

The documentation tells us that we can configure this via a config file in ~/.aws/config, added in version 1.9.17 back in October 2018 – presumably when all the Keep Alive conversations were fresh in folks’ minds.

However, since the Lambda runtime environment disallows writing to that path, we can’t write the config file easily. We might be able to create a custom Docker runtime and place a file in the path, but that’s a bit harder, and we lose some of the benefits of using the AWS prebuilt runtime like startup latency, which when we’re exploring a latency-oriented article, seems like the wrong choice 😁.

Using serverless framework CLI with the serverless-python-requirements (what I’m currently using), or AWS SAM, you can add the updated version of boto3 and botocore, and deploying the updated application allows us to leverage the new setting in a Lambda environment. You may already be using one of these approaches for a more evolved application.
Hopefully 🤞 the Lambda Runtime will be updated to include these versions in the near future, so we don’t have to package these dependencies to get this specific feature.

With the updated packages, we can pass the custom Config with tcp_keepalive enabled (as shown above), and observe more constant performance for the same style of test:

left: much smoother!! right: narrower distribution of values, max 8.50 ms

There’s an open request for the config value to be available via environment variable – check it out and give it a 👍 to add your desire and subscribe via GitHub notifications.

Enjoy lower, more predictable latencies with Keep Alive!

Check out the example code here: https://github.com/miketheman/boto3-keep-alive


Postscript: If you’re interested in pinpointing calls for performance, I recommend checking out Datadog’s APM and associated ddtrace module to see the specifics of every call to AWS endpoints and associated latencies, as well as other parts of your application stack. There’s a slew of other vendors that can help surface these metrics.

Container-to-Container Communication

Question ❓

In a containerized world, is there a material difference between communicating over local network TCP vs local Unix domain sockets?

Given an application with more than a single container that need to talk to each other, is there an observable difference in latency/throughput when using one inter-component communication method over another from an end-users’ perspective?


Background 🌆

There’s this excellent write-up on the comparison back in 2005, and many things have changed since then, especially around the optimizations in the kernel and networking stack, along with the container runtime that is usually abstracted away from the end user’s concerns. Redis benchmarks from a few years ago also point out significant improvements using Unix sockets when the server and benchmark are co-located.

There’s other studies out there that have their own performance comparisons, and produce images like these – and every example is going to have its own set of controls and caveats.

I wanted to use a common-ish scenario: a web service running on cloud infrastructure I don’t own.

Components 🧩

For the experiment, I chose this set of components:

  • nginx (web server) – terminate SSL, proxy requests to upstream web server
  • gunicorn (http server) – speaks HTTP and WSGI protocol, runs application
  • starlette (python application framework) – handle request/response
components

I considered using FastAPI for the application layer – but since I didn’t need any of those features, I didn’t add it, but it’s a great framework – check it out!

As gunicorn server runs the starlette framework and the custom application code, I will be referring to them as a single component later as "app", as the tests I’m comparing is the behavior between nginx and the "app" layer, using overall user-facing latency and throughput as the main result.

nginx 🌐

nginx is awesome. Really powerful, and has many built-in features, highly configurable. Been using it for years, and it’s my go-to choice for a reliable web server.

For our purposes, we need an external port to listen for inbound requests, and a stanza to proxy the requests to the upstream application server.

You might ask: Why use nginx at all, if Gunicorn can terminate connections directly? Well, there’s often a class of problems that nginx is better suited at handling rather than a fully-fledged Python runtime – examples include static file serving (robots.txt, favicon.ico et. al.) as well as caching, header or path rewriting, and more.

nginx is a commonly used in front of all manner of applications.

Python Application 🐍

To support the testing of a real-world scenario, I’m creating a JSON response, as that’s how most web applications communicate today. This often incurs some serialization overhead in the application.

I took the example from starlette and added a couple of tweaks to emit the current timestamp and a random number. This prevents any potential caching occurring in any of the layers and polluting the experiment.

Here’s what the main request/response now looks like:

async def homepage(request):
    return JSONResponse(
        {
            "hello": "world",
            "utcnow": datetime.datetime.utcnow().isoformat(),
            "random": random.random(),
        }
    )

A response looks like this:

{
  "hello": "world",
  "utcnow": "2021-12-27T00:31:42.383861",
  "random": 0.5352573557347882
}

And while there are ways to improve JSON serialization speed, or tweak the Python runtime, I wanted to keep the experiment with defaults, since the point isn’t about maximizing total throughput, rather seeing the difference between the architectures.

Cloud Environment ☁️

For this experiment, I chose Amazon Elastic Container Service (ECS) with AWS Fargate compute. These choices provide a way to construct all the pieces needed in a repeatable fashion in the shortest amount of time, and abstract a lot of the infra concerns. To set everything up, I used AWS Copilot CLI, an open-source tool that does even more of the heavy lifting for me.

The Copilot Application type of Load Balanced Web Service will create an Application Load Balancer (ALB), which is the main external component outside my application stack, but an important one for actual scaling, SSL termination at the edge, and more. For the sake of this experiment, we assume (possibly incorrectly!) that ALBs will perform consistently for each test.

Architectures 🏛

Using containers, I wanted to test multiple architecture combinations to see which one proved the "best" when it came to user-facing performance.

Example 1: "tcp"

The communication between nginx container and the app container takes places over the dedicated network created by the Docker runtime (or Container Network Interface in Fargate). This means there’s TCP overhead between nginx and the app – but is it significant? Let’s find out!

Example 2: "sharedvolume"

Here we create a shared volume between the nginx container and the app container. Then we use a Unix domain socket to communicate between the containers using the shared volume.

This architecture maintains a separation of concerns between the two components, which is generally a good practice, so as to have a single essential process per container.

Example 3: "combined"

In this example, we combine both nginx and app in a single container, and use local Unix sockets within the container to communicate.

The main difference here is that we add a process supervisor to run both nginx and app runtimes – which some may consider an anti-pattern. I’m including it for the purpose of the experiment, mainly to uncover if there’s performance variation between a local volume and a shared volume.

This approach simulates what we’d expect in a single "server" scenario – where a traditional instance (hardware or virtual) runs multiple processes and all have some access to a local shared volume for inter-process communication (IPC).

To make this a fair comparison, I’ve also doubled the CPU and memory allocation.

Copilot ✈️

Time to get off the ground.

Copilot CLI assumes you already have an app prepared in a Dockerfile. The Quickstart has you clone a repo with a sample app – so instead I’ve created a Dockerfile for each of the architectures, along with a docker-compose.yml file for local orchestration of the components.

Then I’ll be able to launch and test each one in AWS with its own isolated set of resources – VPC, networking stack, and more.

I’m not going into all the details of how to install Copilot and launch the services, for that, read the Copilot CLI documentation (linked above), and read the experiment code.

This test is using AWS Copilot CLI v1.13.0.

Test Protocol 🔬

There’s an ever-growing list of tools and approaches to benchmark web request/response performance.

For the sake of time, I’ll use a single one here, to focus on the comparison of the server-side architecture performance.

All client-side requests will be performed from an AWS CloudShell instance running in the same AWS Region as the running services (us-east-1) to isolate a lot of potential network chatter. It’s not a perfect isolation of potential variables, but it’ll have to do.

To baseline, I ran each test locally (see later).

Apache Bench

Apache Bench, or ab, is a common tool for testing web endpoints, and is not specific to Apache httpd servers. I’m using: Version 2.3 <$Revision: 1879490 $>

I chose single concurrency, and ran 1,000 requests. I also ignore variable length, as the app can respond with a variable-length random number choice, and ab considers different length responses a failure unless specified.

ab -n 1000 -c 1 -l http://service-target....

Each test should take less than 5 seconds.

The important stats I’m comparing are:

  • Requests per second (mean) – higher is better
  • Time per request (mean) – lower is better
  • Duration at 99th percentile. 99% of all requests completed within (milliseconds) – lower is better

To reduce variance, I also "warmed up" the container by running the test for a larger amount of requests

Local Test

To establish a baseline, I ran the same benchmark test against the local services. Using Docker Desktop 4.3.2 (72729) on macOS. These aren’t demonstrative of a real user experience, but provides a sense of performance before launching the architectures in the cloud.

arch reqs per sec ms per req 99th pctile
tcp (local) 679.77 1.471 2
sharedvolume (local) 715.62 1.397 2
combined (local) 705.55 1.871 2

In the local benchmark, the clear loser is the tcp architecture, and the sharedvolume has a slight edge on combined – but not a huge win. No real difference in the 99th percentiles – requests are being served in under 2ms.

This shows that the shared resources for the combined architecture are near the performance of the sharedvolume – possibly due to Docker Desktop’s bridging and network abstraction. A better comparison might be tested on a native Linux machine.

Remote Test

Once I ran through the setup steps using Copilot CLI to create the environment and services, I performed the same ab test, and collected the results in this table:

arch reqs per sec ms per req 99th pctile
tcp (aws) 447.57 2.234 5
sharedvolume (aws) 394.55 2.535 6
combined (aws) 428.60 2.333 4

With the remote tests, minor surprise that the combined service performed better than the sharedvolume service, as in the local test it performed worse.

The bigger surprise was to find that the tcp architecture wins slightly over the socket-based architectures.

This could be due to the way ECS Fargate uses the Firecracker microvm, and has tuned the network stack to perform faster than using a shared socket on a volume when communicating between two containers on the same host machine. The best part is – as a consumer of a utility, I don’t care, as long as it’s performing well!

ARM/Graviton Remote Test

With the Copilot manifest defaults for the Intel x86 platform, let’s also test the performance on the linux/arm64 platform (Graviton2, probably).

For this to work, I had to rebuild the nginx sidecars manually, as Copilot doesn’t yet build&push sidecar images. I also had to update the manifest.yml to set the desired platform, and deploy the service with copilot svc deploy .... (The combined version needed some Dockerfile surgery too…)

Results:

arch reqs per sec ms per req 99th pctile
tcp (aws/arm) 475.03 2.105 3
sharedvolume (aws/arm) 451.71 2.214 4
combined (aws/arm) 433.94 2.304 4

We can see that all the stats are better on the Graviton architecture, lending some more credibility to studies done by other benchmark posts and papers.

Aside: The linux/arm64-based container images were tens of megabytes smaller, so if space and network pull time is a concern, these will be a few microseconds faster.

Other Testing Tools

If you’re interested in performing longer tests, or emulating different user types, check out some of these other benchmark tools I considered and didn’t use for this experiment:

  • Python – https://locust.io/ https://molotov.readthedocs.io/
  • JavaScript – https://k6.io/
  • Golang – https://github.com/rakyll/hey
  • C – https://github.com/wg/wrk

There’s also plenty of vendors that build out extensive load testing platforms – I’m not covering any of them here. If you run a test with these, would definitely like to see your results!

Conclusions 💡

Using the Copilot CLI wasn’t without some missteps – the team is hard at work improving the documentation, and are pretty responsive in both their GitHub Issues and Discussions, as well as their Gitter chat room – always helpful when learning a new framework. Once I got the basics, being able to establish a reproducible stack is valuable to the experimentation process, as I was able to provision and tear down the stack easily, as well as update with changes relatively easily.

Remember: these are micro-benchmarks, on not highly-tuned environments or real-world workloads. This test was designed to test a very specific type of workload, which may change as more concurrency is introduced, CPU or memory saturation is achieved, auto-scaling of application instances comes into play, and more.

Your mileage may vary.

When I started this experiment, I assumed the winner would be a socket-based communication architecture (sharedvol or combined), from existing literature, and it also made sense to me. The overhead of creating TCP packets between the processes would be eliminated, and thus performance would be better.

However, in these benchmarks, I found that using the TCP communication architecture performs best, possibly due to optimizations beyond our view in the underlying stack. This is precisely what I want from an infrastructure vendor – for them to figure out how to optimize performance without having to re-architect an application to perform better in a given deployment scenario.

The main conclusion I’ve drawn is: Using TCP to communicate between containers is best, as it affords the most flexibility, follows established patterns, and performs slightly better than the alternatives in a real(ish) world scenario. And if you can, use Graviton2 (ARM) CPU architecture.

Go forth, test your own scenarios, and let me know what you come up with. (Don’t forget to delete your resource when done!! 💸 )

A decade of writing this stuff? Seriously.

In tinkering with my blogging platform and playing with different technologies, I’ve just realized that I’ve been writing online for well over a decade now.

It started a long time ago, when I was writing personal stuffs on a public Open Diary back in 1995, under an alias, which for the life of me, I can’t recall. The site is currently unavailable, and I was curious to see if they still indexed old entries, and see if I could dig anything up from back then.

It was a place that I tossed out whatever I had in mind, a place to jot down the ideas running through my head, a place for a creative outlet with the safety of knowing nothing would ever come back to me, since I lived behind the veil of anonymity (since back then, PRISM was just a dream…), and I was able to express whatever I wanted, in a safe-like place.

After writing there for a couple of years, I was witness to the 1997 Ben Yehuda Street Bombing – I was at a cafe off the street with some friends, when it happened, and went to offer whatever help I could, having had some First Aid training. After spending some 2 hours dealing with things that I’ve pushed far to the back of my mind, I was gathered by a friend, carted to his house, and sat in shock for a few hours, before making my way home.

The next day, I wrote about it on OD, and referenced my friend by first name only.

A couple of days later, a comment came on my post, asking if my friend was ‘So-and-so from Jerusalem’, and if so, that they knew him, and agreed that he was a great help. We began discussing our mutual friend, and eventually met in person.

This was the first revelation I had – you’re never truly anonymous.

We became pals, hung out a few times, and continued to stay up to date with each other for a while.
I did notice that after a while, my writing dwindled, now I knew that there is someone out there who knew who I am, not that I was saying anything outrageous, but the feeling of freedom dropped.

During my time in the Air Force, I wrote extremely rarely, since getting online was near impossible from base, so after discharge in 2000, I pretty much had stopped writing altogether.

In 2003, my friend Josh Brown invited me to the closed community (at the time) of LiveJournal, where it quickly grew into the local social networking site, where we could post, comment, and basically keep up with each other’s lives.
Online quizzes were ‘the thing’ and posting your results as an embed to your post was The Thing to do.

After spending 4 years on LJ, they began providing additional customizations, added features for paid-only users, and I didn’t want to spend any money on that, rather I wanted to host my own site.

So I did, for a while. In 2006, I built my own WordPress 2.0 site (history!), hosted it on my home server (terrible bandwidth) and began on the journey of customized web application administration. Dealing with databases, application code updates, frameworks, plugins, you name it.

I think I actually enjoyed tinkering with the framework more than actually writing.

Anyhow, I’ve written sporadically over time, about a wide variety of things, both on this site, and elsewhere.
The invention of Facebook, twitter, and pretty much any social network content outlet has replaced a lot of the heavier topic writing that went on here.

But it does indeed fill we with some sense of happiness that I’ve been doing this for a long time, and have preserved whatever I could from 2003 until now, and continue to try and put out some ideas now and then.

My hope is that anyone can take the to express their creativity in whatever fashion they feel possible, and share what they want to with the rest of us.

My foray into web development

I like browsing the web. I do it a lot throughout my day.

A lot of people work hard at making the web a cool-looking place. Some sites make simplicity look so easy, that when you look under the hood, it’s all chaos and destruction, folded and crunched together, all to present something really nice and smooth for the end-user.

I’m not a developer – much less a web dev. There’s a lot to know in any field of computing – and in web it’s pretty much the most visible part of computing as a whole, since pretty much anyone anywhere is going to use a web browser to view a site at some point.

I mean, sure, we all learned some HTML – hell, I wrote some sites back in the days of Geocities, and it was awesome to learn about tiling backgrounds of animated GIFs, and when CSS came around, minds blown!

And I left that field for the frontend developers, and went into infrastructure and operations.

And as time passes, you find yourself managing a variety of systems and knowledge, and at some point, you may say to yourself, “I wish I knew how to answer this question…”

And then you write some code to answer it. Voila! You’re a developer, of sorts.

I’m a huge fan of data visualization. Telling stories with pictures dates back millennia, and it’s very relatable to most people. Recently, I wrote a tool to help myself display the dependency complexity of Chef roles, and I found that, while being very useful, the output is very limited, as it’s a static generated image, whereas we live in a web-friendly world where everything is interactive and fun!

So when I came across another hard question I wanted to answer, I thought, “Why not make this a web application?”

This time, the question I wanted to answer was: As a GitHub Organization owner for my company, what human-to-team-to-software-repository relationships do we have, and are they secure?

If you’ve ever managed an Organization in GitHub, there are a few key elements.

  1. An Organization can have many Repositories
  2. An Organization can have many Teams
  3. A Repository can have many Teams
  4. A Team can have many members, but only one permission (read only, read/write, owner)

So sorting out who is on what Team, what access they have, across many repositories, can be a security nightmare. Especially when you have more than 4-5 repositories.

During my first foray into solving this, I cobbled together a command-line tool, using Ruby with the Graphviz library. I’ve like Graphviz for years – it’s straightforward, as structured text gets rendered into a graph and then can be output to a file.

Very straightforward, has some limitations, but basically allows you to store graphs as text, and re-render them when changes happen. Basically, it’s like storing source code and not the binary output.

But since there were some limitations, and I wanted this new question to be more than a command-line tool, something I could share with the world at large, without requiring any client-side installation of any tools or dependencies.

So I spent a lot of time hemming and hawing, looking at web frameworks and trying to figure out some of them, and “how does this work?” came up a lot.

Finally, yesterday I set out to sit down and accomplish this task. I sat in a Starbucks in New York City, and had a Venti. I started banging away at about 11:30. I took a break for a refill and a snack around 1:30, and when I sat down again, I kept hacking away until 9:30pm, when I deemed completion.

The code was written, tested by me locally, pushed to GitHub, deployed to Heroku, DNS name wired up and all. As soon as I completed, I left Starbucks, and heaved a huge sigh – it was one hell of a mental high, I was in “the zone” and had been there for a long time.

You are more than welcome to browse the source code here and the finished project here. I call it the GitHub Organization Viewer, hence “GOVweb”.

I have a bunch of other ideas on how to make this better, how to model the data, which visual style to use, but I think for now, I’m going to leave it for a bit, and see what I think about it in a couple of months.

But all in all, this reinforced my opinion to never be afraid to try tackling a new idea, a new project, a new field you’re unfamiliar with – as long as you can read, comprehend and learn, the world is your oyster.

Spread your wings and fly…

Commercial flight is no longer something I look forward to.

There was a time that I would get excited about traveling through the air, it was different, exotic, something reserved for special occasions.

As I have grown, and flown more, it feels like when you can see beyond the intricacies of a magicians’ web of illusion, and realize that the trick wasn’t that complicated after all. It’s all simple mechanics, and seeing the same trick performed again and again, it becomes routine and mechanical.

I guess that’s true of anything though – take the fact that I am writing this mid-flight, over the Atlantic Ocean, on a device that did not exist a few years ago – a netbook computer – and how much that has become a routine part of our lives and like many technological advances is taken for granted soon enough.

My traveling companion for this trip is my uncle, who rarely flies internationally, and just requested that I don’t write about him. Apparently he doesn’t realize that the request in and of itself a reason to do so – everyone say Hi to Uncle Dennis!

Today’s trip started with us meeting at the airport, and spending a few moments weighing our bags with a mini-bag scale that I carry on trips, to verify that our checked luggage would be within the weight parameters and not incur hefty overweight fees. A short re-distribution took place, where we exchanged some items for others between our bags so that we were both at the appropriate limit of each bag being under 50 pounds (or 22 kilos).

Then we joined the line of the many people trying to be processed at the El Al check-in desks. This is a different procedure than any other airline, as El Al has their own security screening process, which isn’t that secure, I hate to tell you.

Once you finally get around enough of the line to reach an actual security person, they ask you a few standardized questions – pretty much the ones that George Carlin has ridiculed in epic words – before sending you off to the right, to wait in line again, where your checked luggage (only checked luggage!) will be scanned by their heft X-ray machines.

Waiting in this new line showed a degree of inefficient operation that I have grown accustomed to, and just like everything else when it comes to international flight, I simply sigh, suck it up, and ride the waves of despair. There are two large scanners, and only one person loading bags on to them. One line shuts down while we are waiting there due to a large family with a bunch of children having some sort of issue with putting the stroller through, or something else.

Most of the staff are American TSA workers – except two – an older Israeli guy who kept fluttering around and interrupting any kind of flow that was slowly being achieved, and a younger Israeli girl whose sole job, it seemed, was to place a “scanned” sticker on bags that gad been scanned. She couldn’t do that very well, as a family had to come back from their lengthy check in process to ask for a sticker – which she simply gave to them.

Insecurity Note #1 – who is to say that bag was scanned or not? What is to prevent me from taking the same approach, simply not choosing to scan one bag, then coming back to the desk and getting a sticker for the second?

Insecurity Note #2 – once the bags were scanned – only checked luggage, mind you – they are returned to our possession for a lengthy wait for the check-in desk. During this time, I literally could have placed anything at all from my pockets, or even anything larger from my carry-on or backpack into my luggage that now, thanks to the security girl’s ability to focus for a moment, has a nice, official “scanned” sticker on it.

Uncle Dennis and I chat about this, shake our heads in despair and continue on with the show. I could have brought a bag full of firecrackers with me.

The check-in desks never seem to operate fast – every traveler in line in front of me seems to have a very specific set of problems that has never been encountered in the past 40 years of flight, and requires involvement of a supervisor, a manager, and sometimes a quick chat with the captain. However, it must be me, that I take the time to figure out the rules of this stupid game and adhere to them, as the moment I step up, I am typically checked in, have my boarding pass and luggage tags all in under three minutes.

From this point forward is the move to the last security scan before the gates, and the last time beloved ones will see you – gone are the days where they can stand at the door of plane, wave at the aircraft as it taxis away to wait forever in line on the tarmac for clearance to take off. Now they say their goodbyes, and sometimes watch as the mass of humanity gets funneled slowly into another snake maze line, where their passport and boarding pass and given a cursory glance before shunting them into another processing line where the most exciting part of all of this takes place – the metal detector and carry-on x-ray.

Why is this the most exciting? Because it has become a challenge to me to beat this system at its own game. I have been stopped more times than I can count at this gateway.
My shoes are unlaced and come off, my bag pops open and the netbook comes out into a plastic tub, the shoes go in there too. Everything in my pockets – phone, wallet, and change, anything – goes into the bag. My belt flies off into the plastic tub. My jacket is already in there. Everything goes on the conveyor belt.

Then, with a deep sigh, I step forward to the beckoning TSA agent at the metal detector. And with a sense of “I know this thing is going to beep at me”, I walk through and stop, waiting for the questions about my pocket contents, any medical metal hardware or whatever else they can ask me.

Guess what? I won this time. Maybe the machine is malfunctioning. Maybe the sun is shining just the right way right now, or maybe the magnetic poles of the Earth are aligned perfectly for me right at this moment.

No call for a bag check, no red light went off, requiring a roundabout impromptu interrogation by a security idiot, for whom petty power has corrupted beyond all proportions, and no further hold up for me at this station.

Sigh of relief, grab my shoes, bags and laptop. I had almost considered doing this trip in only my pajamas.

Up until this point, I think I’ve waited in about four lines, and there’s one more – boarding the plane. Not the most efficient process to say the least – as this airline does not board by section, row or reason – simply board now. There’s a line that stretches to two gates away for this flight only. We take a seat, and wait for the line to be processed at a snail’s pace.

Once on board, again it seems like people have forgotten all semblance of “how things work”, not realizing that we’re all going to be compressed in this tin can for the next 10 hours or so. Crowding in the aisles, stopping and talking with people already seated, blocking the passage of everyone else from boarding, tossing their carry-on in to the overhead compartments and sitting down.

Before leaving home, I took the time to visit the El Al web site and look at their carry-on luggage policy – and got out the measuring tape to verify that my bag did indeed meet the required size limits, and it did. However, a Boeing 747-400 (how long ago was this particular aircraft made?) center overhead compartments are about 1 inch smaller than the advertised capacity. My frustration of trying to jam my own bag into their compartment were heard for a few rows, and a few people smiled in sympathy, and nod. I guess they had a similar experience at some point.

I walk farther down the plane looking for space in one of the overhead compartments that face the windows, large enough to share, and as I see an opening and begin to raise my bag, a guy tells me that his “seats are here and this room should be reserved for him and his family”. I grumble and move along further down, finally spot a vacancy 10 rows away, rush there, toss my bag in, slam the compartment shut and sigh in relief.

Fighting my way upstream to get back to my seat, I look and see that, of course, I’m sitting next to a guy that has had his (and mine and yours) share of good meals, and that his gut spills over on to the armrest, to squash against my right arm, prompting me to fly with my arms folded in front of me for most of the flight.

The woman in front of me comments loudly that she cannot believe how small the seats and space between rows is. I smile in sympathy and nod.

And we haven’t even begun to taxi to the runway yet.

Time goes by, so slowly

I seem to be letting larger amounts of time slip by between posts, and that kind of makes me sad.

Between having the ability to Tweet, Facebook status update and Google Buzz, i feel that sometimes I just don’t want to write, and that is a Bad Thing.

Writing is a great way to dump some of the thoughts, feelings and ideas from inside this mess of a brain to written word, and in the past has allowed me to review these at a later date to see what the heck I was thinking and talking about.

Now I am not committing to writing regularly, or even on any set schedule, but just doing it now and then seems to help out.

In recent past, I’ve been tinkering with all kinds of technologies – from TCL to python and powershell, from WordPress php and css to Google AppEngine, and even more in the hardware and software realms.

Some of the things I am teaching myself is how to understand enough of the lowest possible level to get the core ideas to then be able to make that jump into the high-level arena, where having the big picture is crucial.

Some of that lies within data visualization, some of it relies on knowing the inner workings of a system, another is how to get data in and out of a management interface, and trying to figure out what is the question you want answered.

I think figuring out these kind of things are the challenges I like most.

Site was down

My site apparently got hacked about a month ago, thanks to me not keeping my site software up to date, and left a small vulnerability open, so something went wrong, and I was offline for about three months, until I sat down, and started fresh, with the same content database. It might take a few weeks until all the software is brought back into play, so let me know if anything is broken, and I’ll try to fix it.

Staying on top of patching software is not easy, especially when you do that by day, and when you get home you don’t really want to look at a computer anymore.

But it’s back, and I’m happy. Life is better.

Climbing to the top

Today, some crazy Frenchman decided to climb the New York Times building. A more detailed story here.

I was set to meet some improv pals at 39th and 8th Ave – one block away from the NYTimes building, and as I approached, throngs of people were in the streets, yelling and pointing.

It seems like there’s a second climber, who also wanted to scale the impressive building.

So naturally, i stopped, yanked my camera out of my bag, and snapped off a few.

See here:

Continue reading Climbing to the top

He’s back, and he’s bad.

It’s been a couple of weeks since I left for my visit home, and while I have to complete my full vacation post (and it’ll be a long one), suffice to say that I am still alive, despite random attempts to change that, and had a great time.

Jet lag is annoying, and I think I’ve finally kicked it, and I started my 201 improv class – it’s gonna be great.

Back in New York City, I am happy to report that it’s still on the map, and hasn’t scurried off and hidden under the grandfather clock in the foyer in my absence.