Starry Wisdom

Avro vexes me every time I use it – and the documentation is helpfully only to a small extent.  Today I was trying to add a field that stored a list (array) of strings and had a default value to an already existing schema. I tried a couple of things…

#doesn't work
{"name": "foo", {"type": "array", "items": "string"}, "default": ["bar"]}

#doesn't work
{"name": "foo", "type": {"type": "array", "items": "string"}, "default": ["bar"]}

Before I realized that the “type” required a list of types in this case, even if the list was one element long. So this is the working pattern:

{"name": "foo", "type": [{"type": "array", "items": "string"}], "default": ["bar"]}

Chalice

Chalice is the “Python Serverless Microframework for AWS”. It allows quick and simple development of REST APIs, and comes with a a deploy tool that does all the work necessary to deploy your lambda, as well as create policy and integrate with the API Gateway. Let’s start out by creating an example app:

$ chalice new-project example-app

And then we can quickly create a simple app that reads in a couple of parameters and creates a JSON response. Note the use of the decorator to declare the route and method:

from chalice import Chalice app = Chalice(app_name='example-app') @app.route('/customer/{customer_id}/order/{order_id}', methods=['PUT']) def register_order(customer_id, order_id):     # imagine inserting this into Dynamo, etc...     return {'customer':customer_id,             "order_id": order_id }

Deploying this as simple as:

$ chalice deploy
Creating role: example-app-dev
Creating deployment package.
Creating lambda function: example-app-dev
Initiating first time deployment.
Deploying to API Gateway stage: api
https://id.execute-api.us-west-2.amazonaws.com/api/

At which point you can access it like this:

$ curl -X PUT https://id.execute-api.us-west-2.amazonaws.com/api/customer/customer01/order/123183123
{"customer": "customer01", "order_id": "123183123"}

See the tutorial, which is quite good, for more information on what you can do inside the apps (such as tying in other AWS services).

Enter Terraform

But what if you want to use Terraform to deploy your infrastructure? The first step is to create a deployment package:

$ chalice package .
Creating deployment package.
$ ls -l deployment.zip
-rw-r--r-- 1 nchazin staff 9022 Oct 18 22:24 deployment.zip

And now we’re ready to code up our terraform. We’ll begin by defining a few variables, which we can store values for in a terraform.tfvars file.

Next we’ll define the lambda itself, along with its associated role, and a policy which allows us to log and monitor with Cloudwatch:

 

 

With our lambda set up, we can create out API Gateway:

 

The final piece is the permission that allows our API Gateway to access the lambda. Note that you can be more specific and limit acess to the specific API method and path if desired by adding them to the source_arn.

Now we can install our API:

And query it at both the stage and prod endpoints:

I met Noah at 4:07 AM on March 27, 2015. He’s pretty cute.

Recently at work I came across a certain long-lived server process that was using an immense amount of memory over time. Every few days it would grow something over 10GB of resident memory, at which time we would restart the process. This was clearly not an ideal solution, especially because this specific server was running on something like eighty servers in forty data centers. So last week, during some relatively down time, I dug into where the memory was going. The first thing I did was spend some time with the heap profiler and heap checker from google’s perftools (https://code.google.com/p/gperftools/?redir=1) (side note – this is still on google code and not github!?). This showed nothing particularly useful. I then resorted to using valgrind on a test box, which, after an excruciatingly slow run, showed no leaks.

Well, I next spent a little time on code inspection, where I found that where there was one place where a leak was possible, it wasn’t likely to contribute to GB of leaked memory over days under the workloads that we have. And then the next thing I did was break out all the inputs to the server – client requests, plus data streams going into it. I went through these one at a time, and finally found that our primary data stream could easily raise the memory usage from a baseline of around 2GB to the 10 GB when I ran it through a test box at an accelerated rate.

Without going into too much detail, this data stream basically gives us a small number of data points for many pieces of data. These data sources usually each contribute their own data to the data point, and this data can change over time. So conceptually there is a two level data structure:

As it was written (disclaimer – I wrote much of the code in this server, but I don’t think I wore this part… although I may have) this was two levels, each implemented by c++11 std::unordered_map – which is essentially a hash table. The lower level map was defined in particular as std::unordered_map<std::string, Pod>. For the purposes of this article, the Pod type was this (and this is actually almost exactly what we were using:

struct Pod {
uint32_t item1{0};
uint32_t item2{0};
time_t timestamp{0};
};


A word about the strings – these were the host ids of the several servers providing the data. So in effect, every “bucket” in the upper level map had a map inside of it with the same several keys in it. Which meant that we were spending a lot of space on the keys. So the quick thing to do was to create an ancillary std::unordered_map<std::string,uint16_t> to map the hostname into integers and therefor the lower level maps become std::unordered_map<uint16_t,Pod>. Testing this showed that memory usage went down by around 3GB, which was a big saving. The math for removing the strings didn’t quite ever add up, but I was fairly happy. 3GB savings from 8GB is about 38% improvement, at the cost of one additional O(1) lookup per insertion. But maybe there was more to do. Why use a map at all when you have the same 5 keys there for *almost* every single lower level map. What if we just used a std::vector<Pod>? For this run the memory used was 5.4 GB below the baseline, or an additional 2.4 GB used. This was a savings of over 5GB, or over 60%!

It wasn’t obvious to me what was going on, so I wrote a little program to test a very simple scenario of allocating 10 million of the lower level “maps” with different strategies – std::unordered_map,std::map and std::vector. Here are the results for memory used compiling with both clang and g++ – all memory is in KB used, as reported by ps -ev on linux.

This gives results similar to my results for our production code at work. And it was at this point that I realized what was going on. When you create a hash table, it is not sized to the actual data in it - there is a lot of extra space used in the table for buckets that are not full - wasted space. This is not a big deal for a fairly dense hash table. However the vector was sized for the exact number of elements inside of it - there is still some overhead for the vector itself, but compared to the empty space in the hash table, it's not much. In hindsight this is fairly obvious, but it was not clear to me or my coworkers when I first started looking into this memory issue.

By the way, my test code is available here.