Starry Wisdom

Entropic Words from Neilathotep

Friday, April 17, 2015

Noah

I met Noah at 4:07 AM on March 27, 2015. He’s pretty cute.

posted by neil at 12:33 pm
under Uncategorized  

Thursday, March 19, 2015

Be careful how you use data structures!

Recently at work I came across a certain long-lived server process that was using an immense amount of memory over time. Every few days it would grow something over 10GB of resident memory, at which time we would restart the process. This was clearly not an ideal solution, especially because this specific server was running on something like eighty servers in forty data centers. So last week, during some relatively down time, I dug into where the memory was going. The first thing I did was spend some time with the heap profiler and heap checker from google’s perftools (https://code.google.com/p/gperftools/?redir=1) (side note – this is still on google code and not github!?). This showed nothing particularly useful. I then resorted to using valgrind on a test box, which, after an excruciatingly slow run, showed no leaks.

Well, I next spent a little time on code inspection, where I found that where there was one place where a leak was possible, it wasn’t likely to contribute to GB of leaked memory over days under the workloads that we have. And then the next thing I did was break out all the inputs to the server – client requests, plus data streams going into it. I went through these one at a time, and finally found that our primary data stream could easily raise the memory usage from a baseline of around 2GB to the 10 GB when I ran it through a test box at an accelerated rate.

Without going into too much detail, this data stream basically gives us a small number of data points for many pieces of data. These data sources usually each contribute their own data to the data point, and this data can change over time. So conceptually there is a two level data structure:

As it was written (disclaimer – I wrote much of the code in this server, but I don’t think I wore this part… although I may have) this was two levels, each implemented by c++11 std::unordered_map – which is essentially a hash table. The lower level map was defined in particular as std::unordered_map<std::string, Pod>. For the purposes of this article, the Pod type was this (and this is actually almost exactly what we were using:

struct Pod {
uint32_t item1{0};
uint32_t item2{0};
time_t timestamp{0};
};

A word about the strings – these were the host ids of the several servers providing the data. So in effect, every “bucket” in the upper level map had a map inside of it with the same several keys in it. Which meant that we were spending a lot of space on the keys. So the quick thing to do was to create an ancillary std::unordered_map<std::string,uint16_t> to map the hostname into integers and therefor the lower level maps become std::unordered_map<uint16_t,Pod>. Testing this showed that memory usage went down by around 3GB, which was a big saving. The math for removing the strings didn’t quite ever add up, but I was fairly happy. 3GB savings from 8GB is about 38% improvement, at the cost of one additional O(1) lookup per insertion. But maybe there was more to do. Why use a map at all when you have the same 5 keys there for *almost* every single lower level map. What if we just used a std::vector<Pod>? For this run the memory used was 5.4 GB below the baseline, or an additional 2.4 GB used. This was a savings of over 5GB, or over 60%!

It wasn’t obvious to me what was going on, so I wrote a little program to test a very simple scenario of allocating 10 million of the lower level “maps” with different strategies – std::unordered_map,std::map and std::vector. Here are the results for memory used compiling with both clang and g++ – all memory is in KB used, as reported by ps -ev on linux.

This gives results similar to my results for our production code at work. And it was at this point that I realized what was going on. When you create a hash table, it is not sized to the actual data in it - there is a lot of extra space used in the table for buckets that are not full - wasted space. This is not a big deal for a fairly dense hash table. However the vector was sized for the exact number of elements inside of it - there is still some overhead for the vector itself, but compared to the empty space in the hash table, it's not much. In hindsight this is fairly obvious, but it was not clear to me or my coworkers when I first started looking into this memory issue.

By the way, my test code is available here.

posted by neil at 9:17 pm
under technology  

Friday, November 14, 2014

Kitchen Renovation – small big things

In the past week, the countertops were put in:

Untitled

They are engineered quartz (Caesarstone brand) and are a huge improvement over the old laminated counters, if I do say so myself.

A few days later, the backsplash (subway tile) was installed:

Untitled

The prep sink is now fully functional, and the main sink is mostly hooked up – we’re in the home stretch now. Our contractor tells us that we should have a functional kitchen by the end of today – inspections are the end of next week.

posted by neil at 9:00 am
under kitchen renovation  

Sunday, November 2, 2014

Kitchen Renovation, And Then…

Well as mentioned, the tile was installed in the past couple of weeks:

Untitled

And the counter top templates were finished, but due to some FURTHER miscommunication, there was a delay before the fabrication could begin of another week and a half. The good news is that we have an install date for the counter tops now, of 11/10, and after that the backsplash tiling, and appliance hookup can happen. There are still a few other details to be done too, but the counters are really the long pole here.

posted by neil at 5:29 pm
under home ownership,kitchen renovation  

Saturday, October 18, 2014

Kithen Renovation – One month, and then some.

It’s been a busy time the past couple of weeks. I spent all of the week before last in Portland for work, leaving Mackenzie to manage the project herself. In that week our fridge was finally delivered, and is now parked in the living room, since the dining room was still filled with cabinet boxes when it arrived:

New fridge - in living room...

Anyway, I came home to most of the cabinets installed, but the apron sink not yet set into the counter… there was some miscommunication about what was necessary before counter top templating, and this caused an additional one week delay in that (it was supposed to be done on 10/8 but instead wasn’t finalized until 10/15. There is about a 10 day manufacturing time on the counter tops… but hey at least all the cabinets are in, the hood is installed, and the walls and ceiling are painted:

Trip added to cabinets, tile underlayment, and hood!

Feature wall

Next week the floor tile will go in, as well as some more detail work, and then, well I don’t know. I think there are still a few more weeks left of the project, but hey, there aren’t as many boxes taking up space all over the house, at least…

Note – full album of the project pics available here

posted by neil at 5:00 pm
under home ownership,kitchen renovation  

Sunday, October 5, 2014

Kitchen Renovation – Small Progress

At the end of the third week, the progress is:

Rough inspections passed, hose bib attached on the deck (this was an add-on job Mackenzie requested which the contractor kindly provided gratis, and drywall taped and mudded:

Untitled

I was hoping to have had some cabinet installation done this week, because the cabinets in their boxes are taking up a tremendous amount of space in the house, but alas, that hasn’t started yet. Now there is a race to get that done so this week we can get the counter tops templates, and the new fridge delivered. I am a bit concerned about either/both of those items happening on schedule, especially since I am going to be out of town all week long for work. Oh well, Mackenzie runs projects bigger than this for a living, I’m sure she can manage to hold down the fort.

posted by neil at 10:27 am
under home ownership,kitchen renovation  

Sunday, September 28, 2014

Kitchen Renovation – 2 weeks in

Well, after two weeks and 1 day of work (a team came in yesterday, Saturday 9/27/2014 to put in drywall), we have drywall up, and most of the plumbing and electrical done. Most, because the inspectors found some minor issues and need to come back, and also some of our minor added work needs to be done. This means at least some of the drywall will have to come down (as you can see it’s not mudded and taped yet) for access and to show the inspectors.

Southeast corner, drywall in - Day '11'

It’s a good thing it’s not done yet too, because there are still rolls of R-30 fiberglass insulation sitting in the attic:

Hey look, rolled up batting I'm the attic

Getting up there is sort of a pain, but as part of a “clean up the pantry of expired canned goods” task it made sense to take a peak up there and get that photo!

Anyway, I’m really note sure what the schedule is right now, but I believe that cabinets can start going in at some point this week – so they might all be up by the end of the week, or it might be early next week. Once this happens, we can get the counter top people in to template, and that ten day or so period begins, while tiling and other detail work can begin. In other words there are at least three weeks left of this project. I’ll be happy when the cabinets start going up, because the whole house is filled them them i their shipping boxes. It’s getting a bit claustrophobic in here!

posted by neil at 12:39 pm
under home ownership,kitchen renovation  

Sunday, September 21, 2014

Kitchen Renovation, Part 2

As always, you can see the full set of pictures here:

Day 2 - It looks like a barn

First of all, we went to the “kitchen warming” party of Mackenzie’s college friend today – they had their kitchen done over the summer and it looks great, so I am looking forward and hopeful for our final results!
However, not too much has happened since the last post, although there has been some plumbing work done. We’re moving the dishwashing sink from the south wall to the west wall – which means a new waste water stack:

New waste stack

As well as some rerouting of the supply lines, which are now going through the attic:

External supply lines - up and over the old, boarded in window on the East wall

Supply lines through the attic

We found out that there is going to be at least a half week delay in getting inspections for the rough plumbing and electrical, which are not going to happen until Friday 9/26. Assuming that all goes well, drywalling will start Saturday 9/27 – so hopefully the cabinets will be hung by the end of that week, and then counters can be templates and the rest of the finishing work begun.

posted by neil at 9:43 pm
under home ownership,kitchen renovation  

Wednesday, September 17, 2014

Kitchen Renovation

Hey do I have any readers left? I wouldn’t be surprised if I lost my 2-3 fans after my 9 month hiatus. Well, serves me right.

Since I moved in to this flat in 2010 I’ve wanted to renovate the kitchen, which was about the only room in this now 105 year old apartment that had not been renovated recently. After much planning, mostly on the part of Mackenzie, we finally came up with a basic design that we liked, which required the fridge to be in the corner, on an angle. This is somewhat unorthodox, but we’ve lived like this since some time in 2011, so clearly we approve.

Well, we started shopping for cabinets in probably March, and it took us until the end of June to really confirm what we want and order cabinets. Based on a delivery late this week – 9/18, we hired the contractor to start demolition on 9/15. Well, the delivery date of the cabinets got moved up a day to 9/17 – but more on that later. Here is a shot of the kitchen from before the work started, after clearing all our stuff out:

Before

Dig that sweet 1980 styling!

After the first day was over the cabinets and walls were down, but the ceiling was still up and the lovely vinyl flooring was still there. It was pretty cool to see the bones of the house, however:

By the end of day 2 the vinyl flooring was up (although there were a couple of layers under it, above the plank subfloor that weren’t ripped up!) and the ceiling had come down. This is a good shot of the ceiling, showing the attic and the roof boards. It sort of looks like a barn:

Day 2 - It looks like a barn

Day three (today, 9/17) was sort of a bit underwhelming – they started the rough electrical and installed the recessed lighting which was a late edition to the project after we realized that since we ripped out the ceiling, we would be foolish to not include them.

Day 3 - Recessed Lighting

There is also some 3/8 inch plywood over the plank flooring.

Day 3 - thin plywood subfloor over the planks

There is some more lumber outside to help reframe some “issues” from previous work. The most egregious of which is this knocking out of the base of one of the studs:

Day 3 - look at that notching

Well, the house has been up a long time with that, so I’m not too worried :).

Anyway, so that’s the current status and I will post more, but there are some adjacent stories worth telling, about delivery of some important parts of the new kitchen.

1) We bought a new fridge, which Mackenzie fell in love with, and met my ideals of french door top, freezer below. We ordered it the friday before labor day and it was supposed to be delivered on 9/11 – but after the delivery truck came about 50 minutes late, we could not accept delivery because the power cord to the brand new fridge was cut almost all the way through. We have delivery rescheduled to next thursday, 9/26. Hopefully that will do down without a hitch… we’ll se.

2) So the cabinets were supposed to be scheduled today, 9/17. This was going to make our lives tricky since the house isn’t all that big and we would have had to store all the boxes until they were ready to be installed. Clearly the room isn’t ready for them, there is plenty of work to do before the sheetrock can go up – its going to be at least Monday before cabinet installation can begin. And we would have to spend the rest of this week and the weekend squeezed out of our house. Well, luck shown on us with this delivery snafu – for some reason they decided to use a full semi truck to bring them to our house, on a small side street in the middle of the City. The driver decided it was not worth risking pulling the trick down the street, due to overhead power lines, after arriving half an hour after the delivery window (terrible traffic today). They’re going to come back Monday morning, in a smaller truck, That should be much better.

posted by neil at 8:35 pm
under home ownership  

Sunday, December 22, 2013

Site Outage

Not that anyone is reading, but the blog has been down for the past week or so, as my web host had been in the midst of a very protracted hardware upgrade, and the temporary status were permission issues on the website that i was not able to fix unit this AM. Service is now restored, so you can return to your regularly scheduled non-updates (book post coming in a week or so though!).

posted by neil at 10:53 am
under meta  
Next Page »

Powered by WordPress