Autorelease pool hell

I spent a lot of time hunting down a problem with a crash in autorelease pools in our Python - Chromium Mac application and I finally found the primary cause. I couldn't find any information at all about this kind of crash on the internet, so I decided to share the experience with you. Brace yourselves, it's going to be long.

Our app was crashing on exit somewhere in AutoreleasePoolPage::pop. Attempting to access memory at 0x00000010 which is very bad. Everybody on the internet says that for Objective C crashes, use Instruments and Zombies, right? Well, zombies were sleeping in the graves peacefuly for me. Nothing found. So let's try the next tool.

I couldn't get Valgrind to work for this app on OS X, because I had a recent machine which supports HW instructions for AES and Valgrind in 32b mode doesn't, which is basically game-over. So I got an older machine, without these instructions, and found ... nothing. Really, no information related to this problem at all. It seems that I'll need to do some more thinking.

In the meantime, I discovered some other problems, largely unrelated to this one. Because of them, I was considering to remove wxPython as the window-creating mechanism from our app and roll my own. This did solve the problem with accessing the USB on OS X, but this crash, in exact same form, persisted even after getting rid of wx. It's starting to get sad, really.

I was forced to roll up my sleeves and dig once more into disassembly. I found that the memory which AutoreleasePoolPage::pop() is accessing is somewhat wrong. But I couldn't find any more information from that mountain of instructions and this is where I got lucky - I found that there is actually source code for this class! Just get the right version of objc from http://opensource.apple.com! This made things a lot easier and clearer. And what's more, after a while I was able to compile this project and use the debug version instead of the system one. How awesome is that!

What I found out is that autorelease pools are nothing more than a block of memory containing pointers. This memory is managed manually by the AutoreleasePoolPage C++ object. And that is rather unfortunate, because there are almost no safeguards at all. If they were using the standard malloc or new, I could employ all kinds of diagnostics, safeguards and checking and they may be able to slap me at the right moment. But with this manual management, we are on our own. This was probably one of the reasons why Valgrind, Guard Malloc and other tricks just didn't work.

Takeway 1: If you manage memory by yourself, you will probably need heavy checking, the kind that malloc has.

So I knew that I'm somehow overwriting my own memory inside an AutoreleasePoolPage. So let's introduce some logging and custom diagnostics. I used dtrace and Instruments to get arguments and call stack at key points in the program (well, basically at every call to AutoreleasePoolPage methods). dtrace is awesome, if you don't know it, you need to. This told me that, indeed, one autorelease pool memory block is getting overwritten by a different one. And I also knew which one! I didn't need much to realize that I'm using the pools incorrectly. It seems they should always be used like a block and nested correctly:

  p1 = [NSAutoreleasePool alloc init];
  p2 = [NSAutoreleasePool alloc init];
  [p2 release];
  [p1 release];

or use this shorthand syntax:

@autoreleasepool {
  @autoreleasepool {
    ...
  }
}

which means creation and destruction in a single function. What I did was to create the block somewhere in one method and release it somewhere in a run loop. This apparently breaks nesting and causes all this headache.

Takeway 2: Always nest autorelease pools correctly.

Thanks to the call stacks from Instruments, I knew which pool is destroying which other pool and now I only need to figure out how to fix it. Phewww.

英文

As for English knowledge, this varies a lot. Some people don't speak at all, some speak at an average level and some speak very well, better than me. I heard that some universities teach entirely in English.

Have I already mentioned that education is really big in HK? As well as status symbols and money. One foreigner living here posted this and I do feel very similarly about it:

Ahh... Hong Kong... it takes your insecurities and enhances them tenfold to the point where you're always self-conscious or you don't care about them. Little kids will point out your every flaw while everyone else is silently judging you for your technological and fashion choices. The populace is hungry for money and will jump on any opportunity to make a buck. In the meantime, little Sally Wu is only getting four hours of sleep every night because her mother wants her to be the perfect student.

It might be the environment I work in, but many local students of Computer Science are trying to pursue PhD and other things people usually do in non-engineering disciplines to try and advance their career. Things that programmers and SW people usually don't do, because they don't need it. Because at least in Europe + US most employers consider real-world experience rather than academic degrees.

Peopleware summary

• In SW development, deadline will not make people more productive, but it will likely make them less productive.
• Parkinson's law (work expands to fill alotted time) is just a joke.
• Productivity cannot be increased by simply taking some (heavily advertised) magic pill, only by improving working conditions. And yet managers are so susceptible to buying magic pills, maybe because they feel desperate?
• Noise and interruptions at the workplace are really annoying - in programming, most meaningful work gets done in the flow and it takes at least 15 minutes to get back there after an interruption.
• When the members of a team work in physical proximity, they tend to discuss things at the same time and go to quiet, flow-mode at the same time as well.
• Are deadlines really so important to justify losing good employees to burnout?
• Price-vs-quality: end users usually don't care about quality that much (because it costs money), but low-quality products reduce the programmers morale. It's best in the long-term to let the developers set their own standard. In some companies, developers can even veto shipping.
• The book is also rather funny or sarcastic: "Before drawing plans for its new Santa Teresa facility, IBM violated all industry standards by studying the work habits of those who would occupy the space." 
• Quiz: what annual turnover does your company have? How much does it cost to replace a person? Scoring: If you had any answer at all, you pass. Otherwise you fail. Most people fail.
• It has come to my attention that some of you, when travelling on expenses, have been travelling economy class. This is not an economy-class organization. This is a first-class organization.
• Managers with low confidence or ego problems may feel the urge to impose their position of status on their lowly subordinates, such as moving the company/office closer to their homes or  suppressing people who are different, because they feel threatened by them (around page 97).
• Hawthorne effect - people are more productive when they do something new or in a different way or different environment (page 120).
• People enjoy working in teams for a common goal. The challenge of work is important, but not in and of itself; it is important because it gives us something to focus on together.
• At one company, they made a team of testers who took their pride in finding as many bugs and problems as possible, often in very mischievous ways. Grown men and women were reduced to tears by watching their programs misbehave under the demented handling of these fiends. The testing team started to dress in black and really enjoyed their work. 
• To build a team: The team members need to get into the habit of succeeding together and liking it. A good manager/leader will set up situations to gain this kind of momentum. 
• Internal competition has the direct effect of making coaching difficult or impossible. This means that internal competition (employee of the year, ...) makes no sense. 
• Introducing change: There will be people strictly against change, strictly for-change and the doubters. And these are actually the only meaningful potential allies of the change.
• To promote the change, forget about rational claims. People take sides based on emotions. So it's much better to emphasise how the old system was working well, how it served so many requests of the customers and then slowly transition into motivating the people to embrace the change (page 211).
• How do you think about salaries? Do you consider them to be expense that, once paid, is gone, like the heat energy in winter? Or do you think of it as an investment? Because replacing people and bringing them up to speed takes a lot of time and money, so it's worth it to try and keep them. (page 217)
• The office and environment you work in influences productivity to a high degree. Some really
 nice inspiration: http://officesnapshots.com/2012/12/27/most-popular-offices-2012/

學

People in Hong Kong consider education to be very important. There are many expensive schools, parents and kids alike obsess over passing entrance exams, you hear phrases such as "he's not highly educated" from girls. But then, being the tall guy I am, I did have a chance to see what other people do on the MTR. Few of them actually read books, despite carrying iPads or Galaxy Note Twos. Everybody is just playing Candy Crush. And if people are not reading on public transport, which is the best time to do so, I doubt they read at any other time. Kindle is my faithful companion on every single journey around the city I take and I've actually finished Heretics of Dune in the month I've been here. In fact, few people here even ever heard of the Kindle. After discussing this with some people, I learned that students are simply overloaded with the school tasks to have any strength left for reading.

There is also an interesting distinction in the quality of services. My tiny apartment is nicely private, but doesn't have any cooking facilities, leaving me dependent on restaurants. I certainly don't need to be afraid of starving, in Tai Wai, in 500 m around my place, there surely are at least 10 Chinese restaurant-o-fastfoods, and I yet have to see them closed (maybe on lunar new year?) so it's just a matter of  overcoming the language barrier. But for some reason, I don't like these places very much, mostly because they don't appear to be very tidy and orderly. And, based on a few observations, the attitude and service quality of the staff is rather far from awesome. Higher-class restaurants in shopping malls, on the other hand, have service that comes a lot closer to awesome. And they also pay close attention to appearance and style. Of course, you could also tell these two classes of restaurants apart just by looking at the bill for the food.

The services in shopping malls even go to such lengths to hire a full-time toilet attendant. He stands there in the hand-washing area wearing a uniform making sure that the place remains clean. There is even a plaque on the wall with his name and photo. 

飯

Thinking that you can just learn 3 cantonese words and just walk in to a local restaurant-o-fastfood is very naïve. Of course, most places, especially on the Westerner-infested Hong Kong Island, have english-speaking waiters, but not where I live. Here you can find authentic chinese restaurants with authentic people who can't speak English.

Anyway, so a guy walks into a restaurant and tries to order using the 3 words, "chicken", "meat" and "rice". It does not work. Instead, he is showered with more questions regarding the type of sauce, the colour of rice and more. The foreigner does not even manage to say "I don't understand".

Fortunately, there was a guest who could speak some English. And when I tried it again in another place, they had a friend on the phone. No starving for that day at least!

So it begins

Right now, as I'm writing, I'm sitting on a plane to Hong Kong where I'm going to work in iPatrolTech.  It's the result of about 4 months of planning and some paperwork and I got the idea that I should change something as early as March 2012. Before that, I was working in a small company in Olomouc. The work was interesting and I was (hopefully still am!) friends with most people there, had fun and everything was comfortable. But I knew I had to step out of my comfort zone while I was still young, so I quit and started looking for new opportunities.

The people in the company understood and before I left, they asked me to tell my friends about the free programmer position. This, unexpectedly, was a great lesson for me.  Their product (MoNET) is not (yet) very famous, but it's quite interesting and can do things that few others can duplicate.  It allows you to create an information system, complete with web, desktop and Android GUI within mere days. And as I said, I liked the company, so I didn't hesitate about trying to find someone for them. I put an ad on the job board at two IT universities in Brno and told about it to everyone I met. And I really didn't expect that I would not find anyone. Even people who didn't appear too bright turned the offer down.  You can see it on the internet every day that there is a shortage of programmers, but as with many things, experiencing > hearing. Now I know much better how valuable my skills are. And, if you're a programmer near Olomouc and would like a full or even flexible part-time job, fire up your email client and let me know at me@rplasil.name!

Let's get back to the plane now. Before I get dinner, which should be marinated lamb with basmati rice, I'm going to describe how I even got the job in HK. My first idea was to get an internship or a job in Sillicon Valley, but soon I found that the H1B visas are already all used up until September 2013, so I had to accept that I'll have to visit the IT Mecca some other time and that I need to start looking somewhere else. Europe, to me, seemed not very interesting. I asked some people from the GE Foundation Scholar-Leader programme for some ideas and they referred me, among others, to an organization called KOPRA. There I found an ad from a startup in HK and that's when the idea first came to my mind. I applied, got a reply, but it didn't work out. But the idea of working in HK was already there. I already visited Singapore for a few days and HK is another Asian tiger. As I was learning kung-fu, Hong Kong seemed to be a very interesting place to live in for a few months. The first company didn't accept me, so what should I do now? Well, of course, start spamming all the others! But to be honest, I first needed some encouragement from my friends, especially from Aleš. But then I started searching, writing to all sorts of places with the idea that I wouldn't stop until I knew it was impossible.  And then I found iPatrolTech, a stealth startup in the Science Park...

I think I would not have the courage to leave my home and go for half a year to literally the other side of the world if I hadn't made a few intermediate steps.  First, I went on Erasmus in Vienna. That meant living away from home for almost half a year. Last year I went to Singapore, Malaysia and Thailand, which gave me at least some familiarity with Asia.

I'm looking out of the window and hope to see the Himalayas. But it seems we are too far south from there. The altitude is about 11 km and it's not particularly warm outside the plane, only -58 degrees Celsius. We are approaching Hong Kong at the speed of 918 km/h. HK here I come!