tag:blogger.com,1999:blog-43017183341487668582024-03-07T20:03:50.261-08:00Dubhrosadubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-4301718334148766858.post-70228844660240504392016-07-22T06:55:00.000-07:002016-07-22T07:13:27.156-07:00How to dive into a large codebase<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
</h2>
<div class="MsoNormal">
<span lang="EN-GB">Getting to
grips with a new codebase can be very difficult. Every software developer has
to dive into unfamiliar code on a regular basis, but to my knowledge there are
no good guides on how to approach the task. My job(s) for the past decade have
involved writing code, but more of my time has been spent reviewing code on
dozens of active projects and learning how to quickly dive into an unfamiliar
codebase has been crucial. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">Many
discussions on this topic focus on how to navigate code in a particular editor.
In this post I want to focus on the general techniques rather than editor
specifics (though I’ll get to my current preferred setup at the end). <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">Survey the directory
structure</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">Start at
the root directory of the project. Most normal projects will have 10 to 20
files and directories in the root. Go through these one by one and make a
1-line note of the purpose and contents of each.</span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Checklist –
at the end of this step you should be able to answer these questions</div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
</div>
<ol style="text-align: left;">
<li><span lang="EN-GB" style="text-indent: -0.25in;">What
files, if any, provide documentation</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">What
files drive the build and deployment system (for projects and languages that
don’t strictly have a build system, there’s usually a deployment system; I’ll
just refer to this as the build system)</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">Where
is the source code and is there a subdirectory structure for source files, if
so, what does the subdirectory structure represent (libraries, components,
executables?)</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">Where
is the test code (usually either commingled with the main source or in a
separate subdirectory)</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">What
are the external build dependencies required to build the project</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">What
are the build targets (usually executables, libraries, tests, documentation)</span></li>
</ol>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">Understand and Run the
Build and Tests</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">Even though
I’m often reviewing code that I’m never going to modify, I still like to start
by verifying that I can successfully build the project outputs and run the main
executable and tests (if those things exist). This step helps identify any
weird dependencies the project has, and means that when you’re finally ready to
edit code you don’t have to break flow to figure out the build system. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">If the
project has a test suite, figure out how to run it. Test suites vary a lot
across languages and projects, and in some cases can be really finicky to get
running, but it’s time well spent. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">Identify the interfaces,
inputs and outputs</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">Every
program is just a way to transform input data into output data. If you “dive”
into the middle of a large codebase and try to figure things out from the
inside out, you will fail, or at least waste a lot more time than you should. Always
start from the outside and work your way in. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">Identify
what the inputs are, and what the outputs are. Make notes describing them –
force yourself to articulate this knowledge. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">Projects
that implement an “official” API ought to be easier to comprehend, and often
they are, but don’t fall into the trap of assuming that all the inputs and
outputs are captured by the API. Many APIs provide a partial account of the
I/O, and in fact you need to understand the backend database interface and the
dataflows into the DB in order to really identify all the relevant inputs and
outputs. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">Make sure
that you identify all the inputs and outputs, that includes log file outputs
and configuration inputs. Many projects have logging outputs that give you a
very useful and comprehensive picture of what the program does. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">Structured Examination of
Code</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">Don’t just “browse”
the code. Write down specific questions that you want to investigate, like “How
are messages filtered and decrypted”. Keep focused on the point you are
investigating, try to avoid being distracted by interesting looking code. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">Make notes
describing the answer to these questions, including a function call graph and
any important data manipulations. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">When you
open a file, page down through the file, all the way to the bottom, spending
about 5 seconds skim/scanning the code per screen. I don’t have a good
explanation, but I find this really helps me to get oriented and get a feel for
the size and shape of the code. You obviously can’t absorb much of the detail
by doing this, but it answers a lot of high level questions like whether the
code is repetitive boiler plate or a bunch of simple functions or a small
number of really complicated functions. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">Understand the branching
structure</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">Thankfully
most modern projects use good distributed version control systems with sane
branching policies. You can usually figure out the branching policy quite
quickly just by looking at the history, but always check the project
documentation for specific information on this. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">Spend 20 minutes reading
the most recent commit messages and diffs</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">I time-box
this activity because for large, long-running projects you could spend an
indefinite length of time reading the changes. 20 minutes doesn’t sound like
much but it’s more than enough to get a feel for the parts of the codebase that
are under active development, which developers are working on those areas, and
whether the development is issue-driven or new-feature. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">Making Notes</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">You have to
make notes as you go, otherwise you will flounder and waste an inordinate amount
of time. If you need to dip in and out of codebases with weeks or months in
between visits, your notes will be invaluable to you the next time through. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">I start
taking notes in Workflowy. If the notes grow a lot, I switch them to a git
repository I’ve called “codenotes” just for this purpose. It has a subdirectory
for every project, with cloning instructions, so I know how to get started next
time around, along with my notes. If you’re spending a lot of time on one large
project, consider writing a readme for developers and adding it to the project’s
own wiki or source control. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<h3 style="text-align: left;">
<span lang="EN-GB">My Personal Setup</span></h3>
<div class="MsoNormal">
<span lang="EN-GB">I use Vim
to read code. I turned off syntax highlighting a long time ago and am convinced
that it’s far easier to quickly read and comprehend code without it. Actually I
use the nofrils color scheme that has no syntax highlighting but does make
comments a very slightly different color to the code. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">I
occasionally use folds (two keystrokes will hide all the code except the
toplevel class and function declarations), but they are not crucial. I use the
NERDTree plugin to browse the directory structure, but again I don’t think it’s
crucial. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">I have set
up a few keyboard shortcuts that make it quicker to load files and switch
between files.</span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="text-indent: -0.25in;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="text-indent: -0.25in;">Buffers:
nnoremap Leader <leader>b :ls<cr> :buffer<space></space></cr></leader></span></div>
<div class="MsoNormal">
<span style="text-indent: -0.25in;">Files:
nnoremap Leader <leader>e q:iedit **/*</leader></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">I’ve used
tags on and off over the years. If you work with languages for which tag
support is mature, then they’re good, but several of the languages I need to
work with are still working out tags support (javascript and others), and the
time needed to set up the finicky toolchain isn’t worth it in my view. There isn’t
enough of a difference between tags and grep in my view for me to spend time on
tags that don’t just work out of the box. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB">I also
occasionally use Atom, Visual Studio, VS Code, neovim, and a few other editors
and IDEs and find them all to be perfectly acceptable, I’m just more productive
in Vim. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<br />
<div class="MsoNormal">
<br /></div>
</div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com1tag:blogger.com,1999:blog-4301718334148766858.post-54448859868317354852016-06-17T08:28:00.001-07:002016-06-17T08:38:26.077-07:00Browsing is Broken Part 3: Privacy<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: left;">
Access Provider Privacy</h3>
<div>
Whenever you connect to the web, you're connecting via some kind of access provider. Most people will think of their ISP (internet service provider) aka your home broadband provider, but these days we're constantly connecting our phones to wifi at work, in cafes, shops and airports. Many phone network providers are teaming up with wifi networks so your phone will automatically connect to wifi spots around your city, and the latest phones support making calls and texts over the wifi connection. </div>
<div>
<br /></div>
<div>
My privacy requirement is that when I connect to an access point, my web traffic is protected from the access provider, and that they can't see what I'm browsing or read the emails or messages that I download over their wifi. You might feel that this is unnecessary; can't we just trust the access providers? I'm not going to get into that here, other than to point out that you're also trusting every individual tech geek that works at those companies, and a lot of small technology outsourcing companies that they will use for IT installation and support. You're also hoping that they haven't been hacked by malicious individuals, and that they never will be hacked (probability:zero). And regarding your need for privacy, even if you are entirely blameless, consider the possibility that one day a friend or relative sends you a "private" message in which they joke about something that looks illegal or sinister when taken out of context. Their privacy is dependent on your privacy. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
VPNs</h3>
<div>
OK, let's start simple here. Say your company has a bunch of computers in two offices in different cities. Each office has its own private network, connecting just the computers in that office to each other. Naturally, you'll want to be able to connect the two networks together (an '<b>inter</b>-office-<b>net</b>work'!). That's the "N" in VPN. So you connect the two with a cable from one office to another. These days, unlike when the telegraph first arrived, you don't lay the cable yourself, you lease one from the phone company. Everything works great, the connection is Private (that's the P), but, leasing a line is really expensive. And since the internet is already available for free, why not use that instead? So you want a private network that goes out over the public internet, so you need some fancy software that creates a Virtual reality (that's the "V") style simulation of a Private Network on top of the public internet. </div>
<div>
<br /></div>
<div>
That's where VPNs come from. These days, you can download a VPN client to your phone or laptop, and connect to a cloud-based VPN server. Now, it's as if you have a cable connecting your device to that server directly, throught the magic of encryption and internet routing. Any traffic that goes over that tunnel can't be accessed by the real devices in between, such as the wifi router in the cafe, because it's encrypted and only the VPN server knows how to decrypt it. </div>
<div>
<br /></div>
<div>
So, VPN clients are a great solution for maintaining privacy from your access provider, right? It's true they provide a potential solution, but there are pitfalls. The VPN client on your phone can stop running, or need to reconnect to the server, while this is happening all your web traffic is susceptible. Even if the VPN stays up and running all the time, you can't always be sure what traffic is routed over it. Remember our two offices VPN example? Well in that situation, IT guys would still route the web traffic from PCs in the office directly to the internet - only traffic destined for the other office's machines would be routed over the VPN. </div>
<div>
<br /></div>
<div>
Most VPN client apps for phones do try to route everything over the VPN, since that's the real reason people use them. But they can still leak information. When you connect to a wifi access point, your device has to talk to it directly in order to get configuration information so that it can actually work (this is called DHCP). If the VPN client refused to let any traffic go to any destination other than over the VPN, you wouldn't be able to connect to the access point in the first place. </div>
<div>
<br /></div>
<div>
Even if you get your VPN configured as tight as possible, it's quite likely that you still leak DNS lookups (remember those from part 2?). So the access provider can't see exactly what data you're transferring, but they can see all the website addresses that you look up in order to connect to them, which is quite a lot of meta data and certainly doesn't constitute privacy. </div>
<div>
<br /></div>
<div>
The larger access providers, such as the big home broadband companies, are aware of the use of VPNs and of course they can detect when your VPN client attempts to connect to a VPN server (since they know the DNS names and IP addresses of the popular VPN services). If they wanted to, it's pretty easy for them to cause these connections to fail by blocking the initial connection, so your client can't reach the VPN server to start the whole encryption process. </div>
<div>
<br /></div>
<div>
It's also possible for the access provider to take the traffic from your VPN client and send it to one of their own servers. This requires some sophisticated NSA level techniques, but it's entirely feasible. A less sophisticated approach requires the attacker to first hack into the VPN servers and get some decryption keys, but that's not at all infeasible - most OSs have security vulnerabilities and it only takes one server to be unpatched for the attacker to succeed. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Proxies</h3>
<div>
Now that you understand VPNs, proxies are a sinch, and we already discussed them in a previous post. Essentially a proxy is a server in the cloud that your browser connects to and sends all its web requests to. It's arguably a little simpler than a VPN, and they are just focused on keeping your browser traffic private, unlike the more general purpose VPN. </div>
<div>
<br /></div>
<div>
Unfortunately, many of the popular browsers and proxies still leak DNS requests. So your web traffic is encrypted, but a snooper can easily tell which sites you're accessing. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Dissatisfaction</h3>
<div>
I'm sure when technically minded people read this, they'll suggest many possible ways of securing your web traffic from the access provider, but I've yet to find anything that a person of basic technical ability can be confident they've configured correctly and be sure they won't leak information or leave themselves open to various vulnerabilities. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com0tag:blogger.com,1999:blog-4301718334148766858.post-44486721045081953312016-06-17T01:08:00.002-07:002016-06-17T01:40:52.641-07:00Browsing is Broken Part 2: Blocking Unsolicited Content<div dir="ltr" style="text-align: left;" trbidi="on">
In part 1, I explained why I want news websites to send me their content directly, instead of passing me off to third-party advert networks that they have no control over. Since that isn't going to happen any time soon, we have to find ways to stop our browsers fetching potentially damaging content from the third-party servers that the media companies refer us to. The most popular options are ad blockers, proxies and blacklists<br />
<br />
<h3 style="text-align: left;">
Ad Blockers</h3>
Adblock is a browser add-in that hides ads. Your browser still fetches and downloads the ad, but then Adblock steps in and stops the ad from being displayed, or the video from being played. For many people, this is a fine solution, and Adblock is justifiably very popular. I used it for a while myself.<br />
<br />
The problem with most ad blockers is that your browser still requests and fetches all that ad content in the first place, and then, the ad blocking software runs, taking up more time and cpu, in order to remove the ad. The advertising networks of course know about ad blockers, so they try to circumvent them, disguising ads, so the adblockers let them through. Some adblockers can really slow down the browser quite a bit, and use up a lot of cpu and memory.<br />
<br />
The advertiser vs adblocker battle feels a lot like virus writers vs virus scanners. Both sides have to run as fast as possible just to stay still, and the users never quite know who is in the lead. When the virus writers jump ahead, the consequences are devastating, in part because the Darwinian selection pressure means that a successful virus has to be immensely sophisticated and difficult to eliminate.<br />
<br />
There has also been a backlash against ad blockers from technology companies. Google removed Adblock from the play store in 2013, and many ad blockers have been removed from the Apple Appstore over the years too. So as a user, you can't rely on these apps being available on your device indefinitely.<br />
<br />
The big advantage that ad blockers have over the techniques I'll discuss next is that they are really simple to install and use. Pretty much no technical knowledge is required by the user, and if the adblocker stops working for whatever reason, the browser usually works fine.<br />
<br />
<h3 style="text-align: left;">
Proxies</h3>
<div>
A proxy is a server that a browser uses to access the internet. So the conversation becomes:</div>
<div>
<br /></div>
<div>
<b>mybrowser</b>: hey, proxy, can you get me the nytimes front page please?</div>
<div>
<b>proxy</b>: sure, I'll go get it</div>
<div>
[proxy talks to nytimes]</div>
<div>
<b>proxy</b>: here you go mybrowser</div>
<div>
<b>mybrowser</b>: thanks proxy! ok, let's look at this html, ok, gotta get a bunch more stuff</div>
<div>
<b>mybrowser</b>: hey proxy, can you get me all this stuff from strange_address_1 through strange_address_200</div>
<div>
<b>proxy</b>: sheesh, sure, whatever, coming right up....ok here you go:</div>
<div>
<b>mybrowser</b>: thanks! ...ah crap this is huge.</div>
<div>
<br /></div>
<div>
Every browser supports configuring a proxy to talk to, because in many corporate networks, the only way to access the web is via a proxy. This helps IT control and monitor web access, and partition bandwidth so that you syncing your iTunes library on your work PC doesn't interfere with corporate email traffic, which would go on a different route. </div>
<div>
<br /></div>
<div>
Once you have all the web traffic going through a proxy server that you control, it's easy to do things like set up a blacklist of sites that the proxy refuses to access. So if a browser requests hidden.malware.site.xyz.com, the proxy says "Access forbidden" or something equally sinister. Naturally, it doesn't stop there, and lots of corporations set up their proxies to stop people accessing facebook. The smart ones let them access facebook but have the proxy log every access and track how much time the employee is goofing off. </div>
<div>
<br /></div>
<div>
The functionality developed for corporate proxies is very close to what we need for unsolicited content blocking, and sure enough, there are ad-blocking proxies that do a great job of removing unsolicited and potentially harmful content. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Cloud proxies</h3>
<div>
Most proxy services are cloud-based; you get the address details of the proxy, you input them into your browser proxy configuration, and from then on, your browser asks the proxy server in the cloud to fulfill all your requests. </div>
<div>
<br /></div>
<div>
There are a couple of problems with this. Sometimes, the cloud proxy server is in a different country to you. So you get the google homepage for Romania (no joke, happens to me a lot when I use privatetunnel), and google asks you all the time if you want to translate the page into Romanian. </div>
<div>
<br /></div>
<div>
When gmail sees you logging in via a proxy, it will probably have a bit of a fit, and ask you to reauthenticate, and prove that you're a human with one of those squiggly text things. Also, this will keep happening, because when you go through proxy services, the server you go through changes regularly. From gmail's point of view, it looks a lot like someone's trying to hack into your account from dodgy locations that keep changing. </div>
<div>
<br /></div>
<div>
Another problem with cloud proxies is that your requests have to make the extra trip to the proxy server. Usually this isn't a big overhead, but it can mean that your browsing feels slower. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Local Proxies</h3>
<div>
A proxy is just a software program, so you can install one on your PC. One of the best is a program called Privoxy, which works really well and can be configured to do whatever you need. Using Privoxy installed locally on your PC has none of the issues of using a cloud proxy. Websites like gmail don't see any difference when you access them. Unlike cloud based proxies, your web requests don't have to jump through a remote server, so your browsing should feel as fast as it does with no proxy - in fact it might feel a bit faster because Privoxy will filter out ads and other content. </div>
<div>
<br /></div>
<div>
The downside of Privoxy is that it can require a bit of technical knowledge to set up and maintain. If you configure your browser to use the Privoxy proxy, then if Privoxy isn't running, you'll get a message saying "Proxy server not accepting connections" or something similar. If you've read this far, and installed Privoxy, I'm sure that won't be a problem for you to figure out, but it may not be something you install for non-technical friends and family. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Blacklists</h3>
<div>
Every time your browser loads a page, it starts by converting the "human" name of the website, like www.example.com into an IP address. It does this by accessing the domain name system, or DNS, which is like a big database that maps all the web addresses on the internet onto IP addresses. If no entry for the human name is found, the browser doesn't know what IP address to send the request to. If it can't find the address for the name you typed into the address bar, you see an error message like "www.bleaurgh223A.com not found". </div>
<div>
<br /></div>
<div>
When your browser gets the html for a page from the primary site, like nytimes.com, it reads the html and fetches any content required to complete the page. The html will have addresses telling the browser where to go. This is how your browser ends up fetching content from adnetwork3vil.dblclack.net when you want to read nytimes.com. It asks DNS for the IP address of adnetwork3vil.dblclack.net and then fetches that anorak ad. </div>
<div>
<br /></div>
<div>
Browsers are very forgiving, and they expect errors to happen. This is good, because if you look at your browser console (a hidden debugging window you can usually access by hitting F12), you'll see that almost every page you load has some errors. Often these errors are due to broken links. The guy who created bestcatpictures2003.com linked to lots of cat pictures on websites that no longer exist. The browser expects this kind of thing, so it just does its best and displays whatever parts of the page it could successfully get. </div>
<div>
<br /></div>
<h4 style="text-align: left;">
Hosts File Blacklisting</h4>
<div>
Before DNS became cloud-based, computers had to have a file that listed all the mappings from human names to IP addresses. This is the "hosts" file, and it's still located in /etc/hosts on most unix systems, and C:\Windows\System32\drivers\etc\hosts on windows. The operating system still checks this file every time the browser makes a DNS request, just in case it has an entry. If it finds an entry, it's much faster than asking the cloud based DNS. These days, the hosts file usually contains just one or two entries, but there's no reason you can't add more.</div>
<div>
<br /></div>
<div>
That's how we create a blacklist. We add entries into the hosts file and map them to bad IP addresses, e.g. 0.0.0.0. When the browser asks for the address of adnetwork3vil.dblclack.net, the operating system checks the hosts file, and finds an entry mapping adnetwork3vil.dblclack.net to 0.0.0.0, the browser tries to fetch content from 0.0.0.0, which fails, but the browser is built to expect such failures, so the rest of the page loads just fine. </div>
<div>
<br /></div>
<div>
A lot of people work to create these blacklist files, and you can download good ones for free on the web. Although the description of how all this works is a bit technical, installing a hosts file is just a matter of backing up your existing file and copying in the new one to the right location. After that, you might want to get the latest version every few months as new sites are added, but there's really no maintenance required. </div>
<div>
<br /></div>
<div>
The main catch with using the hosts file for blacklisting is that you need to have administrator access on your device. For PCs this isn't usually a problem (you definitely have admin access on your home PC), but on an Android phone, it means you need root access, which requires some technical knowledge. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Dissatisfaction</h3>
<div>
None of the methods we have to avoid unsolicited content is entirely satisfactory. I currently use a hosts blacklist on all my devices, and I really like the results. Ad blocker browser plug-ins are the best solution for non-technical users, but Google and Apple have shown that they are opposed to allowing us to use them. At the time of writing (June 2016), adblock apps are available, let's hope it stays that way. </div>
<div>
<br /></div>
<div>
Privoxy is probably the best solution overall - it gives you complete control, and nobody can stop you installing it on your laptop/PC. Unfortunately, in order to get Privoxy working on your phone, you need it to be jailbroken or rooted. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com0tag:blogger.com,1999:blog-4301718334148766858.post-66317910048379754852016-06-16T22:49:00.003-07:002016-06-18T09:16:26.049-07:00Browsing is Broken Part 1: Unsolicited Content<div dir="ltr" style="text-align: left;" trbidi="on">
The websites of many of the major news outlets that I used to read regularly are now overloaded with ads and content from third parties that I just can't tolerate any more. I started to notice how bad things were getting about three years ago when visiting rollingstone.com on my Android phone exposed me to malware that made a charge to my mobile bill. It's not just inconvenient, it's insecure, and ultimately it's lose-lose for the media and their audience.<br />
<br />
I get the business model that online companies need to sell advertising, and in principle I support that absolutely. Heck, I tried signing this blog up for Adsense in the off chance I can finally make a dollar back off Google (they turned me down). What I don't agree with is the way they implement it. Simplistically, when I enter nytimes.com in my browser, the conversation between the computers involved goes something like this:<br />
<br />
<b>myphone</b>: Hey, nytimes.com can I get the front page please?<br />
<b>nytimes</b>: Hang on a sec...just looking you up...<br />
<b>nytimes</b>: Hey adservers, dubhrosa just asked me for my front page, whaddya got for him<br />
<b>adserver_network</b>: oh baby! dubhrosa, I've got a ton of stuff for that guy, I'll send it directly to him if that's ok.<br />
<b>nytimes</b>: Yeah sure, go nuts, I'll send him the headlines and some pictures, I'll leave most of the page for you guys<br />
<b>adserver_network</b>: Great! Last week he bought an anorak from amazon. Maybe he'd like to see a couple more ads for anoraks. Also, a few months ago, he clicked on an ad for septic tank inspection services, it might have been a misclick, and we've shown him about 2000 more of the same ad since then, but hey maybe today's the day. Oh, and he seems to be into cars, so lets put on that video for the new Ford truck that starts to play automatically.<br />
<b>nytimes</b>: ok great, thanks dudes<br />
<b>adserver_network</b>: sure thing nytimes, here's your 0.001c<br />
<b>nytimes</b>: hey thanks! nice tip! you guys are sooo nice!<br />
<b>myphone</b>: ok, here's the html for this nytimes front page, thanks nytimes<br />
<b>nytimes</b>: my pleasure<br />
<b>myphone</b>: ok, in order to display this page, I need to go fetch a crapload of pictures and stuff, let's get that<br />
<b>myphone</b>: hey, strange_name_1 through strange_name_200, can I have this stuff please?<br />
<b>adserver_network</b>: [teehee, they never know it's us] sure! here you go!<br />
<b>myphone</b>: yikes, they're sending me 50 megabytes of crap here, oh well this is gonna hurt my data plan and my battery.<br />
<br />
<br />
Here's how the conversation should go:<br />
<br />
<b>myphone</b>: Hey, nytimes.com can I get the front page please?<br />
<b>nytimes</b>: Hang on a sec...just looking you up...<br />
<b>nytimes</b>: (to self) ok, what ads do I have today that I should show dubhrosa...ok, stick them into the page<br />
<b>nytimes</b>: here you go, this is the front page html<br />
<b>myphone</b>: thanks nytimes<br />
<b>myphone</b>: ok, there's some other stuff I need to download from nytimes to complete the page<br />
<b>myphone</b>: nytimes, give me these pictures and video links please<br />
<b>nytimes</b>: here you go<br />
<b>myphone</b>: thanks!<br />
<br />
The key difference is that in this flow, the nytimes is responsible for storing and serving the advertising content to its readers. The ad content is stored on their servers, and their staff have the ability to control that content. They can still target me with ads they think are relevant based on my previous online activity, but they have full control over the content that is sent in response to my request. They can keep the page size below some sensible limit. They can ensure their readers have a nice experience when browsing their site. I don't think any of this is unreasonable demand. Imagine if a newspaper editor allowed advertisers to scrawl whatever they wanted into the adspace of the newspaper, with absolutely no review by the newspaper staff. Shouldn't media companies, whose brand is so important, take control of what they send to their readers?<br />
<br />
Unfortunately, the way the online ad industry has turned out means that this is unlikely to change, and in order to make browsing tolerable, we have to find solutions. I've looked at quite a few, and that's what I'll be talking about in part 2. <a href="http://dubhrosa.blogspot.com/2016/06/browsing-is-broken-part-2.html">Read Part 2</a><br />
<br />
<br />
<br /></div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com0tag:blogger.com,1999:blog-4301718334148766858.post-48456580914152400682016-01-19T00:23:00.000-08:002016-01-19T00:24:15.295-08:00A Personal Finance Application Wish List<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="Publishwithline">
Many years ago I used Microsoft Money, so when I recently looked for a personal finance app, I was surprised to find Microsoft end-of-lifed the product. Even more odd was that nothing seems to have come along to replace it. It seems that building an app to
track personal finances should be relatively easy, and I’m sure many
programmers would have noticed that it’s a “point of pain”, so why isn’t there
anything really good out there?</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
A bit of digging revealed that there are a couple of good apps, like Mint, and
some of the other spending trackers from the banks. Though many of them are region centric, so if you're not in the US or the UK, you might be out of luck. One of the leading
independent apps appears to be YNAB, and it’s great for what it is, but it’s
not a full featured personal finance application. I now use YNAB (much more on this later). <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The problem may be that there are several distinct
activities that I want from my PFM (personal finance manager), and these
activities get confused all the time. There isn’t a convention that I know of
for what these activities are called, so for now I’m calling them Tracking,
Budgeting, Planning, and Cashflow Planning. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Tracking<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
Tracking is pretty simple – you just record all your
transactions, add some meaningful categories, and at the end of the month you
can look at how much you spent in each category. This was the way I used
Microsoft Money a lot of the time. When I was using it, the mere fact that I
was tracking everything and analyzing it regularly had a definite impact on my
spending. I was much less likely to spend on trivial things. I understood that
my bank balance was just a number, and the real picture could only be seen when
all the commitments that money had to serve were taken into account. <o:p></o:p></div>
<div class="MsoNormal">
Tracking is the least complex feature of a PFM, and it
probably delivers you the bulk of the benefits. Mostly because you’re forming
the habit of regularly analyzing your spending, and breaking the illusion that
the cash in your bank account is money you can spend. Perhaps part of the
reason that there aren’t too many good apps out there is that this is easy
enough to do with a spreadsheet, and many banks and credit cards now offer this
kind of view of spending. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Budgeting<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
To most people, this means writing down all the transactions
you think are going to happen in the next month or longer. Then you either try
to reduce some (if you don’t have enough money to cover them), or you see that
you’ve got enough money and you’re done. Then at the end of the month you check
that you’re within budget (you are almost certainly not), then you feel bad for
a bit, resolve to do better in future, and eventually give up budgeting. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Until I used YNAB for a while, I didn’t realize that my idea
of budgeting was nothing like true budgeting. YNAB is a software version of the
old “envelope” system, where back in the days of paper paychecks and using cash
for everything, the thrifty household would cash their paycheck, then put
amounts of cash into envelopes labelled with the categories they were for. An
envelope for Groceries, one for Heating Oil, another for Clothing and so on. If
you were looking further out, you might have one for Christmas, and another for
the Vacation you were planning for next year. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
There’s a certain genius to this system, and more than any
other, it forces you to break the delusion that the cash you have is the cash
you can spend. Even better, if you do spend too much on something trivial, you
have to physically take cash out of one of the other envelopes to do it; and it’s
kind of tough to let yourself take money out of the Kids College Fund or
Vacation to go bowling.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
YNAB is a very opinionated piece of software (a good thing I think). It’s tough to
learn how to use it in the intended way without spending a lot of time reading
the docs and watching the excellent online videos. I’m thankful I stuck at it, I
don’t think I would have ever really understood what budgeting means otherwise.
That’s probably why there seems to be a steep learning curve for such a simple
piece of software – it forces you to look at budgeting a different way. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Planning<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
I’ve grown to respect the YNAB approach and I hope I’ll
always see budgeting in this new powerful way. However, I also need to plan.
The YNAB way is that you don’t allocate money to any category until you
actually have that money. So when you’re starting out, you rarely budget more
than 1 month out. So let’s say you get paid on the 1<sup>st</sup> of the month,
and your mortgage payment is paid on the 2<sup>nd</sup> of the month. In the
YNAB budget view, you don’t show that mortgage payment on your budget until you
have the cash for it. Now I get why this is right, and I get that forcing you
not to budget until you’ve got the cash is the best way to be realistic and
build up a proper buffer so you’re paying this month’s expenses with the
paycheck from a month or two ago. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
But. I also want to be able to see a plan or a projection of
those transactions. It’s important for me to think beyond the next month and
remember that I pay my car insurance annually in March, and that’s nearly 700.
In YNAB, the way you deal with this is by creating a category / envelope for
car insurance and put enough into it each month so you’ll be able to pay the
700 in March without screwing up your cash position. That’s fine. But there isn’t
anywhere in YNAB where you can see that car insurance of 700 is due in March,
and property tax is due in April, and school supplies are due in September, and
so on. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
I also need a planning system that helps me make big
decisions. Like what happens if I switch jobs to something that pays less? A
short term budget doesn’t give me what I need here. Should my wife go back to work?
What happens if my son gets into an expensive school? These questions require
plans that go months and years into the future, and sometimes, the output is
along the lines of “get a second job or a big promotion buddy, you need way
more cash”. And that’s ok (I’ve done this in the past, many people do), but we
should have tools that make it easier to explore the what-if scenarios. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Cashflow Planning<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
I also need to resort to a spreadsheet in order to figure
out how much cash I’m going to have in my checking account, my family joint
account, and what the balance is going to be on any credit cards we use. The
purist budgeter approach is that this stuff doesn’t really matter – you have
income coming in, when you receive that income, you allocate it to your
expenses, and when you pay those expenses, it doesn’t matter which account it
comes from, what’s really happening is income is being moved to outflows. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Again, this took some getting used to, and when it finally clicked,
it made a ton of sense and I can see why it’s the right way to think when
budgeting. But. I still need something that tells me when our joint account
needs to be topped up so it doesn’t go overdrawn. It would be nice if I could
see when that was likely to happen, and since I can input the expenses I’m
going to be paying this month and next, along with the dates and amounts, I’m
pretty sure the computer could do this for me. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
More urgently, when you first adopt YNAB, you’re probably
living paycheck to paycheck. In fact, you’re probably living off next month’s
paycheck, i.e. spending on a credit card and then paying it off when you get
paid, then repeating the cycle. While you work to fix this, and move to living
off income you already have in the bank, you really need help managing your
credit cards. The most urgent question being: which expenses should I pay with
my credit card this month, so that I don’t put my checking account into
overdraft? Some expenses can’t be paid with credit card, so it’s not good
enough to say that you pay everything out of your checking until that’s used up
and then switch back to credit. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Overspending and
Amendments<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
When you overspend in YNAB, or you add an item to your
budget mid-way through the month, you overwrite the budget with this new
information. There’s no way to see what your original budget was versus how
reality ended up. Other users have pointed this out, and the YNAB response is
that it’s just the right way to do things. I think it may be ok to display the
current budget without all the clutter of what the original amounts were, but
the app should show you how often you underestimate categories and by how much,
and how often you completely omit items. Then, when you’re budgeting next
month, a helpful tip would be text like “You underestimate the total budget by an
average of 15% each month, and regularly overspend in Entertainment by 50%”. I
think it would also be a helpful discipline for the user to be asked to “declare”
that a budget is final. As it is now, it’s too easy to keep tweaking and there’s
no way of tracking what the original intent was.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">The Huh…? Factor<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
The first months I used YNAB, I entered every transaction
diligently, I allocated my income, I amended my overspending as recommended,
and I created a buffer category to start building toward living off old income.
I also cut back on a lot of spending, making pretty significant changes to get
things under control. But when I put all this in YNAB, and looked at the
reports and the budget screen, I had no clue what it all meant. Was I doing
well? Did I have enough cash to cover my main expenses next month, assuming I
got paid as usual? I had no idea, and I still think it’s tough to see. Yes, you
need to stop thinking of the cash in your bank account as the money you can
spend, but it would be nice to be able to move into “budget space” in YNAB, and
then have it translate back into “cash and time” space. Putting it simply, if I
give the computer my current balances, all the upcoming transactions with dates
and amounts, it should be able to tell me what my balances will be in future. I
know this isn’t budgeting, but it’s an important part of what I want from a
personal finance app, and if I find another app that does this, I’m less likely
to enter my transactions and projections into both YNAB and that new app. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Commitments, Covered,
Discretionary<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
In a budget app, I’d like to be able to label some future
expenses as “Hard commitments”. These are things I’m legally obligated to pay,
like rent or mortgage, my internet service, gym fees, anything where I’ve
signed a contract and I can’t just decide to reduce my spending in that
category in the short term. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
When income comes in, the budget app should allocate to the
Hard Commitments automatically, in the order that they will have to be paid. If
there’s income left over, then I can use that to allocate to the discretionary
categories. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
One solution to resolve the conflict between YNAB’s idea
that you only enter the expenses that you have income to cover, and the need to
do this to properly plan future cashflows, might to make it possible to enter
future month’s forecast outflows, but to clearly mark them as “uncovered”. When
income is allocated to outflows, they are “covered”. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
So let’s say you’ve got no income yet this month, and a
couple of hard commitments in your budget for the month along with some
discretionaries. The hard commitments should be colored red, making sure you
can’t miss the fact that if you don’t get some source of funds to pay them, you’re
in trouble. The discretionaries can be colored amber, indicating that they’re
not covered. As income arrives, the hard commitments turn green, as do the
discretionaries. The commitments for future months stay red until you’ve got
the cash to cover them. This way, the budget view will eventually show you with
a month, or two, or three, with all categories in green, making it clear that
you have a buffer and how far into the future it reaches. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Hopes<o:p></o:p></b></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;"><br /></b></div>
<div class="MsoNormal">
The ideal outcome for me would be that YNAB adds functionality
to their app that makes it easier to plan and see a “cash and time” view of my
data. Or perhaps, other apps get built that can read the YNAB data files, so I
don’t have to have separate records. I love the way YNAB just uses dropbox to
sync and share the data files, and it looks like they’ve designed a good format
for the data. Maybe what we need is for this format to be made “open” or
somehow standardized by convention so that app developers can all read the user’s
data, no need to rekey or import. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<w:sdt docpart="01502E358F154E49A7446966E473E3FA" id="89512082" storeitemid="X_5F329CAD-B019-4FA6-9FEF-74898909AD20" text="t" title="Post Title" xpath="/ns0:BlogPostInfo/ns0:PostTitle">
</w:sdt>
<br />
<div class="MsoNormal">
I’m sure there are a bunch of great apps out there that I’m
totally unaware of, or perhaps the right thing is to get a professional
accounting package like Sage or Quicken and do all of this the way the pros do
it. Let me know what I’m missing! <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
</div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com0tag:blogger.com,1999:blog-4301718334148766858.post-76299633063412655972014-07-18T05:18:00.002-07:002014-07-18T07:42:21.800-07:00MH17 flight path history<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
MH17 crashed in Ukraine on the 17th July 2014, with the loss of all on board. According to reports aircraft was flying directly over the the disputed territory, at an altitude of 33,000 feet. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Eurocontrol reportedly said that they had kept the corridor open even as Ukrainian authorities banned any aircraft flying below 32,000 feet. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The historical flight paths followed by MH17 show that the flight on the 17th followed a route further to the north than previous flights, taking it over the disputed territory. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
12-July, almost completely avoids Ukrainian airspace, flies
to the south over the Black sea:</div>
<div class="MsoNormal">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-pDUl8nvhlBg/U8kC65E-NtI/AAAAAAAANyg/camLGwpmLEM/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-pDUl8nvhlBg/U8kC65E-NtI/AAAAAAAANyg/camLGwpmLEM/s1600/1.png" height="297" width="640" /></a></div>
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
13<sup>th</sup> July, slightly further north, well inside Ukrainian space:<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-giWyzrABXRg/U8kC631KgPI/AAAAAAAANyk/fG93ghDR60M/s1600/2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-giWyzrABXRg/U8kC631KgPI/AAAAAAAANyk/fG93ghDR60M/s1600/2.png" height="324" width="640" /></a></div>
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<o:p><br /></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
14<sup>th</sup> July, entered Ukrainian space further north, but routed far to the south of the disputed territory<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-V0xpEwqigNM/U8kC60hlqtI/AAAAAAAANzQ/V6Y38iYRxEo/s1600/3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-V0xpEwqigNM/U8kC60hlqtI/AAAAAAAANzQ/V6Y38iYRxEo/s1600/3.png" height="382" width="640" /></a></div>
<br />
<br />
<br />
<div class="MsoNormal">
15<sup>th</sup> July, similar to the previous day’s route</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-JrJ8Y2NkwTc/U8kC7WN5sRI/AAAAAAAANyo/pkADmX6v6r0/s1600/4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-JrJ8Y2NkwTc/U8kC7WN5sRI/AAAAAAAANyo/pkADmX6v6r0/s1600/4.png" height="384" width="640" /></a></div>
<br />
<br />
<br />
<div class="MsoNormal">
16<sup>th</sup> July, Way farther north, skirting the disputed territory<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-OrFbDXyR-iw/U8kC77pskkI/AAAAAAAANzA/VJDpBPm0ZXg/s1600/5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-OrFbDXyR-iw/U8kC77pskkI/AAAAAAAANzA/VJDpBPm0ZXg/s1600/5.png" height="370" width="640" /></a></div>
<br />
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
17<sup>th</sup> July, Further north still, directly over the disputed territory, with tragic result<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Jm5jAJbu118/U8kC8fcdpbI/AAAAAAAANyw/TnVZ8RedXWk/s1600/6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-Jm5jAJbu118/U8kC8fcdpbI/AAAAAAAANyw/TnVZ8RedXWk/s1600/6.png" height="358" width="640" /></a></div>
<br />
<br />
<br />
<div class="MsoNormal">
As reported in some media, other airlines have routed away from the disputed territory’s airspace for months<o:p></o:p></div>
<div class="MsoNormal">
e.g., Air India Delhi to Frankfurt<o:p></o:p></div>
<div class="MsoNormal">
<o:p></o:p></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-gK4BpW0gpxM/U8kC9DfKjUI/AAAAAAAANy8/1fOFbtwpahA/s1600/7.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-gK4BpW0gpxM/U8kC9DfKjUI/AAAAAAAANy8/1fOFbtwpahA/s1600/7.png" height="368" width="640" /></a></div>
<br />
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<o:p> All images snapped from <a href="http://flightaware.com/">flightaware.com</a>: </o:p></div>
<div class="MsoNormal">
<o:p><br /></o:p></div>
<div class="MsoNormal">
<o:p><a href="http://flightaware.com/live/flight/MAS17/history/20140718/1000Z/EHAM/WMKK">http://flightaware.com/live/flight/MAS17/history/20140718/1000Z/EHAM/WMKK</a></o:p></div>
<div class="MsoNormal">
<o:p><br /></o:p></div>
<div class="MsoNormal">
<br /></div>
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<br />
<div class="MsoNormal">
<br /></div>
</div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com0tag:blogger.com,1999:blog-4301718334148766858.post-4024870349133341512013-08-01T09:06:00.000-07:002013-08-02T08:55:33.430-07:00<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
Good Intentions</h2>
<br />
<h3 style="text-align: left;">
The biggest waste of time I know.</h3>
<br />
I'm one of those programmers who spends a lot of his time wondering "how could I do this better/easier/faster/safer/simpler". Often the conclusion is recognising some common patterns across a codebase or refactor functions into more meaningful units, the bread and butter of keeping your code in shape. Quite often however, "this" becomes programming itself, or more accurately, application development. I start out asking how to make a module better, and trace the question back until I'm questioning the overall architecture of applications in general, rather than the particular program I'm working on.<br />
<br />
For example, a little project I'm working on at the moment has a data access layer. It's a small file of functions that wrap database calls. There's a certain amount of marshalling and demarshalling, and a few hard coded mappings from query results to types. So naturally, I'm asking "why can't I generate all this code automatically and not even have to think about it, and have something automatically generate typed mappings? I could just give it my types and it should figure out everything else!" And so I spend a few days reading about ORMs, trying them out, debugging exploratory code, getting a buzz from seeing something work "automagically" and then getting battered and bruised by dependencies, versioning issues, performance gotchas that are now opaque to my reason, eventually whimpering back to my simple wrappers, cleaning them up for an hour and moving on. Yeah, yeah, there'll be a few bugs in that module and I'll probably come back to it a few times, but doing it this way is highly likely to be better than downing tools and rearchitecting the entire project to use an ORM. (Ok, I knew about ORMs a long time ago, but it's a typical example).<br />
<br />
And that's a real time waster. Probably the biggest waste of time in my entire life.<br />
<br />
Don't get me wrong, about 5% of the time, I'll learn about a method or library or framework or approach that really is better. The next project I design, I'll be able to make an informed decision as to whether an ORM is appropriate or not. (Disclaimer: I said "informed" not "good"). Sometimes it's even relevant to the project. Once in a while, it's not a completely foregone conclusion or utterly obvious, so I actually get to make the decision.<br />
<br />
But 95% of the time, it leads to nothing, even if I find something interesting, it's not applicable to the current project, or the next one, and the one after that will be running on hexacore smart watches with 6G connectivity so it's probably better to wait until we're starting work on that before doing a review and making decisions about what stack to build it with.<br />
<br />
And speaking of stacks, a lot of the irresolvable flamewars are because fanboi A is arguing about how great his language is, and fanboi B is arguing about how great his entire stack is. They're both right, they're just talking about two completely different things. I'll even propose the humbly named "dubhrosa relation of language elegance": the more elegant a language, the less likely it's embedded in a productive stack. Why? Evolution.<br />
<br />
A productive stack is one in which everything important to commercial app development is basically possible and reasonably straightforward, in which you can change code in one part of your stack without creating massive explosions of infeasibility in some other part. Usually these stacks have ugly edges. In some cases, they have entire continents of ugliness, like stacks with PHP in them.<br />
<br />
Perhaps this ugliness is a kind of a genetic trait that hints at what drove its creation. Take perl, or PHP. They're both ugly. There are pockets of elegance and undoubtedly they are "powerful", but in general, you can tell they grew without any especially well-grounded master design. You can also tell that the people driving the development of these kinds of languages were really focused on making stuff work. I can be fairly confident that perl and PHP will allow me to connect to any major database. There might be a few ugly bits, but I know it's really unlikely that I'll find myself boxed into a corner, like not being able to connect to a major db, or being able to, but ending up rewriting the standard library for doing so because it's such poor performance. I don't have the same expectation with Haskell or OCaml, for instance.<br />
<br />
So, in conclusion, if you're building an app these days, you should pick a stack, a popular, well used one, and stick with it for the duration of the project. Learn to love the one you're with. When you encounter overwhelming ugliness, put it on your workflowy list under "There must be a better way - discuss", write the code that makes you cringe, comment it so everyone knows you're aware of how ugly it is so there's no risk to your programmer-tribe status levels, and move on.<br />
<br />
<h3 style="text-align: left;">
Gratuitous blogpost controversial statements that I probably don't really mean exactly </h3>
<br />
Let's put some languages in order of ugliness:<br />
<br />
Python,PHP, perl, Java, Ruby, C++, C#, F#, Haskell<br />
<br />
(yeah, yeah, Python really is way uglier than C++, just not at first glance, it's the beer-goggles language. Its ugliness is hidden under a layer of cakey makeup and whitespace and bitchy bravado. It tells you how much you deserve list comprehensions and before you know it you're in too deep. You're shouting at the screen "if I wanted a glorified dictionary wrapper masquerading as a programming language I could have built my own! a fast one!". It's uglier than PHP in a deep way. PHP sits in the corner of the library sniffling and covered in acne but is honest and friendly and if you ask him out on a date he says "are you sure?" three or four times. Then he brings you to the fairground and you have a great time up until you fall off the rollercoaster (did he get too excited and push against the guardrail? you'll never know for sure) and break your arms and wake up in hospital and somehow you've caught his acne and his cold but he's there beside your bed with video games and taytos. Python, by contrast, roofies you with syntax and when you see through it, tells you that you're inadequate if you don't understand how great it is, and when you finally leave, he spams your twitwall for months.)<br />
<br />
And yes, Haskell is the most elegant, by about a billion miles, don't even try to argue with me on this one. Haskell's community is fantastic, but its ecosystem sucks. Configuration management can be tricky (google cabal hell), and it doesn't fit well in any of the widespread stacks. I've tried using Haskell in a web app, first using the Haskell webapp frameworks, and then just as a CGI within Apache/postgres, pausing only to build a custom json protocol with session oriented connections and a custom client. Oh how I wept. It was like finding the partner of your dreams, perfect in every way, except they wake up and stab you in the back of your head at 3 in the morning on a regular basis. I say this in part because it's sufficiently vague that the absolutely lovely and helpful Haskell guys don't implode in exasperation because I tried version 0.1.3.2.3 of yesod and the random stabbings were fixed in 0.1.3.2.5 (and I'd explain I couldn't use that because I was using a version of the bystring library hand picked to be compatible with a particular version of the vector library that meant I couldn't upgrade to that version of yesod even if I wanted to, oh wait that's fixed now but I need to build ghc from source you say? How about the x64 bug?), and in part because it's entirely true.<br />
<br />
There's also the little discussed, but very important, matter of a project's "moron impact factor" that you need to consider when understanding the successful stacks. Think of it this way: every successful project basically never completes, it's a successful application so people are using it, asking for new features, and burdening the system with ever increased load. At some point, you'll have new programmers, old ones might leave, or you're adding because there's just more work to be done. The law of large numbers and cretinous recruitment agents means that at some point some very stupid programmers will work on your application. The amount of damage they can do in a given period (moron detection time) is a characteristic of the stack you're using. This is the hidden value of verbose languages - it takes stupid programmers so long to wade through all the verbose code, that they can't do as much damage, there simply aren't enough hours for them to get to it. What's more, it's easier to make your peace with adding a mediocre programmer to the team to add some feature, because the stack is pretty ugly anyway, so we're all acclimatised to the necessity of ugliness, so we can tolerate ugly but functional code written by a mediocre programmer who was drafted in.<br />
<br />
Please let me know when my Haskell stack is ready. Until then I'll be in a darkened office rubbing my temples and sighing.<br />
<br />
<br /></div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com2tag:blogger.com,1999:blog-4301718334148766858.post-73861842456466463802012-12-31T09:31:00.001-08:002012-12-31T14:28:34.741-08:00XVoice: speech control of Linux desktop applications<div>
<br /></div>
<h2>
XVoice</h2>
<div>
An open source speech control project up for adoption</div>
<div>
<br /></div>
<h4>
The Early Days</h4>
<div>
Around the end of the 1990's IBM released a Linux version of their ViaVoice speech recognition engine. It was always a beta product, it never had the full set of features of the original Windows program, but at the time it was the only good recognition engine for Linux, so I started playing around with it. </div>
<div>
<br /></div>
<div>
Soon I discovered an open source project, XVoice. XVoice was an application that used the ViaVoice engine for speech-to-text, but then used the resulting text to control the Linux desktop. It was a hack that used a bunch of programs in ways they were never designed for, and it achieved something rather exciting: you could speak to Linux and it would do your bidding. </div>
<div>
<br /></div>
<div>
One of the great features of the ViaVoice engine was that it allowed you to define a grammar, and then the engine would match whatever speech was input against that grammar. This meant that without training, recognition rates were near-perfect for a domain specific grammar (nicely defined in BNF). </div>
<div>
<br /></div>
<h4>
Progress and Success</h4>
<div>
After a few months of regular development in early 2000, XVoice had proper support for user-defined command grammars. These grammars mapped spoken commands to keystrokes, and you could have multiple grammars, one for each application. XVoice had some (hacky) heuristics; you could specify a regex that would match against the window title, which then would automatically load the right grammar file. You could control the mouse too, XVoice split the screen into ever smaller 3x3 grids that you navigated until the mouse was where you wanted it. The grammars were hierarchical so you could include the grammar for spelling out numbers in your emacs control grammar, and they supported pattern substitution, so the command sent to an application could include some of the words you said. </div>
<div>
<br /></div>
<div>
There were some quite motivated users who contributed a lot to the development. One was a programmer who used Vi but had severe RSI that was making it difficult for him to work. He defined a comprehensive Vi grammar that allowed him to program, and interestingly, claimed he was more efficient because he was using higher level Vi commands than he would normally. Some Emacs users had huge grammars that let them read news, send emails, program in lisp and who knows what else. </div>
<div>
<br /></div>
<div>
As an aside, my experience working on XVoice left me in no doubt that for regular people, voice control of your computer is a fun trick, but the only people who would use this on an ongoing basis are those who have no other choice. Talking for several hours a day is physically difficult, and even with the clever grammars some people designed, it's not something you'd choose to do unless you had to. We had at least a few quadriplegic users who were starting to use the system with some success, for them, XVoice was the only way they could operate a Unix machine. This realization made the future of the project clearer: we'd focus on features that helped people who couldn't type at all or only with great difficulty. </div>
<div>
<br /></div>
<div>
As a programming project, working on XVoice was just great, and I learned a lot from the other programmers who were much more experienced and capable than I was. By the end (version 0.9.6 I think...) it was a really cool program, being used by people who really needed it to work. We encountered and solved some of the problems of voice control of graphical user interfaces. The command grammar was a pretty elegant and extensible system, and I recall that the sense at the time was that most of the interesting work ahead lay in defining bigger and better libraries of grammars. </div>
<div>
<br /></div>
<h4>
An Abrupt End</h4>
<div>
Unfortunately IBM didn't seem very interested in ViaVoice on Linux. We had some contact with the developers, who were quite helpful, but the "official" IBM people would never tell anyone what the plans for continuing Linux support were. The only contact we had with them was when they demanded that we make it clear on the XVoice related websites that we didn't distribute ViaVoice with XVoice, that people had to buy it separately (which we always did). </div>
<div>
<br /></div>
<div>
Then one day IBM discontinued ViaVoice for Linux. It just disappeared from their website. At the time, CMU Sphinx was the only plausible candidate for an open-source speech-to-text engine that we could use instead of ViaVoice, but it wasn't very mature and had some issues that would have been tough to work with. The main coders on the project had personal or work issues that meant they couldn't work on XVoice for a while, and so the project lost momentum. </div>
<div>
<br /></div>
<h4>
Wistful Thinking</h4>
<div>
Every once in a while when I read an article about speech to text, particularly about command and control systems, I wish that we'd had the time to rework XVoice to work against an open source engine. It's disappointing that there's still no clear (open source or other) solution for people who can only interact using speech. Many big tech companies pay lip service to accessibility, but in this case the big boys didn't do anyone any favors.</div>
<div>
<br /></div>
<h4>
The Future</h4>
<div>
I think it's important to distinguish between projects that focus on building better general recognition accuracy for dictation, and accessibility oriented command and control systems. Complete control with speech is a difficult problem, I don't think you solve it as part of a larger generic command and control platform. You have to focus on accessibility and talk to users who have real accessibility issues, and get them to work with you to overcome them. If you are working on command and control, it's worth remembering that these people are the only ones who will be still using your software when the novelty wears off and their throat is sore. In the final few months, that's where XVoice was focusing, there were a bunch of awkward problems we'd need to fix, but it was pretty exciting. </div>
<div>
<br /></div>
<div>
The key feature that XVoice or any other command and control system relies on is the ability to feed context-specific grammars to the recognition engine on the fly. The underlying accuracy of the engine isn't very critical if the grammar is sufficiently constrained. All modern engines are likely good enough. But as far as I know, the current HTML5 implementations of speech input don't yet support setting grammars. CMU Sphinx appears to, but it's not clear how well it works in practise, their configuration files seem quite complex. </div>
<div>
<br /></div>
<div>
The XVoice code is all GPL, it's on Sourceforge and now on <a href="http://github.com/tdoris/xvoice">github </a> so please feel free to go nuts. Before today it was about 10 years since I looked at it but the docs are actually pretty good and the code isn't as bad to read as I expected. It's mostly pre-RAII C++, so would need a cleanup and a dose of smart pointers to bring it up to modern standards. Even if the project isn't resurrected, the ideas around how command grammars are structured and used might be useful to another project, or the code for generating X events could be reused. There's a sample set of grammars in the modules subdirectory that make for interesting reading - there's even one for "netscape", how quaint. </div>
<div>
<br /></div>
<div>
<br /></div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com4tag:blogger.com,1999:blog-4301718334148766858.post-22382577271805829842012-12-23T15:17:00.000-08:002012-12-23T15:17:37.681-08:00Lessons learning Haskell<br />
<br />
<h2>
Lessons learning Haskell</h2>
It's often claimed that learning Haskell will make you a better programmer in other languages. I like the idea that there's no such thing as a good programmer, just a programmer who follows good practices. As soon as we stop following good practices we suck again. So, Haskell must introduce and indoctrinate better practices that we carry back to our other languages. Right? I think it's true but it's not obvious, so I've written this article to outline some of the habits and practices that I think changed after I used Haskell for a while. <br />
<h3>
Purity</h3>
Haskell makes your functions pure by default. You have to change the type signature if you want to do IO, and mark every function between your function and main() as tainted with IO. This forces you to be conscious of IO. It encourages you to keep functions that do IO as high up the stack, close to main, as possible. Purity also means you can't read or write global variables, that's just another type of IO. If your function needs some data, you pass it as a parameter. So the type signature of a pure function is a complete itinerary of everything it can access, and therefore is a very good spec for what the function does in most cases.<br />
<br />
Experienced programmers who pay attention already know that IO and global vars mustn't be taken lightly. Every IO operation is a potential source of errors, exceptions, and failures. Functions that do IO are difficult or impossible to test. Programmers know this, but Haskell makes sure you never forget it when it matters. It incessantly shunts you in the direction of keeping the call-stack of IO-doers as small as possible. <br />
<br />
When I go back to my other languages, I now put all my IO in top-level functions that are called directly from main or the event loop. I gather every scrap of data I need to do the computation. I marshal the data into typed structures and pass it all into a pure function that does the work. Then the structure that's returned is demarshalled and transmitted, displayed or stored as required. If I need to do some computation to determine what data I need to fetch, I make sure this is not commingled with the IO functions, so my code fetches data, calculates what else needs to be fetched, fetches that data, and so on.<br />
<br />
This has some nice effects. The IO doers are isolated and distinct. Error handling and exception catching is clearer and simpler. The compute code is pure. This makes it much easier to test, debug, and understand. <br />
<br />
<br />
<h3>
Clean Syntax for Static Types</h3>
<br />
<span style="font-family: Courier New, Courier, monospace;">boo :: Map Integer String -> String -> Integer</span><br />
<br />
I have no idea what the word "boo" is supposed to mean when used as a function name. But I can be almost certain what this "boo" does. It takes a map that has Integer keys and String values, and a second argument that is just a String, and it returns an Integer. So I'm fairly sure that this function does a reverse mapping - you give it a String value and a Map, and it finds the Integer key for that value.<br />
<br />
A lot of this depends on the fact that the signature tells me that there is no IO going on. If the bit at the end of the line was "-> IO Integer" instead of "-> Integer", all bets are off. The function could be sending the Map and String to launch control, and -> IO Integer could be the number of seconds it took to get a response, or the price of a gallon of gas in pennies (hence "boo", perhaps). The point is, you can't confidently reason about a function from its signature if IO is involved.<br />
<br />
The Haskell type signature of a function is particularly clear and easy to follow. Functions just map one type to another "foo :: Author -> DateOfBirth". Parameterized types just list the parameter types "Map Integer String". There are very few boilerplate tokens for the eye to scan. <br />
<br />
But how has this changed what I do in other languages? I now sketch out the design for larger components in this Haskell signature notation. Particularly if I'm writing a library with a public API. I've shown these sketches to other developers as we discuss the design of a program, and they get it. Most of the time, I don't mention that the notation is Haskell. The only slight oddity for them is the use of -> between function "inputs". They expect foo :: A,B -> C instead of foo :: A -> B -> C. But they get over it immediately, and I have never had to mention currying or partial application, since they're usually just pleased that the notation is clearer than anything else we've ever used.<br />
<br />
<h3>
Container Operations</h3>
<br />
I think one of the reasons I started using Lisp, then Erlang and then Haskell was that I must have typed "for (size_t i=0; i<..." just about a million times and I was sick of it. C++ teases with approximations to map, filter, fold, scan, just enough so that you'll try them for a few months until you eventually give up or your colleagues smack you. When I want to filter items from a container, I don't want to start by saying "for(size_t i=0...". I want to say "filter f xs" and I want my colleagues to read that too.<br />
<br />
It might seem like an overreaction. But even in big classfull C++ projects, where I was senior developer, I spent my days writing functions. Functions consisting of loops and branches, because C++ didn't do a great job of accommodating "operations on containers". Despite all the guff written about STL separating iterators from algorithms from containers (from allocators...ahem), nobody provided a simple set of primitive container operations that regular programmers would use. <br />
<br />
Using Haskell for a while, the effect goes further. It forced me to think of every such problem as a chain of the primitive list operations, maps, folds, filters and scans. Now I always think in these terms. I "see" the transformation of a container as a simple sequence of these operations. Before, I would have thought in terms of munging multiple actions into the body of a single loop.<br />
<br />
I see things using higher level concepts, and I write my comments and code with these in mind. I usually still reach for the trusty for loop in C++, but I'll factor together common container operations where appropriate into higher level functions. Filtering is a really common example that seems to come up all the time.<br />
<br />
<h3>
What's Changed</h3>
<br />
Using Haskell definitely gives you a lot of warm fuzzy feelings, (until your filehandle is closed before you've actually read the data because you didn't ask for a result, silly). Part of the joy of the language is that it forces you to take a new approach to problems you've solved conventionally before. When the answer clicks and you see that the new approach is more elegant and powerful and general than what you've been using all these years, it's hard not to smile with sheer pleasure.<br />
<br />
In real world commercial software projects, if you don't properly test your code, and do code reviews, it doesn't matter what language you use, you're leaving the big wins on the table. A team that does these things well consistently will beat any team that does not, regardless of what language or technology stack they're using.<br />
<br />
Using Haskell changed my practices so that the code I write is easier to test and easier to code review. There's a bunch of other stuff too, some of it can be articulated, some will probably always be just a "warm fuzzy".<br />
<div>
<br /></div>
dubhrosahttp://www.blogger.com/profile/08495570833321981688noreply@blogger.com9