I know, tests are important. I know, the earlier I start writing automated tests the better. I know, a good regression test basis is invaluable. I know, tests are my security net, when I plan major refactorings. I know, tests enable progress. I know, tests give me another healthy perspective of the functionality I have to develop. I know, tests can help in proving that my application is actually doing what it is supposed to do. I know …
But I also know, how painful it can be to keep tests green, when changing the application code. I know, how difficult it is to write tests in some cases. I know, thoroughly tested applications tend to have much more test code than application code. I know, test code doesn’t provide any direct customer value. I know …
I know it is a trade off. I admit, I hesitate to write tests. Especially when trying new things out. While learning those new things, the application’s code will change dramatically. And I always do new things. Things I already know well, tend to bore me. I’m always striving to learn something new. Not everything new, but some major aspect.
My strategy is to skip tests until a hard nut appears. Then I look into the testing possibilities briefly (in case I engage a new base technology). If there is nothing simple and obvious, I try to crack the nut without the safety net. But then, usually not much later there appears a second hard nut. When this happens, I put some more energy in providing tests for both of the nuts I found so far. The goal: assuring that shell regenerating attempts are becoming visible, when trodding on.
Well, that was a lengthy intro.
The project proposal management app, I mentioned in earlier posts, just gave me the second hard nut to crack. Time for tests. The first nut was the handling of tree sets – basically moving arbitrary things around like in a file browser. I implemented that before, and it shouldn’t be that difficult. I had some problems, though, might be, that the chosen representation was a little unlucky, or that there were to many capabilities and constraints I pursued. Might be the combination. It works now, but it was a longer journey and it feels really vulnerable to changes.
Since unit testing in Meteor apps is far from being simple and obvious, I waited for the second hard nut. This one hit me while working on adding fulltext-search. There is no ready-to-use library I know of, so I decided to roll my own. This really exited me. But soon I realized, that I should be able to prove, that the different aspects of the search engine are working as expected. Manual testing this functionality through the user interface is no option, since there are many things happening under the hood. I wanted to test those in isolation. Well that calls for unit testing the different functions that are doing the work.
I mentioned it earlier: unit tests are no core part of Meteor, for now. There are plans to provide some official way of testing the app with version 1.0 of Meteor. Till then, you have to hack a bit. Well, what actually is the problem? There are numerous testing libraries available for Node.js, why can’t you simply use them? I asked myself, did some research and found out, that the basic tools are there. I settled on Mocha as the primary driver for tests. It seems to be everybody’s darling right know. I further decided to use Sinon.js, which seems to be the divine being, between the ordinary mocking/stubbing/spying frameworks out there.
So when I want to test the function: tokenize in the file: fulltext.coffee, how do I do that? Well Meteor does help a little. You can create a tests folder inside your apps root folder. This folder will be ignored by Meteor, so it’s a good place for your test code, without fearing that it might be mixed with your app’s code. There is even another place. Since search is a quite common feature I decided to make a Meteor package. The package folders are safe for test code as well.
In the “packages/fulltext” folder I decided to take the default route for mocha and created a test folder (notice the missing ‘s’) for my test code. On the root level lies the file that contains the function to be tested. So far so good. But how do I actually access the to-be-tested function from my test? I tried the obvious:
testling = require '../fulltext.coffee'
Ran the test from the package root with:
mocha --compilers coffee:coffee-script
And mocha answered with this error:
/home/dirk/Development/ProPro/ProPro/packages/fulltext/fulltext.coffee:96 FulltextIndex = new Meteor.Collection('fulltextindex'); ^ ReferenceError: Meteor is not defined at Object.<anonymous> (/home/dirk/Development/ProPro/ProPro/packages/fulltext/fulltext.coffee:96:25) at Object.<anonymous> (/home/dirk/Development/ProPro/ProPro/packages/fulltext/fulltext.coffee:107:4) at Module._compile (module.js:449:26) at Object.loadFile (/usr/local/lib/node_modules/coffee-script/lib/coffee-script/coffee-script.js:179:19) at Module.load (module.js:356:32) ...
This didn’t bother me much at first, since I didn’t start Meteor, the missing global variable wasn’t really a surprise. The fun began as I tried to make a replacement (mock, stub, whatever) for Meteor and pass that to the to-be-tested file. Running Meteor and somehow executing the tests from the inside, wasn’t an obvious path either. So I tried another naive solution: defining a Meteor variable before requiring the to-be-tested file. But unfortunately require doesn’t pass the context down.
Well, the last resort, make some bridge code yourself. I called it testling and you can find it here and it’s also available from npm:
npm install unit-testling
You use it similar to require: pass the relative path to the to-be-tested file as the first parameter. You can optionally pass a second parameter: it expects an object and injects its attributes into the context of the file. The following gist shows an example.
I know, this post is already quite long. But there is one aspect I learned, as I developed testling, that is worth mentioning and will finish this post: resolving relative paths correctly. The problem: I wanted to be able to enter the path to the to-be-tested file, relative to the test file. There is no direct way to achieve this, the context only knows about the path to the executable and the seed script, which is not the test, but mocha. To resolve paths correctly one has to look at the call stack. I took this advice and parts of the code from an answers given here.
While working on the impossible-hangman game I stumbled over the problem of accessing the hosts filesystem from Meteor. I intended in earlier versions of the game, to load the dictionary from a file. Meteor sits on top of Node.js, so I wondered if simply using the fs-module, like I would do in plain Node, might solve the problem. Unfortunately, you cannot use require in your app code. There has to be a way to include the module and there was a solution that worked until version 0.6.0 of Meteor:
fs = __meteor_bootstrap__.require 'fs'
But the more recent versions of Meteor don’t seem to support this way any longer. Instead they embrace Node’s modules system: npm, but only in packages. So you can not do:
in your app code. You have to create an app-private package. This is actually very simple. Create a subfolder called packages and in it, create another folder fs. In the latter folder you have to put a package.js file and a file providing the sources for the packages. The following gist shows the package definition. There is a little overhead, though, because I’m using CoffeeScript for the package’s source code. Absolutely unnecessary, actually:
The next gist shows the code of fs.coffee.
Now I can access fs and path in my apps standard code files. Keep in mind though, that this won’t work if you plan to deploy your app to your_app.meteor.com. I guess some security restrictions are in place there. Here is an example of using both modules an app code:
I can hardly resist to vomit into a corner, everytime someone comes up with a new and fancy 3 to 5 letter acronym for some stupid simple concept. BYOD, which stands for “Bring Your Own Device”, is another glory gory example.
If you haven’t heard of it, feel blessed, and close your ears for anyone mentioning it to you. It might be, that this is only some local movement, though. You know, we Germans tend to come up with such and similar stupid things by ourselves, from time to time. Most prominent and irritating example is the word handy meaning mobile phone.
Currently and for more than a year now (if I remember right) our (IT) press is polluted with articles about BYOD. Discussing the pros and cons, what you can/should/shouldn’t do about it, what rules to empower, … and on and on and on.
I bring my own devices, for more than four years now. Well before the acronym, and it’s hype came to existence. I do this, because my employer isn’t willing/able to provide me with the devices I enjoy working with. I tried a few times to get what I wanted, but failed, even with such a simple thing as a keyboard. I’m quite picky regarding certain things, I admit.
We have a corporate shopping cart, where you have to pick your equipment from. And a defined set of devices supported by our inhouse support crew. All that stifles your choices and there is nothing fancy left. Orderly things a bigger company is supposed to have? I guess not, much money is burned in managing that stuff. A little more trust in the capabilities of the employees – or enabling them, or get rid of those employees who don’t get it – would cost much less and would result in superior safety.
Nevertheless, I started to pay for my gear, with my own money. Since I’m supposed to know what I’m doing, it’s tolerated and I never had any issues. But I wouldn’t BMOD, if my employer would provide me with the things I need.
BYOD boils down to a lame excuse. Provide your employees with the equipment they think they need and the whole concept evaporates.
I tried, again, to participate in Olimex’s weekly programming challenge, I failed, again. I missed the deadline. My weekends are simply to packed. On the other hand, I intentionally allowed me the time to tune the solution into something presentable. For the impatient, look here. The sources are over at GitHub.
At first I thought about taking the very basic route. No libraries or frameworks, just plain and simple HTML, CSS and CoffeeScript. But the first task: “putting the necessary boilerplate into the HTML file” made me hesitate. I quickly backed up to the, by now familiar path Meteor provides.
I always wanted to experiment a little bit with HTML5 canvas, and this challenge gave me the opportunity. While researching, I stumbled over fabric.js. Unfortunately, the integration wasn’t as straight forward as I had hoped and as I looked into the documentation, I couldn’t find a really strong reason to use the library. The task of drawing a hangman looked really simple. Plain canvas was supposed to be easy enough and straight forward. For the curious, the ‘hangHim()‘ function does the drawing.
I don’t mind to bother you with the whole journey. Some aspects I found out, will be covered in separate posts, because I didn’t need them in the final solution. Some overly complicated things are just brainchildren of my own stupidity. I should talk about the main glitch I programmed. Well, the challenge was to make an impossible hangman game, by switching the searched-for word, in case the player would actually find a letter of it. That’s mean, heh?
My crank mind settled early on the idea of making inverse indexes: for each letter of the alphabet create an array of words, that doesn’t include the letter. This way, I convinced myself, I could later intersect those word arrays easily and at low cost. I thought to be really clever. To make it worse, I decided to use Meteor Collections for the static set(s) of words. With the full dictionary – the one linked from the challenge (around 130.000 terms) – that resulted in around 16MB of data to be transferred to the client. A nightmare for any webapp.
I guess, despite the missing comments, the code speaks for itself. If you have any questions feel free to ask.
I decided against Arduinos for my second self-made keyboard. Besides being a budget driven decision, the chosen Pinguinos outperform the Arduino Unos in every regard. The fact that I prefer to experiment with new things, instead of taking well-known paths, may have been the primary factor for the decision to switch.
Some weeks ago – shit, time really flies by – I completed the wiring of the right hand keyboard-matrix and soldered it to the Pinguino. I could have taken the first steps in programming the controller since then. Unfortunately I’m so easily distracted, that some things take much longer than they should.
I installed everything needed and started to write some code. But along the way I realized what a pain it is to set the development environment up. I made the Pinguino IDE work somehow. Not on my Mac. But on my old Ubuntu computer. In the end I couldn’t recall everything I did to make it work.
As a reference for myself as well as any other new Pinguino developer I decided to install everything from scratch in a fresh virtual machine and write those steps down. My host’s OS is Ubuntu 12.04 and I’m using VirtualBox. You don’t have to follow this path and might want to install everything into your main operating system. But you should consider using a VM. I learned over the years that it’s really convenient to being able to cleanly throw experiments away, without screwing your main working environment.
Installing VirtualBox and a Virtual Machine
The basic installation of VirtualBox is actually quite easy. And the result is sufficient for most situations. I wrote the following command:
sudo apt-get install virtualbox-4.2
into the shell, executed it by pressing <ENTER> and apt installed everything needed.
Afterwards, I started the Virtual Box Manager and created a virtual machine, by defining some basic settings: name, type and version, which I set to ‘pinguino-dev’, ‘Linux’ and ‘Ubuntu 64bit’. Then Memory, HDD and Processor, where I was basically following the recommended settings. I decided for the 64bit version of Ubuntu and attached the freshly downloaded Ubuntu 12.10 64 bit ISO image to the VMs virtual CDROM drive. Now the first problem popped up, the computer crashed everytime I meant to run the VM.
Google taught me that VirtualBox tends to crash, if you try to run a 64 bit virtual machine on a 32 bit host operating system. After I had fixed that (by using the 32 bit version of Ubuntu) and enable PAE in the VMs system processor settings, the VM finally started up.
The installation of the basic operating system – basically following the recommended settings -took some time, but went seamless. Afterwards, I had a fresh and clean system running in the VM and I could start installing everything necessary to run the Pinguino IDE.
Switching to LXDE
Well, almost … my old computer only has 2GB of RAM. Because of this, the guest system was reacting really slowly and bogged down the host as well. I decided to switch to LXDE as the windows manager for the VM. The following command installs it, along with a login manager.
sudo apt-get install lxde lxdm
While running the above command, I was prompted to select the default display manager. I switched to LXDM. After rebooting, the new login manager appeared, and I selected LXDE as the window manager while logging in. Now I had a system I could work with, without getting annoyed after a few seconds.
If your computer is low on resources like mine, you should consider downloading an adapted Ubuntu version (e.g. Lubuntu) in the first place.
Getting the Pinguino IDE
I decided to use the latest version of the Pinguino IDE and downloaded therefor the sources from the subversion repository hosted at google code:
sudo apt-get install subversion svn checkout http://pinguino32.googlecode.com/svn/ide/ pinguino32-read-only
Then the required libraries:
sudo apt-get install python-usb python-wxgtk2.8 python-svn
After this had completed I could start the IDE, with:
cd pinguino32-read-only/x.4 ./start_pinguion.sh
Routing USB devices to the VM
The next step was connecting the Pinguino board with the computer. I fetched a Mini-USB cable and did so. I had to enter the boot loader mode. This way the board was recognized by the USB subsystem. This can be validated with:
The output should contain a device, named “Microchip” something. To actually enter the boot loader mode I had to hold down the button labeled <BUT> and press and release the button labeled <RST> once – this might be different on different boards. The two attached LEDs should flash alternately as a result.
That was the easier part. To make the board programmable in the VM I had to take a few extra steps. By now the USB subsystem should be totally invisible to VirtualBox. You can check that with:
VBoxManage list usbhost
At the current state, this did output nothing, literally. I needed to install a few things. I downloaded the “VirtualBox Extension Pack” (from here) on the host. This enables the VM to access the host systems USB 2.0 devices. I installed it by executing the downloaded file.
It was also necessary to install the so called “Guest Additions” in the VM. In the VMs menu I clicked “Devices -> Install Guest Additions”. This downloads the appropriate version and mounts it to the VMs CDROM drive. There, I executed ‘VBoxLinuxAdditions.run’ and did get some error message. Installing the following packages solved the problem and the Guest Additions installed successfully.
sudo apt-get install build-essential linux-headers-`uname -r` dkms
But still the command “VBoxManage list usbhost” showed nothing. I had to change my users group membership and put the UDEV rules in place, so that VirtualBox has the proper rights to access the USB devices.
I copied the matching UDEV rules from the subversion repository of the Pinguino IDE to the right system folder. I did that on the host as well as on the VM. You might share a folder between both systems to access the files. Or you check the sources out twice:
cp ~/pinguino32-read-only/x.4/extra/rules/26-microchip.rules /etc/udev/rules.d/ cp ~/pinguino32-read-only/x.4/extra/rules/41-microchip.rules /etc/udev/rules.d/
I added the group “microchip” to both systems:
And then I added my user to the created group and to the vboxusers group. The latter is only necessary on the host OS:
usermod -a -G microchip dirk usermod -a -G vboxusers dirk
Afterwards, I shut down the VM and rebooted the host system. After trying to restart the VM I got the following error:
After executing the proposed command in the terminal, which took a while to finish, I tried again and the VM started successfully up. I tried VBoxManage at the host, and the USB devices were found, even my Pinguino in bootloader mode. You can control the devices attached to the VM in the bottom-right corner of the VMs window. I attached the Pinguino and after starting the Pinguino IDE, I could theoretically upload my programs to the board. Unfortunately, the IDE behaved somewhat buggy. Starting the IDE with:
yielded better results. And I successfully uploaded the sample blinky code to the board. That’s it. Now I will go on and code the keyboard polling and post about it later.
I’m just asking myself, what is the amount of comments that anyone bothers to actually read? A moment ago I read this post from Rara and I was about to comment, because I felt so … akin (a new word for me, hope this makes any sense).
I scrolled down willing to express my thoughts on the matter and realized, that there are already countless comments. Far before I reached the bottom of the page, where I actually could post my comment, I decided against commenting. The reason: I think it’s polite to read the comments of others. In case someone made a similar point, where I can build upon or if my comment would be simply redundant. Both cases do actually happen quite often.
Commenting without reading what others say – that crosses my mind whenever I find a post with a lot of comments, more then ten root comments I consider a lot – seems unfair and disrespectful regarding those that already left their thoughts.
If the matter is important and I have the time, I read even hundreds of comments, but swiftly. In case it’s just a funny coincidence, or something I riddled myself with, like in the case of Rara’s post, well I don’t comment. That feels rude and disrespectful too.
Because, I think Rara would have appreciated my comment – well at least I try to tell that myself. Might be, because I don’t get many comments … let me think … if I would get hundreds of comments a day … mmhh … that wouldn’t make any difference.
I think every thought-through comment, indicating the commenter has read the post, is more than welcome to any blogger. If I’m wrong here and you feel molested, please tell me and I will immediatoly stop.
What made me feel akin to Rara and started this post? She talks about her urge to fix things she comes across. More precisely the things she’s an extended knowledge of. Well I often have the same urge. For her it’s advertisement, homepages, design and signages. For me it’s software architecture, user interface design and more general small and large scale complex systems. I stumble across so many overly complicated things, that could have made way simpler if anyone just had put any thought in simplicity when inventing the thing in the first place or changing it through the decades of it’s usage.
Do you know the book Designed for Use. Striving for simplicity in UI design could be its caption. Highly recommended.
I learned yet again, that I’m living in an ivory tower. Actually, there is only a very limited number of places I usually go. My work place, quite intelligent and grown-up people in the majority. My home, my almost sane family. Our neighbourhood, one of the better in Krefeld. An idyllic world. But from time to time I have to leave that capsule of righteousness and hilarious problems. Last Saturday such a time had come.
Amazon notified me just a few days earlier that they shipped One Piece #66. This one was finally supposed to finish the Fishmen Island Arc. I made a habit out of collecting the books until they conclude an arc, and than read them altogether. Otherwise, those cliffhangers drove me crazy and I couldn’t wait two to three month until the next one came out. I was forced to spoil for what happens next. That has always taken a little bit the fun out of it.
I was happy. Now I could finally consume the other four books laying on my bedside table.
The Message of Doom
Arriving at home on Friday afternoon, I found one of the most feared things in our idyllic-world capsule: a message from the mailman. Actually Amazon ships their books with DHL and they are not allowed or unable or whatever to use a mail box. We have an opening in our front door for letters and stuff. This tiny parcel could easily fit, and no one except the people living in the house (only us) could pick the parcel up. So where is the point in denying to use it? Despite this, their are usually neighbours accepting our parcels. But sometimes you have simply bad luck.
I mentally prepared to visit the central post office. That literally took me down for the rest of the day. The last time I was there, not so long ago, I spent two hours in the line for a similar small parcel. It always happens on Fridays, and you cannot pick them up before twelve o’clock the next day. The opening hours on Saturdays leave only one hour, to do so. And since DHL created those dreaded Pack Stations, there is only the main post office left. I was cursing them. They never really made the leap from a governmental service to a customer friendly company.
The Odyssey Begins
I took my Kindle with me and hoped to read another good chunk of The Emperor’s Edge, which I can highly recommend to any fantasy book fan. When I entered the hall the line actually wasn’t that long, only once through the 60 meters wide hall. I estimated around an hour of waiting. Luckily they don’t kick you out at one o’clock, they merely don’t let new customers in and handle the line till the end.
Reading was hard from the beginning, since two girls were behind me that chatted constantly as if they haven’t seen each other for years.
After ten minutes – the line moved around 5 meters – I noticed a young woman (mid 20′s, I would guess) who paced back and forth between a counter and the form fill out area (sounds like chill out area, but is quite the opposite) ignoring any line and almost jumped over the barrier that separated the counter area from the waiting line area. She wore a tracksuit in light blue. You know one of those, hip-hop style thingies. I’m not aware of the correct technical term.
The next thirty minutes I was able to read. The chatter behind me was so constant I could easily blend it out and was sucked into the story.
About Nazis and Bulgarians
But then it was getting real loud. The post officer was shouting at the tracksuit woman. At first I couldn’t understand what they were saying. But as the loudness grew I caught a few words: “… said, not going to pay … can not read … need ID …”. I dismissed it and tried to get back to the story. I couldn’t. They were getting louder and louder, and as I was getting closer and closer I was able to understand clearer what they were talking about.
The post officer, was denying to pay out 2.000€. She was a Bulgarian citizen and had her Bulgarian ID with her. The post officer claimed not being able to read her ID and without an ID he wouldn’t give her the money. Apparently she had something, that normally, provided she had a readable ID, would legitimate her to receive the money. That part was unclear to me, because I didn’t witness from the beginning. She argued that she desperately needed the money now. The post officer said that he understands, but can do nothing without a valid ID. As it turns out, he was already the manager in charge. Nevertheless he involved a second officer to witness the not readability of the ID. But eh, what should he do, question his boss? That wouldn’t happen. After some more minutes back and forth and her pointing fiercely at the part of her ID where she claimed all information was available in latin letters, she accused him of being a Nazi. She didn’t use the word, but she screamed under tears: “YOU ONLY TREAT ME THIS WAY, BECAUSE I’M BULGARIAN … THAT NEVER HAPPENED TO ME, BEFORE”. He felt offended and dismissed her. By then, it was my turn, and I was called to the counter right besides both of them. They exchanged a few more not so nice, but still somewhat professional, words. Then she walked off, under heavy tears. The whole hall had witnessed the spectacle.
What Did Just Happen?
I tried to tell the facts as emotionless as possible, so far. Now my interpretation: the post office still employs the laziest, stupidest and most ignorant people you can find on earth. There are exceptions of course. But not him. He denies to decipher an ID that’s different from what he normally sees. She had a somewhat questionable outfit at least in the eyes of an old stubborn post officer having only a few years before retirement. Her German was broken, but she seemed to understand everything. He feared being duped and made a decision early on, which he couldn’t withdraw. Pride … you know. Since the whole hall has witnessed.
As I was waiting for my parcel to be found in some dungeon, there was a call from the door. “She is back, and brought her boy friend. What shall I do?” The managing post officer was serving a different customer by now, continously muttering about being accused of being a Nazi and that he is unable to read cyrillic and that’s always the same with those Greeks. He shouted back: “Let them in”. Shortly after, I received my parcel and found myself in front of closed doors. Another post officer had kept the doors locked. The woman and her boyfriend sill outside. He looked almost as expected: similar age as her, a tracksuit, barrel chested, thick necked, short hair, grim face. If he would have grown bigger than 1.7 meters, I would have been a little scared by now.
Trying to Escape
The post officer guarding the door urged me to wait. She called out again for the manager. He said again to let them in, but to wait a second to finish his current customer. While I was waiting to get released, a woman nearby (almost at the end of the line), said to herself: “Ah. That’s so sad.” The door guard instantly snapped at her: “What is sad?”. The woman, apparently feeling attacked, answered after a hesitating moment in a low somewhat trembling voice: “Everything. Everything in this country.”
As it took more much more than a second to finish his customer, and I couldn’t bear to stare at the couple outside of the glass door any longer, I opened the parcel and found out, that the Fishmen Island Arc is indeed going to end in this book. The door guard finally let me out and the tracksuit couple in. I was lucky to miss the coming moments at the dreaded post office.
Afterwards I was blaming myself. Maybe I could have helped her in the first place. But thinking again I dismissed that thought. Other encounters with this stubborn post officers have proved me, that you can’t teach them anything. And well, I don’t look that trustworthy either. He would have accused me to conspirate with her, be it just silently.
I confess I’m also a coward and have learned quite early to keep myself out of trouble. But that was not the biggest hindrens, it was more the fact, that I felt like a spectator, not really there, disconnected, unable to interact. Like glimpsing through binoculars from my ivory tower down onto the real world. Have you ever experienced a similar feeling of disconnectedness? Should I try to you keep myself more “grounded”? What’s the point, why do I blog about this encounter?