A few more thoughts on TV series

I recently got my hands on the first few seasons of “Orphan Black”—and was initially very impressed: A novel concept, wonderful performances* by Tatiana Maslany, and characters put into interesting situations (see excursion below). Series like these prove that it is not necessary to just dust of the same old idea or franchise to squeeze out a few extra dollars. (Cf. previous posts, e.g. [1].)

*She plays a handful of central characters, and another handful of less central, that are clones, managing to bring over so different personalities and traits that, looks aside, they might as well be played by different actresses. She even, on several occasions, plays one clone pretending to be the other in a manner in a realistic manner, actually hitting A-pretending-to-be-B. (Similar scenarios often end up with an actor/actress playing this almost exactly as either A or B, or sometimes trying to do a realistic A-pretending-to-be-B and failing badly.)

However, approaching the middle of the second season, I am less enthusiastic, the series having lost some of its initial strengths and entered several hackneyed conspiracy and intrigue lines. The Dyad institute might be unavoidable, seeing both that its works are central to the premise of the show, including explaining why there are clones, and that some type of antagonist is needed. The Proletheans, on the other hand, are just unnecessary. Similarly, what is the point of turning “Mrs. S” from a more-or-less regular foster parent into an extremely shady, possibly criminal, possibly terrorist, character, with involvement in the clones’ early history? Why not try the novel idea of not having every second character have a “surprising” dark past?

This is paralleled by my very recent watching of the fifth season of “Grimm”: While never a candidate for an all-time great, it remained quite enjoyable while it focused on the “monster of the week” format and the exploration of the series mythology. However, it had had long excursions into global intrigues and whatnot, and with the fifth season this area exploded—as did the cliches. We now have a state of semi-war between various parties, the hero’s former girl-friend going “Dark Phoenix” and working for a secret organization, several secret organizations, an extremely powerful magic child causing trouble,… The destruction of the “Wesen Council” is not only a hackneyed destruction-of-the-potential-saviors/-allies-in-advance, it also very closely parallels the specific destruction of the “Watchers’ Council” on “Buffy”. The events of the penultimate episode took me to the point that I did not even bother watching the last episode—and will not bother with the concluding sixth season. A particular weakness, committed by many other series, is the explosion of the number of originally rare beings (here “wesen”*), to the point that it would be virtually impossible for “civilians” not to be aware of them, had they existed in reality.

*One point that annoyed me from the start: This German import, roughly “being”, is invariably pronounced like the word “wessen” (“whose”). Foreign pronunciations can be hard, but when one specific term is used several-to-many times per episode, the minimal effort of just once asking someone proficient in German for feedback is not too much to ask. (Virtually all German words used in the series are mangled or semi-invented, but most are used in just one or several episodes, and most are used by speakers who, in a real-life scenario, could not be expected to know better. “Wesen” is mispronounced even by the wesen themselves and even by purportedly German characters.)

Not every series has to deal with dark conspiracies, threats to the world as a whole, insurrections against the current order (be it by the antagonists or the protagonists), … Indeed, most series would be a whole lot better if they were left out!

Similarly, not every new season needs to up the stakes, invent greater threats, whatnot.

Similarly, there is no need for a series to continually reinvent it self: Most reinventions work worse than the original and even those that do work well risk alienating the original fan base. Usually*, it is better to stay with a single great concept. True, this can eventually lead to viewers growing tired of and abandoning a series, but nothing lasts forever. Good new ideas that do not fit the original format can be explored in a new series, while the original series runs its natural course at full quality.

*Doing a quick brain-storming, I actually could not name a single exception of a major and lasting change in concept/premise/setting/… that had a positive net-effect. (But I am certain that they do exist. The list of smaller changes causing an improvement, e.g. a strong new supporting character, is considerably longer.) The closest I came up with was “Chuck” and the addition of intersect-provided physical abilities. These made for both many interesting plot developments and a lot of entertaining action scenes; however, I still consider the earlier series more interesting and enjoyable. A case could possibly be made for some of the developments on the various “Stargate” series.

Excursion on “interesting situations”:
As I realized watching “Orphan Black”, one of the things that I appreciate the most in fiction is protagonists being put in (in some sense) interesting and unusual situations (mostly based on their own frame of reference). “Orphan Black” e.g. has the main protagonist among the clones see another clone die—and take over her identity with no previous information. Ensuing experiences include having to get through a hearing about a lethal shooting committed by her police-woman alter ego and trying to keep parts of her “old” life going in parallel to the “new” life. This applies in particular when learning and personal development are involved in variations of the “Bildungsroman” theme. A somewhat recent example is the movie (I have not read the book) “Divergent”: I was fascinated by the heroine’s move from the highly specialized faction she was born in into another and her efforts to cope in the new environment, including having to hold her own against people who had lived in that environment since childhood. Unfortunately, this part of the movie was not explored in the depth I would have preferred—having to leave room for conspiracy and insurgency… A similar trend is seen in “Counterpart” (mentioned as “very promising” in my recent post on “Back to the Future”): Two alter egos (or counterparts…) from different realities meet each other and eventually switch places. Early on these situations are at the core of the series; by now, still in the first season, conspiracies and whatnot dominate.

In all fairness, it could be argued that the use of an “interesting situation” also borders on the hackneyed in the genres I tend to watch/read the most. Consider e.g. how the likes of Bilbo/Frodo or Luke Skywalker are torn out of an idyllic existence for great adventures, how any amount of earth humans are transplanted to unknown worlds (most notably in the “Narnia” series), how the ignorant-of-magic Harry Potter finds himself in Hogwarts with minimal preparations, or, looking at some of my posts on fiction, e.g. the early events of “iZombie”, “Grimm”, and “Orphan Black”. However, this is a point where I am willing to give a lot of leeway—not only because I enjoy the situations, but also because they have a narrative advantage of being able to explain a new world to the viewer/reader without jumping through hoops: Explain the world to the protagonist and the audience receives the same information.* To boot, many of these situations are radically different from another, while e.g. fictional conspiracies have a great degree of fungibility.

*There are examples of doing it otherwise that work well. An extreme case is the “Malazan Book of the Fallen”.

Excursion on (dis)similarity of alter egos: A common problem in fiction is that alter egos are far further apart from each other than they realistically should be. This has narrative advantages; however, it is also potential danger in that it misleads the broad masses on topics like personality development, perpetuating the outdated “tabula rasa” models and their highly negative political influence. “Counterpart” does a reasonable job in that the differences between the main protagonist and his alter ego are small enough to be explained by different events and developments in their lives. “Orphan Black”, on the other hand, shows so extreme differences that the clones basically have no more in common than a group of randomly selected individuals—something considerably less realistic than human cloning.

Advertisement

More on password security / Follow-Up: My recent problems with Unitymedia

An expansion on the password security issues briefly mentioned in my previous post.

Returning to and setting the hotspot password, I was faced with the following rules (paraphrased into English):

1. At least 8 characters, at most 64.

   A lower limit of 8 characters is very weak in today’s world, and negates much of what is attempted to be gained by the other rules.

   (Any upper limit is sub-optimal, but it is hard to avoid having a limit somewhere, 64 should be enough for these purposes, and compared to some idiots who actually put upper limits of e.g. 16 characters, it is quite good. Some banks go to extreme lengths to increase security with various TAN-mechanisms, yet leave the online-banking password/PIN at exactly 5 characters…)

2. At least one upper-case letter.

   This is obviously geared at nitwit users who chose too easy passwords, up to and including “password”. However, it also reduces the search space, making life easier for crackers of random passwords—and it poses a problem during password generation: Especially with shorter* passwords (say 12 characters) and in combination with the following two items, there is a non-trivial risk that a randomly** generated password will not be conformant. To boot, such restrictions only look at one aspect of a password and a password of 11 characters made solely from lower-case letters will be more secure than 8 characters mixing upper/lower case and digits.***

   *And note the below item where a longer password will be more likely to be non-conformant: Unitymedia has us coming and going.

   **A randomly generated password is almost always the best choice from a security point of view. A randomly generated “ertya123456dmqpdfe” will be more secure than a manually chosen “consTituti0nal_amendMent”, despite the conspicuous digit sequence and other violations of these rules, and despite being shorter. To boot: If everyone used random passwords, these rules would be entirely redundant and dictionary attacks (cf. below) pointless.

   ***Assuming 26 letters, the former has 26^11 combinations, while the latter has (2 * 26 + 10)^8 combinations. The former wins by a factor of roughly 17…

3. At least one lower-case letter.

   As above.

4. At least one digit (literally, “number”/“zahl”).

   As above.

5. Special characters are allowed (e.g. !@#+-=).

   Good: They should be.

   However, it is extremely unlikely that Unitymedia is set up to handle all variations (cf. the next item) and a complete listing should be given. Also, someone who does use special characters increases the risk of violating one of the preceding rules with an automatically generated password.

6. No spaces.

   An arbitrary restriction that should not be needed with correct password handling by Unitymedia, and which reduces the search space. This might be an attempt to protect users against confusion arising from (accidental) leading or trailing spaces, but, if so, the rule should not apply within the password. More likely, there is some deficiency in Unitymedia’s systems that falls on its face when spaces are used.

7. No consecutive letters or digits [literally, “numbers”/“Zahlen”] (e.g. 123, abc).

   Firstly, this is a very unclear rule, making it hard to determine what the actual restrictions are. What is almost certainly meant, based on knowledge of common password errors and the examples, is that there must be no string of one letter (or digit) followed by another which is “one higher”. That is not what is said, however: The most reasonable literal interpretation would exclude e.g. “145” and “azt”, because we have digits or letters following each other in the string. Other potential interpretations are possible, however. The examples used make it unclear if e.g. “12” would be OK.

   Secondly, this rule is highly problematic for those using password generators: With a long password, the chances that a perfectly random password does contain one or several such combinations is fairly high, even assuming a minimum of three characters. Assuming just two characters, automatic generation will fail very often.

   Thirdly, without additional protection against e.g. “321” or “135”, this rule is toothless.

   Fourthly, even non-random passwords are weakened, because the search space can be reduced.

8. Must be different from the customer-area password.

   Strictly speaking, it is a good thing to use different passwords for different objectives. However, without also banning trivial variations (e.g. just adding a “1” at the end), the benefit of this is small. It it also well-known that the more passwords users have, the more likely they are to write them down or cheat* in other ways, thereby turning the security advantage into a disadvantage. This risk is particularly large with unsavvy users, which is exactly the group these rules are so obviously targeted at. Of course, a much worse error would be to use the same password for two entirely separate services, e.g. Unitymedia and Hotmail; however, here there is no restriction**.

   *E.g. through using trivial variations, foregoing random passwords in favour of “dictionary” passwords, resorting to personal facts, …

   **For practical reasons, such a restriction would likely have to be limited to an admonition. This admonition, however, could well bring more benefit than the Unitymedia-internal technical restriction…

   (It also very, very slightly reduces the space of available passwords/the randomness of possible passwords. In this case, it is highly unlikely to have any practical effect, but similar rules would be detrimental in a cryptographic context.)

Two additional weaknesses are:

Firstly, no mention is made of what happens with letters not normally present in German, e.g. “å”, or Unicode variations of letters that happen to look the same but are considered different. This is not only a major source of insecurity for the foreign user (for instance, a Chinese user might prefer to have an all-Chinese password), but also makes it very hard to judge search spaces. For simplicity, I go with the English alphabet and 26 letters above.

Secondly, the single greatest danger is the use of passwords vulnerable to a dictionary attack, e.g. “consTituti0nal_amendMent”. These, however, are not banned. A dictionary of, say, 100,000 words is almost certain to contain “constitutional amendment”. It has 24 letters (including the space). Allow a geometric average of three* variations per letter. We could now take 3^24 * 10^5 as an estimate of the randomness of this password. This is a smaller number than 26^12, corresponding to a perfectly random string of 12 lower case letters. It is actually almost as weak as a mere 8 random characters from a 100 character space, as could be approximately achieved by mixing upper/lower case, digits, and special characters from a regular German keyboard.

*Most will only have two, the upper and lower case, for such naive transformations. Some will have three, e.g. “o”, “O”, “0”. Some could have more, e.g. the space being replaceable by any special character. Also note that the above randomness estimate is likely on the generous side, because most other words in the dictionary will be considerably shorter. (The above, however, is only intended to give a rough ballpark figure—not as a stringent mathematical analysis.)

As an aside, what weaknesses are the more severe can depend on the type of attack attempted and the surrounding circumstances. Is the attacker looking to crack one specific account or any account? Does he have access to e.g. a set of hashed passwords that he can attack off-line or does he need to attack through the log-in masks? Etc. Notably, if a non-random password is used and a specific account is attacked, then “social engineering” is likely to work better than a dictionary attack.

My recent problems with Unitymedia

Incompetent and user/customer hostile businesses has been a recurring theme in my writings. My experiences with Unitymedia rank among the very worst, however.

Problems until recently were mostly limited to abusing my email address for spam. However, the developments in the last two weeks are utterly inexcusable. Even in a somewhat abbreviated listing:

1. Visiting my apartment in Wuppertal for pre-sabbatical preparations*, where I have had an Internet connection from Unitymedia for almost a year-and-a-half, I find that the connection no longer worked. This after my only using it for a total of roughly two months, due to my long absences, and now that I finally was going to use it on a daily basis for the foreseeable future.

   *Cf. a previous post.

2. I attempt to trouble-shoot through the web interface of the router—only to find that the web interface simply does not work with my Firefox. This without any messages as to why, no “please activate X”, or anything indicating that something was amiss—apart from things not working. For instance, a button that was to be pressed was not visible; for instance, after finding the invisible button and pressing it, nothing happened.

   This state persisted after I had verified that all the likely complications, including cookies, JavaScript, and images, where activated and functioning.

   Having limited time, I (temporarily) gave up and focused on other things.

3. Back in Cologne, I tried to log in to the customer area of Unitymedia’s website. This was not possible, with repeated errors of
   

   Bad Request

   Your browser sent a request that this server could not understand.

4. I also investigated Unitymedia’s WiFi hotspots*, hoping to use them as a work-around. This was fruitless, with no information easily found (but compare below).

   *Every (WiFi-)router is per default enabled as a hotspot for other Unitymedia customers, implying that they can access the Internet without extra cost when away from their own routers. (Assuming that another router is sufficiently close by.)

5. I now contacted customer service per email, giving a detailed record of events and including my last invoice number (the customer number not being obvious from any of the information available in Cologne).

   The result was a pure boiler-plate email claiming that my customer account could not be found based on the data given—utterly absurd since I copy-and-pasted the invoice number. (And have subsequently verified that I sent the correct number.)

   Worse: This email committed many of the sins I discuss in a previous post, including altering the subject line and not including the original message—and added one entirely new: The sender was replaced by a “no-reply” address in an ongoing conversation. These are inexcusable in any context (cf. the linked-to post), but in an ongoing conversation?!?!? Effectively, I have to go back to a previous message and copy the recipient address from there in order to reply!!! An absolute and utter travesty of email use.

   Whether Unitymedia is just utterly incompetent or are deliberately trying to sabotage email communications*, I do not know. Either which way, this is so far beyond the acceptable that the decision maker should be summarily fired for this alone.

   *For some reason, many businesses appear to be extremely email adverse and/or view email as a pure one-way channel for them to send messages, mostly spam, to their customers. Common problems include hiding email addresses, taking any chance to ask the customer to call customer service instead, trying to divert customers to Facebook instead of email, … On a few occasions, I have even had emails to officially publicized addresses be given an automatic response of “please use our contact form instead”.

6. I sent back further (redundant!) information, and now Unitymedia apparently did manage to find me. However, instead of addressing the issues at hand, a message amounting to “we tried to call you; please call us back” was given—something which is entirely pointless, seeing that I am not in Wuppertal at the moment… Worse: Going by the time the email was sent, I and most others would have been at work in the first place—if they had reached me (or whomever) it would have done no-one any good, making the phone call a waste of time. I had described the events in sufficient detail and without being in the presence of the router, there is very little else that I could reasonably have added or tried.

   To boot, I currently have no use for a cell phone and have let my pre-paid SIM expire. Apparently, however, someone who does not have a cell phone is not allowed customer service…

   Also to note: The information that I had additionally requested should have been given per email, not per telephone. If I had wanted information per telephone (extremely error prone), I would have called myself; I sent an email and both common sense and common courtesy requires a reply by email.

   Almost needless to say, this reply also committed all the above email sins…

   As an aside, there are quite large bootstrap problems involved by now: Almost any attempt to open a contract requires leaving a phone number and/or email address—quite often “and”; often specifically a mobile phone number. This even when there is no actual justifiable need; this even when the contract is for e.g. telephone services. When I moved to Düsseldorf in 2011 (?), for instance, the provider I first turned to for telephone and Internet services (Deutsche Telekom) required a pre-existing phone number to even leave the first screen of the process. We could be approaching a state where e.g. an immigrant simply is stuck, not being able to get basic services because he does not already have basic services.

7. I replied correspondingly, including pointing to the fact that most of the checks Unitymedia should do could or even must be done without my involvement. (For instance, checking that everything is OK with my contract does not require my involvement; correcting errors in Unitymedia’s web pages must not involve me.) This email is still unanswered.
8. Today, I had grown tired of waiting and not wanting to risk further delays, seeing that I only have the Cologne apartment (and the Internet connection there) until the end of next week, I installed Chromium*, hoping that this would work with the atrocious web pages of Unitymedia. Well, to some approximation, it did. After various hitches, including a password field that refused my securely generated password** and an incorrectly constructed confirmation email***, I finally managed to register and login in.

   *An open source version of Chrome.

   **The best approach to secure passwords is complete randomness. Restrictions like “must contain a digit” can be helpful in slightly protecting idiots who try to use “password” as the actual password, forcing them to move to e.g. “pasSw@w0rd”. However, the emphases is on “slightly” and these restrictions lower the security of random passwords. (Since they are no longer completely random.) The procedure of Unitymedia is made a mockery by insisting on a “security question”, which very, very significantly lowers the security of the password mechanism: A glass window next to a steel door. (The considerably better, even if not perfect, way is to have the ability to send an email with a “reset” link to a pre-defined email address.) As for the security question, I originally tried to use (approximately) “security questions are a bad idea” as the answer. This was rejected as invalid, with no indication of why. (Length? The spaces? After replacing it with a shorter, random string without spaces it worked.) Complete and utter idiots!

   ***The (HTML) email was so poorly written that it did not even render in my email client, appearing to be entirely empty; I was forced to save the email to a text file and to open it manually in a browser. The actually needed contents where several lines of text and a link; the actually provided contents were an order larger due to the inclusion of various information about who was the CEO and whatnot; the actual size of the HTML code was 61406 (!) characters, compared to 1657 for the actual text. (The latter, imprecisely, measured through copying the text from my browser and copying it into the Linux “wc” tool; the former not including several external images, which are incidentally a big “no no” when using HTML emails.) Running the HTML through tidy, a HTML validator, gave no less that 159 (!) warnings.

9. After navigation through the visually horrifyingly designed pages, with their illogical structure, dodging repeated annoying and uninteresting messages that Unitymedia had wonderful new offers for me, and generally being on the very, very end of my patience, I finally found instructions for how to use the hotspots—with smart phones. A use with computers, even notebooks, is apparently not on the agenda (but I assume that the instructions are sufficiently adaptable that it will be possible).

   However, before use I had to activate the functionality, set a password, and whatnot. Before submitting the corresponding form, I clicked on the link for the Terms-and-Conditions—only to unexpectedly find myself looking at a PDF document within Chromium. I closed it to download and reopen it in a proper PDF viewer—only to find that the tab with my data was gone. (Apparently, the PDF had opened in the same tab.) I re-opened the tab and went back to the original page—only to find that the data I had entered were gone. At this point, I just gave up, wanting to save my blood pressure from a complete disaster.

   (This is of course only partially the fault of Unitymedia. Most of it likely falls on a weird default behavior from Chromium, which incidentally proved to be very frustrating and limiting in other regards too, e.g. in the use of annoying animations and filling the “new tab page” with a redundant Google search page, neither of which appeared to be possible to deactivate through the main settings.)

The web pages of Unitymedia could basically be used as an example for aspiring web designers of how not to do it. I will not attempt a detailed analysis (because that would require me to go back and look at them in corresponding detail, for which I have neither the time nor the patience). However, I do note especially on the visual side the need for excessive scrolling to reach any content, any screen typically containing just a few lines of text—and large, uninteresting images or large swatches of even less interesting empty space. Technically, they provide an excellent example of why Ajax/DHTML/whatnot are rarely a good idea and why it is almost always better to develop regular HTML pages, using vanilla forms, and possibly some very minor piece of JavaScript for some special tasks. Content-wise, the pages are confusing, making it hard for even a very experienced surfer to find the right information. By and large, I would liken the visit to trying to find useful product information in a supermarket flyer.

The pointlessness of missing the point

A continual source of puzzlement to me is how people can take a certain idea/concept/methodology/whatnot and complete miss the point, often to such a degree that the disadvantages are kept and the advantages removed—leaving us worse off than without it. Terry Pratchett once discussed this phenomenon with the example of how the British* had taken fast food and entirely removed the “fast” aspect, leaving them with slow fast food… There is also a considerable overlap with “cargo cults”.

*I have spent less than a week in Britain and do not vouch for this being true. However, in an interesting parallel, it is my impression that the German versions of McDonald’s and Burger King have grown a lot slower over the years.

An example that I regularly encounter and that prompted me to write this post:

Alpine, my usual email client*, handles locks on folders in a manner that is contrary to the concept of locking. Specifically, if one instance of the program has opened a folder with a write-lock (which is the default) then a second instance opening the same folder will simply take over the lock… What do the nit-wit authors believe is the purpose of a lock?!? To boot, this operation can take some time, delaying further work—and after the delay I have two instances both in the wrong state..

*Yes, Mutt is probably the better choice, but I never have gotten around to make the switch.

True, in this manner there is still some protection in that concurrent writes will not corrupt the folder; however, this would be better solved with a latch* to begin with.

*Since terminology can vary from context to context: By “latch” I mean a very temporary variation of a lock that is set immediately before an action and released immediately after it; by “lock” a full lock that is set at some time to prevent others from performing a certain operation, possibly hours before an action takes place, and released at later time, possibly hours after an action has taken place—and with no guarantee that an action actually does take place. Alpine uses locks and not latches in this terminology.

What would be the proper way to handle the situation? (Assuming that a peaceful coexistence, e.g. through use of latches, is not possible.) The ideal way is likely to prompt the user whether a lock takeover should be attempted. Next best version would be to simply tell the user that the folder (in the second instance) is in read-only mode because of an existing lock. (For Alpine, the latter must of course be complemented with the ability to manually remove locks, in case of a software crash leaving a lock in existence. In many other cases, say a good DBMS, such locks would be cleaned out automatically.)

At any rate, it is quite clear that the lock should per default remain with the instance that locked the folder. The opposite is like a locked physical door changing its lock and handing out a new key to the next person who pushes down the handle…

Computer/software usability is an ample source of other examples. Consider e.g. w3m, a text-based browser that I often use when logged into a server: Among the problems are that is has several ways to exit. One, the input “Q”, exits unconditionally; another, the input “q”, prompts the user whether w3m should quit or not.*

*Fortunately for me, I almost always want and deliberately use “Q” to begin with, making the danger for me personally small. Others can be less fortunate.

Both individually are fine, but taken together they are a mess: The reason for the prompt is to prevent an accidental exit from the application. However, by keeping both, the gain from the prompt is negated: It is still possible to accidentally exit in a very trivial way. True, “Q” typically involves one key more than “q” (specifically, the “shift” key); however, this difference is negligible in many cases, especially with an experienced typist in automatic mode. To boot, it disappears if caps lock is on, which happens quite often for inexperienced typists. Worse: There is another input, CTRL-q, which is very similar in form and likely to be used often (implication: close the current tab)—consider the risk of going for the one and picking the other (e.g. through a memory lapse).

Correct behavior: Have one single command which either prompts or does not prompt based on a setting.

Alternatively, do it the Vi/Vim way and have different commands that are too different to pose a great risk and to unusual to pose a major danger. For instance, to accidentally mistype “:q!” or “ZZ” is a lot less likely than “Q”—and neither is very likely to be confused with “:q”. (Yes, there is still a risk that an experienced typists simply has a brain fart and inputs e.g. “ZZ” wanting to cause some other action; however, the risk is smaller and it is not possible to cover all eventualities without disproportionate cost.)

A more complex example is provided by Scrum, which is usually implemented in a very “to the letter” manner, without an understanding of “why”. The result is then, for instance, that a meeting is held because Scrum prescribes the meeting—but that the purpose* of the meeting is not fulfilled. Indeed, Scrum is often run in a manner very much like “cargo cults”: Imitate the shape of something and hope that magic happens, even when the shape is the only similarity with the real thing.

*Usually continual improvement through “inspect and adapt”.

Similarly, naive project managers often have no eye on what they should achieve: They hold status meetings that do not communicate the status (or does so in a wasteful manner); they make elaborate spread sheets that have no connection to reality, seemingly for the purpose of having a spread sheet*; they spend hours in MS Project and only produce garbage**; they insist that certain project step be marked as completed to fulfill a time plan when the step is not actually complete***; etc. A telling issue is when a project starts to run behind schedule—and the time spent in meetings is increased as a consequence. More time in meetings means less time available to perform; less performance means that the project falls back even further. Do the benefit from the meetings compensate? Very rarely—most of the time they are empty actions to demonstrate that something is being done.

*Possibly, they do have some real purpose, but then they must focus sufficiently on achieving that purpose. They rarely do…

**There is no point in having an elaborate project plan when it does not help with the execution, controlling, reporting, whatnot of/on the project.

***If the step is there for a reason, then they either should wait until the step is actually complete or make sure that everyone understands that the plan proceeds with the step INcomplete. By marking the step as complete when it is not, they spread misinformation and increase the dangers involved.

Many other examples in many areas can be found.

My advice: When you do something, take the time to occasionally consider why you are doing it, what the actual purpose of the activity is—and whether your current actions actually are helpful towards that purpose. (If you do not know the purpose, you had better find out…)

My upcoming sabbatical

For what seems like an eternity, I have been trying to arrange a sabbatical. This has been easier said than done: After my original main project with my current customer, I have received extension after extension, due to a mixture of new projects and problems with filling a permanent position. (Hiring good software developers in the current Germany is quite hard, demand exceeding supply considerably.)

While a “luxury problem”, the delays have grown frustrating in their own right, there being so much that I want to do, including reading, writing, studying, watching movies, fixing my web site, fixing my apartment, traveling a bit, … To boot, my December vacation did not leave me rested, through a mixture of too much ambition* and the need to catch up with a number of left-over tasks, including tax filings and other bureaucracy; the last few months have left me correspondingly over-worked and lacking in energy.

*In turn rooted in my wish for a sabbatical: With these long delays, I wanted to do the most of what time I did have free, resulting in a (with hindsight) faulty priorisation.

I am almost there: I managed to keep the last extension down to a minimal ten days, of which only four remain. March is a mix of work, preparations*, and recuperation; from April on, I can finally get started.

*Address changes, moving things from Cologne to Wuppertal, ensuring that this-and-that works in Wuppertal, …

Exactly how long this sabbatical will be is yet to be determined, but it will be at least until the end of 2018.

There will likely be a considerable increase in my post rate during this period; however, I hope to eventually switch all my writings back to my web site, and to abandon this WordPress bullshit. (Except to keep existing contents available and, possibly, to inform about new writings published elsewhere.) This too is somewhat open-ended, however, the switch being more time consuming than urgent.

My experiences with professional associations and similar groups

There are a great number of professional or amateur organizations, associations, whatnot in the IT, computing, engineering, …, areas that on paper could be of interest for someone like me. In reality, they appear to be mostly a waste of money for the typical* member, and do not always work in the interest of the members. At least some (notably VDI, cf. below) have gone down the road of FIFA-ization (cf. e.g. parts of [1]).

*For those members who have the time and inclination to be highly active participants, the situation might be different—if active participation is at all possible. However, realistically speaking, the typical member will not be a highly active participant.

To date, I have been a member of or considered membership in five different organizations of note. I will discuss these briefly and incompletely* below:

*Including because I probably only remember a minority of the problems encountered and because my memory will often be vague on the details of the things I do remember.

1. ACM: This was my first membership, some time close to the millennium switch.

   I was mildly disappointed with the member magazine, the contents often being too superficial, never digging down in a manner that brought real benefit, and leaving the impression of being more of an abstract introduction-for-executives than educational-matter-for-engineers.* I was outright annoyed at their political activities in the name of the members, especially with regard to the then hot topic of DMCA: While I do consider (at least parts of) the DMCA a bad thing, it is not a given that all members do or did—and using their money for political purposes was unethical. To boot, while I did oppose the DMCA, who knows what ACM would have next campaigned for or against that I would not have agreed with?

   *These problems have been quite common with membership magazines over the years. With hindsight, I even suspect that I would have see ACM as a positive surprise, had I encountered their magazine after e.g. that of VDI.

   With the above I could live, however. What came next was so outrageous that I that could not:

   As I went online to renew my membership, I was met with a form that was very obviously intended for the user to just scroll through and to click “OK”. Half-a-second before my mouse button went down, I was halted in my tracks: Why was the amount almost three times as large as it should be?!? On closer inspection, I found that the the ACM had sneaked in pre-selections for a second membership somewhere else, as well as two (!) “voluntary” donations. Such methods are basically inexcusable when used by a commercial business—but when used by an allegedly not-for-profit professional society?!? I left right then and never looked back: The default should obviously be a renewal with identical content. Any and all change should be made through an explicit action by the users. (I am not entirely certain, but I suspect that the course taken by ACM would be outright illegal in today’s Germany.)

2. VDI: I became a member a few years back—and had enough within just a few months.

   First, this organization is strongly geared at running its own political and other agendas in a very top–down manner. The members appear to be viewed primarily as a source of money, and their interests are ignored. A common criticism from others is that VDI is more of an interest organization for the industry than for engineers (contrary to the name and ostensible purpose). I would tend to concur, but would also point to the common problem of large and old organizations over time becoming an end in themselves rather than a means—or becoming a means to the end of furthering the careers of its governance.

   Second, there were a number of almost absurd problems around membership friendliness, starting with the fact that the membership charter (or whatever the appropriate term is) was only available to those who already were members. In effect, becoming a member is like entering a contract without being given prior access to the T & Cs…

   Third, the web pages, emails, whatnot, were on such an amateurish level that it makes any engineer cringe. For instance, I regularly received HTML emails that were so poorly written that my email client could not render them—and what engineer sends HTML emails to begin with? Other emails had such absurd problems as the greeting appearing in the “To” header before the email address.* The web pages were often lacking in information but filled with uninteresting and wasteful images of the cheapest advertising kind. Know your audience: Engineers are among the smartest people and most rational thinkers around. You might get away with treating the average citizen like a complete idiot (but still should not!), but here? To boot, there were numerous functional problems with the web pages too.

   *And, yes, I have verified that this was not an issue with my email client. The actual raw message had this screwed up.

   Generally, I had the impression that there was very little of an engineer mentality (including a focus on substance), and much of a business-student mentality (including a focus on appearances). Dilbert would not have felt at home; his pointy-haired boss might have.

3. VDE: So far, knock-on-wood, the least disappointing organization and the only one where I am still a member. There are some VDI-like tendencies, but they are nowhere near as strong and there is much more of the engineer mentality I found wanting at VDI.

   To date, I do have two complaints:

   First, sloppy handling of address and bank data causing me a considerable extra effort around the time of the first (?) membership renewal. (Unfortunately, many Germany organizations, businesses, whatnot, are thrown off when a member/customer moves or changes bank. Do not ask me why.)

   Second, an extensive prospect trying to sell a trip to China under the guise of an educational excursion (possibly, to a power plant?). From overall context, it is quite clear that VDE (deliberately or out of naivete) became an advertising tool for a travel business. This included aspects like the price being considerably over the top* and the suggestion that the members should let family and friends join them**. (This to be contrasted with the quite legitimate other, intra-German, excursions they irregularly organize.)

   *Out of interest, I did a few online comparisons, and the price was quite unfavorable. This is the more negative as one would normally expect a better price when one organization brings many travelers to the table than when one single traveler orders for himself.

   **If the main reason for the trip was educational purposes, family and friends should be left at home. The educational part does not (or only considerably more weakly) apply to them, and their presence is likely to be a distraction for the main traveler. The real reason? Almost certainly that they wanted filled seats, irrespective of who filled them, and better profits.

4. CCC*: This is an organization of a very different character, currently best known for a strong focus on political work/lobbying/education for consumer/citizens rights, but with no real member benefits. I originally became a member to give them some economic support when it came to fighting for protection against unethical governmental snooping, for more freedom of information, and similar.**

   *These experiences are the most recent and what moved me to write this post: Do not be mislead by the length of the discussion relative the older experiences.

   **Note that this is a very different situation from both ACM and VDI above: Political activities are not an ostensible purpose of ACM and they took place over the head of the members. VDI, meanwhile, engaged in activities that were not, or only coincidentally, in line with the members’ interests.

   Unfortunately, the CCC turned out to be such complete amateurs with regard to the membership that I not only find it better to not be a member, but also see myself doubting their credibility in other regards too. I am willing to make some allowances for idealism and volunteer work, but somewhere there has to be a limit, and if CCC is unable to stay clear of this limit, well, that does not send a positive signal. I note that CCC, according to the linked-to Wikipedia page, was founded in 1981 and has 9000 members—even a largely volunteer organization of that age and size simply has to have mastered the basics.

   For instance, I have repeatedly received highly unexpected emails that my account would be in arrears since six months back, despite never receiving a notification that any amount was due. No invoice, no statement of renewal, no nothing. Notably, CCC does not support the German standard of Lastschrift or any other payment form that allows the CCC to initiate payment on their own.* That it takes half an eternity to send a reminder is depressing in its own right. Either they, as a group of computers professionals and enthusiasts!, have failed to implement even the most basic of automatic invoice and dunning processes or they are negligent in terms of cash flow…

   *Which makes interaction with the CCC in this regard highly unusual by German standards. With the one major exception of rent payments, virtually all recurring, and many ad hoc, payments are initiated by the recipient, with the possibility of the payer canceling any unwarranted payment after the fact. Correspondingly, people do not expect to actively take steps to pay without receiving a bill. Of course, even when no active steps are required, a bill (or corresponding document) should be sent as a matter of course. Alas, the CCC sees this differently.

   For instance, the member magazine saw its last issue in 2014 (!), yet CCC still mentions the magazine as an advantage of being a member… This to the point that the in-arrears message claims as the one practical consequence “die Datenschleuder wird nicht mehr versandt” (“[magazine] will no longer be sent”). This as opposed to a membership not in arrears where the magazine is not sent either… Now, I do not give two hoots about the magazine*—but I am very concerned about the general attitude and the lack of professionalism displayed. If the CCC over the course of four years has been unable to even just hack together a yearly pro-forma edition of four pages, what does that tell us? If they give members misleading information, what does that tell us? For that matter, when things have taken such a turn, I would simply have officially discontinued the magazine and removed references to it from membership information (apart from the old-issues archive). That the CCC has failed to do so, is a further sign of lacking judgment.

   *But out of sheer annoyance required them to send one, even be it a back issue, for me to pay the outstanding amount.

   Now, if I had trust in their ability to do well in other areas, this might still be tolerable (my membership being intended as an act of support)—but I do not. There tends to be a strong correlation between incompetence in one area of an organization and incompetence in other areas.

   For those who still want to support the CCC: Do not become a member. Instead opt for a one-off donation. The money gets where it should go and you are free from future hassle.

5. IEEE: I have repeatedly considered membership. Every time I have found myself in the conundrum that IEEE requires its members to adhere to a certain code, while not telling the prospective members what that code is…

A similar area is qualifications like Eur Ing: While the idea behind them is not a bad one, the benefits appear to be limited*, the costs and efforts high, and there is a significant risk that these are just a way for someone to get money from others with a low effort… I have deliberately chosen to stay clear of these for now.

*The main benefit, on paper, would be an added professional credibility and/or a few pre-/post-nominals. Unfortunately, since hardly anyone (in at least Germany) knows of these titles and organizations, this benefit will be small in practice. (However, I have the impression that the situation can be different elsewhere, where e.g. the U.K. has similar country-specific titles that are considerably better known and far older.) For that matter, Germans rarely use any post-nominals and usually only post-graduate academic pre-nominals to begin with.

Aside on renewals:
The almost exclusive form of renewal in Germany is the automatic: If the member/customer/whatnot does not give notification otherwise within a certain time-frame, renewal takes place. In other countries, explicit renewal can be more common, as with ACM above: The member/customer/whatnot is asked whether he wants to renew and if he does not actively do so, no renewal takes places. I find the latter to be much preferable to the former, for ethical reasons, for reasons of control, and (in mutual interest) for clarity of communication. To boot, situations like with CCC above could no longer occur. I would go as far as claiming that many German companies use this strategy in the deliberate hope of customers accidentally missing deadlines and thereby being caught in an unwanted relationship for one more year*, especially with an eye on this being the typical minimum renewal period: If this was not the purpose, why not allow a discontinuation with, say, two weeks notice at any given time?

*As opposed to more legitimate reasons, e.g. saving the costs for the additional communication.

Where did those millions go?

The last ten days have been … interesting … work-wise, following a production release, and allowing some observations to be made and lessons to be drawn.

(Below, I am deliberately skimpy with contextual details to avoid a direct or indirect leak of internal information relating to my customer, as well as reducing the risk that someone identifies said customer.)

1. The main event was several million (!) Euros more than intended being paid out to various parties, possibly including some misallocation of money between parties. Correcting this was fortunately possible, but it required a lot of additional efforts and caused face loss for my customer vis-a-vis its customers, the Development vis-a-vis the Payment department, and the both the Development and Payment departments vis-a-vis the executives.

   The reason for this was a single line of code* left in place, when it should have been removed, that caused the state of some records to not be updated appropriately, in turn causing them to be processed a handful of times each day (instead of one single time on one single day). How the hell did something like that get past not only the original changer (me), but also pass the code review and extensive tests by Quality Assurance? How did the resulting unexpected payment instruction not catch the attention of the (usually pedantic) Payment department until several days had gone by?

   *Specifically, a considerable part of the processing had been moved from using individual sets of database tables for data delivered from individual third parties to using one set of unified tables for all third parties. An SQL UPDATE that should just have altered all records given as input also checked whether a Foreign-Key field was filled—a check that might (cf. below) have once been necessary, but was now redundant. Worse, the test was now faulty, because the old tables had used “-1” to indicated the absence of the Foreign-Key while the new tables used a NULL-value. Since the check still used “-1”, no update took place.

   As is often the case, there has to be a confluence of a number of unfortunate circumstances. (Read up on the Chernobyl accident and note how many different screw-ups took place leading up to the event.)

   Here the problems include (but are likely not limited to):

   • My making the original change long before the filling of the unified tables was actually implemented (by someone else) and running my developer tests with correspondingly faulty data.
   • The left-in line of code looking very innocuous, possibly even like a safety measure, which would make it far easier to miss for the code reviewer, especially with no change being present compared to the original code: Very often, it is a good idea to write UPDATE statements in a manner that prevents unintentional changes by checking for an old value*. In this case it was not, because irrespective of the value at hand the records had been processed.

     *Consider writing a statement that corrects some incorrect data for a predefined set of records, and assume that, before the statement is actually executed, one of the entries is altered and processed further through some other mechanism or an intervention by someone else. The results can be quite problematic, and it is better to add a check that the records are still in the state they are assumed to be, e.g. in that a “SET a=[new value]” is accompanied by a “AND a = [old value]” in the WHERE clause.

   • The original code, pre-dating my change, was also almost* certainly faulty, because the check had the same dangers as in the previous item. The problem had simply never manifested, due to the Foreign-Key always having had the expected value until now. (Or had it? If the event was rare enough, it might have gone unnoticed…)

     *I do not have the code at hand, and there might conceivably have been some legitimate reason for its presence in the original; however, as I remember the corresponding statement, there was not—and even if there were, it was a sloppy solution to begin with.

   • The corresponding functionality was never tested by QA, limiting the tests to my previous developer tests (cf. above). Why was it not tested? Well, for the entirety of the test phase, this functionality ran into errors due to missing master data in a sub-system, preventing this specific line from even being reached for the test set chosen by QA*. The developer who would normally have been responsible for amending the data was on a month-long leave, and his replacement was his predecessor—who had moved on to project management and was swamped with the administrative parts of the same project. Despite several reminders over weeks, the data never were amended (and QA had their hands full testing other parts).

     The day before the end of the tests, I explicitly asked QA to do at least an approximate test by removing the sub-system from the equation** so that we had at least the equivalent of a smoke test (which would have been enough to discover this particular error). But here is the real kicker: As I come in the next day, QA tells me that the project manager (the very same as above!) had told them to not perform this test…

     *Inputs copied from the production system, intended to ensure that the test was as realistic as possible. Unfortunately, this often requires amending the master data in the QA systems, for which there is currently no way short of manually finding and copying the right set of entries. (I am not an enthusiastic supporter of this approach, but it is not my decision.)

     **Specifically, that we were to temporarily replace the interface to the sub-system with a dummy that delivered some approximation of the data for a very limited set of inputs.

     As an aside, my main contribution to this debacle was not that I made the original mistake—things like that happen every now-and-then and is the reason why code reviews and QA tests are needed. No, my main error was in not insisting that a test took place.

     Unfortunately, this is not the first time that an explainable, harmless error (here the missing master data) has lead to QA not discovering another, actually harmful error that was “hidden” by the first error.

   • Due to previous delays and the urgency of the deadlines, there were several complications, including the impossibility to extend the QA phase and the fact that the decision was made to go live even with a QA report clearly stating that there were untested areas (including the above) and that this was a high risk endeavor.
   • Production releases are theoretically only made after the informed approval of at least one of the two main executives, who are responsible for a final risk–benefit decision and whatnot. In practice, they rubberstamp all or almost all requests at a speed that forbids an informed decision.
   • The head of Payments had spotted a (still small) discrepancy on the very first day after the installation—but was so swamped that he, by his own statement, postponed the investigation.

   As an aside: I am a great fan of automatic and repeatable tests of various kinds. It is possibly that such would have helped finding the error. However, they are basically not a topic for this very non-agile customer, there is almost no infrastructure for them (apart from what little I have written myself), and even I rarely actually have the time with the hectic schedules that abound. This is the one thing from the Java world that I miss…

2. In a further complication in the same rough area (and hit by the same instance of not-tested-by-QA) it turns out that some amounts had the wrong sign from the perspective of the next system in the chain of processing.

   An older piece of code (that we knew was faulty) calculated amount A from the absolute value of amount B and the sign of amount C—something that possibly might have been reasonable when the system was originally written, processing a more limited range of business cases from another source, but was now heavily outdated. After investigations and long discussions (including with the domain expert from and co-head of Product Management) the decision was made to ignore amount C and to use the negated sign of amount B instead. This made for a consistent and plausible calculation and preserved compatibility with the old results in the vast majority of cases—after all, if the old results had not been correct most of the time, the problem would have been fixed much earlier.

   Alas, no: The calculated values were only used in a minority of all cases—and this minority required the (unnegated) sign of amount B. The other cases had almost all been the exact opposite of what they should be for many years—but since the values were not actually used, no-one had ever noticed.

3. Generally, it is often very small details that cause problems in software development, probably because they are so easy for both developer and code-reviewer to overlook. Above we have one example with a single line too much and another with a single character (a “-”) too much. A few weeks earlier, a colleague had in three cases used one of two different functions that only differed in that the name of the one contained an “n” (for n-umber or n-umerical) and the other an “a” (for a-scii, i.e. text) and that the one assumed that leading zeroes should be removed from input. One of the three cases correctly used the n-version, the second correctly used the a-version, the third accidentally the a-version when it should have used the n-version. Again: One single character—with very incorrect calculations as a result.
4. The quite extensive changes made with this production release caused several prior errors that had gone undiscovered, typically for years, to manifest due to harder checking and less leniency.* This included, albeit mostly shortly before the release, the discovery that a third party had incorrectly passed us certain data in a pre-anonymized form that we needed in its original version, anonymization taking place in our system.** The effect was that several things that we needed to calculate based on this data were occasionally miscalculated, including some cash-flow dates.

   *Being lenient with e.g. input from other computer systems (as opposed, possibly, to “from human users”) is usually a big mistake. It is better to “fail fast” so that the problems with the input can actually be fixed, and do not remain hidden and potentially harmful for years.

   **Specifically, a string of characters that needed to be compared to configured ranges of characters to identify certain characteristics. A part of this string was overwritten with zeroes in a blanket manner, implying that the comparison often picked the wrong range or, more rarely, did not find any range at all. The latter case was sufficiently rare that we had hither-to assumed that it was a case of either the third party rarely receiving incorrect input, it self, or slight discrepancies in the range definitions used by various parties. (The ranges all coming from yet another third party on a regular basis, allowing for the possibility of small, temporary differences in definitions among different parties, depending on exactly when the definitions were received and imported.) The value it self, as opposed to the results of the calculation, had not caused any alarm for the simple reason that we normally never see these values: They are used for the calculations and then immediately anonymized, and, because we use the same anonymization algorithm as the third party, it is later impossible to tell that the data had been pre-anonymized.

5. A strictly speaking undramatic and, in it self, unimportant issue was a temporary performance drop after the installation. This was easily resolved on the same day, but contributed (as did the interruption of services needed for the original installation) to Payment having its schedule delayed by several hours and indirectly to Payment not acting on the discrepancy they found (cf. above). The main hitch here is lack of communication: Payment had, as became obvious, no idea of the scope* of the changes going live that day, and were left unprepared for delays that were entirely within the expected from a Development point of view. Had they been sufficiently made aware that this was not just another run-of-the-mill release, things would have gone smother, the interdepartmental irritation would have been considerably smaller—and we might have discovered the main problem on the first day.

   *A very major project following the outdated “Waterfall” model that required correspondingly very major changes to the code. There is something to be said for the short release cycles of e.g. Scrum…