Setec Astronomy

Writing Software That Can Kill

Thu, 07 Jan 2016 00:00:00 -0500

Over the last few articles, we've discussed whether software engineers have ethical responsibilities over the code that they write. We've talked about how badly written code can lead to harm, how failures — even lethal ones — don't necessarily lead us to write better software.

Let's say, for the moment, that you believe that we do have a moral obligation to the public around the software that we build. I want to discuss a particularly sticky question: should we build software that could kill people?

Narrowing the Scope

It's important to be precise in definitions, here, since there are several kinds of software that can be very dangerous, but that don't have the same moral question mark.

First, and most obvious, is software that powers systems directly affecting human life. Software that runs power plants, powers pacemakers, keeps the electrical grid humming, and lets planes fly on autopilot. More generically, this is software which can cause life-threatening harm to people if it functions improperly, but should not cause harm when working correctly. (The term of art here is "life-safety critical software".)

The next category of lethal software is software whose purpose requires life-threatening harm, but can be more or less harmful depending on its proper function. Here, we put a lot of software in the defense world; a properly functioning missile guidance system will certainly be lethal, but an improperly functioning one can be far more so.

The third category is software cannot be directly harmful, either in normal or abnormal behavior, but can indirectly cause life-threatening effects. Most of the software in the world falls into this category; as previously discussed, even something as innocuous as a hookup app can be lethal if put into the wrong hands without proper precautions in place. Banking software, for example, can't directly injure someone, but an abnormally functioning one can certainly cause harm indirectly — even harm on a global scale.

The last category is the one I want to discuss further: software which is not intrinsically intended to do harm, like some military software, yet can directly result in harm even when working correctly. This is software for which the risks and the proper function are inexorably combined with no ability to separate them. This is the area where much of the controversy both in and outside of the software community lies.

The Dangers of Secrecy

The most commonly discussed software with these properties is encryption. If well-implemented and well-designed, it can resist even powerful adversaries attempts to intercept communications. It has been a well-respected tool used since ancient times, with pen-and-paper ciphers being used for millennia. But this story can be told from two perspectives, both equally correct.

Encryption can save lives. Tor, one of the most famous anonymization tools used in modern times, has been used by members of the media to deliver news into areas where information is heavily censored and controlled, by law enforcement to protect undercover officers from detection, by human rights activists to report on torture and other abuses, by whistleblowers both famous and infamous, and by the military to help field agents and intelligence assets whose lives are at stake. [1]

The same story can be told from a very different perspective, however. Encryption can be used to protect pedophiles distributing child pornography, as a tool to help hide bot-net command-and-control traffic, and a way for terrorists to coordinate attacks against innocent people. It can be used to help criminals escape prosecution, and it can be used by governments to hide abuses and human rights violations. It can be used to distribute information on topics ranging from bomb-making to hate-speech, from money laundering to getting away with murder.

Encryption can help save lives, and encryption can help end them.

Security Through Surveillance

Encryption is not the only technology that presents this kind of duality. Surveillance software, too, can cut in both directions. Software that monitors internet connections can help prevent the spread of malware, reduce the likelihood of corporate espionage, and protect countless people's personal information from being leaked by a careless or malicious employee.

On the other hand, it can also be used as a tool to hunt down whistleblowers, silence protesters, choke off dissent, and intercept people's most personal conversations and information.

Used for either of these purposes, it is working as designed. It would be neigh-impossible to design a system such that it could both block pornography and report on such and yet could not be able to block and report on people visiting political websites. Like a firearm, it is as much about who uses it as how it works.

Deciding to Build

How should we, as software engineers, weigh these potential costs in our mind? Should we simply forgo the potential civic benefits these kind of software can have because of the potential for use by bad actors? If not, how can we look ourselves in the mirror when these tools are inevitably used for evil?

Each circumstance is different, but the following set of questions have been useful to me when I've struggled with weighing these questions in the past.

Who is meant to use the software?

Software which is intended for use by the general public has stronger moral character than those intended for use by a limited set of people, such as corporations, IT departments, or governments. Software can change the balance of power that already exists; centralizing power in the hands of the few can be dangerous. Here we would weigh software like PGP, designed to work in a decentralized fashion between individuals, as more moral than software like S/MIME, where power is centralized in the few. (S/MIME where certificates and keys are generated by a third party, fails this test even more significantly.)

How substantial are the resources required to run the software?

Software which requires few resources has better moral character than software which requires many. If the software requires substantial computing resources, it will likely end up centralizing in the hands of the few, even if there is no de jure requirement for it. Here, though, resources can be non-physical as well; if the software requires network access that is not easily obtained to usefully function, such as the ability to observe all traffic on a link, or be distributed across many points of presence, its moral character is weakened.

Does the software have more than one category of users?

In general, software that has mixed categories of users require more scrutiny than those which only have one. An example, here, would be a software that has "administrators" and "users", or "customers" and "end-users". It is especially concerning if one group of users is given privileged access over others.

As an example, an IDS system which monitors for malware communications and reports infections to an administrator would be considered to have two groups of users -- the IDS administrators, and the end-users who it monitors. This would be judged less moral than a local installation of a malware detection system which monitored and reported to the same person.

Do you, the developer, have a direct financial interest in the software?

Software which you, the developer, have a direct financial interest in (whether you are being paid to write it, or are selling it yourself), is less moral than software you have only an indirect, or no, financial interest in.

This is probably one of the most difficult rules to weigh, and on its surface can be paradoxical. Why should I hold software that I write for work to a higher moral standard than software I write for free? Code is code, whether I'm paid for it or not. All else being equal, what does it matter if I get paid?

It is a fact of life that those who are responsible for paying our salaries have a measure of control over us. Whether explicitly ("I order you to...") or implicitly ("It's important to the team that..."), this pressure can induce us to weaken our resolve. We can even do this to ourselves; I, certainly, have done things I wasn't happy about knowing that my employers would want it done, without them applying any coercion of any form. Even the self-employed are not exempt from this psychological pressure; the potential for money is a very tempting one in itself.

This metric is meant to provide push back against those pressures, both internal and external. The red flags raised in other questions should be considered more severely whenever you are in a context where your personal interests are directly involved.

Working An Example

For our first example, let's use Tor. While it's a generally respected tool, it has had its controversies; the owner of Freedom Hosting, at the time the largest web host designed for use with Tor, was arrested and called him "the largest facilitator of child porn on the planet."

Who is meant to use Tor?

Tor is designed for use by the general public. Though companies are free to use the Tor network, it is designed to resist any one person from gaining too much control over the network as a whole.
How substantial are the resources required to use Tor?

Tor requires no substantial or privileged resources to gain the secrecy it controls. Though it does depend on having a number of high-bandwidth connections to the internet along its backbone, the network can be built from an aggregation of common high-bandwidth contributors.
Does Tor have multiple categories of users?

Yes, Tor does have different categories of users. However, no group is in a privileged position over any other group. (A malicious exit node can intercept unencrypted traffic exiting the Tor network, but this is not particularly a more privileged position as far as Tor itself is concerned.)
Does anyone have a direct financial interest in the software?

Tor is shepherded by the Tor Project, a 501(c)(3) corporation that relies on grants and donations from the general public. Though they have employees, there is a substantial volunteer base who would be in a good position to detect any malicious behavior.

Getting Advice

Struggling with these sort of questions in isolation can be difficult, if not outright draining. One of the things I've found very helpful is talking them out with other people who understand the issues at hand. Whether it is co-workers, friends, or an advocacy organization like the ACLU or EFF, you should talk to someone if you are weighing a problem like this. If you have no one else to talk it over with, I'm more than happy to be a sounding board.

Asking these questions can be difficult, but I have no doubt that you, and the world, will be the better for the asking.

Discuss this post on Hacker News

The Price of Progress

Sun, 15 Nov 2015 00:00:00 -0500

After my last article, I had a lot of interesting conversations with different people on whether software development is a form of engineering, as well fascinating conversations on ethics in general. Not long after my last post, in fact, The Atlantic did an interesting piece covering some of the same material called Why Computer Programmers Should Stop Calling Themselves Engineers (I have no delusions that their article is whatsoever related to mine; the timing is coincidental.) The general comments on the article boiled down into a couple of broad categories:

The Goggles^W Codes; They Do Nothing!

One of the most common critiques that I read was that engineers shouldn't have a code of conduct because they don't work. Several people commented that, considering the lack of licensing (and thus the low barrier to entry), as well as the low current penetration of people who would commit themelves to a code of conduct, they would only punish those who would subscribe to them. Just like vaccines, codes of ethics only work when enough people are willing to commit themselves to it; before then, those who do can simply be dismissed and replaced with minimal disruption.

That's very true. Practically, I think these sorts of refusal would only convince some (maybe even small) set of those pushing for unethical action. As is commonly said in the physical security world, "locks are on doors only to keep honest people honest." I think there is a broader philosophical question about whether or not one should continue to refuse to engage in unethical behavior even when believing that doing so is futile— but that's a subject for someone better educated than I to speak on.

Programming Isn't Ready

Another one of the common critiques that I heard was that software is too new or underdeveloped of a field to put these kind of standards on yet. Other forms of engineering work on very well-studied principles with hundreds or thousands of years of study and discipline, but software is a relatively new phenomenon, with radical changes in our understandings and the state-of-art coming on the span of years, rather than decades or even centuries. The oldest buildings still standing are 4000 years old, while the oldest software application still running is, at most, 60 years old. (In all likelihood, this number is conservative, and the oldest extant software is closer to 50, or even 40 years old.) Some also argued that the class of problems are different; physical construction has well-defined and understood limitations, with changes after construction has started being relatively expensive to do. Software, on the other hand, is rapidly iterated on in a very different manner, and the flexibility creates far more potential for ill-defined specifications and mistakes in implementation.

Again, true. It's certainly not in dispute by anyone that software engineering is a newer field than engineering for other disciplines; even those who would argue that "modern" engineering is relatively recent in terms of the span of history certainly wouldn't argue that it's as new. Still, this seems to me to be a relatively uninteresting argument; just because something is new, does that mean we should not strive for better?

A good friend of mine, Michael Stone, provided what I believe to be a more compelling variant of this argument. With apologies in advance to him for any misstatements of his eloquent point: Software failures, unlike failures in other disciplines firmly defined as "engineering", aren't generalizable. Unlike in engineering, where failures can often be traced back to a more general category of problem, software system failures don't share the same kind of fundamental design failure. At most, the community is able to point to certain kinds of failures which are common — off-by-one errors, buffer overflows, etc. — but we don't respond in the same way.

Take the case of the Tacoma Narrows bridge collapse as an example. After the Tacoma Bridge collapsed because of aeroelastic flutter, the state of the art regarding bridge design was fundamentally changed. One of the authors of the report from the Federal Works Agency Commission that investigated the collapse is telling enough that I think it bears reproducing here wholesale:

The Tacoma Narrows bridge failure has given us invaluable information...It has shown [that] every new structure [that] projects into new fields of magnitude involves new problems for the solution of which neither theory nor practical experience furnish an adequate guide. It is then that we must rely largely on judgment and if, as a result, errors, or failures occur, we must accept them as a price for human progress.

— Othmar Ammann, The Failure of the Tacoma Narrows Bridge

I can find no examples of bridge collapse attributed to aerodynamic failure after Tacoma Narrows. Action was taken to reinforce similar bridges shortly after the collapse, and aerodynamic flutter is considered in structural engineering of everything from buildings to airplanes today.

As famous as the Taacoma Narrows collapse is in structural engineering, we can find a counterpart in software failure: the case of the Therac-25. The causes of Therac-25 are interesting enough reading that I suggest reading Professor Nancy Leveson's report in full, but the fundamental failure was a race condition (with many exacerbating factors). Therac-25 killed three people and injured three more, while the Tacoma Narrows failure killed no people, and only a single dog.

Unlike in Tacoma Narrows, however, Therac-25 did nothing to fundamentally change the state of the art in software programming. Race conditions, certainly, are a known problem -- in some cases, one that programming languages, techniques, and frameworks strive to solve. Even still, race conditions failures in software programs are still so common that I need not reach past personal knowledge to find examples. Paying the "price for human progress" doesn't seem to substantially advance the state of the art in computer programming. The same class of errors that caused Therac-25 to become lethal in 1987 led to the Northeast Blackout of 2003, affecting over 55 million people and leading to several deaths. There are countless more examples of financial losses and physical harm that can be attributed to race conditions, yet we still write programs vulnerable to these sets of programs today.

Blame is the Enemy of Safety

Whether we have codes of conduct or not, whether software programming is an engineering discipline or something else entirely, it seems to me that there are few who would seriously argue that the current state of the programming world is perfect and that we have nothing to improve. As Professor Leveson would, no doubt, remind us: blame is the enemy of safety. If we, as an industry and a profession, want to build safer, more reliable systems, we must give up the idea that we can prevent accidents if we only sacrifice enough people who write bad software on the altar of "reasons for outage" and "post-mortem analysis". I cannot recommend reading "Engineering a Safer World" by Nancy Leveson enough.

If we are not engineers, then perhaps we are craftspeople; if so, I hope we can become proud of our craft and treat it with the respect and care that it deserves.

Executing Software Engineers for Bugs

Sun, 01 Nov 2015 00:00:00 -0400

There is an apocryphal tale I've heard many times about how, in Ancient Rome (or, in some tellings, Greece), the engineers responsible for the construction of an arch were required to stand underneath it as the final wooden supports were taken out. In some version of the story, not only was the engineer required to stand underneath it, but so was the engineer's family. In the case of the Veresk bridge in Iran, some believe that the Shah forced the chief engineer to stand under the bridge with his family as the first train went over it. Whether any of these instances actually happened or not, the stories tell us something about the cultural perspective of the obligations that engineers have to society as a whole.

In 1920s, after the Quebec Bridge disasters, a professor at the University of Toronto convinced the Engineering Institute of Canada to come up with a ceremony and a code of ethics that all its graduating engineers would be required to undergo. This ceremony became the Ritual of the Calling of an Engineer. This ritual is similar to a much more ancient tradition of the Hippocratic Oath undertaken by physicians which states their obligations to those under their care above and beyond their obligations to anyone else. It seems clear, at least to me, that there is a cultural understanding of the obligations of civil engineers and architects to the general public. Take the case of the Sampoong Department Store Collapse in 1995, the most lethal building collapse since the collapse of the Circus Maximus in 140 CE. In this instance, not one but two different construction firms refused to complete work that they believed to be unsafe and resigned -- or were fired -- after they did so. Five years after the building opened to the general public, it collapsed, killing more than 500 people.

All of the major professional engineering societies have codes of ethics that they require their members to obey which include statements of engineer's obligations to the general public. "Engineers shall hold paramount the safety, health and welfare of the public and shall strive to comply with the principles of sustainable development in the performance of their professional duties," from the American Society of Civil Engineers, is one example, but the language is similar between them.

Even the professional societies for software engineers have such statements; the Association for Computer Machinery: "Software engineers shall act consistently with the public interest... [approving] software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good." The Institute of Electrical and Electronics Engineers: "We, the members of the IEEE... accept responsibility in making decisions consistent with the safety, health, and welfare of the public, and to disclose promptly factors that might endanger the public or the environment."

Unlike our brothers and sisters in the other engineering disciplines, though, software engineering does not have the same history of mandatory formal education. Indeed, many hackers -- myself included -- take pride in the fact that the software community takes in both the self and formally-trained. Perhaps for that reason, we do not have anywhere near the same level of membership in professional societies as the other engineering disciplines.

I would wager good money that almost every single person reading this has written code that was deployed into production before they were happy with how it was written and confident that it had no bugs in it. For that matter, I would wager the same amount that most of us have had code deployed to production which we knew had bugs in it, but were pressed to do so anyway.

I can see the objections many of you are writing me right now: "But, it's not the same! Sure, people who program code that go into medical devices, or power the space shuttle, they should definitely be careful. But I just write code that powers a video game, or an advertising network, or a new social network for dogs. It can't hurt anyone!"

Let me be clear up front: You're right, in part. Code which has life-safety implications should be held to a very much higher standard than code which isn't. I certainly hope that the code which is inside an implantable defibrillator has gone through a hell of a lot more code checking and QA than the thrown-together code I've written for online advertising. But making the leap from that to "my code can't harm people" is a bridge too far.

Look at the recent spat of security vulnerabilities that have leaked countless people's personal information. From OPM to Target, T-Mobile to Home Depot, it seems that it's never more than a couple of weeks between a new report of some catastrophic information leak these days. But it's not only security vulnerabilities that we have to think about. Let's take a hookup app as an example. On the surface, there's not a whole lot to be concerned about here. People are free to give as much -- or as little -- information about themselves as they want; they can select pictures which show their faces, or not, and the only bit of information revealed about them other than what they choose is the number of feet away that a user is from you. Not particularly sensitive information.

However, it's quite easy to turn this relatively harmless piece of software into a very dangerous tool for the users. Take Grindr, for example. In 2014, it was publically revealed that you could query the Grindr API without any authentication and retrieve a list of users near you, along with their distances to you, out to several decimal places of accuracy. In addition, you could query the API -- again, without any authentication -- and give it an arbitrary latitute and longitude of /your/ current location. These two things combined meant it was trivial to triangulate the position of arbitrary users within a couple feet of accuracy [1]. Because Grindr targets the gay community, this quickly became a life-threatening vulnerability. Police in Egypt were recorded using this functionality to track down and arrest -- or extrajudicially punish -- gay people [2]. Though they disabled the "distance" feature for several months, it has been reactivated, though blocked in several countries. It is still possible to triangulate the position of arbitrary Grindr users today.

This is only one example of how a seemingly harmless feature can be extremely dangerous if they're not carefully coded and the ramifications thought out carefully. There are countless more out there; I'm sure every one of us can think of some feature that we have written at one point or another that could be misused to the detriment of our users.

Do we, as software engineers, have the same kind of moral responsibility as our fellow engineers in other disciplines? Should we refuse to build systems which can be turned against users, even if it means our jobs? How do we, as a profession and as a community, balance our responsibilities to our employers and to the general public? If we were required to "stand under the arch", as it were, how many of us would have pushed back against things that our employers asked us to do?

Discuss on Hacker News

MEGA is genius

Sun, 18 Oct 2015 00:00:00 -0400

Let me say this right off of the bat: I'm not weighing in on the business model or the controversies of MEGA when I say that its model is genius. What I am specifically talking about is only its security model. MEGA uses an often misunderstood scheme of encryption as part of its threat model and mitigation that is nothing short of brilliant. This isn't, of course, a new scheme - plenty of others have done similar in the past (Spideroak, for example, among many others.) Still, it's brilliant.

Before we delve into the specifics of MEGA, let's work with a hypothetical example that can better show some of the balancing act that is involved, not just between security and usability, but also with whose security we are discussing.

Imagine a particular kind of chat service that enables pseudonymous conversations between two, randomly paired users of the service. These users can have a conversation of whatever length they desire, and at the option of either of the users, be rematched to a new random pairing. Let's set out some of the attributes that we would like this service to have.

Only the sender and recipient of a message can access and read that message.
Users cannot send messages that have no useful purpose - aka, spam.

Even with that very limited set of constraints, we run face-first into an immediate conflict. The usability demand ("no spam") requires that the server operator can, in some way, monitor the content of the messages, flying in the face of the demand for privacy ("no spying"). How can we resolve this? Thinking a bit more about what the purposes of these restrictions are, let's break these constraints out a bit more to see if we can come up with an area that we are OK with compromising on.

When designing the threat model, there are two different sets of reasons that we might want to ensure that the server operator is unable to read the messages of its users. On the one hand, there is the simple case - we want users to have privacy and for their messages to be secure. Simple. But there is a second, hidden reason that we might want to have this property: we, as the server operator, don't want to be put into a position where we /have/ to monitor our users messages.

Let's assume for a second that we simply off-load the burden for securing the messages onto our users, in the same way that email providers do. The users can use PGP or similar for whatever messages they want to remain private, and for messages that they don't care, they can just use plaintext. (Ignoring, for a moment, the problems of actually educating the users on how to use PGP or similar.) Let's say that a user ends up engaging in some kind of criminal activity using this service, somehow - harassment, maybe, or something else undesirable. It is quite reasonable to assume that the investigating officers will come knocking on the server operator's door with a subpoena, wanting whatever records that the server operator has. Or, more dramatic, they may want to wiretap those users messages into the future. Another possibility, a copyright owner may want us to block certain copyrighted content from being shared on the network.

This is, for many reasons, not a great position to be in as the server operator! We are having to do a bunch of extra work, both political and technological, including some that might conflict with personal beliefs. Let's make this explicit in our goals:

Users can only access messages for which they are the sender or recipient.
The server operator cannot monitor the content of any user messages.
Users cannot send messages that have no useful purpose - aka, spam.

Now, it is clear that the real conflict in goals here is between the latter two, not the former. How can we resolve this conflict? We will need to weaken one of our security goals in order to resolve this problem. How can we do so, without trapping us in the earlier bind? Let's think about the goals of our attacker - in this case, the spammer.

A spammer wants to send their messages to as many users as possible.
A spammer wants either to:
- gain some monetary advantage from the messages, even if indirectly ("commercial spam")
- disrupt the service, either through technological means or by scaring off legitimate users ("lulz spam")

Clearly, regardless of the reason, spammers are incentivized to not have extended conversations with the people they are communicating with. They wield a shotgun, not a sniper rifle. This gives us multiple advantages - not only can we apply technological mechanisms such as rate limiting to interfere with their aims, we have a potential option for weakening one of our security goals without causing a major loss. Since there is not a substantial amount of "metadata" to be leaked in a service that's randomly pairing users - since they can't choose who they are communicating with - our core goals around privacy of the users is aimed at the content of the messages. For spammers, who are likely to want to spew their message and re-pair as fast as possible, it means there is not a whole lot of value to communicating for an extended period of time (thus causing messages to contain more and more private information). How about structuring our goals to recognize that?

Users can only access messages for which they are the sender or recipient.
The server operator cannot monitor the content of any user messages, unless one of the participants chooses to enable them to.
Users cannot send messages that have no useful purpose - aka spam - without detection.

If messages can be revealed by one of the participating users - with, say, a "Report" button - the server operator is able to gain information about people misusing the service and take action against them, without having to risk a broad ability to access the content of messages.

With this weakened set of goals, there is a pretty clear path towards how one can construct a system to achieve them. Let's turn our attention back to MEGA. MEGA is the successor to Megaupload, which was taken down after a Department of Justice investigation into criminal copyright infringement in 2012. One of the prime elements of the controversy around Megaupload was that they had tools in place to seek out certain kinds of content across the network and remove it, regardless of the links to it (as they did with child pornography), and yet they refused to use those techniques to remove copyrighted files. Their ability to access the content that was uploaded to their system caused them to be put into a position where they were being forced to use it - or suffer the consequences.

MEGA, its successor, advertises that it uses client-side encryption to ensure that they cannot read the content of the files which are uploaded to them. This, however, is not to ensure user privacy. In the (vastly) most common usecase, the encryption key is simply provided to those who would access the files as part of the URL and publicly posted. It is obvious that linking to a file, unencrypted, at the path example.com/fileA is no more or less secure than linking to a file, encrypted, at the path example.com/fileA#encryption-key - for the users. It is, however, a tremendous difference to the server operator's ability to make statements about whether or not they are able to access the content stored on their system.

This is a "hack" in the traditional sense, twisting the normal purpose of encrypting prior to upload -- to make the information accessible only to authorized people -- into a different security model altogether, one that blinds only a single person -- the operator -- to the content. For better or worse, MEGA is genius.

TOFU, Keybase, and Other Vegan Security Choices

Sat, 08 Aug 2015 00:00:00 -0400

When we left off last article, we examined the two most common security models used today and found them both lacking. But fear not, brave readers - there are yet more key distribution solutions out there. Surely one of them isn't total shit!

TOFU, or "eh, it's probably alright"

There is one other method of key distribution that is commonly used. Made popular by OpenSSH, TOFU, or "trust on first use", has a slightly different attitude to the problem of key distribution than other solutions. The hardest problem in key distribution is bootstrapping your trust from nothing. At the end of the day, you need to choose something to build your trust on top of: the certificate authority model chooses a list of preselected authorities that are burned into the operating system, and the web of trust model bootstraps based on a network of people you trust in person. TOFU looks at this problem and tosses up it's hands. "Fuck it," TOFU says. "It's probably alright."

Trust on first use provides its security guarantees by protecting all of your connections /after/ the first one. The first time you connect, you are prompted with a key fingerprint of the server and asked if it's the right server.

The authenticity of host 'xmenrevolution.com (97.107.134.16)' can't be established.
RSA key fingerprint is b9:2b:30:7c:a8:89:39:86:9b:13:19:fe:1b:c4:ec:a0.
Are you sure you want to continue connecting (yes/no)?

It is the users responsibility to verify this fingerprint through some out-of-band method. Checking with the server operator, looking at SSHFP records in DNS, or sacrificing virgins to those beyond the veil for the answer. Whatever works. After this first connection, the certificate is saved with a mapping to the host that you connected to. On every subsequent connection, it verifies that the server you think you are connecting to has the same fingerprint as the one you first connected to. If the key changes, it blocks the connection with a warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
b9:2b:30:7c:a8:89:39:86:9b:13:19:fe:1b:c4:ec:a0.
Please contact your system administrator.
Add correct host key in /tmp/asht to get rid of this message.
Offending RSA key in /tmp/asht:3
  remove with: ssh-keygen -f "/tmp/asht" -R xmenrevolution.com
RSA host key for xmenrevolution.com has changed and you have requested strict checking.
Host key verification failed.

Sarcasm aside, this is not necessarily an unreasonable approach. In order to subvert it, an attacker must be constantly in a position to sit between the user and the server, always ready to intercept the connection. If the attacker misses the window of opportunity, they won't be able to use a man-in-the-middle attack in the future without being noticed. Even for an adversary with extensive control, this is a difficult - though certainly not impossible - proposition. TOFU strikes a utilitarian balance between security and usability, but it's far from a silver bullet. It can't be easily used for asynchronous communications, like e-mail, because the window for an attacker is lengthened significantly. It doesn't help at all in the face of a global adversary, or one that is able to attack the network at the point of connection between the server and the internet. It is difficult to imagine it being used for a replacement for the CA model used in SSL/TLS without sacrificing security in a substantial way.

Keybase

One company out there aims to solve some of the problems with the complexity of the web of trust by creating a publicly audited directory of people's PGP keys. It does so not only through a social mechanism, but also by letting people post claims to their identity through attestations on accounts that they control. The Keybase software then lets you verify that these attestations are still in place when you retrieve the copy of their public key. Even if an attacker compromises the person's Keybase account, it is probably difficult for them to compromise all the associated accounts and post new attestations with a key controlled by the attacker. For example, my own Keybase profile has links to Twitter, Github, Hacker News, and this website. An attacker would have to either compromise all of those accounts to post new attestations, or provide attestations to other profiles that could be confused with mine, as well as my Keybase account. (Or Keybase itself.)

Keybase also lets you "track" someone's identity by signing an attestation yourself that you have verified someone's information. This attestation can be used by yourself in the future, if you change computers and no longer have access to the cached information. It can also be used by others to validate that other people have been watching the profile for changes for a period of time. The idea being, even if an attacker manages to compromise all the accounts, the more people that are watching, the more likely the compromise would be noticed and reported. Keybase also uses the Bitcoin blockchain to make attacking Keybase servers more difficult[1]; in the end, it is likely an attacker would choose to target an individual user rather than the Keybase servers itself.

On the surface, it seems that it would be very difficult for an attacker to simultaneously compromise all of the associated accounts on Keybase. Many of the websites that can be linked support two-factor authentication and have strong password requirements. If it's difficult to break into one, surely it's difficult to break into all of them, right?

Well. Sort of. Let's take my profile as an example. I have four accounts linked on Keybase - Github, Hacker News, Twitter, and this website. My email accounts that are supported in my key are a Gmail account and an account hosted on the same server as this website. That would mean, you would need to compromise at least four different accounts, two of which have two-factor authentication, and all of which have strong passwords that are not shared between any accounts. (Password reuse rates have been seen as high as 50%, making the average user's case likely much easier than attacking mine. [2])

There is another link between all of these accounts, though. Password reset mechanisms. Many, many websites - including most of the above mentioned - use email as the out-of-band authentication mechanism. An attacker that was able to compromise an email account could reset the passwords on the other accounts and gain access to them that way and "bootstrap" their access up to other websites. This is a very common tactic and has been used to great effect, including in the Twitter hack in 2009 [3]. "Separate" accounts may not be so separate after all, weakening the fundamental security that Keybase is built on top of. Whether such a compromise could go unnoticed is a more nebulous question.

DNS, DANE, and SSHFS - oh my!

One option that has seen some popularity recently is validating the certificate of a server through the DNS infrastructure. While a little bit of a strange choice at first glance, after mulling it over for a sufficient amount of time, it can make some sense. DNS is the "directory-book" of the internet; why not extend it to give clients a picture not only of what server they want to contact, but also how to verify that server?

Lots of reasons, it turns out. DNS, by default, isn't authenticated at all. This means even trivial attackers can pass you whatever records they feel like. So much for security. Relatively recently, the DNS infrastructure has begun to be signed through a process called DNSSEC. DANE, the standard for putting TLS certificates into DNS, requires that DNSSEC be used in order to trust the results. Unfortunately, the roll-out of DNSSEC has been slow, with many top level ccTLDs not yet signed, and with most DNS resolvers still not verifying the chain.

Even if DNSSEC was fully deployed, it's really just a microcosm of the CA model. Instead of trusting a whole laundry list of domains, though, you must trust a different set of people: the root keyholders, the TLD keyholders, and the nameserver keyholders. If you mistrust anyone in the chain - and the US government is (substantially) in control of the root - then you can't trust results coming from DNS. DANE just ends up being another form of the certificate authority model.

What's next?

We've looked at a handful of different problems with widely-deployed crypto out in the wild - from user experience to key distribution. At this point, you might be feeling somewhat depressed about the state of security on the internet; I can't particularly blame you. A truly secure system isn't something that you can really ever reach, though. A recurring joke in ##crypto is a statement of the form "Best security practices recommend you X", where X is something of the form "encase your system in cement", or "cool your system with sulfuric acid". Security and usability are opposite ends of the spectrum; a well-secured system has struck the right balance between them for its particular threat model. Security needs to exist in the world, not a vacuum. So don't fear, and don't reach for the impossible. Find the right balance of compromises for you.

This is the last article in the "GPG is awesome; GPG is terrible" series. If you have comments or questions, please feel free to reach out to me, over IRC, email, or IM. I will continue covering security related topics in the future; if you have suggestions of things you'd like to read, please feel free to recommend them.

Compromisable Authorities and the Web of Mistrust

Sat, 01 Aug 2015 00:00:00 -0400

The last article walked through the key generation process and all of the traps that will end in an insecure key. Even after that, though, a new key pair doesn't do you any good whatsoever if you can't get it to the people you want to communicate with. As the saying goes, "encryption is easy; key distribution is hard."

There are two main approaches to key distribution: the Certificate Authority model, and the Web of Trust model. Both have slight variations and extensions as they are implemented in the "real world", but they boil down to roughly the same fundamentals.

Certificate Authorities

The Certificate Authority (or CA) model is the basis for the majority of encryption used on the internet today. From corporate intranets to securing your bank's website, Certificate Authorities is the method of choice for making sure that the key you are using to talk to someone actually belongs to the person you want to reach. SSL/TLS, which secures (essentially) all encrypted HTTP traffic in the world, mandates the use of the CA model.

The Certificate Authority model is very simple. Baked into your web browsers and your operating systems is a list of companies which you trust to validate identities on the internet, and so-called "root certificates" for those authorities. You might be surprised to know just how many of them there are, and with names that you probably have never even heard of. At the time of this writing, there are 180 root certificates that are distributed by the Mozilla Foundation with their Firefox browser, and other browsers and operating systems will have similarly large counts, though some of the specifics may vary slightly.

The complete list has many names that are recognizable - the governments of France, Japan, Spain, Taiwan, The Netherlands, and Turkey, for instance - as well as many names that aren't nearly so familiar. Did you know that companies like Atos, Buypass, Camerfirma, Izenpe, and WISeKey can all sign certificates that will be instantly trusted by your computer? This is not to say that any of these companies is necessarily untrustworthy - but it also points out the fundamental flaw of Certificate Authorities: you need to have at least one agency that you trust, fundamentally, to authorize all the others. - and you need it at the very beginning.

If it's not included in your browser, or in your operating system, you have no way of determining if any list of certificate authorities that you retrieve in the future is authentic. These CAs, well-meaning as they may be, are a target for attack. Hackers have compromised them in the past, issuing fake certificates which were used to impersonate Google, among other things. It is commonly believed that the United States Government stole certificates for use in distributing targeted malware to disrupt the Iranian nuclear program. CAs have also made mistakes, issuing certificates with more authority than they were intended to have. These CAs are large targets precisely because of the pedestal on which we put them.

Slimming down the list of CAs is certainly a good idea in theory, but in practice it is very difficult. All the certificates issued by those CAs -- and all the certificates issued by their delegates -- need to be reissued. Nor does it answer the solve the harder question: how do we choose who to trust?

Web of Trust

Many people would draw a parallel between the web of trust and communism: great in theory, but terrible in practice. The web of trust takes a decidedly more personal approach to solving the problem of key distribution - or, as today's Silicon Valley VC-hotbed would say if it was reinvented today, "a social network for key distribution."

Whereas every person who uses a particular version of an operating system or browser works from the same, universal set of CAs, no two people have the same view of the web of trust, even if they have an overlapping layout.

The web of trust, most commonly known as the PGP web of trust, is built on the concept of "nearness". Each person starts out trusting no one but themselves, and their own keys. Through a process of in-person verification of stranger's keys, potentially at a ceremony the like of which would once have resulted in a burning at the stake due to the cryptic incantations, the goal is to establish a link of verified keys and trusted parties between you and anyone you would want to communicate with. The hope is, with enough people that you have a trust relationship with, you can reach anyone that you might want to talk to through a minimum of hops.

If you have read that paragraph a couple of times and still have no idea what the fuck it means, have no fear: that's a quite common for first exposure to the web of trust. Unlike the Certificate Authority heirarchical architecture, the web of trust is a distributed mesh of relationships, and it is far from simple.

There are two fundamental operations in the web of trust: verifying (or attesting to) a key and assigning trust to an identity. Let's start with the more common one - verifying a key.

Key Signing

You and a friend (named Alice, by convention) have both managed to generate a secure key and you now want to be able to communicate securely. Key exchange is relatively simple if you can meet in person - but how can you leverage people who you meet in person to talk to people that you can't?

You and Alice meet in person and exchange a copy of each other's public keys. This can be done by giving each other a USB stick with the copy of the public key on it, or more commonly, giving each other a piece of paper with a relatively short "fingerprint" of the key that you can use to retrieve the key from a server while trusting that it hasn't been altered. If you don't know Alice that well, you might also examine a few copies of her identification, to ensure she really is the person she says she is.

When you go home that night, you "sign" a particularly formed statement that PGP compatible systems know how to read which states that you believe that Alice's key, with the fingerprint that you received from her earlier, actually belongs to her. You then send this attestation to Alice for her to distribute publically. Once she does so, anyone who retrieves a copy of her key will also receive your attestation.

Leveraging Trust

You go around to all of your other friends and exchange keys with each other. At the end of the day, you build up a nice network of people that you have signed keys from. How does this help you?

While you have signed a bunch of different people's keys, you still don't have a mechanism to reach anyone whose keys that you haven't met yourself. However, there's a simple thing you can do to extend your network. As well as attesting to one of your friend's keys, you can also tell your PGP system that you trust this particular friend to verify identities on your behalf. You can even publish this statement of trust publicly. When you do so, other people who trust you can leverage your verifications to reach people that you have met and they haven't, and vice-versa.

Great! Everyone publishes their trusts, everyone publishes their verifications, and we can reach everyone that we want to. This is great! No single point of failure to compromise, leveraging real-world relationships. How social! How Web 2.0!

Getting Stuck in the Web

So, that's it. The web of trust is great. Problem solved. Right? ... Right?

Not quite. The largest network of PGP servers, SKS Keyservers, has around four million keys in its database [1]. Of those four million, the largest group of keys for which there is a validation path to every other key in that group is less than 60,000 [2]. That means, only 1.5% of keys on the key server can actually be validated through the largest web of trust (called the "strong set"). All of the other 98.5% of keys are disconnected - either signed by no one, or signed by a small group of people that has no connection to the larger group.

This problem is called the "small world" problem, and it is the giant hole in the web of trust. The web of trust depends on a large interconnected graph, with published trust levels, in order to "walk" the path to the key of someone you want to talk to. In practice, even if you manage to get into that 1.5% of the strong set, you are probably too "far away" on the graph to use it all that effectively.

This is a bit confusing, so let's work through a real-world example. Let's say I want to send an email to Linus. I have a key in the strong set, and so does he, so I should be able to leverage that web of trust to retrieve a key that I can be confident he actually controls, right?

Well... sort of. No one that I trust fully to validate keys on my behalf has signed Linus' key. People who I trust fully have signed keys that have signed keys that have signed Linus' key, but nothing direct. When I ask GPG what the trust level of Linus' key is, it tosses up its hands:

pub   2048R/0x79BE3E4300411886 2011-09-20
      Key fingerprint = ABAF 11C6 5A29 70B1 30AB  E3C4 79BE 3E43 0041 1886
      uid                 [ unknown] Linus Torvalds <torvalds@linux-foundation.org>
      sub   2048R/0x88BCE80F012F54CA 2011-09-20

Digging into this further, GPG (by default) uses the following pattern to dictate how trusted a key is:

A key is trusted if it has been signed by you, or signed by someone you trust fully; OR
A key is trusted if it has been signed by three people who you marginally trust; AND
The path between the key and you is 5 steps or shorter.

What does this mean for me emailing Linus? Well, #1 isn't valid - I haven't signed Linus' key, nor has anyone I fully trust signed it. No one that I marginally trust has signed his key.

One technique that is (somewhat controversially) used is the idea of transitive trust. Most frequently, this is stated that trust is transitive at one level less than the parent. That means, if I trust someone fully, and they sign someone's key (let's say, Bob), then I marginally trust Bob to sign keys on my behalf. Trust, in reality, doesn't really work this way - but let's pretend for a moment. Does that help me talk to Linus?

It gets me closer, but not close enough. I have three different paths of length three between Linus and myself. Of those three paths, two of them start with someone who I trust fully, and one of them starts with someone I trust marginally. This means that two of the people who have signed Linus' key get assigned the ability to produce marginally trusted signatures, and one has an untrusted signature. Two marginal signatures isn't enough to satisfy rule #2, so the trust level remains unknown, and it's not "safe" for me to email Linus and have confidence that the key really belongs to him.

Now what?

Well, shit. We covered the two most common models for key distribution, and we have a long list of why they both suck. Now what? In the next article, we will cover some of the less common approaches to key distribution and see if any of those look better.

GPG is awesome; GPG is terrible

Sun, 12 Jul 2015 00:00:00 -0400

When you read about GPG (aka GnuPG) these days, you will find a bunch of very intelligent people saying two completely opposite things. On the one hand, we hear that GPG is an extremely effective tool against surveillance; it was one of the primary tools used by Edward Snowden in his efforts to release information about illegal activities by the NSA [1]. On the other hand, we hear that using GPG correctly is an incredibly difficult (if not almost impossible) process, and that is uses a protocol that is increasingly showing its age [2]. While these appear to be conflicting propositions, I believe that they are both true. GPG is awesome; GPG is terrible.

For a moment, let's leave aside the problems with the GPG protocol itself - lack of forward secrecy, lack of deniability, overly complex controls with insane defaults (yes, CAST5 is still the default symmetric cipher), compatibility with ancient legacy standards, to name a few - and focus on what I think is a larger problem with GPG: its user experience around key management.

Let's walk just through the generation step together with a fairly recent version of GPG:

hlieberman@agartha:~$ gpg --gen-key
gpg (GnuPG) 1.4.19; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection?

While the default that it suggests is sane, someone who is not familiar with GPG is likely to be confused about what exactly the differences are. Even for someone familiar with the algorithms, half of the choices will give you a key pair that you can't receive encrypted email with. But, OK, let's just blindly hit enter and accept the defaults.

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)

While there's certainly nothing wrong with a 2048 bit key, these days new keys should probably be 4096 bits. Again, let's accept the default.

Please specify how long the key should be valid.
0 = key does not expire
<n>  = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)

This is one of the more common mistakes I see for people generating GPG keys; one I have made myself before. There is almost no reason to generate a key that has an expiration date longer than a year, and no reason to generate a key with no expiration date set. Expiration dates can always be changed in the future - even after a key has expired. However, they provide an important safeguard in the event that you lose access to both the private key and a revocation certificate. Especially when you're new to GPG, it's not unusual for you to need to toss the first key that you create, and you don't want a key sitting around on the keyservers forever that you can't remove. It's good practice to make sure that your key has a somewhat close expiration date[3].

Let's accept the default and move on.

Key does not expire at all
Is this correct? (y/N)

Uh-oh. Is this an error message? Is it warning me about a bad practice? Let's go back and set an expiration date.

Is this correct? (y/N) N
Key is valid for? (0) 3y
Key expires at Wed 11 Jul 2018 07:08:41 PM EDT
Is this correct? (y/N)

Guess it's just a warning by default to make sure it's parsed what you want correctly; I understand now.

Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Harlan Lieberman-Berg
Email address: hlieberman@setec.io
Comment: This is just a test key.
You selected this USER-ID:
    "Harlan Lieberman-Berg (This is just a test key.) <hlieberman@setec.io>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?

Using a comment isn't a great idea in most cases. It either provides repetition of information already in the email address, or is used to tell people things that should better be elsewhere [4]. (Why do you have a key with the comment "Don't use this one" that's not been revoked?)

From here on, it's relatively smooth sailing. But did this cause me to generate a key that's secure? Well... sort of. If you're using a new enough version of GPG, you should be OK, but even fairly recent versions don't follow best-practices. You probably used SHA-1 for your self-signature, instead of something stronger; you may even have used MD5. Your preference order for ciphers probably includes stuff you don't want -- for this particular version of GPG, it includes 3DES as a preferred encryption algorithm. Similarly, it prefers SHA-1 above SHA-512.

You didn't generate a revocation certificate, though doing so is very much recommended. You also aren't following the current best-practice of having an entirely offline primary key - and using the same key for signatures and key management to boot.

This was hardly a great success, and we haven't even gotten the key anywhere that we can use yet. In the next post, we will focus on the next set of problems: securely integrating into the key infrastructure.

Solitude

Sun, 28 Jun 2015 00:00:00 -0400

Looking back on the months that I spent working on AdvertiSin, I am struck by how inaccurate my estimates of what my problems would be were. Before I started working on AdvertiSin at all, I spent a long time looking at the feasibility of the business. The classic MBA outlook: identify the market, identify the competition, identify the differentiating factor, the value proposition, and so on and so on. I looked at timelines, at financial forecasts, and examined whether I really thought the idea was workable. I did - and still do. When it came time to work on AdvertiSin full time, I leapt into the brink gladly, confident that I had a good grip on what the risks were - at least as much as such a thing is possible, where startups are concerned.

I was wrong. In the end, the thing that was by far the most difficult for me to struggle with was a risk I hadn't even considered the broad category of when I set off on this project back in January of 2014: me. When I was weighing things, I was focused on the idea of the business itself, and the implementation of it. I, personally, didn't come into it, except with how I thought my skills could manage to do this or that.

In retrospect, I did get some very good advice that I didn't pay enough attention to. Both of the founders of my then-employer, Alden and Venkat, warned me about the effects of solitude on startup founders. They warned me about the sense of isolation, about the feeling in the pit of your stomach that can happen a couple of weeks in of "oh, fuck, what have I done?"

I watched for that feeling in the back of my mind, but it didn't manifest anything like I thought it would. It wasn't a hollow feeling, or a panic that struck late at night. I was so focused on what I was doing, I forgot the things I learned as a manager; the symptoms of burn-out sneaking up on me. I didn't even realize what was happening when I first started seeing my useful working hours cut back, and then back further. When ten hours of useful thought became eight, became six, became four.

When I finally figured out what was happening, I was already crispy around the edges. I took steps to mitigate it - joined a co-working space, made an effort to talk more to people who were advising me - but I wasn't able to recover enough of my personal momentum to justify continuing full time. Putting AdvertiSin back on the shelf, if only just for now, wasn't an easy decision to make, but I think it was the right one.

I hope that other founders learn from my lesson. When you are looking at your startup, don't just look at the business, but look at yourself. Startups are a wild ride, and going it alone makes the peaks and falls that much harder on you. Take breaks, and keep up your social life. The business can't survive without you, but it will consume you if it can.

wdiff

Mon, 09 Mar 2015 00:00:00 -0400

This week, while helping out the DebConf 15 team with some proofreading, I encountered a kind of diff I've never worked with before. A wdiff, or word diff, is intended for displaying word differences between text files. In the context where you are editing free-form text, rather than something like code, it makes a lot more sense than a more traditional unified or copied diff.

So far, so good! But, unfortunately, neither I nor the people who I was working with were able to find any tool out there that actually lets you apply a wdiff patch to a file - or even to blindly discard or accept all the changes inside a wdiff file.

As with most problems, it's nothing a sufficiently gross application of Perl can't solve. In case anyone in the future runs into this problem, I'm including the little Perl one-liner that will take a wdiff file on stdin and output the document with all changes applied. It's not pretty, and I haven't tested it that thoroughly, so use it with caution.

perl -ne '$_ =~  s/ ?\[-(?:[^\]]+)-\]//g; $_ =~ s/ ?\{\+([^\}]+)\+\}/ $1/g; print $_'

DebConf Retrospective

Mon, 01 Sep 2014 00:00:00 -0400

Once more, back in the air, just a bit beneath the route that I took on my way out to Portland. The last nine days have been hectic and, in many ways, very different than I was expecting. In other ways, though, it was exactly what I expected. To-do lists definitely breed, that's for sure.

I've been to many other tech conferences before, but this didn't have nearly the same feel to it. It was more than simply a bunch of engineers with some common interest gathering together; as cliché as it is to write, it felt more like a gathering of friends. Sure, there were talks aplenty, and those were familiar enough. Perhaps it was just the regularity - the number of years that some of the other conferencegoers had been attending different DebConfs all over the world.

I think leaving it at that, though, would be a disservice; it's not the first time I've thought that Debian was one of the most welcoming communities that I've been a part of in the free software movement. I certainly made new friends, and even as I fly back home, I'm eagerly looking forward to the next DebConf in Heidelberg.

Emotional sappiness through, let's turn talk to the tech. The most significant thing that I worked on while at DebConf in terms of technical complexity, if not impact, was the debugging of a build failure in NanoMsg::Raw, thanks in a very large part to help from my friend olasd - Nicolas Dandrimont. The build failure wasn't an easy one for me to understand - it was a segfault coming from somewhere inside either the XS, inside the underlying library, or as some race condition somewhere in between. Adding to the complexity was the fact that the NanoMsg::Raw package already had some test failures that were false positives, and the underlying library with its own set of test suite difficulties.

We finally narrowed the intermittent failure down to an uninitialized variable in the XS code in NanoMsg::Raw that was causing two of the builds to fail - the primary cause of the bug. The fix ended up being two lines - initializing the variable and switching from an mPUSHs to an mXPUSHs to make sure there was enough room on the stack. Of course, one thing so often leads to another, and the package still fails to build properly due to another failure, as well as various architecture-specific problems that the upstreams are working on fixing.

That and getting a few moments to upload updated versions of packages for the pkg-perl team was the only things that I had planned on getting done that I had any chance to do, but I found a few other projects that are very interesting. I worked with Raphael Geissert on some changes to the code editor for sources.debian.org. The version currently there, codeeditor, was causing tremendous javascript slowdowns when large amounts of code were loaded into it - almost up to ten seconds, on resource-constrained hardware. Raphael and I tried to dig into the why for several hours, but we ended up with the belief that the problems were structural, not a minor bug, in how the codeeditor system was doing the actual highlighting and formatting. The package was using individualized spans to do it, causing memory to jump into the hundreds of megabytes when large files were opened, as well as the achingly slow render time that drew our attention in the first place.

In the end, we decided to jump ship to another, somewhat similar library: Ace, the editor that Cloud9 develops and the descendant of the Mozilla Skyline project. Integration isn't fully complete yet, but the initial results look great. The performance is fantastic, the quality is excellent, and all in all, the Ace editor seems to be a very well thought out piece of tech. I hope to work with Raphael on finishing the last bits of it off in the next coming weeks, and I hope that you will be able to see the results soon for yourself.

A lot more than that happened, and far more than I could write down here. Even though the conference is barely over, I am eagerly looking forward to seeing everyone again at DebConf 15 in Heidelberg.