Trojitá

Trojitá, a fast Qt IMAP e-mail client, has a shiny new release. A highlight of the 0.7 version is support for OpenPGP ("GPG") and S/MIME ("X.509") encryption -- in a read-only mode for now. Here's a short summary of the most important changes: ... [More] Verification of OpenPGP/GPG/S-MIME/CMS/X.509 signatures and support for decryption of these messages IMAP, MIME, SMTP and general bugfixes GUI tweaks and usability improvements Zooming of e-mail content and improvements for vision-impaired users New set of icons matching the Breeze theme Reworked e-mail header display This release now needs Qt 5 (5.2 or newer, 5.6 is recommended) As usual, the code is available in our git as a "v0.7" tag. You can also download a tarball (GPG signature). Prebuilt binaries for multiple distributions are available via the OBS, and so is a Windows installer. The Trojitá developers [Less]

Are you interested in cryptography, either as a user or as a developer? Read on -- this blogpost talks about some of the UI choices we made, as well as about the technical challenges of working with the existing crypto libraries. The next version ... [More] of Trojitá, a fast e-mail client, will support working with encrypted and signed messages. Thanks to Stephan Platz for implementing this during the Google Summer of Code project. If you are impatient, just install the trojita-nightly package and check it out today. Here's how a signed message looks like in a typical scenario: Some other e-mail clients show a yellow semi-warning icon when showing a message with an unknown or unrecognized key. In my opinion, that isn't a great design choice. If I as an attacker wanted to get rid of the warning, I could just as well sign a faked but unsigned e-mail message. This message is signed by something, so we should probably not make this situation appear less secure than as if the e-mail was not signed at all. (Careful readers might start thinking about maintaining a peristant key association database based on the observed traffic patterns. We are aware of the upstream initiative within the GnuPG project, especially the TOFU, Trust On First Use, trust model. It is a pretty fresh code not available in major distributions yet, but it's definitely something to watch and evaluate in future.) Key management, assigning trust etc. is something which is outside of scope for an e-mail client like Trojitá. We might add some buttons for key retrieval and launching a key management application of your choice, such as Kleopatra, but we are definitely not in the business of "real" key management, cross-signatures, defining trust, etc. What we do instead is working with your system's configuration and showing the results based on whether GnuPG thinks that you trust this signature. That's when we are happy to show a nice green padlock to you: We are also making a bunch of sanity checks when it comes to signatures. For example, it is important to verify that the sender of an e-mail which you are reading has an e-mail which matches the identity of the key holder -- in other words, is the guy who sent the e-mail and the one who made the signature the same person? If not, it would be possible for your co-worker (who you already trust) to write an e-mail message to you with a faked From header pretending to be your boss. The body of a message is signed by your colleague with his valid key, so if you forget to check the e-mail addresses, you are screwed -- and that's why Trojitá handles this for you: In some environments, S/MIME signatures using traditional X.509 certificates are more common than the OpenPGP (aka PGP, aka GPG). Trojitá supports them all just as easily. Here is what happens when we are curious and decide to drill down to details about the certificate chain: Encrypted messages are of course supported, too: We had to start somewhere, so right now, Trojitá supports only read-only operations such as signature verification and decrypting of messages. It is not yet possible to sign and encrypt new messages; that's something which will be implemented in near future (and patches are welcome for sure). Technical details Originally, we were planning to use the QCA2 library because it provides a stand-alone Qt wrapper over a pluggable set of cryptography backends. The API interface was very convenient for a Qt application such as Trojitá, with native support for Qt's signals/slots and asynchronous operation implemented in a background thread. However, it turned out that its support for GnuPG, a free-software implementation of the OpenPGP protocol, leaves much to be desired. It does not really support the concept of PGP's Web of Trust, and therefore it doesn't report back how trustworthy the sender is. This means that there woldn't be any green padlock with QCA. The library was also really slow during certain operations -- including retrieval of a single key from a keystore. It just isn't acceptable to wait 16 seconds when verifying a signature, so we had to go looking for something else. Compared to the QCA, the GpgME++ library lives on a lower level. Its Qt integration is limited to working with QByteArray classes as buffers for gpgme's operation. There is some support for integrating with Qt's event loop, but we were warned not to use it because it's apparently deprecated code which will be removed soon. The gpgme library supports some level of asynchronous operation, but it is a bit limited. Ultimately, someone has to do the work and consume the CPU cycles for all the crypto operations and/or at least communication to the GPG Agent in the background. These operations can take a substantial amount of time, so we cannot do that in the GUI thread (unless we wanted to reuse that discouraged event loop integration). We could use the asynchronous operations along with a call to gpgme_wait in a single background thread, but that would require maintaining our own dedicated crypto thread and coming up with a way to dispatch the results of each operation to the original requester. That is certainly doable, but in the end, it was a bit more straightforward to look into the C++11's toolset, and reuse the std::async infrastructure for launching background tasks along with a std::future for synchronization. You can take a look at the resulting code in the src/Cryptography/GpgMe++.cpp. Who can dislike lines like task.wait_for(std::chrono::duration_values? :) Finally, let me provide credit where credit is due. Stephan Platz worked on this feature during his GSoC term, and he implemented the core infrastructure around which the whole feature is built. That was the crucial point and his initial design has survived into the current implementation despite the fact that the crypto backend has changed and a lot of code was refactored. Another big thank you goes to the GnuPG and GpgME developers who provide a nice library which works not just with OpenPGP, but also with the traditional X.509 (S/MIME) certificates. The same has to be said about the developers behind the GpgME++ library which is a C++ wrapper around GpgME with roots in the KDEPIM software stack, and also something which will one day probably move to GpgME proper. The KDE ties are still visible, and Andre Heinecke was kind enough to review our implementation for obvious screwups in how we use it. Thanks! [Less]

Hi all, we are pleased to announce version 0.6 of Trojitá, a fast Qt IMAP e-mail client. This release brings several new features as well as the usual share of bugfixes: Plugin-based infrastructure for the address book, which will allow better ... [More] integration with other applications Usability improvements in the message composer on several fronts Better keyboard-only usability for those of us who do not touch mouse that often More intuitive message tagging, and support for standardized actions for junk mail Optional sharing of authentication data between IMAP and SMTP Change to using Qt5 by default. This is the last release which still supports Qt4. Improved robustness on unstable network connections The old status bar is now gone to save screen real estate IMAP interoperability fixes Speed improvements This release has been tagged in git as "v0.6". You can also download a tarball (GPG signature). Prebuilt binaries for multiple distributions are available via the OBS, and so is a Windows installer. This release is named after the Aegean island Λέσβος (Lesvos). Jan was there for the past five weeks, and he insisted on mentioning this challenging experience. The Trojitá developers [Less]

It is that time of the year again, and people are applying for Google Summer of Code positions. It's great to see a big crowd of newcomers. This article explains what sort of students are welcome in GSoC from the point of view of Trojitá, a fast Qt ... [More] IMAP e-mail client. I suspect that many other projects within KDE share my views, but it's best to ask them. Hopefully, this post will help students understand what we are looking for, and assist in deciding what project to work for. Finding a motivation As a mentor, my motivation in GSoC is pretty simple — I want to attract new contributors to the project I maintain. This means that I value long-term sustainability above fancy features. If you are going to apply with us, make sure that you actually want to stick around. What happens when GSoC terminates? What happens when GSoC terminates and the work you've been doing is not ready yet? Do you see yourself continuing the work you've done so far? Or is it going to become an abandonware, with some cash in your pocket being your only reward? Who is going to maintain the code which you worked hard to create? Selecting an area of work This is probably the most important aspect of your GSoC involvement. You're going to spend three months of full time activity on some project, a project you might have not heard about before. Why are you doing this — is it only about the money, or do you already have a connection to the project you've selected? Is the project trying to solve a problem that you find interesting? Would you use the results of that project even without the GSoC? My experience shows that it's best to find a project which fills a niche that you find interesting. Do you have a digital camera, and do you think that a random photo editor's interface sucks? Work on that, make the interface better. Do you love listening to music? Maybe your favorite music player has some annoying bug that you could fix. Maybe you could add a feature to, say, synchronize the playlist with your cell phone (this is just an example, of course). Do you like 3D printing? Help improve an existing software for 3D printing, then. Are you a database buff? Is there something you find lacking in, e.g., PostgreSQL? Either way, it is probably a good idea to select something which you need to use, or want to use for some reason. It's of course fine to e.g. spend your GSoC term working on an astronomy tool even though you haven't used one before, but unless you really like astronomy, then you should probably choose something else. In case of Trojitá, if you have been using GMail's web interface for the past five years and you think that it's the best thing since sliced bread, well, chances are that you won't enjoy working on a desktop e-mail client. Pick something you like, something which you enjoy working with. Making a proposal An excellent idea is to make yourself known in advance. This does not happen by joining the IRC channel and saying "I want to work on GSoC", or mailing us to let us know about this. A much better way of getting involved is through showing your dedication. Try to play with the application you are about to apply for. Do you see some annoying bug? Fix it! Does it work well? Use the application more; you will find bugs. Look at the project's bug tracker, maybe there are some issues which people are hitting. Do you think that you can fix it? Diving into bug fixing is an excellent opportunity to get yourself familiar with the project's code base, and to make sure that our mentors know the style and pace of your work. Now that you have some familiarity with the code, maybe you can already see opportunities for work besides what's already described on the GSoC ideas wiki page. That's fine — the best proposals usually come from students who have found them on their own. The list of ideas is just that, a list of ideas, not an exhaustive cookbook. There's usually much more what can be done during the course of the GSoC. What would be most interesting area for you? How does it fit into the bigger picture? After you've thought about the area to work on, now it's time to write your proposal. Start early, and make sure that you talk about your ideas with your prospective mentors before you spend three hours preparing a detailed roadmap. Define the goals that you want to achieve, and talk with your mentors about them. Make sure that the work fits well with the length and style of the GSoC. And finally, be sure that you stay open and honest with your mentoring team. Remember, this is not a contest of writing a best project proposal. For me, GSoC is all about finding people who are interested in working on, say, Trojitá. What I'm looking for are honest, fair-behaving people who demonstrate willingness to learn new stuff. On top of that, I like to accept people with whom I have already worked. Hearing about you for the first time when I read your GSoC proposal is not a perfect way of introducing yourself. Make yourself known in advance, and show us how you can help us make our project better. Show us that you want to become a part of that "we". [Less]

Hi all, we are pleased to announce version 0.5 of Trojitá, a fast Qt IMAP e-mail client. More than 500 changes went in since the previous release, so the following list highlights just a few of them: Trojitá can now be invoked with a mailto: URL ... [More] (RFC 6068) on the command line for composing a new email. Messages can be forwarded as attachments (support for inline forwarding is planned). Passwords can be remembered in a secure, encrypted storage via QtKeychain. E-mails with attachments are decorated with a paperclip icon in the overview. Better rendering of e-mails with extraordinary MIME structure. By default, only one instance is kept running, and can be controlled via D-Bus. Trojitá now provides better error reporting, and can reconnect on network failures automatically. The network state (Offline, Expensive Connection or Free Access) will be remembered across sessions. When replying, it is now possible to retroactively change the reply type (Private Reply, Reply to All but Me, Reply to All, Reply to Mailing List, Handpicked). When searching in a message, Trojitá will scroll to the current match. Attachment preview for quick access to the enclosed files. The mark-message-read-after-X-seconds setting is now configurable. The IMAP refresh interval is now configurable. Speed and memory consumption improvements. Miscellaneous IMAP improvements. Various fixes and improvements. We have increased our test coverage, and are now making use of an improved Continuous Integration setup with pre-commit patch testing. This release has been tagged in git as "v0.5". You can also download a tarball (GPG signature). Prebuilt binaries for multiple distributions are available via the OBS, and so is a Windows installer. We would like to thank Karan Luthra and Stephan Platz for their efforts during Google Summer of Code 2014. The Trojitá developers Jan Kundrát Pali Rohár Dan Chapman Thomas Lübking Stephan Platz Boren Zhang Karan Luthra Caspar Schutijser Lasse Liehu Michael Hall Toby Chen Niklas Wenzel Marko Käning Bruno Meneguele Yuri Chornoivan Tomáš Chvátal Thor Nuno Helge Gomes Hultberg Safa Alfulaij Pavel Sedlák Matthias Klumpp Luke Dashjr Jai Luthra Illya Kovalevskyy Edward Hades Dimitrios Glentadakis Andreas Sturmlechner Alexander Zabolotskikh [Less]

Some of the recent releases of Trojitá, a fast Qt e-mail client, mentioned an ongoing work towards bringing the application to the Ubuntu Touch platform. It turns out that this won't be happening. The developers who were working on the Ubuntu ... [More] Touch UI decided that they would prefer to end working with upstream and instead focus on a standalone long-term fork of Trojitá called Dekko. The fork lives within the Launchpad ecosystem and we agreed that there's no point in keeping unmaintained and dead code in our repository anymore -- hence it's being removed. [Less]

Some of the recent releases of Trojitá, a fast Qt e-mail client, mentioned an ongoing work towards bringing the application to the Ubuntu Touch platform. It turns out that this won't be happening. The developers who were working on the Ubuntu ... [More] Touch UI decided that they would prefer to end working with upstream and instead focus on a standalone long-term fork of Trojitá called Dekko. The fork lives within the Launchpad ecosystem and we agreed that there's no point in keeping unmaintained and dead code in our repository anymore -- hence it's being removed. [Less]

One of the improvements which were mentioned in the recent announcement of Trojitá, a fast Qt e-mail client, were substantial memory savings and speed improvements. In the rest of this post, I would like to explain what exactly we have done and how ... [More] it matters. This is going to be a technical post, so if you are not interested in C++ or software engineering, you might want to skip this article. Planting Trees At the core of Trojitá's IMAP implementation is the TreeItem, an abstract class whose basic layout will be familiar to anyone who has worked with a custom QAbstractItemModel reimplementation. In short, the purpose of this class is to serve as a node in the tree of items which represent all the data stored on a remote IMAP server. The structure is tree-shaped because that's what fits both the QAbstractItemModel's and the IMAP way of working. At the top, there's a list of mailboxes. Children of these mailboxes are either other, nested mailboxes, or lists of messages. Below the lists of messages, one can find individual e-mails, and within these e-mails, individual body parts as per the recursive nature of the MIME encapsulation. (This is what enables messages with pictures attached, e-mail forwarding, and similar stuff. MIME is fun.) This tree of items is used by the QAbstractItemModel for keeping track of what is where, and for issuing the QModelIndex instances which are used by the rest of the application for accessing, requesting and manipulating the data. When a QModelIndex is used and passed to the IMAP Model, what matters most is its internalPointer(), a void * which, within Trojitá, always points to an instance of some TreeItem subclass. Everything else, like the row() and column(), are actually not important; the pointer itself is enough to determine everything about the index in question. Each TreeItem has to store a couple of interesting properties. Besides the usual Qt-mandated stuff like pointer to the parent item and a list of children, there are also application-specific items which enable the code to, well, actually do useful things like printing e-mail subjects or downloading mail attachments. For a mailbox, this crucial information might be the mailbox name. For a message, the UID of the message along with a pointer to the mailbox is enough to uniquely identify everything which is needed. Lazy Loading Enter the lazy loading. Many people confirm that Trojitá is fast, and plenty of them are not afraid to say that it is blazingly fast. This speed is enabled by the fact that Trojitá will only do the smallest amount of work required to bring the data over the network (or from disk, for that matter). If you open a huge mailbox with half a million messages, perhaps the GMail's "All messages" account, or one's LKML archive, Trojitá will not start loading half a million of subjects. Instead, the in-memory TreeItem nodes are created in a special state "no data has been requested yet". Trojitá still creates half a million items in memory, but these items are rather lightweight and only contain the absolute minimum of data they need for proper operation. Some of these "empty" nodes are, eventually, consulted and used for item display -- perhaps because a view is attached to this model, and the view wants to show the recent mail to the user. In Qt, this usually happens via the data() method of the QAbstractItemModel, but other methods like rowCount() have a very similar effect. Whenever more data are needed, the state of the tree node changes from the initial "no data have been requested" to "loading stuff", and an asynchronous request for these data is dispatched. An important part of the tale is that the request is indeed completely asynchronous, so you won't see any blocking whatsoever in the GUI. The QTreeView will show an animation while a subtree is expanded, the message viewer might display a spinner, and the mail listing shows greyed-out "Loading..." placeholder instead of the usual message subjects. After a short while, the data arrive and the tree node is updated with the extracted contents -- be it e-mail subject, or perhaps the attached image of dancing pigs. As the requested data are now here, the status of the tree node is updated from the previous "loading stuff" into "done". At the same time, an appropriate signal, like dataChanged or rowsInserted, is emitted. Requesting the same data again via the classic MVC API will not result in network requests, but everything will be accommodated from the local cache. What we see now is that there is just a handful of item states, yet the typical layout of the TreeItem looks roughly like this: enum class FetchingStatus { INITIAL_NOTHING_REQUESTED_YET, LOADING, DONE, FAILED }; class TreeItem { TreeItem *m_parent; QList m_children; FetchingStatus m_status; }; On a 64bit system, this translates to at least three 64bit words being used -- one for the painter to the parent item, one (or much more) for storage of the list of children, and one more for storing the enum FetchingStatus. That's a lot of space, given we have just created half a million of these items. Tagged Pointers An interesting property of a modern CPU is that the data structures must be aligned properly. A very common rule is that e.g. a 32bit integer can only start at memory offset which is a multiple of four. In hex, this means that an address, or a pointer value, could end with 0x0, or 0x4, or 0x8, or 0xc. The detailed rules are platform-specific and depend on the exact data structure which we are pointing to, but the important message is that at least some of the low bits in the pointer address are always going to be zero. Perhaps we could encode some information in there? Turns out this is exactly what pointer tagging is about. Instead of having two members, one TreeItem * and one FetchingStatus, these are squashed into a single pointer-sized value. The CPU can no longer use the pointer value directly, all accesses have to go via an inlined function which simply masks away the lowest bits which do bring a very minor performance hit, but the memory conservation is real. For a real-world example, see this commit in Trojitá. Using Memory Only When Needed Back to our example of a mailbox with 500k messages. Surely a user is only going to see a small subset of them at once, right? That is indeed the case. We still have to at least reserve space for 500k items for technical reasons, but there is certainly no need to reserve space for heavy stuff like subjects and other headers. Indeed, in Trojitá, we track the From/To/Cc/Bcc headers, the subjects, various kinds of timestamps, other envelope items and similar stuff, and this totals a couple hundred bytes per each message. A couple hundred bytes is not much (pun intended), but "a couple hundred bytes" times "half a million" is a ton of memory. This got implemented here. One particular benchmark which tests how fast Trojitá resynchronizes a mailbox with 100k of messages showed immediate reduction in memory usage from previous 45 MB to 25 MB. The change, again, does come with a cost; one now has to follow one more pointer redirection, and one has to perform one more dynamic allocation for each message which is actually visible. That, however, proves to be negligible during typical usage. Measure, Don't Guess As usual with optimizing, the real results might sometimes be surprising. A careful reader and an experienced Qt programmer might have noticed the QList above and shuddered in horror. In fact, Trojitá now uses QVector in its place, but when I was changing the code, using std::vector sounded like a no-brainer. Who needs the copy-on-write semantics here anyway, so why should I pay its price in this context? These data (list of children of an item) are not copied that often, and copying a contiguous list of pointers is pretty cheap anyway (it surely is dwarfed by dynamic allocation overhead). So we should just stick with std::vector, right? Well, not really. It turned out that plenty of these lists are empty most of the time. If we are looking at the list of messages in our huge mailbox, chances are that most of these messages were not loaded yet, and therefore the list of children, i.e. something which represents their inner MIME structure, is likely empty. This is where the QVector really shines. Instead of using three pointers per vector, like the GCC's std::vector does, QVector is happy with a single pointer pointing to a shared null instance, something which is empty. Now, factor of three on an item which is used half a million times, this is something which is going to hurt. That's why Trojitá eventually settled on using QVector for the m_children member. The important lesson here is "don't assume, measure". Wrapping up Thanks to these optimization (and a couple more, see the git log), one particular test case now runs ten times faster while simultaneously using 38% less memory -- comparing the v0.4 with v0.3.96. Trojitá was pretty fast even before, but now it really flies. The sources of memory diet were described in today's blog post; the explanation on how the time was cut is something which will have to wait for another day. [Less]

One of the improvements which were mentioned in the recent announcement of Trojitá, a fast Qt e-mail client, were substantial memory savings and speed improvements. In the rest of this post, I would like to explain what exactly we have done and how ... [More] it matters. This is going to be a technical post, so if you are not interested in C++ or software engineering, you might want to skip this article. Planting Trees At the core of Trojitá's IMAP implementation is the TreeItem, an abstract class whose basic layout will be familiar to anyone who has worked with a custom QAbstractItemModel reimplementation. In short, the purpose of this class is to serve as a node in the tree of items which represent all the data stored on a remote IMAP server. The structure is tree-shaped because that's what fits both the QAbstractItemModel's and the IMAP way of working. At the top, there's a list of mailboxes. Children of these mailboxes are either other, nested mailboxes, or lists of messages. Below the lists of messages, one can find individual e-mails, and within these e-mails, individual body parts as per the recursive nature of the MIME encapsulation. (This is what enables messages with pictures attached, e-mail forwarding, and similar stuff. MIME is fun.) This tree of items is used by the QAbstractItemModel for keeping track of what is where, and for issuing the QModelIndex instances which are used by the rest of the application for accessing, requesting and manipulating the data. When a QModelIndex is used and passed to the IMAP Model, what matters most is its internalPointer(), a void * which, within Trojitá, always points to an instance of some TreeItem subclass. Everything else, like the row() and column(), are actually not important; the pointer itself is enough to determine everything about the index in question. Each TreeItem has to store a couple of interesting properties. Besides the usual Qt-mandated stuff like pointer to the parent item and a list of children, there are also application-specific items which enable the code to, well, actually do useful things like printing e-mail subjects or downloading mail attachments. For a mailbox, this crucial information might be the mailbox name. For a message, the UID of the message along with a pointer to the mailbox is enough to uniquely identify everything which is needed. Lazy Loading Enter the lazy loading. Many people confirm that Trojitá is fast, and plenty of them are not afraid to say that it is blazingly fast. This speed is enabled by the fact that Trojitá will only do the smallest amount of work required to bring the data over the network (or from disk, for that matter). If you open a huge mailbox with half a million messages, perhaps the GMail's "All messages" account, or one's LKML archive, Trojitá will not start loading half a million of subjects. Instead, the in-memory TreeItem nodes are created in a special state "no data has been requested yet". Trojitá still creates half a million items in memory, but these items are rather lightweight and only contain the absolute minimum of data they need for proper operation. Some of these "empty" nodes are, eventually, consulted and used for item display -- perhaps because a view is attached to this model, and the view wants to show the recent mail to the user. In Qt, this usually happens via the data() method of the QAbstractItemModel, but other methods like rowCount() have a very similar effect. Whenever more data are needed, the state of the tree node changes from the initial "no data have been requested" to "loading stuff", and an asynchronous request for these data is dispatched. An important part of the tale is that the request is indeed completely asynchronous, so you won't see any blocking whatsoever in the GUI. The QTreeView will show an animation while a subtree is expanded, the message viewer might display a spinner, and the mail listing shows greyed-out "Loading..." placeholder instead of the usual message subjects. After a short while, the data arrive and the tree node is updated with the extracted contents -- be it e-mail subject, or perhaps the attached image of dancing pigs. As the requested data are now here, the status of the tree node is updated from the previous "loading stuff" into "done". At the same time, an appropriate signal, like dataChanged or rowsInserted, is emitted. Requesting the same data again via the classic MVC API will not result in network requests, but everything will be accommodated from the local cache. What we see now is that there is just a handful of item states, yet the typical layout of the TreeItem looks roughly like this: enum class FetchingStatus { INITIAL_NOTHING_REQUESTED_YET, LOADING, DONE, FAILED }; class TreeItem { TreeItem *m_parent; QList<TreeItem*> m_children; FetchingStatus m_status; }; On a 64bit system, this translates to at least three 64bit words being used -- one for the painter to the parent item, one (or much more) for storage of the list of children, and one more for storing the enum FetchingStatus. That's a lot of space, given we have just created half a million of these items. Tagged Pointers An interesting property of a modern CPU is that the data structures must be aligned properly. A very common rule is that e.g. a 32bit integer can only start at memory offset which is a multiple of four. In hex, this means that an address, or a pointer value, could end with 0x0, or 0x4, or 0x8, or 0xc. The detailed rules are platform-specific and depend on the exact data structure which we are pointing to, but the important message is that at least some of the low bits in the pointer address are always going to be zero. Perhaps we could encode some information in there? Turns out this is exactly what pointer tagging is about. Instead of having two members, one TreeItem * and one FetchingStatus, these are squashed into a single pointer-sized value. The CPU can no longer use the pointer value directly, all accesses have to go via an inlined function which simply masks away the lowest bits which do bring a very minor performance hit, but the memory conservation is real. For a real-world example, see this commit in Trojitá. Using Memory Only When Needed Back to our example of a mailbox with 500k messages. Surely a user is only going to see a small subset of them at once, right? That is indeed the case. We still have to at least reserve space for 500k items for technical reasons, but there is certainly no need to reserve space for heavy stuff like subjects and other headers. Indeed, in Trojitá, we track the From/To/Cc/Bcc headers, the subjects, various kinds of timestamps, other envelope items and similar stuff, and this totals a couple hundred bytes per each message. A couple hundred bytes is not much (pun intended), but "a couple hundred bytes" times "half a million" is a ton of memory. This got implemented here. One particular benchmark which tests how fast Trojitá resynchronizes a mailbox with 100k of messages showed immediate reduction in memory usage from previous 45 MB to 25 MB. The change, again, does come with a cost; one now has to follow one more pointer redirection, and one has to perform one more dynamic allocation for each message which is actually visible. That, however, proves to be negligible during typical usage. Measure, Don't Guess As usual with optimizing, the real results might sometimes be surprising. A careful reader and an experienced Qt programmer might have noticed the QList above and shuddered in horror. In fact, Trojitá now uses QVector in its place, but when I was changing the code, using std::vector sounded like a no-brainer. Who needs the copy-on-write semantics here anyway, so why should I pay its price in this context? These data (list of children of an item) are not copied that often, and copying a contiguous list of pointers is pretty cheap anyway (it surely is dwarfed by dynamic allocation overhead). So we should just stick with std::vector, right? Well, not really. It turned out that plenty of these lists are empty most of the time. If we are looking at the list of messages in our huge mailbox, chances are that most of these messages were not loaded yet, and therefore the list of children, i.e. something which represents their inner MIME structure, is likely empty. This is where the QVector really shines. Instead of using three pointers per vector, like the GCC's std::vector does, QVector is happy with a single pointer pointing to a shared null instance, something which is empty. Now, factor of three on an item which is used half a million times, this is something which is going to hurt. That's why Trojitá eventually settled on using QVector for the m_children member. The important lesson here is "don't assume, measure". Wrapping up Thanks to these optimization (and a couple more, see the git log), one particular test case now runs ten times faster while simultaneously using 38% less memory -- comparing the v0.4 with v0.3.96. Trojitá was pretty fast even before, but now it really flies. The sources of memory diet were described in today's blog post; the explanation on how the time was cut is something which will have to wait for another day. [Less]

Summary An SSL stripping vulnerability was discovered in Trojitá, a fast Qt IMAP e-mail client. User's credentials are never leaked, but if a user tries to send an e-mail, the automatic saving into the "sent" or "draft" folders could happen over a ... [More] plaintext connection even if the user's preferences specify STARTTLS as a requirement. Background The IMAP protocol defines the STARTTLS command which is used to transparently upgrade a plaintext connection to an encrypted one using SSL/TLS. The STARTTLS command can only be issued in an unauthenticated state as per the IMAP's state machine. RFC 3501 also allows for a possibility of the connection jumping immediately into an authenticated state via the PREAUTH initial response. However, as the STARTTLS command cannot be issued once in the authenticated state, an attacker able to intercept and modify the network communication might trick the client into a state where the connection cannot be encrypted anymore. Affected versions All versions of Trojitá up to 0.4 are vulnerable. The fix is included in version 0.4.1. Remedies Connections which use the SSL/TLS form the very beginning (e.g. the connections using port 993) are secure and not vulnerable. Possible impact The user's credentials will never be transmitted over a plaintext connection even in presence of this attack. Because Trojitá proceeded to use the connection without STARTTLS in face of PREAUTH, certain data might be leaked to the attacker. The only example which we were able to identify is the full content of a message which the user attempts to save to their "Sent" folder while trying to send a mail. We don't believe that any other data could be leaked. Again, user's credentials will not be leaked. Acknowledgement Thanks to Arnt Gulbrandsen on the imap-protocol ML for asking what happens when we're configured to request STARTTLS and a PREAUTH is received, and to Michael M Slusarz for starting that discussion. [Less]