tag:blogger.com,1999:blog-75589582024-03-19T14:54:32.939+03:00Vladimir PrusVladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.comBlogger48125tag:blogger.com,1999:blog-7558958.post-73148366725553568572016-05-31T21:59:00.000+03:002016-05-31T21:59:10.806+03:00Desktops and Startups<p>I spent a good part of 2015 working on a desktop app for a startup, not a typical combination these days. If you're building a mobile app, there are multiple companies offering platforms for every task you need, and a of recommendations which platform to choose. On desktop, you have to build or pick everything yourself. In this post, I'll share some advice.</p><h2 id="ui-framework">UI framework</h2><p>What does one use to create cross-platform desktop applications these days? Let’s first look at standard options at each platform.</p><p>On Windows, there are two modern technologies:</p><ul><li><p>XAML, .NET and C#. Good stack if you like C#. In our case, we have a shell extention too, which per Microsoft guidelines should not be written with managed code, but doing that one part in C++ is not very hard.</p></li>
<li><p>XAML, Windows Runtime and C++/CX. This is newer technology, apparently coming from the part of Microsoft that prefers C++. The C++/CX language is native, with no CLR runtime involved, and is pretty close to standard C++, and is therefore attractive if you're a mostly C++ engineer. On the downside, you're limited to Windows 8.1 and later. Also, the application will be a Metro-style one—while there we announcements that desktop apps will be possible soon, that did not happen yet. Shell extension or tray icon will likely be a big problem.</p></li>
</ul><p>On OSX, there’s only one native stack, although you have the choice between Objective C (not a language I want to write core logic in, ever) and Swift (not a language I want to write core logic in, yet).</p><p>Two technologies attempt to be fully cross-platform:</p><ul><li><p>Qt and C++ (using either Qt Widgets or Qt Quick for UI). Works, in theory, on all Windows versions and OSX versions in existence, with Qt Widgets extemely stable and Qt Quick maturing.</p></li>
<li><p>Xamarin and C# (using Xamarin.Forms and XAML). Also a nice stack, and compiles to native code, but requires Windows 8.1 or later.</p></li>
</ul><p>Given the above, what do you do if you definitely want to reuse core logic level? Since we wanted to support Windows version prior to 8.1, this rules out C#, and we actually already had ths level written in C++ with Qt. And while putting XAML UI on Windows and Objective-C UI on OSX on top of C++ core would be possible, that would add fair number of incidental complexity. Therefore, I went with the Qt option, and since the UI was not going to be extremly complicated, decided to give Qt Quick a try.</p><p>Of course, this was a tradeoff between development speed and native look. I have originally produced Windows version, and then creating first OSX version took about a day—clearly good at a startup. On top of that, any interface changes were immediately available for both platforms, just a rebuild away. The UI did not look <em>exactly</em> native on OSX, but close enough for most people, and on Windows, there’s already enough of style variety that nobody had any concerns.</p><p>On the negative side, it turned that Qt Quick is not as mature as I’ve hoped. I have covered this in detail in the <a href="http://blog.vladimirprus.com/2016/05/qt-quick-on-desktop.html">previous post</a>.</p><p><em>Conclusion: Qt with Qt Quick is fine if you have little UI and is already using C++. Xamarin might be a viable option as well</em></p><h2 id="installer">Installer</h2><p>We wanted the installation and update to be as transparent as possible, with no questions asked. On OSX, using DMG for initial installation is the standard solution, and autoupdate can be handled by a library called <a href="https://sparkle-project.org/">Sparkle</a>.</p><p>On Windows, Microsoft documents <a href="https://msdn.microsoft.com/en-us/library/zebw5zk9.aspx">two options</a>. Windows Installer (MSI) is the standard technology, but does not support autoupdate, and any custom solutions has to display prompt dialogs. ClickOnce is a newer technology for .NET applications, supporting installing into per-user location without touching any system directory and transparent updates. Sadly, it does not appear to support native C++ applications, only C++ applications built for CLR. I’ve ended up writing a custom installer from scratch.</p><p>The installer downloads an update manifest (in the Sparkle format), then downloads an archive for the most recent version, checks signature, unpacks to a subdirectory of users <code>AppData</code> folder, and run the binary from there. Auto-update works much the same—detect that update manifest has changed, and start the installer again. When the new version of the application starts, it uses IPC to ask any previous version to shut down.</p><p>This approach works suprisingly well—most installation attempts succeed, and most auto-update attempts do indeed update. It is interesting to note how diverse environments are over the world. For example, downloading 20 megabytes is almost instance for me, but it takes minutes, and sometimes several tries, for users in other locations. Adopting binary deltas, like Chrome does, sounds a good idea.</p><p><em>Conclusion: on Windows, custom installer is a viable option for a native binary that does not modify system, and on OSX, Sparkle works out of the box.</em></p><h2 id="signing">Signing</h2><p>I first run into Windows binary signing at Mentor, when we were building our <a href="http://blog.vladimirprus.com/2014/03/eclipse-p2-product-installer.html">P2 installer</a>, and at one point, the IT-mandated anti-virus software started deleting our own binaries. It would only stop when we got a code signing certificate. This time, couple years later, we had as much as three obstacles. First, Chrome would flag our installer as risky on download. Second, Windows SafeFilter would refuse to run it. Finallay, finally the antivirus software would randomly wake up and break things. Naturally, we had to get code signing certificate too.</p><p>The code signing certificate involves organization validation. Not much of a problem for a large company whose address can be validated by Google maps. But a small startup with no landline, no office number on the door, and no utility bills is deemed suspect by the registars. It took a while for our CEO to build appropriate chain of trust.</p><p>I would say the whole system is quite pointless. Surely, organization validation makes it possible to pass an address and a phone number to police, in case good guys become bad overnight. However, it does absolutely nothing to protect against good-willing but incompetent guys — for example who leak signing keys, build on virus-infected machines or do not secure auto-update. And it does nothing to protect against really-bad guys, who surely can fake addresses and phone numbers.</p><p><em>Conclusion: if you plan to publish windows applications, get a code signing certificate as soon as possible.</em></p><h2 id="crash-reporting">Crash Reporting</h2><p>Crash reporting is part of Winwdows for a while, but by default, only Microsoft gets to look at crashes. One can apply for a developer account with permission to look at your own crashes, but the instructions start with ‘get extended validation certificate from one of these 3 CA’. Extended validation is even more messy than organization validation, and Microsoft could well not approve us anyway, so I looked at other options.</p><p>Chrome includes a library called <a href="https://chromium.googlesource.com/breakpad/breakpad/">Breakpad</a> and while they have no server-side, Mozilla fills the void with <a href="https://wiki.mozilla.org/Socorro">Socorro</a>. The setup could be easiert—in particular there are no official Ubuntu packages, one should use specific deployment system, and it requires ELK stack. However, after a few hours I could get the first test crash reported, and was going to start using it for real when I came across <a href="http://drdump.com">DrDump</a>.</p><p>That service provides a library to handle crashes, including UI, cloud backend to store crashes, and a web app to review the crashes. On a crash, minidump is sent automatically and user is prompted to submit a full dump, unless one was previously collected for the same stacktrace. Overall it took about an hour to integrate and play with, and it worked quite satisfactory since then. It was surprising, when asked about pricing, to hear that the service is free—reportedly because with matching crash stacktraces, the disk storage required for full dumps is quite low. I doubt it would be still free for an app with million users each hitting unique bugs, but it remains free for us.</p><p><em>Conclusion: DrDump is a fine solution for an early stage startup. Breakpad and Soccorro will be also fine when you have time for devops.</em></p><h2 id="analytics">Analytics</h2><p>The mechanics of collecting analytics is easy. We use Mixpanel for mobile analytics and wanted to use for desktop apps as well. On mobile, there’s SDK that sends events, collects system properties and manages event queue. There’s no such option for desktop, but there is a REST API, which is sufficient. Adding basic system details like OS version is easy, and we did not bother implementing event queue. Surely some events do get lost, but it is not a big practical problem.</p><p>With the data in, Mixpanel makes it very easy to show a chart of event counts, filtered and segmented as you wish. Whether it is useful is not clear.</p><ul><li><p>No chart comes with any statistical analysis. Want to check whether Windows 7 users really have lower conversion rate, and you need to do the math using other tools.</p></li>
<li><p>Box plot, the standard way to look at a relation between a continuous and a categorical variable, is nowhere to be seen.</p></li>
<li><p>You can only look at 90 days of data.</p></li>
</ul><p>In essence, given that drawing basic charts is frictionless while statistical analysis demands external tools, Mixpanel lures you into making simplistic analysis. In the end, I have just exported everything and looked at data in R.</p><p><em>Conclusion: you likely don’t need any analytics services. It’s easy to send data into a database, aggregate it, and analyze using a decent statistical package.</em></p><h2 id="summing-it-up">Summing it up</h2><p>While there's no compherehensive platform that helps to quickly launch a desktop application, there are enough pieces of technology that can be put together. The most important I've learned are:</p><ul><li><p>Pick a cross-platform UI framework; you probably don’t have time to write separate code for different operating systems. I found Qt to work OK, but you might want to also consider Xamarin.</p></li>
<li><p>Get code signing certificate early</p></li>
<li><p>Do your own analytics. Decide what will be most important when you launch, figure what statistical tools you’d need, and create events to support that.</p></li>
</ul>Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com0tag:blogger.com,1999:blog-7558958.post-84130059896091748222016-05-19T11:08:00.000+03:002016-05-19T16:11:36.282+03:00Qt Quick on Desktop<p>I worked with Qt quite a bit over the years, but it was only in 2015 where I had a chance to do substantial work with Qt Quick. I wanted to share some impressions, specifically for desktop applications.</p><h2 id="summary">Summary</h2><p>Here is an advance summary of the key points, starting with advantages:</p><ul><li>Qt Quick is now a mature way to build desktop application that either use standard-looking desktop controls, or have relatively simple custom UI.</li>
<li>Data binding is quite pleasant in all ways. However, it's only limited to binding UI properties to expressions over model properties. Dynamically changing UI structure is possible, but is rather convoluted.</li>
<li>Animation system is solid and support for GL effects is much more convenient than writing GL directly.</li>
<li>For a pleasant surprise, it has a state machine built in, thought with some quirks.</li>
</ul><p>But not everything is perfect:</p><ul><li>The set of standard controls and styles could be larger. If you wish to achieve Metro design or Material design on desktop, you might need to use third-party extensions or do it yourself.</li>
<li>Styling merchanisms (in Qt Quick Controls 1) are quite limited, having neither inheritance nor attributes, and Qt Quick Controls 2 have something else entirely.</li>
<li>There are two different layout mechanisms, each with its own quirks.</li>
<li>As of Qt 5.5, High DPI support involved doing the math yourself. This might have improved since.</li>
<li>Open GL is required, and especially on Windows, the set of possible GL configuration is large, the documentation is imperfect, and there are "interesting" differences in behaviour.</li>
</ul><h2 id="what-is-qt-quick">What is Qt Quick?</h2><p>Let's clarify some terminology first:</p><ul><li>QML is a language that defines a tree of objects, along with some property bindings and executable code. It uses custom language for the tree proper, and Javascript for expressions and functions</li>
<li>Qt Quick is a set of basic visual components, and an engine to render them</li>
<li>Qt Quick Controls is a set of standard UI controls and layouts.</li>
</ul><p>These are pretty much always used together, so I'll use "Qt Quick" throughout regardless of what layer I really talk about.</p><h2 id="qml-and-qt-quick">QML and Qt Quick</h2><p>In order to illustrate how Qt Quick UI is put together, I'll use a simple part of registration UI, shown below</p><p style="text-align:center"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7DSap7WoFynGrRjjDp1R2y08C4EzDJDN5-u-9m1x2r11o2H9j0O7svqejrnaHrHN75sm2V3GGx1HqXP8Vjq-Xxc46JdBrI-G1fBVs1covsOaqsoQIAyqtAfKFjX3yWFYAAU08/s1600/registration.gif" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7DSap7WoFynGrRjjDp1R2y08C4EzDJDN5-u-9m1x2r11o2H9j0O7svqejrnaHrHN75sm2V3GGx1HqXP8Vjq-Xxc46JdBrI-G1fBVs1covsOaqsoQIAyqtAfKFjX3yWFYAAU08/s400/registration.gif" /></a></p><p>As you start typing a phone number a decorative line under the input fields turns into progress bar, and as you submit the form, the progress bar becomes indefinite one, and when error is returned, the hint below the input fields becomes an error display.</p><p>The progress bar is a custom component that I won't discuss in detail, but once but once written, it can be easily used:</p><pre><code>CustomInput {
id: phoneNumber
}
CustomPercentageLine {
percentage: model.validPhoneLength == -1 ? 1 : phoneNumber.text.length/model.validPhoneLength
animated: model.working
}</code></pre><p>Here, <code>model</code> is an instance of C++ class that we've injected into QML. The binding expression checks whether a phone has fixed size, and if so, computes progress bar percentage. And if model is busy validating phone number, and its <code>working</code> property is true, the <code>animated</code> property of the view is also updated, causing progress bar to show indefinite state. Similarly, this is how hint and error display is implemented</p><pre><code>CustomLabel {
text: model.error ? model.error : "We'll confirm your number by sending a one-time SMS"
}</code></pre><p>Data binding is certainly good, and QML offers certain syntactic convenience, as we can use JavaScript expressions with no escaping - unlike XML-based templating engines. This gets particularly important for larger expressions, like the one below:</p><pre><code>CustomLabel {
text: {
if (model.error) { return model.error; }
if (model.state === "resendingPin") {
return "Enter the previous code or wait for a new code to be sent";
} else {
return "Enter the code";
}
}
color: model.error ? "#ce4844" : "#999ba4"
}</code></pre><p>The <code>animated</code> property of my custom progress bar is used to indicate indefinite progress, and is shown by three rectangles moving across the blue progress line. Making the rectangles move is easy with the animation framework, for example there's the animation definition for the first rectangle:</p><pre><code>NumberAnimation {
id: animation1
target: rectangle1
property: "x"
duration: 2000
easing.type: Easing.OutCubic
loops: Animation.Infinite
from: -6
to: parent.width
}</code></pre><p>Animation for the second and third rectangles is similar, but they should start with a delay, so we need to use a separate timer item:</p><pre><code>Timer {
id: timer2
interval: 500
onTriggered: animation2.start()
repeat: false
}</code></pre><p>As soon as the <code>animated</code> property of progress bar is set to true, we start animating the first rectangle, as well as the timers that will start animating two others:</p><pre><code>animation1.start();
timer2.start();
timer3.start();</code></pre><p>It would be more convenient if animation supported delayed start without auxilliary timers, but with exception of that, the mechanism is OK.</p><p>For managing larger UI changes, the state machine framework comes handy. For example, upon startup we show a spalsh screen and attempt to connect to backend server and check for authorization, showing the login screen only if necessary. If we can't connect quickly, we'll show a separate screen with progress bar. Here's the relevant code:</p><pre><code>DSM.State {
id: initial
DSM.TimeoutTransition {
targetState: waiting
timeout: 10000
}
DSM.SignalTransition {
signal: model.connectedChanged
guard: model.connected && !model.isAuthenticated()
targetState: needPhone
}
}</code></pre><p>It illustrates two features. First, it's possible to transition to a different state simply after a timeout. Second, it's possible to transition to a different state on a signal, if a particular condition is met. However the second example also shows one of the largest inconveniences: you can't easily transition when a particular expression becomes true. I'd much rather not bother thinking what signals are emitted, and write this:</p><pre><code>DSM.SignalTransition {
guard: model.connected && !model.isAuthenticated()
targetState: needPhone
}</code></pre><p>After all, tracking changes to expression already works for properties. There is a workaround to get this effect with auxilliary items, but it ought to be a standard feature. Still, having state machines as standard feature much simplifies UI logic.</p><h2 id="controls-styling">Controls Styling</h2><p>I'll start the critical part of my post with controls and styling. The below screenshots shows the controls gallery example with the two styles available to Windows desktop application in Qt 5.5.</p><p style="text-align:center"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY8ZCYKeiOd6CtDu-r_Vldkcy44L_B0YPIYQLVVEOiWI3tAJ0kOirX7DNfwprMC3iFyPj4hO5_2VeEni_jdk8UF_z_LdWDp5cq5-amXh6vbEqn_VozOT-perjBMIUAz1XuWH6N/s1600/qt-quick-control-styles.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY8ZCYKeiOd6CtDu-r_Vldkcy44L_B0YPIYQLVVEOiWI3tAJ0kOirX7DNfwprMC3iFyPj4hO5_2VeEni_jdk8UF_z_LdWDp5cq5-amXh6vbEqn_VozOT-perjBMIUAz1XuWH6N/s640/qt-quick-control-styles.png" /></a></p><p>You might wonder what is the difference between Desktop and Base styles. The Desktop style, which is the default, actually uses Qt Widgets style engine to render everything, so it looks <em>exactly</em> the same as the standard Qt Widgets. It's good if you want to take advantage of Qt Quick without changing the look, but creating custom styles in C++ certainly does not look very attractive.</p><p>One can easily switch to the Base style, implemented entirely in QML, but it it slightly less polished, and has quite of lot of hardcoded styling, such as:</p><pre><code>Rectangle {
radius: TextSingleton.implicitHeight * 0.16
border.color: control.activeFocus ? "#47b" : "#999"
}</code></pre><p>The literal <code>0.16</code> is repated in 8 places over several files in the base style definitions, while <code>#47b</code> is found at 14 locations - not what I'd call solid engineering, compared to say most CSS frameworks or Android style system, where you can make consistent changes with a few attribute declarations. There are various third-party solution that offer modern styles for Qt, including <a href="https://developer.ubuntu.com/api/apps/qml/sdk-15.04.5/">Ubuntu Components</a> and <a href="https://github.com/papyros/qml-material">Papyros</a>, but these are not actually styles -- you need to use their own buttons and inputs and other components, you can't just restyle your existing code.</p><p>The set of controls is quite standard too. For example, there's not even implementation of filtered list view.</p><h2 id="layouts">Layouts</h2><p>Qt Quick offers two approaches to layout. Anchor-based layout allows you to position an item relatively to another item - for example you can fill parent and add margin, or you can put an item to the right of another item. It's easy, but not very adaptive.</p><p>There's is also dynamic layout, where desired size of items and available geometry is used to place items on screen. That's what I used, and it mostly works, except for annoying inability to set margins - as the margins feature in anchor layout is actually specific to anchor layout. I ended up writing a custom component that specifically adds margin around content.</p><h2 id="not-invented-here">Not invented here</h2><p>Both styling and layout issues above makes me thing that Qt Quick is being too original. It's a declarative UI language, so looking at CSS and trying to be more conceptually similar would have helped. Sure, CSS is not perfect, but a lot of people know how to use it to style everything from text to buttons to toolbars, and even create random <a href="https://css-tricks.com/examples/ShapesOfCSS/">geometric shapes</a>. Common behaviours like borders, margins, and shadows are trivial to accomblish. Android also has declarative UI with what I find a better style system and layout mechanisms. XAML is also a popular solution. It would be great if Qt Quick was at least conceptually similar to one of these.</p><p>Also, given that QML is heavily based on JavaScript, one would imagine JavaScript modules would work. Maybe not the fancy async module systems, but just standard CommonJS modules. They don't work, with no progress since 2011, as you can see in the <a href="https://bugreports.qt.io/browse/QTBUG-22356">issue</a>.</p><p>Overall, it would be great if Qt Quick were more aligned with other technologies. I have no idea whether it was possible given actual development timelines.</p><h2 id="quality-of-implementation">Quality of implementation</h2><p>There are a couple of issues that were important, but not fundamental.</p><p>As of Qt 5.5, we found that High DPI support is not quite good. Every time you specify a position or size, it's in pixels, so for good look on a High DPI system, one needs to have an utility function to scale pixels according to physical resolution. This might be fixed in Qt 5.6, but I haven't had a chance to try.</p><p>Qt Quick uses Open GL for all rendering. Of course, it creates lots of possiblities, but also configuration problems. Say, on Windows you might end up with 5 different GL implementations, not counting different video card vendors. We ended up with quite some crashes from users that had 'GL' in backtrace, and could not be reproduced. We also had fun getting things to work under Virtual Box, at one point even reverting particular Windows update for things to start working. This should eventually improve, but beware for now.</p><h2 id="wrapping-up">Wrapping Up</h2><p>I found Qt Quick to be a pleasant framework for developing a desktop app with a relatively simple interface. There were some limitations and issues, but they probably will eventually be fixed.</p><p>The biggest concern is that the set of controls and styles is quite basic, and that the styling mechanism is not quite flexible. Furthermore, the new direction is is Qt Quick Controls 2, and it is both focused on mobile devices and uses a different styling approach, so unlikely to bring improvements on desktop. If I were to use Qt Quick on a larger project, I would probably use a third-party controls library.<br />
</p><p>Qt remains the best framework for desktop applications using C++. Whether it's the best desktop framework overall, given that Xamarin recently was open-sourced, remains to be seen.</p>Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com5tag:blogger.com,1999:blog-7558958.post-51756600190690394442015-06-26T22:48:00.002+03:002015-06-26T22:48:51.976+03:00iPhone app clearance<div dir="ltr" style="text-align: left;" trbidi="on">
I moved from iPhone to an Android phone recently, and among dozens of iOS apps I've tried over couple years, some deserve to be mentioned. I include links where appropriate for convenience, but have no affiliation with any of the developers.<br />
<h3 style="text-align: left;">
Productivity</h3>
<div>
<a href="http://dayoneapp.com/">DayOne</a> is a very nice journal application. I've used it to record what I've done each day, and then review entries weekly and copy into a different application. It served that purpose very well - the UI is clean, there's reminder feature, and swipe navigation between entries. It also supports sync via dropbox, so I could make entries on a phone and review on a tablet.</div>
<div>
<br /></div>
<div>
<a href="https://itunes.apple.com/en/app/doc-scan-scanner-to-scan-pdf/id453312964?mt=8">DocScan</a> is an app to take a shot of a document with camera and convert it into PDF, as if produced by flatbed scanner. Its primary benefit is perspective correction that works very well. The files can obviously be sent by email, or added to cloud storage, though I found the number of required clicks a tad large. There's also some automatic conversion to black-and-white but it never produced results I've liked, so I always disabled it.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Fitness</h3>
<div>
Pretty much every fitness app is about using accelerometer and GPS to track steps and runs, which is quite boring, but I've came across two kinds of different apps.</div>
<div>
<br /></div>
<div>
<a href="https://www.runtastic.com/en/apps/pushups">Runtastic Push Ups</a> is a push ups trainer and tracker. You put it on the floor, and it uses proximity sensor to count your push ups. Or, you can press the screen with you nose. That sound a bizarre idea, but I found it a fun in practice. They also make two other apps - Pull Ups and Squats - that use accelerometer to track something other than steps.</div>
<div>
<br /></div>
<div>
There are also apps that flash and built-in camera to measure heart rate. <a href="http://www.cardiio.com/">Cardiio</a> does exactly that, while <a href="https://play.google.com/store/apps/details?id=com.azumio.android.stresscheck&hl=en">Azumio Stress Check </a>determines your stress level using heart rate variability. The latter can be quite entertaining.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Meditation</h3>
<div>
In this saturated app category, I've picked <a href="https://itunes.apple.com/us/app/deep-relax-your-best-companion/id614153824?mt=8">Deep Relax</a> and <a href="https://itunes.apple.com/us/app/self-a-meditation-machine/id492152473?mt=8">Self</a>. The former has about 40 different sounds you can mix, and set an arbitrary timer. The latter is beautifully minimalist, maybe too simple for me.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Lifetracking</h3>
<div>
I've tried a crazy number of those, and only a few were still installed after 5 minutes. Those that survived were Step Journal, Charge, TnS, Lumen Trails and rTracker, but none of them ended up actively used. In particular, rTracker, which is widely praised, ended up ugly app that could not draw sensible charts.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Games</h3>
<div>
This mini-review is concluded by the single game app I found installed, called <a href="https://play.google.com/store/apps/details?id=com.soren.musictiles&hl=en">Music Tiles</a>, where one clicks on black tiles that scroll from the top, producing sounds, and trying to go as far as possible. That's surprisingly addictive, especially on long-haul flights.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<b><br /></b>
<b><br /></b></div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com7tag:blogger.com,1999:blog-7558958.post-21908568456350842792015-06-01T22:29:00.000+03:002015-06-01T22:29:04.789+03:00Last Nine Years<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Exactly nine years ago today, on June 1 2006, I have joined CodeSourcery, as the first Eclipse engineer. I had zero Eclipse and Java experience, but knowing KDevelop and GDB was deemed sufficient. Look like I did passably well, and for my part, I’m happy to have played a small part in a huge change to open-source embedded development tools.</span></div>
<b id="docs-internal-guid-3eeae1de-b096-e217-e064-c77b5e96951c" style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Every current GDB tutorial for embedded development say to just load your binary to the target. It was my first big project, in 2006, to make it work, since GDB knew nothing about flash memory. I’ve ended up teaching it about memory maps, translating memory writes into flash erase and programming operations, throwing together support for some ColdFire chip, and finally adding a single checkbox in the UI.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GDB non-stop mode was entirely done by CodeSourcery. In this mode, each thread can be independently stopped, and examined, while others are running. There I’ve contributed to asynchronous processing of commands and reworking breakpoint machinery. We’ve made GDB handle breakpoints in constructors and function templates, implemented tracepoints, and different flavours of OS awareness. I was also part of initial prototyping for Python scripting.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">On Eclipse side, we made just as many changes, but had less luck submitting them upstream, so describing them is similar to a research paper - it tells what’s possible, but you’re on your own if you want an implementation. Still, we’ve made Eclipse scan for hardware debug device automatically, modified project wizard to include debug settings and create projects you can immediately debug, implemented a IDE editor for hardware board descriptions, and modified register view to effectively deal with thousands of memory-mapped registers. Among that, I did manage to create and submit a new Eclipse CDT view - OS Resources - that shows tables of different objects on the debugged system.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Between Eclipse and GDB, there’s a small interface called GDB/MI. It also saw significant changes, becoming less stateful, adding new notifications (so that Eclipse view don’t have to explicitly pull the data on each stop), and improving variable access methods.</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In November 2010, CodeSourcery was acquired by Mentor Graphics and our product went on to became Sourcery CodeBench, the decision based in part on progress made by open-source tools in the previous years. Understably, a lot of work after that went into integration with other products - including Mentor’s hardware debug devices, profiling tools and Mentor Embedded Linux. Personally, I went on to lead the IDE team, learning how to run a full distributed team across 12 time zones. We were less active in the open-source for a while, but gradually returned, and one of the biggest recent contribution is a product installer based on Eclipse P2 we’ve announced in 2014. </span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And then technology went full circle. The most recent open-source contributions from CodeSourcery team are patches for LLDB-MI, a bridge between LLDB and Eclipse.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In 2006, I’ve joined CodeSourcery in part because at the previous position, there was no longer anything to learn. Over years, I worked with the best people in each area: Daniel Jacobowitz and Pedro Alves on GDB, Carlos O’Donnel on Glibc and GCC, Mikhail Khodjaiants on Eclipse and of course Mark Mitchell, CEO who wrote a C++ frontend once. It was a great experience. Now, it's time to learn something new. See you there!</span></div>
</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com2tag:blogger.com,1999:blog-7558958.post-14129689948306140322015-03-30T15:50:00.000+03:002015-03-30T15:50:49.503+03:00Branding Eclipse Products<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="clear: both; text-align: left;">
Last year we worked on a new Eclipse-based IDE, in particular creating product branding from scratch. Despite visual editors and several existing online tutorials, that still proved confusing, so I've decided to documented what we've learned.</div>
<div style="clear: both; text-align: left;">
<br /></div>
<div style="clear: both; text-align: left;">
In this post, we'll review branding of a simple product. It has:</div>
<div style="clear: both; text-align: left;">
</div>
<ul style="text-align: left;">
<li>One functional plugin, and one functional feature including that one plugin</li>
<li>One product plugin and one product feature including the product plugin and functional feature</li>
<li>Product configuration</li>
</ul>
<div>
To make things simpler:</div>
<div>
<ul style="text-align: left;">
<li>I use artwork we used in our products.</li>
<li>There's no localization of strings</li>
<li>The welcome/intro screen is almost neglected, since we use custom HTML for that, and it might be best approach for any new product.</li>
</ul>
<div>
Of course, I assume you know what is plugin and what is feature.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Functional features</h3>
</div>
<div style="clear: both; text-align: left;">
</div>
<div>
Functional feature, along with contained plugins, implements some useful behavior that you can potentially install into any Eclipse-based product. For example, EGit feature is equally useful for Java and C++ IDE. For proprietary products, it is often tempting to just mix everything together, but creating separate features is often beneficial. The first example is of exactly such standalone feature, see <a href="https://github.com/vprus/eclipse-branding/releases/tag/functional-feature">sources</a>. </div>
<div>
<br /></div>
<div>
The important files in the plugin are META-INF/MANIFEST.MF, about.ini and about.html. For the feature, feature.xml and about.html are important. In particular, feature.xml relays most of the branding to a 'branding plugin', which, in our case, is the lonely functional plugin. </div>
<div>
<br /></div>
<div>
Starting with that source, you can import it into your PDE, click on feature.xml, export feature into a directory, install it into a second Eclipse installation, and then "Help->About" dialog will look like this:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1_GmuyfiIuy3WqjAjQOIAIdn8NKXLoXJLU2hYiw2VwAS98a1j3PEiHE6ZSofz68EK23xUX0VeSAtHhJkhKhaMk1tPQTn6pkl_gHoiygizfmjC5aQGIM-MANNBAQqSDT7vLfga/s1600/feature-about.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1_GmuyfiIuy3WqjAjQOIAIdn8NKXLoXJLU2hYiw2VwAS98a1j3PEiHE6ZSofz68EK23xUX0VeSAtHhJkhKhaMk1tPQTn6pkl_gHoiygizfmjC5aQGIM-MANNBAQqSDT7vLfga/s1600/feature-about.png" /></a></div>
In the row of icons that represent feature providers, there is our "crystal ball" logo. The icon is defined by the branding plugin, via featureImage attribute in about.ini. Should you have multiple feature with the same icon (pixel-wise), they will be merged in this dialog. Clicking "Installation Details" gives us in-depth information:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi0L6izziuUfGxWbWCTzUTLxuAaRWKpEzLW3n8VJwr4jdZy1vcNMvCzH3r-JprqxjpjYjqfACR0jZEB7JB7-ceUisqpYtKDO2l6UwQkpsDutZgG2w0yudjhO4na-ubh8mrx3HG/s1600/feature-ius.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi0L6izziuUfGxWbWCTzUTLxuAaRWKpEzLW3n8VJwr4jdZy1vcNMvCzH3r-JprqxjpjYjqfACR0jZEB7JB7-ceUisqpYtKDO2l6UwQkpsDutZgG2w0yudjhO4na-ubh8mrx3HG/s1600/feature-ius.png" /></a></div>
This dialog shows root P2 installable units in the Eclipse instance. The 'name' column is taken from 'label' attribute of feature.xml, and the description below comes from the 'description' element in feature.xml. The same description is shown when installing the feature. We can also use the "Properties" button to see more details from feature.xml, like copyright and license:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFeejzgQpojBuXlxehil0cI0HKIK_TkMkaufCN2sdn2xrakcHe6T1XTCzed0wCt97uhy-ySlBG2J1uVveY1NFtsre7jzRvubDa-nIFlV8BLTMOcsLgKJs7ToeTSlKRcjG_rPtF/s1600/feature-properties.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFeejzgQpojBuXlxehil0cI0HKIK_TkMkaufCN2sdn2xrakcHe6T1XTCzed0wCt97uhy-ySlBG2J1uVveY1NFtsre7jzRvubDa-nIFlV8BLTMOcsLgKJs7ToeTSlKRcjG_rPtF/s1600/feature-properties.png" /></a></div>
The above is fairly reasonable. The features tab, however, brings some surprises:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjs77Lfa5ZfGK_g5r_GM7bJhAkBk5J0VBW6pxAlOKs9nD32ZKa6ZA4k2wrWb_WW_wZpk9kWSdefyDcDjzyXE1aMy1YdWENDfovN1VpUQhrpDFPdz_0r4799HjZw6KZHffFYtFQH/s1600/feature-feature.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjs77Lfa5ZfGK_g5r_GM7bJhAkBk5J0VBW6pxAlOKs9nD32ZKa6ZA4k2wrWb_WW_wZpk9kWSdefyDcDjzyXE1aMy1YdWENDfovN1VpUQhrpDFPdz_0r4799HjZw6KZHffFYtFQH/s1600/feature-feature.png" /></a></div>
The "Feature Name" column is coming from branding plugin, the Bundle-Name attribute of MANIFEST.MF. The description below is composed from 'label' attribute in feature.xml and 'aboutText' attribute in plugin's about.ini. The icon is also coming from about.ini - and if you specify icon attribute in feature.xml, it is ignored. Finally, the "License" button opens about.html file in the feature directory - which is generally different from license attribute in feature.xml. Clicking the "Plug-in Details" button shows branding information for plugins, which is rather simple:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEHxwOUJ9Fs6CwwpY4Rmslo5droyU9cCb3-QwnPSrPuv1acoR9xX-Rthmue_KIhXW1CI7yLfZn99SKtix_Tn1W-YUnK9UpWmRDtbB65fRVWdn1JyQBM9-Ss7q9W5cDP_ho7mrs/s1600/feature-plugin.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEHxwOUJ9Fs6CwwpY4Rmslo5droyU9cCb3-QwnPSrPuv1acoR9xX-Rthmue_KIhXW1CI7yLfZn99SKtix_Tn1W-YUnK9UpWmRDtbB65fRVWdn1JyQBM9-Ss7q9W5cDP_ho7mrs/s1600/feature-plugin.png" /></a></div>
The "Provider" and "Plug-in Name" fields correspond to Bundle-Vendor and Bundle-Name in MANIFEST.MF. The "Legal Info" button opens about.html in plugin's root directory. As an aside, I'm not sure why it's called "License" for features and "Legal Info" for plugins and "License Agreement" for installable units.<br />
<br />
<h3 style="text-align: left;">
Products</h3>
<div>
Products put together a set of functional features that make sense for a particular audience, and add particular overall branding. Physically, product consists of a product feature and product plugin, organized the same way as functional feature and plugin. The example source for that is <a href="https://github.com/vprus/eclipse-branding/releases/tag/product-feature">here</a>, which you can again import into PDE, export into P2 repository, and install into separate Eclipse instance, and then run it with the "-product com.codesourcery.seed.product.product" command-line option.</div>
<div>
<br /></div>
<div>
The key element of product branding is this extension in plugin.xml:</div>
<div>
<blockquote class="tr_bq">
<product name="Example Eclipse Product" application="org.eclipse.ui.ide.workbench"><br /> <property name="appName" value="Example Eclipse Product"/><br /> <property name="windowImages" value="images/csl16.png,images/csl32.png,images/csl48.png"/><br /> <property name="aboutImage" value="images/IDE_about.png"/><br /> <property name="aboutText" value="About text for the example product."/> <br /> ...<br /></product></blockquote>
</div>
<div>
The first two properties define outside appearance of the product - it's name, shown in the window title, and its icon, shown in taskbar, or launcher, or window switcher, depending on your OS. The other two attributes affect the about dialog box, making it look like below:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXxf2d5OK-9JWmndBQdBPgltH0iJeXkF1wMgUeUiBybR73_k2_LlH51k8b1ZtSdJekRv2EzsI0Vzgaq_p9fO4iyPBnxN2Rzwmge30D6DP_hOn1IngisuZY-jyQPV8KkI-__Yd1/s1600/product-about.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXxf2d5OK-9JWmndBQdBPgltH0iJeXkF1wMgUeUiBybR73_k2_LlH51k8b1ZtSdJekRv2EzsI0Vzgaq_p9fO4iyPBnxN2Rzwmge30D6DP_hOn1IngisuZY-jyQPV8KkI-__Yd1/s1600/product-about.png" /></a></div>
<div>
Now it actually looks like a custom product! The installation details in this case are almost the same, except that it has two features, one depending on the other:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpiK6a6ELdPwjnttpp0mTDmzVUa-FXVUr5sNwqa1gYP28U47LUk_nfVdavNc10ONFe57pgvEiifQOdxSS790R9pHohNPDnHD_vk99czI5ZmEk5n1rsVjdNxQMpT2ninERXslJa/s1600/product-ius.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpiK6a6ELdPwjnttpp0mTDmzVUa-FXVUr5sNwqa1gYP28U47LUk_nfVdavNc10ONFe57pgvEiifQOdxSS790R9pHohNPDnHD_vk99czI5ZmEk5n1rsVjdNxQMpT2ninERXslJa/s1600/product-ius.png" /></a></div>
<div>
There are several other properties in product definition that are related to welcome screen, but as I've said, we replace it completely, so I'm not going to describe it. The example source code has some definitions if you're interested.</div>
<br /><div>
<br /></div>
<div>
<b>Launcher and Product Build</b></div>
<div>
The product feature we've built can be exported from PDE (or built with Maven, if you wish), and installed into Eclipse, but we usually want to build a complete product that can be immediately run. We need product configuration (.product file) for that, and it's covered in detail elsewhere. As far as branding goes, we only need two details:</div>
<div>
<ul style="text-align: left;">
<li>Showing custom splash screen on startup</li>
<li>Starting our product</li>
</ul>
<div>
The product configuration specifies them in fairly direct way - the product is specified as attributes of the top-level 'product' element, and the splash screen becomes a command-line attribute to the launcher. In the exported product directory, two files control this behaviour. First, the eclipse.ini files in the root directory contains '-showsplash com.codesourcery.seed.product' for the splash screen. Second, the 'configuration/config.ini' specifies the product to run. That almost completes our product branding. </div>
</div>
<div>
<br /></div>
<div>
Almost, because while product extension point can specify window icon and similar properties, the .product file also can specify those. When you do product export in PDE, the properties from .product files are copied into product extension point, so unless you duplicate them, you get product with no window icons. This problem is accounted for in the <a href="https://github.com/vprus/eclipse-branding/releases/tag/launcher">final version</a>. We don't have this problem in practice, since we built the final product from the command line, and so splash screen and product id is the only branding we need in the '.product' file.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Do it yourself</b></div>
<div>
I have put together a seed Eclipse product over at <a href="https://github.com/vprus/eclipse-branding">GitHub</a>, and you are free to use it if you are creating a new product. I would suggest these tips:</div>
<div>
<ul style="text-align: left;">
<li>Use high-resolution artwork, preferably created from vector originals, and keep those originals.</li>
<li>Having license in every about.html and every feature.xml is awkward. Either automate it, or refer to documentation for license terms.</li>
<li>Use the same label for each functional feature and its branding plugin</li>
<li>If you can get HTML support working on your target systems, use custom HTML instead of default welcome screen.</li>
</ul>
<div>
Hope this help!</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Acknowledgements</h3>
<div>
Dmitry Kozlov has worked with me on this, while <a href="http://www.mentor.com/embedded-software/sourcery-tools-services/">Sourcery Services</a> allowed me to take time to summarize our experience.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com2tag:blogger.com,1999:blog-7558958.post-25775807855011054282015-01-27T04:00:00.000+03:002015-01-27T04:00:02.074+03:00Lean Analytics<div dir="ltr" style="text-align: left;" trbidi="on">
Last year, I often needed to display and analyze timestamped events, such as product evaluations, issue tracker activity or credit card expenses. After trying a few approaches, I've ended up writing a JavaScript library called Lean Analytics. It's based on dc.js, crossfilter.js and D3.js, and looks like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVDTjsMN7t9NUx2X_7sB45zo7fGhB6BfSqxBGU5qdzFgxLhOH0HMxnFdY-8M0LyX1EaluFZ4sBFIFQiZrlcN9F2GdICoiQAI77yvPGSfrXsDg6RclZEL9NJuOdMGihpz7jO15I/s1600/lean-analytics.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVDTjsMN7t9NUx2X_7sB45zo7fGhB6BfSqxBGU5qdzFgxLhOH0HMxnFdY-8M0LyX1EaluFZ4sBFIFQiZrlcN9F2GdICoiQAI77yvPGSfrXsDg6RclZEL9NJuOdMGihpz7jO15I/s1600/lean-analytics.png" height="355" width="640" /></a></div>
<br />
The easiest way to understand it is to just play with the <a href="http://vprus.github.io/lean-analytics">demo</a> or take a look at the demo <a href="https://github.com/vprus/lean-analytics/blob/master/demo/expenses/index.js">source code</a>. Below I'll explain what it is, when you'd want to use it, and when not.<br />
<br />
<h3 style="text-align: left;">
Overview</h3>
The primary goal was to just visually show the trends in already collected, but rather dry data. The amount of data is fairly small, dimensions are few, and there's no need to extract hidden correlations between dozens of values nor there's a need for dedicated analysts to tweak the charts on a full-time basics. Rather, I wanted it to be extra easy to chart new type of data, don't store anything in the cloud, and embed the charts in existing web apps.<br />
<br />
The library itself is bundled into a single JavaScript file, plus you need to include 3 CSS files. You also need to write code to define where do get data, what metrics to show, and how to group your entries - all of which is <a href="https://github.com/vprus/lean-analytics/blob/master/demo/expenses/index.js">straightforward</a>. For that, you get a lot of fine-tuned visuals:<br />
<br />
<ul style="text-align: left;">
<li>Chart showing main metric (such as transaction amount) aggregated per week, as well as derived metric (such as trendline). There are also dropdowns to select desired metrics.</li>
<li>Compact linear charts showing distribution of the chosen metric over categories.</li>
<li>Tabular view of the data.</li>
<li>Filtering of main chart by category values in real time in your browser. The filters are even stored as part of URL, so you can share links easily.</li>
<li>Buttons to select time ranges.</li>
<li>Automatic progress and error reporting for loading data.</li>
</ul>
<div>
The charts are meant to replace a div in your host HTML document, and they use Bootstrap for styling, so probably will work just fine inside your internal webapps.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
Alternatives</h3>
<div>
DC.js is the foundation for Lean Analytics, and together with crossfilter, does all the hard stuff of filtering data in real time in your browser. It can be used to create way more interesting visualizations, but it requires a considerable amount of code to configure all the details - way more that I was comfortable with.</div>
<div>
<br /></div>
<div>
Several libraries are implementing charts on top of D3, such as NVD3 and C3. Sadly, those are not integrated with crossfilter, and are somewhat in a state of flux.</div>
<div>
<br /></div>
<div>
Google Charts is very solid as far as charting goes, but does not support any crossfiltering either.</div>
<div>
<br /></div>
<div>
Kibana is a full-blown dashboard solution, on top of ElasticSearch. It's certainly great for serious data analysis, but is both not trivial to setup, and is not embeddable in webapps.</div>
<div>
<br /></div>
<div>
Mixpanel is fairly nice, but it's a cloud service, and I did not want, or could not, put data in the cloud.</div>
<div>
<br /></div>
<div>
Zenobase, finally, is a very nice solution specific to lifetracking, to answer questions like "how is my blood pressure correlated with weight". It is inspiring in some ways, but is also a cloud service, and too specified for life tracking to be directly useful.</div>
<div>
<br /></div>
<h3 style="text-align: left;">
<b>Conclusion</b></h3>
<div>
If you want to chart timestamped events with numeric values that are naturally aggregated over weeks, and you want to filter data by categories in real time, and the amount of data is not very large, give Lean Analytics a try.</div>
</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com7tag:blogger.com,1999:blog-7558958.post-53357551187213227882014-12-23T19:12:00.001+03:002014-12-27T09:36:44.986+03:00Calendars and Timezones<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
It's boring to talk about calendars and time zones, but apparently major software companies still get this wrong, and a fair number of people end up very confused. For the latest example, in summer 2014, the time zone in Moscow, Russia was UTC+4, and it were to stay this way all year. Then the government decided to switch to UTC+3 in Autumn anyway, and I woke up with iPhone showing wrong time.<br />
<br />
<b>iOS: hardcoded time zone</b><br />
<br />
Since iOS has time zone as part of OS, it still thought Moscow is UTC+4, so the clock was one hour later. That was easy to fix, I've changed iPhone time zone to a nearby UTC+3 one. And then, the calendar events started to randomly misbehave, showing one hour off, in different directions.<br />
<br />
<b>Google Calendar: works just fine</b><br />
<br />
Suppose I create an event in Google Calendar, via web, at 18:00 Moscow (UTC+3). The invitation email to guests has a total of 4 MIME parts:<br />
<ul style="text-align: left;">
<li>HTML part describing the event, with buttons to accept or decline</li>
<li>text part with a plain-text version of same</li>
<li>attachment with the application/ics content type, and invite.ics name</li>
<li>invisible part with text/calendar content type, and same content as invite.ics</li>
</ul>
<div>
Applications that understand invites will look at one of the last two parts, and see this:<br />
<blockquote class="tr_bq">
<span style="font-family: Arial; font-size: 15px; white-space: pre-wrap;">DTSTART:20141210T150000Z<br />DTEND:20141210T160000Z</span></blockquote>
</div>
That is, the invite specifies event type as 15:00 UTC, with no timezone information, which is the correct time.<br />
<br />
<b>iOS: using time zone name</b><br />
<br />
When I access my Google Calendar from iPhone, I see the same event on 19:00. Apparently, when accessing Google Calendar via Exchange protocol, iOS receives time zone information of the event, as "Moscow", checks its outdated database, and decides the time is 18:00 UTC+4 - one hour later than it should be.<br />
<br />
<b>Exchange: totally confused</b><br />
<br />
Like iOS, our corporate exchange did not know about recent changes, and still thought Moscow is UTC+4, I've switched it to a time zone calle<span style="font-family: inherit;">d "<span style="white-space: pre-wrap;">(UTC+03:00) Kaliningrad, Minsk". When I create event at 18:00 the invitation email has 2 MIME parts:</span></span><br />
<ul style="text-align: left;">
<li><span style="font-family: inherit; white-space: pre-wrap;">text part that briefly say "When: Saturday, December 06, 2014 19:00-20:00. (UTC+03:00) Minsk", which looks right </span></li>
<li><span style="font-family: inherit;">invisible part with text/calendar content type</span></li>
</ul>
<div>
That calendar data is just the opposite to what Google does, since it names time zone:</div>
<blockquote class="tr_bq">
<span id="docs-internal-guid-8154a6b0-77d2-7883-33c5-83ab3c5f3a78"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">DTSTART;TZID=Kaliningrad Standard Time:20141205T180000</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">DTEND;TZID=Kaliningrad Standard Time:20141205T190000</span></span></blockquote>
Not only that, but the name of timezone is different from the text part, and the timezone itself is also defined inside the content, as UTC+2, so that makes for 18:00 UTC+2 - one hour earlier than it should be. I don't have a good theory how it can be that broken.<br />
<br />
<b>Solution</b><br />
<br />
What I ended up doing, and what you should do in similar situation, is find a timezone with the right offset and DST rules, but far away geographically and politically. I've ended up using Madagascar iOS and Nairobi in Exchange. Then, you should make sure that every single calendar system you use is switched to that timezone:<br />
<ul style="text-align: left;">
<li>In Google Calendar, under Calendar Settings, modify Country and then Time Zone.</li>
<li>On iOS, modify Settings → General, Date & Time.</li>
<li>On iOS, also modify Setting → Mail, Contacts,Calendars → Time Zone Support.</li>
<li>In Exchange, or rather Outlook Web App, modify Settings → Regional → Current Time Zone. You can also go to Settings → Calendar and click "change your work week to the current time zone", though that does not matter much.</li>
</ul>
<br />
<br />
<br />
<br />
<div>
<br /></div>
</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com0tag:blogger.com,1999:blog-7558958.post-86160892482157848052014-10-28T19:07:00.000+03:002014-10-28T19:07:35.564+03:00Psychology of Software Development<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<br />
In 1987, Tom DeMarco and Timothy Lister wrote <a href="http://www.goodreads.com/book/show/67825.Peopleware">Peopleware</a>, and it goes like this:<br />
<blockquote class="tr_bq">
The major problems of our work are not so much technological as sociological in nature.</blockquote>
25 years later, it looks still true. I have mostly worked around open source, where I do recall big public clashes, like gcc versus egcs, or glibc versus eglibs, caused by social reasons. Ongoing systemd saga is mostly social. Everywhere, design discussions become personal ones, with no way to decently get out. Regardless of technology, good maintainers often have particular character traits. Now that's just me, and the world of open-source and cute nonsense hacks. Do real psychology researchers have anything to say about real professionals?<br />
<br />
Already in 1975, <a href="https://www.goodreads.com/book/show/66354.Flow">Flow</a> and <a href="https://www.goodreads.com/book/show/1041225.Intrinsic_Motivation_and_Self_Determination_in_Human_Behavior">Intrinsic Motivation</a> began to question classic ways to run a business. Since then, psychologists found many pieces of the puzzle, showing that sociology and psychology are indeed important. Managers often don't care. Waterfall was sometimes replaced with agile, with its "Individuals and interactions over process and tools", but that soon became a new cargo cult. In the original curgo cult (<a href="https://www.goodreads.com/book/show/5544.Surely_You_re_Joking_Mr_Feynman_">Surely You're Joking, Mr. Feynman!</a>), natives sat with fake wooden headphones, waiting for airplanes with goods to arrive. These days, we sit with actual headphones, hold all the right meetings and wait for team to "self-organize" and for the breakthrough to happen.<br />
<br />
Let me try to give an index of current psychology that apply to software engineering, so that if you want to a better software developer, or engineering manager, you can have an easy starting point. The material here follows the <a href="http://2014.secr.ru/program/submitted-presentations/psychology-of-software-development">SECR 2014 talk</a> I recently gave, but is edited to be more hands-off and is in English.<br />
<b><br /></b>
<b>The puzzle pieces</b><br />
<br />
We like to think that evolution meant to create humans all the time, but it's not a smooth directed process. Rather, it's a random competition between every two different genomes - including between viruses and animals, animals and humans, humans and other humans, and even humans of different genders - with no ultimate winner (<a href="https://www.goodreads.com/book/show/16176.The_Red_Queen">The Red Queen</a>). Multiple behavior strategies were formed in humans as the result, with emotions to activate them, and all of us have a custom mix of them to use. Each strategy is good enough, statistically, to stay in human genes, but no single one is guaranteed to work for any particular human. Worse, these strategies were created to work in prehistoric environments, which no longer exist (<a href="https://www.goodreads.com/book/show/681941.The_Moral_Animal">The Moral Animal</a>). No wonder that we're driven by emotions, and nudges, and often make wrong decisions.<br />
<br />
We know that all kinds of emotions are important to work, such as motivation (<a href="https://www.goodreads.com/book/show/6570502-switch">Switch</a>), happiness (<a href="https://www.goodreads.com/book/show/9484114-the-happiness-advantage">The Happiness Advantage</a>) as well as overall ability to deal with emotions in oneself and others (<a href="http://emotional%20intelligence/">Emotional Intelligence</a>). Doing the right things is easier with right habits (<a href="https://www.goodreads.com/book/show/12609433-the-power-of-habit">The Power of Habit</a>) and with helpful nudges (<a href="https://www.goodreads.com/book/show/2527900.Nudge">Nudge</a>), but at times, willpower need to be involved (<a href="https://www.goodreads.com/book/show/10865206-the-willpower-instinct">The Willpower Instinct</a>).<br />
<br />
Behind the surface, our brains are nothing but neural networks, and those are not perfect. They can quickly learn, and then respond very quickly, basically automatically. The clusters of strong neuron connections created by learning, called attractors, get stronger each time they are used, and we get progressively efficient. But that's only when learning has a proper feedback loop - fast, positive and consistent. Otherwise, we form false attractors - that fire fast, and get stronger with each use, but produce wrong results. We're particularly prone to create false attractors when feedback is slow, and probabilistic.<br />
<br />
For thinking, false attractors are responsible for thinking biases (<a href="https://www.goodreads.com/book/show/11468377-thinking-fast-and-slow">Thinking Fast and Slow</a>), such as coming up unrealistic estimates, siding with other's unrealistic estimates, picking up first design that comes to mind without checking, and trying hard to avoid past mistakes that are unlikely to happen again. For emotions, seriously false attractors require attention of a psychotherapist, while in common case, we can get excited or upset just because we were in similar situation before, not because the response makes sense today (<a href="https://www.goodreads.com/book/show/35711.A_General_Theory_of_Love">General Theory of Love</a>). At the intersection of thinking and emotions, we often get excited about rewards, and regret later. One classic example is quarter-end rush to sign deals - and then figure out how to get them done. This is one case where introverts can be extremely helpful at work, since they don't get excited about rewards all that much (<a href="https://www.goodreads.com/book/show/8520610-quiet">Quiet</a>). We do have a capacity for slow, more formal and symbolic thinking, but it's much harder, and the need to use it often can only be pointed by others.<br />
<br />
<br />
<b>The picture</b><br />
<br />
There is no final picture, and many pieces don't fit. Many are broken, despite passing statistical tests, because <a href="http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124">most published research findings are false</a>. Many are discolored, having so small effect it's not clear where to put them. The sampling bias is caused by using mostly students for experiments. The measurement bias is here - say, how would anybody measure programmer's performance? I would hope nobody counted lines of code written or tickets closed. And many psychological theories are just not good scientific theories, if we use the definition that Steven Hawking gives in <a href="https://www.goodreads.com/book/show/3869.A_Brief_History_of_Time">Brief History of Time</a>:<br />
<blockquote class="tr_bq">
good theory [...] must accurately describe a large class of observations on the basic of a model that contains only a few arbitrary elements, and it must make definite predictions about the results of future observations</blockquote>
Surely physics cannot predict the future of entire universe, but it is fairly good at predicting where a rocket would go if an engine is fired. Psychology, in contrast, has a large set of observations but struggles with useful predictions or interventions. Influential books are particularly at fault, explaining how flow, and happiness and emotions are important, but failing to mention that finding flow in gardening, and walking around smiling might not be great strategy for a person or society. Evolutionary psychology (<a href="https://www.goodreads.com/book/show/681941.The_Moral_Animal">The Moral Animal</a>) holds some promise here, having small foundation, and trying to predict everything, but it's too early to judge its success.<br />
<br />
<b>The corners</b><br />
<b><br /></b>
Not having a complete picture is unfortunate, but looks like the corners are filled in - we know few important areas where neglect is detrimental.<br />
<br />
<ul style="text-align: left;">
<li><b>Emotions</b>. Emotions matter, and while Real Professional can do unpleasant things for a while, without emotions, habits and nudges, he will likely give up. I'd conjecture that often, team opinion and motivation about possible new features is more important than the estimates of market value.</li>
<li><b>Trust</b>. Is everybody in the same boat, heading in the same direction, sharing the same prizes, and actually rowing when needed?</li>
<li><b>Feedback</b>. For automatic thinking to work well, we need feedback that is positive, timely, and consistent. That does not happen by itself in complex areas, say quality, or architecture design, and need to be explicit part of the process - be it review of key architecture decisions few months down the road, or having low defect count celebrated.</li>
<li><b>Slow thinking</b>. Automatic thinking will eventually make mistakes, and then start repeating them. If you have orderly predictable Scrum process, you can easily become too complacent. There must be time to make complex technical decisions, and time to review direction from the basic principles.</li>
</ul>
<div>
Do these corners receive systematic attention on your team? Are they among process metrics? Probably not, and probably some of them need improvement. Lacking full picture, we don't know how much effect any changes will have. Maybe, things are already not too bad in ad-hoc way. Maybe, some are totally broken. Anyway, we're software engineers, we're good at setting up processes, and if we keep these four corners in mind, we can make our processes work together with our psychology, and not against it.</div>
</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com1tag:blogger.com,1999:blog-7558958.post-1848841953217730862014-07-31T11:55:00.001+04:002014-12-21T21:33:00.103+03:00Text Baselines in HTML<div dir="ltr" style="text-align: left;" trbidi="on">
I usually write HTML as part of a some quick hack, often personal, so I take every shortcut I can. But one thing I cannot stand is misaligned text - where two text elements appear next to each other horizontally, but their text baselines are not perfectly aligned. This is a much-discussed topic, but somehow most of the solutions that professional designers discuss don't work for me.<br />
<div>
<br /></div>
<div>
For example, take a blog post called <a href="http://alistapart.com/article/settingtypeontheweb">Setting Type on the Web to a Baseline Grid</a>. It's fairly detailed about global structure of the page, which is indeed nice, and it comes with an example, and when you look closely, you see this clear misalignment:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwzTN8f8lUpv85AS_4XJThv4h6vm-zoaEhZXaiKb2z1AEmR_CJhYtI-jmRMRVEIU5azJbnuUpb2bb26D-YEdchn3nWFMSvUEZ6eThyphenhyphencu5HxsD-kBEi7D3xLNr7S4v_HtngUBV6/s1600/baseline-1.png" imageanchor="1" style="clear: left; float: left; margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwzTN8f8lUpv85AS_4XJThv4h6vm-zoaEhZXaiKb2z1AEmR_CJhYtI-jmRMRVEIU5azJbnuUpb2bb26D-YEdchn3nWFMSvUEZ6eThyphenhyphencu5HxsD-kBEi7D3xLNr7S4v_HtngUBV6/s1600/baseline-1.png" /></a></div>
<br />
<div style="clear: both;">
Incidentally, aligning baselines of two text elements with different font size is what I wanted to do yesterday, so let's see how to do it, starting with a case that works just fine. Here are the essential parts of HTML:</div>
<blockquote class="tr_bq">
<body><br />
<div><br />
<span class="big">Hello</span><br />
<span class="small">World</span><br />
</div><br />
</body></blockquote>
<div>
<br /></div>
<div>
The way it looks, with boxes added for exposition, is this:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0EQisc8cWIWADfiWQCmIxXtzmLR7NSok5lf4GMlmD-JjHDoSBt8fLQ2zUbRxRZ3jL6vqn8Il-6_LBzfXVVZYsxuAPPcr5ZiAe22fcKkHsKBYGHfybDyh9z_C79LQj61tsgwn4/s1600/baseline-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0EQisc8cWIWADfiWQCmIxXtzmLR7NSok5lf4GMlmD-JjHDoSBt8fLQ2zUbRxRZ3jL6vqn8Il-6_LBzfXVVZYsxuAPPcr5ZiAe22fcKkHsKBYGHfybDyh9z_C79LQj61tsgwn4/s1600/baseline-2.png" /></a></div>
<br />
<div>
Each of the span elements creates a box with text. The height of each box is equal to CSS font-size property - which is 36px for the first box and 24px for the second. They are then laid out using <a href="http://www.w3.org/TR/CSS21/visuren.html#inline-formatting">inline formatting context</a> that by default aligns baselines of each box - so the smaller box is moved down. If you examine the <a href="http://jsfiddle.net/2PZ8d/1/">example 1</a> you'll see that second span element as offsetTop property of 11px - which is exactly what is needed to align baselines, helpfully computed by the browser.</div>
<div>
<br /></div>
<div>
If we were to compute that magic value of 11 ourselves, we'd be in trouble. The CSS font-size property effectively only tells the height of the text box. Although the size of actual letters (x-height and cap height) is proportional to box height, the proportion depends on the font. The position of the text baseline in the text box also depends on the font, so we have nothing to work from. For a very detailed discussion, see <a href="http://www.thomasphinney.com/2011/03/point-size/">Point Size and the Em Square: Not What People Think</a>. There are libraries that try to compute font metrics, such as <a href="https://github.com/Pomax/Font.js">Font.js</a>, but they do it by drawing font on the canvas and then looking at pixels - rather roundabout way.</div>
<div>
<br /></div>
<div>
If it's hard to compute the right offset for the smaller box, maybe we can make the smaller box have the same height? There is the line-height CSS property indeed, which can be larger than font-size, but it adds the extra height equally on the top of the bottom. In our case, it will add (36-24)/2=6 pixels at the top, very different from the desired 11, as shown by <a href="http://jsfiddle.net/2PZ8d/2/">example 2</a> and the below screenshot.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqWAPRv1_NpnbMMRf3zEKxuul5J8i5xROFbiSczPBAZh5rqmkUzDBo5S0nIeGhC6B5x1JLXGUuvQq__ETuvQS_lNb4yxj5xGxEkBTyTPNVuX1xO82gEKPMYo51w6WLFzgCqR8W/s1600/baseline-3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqWAPRv1_NpnbMMRf3zEKxuul5J8i5xROFbiSczPBAZh5rqmkUzDBo5S0nIeGhC6B5x1JLXGUuvQq__ETuvQS_lNb4yxj5xGxEkBTyTPNVuX1xO82gEKPMYo51w6WLFzgCqR8W/s1600/baseline-3.png" /></a></div>
<br />
<div>
The bottom line is that when text elements with different font size are inside single inline formatting context, they are perfectly aligned by default, and replicating such alignment manually is hard. But what if want alignment outside inline context - for example if I want to send the second text element to the right?</div>
<div>
<br /></div>
<div>
<b>Floats</b></div>
<div>
<b><br />
</b></div>
<div>
One way to send an element to the right is "float: right" property, but <a href="http://jsfiddle.net/2PZ8d/3/">example 3</a> looks rather bad.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf5WA9RJqAOn0oCDgjojICvIQS39Yt7njMzyNUA0Hyw52a7cUf0EnKPhkZH_c-rXj9_T-DSOu0AWkbschCM-ZYt4JhTAv1CSCaR-33RkHVgLmskh4S4hO85l606D1rIUrrR8WM/s1600/baseline-4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf5WA9RJqAOn0oCDgjojICvIQS39Yt7njMzyNUA0Hyw52a7cUf0EnKPhkZH_c-rXj9_T-DSOu0AWkbschCM-ZYt4JhTAv1CSCaR-33RkHVgLmskh4S4hO85l606D1rIUrrR8WM/s1600/baseline-4.png" /></a></div>
<br />
<div>
This is not a browser bug - CSS spec is clear that when element is floated to the right in inline context, it will be aligned to the top, and this behavior cannot be changed. As explained above, shifting the element manually is rather unpractical. What we can do though, is to add a strut - text element with the large font size, serving only to force the alignment. Here's <a href="http://jsfiddle.net/2PZ8d/4/">example 4</a> and a screenshot:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiylzE1jFQpxNskjRr6TuNkFS8lW6XHCZxbLpWZ-0MJPVc6ywW5K-CW9WRIe_vjDIW8rMyWSmSnxolF-qA7_UgGGVX737f_BT0PDNe2oqjtMuh47nZUbwJCHslDUN49OnprXgx/s1600/baseline-5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiylzE1jFQpxNskjRr6TuNkFS8lW6XHCZxbLpWZ-0MJPVc6ywW5K-CW9WRIe_vjDIW8rMyWSmSnxolF-qA7_UgGGVX737f_BT0PDNe2oqjtMuh47nZUbwJCHslDUN49OnprXgx/s1600/baseline-5.png" /></a></div>
<br />
The final observation is that if inline box has no content, the browser adds true <a href="http://www.w3.org/TR/CSS21/visudet.html#leading">strut</a> - of zero width. So, here's <a href="http://jsfiddle.net/2PZ8d/5/">example 5</a> - with all borders removed, just perfectly aligned text<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHRKV1uFWqIIY7bfZ_OZNQa1j_mXM_ehnZJW4lLJb8yvDWJjQXObeay0ZPyJDww3ZVrKcUbZSAsRU5lVytMJ6cglPhwvLoBbYVJrPODaH01ItQSgABF95_ZK6NLNM1yyvFu0Di/s1600/baseline-6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHRKV1uFWqIIY7bfZ_OZNQa1j_mXM_ehnZJW4lLJb8yvDWJjQXObeay0ZPyJDww3ZVrKcUbZSAsRU5lVytMJ6cglPhwvLoBbYVJrPODaH01ItQSgABF95_ZK6NLNM1yyvFu0Di/s1600/baseline-6.png" /></a></div>
<br />
<b>Using positioning</b><br />
<b><br />
</b> Instead of floats, one can use either "position: relative" or "position: absolute" on the second span. Using absolute position is fairly easy - with "right: 0" it will be moved to the right border of the container (<a href="http://jsfiddle.net/2PZ8d/6/">example 6</a>) but also move to the top - just like happened with float. Strut can be used to fix this, likewise. When using relative positioning, we don't need to do anything about baseline - everything remains aligned by default. However, we need to figure out relative horizontal shift - not too hard, but requires JavaScript (<a href="http://jsfiddle.net/2PZ8d/7/">example 7</a>).<br />
<br />
<b>Using text alignment</b><br />
<b><br />
</b> I have intentionally used a simple example here, but if displaying two words on different sides of a page is exactly what you need, you can just give width to both span elements, so that they cover entire width, and then make the second element right-aligned.<br />
<br />
<b>Conclusions</b><br />
<b><br />
</b> It is unfortunate that CSS neither supports baseline alignment across different blocks, nor exposes font metrics is a usable way. If you know exactly what fonts are used, you can compute offsets by hand. In other cases, use struts or relative positioning. </div>
</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com0tag:blogger.com,1999:blog-7558958.post-90640728925490364282014-07-09T19:58:00.001+04:002014-12-21T21:33:14.235+03:00Rectangles with SVG<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: justify;">
One would think drawing a rectangle using SVG is easy - it's a basic shape, and there are numerous tutorials with straightforward instructions, such as:</div>
<blockquote class="tr_bq" style="clear: both; text-align: justify;">
<rect x="10" y="10" width="100" height="100" stroke="black" stroke-width="1" fill="white"/></blockquote>
<div class="separator" style="clear: both; text-align: justify;">
What is rarely explained is the exact meaning of the coordinates, and dimensions.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
SVG operates in a pure mathematical coordinate system. Coordinates are real numbers and the lines of the shapes are infinitely thin. Width and height of a rectangle are distances between these infinitely thin lines. The stroke is applied alongside these mathematical lines, with equal width on each size. So, the above rectangle is actually 101 units wide - with 0.5 unit of stroke applied to the left of left line, then 100 pixels to the right right and 0.5 unit of stroke to the right. And there's a total of 1 unit of stroke inside the rectangle, so the empty inside of the rectangle is 99 units wide.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
Coordinates don't need to be integer, and it can even be more logical to have fractional coordinates. The picture below shows a rectangle with extra content that is scaled up, in SVG, for exposition. The grid lines corresponds to integer coordinates in the unscaled world. The y coordinate of the rectangle is 2.5. With 0.5 of stroke added on both sides, it nicely occupies space between y=2 and y=3.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEoZF0dZWqDgrHFIQqn-9BbKU3dxUZ8FrytOXLLwMhqRz2pllYT8f4d3DAdSrYPYIQD80uPcNk6RCMYrxu2pt4I15Jk_FyDlFBBuOuNUqJySJ4x4uuNHblqshnEKCGw4TheArN/s1600/g3257.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEoZF0dZWqDgrHFIQqn-9BbKU3dxUZ8FrytOXLLwMhqRz2pllYT8f4d3DAdSrYPYIQD80uPcNk6RCMYrxu2pt4I15Jk_FyDlFBBuOuNUqJySJ4x4uuNHblqshnEKCGw4TheArN/s1600/g3257.png" /></a></div>
<br />
But when we try to display SVG using default coordinate system, when one uint is a pixel, things break down. There, integer values of coordinates fall between the pixels, and the browsers cannot quite agree about how to apply 0.5px stroke on both sides of a line. The below picture shows rendering of rectangles by Chrome and Firefox, using both whole-pixel and half-pixel coordinates.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjM0F4fjXSTWQrJqn7i3EtuUKSn3yc76bTBt9tdxrK8IJPZr71SveWk370Q4Kja-iwSDOGMDgCWZ60vY-HRjMuWu4tWh4wRNdP9Mrwcj9FByiZyb0kA9uxvW6psIhPm5b_bgD3b/s1600/svg-rectangle-pixels.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjM0F4fjXSTWQrJqn7i3EtuUKSn3yc76bTBt9tdxrK8IJPZr71SveWk370Q4Kja-iwSDOGMDgCWZ60vY-HRjMuWu4tWh4wRNdP9Mrwcj9FByiZyb0kA9uxvW6psIhPm5b_bgD3b/s1600/svg-rectangle-pixels.png" /></a></div>
Chromium does a logical thing. With integer coordinates, it applies 0.5px stroke on both sides of the line by making the pixel half-saturated - so we get line that is twice as wide and twice less dark. With half-pixel coordinates the mathematical lines go through the middle of pixels, so 0.5px stroke on both side result in a single pixel painted with the stroke color. What Firefox does, I don't know. If you look carefully, vertical and horizontal lines are pained differently, and top line is painted differently from the bottom line. Both with integer and half-pixel coordinates most of the lines are two-pixel, with different saturation of pixels. This looks like explicit programming, but I miss the logic. Of all of these I prefer Chromium rendering with half-pixel coordinates.<br />
<br />
Of course, drawing a rectangle was not my goal. Rather, I was about to draw program control flow graphs using <a href="http://jointjs.com/">JointJS</a> and <a href="https://github.com/cpettitt/dagre">Dagre</a>. Overall, these libraries are rather good - I could quickly produce a reasonable prototype and JointJS code was easy to read and extend. However, it also effectively forces integer coordinates. There is one place in code where it did rounding unintentionally, and another place where coordinates are rounded explicitly. Will certainly work around this locally, but it would be great if people building on top of SVG pay attention to this issue.<br />
<br />
If you want to follow along, here is the <a href="http://bl.ocks.org/vprus/cbd156d6939ffdf623ae">first example</a> and <a href="http://jsfiddle.net/LYaYk/1/">second example</a>.</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com0tag:blogger.com,1999:blog-7558958.post-79297304299506987542014-04-21T14:00:00.000+04:002014-04-21T14:00:00.669+04:00IRC visualization using D3<div dir="ltr" style="text-align: left;" trbidi="on">
In order to learn <a href="http://d3js.org/">D3</a>, I've created some basic visualization of IRC messages, with pictures like below. GitHub has the <a href="http://github.com/vprus/data-irc">sources</a> and <a href="http://vprus.github.io/data-irc">demo</a>.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsX-bOhjsm0VemQhKt2V3nvRLp3VKb1tsBIBZm_ZNaFEdkAevNdIdLZqdiqsgijyXDHYHKI-GsNrDTQUFPTacjaSu73OMADwbNC4seIp87V9CbvHApXJdm2KngfyCr8V19M4tq/s1600/Screenshot+from+2014-03-27+10:12:28.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsX-bOhjsm0VemQhKt2V3nvRLp3VKb1tsBIBZm_ZNaFEdkAevNdIdLZqdiqsgijyXDHYHKI-GsNrDTQUFPTacjaSu73OMADwbNC4seIp87V9CbvHApXJdm2KngfyCr8V19M4tq/s1600/Screenshot+from+2014-03-27+10:12:28.png" height="298" width="320" /></a></div>
I started with an internal IRC channel, and thought there might be 2 or 3 group of people mostly talking to each other. The created graph was a perfectly round cloud. Some surprises in the center, but that's it. The histograms showed that some people are very chatty, and pairwise histogram of messages was even more telling - 1300 pairs that only exchanged a few messages, and 60 very active paths. Neither showing just the top 20 users nor excluding top 20 users revealed any grouping. Then I've made the each node in the graph be pulled to the only another node - the top recipient of messages - and got the above picture. There are some groups visible, and the grouping is indeed around functional areas.<br />
<br />
The data in the demo is artificial, but I've tried to recreate the same distribution, and the results are fairly close. Trying on logs for some other channels, mostly open-source, was a failure. The histograms are similar - few people with a large number of messages - but the graphs had no grouping at all.<br />
<br />
D3 itself is fairly nice framework, as many of its examples show already. Nice data binding mechanism, animation, and a lot of utilities. Still, it's a construction kit. There are no charts one can use. of out the box and even things like margins are copy-pasted between all examples. It was also required to fine-tune everything to the data, from chart dimensions to margins to parameters of the graph layout algorithm. If one tries on another data set, the axes of histograms break down, and the nodes of the graph disappear to the sides of the screen.<br />
<br />
There are libraries that try to create reusable charts on top of D3. <a href="http://c3js.org/">C3</a> appears to have nice standard charts, but completely hides all of D3's power. <a href="http://nvd3.org/index.html">NVD3</a> is quite promising, but has no documentation and is apparently in the middle of extensive rewrite. <a href="http://nickqizhu.github.io/dc.js/">DC</a> is meant to be a frontend to <a href="http://square.github.io/crossfilter/">crossfilter</a>, but not so useful with arbitrary data - and in another context, I just could not get it to create charts I want.<br />
<br />
I will likely use D3 in future, though will need a personal set of chart helpers first.</div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com0tag:blogger.com,1999:blog-7558958.post-5026689081089561962014-03-20T12:19:00.000+04:002014-12-21T21:32:30.687+03:00Eclipse P2 Product Installer<div dir="ltr" style="text-align: left;" trbidi="on">
Last year, the Sourcery CodeBench team have implemented a new installer, using Eclipse P2. This week, at EclipseCon 2014, two of key developers gave a talk about it, and announced that all of the source code is available under Eclipse Public License, at <a href="https://github.com/MentorEmbedded/p2-installer">https://github.com/MentorEmbedded/p2-installer</a>.<br />
<br />
Before, we used certain commercial installer technology, and it was not much fun. There were annoying bugs (like randomly creating corrupted installers), and there were fundamental issues. It did not know anything about Eclipse components, so it could only install everything. And it used an XML-based description, which worked nice for simple projects, but became a nightmare to create or generate. We decided it's best to start from square one.<br />
<br />
The P2-based installer has everything you would expect. There are pages to select components, to review licenses, and to select installation path. It can create shortcuts, change environment variables and supports uninstall. All components to install are provided in a P2 repository, and you can use bundled P2 repositories as well as remote. So, for example, you can include core functionality in the installer users download, and provide optional components via HTTP, and the installer supports selecting of optional components. It is also extensible using Java, both via P2 touchpoints and using installer modules that contribute custom pages. There's also an utility to pack everything together in an executable file you can download and run.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDAvYfQ9vJOcBofjPE4JY-cixWl71gvjxm_z_L-eN9F0JHUJAvYAx9SC3P1mD3_cwKHmUpmCbjbuG1C3Od_VG75VHvUa1NKjlaeAmLwvfgJd4UOOhg2hhHltXAVAhyphenhyphenwlvRuEnH/s1600/p2-installer-post.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDAvYfQ9vJOcBofjPE4JY-cixWl71gvjxm_z_L-eN9F0JHUJAvYAx9SC3P1mD3_cwKHmUpmCbjbuG1C3Od_VG75VHvUa1NKjlaeAmLwvfgJd4UOOhg2hhHltXAVAhyphenhyphenwlvRuEnH/s1600/p2-installer-post.png" height="256" width="400" /></a></div>
<br />
<br />
We hope it will be useful for anybody building products based on Eclipse platform. Kudos go to Mark Bozeman, Mike Wrighton and Richard Memory who worked hard on this. I am also thankful to management of Mentor Graphics' Embedded Software Division, who supported releasing this project into open source. Enjoy!<br />
<br />
<br />
<br />
<br /></div>
Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com7tag:blogger.com,1999:blog-7558958.post-41595800286012211302009-06-06T21:24:00.006+04:002009-09-19T18:49:46.688+04:00STL VisualizationAs of today, KDevelop can nicely display std::vector<std::string>. I'll probably omit the obvious snapshot, and will point to a <a href="http://lists.kde.org/?l=kdevelop&m=125326438617051&w=2">mailing list post</a> with instructions for trying it. Instead, I'll tell the story of this feature.
<p>
For its entire history, GDB did not have any official way to display types from the C++ Standard Library in a sensible way. Several third-party scripts appeared, written in GDB's internal scripting language. However, they were fairly limited. You had to explicitly run those scripts, and all you got was text output without structure, making robust IDE integration impossible. Also, GDB's scripting language is itself unpleasant, and does not even have access to internal data structures and functions. It was clear that we need a way to write pretty-printers using real scripting language, with full access to GDB data structures, and proper integration with frontend interface.
<p>
The first prototype of Python-based pretty printing was written by myself during free hack slot at a CodeSourcery company meeting. It took maybe 4 hours, if not less, and could display std::string as string automatically. Some 4 hours more lead to the <a href="http://sourceware.org/ml/gdb/2008-02/msg00140.html">first public prototype</a>. This version could automatically display std::vector as "[1,2]". The <a href="http://sourceware.org/ml/gdb/2008-04/msg00189.html">second prototype</a> could finally display elements of std::vector as children, like one would expect in a variables tree of a frontend, and even report when new elements are added to the vector. However, this version took a couple of days of work, exposed a mere 4 functions from GDB to Python, and was a mess internally. It was clearly already outside the "quick hack" range.
<p>
Those prototypes would never turn into anything, were it not for <a href="http://tromey.com/blog/">Tom Tromey</a> and <a href="http://blog.bauermann.eng.br/category/english/">Thiago Bauermann</a>, who started a project to add complete Python scripting to GDB. This is much more ambitious than just pretty-printing. In particular, it includes defining new commands in Python, with full access to GDB internals. You can read more details in a <a href="http://tromey.com/blog/?p=494">post series</a> by Tom.
<p>
Pretty-printing became a part of that large effort, and was greatly improved. One of the most notable change was incremental fetch of children. According to the C++ standard, an object does not exist until its constructor has exited. However, gcc debug info just lists all local variables in a block. A naive pretty-printer, when invoked on such a variable, would likely go into uncharted part of memory trying to fetch all children, and never return. To fix this, the Python pretty-printers were designed to use incremental fetch, using Python iterators, and GDB MI interface was also adjusted to be more incremental (yes, it's a <a href="http://vladimir_prus.blogspot.com/2007/02/debugger-stories-stack-widget.html">trend</a>). Beyond that, we've spend at least 3 weeks iterating on finer details. The GDB patch was finally checked in on Sep 15, and KDevelop4 patch shortly after.
<p>
This is still early implementation, and might have bugs, but now it's out for everybody to try.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com6tag:blogger.com,1999:blog-7558958.post-83687189198254413282009-06-01T14:41:00.004+04:002015-10-27T10:57:39.312+03:00Linking 101<div dir="ltr" style="text-align: left;" trbidi="on"><p>Recently, I see more and more people having trouble with link-time errors—as if such an error is the worst kind of luck and cannot be fixed by mere mortals. There are many possible reasons, including Java as default language in universities, and alarming spread of header-only-philia, but that's for another post. Here, I want to give a simple diagnostic procedure for link-time errors. <br />
Let's lay some groundwork first. If your job is programming in C++, you need to know what the <code>-I</code> and <code>-L</code> options do, and how they are different. Also, given a full path to a library file (with <code>.a</code> or <code>.so</code> or <code>.lib</code> extension), you should be able to link to that file—in two different ways. If you don't know any of the above already, all hope is lost—you might want to consider other occupations. Otherwise, let's look at the diagnosis steps for most common error—<code>'undefined symbol'</code>.</p><p>First, understand where the missing symbol is supposedly defined. Educated guess is usually fine. For example, a symbol named <code>boost::system::foobar</code> is most likely contained in the Boost.System library (and it's surprising how many folks fail to guess so). Then, find how you are supposed to link to that <em>logical</em> component, using documentation for the component or the corresponding Linux package. For example, you might decide to add <code>-lboost_filesystem</code> to the linker command line.</p><p>Second, make sure that used <em>physical</em> library file is the right one, and that the linker is not picking a different version of the library from a directory you don't expect. On Linux, you can use the <code>-t</code> flag for the GNU linker (or use <code>-Wl,-t</code> on gcc command line). This will print full paths for every library used, including those specified with the <code>-lfoo</code> syntax. For static linking, this will also tell which object files from the static libraries were used. If you get error when running the application, you one can use the <code>LD_DEBUG</code> environment variable. If you set that variable to <code>help</code> prior to running your program, you'll get a list of possible values. The most handy value in our case is <code>files</code>. <br />
On Windows, the <code>/VERBOSE:LIB</code> option to the Visual Studio linker will produce comparable diagnostics.</p><p>Third, if you seem to link to the right library, there are three further possibilities. First, maybe the library actually should not include the symbol. This can happen if you use wrong headers during the compilation, and can be debugged by passing the <code>-save-temps</code> option to gcc and checking the generated <code>.ii</code> file. Second, the symbol might be almost there—but slightly different—either using different calling convention (on Windows), or <code>wchar_t</code> mode (also on Windows) or a somewhat different types of parameters, or different namespace. In that case, you'll have to make sure the compilation options of the application match library's requirements. Finally, it could be that the library actually lacks the symbol due to library bug, and you have to complain to maintainer. To distinguish those cases, you need to manually examine the list of library symbols. With gcc, the 'nm' command will do for static libraries, while 'readelf' can be used on shared libraries (Unix only). On Windows, <code>dumpbin.exe /symbols /out:symbols.txt somelib.lib</code> can be used.</p><p>That's it for the common case. Below, I list some relatively common specific problems. The list does not claim to be complete, so if you know some other cases, drop me a line.</p><p><span style="font-weight: bold;">Static linking</span>. For static linking, the order of libraries on the command line matters, so if you don't see the linking grabbing the object file with your symbol, you might want to either reorder the libraries or use the <code>--start-group</code> option. See <a href="http://sourceware.org/binutils/docs/ld/Options.html#Options">ld documentation</a> for details and note that the performance cost of the <code>--start-group</code> option might not be a concern these days.</p><p><span style="font-weight: bold;">References to vtable</span>. The GNU C++ compiler sometimes reports unresolved reference to 'vtable for SomeClass'. This generally is a pure way to say that the first non-inline method of SomeClass is not defined. See <a href="http://gcc.gnu.org/faq.html#vtables">GCC FAQ</a></p><p><span style="font-weight: bold;">Windows DLLs</span>. On Windows, if an application wants to use a function in DLL, then both DLL and the application should record this intention, using <code>__declspec(dllexport)</code> and <code>__declspec(dllimport)</code> pair. If either party does not do so, linker complains. With mingw, a typical error is <code>undefined reference to `_imp___WHATEVER'</code>. It means that the library is static, whereas the applications wants to use shared library.</p><p><span style="font-weight: bold;">Windows import libraries</span>. On Windows, it's not possible to directly link to a DLL. Instead, an import library is created and used—typically by passing <code>/IMPLIB</code> option to the linker. If the linker does not report any errors, but does not produce import library either, it's a sure sign that you have not exported any function from the DLL, and have to check the logic that adds <code>__declspec(dllexport)</code></p><p><span style="font-weight: bold;">64-bit compilation</span>. When building 64-bit applications with GCC, you can get an error that say something about "relocation R_X86_64_32", and suggesting the -fPIC option. The issue here is that 64-bit applications should include only code compiled with -fPIC, and if you link against any static libraries, those libraries should also be compiled with -fPIC. On Windows with Visual Studio, if you try to use 32-bit libraries when building 64-bit application, you won't see any warnings, just undefined references. If you look at symbols at the library, and see exactly the symbols that are reported as undefined, 32/64 mismatch is the most likely reason</div></p>Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com10tag:blogger.com,1999:blog-7558958.post-3202326280648892022009-05-20T20:22:00.004+04:002009-05-20T20:40:29.383+04:00KDevelop error displayFor quite a while I wanted KDevelop to display compilation errors directly inside the editor, as opposed to separate window you have to click in. It works now, as shown below.
<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikIadxU5yI9CHuTCdmZumFTGQbh1twHO3g9XyDH-HVbwzJk5sz9mPn8wi5HfoCGr7n7kpVeUKwuanSNzYx0krpQnnOfUG-WpW93o1yC3eNyNCN7INDyWSMyEbbPyFFom9TnVhq/s1600-h/inline_error.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 400px; height: 288px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikIadxU5yI9CHuTCdmZumFTGQbh1twHO3g9XyDH-HVbwzJk5sz9mPn8wi5HfoCGr7n7kpVeUKwuanSNzYx0krpQnnOfUG-WpW93o1yC3eNyNCN7INDyWSMyEbbPyFFom9TnVhq/s400/inline_error.jpg" alt="" id="BLOGGER_PHOTO_ID_5337942626227726130" border="0" /></a>
This was implemented by Ivan Ruchkin, a student at Moscow State University, who will be defending a term paper about various KDevelop-related work tomorrow. The patches will be posted to appropriate mailing lists right after that.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com14tag:blogger.com,1999:blog-7558958.post-69343887353861125282008-04-19T19:42:00.005+04:002008-04-19T20:30:16.478+04:00Variable tooltips<span style="font-family:georgia;">The most voted-for feature request for KDevelop3 debugger was variable tooltips. I don't think KDevelop3 will ever get them, but KDevelop4 will, as shown below.</span>
<p>
<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQeF2f31aIbg7ZdNAKB49CrV-ZDFNsy1Vw3T3kG63udNMXu1RrnKac7I-crdJz__1VEv0ZY9QU5Fu8y-6DxYbPcw6mTSllw2ztVIbCbR2lRRWttmybeIKtRFBXiFxkrkGOIMLH/s1600-h/variable_tooltip.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQeF2f31aIbg7ZdNAKB49CrV-ZDFNsy1Vw3T3kG63udNMXu1RrnKac7I-crdJz__1VEv0ZY9QU5Fu8y-6DxYbPcw6mTSllw2ztVIbCbR2lRRWttmybeIKtRFBXiFxkrkGOIMLH/s400/variable_tooltip.png" alt="" id="BLOGGER_PHOTO_ID_5190982393839103202" border="0" /></a>Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com9tag:blogger.com,1999:blog-7558958.post-42573841933468862912007-12-21T21:47:00.000+03:002007-12-24T23:24:51.717+03:00Debugger stories: pending breakpointsKDevelop 3.5 has a subtle bug. Sometimes, when you step over a function call, you don't stop on the next line. Instead, the application is resumed until it hits a breakpoint, or exits. This bug, in fact, is consequence of how breakpoints in shared libraries are implemented.
<p>
Suppose you've just started a debugger, and try to set a breakpoint on a function in a shared library. The library itself might not be loaded yet, in which case GDB cannot find the address of the symbol to set the low-level breakpoint. To handle this case, starting with version 6.1, GDB supports <span style="font-style: italic;">pending breakpoints</span>. Such breakpoints don't correspond to any address in program, they only keep the specified breakpoint location as string. Whenever a new shared library is loaded, GDB tries to re-parse breakpoint location again, and if that succeeds, creates an ordinary breakpoint.
<p>
Now, this does not work when using the MI interface, for a couple of reasons:
<ul><li>When a pending breakpoint is resolved, it is deleted, and new one is created. And GDB fails to inform MI frontend about this.
</li><li>It's actually not possible to create pending breakpoint using MI at all.</li></ul><p>
Because of these issues (and a bit of historic reasons) KDevelop 3.5 simulates pending breakpoints. GDB is asked to stop whenever a shared library is loaded, and when that happens, KDevelop tries to reinsert breakpoints. This works pretty well, except for the bug I mention in the beginning. Suppose you're stepping over a function call (this uses the "next" command on GDB level). The function opens some shared library, and which point GDB stops and KDevelop tries to reinsert breakpoints. After that KDevelop would like to continue the "next" operation, but it's already aborted by GDB. All we can do is continue the program.
<p>
But it's not longer the case today. As I wrote <a href="http://vladimir_prus.blogspot.com/2007/11/breakpoints-in-constructors.html"> earlier</a> GDB was recently modified so that a breakpoint can correspond to several addresses, such as of template instantiations. A breakpoint is re-evaluated each time a shared library is loaded or unloaded, and locations are added to breakpoint and removed as appropriate, but it remains the same breakpoint. The nice side effect is that pending breakpoints are now just breakpoints with zero locations, that are reevaluated just like other breakpoints, and don't ever change their number.
<p>
In addition to that, I wrote patches to add pending breakpoint support to MI -- which mainly involved getting rid of two parallel breakpoint-setting code paths -- one for MI and one for CLI. Thanks to review of Joel Brobecker and Daniel Jacobowitz, those patches went in GDB CVS eariler this month. KDevelop 3.5 SVN was modified to automatically detect and use this GDB feature. So, if you're willing to build CVS HEAD of gdb and KDevelop from KDE 3.5 branch, you can finally have breakpoints in shared library just working.
<p>
This was probably my last KDevelop 3.5 commit. KDevelop 4 is ahead.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com0tag:blogger.com,1999:blog-7558958.post-79863611475438721742007-11-26T21:30:00.000+03:002007-11-26T21:37:22.532+03:00Breakpoints in constructorsPresently, no release of GDB properly handles breakpoints in contructors. This summer, I've worked on fixing that, and while it took longer than expected, it was eventually done, just in time for <a href="http://www.codesourcery.com/gnu_toolchains/sgpp">Sourcery G++</a> Fall release. The patches were also submitted for GDB FSF, missed the window for 6.7, but will be present in 6.8 release.
<p>
The underlying problem with breakpoints in constructors was that gcc generates two distinct function bodies for a constructor. One is a regular one that constructs the entire object, including all bases. Another one constructs everything except for virtual base classes. As it happens, gcc emits both constructors even for classes that have no virtual bases at all. GDB was not prepared that a given function name or source line corresponds to several addresses
in program, so it picks one. And usually it picked the wrong one.
<p>
Constructor is the most common case, but is not the only one. If you set a breakpoint in a function template, you can have multiple template instantiations that correspond to a source line. An inline function can be inlined in multiple places, and lead to exactly the same problem.
<p>
The solution, obviously, is to teach GDB that a breakpoint can correspond to several addresses, and then create multiple-location breakpoints when needed. Now, whenever a user creates a breakpoint that resolves to a source line, GDB traverses line tables for all modules, and if it finds another address for the same line, that address is added to breakpoint. For a template or inline function, you can end up with quite a lot of locations, so you can review list of locations, and disable the unwanted ones.
<p>
The nicest bit of this is interaction with shared libraries. Say, you've set a breakpoint inside function template. If you load a new shared library, and it contains an instantiation of that function template, a new location will be added to the breakpoint, transparently. If a library is unloaded, the location will become 'pending', until you load the library back.
<p>
The side effect of this work was a serious improvement in the way breakpoints in shared libraries work, but that's a topic for another post.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com8tag:blogger.com,1999:blog-7558958.post-21735522919819660952007-02-01T00:28:00.000+03:002007-02-01T01:11:52.463+03:00Debugger Stories: Stack widgetIn KDevelop 3.4, the stack widget was not changed much. I can remember just two changes—one that is apparent and one that is subtle.
<p>
The apparent change is that we actually parse gdb output, and show it it a readable way, while in KDevelop 3.3 the stack frame formatting was entirely at mercy of gdb's "backtrace" command.
<p>
<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2-kadH2TGfzhoex2HTcpUbWauw3epTPrgk1Z_Py999BGJmpf00TrMkcCAvtIF5ibfGZ-lR3C4lIuG-zzN0G4K6Am_0t7o_rI92Y-hP89JKGCjKMdn10DtBKCFumwGvjo5A0dS/s1600-h/stack.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2-kadH2TGfzhoex2HTcpUbWauw3epTPrgk1Z_Py999BGJmpf00TrMkcCAvtIF5ibfGZ-lR3C4lIuG-zzN0G4K6Am_0t7o_rI92Y-hP89JKGCjKMdn10DtBKCFumwGvjo5A0dS/s400/stack.png" alt="" id="BLOGGER_PHOTO_ID_5026317622361323042" border="0" /></a>
<p>
The subtle change is at the bottom of the screenshot—that "(click to get more frame)" thing. When a program stops, KDevelop fetches very few frames from gdb. If you click on that last item, then another chunk of frames will be fetched.
<p>
This behaviour is needed for two reasons. First, if your program is stuck in infinite recursion, and you try to interrupt it from KDevelop, in KDevelop 3.3 you're out of luck. As soon as the program is interrupted, KDevelop asks gdb for the list of all frames. Since your program is in infinite recursion, the number of frames is very large, and gdb is not very speedy stack-walker. So, you get to wait 5 mins for the stack to be shown. With incremental display, in a few clicks you'll see what function went astray.
<p>
The second reason is embarassing. Even without infinite recursion, getting the list of frames from gdb takes a lot of time. Something like half-a-second for getting 30 frames is not unheard of. Ideally, we'd fix gdb, but since we need incremental fetch anyway, fetching sufficiently small number of frames initially greatly improves responsiveness.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com3tag:blogger.com,1999:blog-7558958.post-1149068079293802372006-05-31T13:12:00.000+04:002006-06-02T19:45:36.546+04:00Debugger stories: WatchpointsOne of my faviourite debugger changes in KDevelop 3.4 is proper watchpoint handling. Before explaining it, some introduction is due.
<p>
Say you're debugging and see that the 'foo' field of 'pointer_to_some_data' is completely bogus. You are sure that it was valid some time ago, like when its containing object was constructed, so the question is where the corruption happened. That's exactly what watchpoints are for. You set breakpoint at a code where 'foo' is known to be valid, and then ask the debugger to stop whenever the value of 'foo' changes. The debugger in turn writes the address of 'foo' to a special processor register, and processor will call back the operating system, and then debugger, when 'foo' changes.
<p>
Except that GDB does not work this way by default. If you say:
<blockquote><code><pre>
watch pointer_to_some_data->foo
</pre></code></blockquote>
there are two interpretations. First is to stop when memory location referred to by <code>pointer_to_some_data->foo</code> is modified. Second is to stop when the value of the <span style="font-weight:bold;">expression</span> <code>pointer_to_some_data->foo</code> changes, which can happen also if <code>pointer_to_some_data</code> changes. Obviously, when debugging memory corruption, you care about memory address, and <code>pointer_to_some_data</code> is just a way to specify the memory address. Alas, by default GDB uses the second interpretation, so to set watchpoint on address you should use:
<blockquote><code><pre>
print &(pointer_to_some_data->foo)
watch *$
</pre></code></blockquote>
<p>
But the problem is not just that you'll get false hits when <code>pointer_to_some_data</code> changes. The thing is that if that variable is a local one, or a function parameter, then GDB will immediately <span style="font-weight:bold;">remove</span> watchpoint when you exit the containing function. So, for KDevelop user it will be like that: you pick a local variable in a variables widget, you expand it, right-click on some member, select "Toggle watchpoint", and continue. The watchpoint you've just added immediately goes away.
<p>
KDevelop 3.4 solves this problem in a radical way. All watchpoints are address watchpoints. For any expression you enter, address is computed and watchpoint is set on address. Expression without address (rvalue) can't be watched and you'll get an error message if adding watchpoints for rvalue. Additionally, when the application exits, all watchpoints are disabled, because data addresses can well be different on the next run. When user decides to enable a watchpoint, the address of expression is evaluated again, and a new watchpoints is set to that address.
<p>
Hopefully this will make watchpoints more usable for the ordinary programmer.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com1tag:blogger.com,1999:blog-7558958.post-1146554292468856882006-05-02T11:17:00.000+04:002006-05-31T18:42:46.876+04:00Introducing MI branchFor recent months, I was working on an internal reorganization of KDevelop debugger, informally known as "MI branch". Now that it mostly works, it's time to describe the goals and results.
<p>
The original goal was to use a different interface with GDB, called "MI", that's specifically meant for frontends. In MI mode, gdb output can be easily parsed into DOM-like structure, and examined in a nice C++ way, something like:
<blockquote><code><pre>
(*last_stop_result)["value"]["old"].literal()
</pre></code></blockquote>
Before, KDevelop was parsing GDB output intended for humans, and could in some cases misinterpret it. Like thinking that application is running, when it's actually stopped. This unreliability was the primary reason for switching to MI.
<p>
But MI is not a silver bullet. Both John Birch (original author of debugger part), and I had reservation about maturity of MI, which proved true eventually. I'll talk about this later, but basically, using MI does not automatically make debugger better, or faster, or anything, contrary to what many think. However, since using different protocol is a big code change already, I've sneaked in a number of architectural and GUI changes, hopefully for better.
<p>
So in the end MI branch had two goals:
<ol>
<li>Fix all glitches. Do you know that in some cases KDevelop 3.3 shows only half of local variables? Or that setting watchpoint in a natural way is rarely what you want? Or that if a program is stuck in infinite recursion, KDevelop will take minutes to show the stack? Each issue is not very significant in itself, but together they make user experience not confortable.
<li>Cleanup internal architecture. Original architecture was a bit too centralized and adding new features required a lot of work. And since many cool features come to mind, it's better be fixed quick.
</ol>
<p>
The "MI branch" itself is already merged to KDevelop 3.4 branch. In future posts I'll describe all changes the debugger has compared to 3.3 release. Stay tuned.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com4tag:blogger.com,1999:blog-7558958.post-1146133234603649902006-04-27T13:48:00.000+04:002006-07-07T18:45:37.220+04:00Non-constant sizeQuite some time ago, when I was learning <a href="http://www.sgi.com/tech/stl/">STL</a>, all information sources stressed the importance of learning complexity guarantees that methods of various containters make. One specific subtle thing is that the <code>std::list<>::size()</code> method runs in <span style="font-style: italic;">linear</span> time, not in constant time. It was explicitly designed that way for a reason, described in <a href="http://www.sgi.com/tech/stl/FAQ.html">STL faq</a>, but what matters to an ordinary programmer is that testing lists for emptyness should be done with the <code>empty</code> method, not by comparing size to zero, otherwise it's easy to kill performance.
<p>
Today, I run into another case where non-const-time <code>size()</code> matters. I was testing KDevelop on some testcase, and noticed that getting list of stack frames from gdb takes a lot of time. I've added some profiling code, and found that a one command takes 200ms to execute. Adding profiling code to gdb revealed that gdb itself takes some 70ms. Of course, that's not ideal, but even larger fraction of time was apparently spend in KDevelop, ehm, parsing the response.
<p>
So I've quickly put up a testcase that repeatedly parses a specific response, and ran it under <a href="http://kcachegrind.sourceforge.net/cgi-bin/show.cgi">callgrind</a>. Ten minutes later I've got a profile with <code>strlen</code> on top. It turned out that the parsing code was using <code>QCString</code> and calling it's <code>length</code> method at least one for each token, and for certain tokens -- once for each character. The <code>length</code>, in turn, just calls <code>strlen</code>. Since the input string was 20K in size, most of runtime was spend measuring the size of that string.
<p>Another unexpected behaviour was found in the <code>QCString::mid</code> function. Internally, it also calls <code>length</code>, and <code>mid</code> was called once for each token.
<p>After uses of non-const-time methods were reduced to minimum, the parsing time my test case decreased 40x. No so bad, I think. The only problem is that time spend in gdb is still to high for a GUI, and that won't be that easy to fix.</span>Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com4tag:blogger.com,1999:blog-7558958.post-1144319222182036772006-04-06T13:57:00.001+04:002010-09-17T08:17:16.189+04:00Printf debugging 2006One of the oldest methods of debugging is "printf debugging" -- putting various print statements in the code and then staring at the output. That's useful not only if you can't use a debugger. If the program does not crash, but produces wrong results after long computation, it's hard to figure where exactly the problem lies. In that case, printing intermediate data can be a very efficient method.
<p>
The only problem is that after adding print statements the program must be recompiled, and after debugging print statements must be removed. But it's possible to make gdb into printing machine using so called "breakpoint commands". Each breakpoint can have a list of commands that will be executed when breakpoint is hit. The commands can include printing and "continue". Here's a simplified example of gdb script I've used recently:
<blockquote><code><pre>
break main.cpp:1353
commands
print ('lvk::nm_model::NM'*)this
printf "Entering 'run', proc %d\n", $->processor_number
continue
end
run
</pre></code></blockquote>
<p>
After putting this to a file "script", gdb can be run as:
<blockquote><code>
gdb -batch -x script <span style="font-style:italic;">name_of_program</span> > log
</code></blockquote>
producing logs of variable values as certain points of the program.
<p>Starting with version 3.3.0, similar functionality is available in <a href="http://kdevelop.org">KDevelop</a>. Just click on the "Tracing" column in breakpoints window, select variables to print and click OK.
<p align="center">
<img src="http://lvk.cs.msu.su/~ghost/images/tracing_dialog_shadow.png"></img></p>
<p>More screenshots <a href="http://kdevelop.org/graphics/screenshots/3.3/tracing2.png">here</a> and <a href="http://kdevelop.org/graphics/screenshots/3.3/tracing_output.png">here</a>.
This is a beginning, future KDevelop version will allow to specify custom commands for breakpoints.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com118tag:blogger.com,1999:blog-7558958.post-1144161590394391382006-04-04T18:23:00.000+04:002006-04-04T18:41:13.196+04:00Unlucky numbers: 48, 58 and 388One day I've got a remarkable bug report:
<blockquote>We're calling the function from your library, and after 48 successfull calls, it crashes. Can you look into this?</blockquote>Initially, I was curious how did they count that '48'. It turned out that the application was repatedly doing the same action, calling my code alogn the way, and counting the number of repeatitions, so '48' was the exact number, and the thing conistently crashed after 48 calls.
<p>
The bug report did not include any calling code, so I've asked for the code to be sent, and went home, and while in a bus decided that either it's fixed size buffer in the calling code, or resource leak, like file descriptors leak.
<p>
And sure thing, next morning I looked at the only thing where I used plain <code>FILE*</code> in order to use Bison-based parser, and there was missing <code>fclose</code> call. Feeling rather smart, I've sent the fixed version back.
<p>
After 30 minutes new bug report arrived saying that the code fails after 58 calls. And this time, the bug does not reproduces for me. After several tries I found out that for me, the unlucky number is actually 388, so I need to wait a bit to reproduce the bug.
<p>
This was resource leak too, though a subtle one. The library was calling external tool, and modified the <code>PATH</code> environment to make sure the tool is found. As the result, the length of <code>PATH</code> variable steadily increased, and finally some OS limit would be reached. After that the value of <code>PATH</code> becomes completely bogus and the external tool won't be found.
<p>
I'm really glad we have <a href="http://valgrind.org">Valgrind</a> so at least memory leaks don't require any magic to debug.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com8tag:blogger.com,1999:blog-7558958.post-1134804443200721952005-12-17T10:12:00.000+03:002005-12-17T10:32:03.236+03:00Not my bugOften most debugging time is spent of pretty uninteresting bugs.
<p>
For example, most part of last Friday I was staring into debugger backed up by MIPS simulator, trying to figure out why exactly one test of 10 run decided to crash on a MIPS board. Let me first show you the code that was found guilty in the end:
<blockquote><pre><code>
static void itoa_really(unsigned long long value, unsigned int base)
{
static char digits[] = {'0', '1', '2', '3', '4', '5',
'6', '7', '8', '9', 'a', 'b', 'c',
'd', 'e', 'f'};
if (!value)
return;
else {
long rem;
itoa_really(value / base, base);
rem = value % base;
target_putchar(digits[rem]);
}
}
</code></pre></blockquote>
This is extremely basic code, but yes, it did broke everything.
<p>
Initially, the crash happened on a real MIPS board, on a pretty complex test. It was reduced to a
test printing just two 64-bit values, and crash depended on the printed values. Say, printing <code>0</code> worked, but printing <code>0x11223344556677</code> crashed.
It was not possible to debug the problem on the board, and the few floating licences for simulator were in use, so I tried to solve the problem by poking around. At one time I've added an extra line of code before call to "target_putchar", and the line was:
<blockquote><pre><code>
rem = 3;
</code></pre></blockquote>
Strangely, the crash was gone.
<p>At this point I though that maybe there's some really wrong with 64 on 32 division, and 'rem'
can become wrong, so I replace that "rem = 3" line with:
<blockquote><pre><code>
if (rem < 0) { rem = 0; }
if (rem > 15) { rem = 15; }
</code></pre></blockquote>
The crash appeared again.
<p>Another try was:
<blockquote><pre><code>
if (rem < 3) { rem = 3; }
if (rem > 3) { rem = 3; }
</code></pre></blockquote>
and the crash was still there.
<p>Finally, with:
<blockquote><pre><code>
if (rem <= 3) { rem = 3; }
if (rem > 3) { rem = 3; }
</code></pre></blockquote>
everything worked. Of course, all values was printed as "3333.....333", but it did not crash. Completely buffled, I decided simulator is the only help and eventually we've grabbed the license.
<p>A couple of hours later, I found the one-character fix (literally). The linker config file specified mere 1KB of stack size, and when printing too large values, <code>itoa_really</code> recursed too deep, overflowing the stack, and overwriting a bunch of program variables. Changing to 4KB of stack set it all right. And what's even more funny, I never ever touched that linker config file, and have no idea what kind of programs are supposed to work on such stack space diet.Vladimir Prushttp://www.blogger.com/profile/15916897773452934224noreply@blogger.com8