HyperScience

Maxima and IMaxima on Emacs

Sometimes (actually less frequently than I should) I try to do some maths. I tend to do more of it in teaching than in my research, as my research is mostly experimental and usually I’m using someone else’s maths. Most of my research calculations are numerical, and J works really well and efficiently for these sorts of things. However sometimes you need to use analytical maths, and when that happens and you forget the calculus you learned as an undergraduate, then computational algebra systems (CAS) can be very useful.

Probably the most well known CAS is mathematica, which became popular at the time when Macintosh GUI-based applications were coming out, but mathematica was at least in part the outgrowth of existing primarily text-based CASs such as Macsyma, Derive and others. Many of these started their lives as commercial packages, but were either discontinued because of mathematica’s dominance of the commercial space (like MathCad, which I used during my PhD) or were open-sourced. There are several open-source CASs around now, with the most common being maxima, SageMath and Fricas. Here I’m going to be talking about maxima.

Maxima is a descendent of Macsyma, a lisp-based CAS first written in the 1960s and released into the public domain. I have chosen this CAS, as it has good hooks into emacs, in particular a nice plugin that is part of the maxima package called imaxima.

There is another package called wxmaxima that contains a GUI interface to maxima, and many people use this because it has lots of nice icons and is easy to use. However, as an emacs user, I prefer to use the maxima REPL within an emacs buffer, and fortunately maxima comes with a file imaxima.el that allows all the power of maxima to be available from an emacs buffer.

Installation

First, install maxima itself. On linux systems, use your favourite package manager (eg apt on Debian systems). In my case, I use Manjaro, so I use pacman as the package manager. You can see the maxima-based packages using

pacman -Ss maxima

This produces

extra/maxima 5.46.0-2 [installed]
    A sophisticated computer algebra system
extra/maxima-ecl 5.46.0-2 [installed]
    ECL backend for Maxima
extra/maxima-fas 5.46.0-2
    Maxima FAS module for ECL
extra/maxima-sbcl 5.46.0-2 [installed]
   SBCL backend for Maxima

on my system. As I use sbcl, maxima-sbcl provides a back-end for that Lisp interpreter. Installation on my system is done typing

pacman -S maxima maxima-ecl maxima-sbcl

on the command line.

If you have the locate command installed (it can also be installed via pacman on Manjaro) then after typing

sudo updatedb

on the command line, followed by

locate imaxima.el

produces

/usr/share/emacs/site-lisp/maxima/imaxima.el

as an output. Then, to install imaxima, I put the following lines in my init.el file

;-----------------------------------------------------------------------
; imaxima - interactive computational algebra system
;-----------------------------------------------------------------------
(push "/usr/share/emacs/site-lisp" load-path)
(autoload 'imaxima "imaxima" "Maxima frontend" t)
(autoload 'imath "imath" "Interactive Math mode" t)
(setq imaxima-fnt-size "Large")

You may need to change the path based upon your locate command. These commands run the imaxima and imath functions that allow interactive use of imaxima and inline insertion of latex-quality equations in the output.

Examples

Once installed, typing M-x imaxima starts the interactive shell for imaxima. I’m not even going to try to make a maxima tutorial here, as there are already several available. Figure 1 contains an example of calculation of normal unit vectors for a vector function at a given location. Note that the equations are nicely typeset.

imaxima.png

Figure 1: Maxima session showing vector calculations

And of course no demo would be complete without the arbitrary plotting of a pretty 3D function, in this case a hyperboloid.

MaximaFuncPlot.png

Figure 2: Maxima with pretty plot

The M-x maxima command also works if the maxima package is installed (see next section), but the output is in text form, which is not as aesthetically appealing. The text below is the equivalent maxima output to that in Figure 1 for imaxima.

(%i92) r(t) := [t, cos(t), sin(t)];
(%o92)                    r(t) := [t, cos(t), sin(t)]
(%i93) limit(r(t), t, 2, plus);
(%o93)                        [2, cos(2), sin(2)]
(%i94) limit(r(t), t, 3, minus);
(%o94)                        [3, cos(3), sin(3)]
(%i95) diff(r(t), t);
(%o95)                       [1, - sin(t), cos(t)]
(%i96) define(rp(t), diff(r(t), t));
(%o96)                  rp(t) := [1, - sin(t), cos(t)]
(%i97) load(eigen);
(%o97)          /usr/share/maxima/5.46.0/share/matrix/eigen.mac
(%i98) uvect(rp(t));
                     1                           sin(t)
(%o98) [---------------------------, - ---------------------------, 
                2         2                    2         2
        sqrt(sin (t) + cos (t) + 1)    sqrt(sin (t) + cos (t) + 1)
                                                            cos(t)
                                                   ---------------------------]
                                                           2         2
                                                   sqrt(sin (t) + cos (t) + 1)
(%i99) trigsimp(%);
                             1       sin(t)   cos(t)
(%o99)                   [-------, - -------, -------]
                          sqrt(2)    sqrt(2)  sqrt(2)

It does the job, but the text is harder to read, at least for me, than the nicely typeset output.

Other Computer Algebra Systems

In addition to imaxima, there is a package on ELPA called maxima, which installs a major mode for editing maxima files. Also there is an interface to the Fricas CAS, called Frimacs, which I have not used, but which is worth investigating. Fricas is descended from Axiom, another commercial CAS, and is apparently strong at automatic integration of functions.

Conclusion

There is no question, these CASs can be very useful, either for learning algebra and calculus, or for using them in mathematics, physics and engineering applications. They can be used for anything from a simple calculator alternative (although M-x calc is much better for this) to a solver of integrals, derivatives and linear algebra problems, and for plotting of functions. Often the output of calculations is not simplified in the way one might expect when doing it by hand, but there are simplifying commands that can get around those limitations. It’s also a great tool when I can’t remember my integration techniques. I’ll be trying to use it more in my maths calculations in future now that it’s set up.

New Daily Driver: the Odroid N2+

I have always had a soft spot for fanless ARM single-board computers (SBCs) because they are quiet, portable and consume very little power compared to a typical laptop or desktop machine. A typical desktop computer will consume from 100 to 500 Watts of power, while a typical laptop consumes 60 to 90 Watts. An ARM SBC can consume anything from 6 to 30 Watts, which is considerably less than either of the more common formats. They also have less in the way of hardware monitoring than intel-based CPUs, and can run linux, which is my preferred operating system.

Until fairly recently, however, these machines have been too slow to operate well as standard work computers because package availability was sloppy and memory and CPU availability were at the low end of what you would typically need to get the job done. Also graphic processor support, the bane of linux, is particularly bad for these devices, as they tend to have commercial GPU drivers (as phones are their main application market).

As I mostly use open-source software for my research, and my graphical needs are fairly simple, I have proved to my own satisfaction that I can use these boards to do real work, albeit with a performance penalty compared to a modern i5 or i7 intel chip laptop. My first arm machine was an Odroid XU4 which I brought with me on sabbatical and used for writing papers and reports over a 5-month period. The only problem I had with that machine was that it would get into funny states after updating the OS, and it required a fan. Subsequent to this, I purchased a Pinebook Pro, which I could use as a laptop but which was a slower than the XU4, making the experience a little too frustrating to persevere with in the longer term, though I still use it from time to time.

Now Hardkernal, the maker of the odroid machines, has a new ARM64 SBC which is more powerful than the XU4, the Odroid N2+. This device is marketed as an alternative to the Raspberry pi 4 (which I have not used), being more powerful and more expensive. I purchased mine for USD86 with a plastic case, wireless dongle, and 128 GB emmc card (note that if you are going to use a computer seriously, having as much solid state storage as possible is very helpful). The device comes with 4GB of memory which, as the old Rolls-Royce acceleration specs used to say, is `adequate’.

This device uses more power than the XU4, requiring 12V and 2 A, rather than 5V at 3A. But this is not so surprising given the extra speed of the newer device. It also is by default fanless, although a fan is available for high load applications. So far in using it for my work, the heatsink has not got much more than warm. Although the device can apparently be overclocked to 2.4 GHz, I have not attempted overclocking it.

Initially if you purchase the device from the hardkernel web site, the emmc chip comes with ubuntu mate installed as the recommended operating system. As I like manjaro better than ubuntu, after playing around a bit with the default I used etcher to implement a manjaro sway windowing environment that has been compiled specifically for this computer. After a successful install I noticed that the screen I was using with the N2+ (a QHD 32″ lenovo monitor) would flicker randomly, which was very irritating. In case it was a problem with the Wayland system, I installed manjaro XFCE, which uses X11 rather than Wayland, but when I tried the XFCE version of manjaro, the flickering still occurred. So after a couple of unsuccessful installs, I went back to the original Ubuntu mate installation, which does not cause the flicker problem on my monitor, presumably because hardkernel installed the correct graphics drivers.

I really like tiling window managers that you can control via the keyboard (hence my initial desire to use sway), so once I had mate installed and the default user account removed I installed i3. The i3 window manager seems to work really well under ubuntu, and I was able to set things up just the way I like them. One of the things I don’t like about ubuntu and other Debian-based distributions is the slow turnaround time, as several applications require very up-to-date versions to operate properly (like my University’s owncloud server). However I was pleasantly surprised this time that most programs installable by apt were able to work without causing me problems because of their age.

Here is a picture of what the configuration looks like. The image shows emacs, a translucent shell window (terminator, using powershell) and a web browser all open on the same workspace. The little icons on the bottom right show the other three workspaces that can be used.

Figure 1: i3 configuration

Because i3 is pretty lightweight compared to many window managers, the transition between workspaces and switching between applications is very fast. Using the emacs daemon makes editing very fast too. Once you get used to it, the keyboard-driven workflow associated with i3 and emacs is pretty hard to beat.

In total, with the blind alleys caused by trying the other distros, it took about 10 hours to completely set up the N2 the way I want it. Now I can use all the tools that I use on the intel laptop for my research work, and apart from taking a little longer to load programs, I don’t experience much lag at all compared to my i7 laptop. I was able to connect to my cloud service and to run all the codes I need to, either using snaps, apt, or in a couple of cases compiling from source. The experience is no worse than my usual linux installation experience (best described as me trying lots of permutations of random things based upon internet searches until something works). My existing i3 and emacs configurations were basically able to be transferred directly to the new computer with very few changes necessary. Because all my work in progress is on the cloud, this means that I can work on my project either on my laptop or on this SBC with seamless results, as I have the same applications installed on both machines.

In summary, I’m impressed. This blog post was written on it using emacs org2blog.  It’s possible that I might get bored or frustrated and stop using this machine for work, in which case it will be used as a lab device for transferring data from instruments, or as a connected diary device.  But at the moment I can’t much tell the difference between working on this machine and working on my laptop, and that’s a very good sign.  It’s really impressive how far ARM64 support has come in linux.  For a total outlay of less than AUD200, this is a really fun-to-use laptop replacement, provided you have access to a HDMI monitor and as long as you don’t mind shutting it down between moves (because it does not have a battery).  4GB is not a lot of memory, but in linux I have yet to reach a limit that affects my work in terms of available memory.

The small size of this machine means that, with a big rechargeable battery, it could be made into a very nice portable computer provided you have access to a TV or monitor with HDMI support, which is pretty much everything these days.  I have some ideas about form factors that I hope to have time to try out one day…

First Reproducible Research Paper

I have been playing around with reproducible research in org-mode. As an example for students, I have produced a paper written entirely in org-mode and containing all the required calculations within the document itself. The diagram was made using DITAA and the values were calculated and plotted using calls to elisp and python routines. The paper is formatted as if if were a paper in the Springer Journal Shock Waves, as I wanted to demonstrate the ease of using org-mode with a \(\LaTeX\) style for journal papers. It should be possible, though not necessarily straightforward, to change the style to suit whatever form of journal paper was required. I just chose this one because I like the Springer journal format and fonts.

The paper is on the eternal problem of whether your tea will be cooler after 10 minutes if you mix the milk and the tea immediately and then waiting 10 minutes or by waiting and then adding the tea. You’ll have to read the paper to find out the answer! This is a simple enough problem that allows for a demonstration of how equations and figures are generated and presented using the org-mode markup features.

It was an interesting experience to write this paper. It took me around 2 days of full-time work to get it up and running, but now that I’ve done it, the process should be much quicker for an actual paper. It was not always straightforward to get working either: I found the referencing of figures and tables was hit-and-miss. I’m not sure whether this is normal, or something to do with my configuration, but I often found myself looking at and compiling the .tex files produced by org to determine where my labels ended up and why they were sometimes not found. But eventually I did get it working.

My knowledge of python was not really good enough to allow me to do much in the way of calculating and plotting data, and I was not able to work out how to call a python routine to put numbers in a table. I therefore decided to do most of the table calculations in elisp, where the interface to the org-table is quite seamless. Although I find mathematics in lisp a little awkward, I was able to get it all working with a minimum of fuss, and elisp is pretty easy to debug in emacs.

It’s certainly very neat to be able to populate the text with computed numbers that can change whenever the input parameters for the paper change. And having the plots automatically update when the data is changed is also wonderful. For me, this is the way to properly write a paper, even though there is certainly more groundwork that needs to be done to get the paper written. It may not also work so well for papers where there is a lot of computational work, or where commercial gui-based software is used. But most of my papers contain only small calculations using scripting languages called from the command-line, and org-mode is perfect for that workflow.

An annoyance that I was not expecting is that for the Springer journal file, the abstract occurs in the preamble, so I could not just include a bunch of #+LATEX_HEADER: commands. Instead I needed to use a \input command to include the \(\LaTeX\) within the document.

My plan is for this paper to form a template for a document on how to set up emacs and org-mode from scratch in a new linux distribution, so any student gets a head-start in how to make a reproducible research paper. I could have added more bells and whistles, but I deliberately chose a minimal useful set to not cause unnecessary confusion.

Here is the pdf of the paper

Here is the org-mode file

Here is the bibliography file

Here is the \(\LaTeX\) header file

Because of the wordpress limits on file extensions I had to change all but the pdf file to a .txt extension.

Sometime soon I plan to write the installation from scratch document that allows one to go from a new installation of ubuntu to being able to produce this document.

Getting Gnuplot Working in J

I have been trying to get Gnuplot working in J under linux. It should be easy, as there are at least two packages in J for using Gnuplot. However, it appears both of these packages presume that the user is operating the language under windows.

The motivator for this is having a way of visualising data in J. The plot package is what I would usually use, but the plot package seems to be incompatible with the i3 window manager in linux. In JQt when I try to plot something the plot window seems to be generated and then immediately vanishes. I have not been able to work out why. At least in gnuplot the window is generated and stays in place until dismissed. Using, gnuplot, although more work, also generates better-looking plots.

My particular interest is in loading, processing and plotting data files. These are usually in tab-delimited or in comma-delimited form, so let’s make an example file using the following data that gnuplot can easily read.

For this example we will look at comparing two models for the drag on a spherical object travelling at supersonic speed, such as a meteor. If you want to know more about drag on objects travelling at high speed you might want to look at this paper. The drag on a sphere is usually characterised in terms of a non-dimensional quantity known as the drag coefficient, which is the quantity we have tabulated in our data file. The definition of the drag coefficient is

\begin{equation}
C_D = \frac{D}{0.5 \pi \rho u^2 r^2}
\end{equation}

where \(C_D\) indicates the drag coefficient, \(\rho\) is the gas density, \(u\) is the gas velocity and \(r\) is the diameter of the sphere.

This data is saved in the comma-delimited text file ‘DragDataSpheres.csv’. The first column contains the Mach number of the flow, and the second column contains the drag coefficient.

0.22703679008802968, 0.4754504865676933
0.4289222926495784, 0.5029886144003781
0.612450731328545, 0.52777191664627
0.8164462412329196, 0.693557725580885
0.8726990369926486, 0.779240903758345
0.984149624840694, 0.8814834195623001
1.1685642666036478, 0.9643307478710026
1.4622350885781084, 1.0056430036376527
1.7918182355274597, 0.9999307917591553
2.084687254711646, 0.9887089287070719
2.651941628756867, 0.9552155161499962
2.890034857321303, 0.9550838516918039
3.0728458935037084, 0.9328629423630395
3.201345340681785, 0.9521467214705908
3.347653249833309, 0.9382409290784333
3.494045559278546, 0.9298650439303529
3.6039347416929006, 0.9298042757188796
3.7687685153144335, 0.9297131234016695
3.842365571432188, 0.951792240236996
3.97018981626056, 0.9268367613919297
4.097971860942075, 0.8991163289248247
4.372905817712245, 0.9127891765063343
4.83077741110539, 0.9125359756251952
5.05042917549353, 0.9041195783361327
5.251766076145945, 0.8957133090823157
5.892955107483774, 0.9064186423368754
6.5339753382341765, 0.9060641611032807
6.9369445405670005, 0.908606297949917
7.339913742899824, 0.9111484347965532
7.852476726619007, 0.8942751280774458
8.786872378315877, 0.9158782272562309
9.556138855363217, 0.918217803397956
9.775537418870218, 0.8932116843766618

There is an empirical formula to fit this data, and it is given by a sum of two exponentials:

\begin{equation}
C_D (M) = 2.1 e^{-1.2(M+0.35)}-8.9e^{-2.2(M+0.35)} + 0.92
\end{equation}

In hypersonic theory there is an approximate method for calculating drag, due to the great Lester Lees, called the modified Newtonian method. There’s an interesting story to Newtonian methods. As the name suggests, the model was first postulated by Isaac Newton as a way of measuring drag forces at subsonic speeds. The fluid was considered as a series of particles that hit the object like bullets fired from a gun. When the particles collide with a surface they slip along the surface, only imparting the perpendicular component of their momentum to the object. This model never really took off because it was very inaccurate at subsonic speeds. However the approximation becomes pretty good at hypersonic speeds, and so is often used as a simple way of determining the drag force, because it has the advantage of being simple to determine analytically. So I guess that proves that Newton really was ahead of his time. Lees took Newton’s idea and made a modifying calibration factor that removes an offset inherent to the model by fitting the relationship to the pressure coefficient at the stagnation point of the object, which makes it quite accurate. For a sphere, the drag can be computed using the Modified Newtonian method using the equation

\begin{equation}
C_D = \frac{1}{2} \left( \frac{2}{\gamma M^2} \left( \left[ \frac{(\gamma + 1)^2 M^2}{4 \gamma M^2 -2(\gamma-1)} \right]^{\frac{\gamma}{\gamma-1}} \left[ \frac{1-\gamma + 2 \gamma M^2}{\gamma+1} \right] -1 \right) \right)
\end{equation}

as explained in this paper.

We would like to prepare the data in J, then plot it and save it to a graphical file using Gnuplot. First, let’s just plot the first set of data from the file. The following J code sets the directory and then loads the CSV file columns into the variables M_data and CD_data.

1!:44 '/home/sean/ownCloud/Org/blog/Gnuplot/'
load 'trig plot files csv'

NB. Read the data
a =: 5 }. readcsv 'DragDataSpheres.csv'
'M_data CD_data' =: 0 ". each |:(0,1){"0 1 1 a

text =: 0 : 0
set term qt
set style data points
set datafile separator ","
set pointsize 2
plot "DragDataSpheres.csv" using 1:2 pt 6
set ylabel 'Drag coefficient'
set xlabel 'Mach number'
set nokey
pause 5
set term png
set out 'testplot.png'
rep
set out ''
set term qt
quit
)

text fwrites 'testplot.gnu'
2!:0 'gnuplot < testplot.gnu'

testplot.png

Figure 1: Drag coefficient plot

We put all of our gnuplot instructions in an adjective definition called ‘text’. The ‘pause 5’ instruction pauses for 5 seconds to give us enough time to examine the plot before it goes away. Now that we have this, we can plot our two functions

NB. C_D (M) = 2.1 e^{-1.2(M+0.35)}-8.9e^{-2.2(M+0.35)} + 0.92
CD_Fit =: (2.1 *  (^_1.2*(M_data+0.35))) - (8.9* (^_2.2*(M_data+0.35)))) + 0.92
CD_Fit

gamma =: 1.4
term1 =. 2%gamma * *: M_data
term2 =. (((*: M_data)*(*:gamma+1))%((4*gamma* *: M_data)-2*gamma-1))^gamma%gamma-1
term3 =. ((1-gamma) + 2*gamma* *: M_data)%(gamma+1)

CD_Mod_Newt =: 0.5*term1*(term2*term3)-1

(|:M_data,CD_data,CD_Fit,:CD_Mod_Newt) writecsv 'Compar.csv'

Note that I split the more complex modified Newtonian form into three separate terms before combining them. This eases debugging and also makes the probability of incorrectly parsing the right-to-left evaluation of the J code lower.

The three plots can be plotted together using the following gnuplot command…

text =: 0 : 0
set term qt
set style data points
set datafile separator ","
set pointsize 2
plot "Compar.csv" using 1:2 pt 6 t 'data', '' using 1:3 with lines lw 2 t 'empirical fit', '' using 1:4 with lines lt 4 lw 2 dashtype 2 t 'modified Newtonian'
set ylabel 'Drag coefficient'
set xlabel 'Mach number'
set yrange [0:1.2]
pause 10
set term pngcairo
set out 'Compar.png'
rep
set out ''
set term qt
quit
)

Compar2.png

Figure 2: Drag coefficient plot

Note that it’s better to use the pngcairo terminal type than png. For some reason the latter will not do dashed lines, whereas the former will.

Physically, for those interested, it shows that the modified Newtonian theory is not as good a fit as the empirical curve fit at low Mach number, but it fits the behaviour well at Mach number of 5 or greater. The curve fit won’t help you at all if the ratio of specific heats \(\gamma\) is not 1.4, but the modified Newtonian method can account for that, which makes it very useful in predicting drag.

To summarise: by loading data, doing the appropriate calculations and saving the data to a text file, we can plot the data from J using a system call to gnuplot, which will plot the raw data and any manipulations we choose to perform, combining the power of J for data analysis with the rich plotting functionality of gnuplot.

NB. Here is the full J source code

Org-babel for J

As part of my emacs org-mode work flow, I have been using org-babel for a while. This allows you to insert code blocks into org buffers and have those blocks be executed when your file is compiled. This is a really handy method for doing reproduceable research. For example, you can call the source code function in R to do the statistical calculations for data in a table. If the data in the table changes, so will the calculation of the output data change. This prevents the perennial problem of having data in one file (typically a spreadsheet) and not knowing whether the document you generated for a paper used the 12th of September or the 15th of September version of the spreadsheet. By having explicit links to data and to the algorithm that manipulates that data, you can explicitly record the calculations you used to produce your data. And so can anyone else if they want to. This is very important for producing believeable data.

Org-babel is built into emacs org-mode, and supports an amazing array of programming languages, from compiled languages like C to interpreted languages like python or MATLAB, to specialised scripting languages like awk or gnuplot.

The best feature for me is that org-mode can read from or write to org tables, allowing a seamless integration between code and document. However, this capability differs between programming languages. Some languages, like python and common lisp, seem to be very well catered for in this regard. However my favourite programming language, J, is rather less well catered for. In particular, there does not seem to be a built-in way to pass variables to and from the code block. Instead, you can run your code as if it were a script, and the source block will provide the last calculated value as an output. For example,

#+BEGIN_SRC j :exports both
 NB. The square root of the sum of the squares of the numbers
 NB. between 1 and 10
	    [a =: %: +/ *: 1 + i.10
#+END_SRC
#+RESULTS:
: 19.6214

The output, as stated by the comment, produces the Euclidean norm of the integers between 1 and 10 inclusive, which is 19.6214, and displays it as the result from evaluating the source block. However, for other programming languages one could supply a variable argument using the :var command in the header, to pass a variable argument to a function. So, for example, the 10 in the example above could be replaced by each of the values in the column of a table.

Like most things in emacs, the code for executing commands in code blocks is available as elisp. So, in theory, it should be possible to modify the existing elisp export code to pass variables, including rows and columns in tables, to a J function. At the moment though, my understanding of elisp is not sufficiently good to be able to work out how to do this, but it sounds like a very useful thing to do, and necessary if J is to be seriously used from within org-mode. If anyone has managed to do this, I’d be very interested to know how it’s done. If not, I’ll need to learn some more elisp and try to reverse engineer how it’s already been done for MATLAB code to see if I can do something equivalent for J.

Oh, and happy new year for 2021. I wanted to get one more blog entry done before the end of 2020, as an old-year’s resolution…

Marking pdf Assignments in Emacs

Most of the undergraduate student assignments I need to mark are numerical or analytical in nature.  The assignments are for small classes, and are often written on paper, and even if submitted electronically they are relatively easy to mark up on a tablet, because the answer is either correct or it isn’t.  But I recently had to mark a large number of assignments, for a course on the social context of engineering, where answers to questions are in paragraph form, and are fairly open-ended.  This requires explanation on my part for why marks were not allocated and suggestions on what would have made the answers better.

First time I did this I used a tablet and pen, and although it worked, it was very time consuming and messy to add comments.  Using Adobe Acrobat under Windows was not better, as I find the comment interface unwieldy when I have to use it a lot.  The most tedious aspect of this was that many of the comments were the same for large numbers of students, requiring you to type the same thing over and over again: this year I had around 75 students, with 6 pages and on average around 6 individual comments per page – a total of 2700 comments! Many of the mistakes were common to several students in the class.

In an attempt to improve the experience of marking these assignments, I looked at doing the marking in emacs, using two very simple tools: the pdf-tools package maintained by Andreas Politz, and the built-in registers of the editor.  If you do much marking up of pdfs, this may be of interest to you.

If you use emacs, and don’t currently use pdf-tools, you should.  It’s an easy install from the melpa archive, and it is a lot faster than the default pdf viewer DocView.  Most importantly, it has commands that are useful for marking up and annotating pdf files.  I have added the following commands to my init.el file to allow single-key commands in pdf-tools such as ‘h’ for highlight text, ‘s’ for strikeout and ‘t’ for text comment, as well as ‘d’ to delete existing comments.  I also have changed the default strikeout line from red to green, because red is so judgemental…

configuration of pdf-tools

Configuration file settings for pdf-tools

When this is loaded, opening the pdf file provides a view just like that of a normal pdf-viewer, scaled to fit in the emacs window.  If I select text and hit ‘h’ I highlight that text in the pdf and get to edit a comment on the highlighted text in the minibuffer.  Pressing C-c C-c exits from the minibuffer and the comment can be seen when the cursor rolls over the highlighted text in the pdf.  Pressing ‘t’ does the same, but produces an icon in the pdf instead of the highlighted text.

Annotating in pdf-tools

Annotating in pdf-tools

Once mastered, this is a very quick way of adding comments or deleting text is very fast, as only a few keystrokes are required to add annotations, which are automatically timestamped and user stamped.  Note that I changed my user designation to anonymous here, as sometimes (eg if using for annotating review papers) it’s not desirable to identify.  By default it’s your username.

As I hinted previously, what makes this even more useful is the effectiveness in combination with emacs’ registers.  I had not really used these much previously, as I had no idea what one was supposed to do with them, but populating many pdfs with identical comments is one example of where they can be useful.

While editing a comment in the minibuffer, you can select it and type C-x-r-s-1 and it will save that text to register 1.  Then the next time you want to insert that text in a comment, you type C-x-r-i-1 and it will insert the text.  The box below gives the general idea.

Insert comment from register

Using emacs registers to recycle comments

The process can then be repeated as many times as necessary when needed in a comment.  The usual C-x C-s will save the pdf for you.  Another handy command is C-c C-a l, which lists the annotations.

This procedure is not flawless: there is a bug that comes up every so often (but not always) when editing existing annotations with pdf-tools, where the link to the animation is lost and the updated animation cannot be edited.  If this happens, kill the annotation minibuffer and make and delete another comment, then try again.  This works for me.  The only other thing I’d like to see from the excellent pdf-tools package for marking up documents is a carat symbol for insertion of text.  But I can use it perfectly well in its current form.

You can find out more about registers at https://www.emacswiki.org/emacs/Registers.

The source for pdf-tools is currently at https://github.com/politza/pdf-tools.

 

 

Research notes

One of the problems I have been engaged with for many years now is the idea of how to keep a consistent record of my research that I can come back to years later and pick up where I left off.

Having thought a lot about this, I have some important criteria for such a record.  Lab notes must be

  1. Future-proof (not dependent on proprietary software);
  2. Based fundamentally on ASCII text;
  3. Flexible enough to cope with a variety of experiment structures (one-off experiments, repetitive experiments, numerical experiments etc);
  4. Easy to input and maintain;
  5. Able to time-stamp the state of an experiment at the time it was performed;
  6. Able to incorporate the source code used to analyse the data;
  7. Able to facilitate collaboration with my group members and external colleagues;
  8. Easy to transform into outputs like papers and presentations;

I have tried many different notetaking systems to achieve this, including commercial systems like LabArchives, Microsoft OneNote, Microsoft SharePoint, My own TikiWiki webpage, Zim desktop wiki, Tiddlywiki and others I have forgotten, but none of them could give me what I need according to the above 8 points.  I believe I now have a system that works the way I need it to for lab notes, though I know it is not for everyone: emacs org-mode.  I hope to write up a little more about org-mode in the future, and why I think it’s a great way to produce consistently high-quality, self-documenting research outputs.