Org-mode Signature Code

I’ve been trying to learn a little elisp lately, and have written a useful routine that I thought someone might like. It’s interesting because it uses both elisp and org-mode tables.

Lately I have been setting up mu4e to read my personal email. As part of using that I wanted to have a signature that would generate a wise saying from a country, but to make it a bit more interesting I wanted the signature to choose a randomly selected saying from a different country each time I generated the signature.

I have an org-mode file that contains a two-column table. The first column contains the country and the second contains the quote.

#+NAME: SayingsTab
| Afghanistan           | Don't show me the palm tree, show me the dates                                      |
| Afghanistan           | No one says his own buttermilk is sour                                              |
| Afghanistan           | What you see in yourself is what you see in the world                               |
| Africa                | The tree that has been axed will never forget                                       |
| Albania               | Don't put gold buttons on a torn coat                                               |
| Albania               | Patience is the key to paradise                                                     |
| Albania               | When you have no companion, consult your walking stick                              |
| Algeria               | Do bad and remember, do good and forget                                             |
| America               | A pessimist is a person who has lived with an optimist                              |
| Armenia               | You cannot start a fast with baklava in your hand                                   |
| Australia             | There are none so deaf as those who will not hear                                   |
#+name: quote
#+begin_src elisp  :exports results :var sayings=SayingsTab
	(defun genquote()
      "Generate a random quote from an org-mode table.
  The table contains the country name in col 0 and the quote in col 1.
  Add more quotes to the table as needed, the function
 selects from the current number of rows."
   (let ((selection  (nth (random (length sayings)) sayings)))
	  (message "As they say in %s, \"%s\"." (car selection) (cadr selection))))

Running the source block here generates a random quote from the list in the table as a message. Hitting C-c C-c in the code will run it and generate a quote like

As they say in America, "A pessimist is a person who has lived with an optimist".

I like using the org-table because the new items can be sorted alphabetically, and its easy to add new sayings when I find them.

What I’d like to be able to do is to load the org table using an elisp function and then do the same thing as part of a signature-generating function. Currently I can’ do that, but I can read the text from a tab-delimited text file. Fortunately, I can generate the tabular data that can be read by elisp by putting the point at the table and using the M-x org-table-export command. I save the table to a tab-delimited file called sayings.txt whenever I add to the org-table.

Then in my init.el file, I define the following elisp function to generate an email signature using the data in the text file.

(defun signature()
   "Generates a random quote from a tab-delimited file for use as an email signature quote. 
	       Table of quotes is in ~/ownCloud/Org/sayings.txt
	       The tab-delimited table contains the country name in col 0 and the quote in col 1.
	       Add more quotes to the table as needed, the function
	       selects from the current number of rows."
   (setq sayings (read-lines "~/ownCloud/Org/sayings.txt"))
   (setq sel (nth (random (length sayings)) sayings))
   (insert "\n-- \n" "John Doe \n \n" "john.doe@google.com \n" "Ph: 867 5309 \n")
   (insert "As they say in ")
   (insert (remove-trailing (car (split-string sel "\t" t))) ", '")
   (insert (remove-trailing (cadr (split-string sel "\t" t))) "'. \n")

This should produce something like

John Doe 
Ph: 867 5309 
As they say in Malaysia, 'One buffalo brings mud and all the herd are smeared with it'. 

where the last line is the randomly generated quote. So then, when I’m editing my email, I just need to call the signature function when editing an email in mu4e and I get my signature with a new quote every time.

If you know how to read direct from the org-table, rather than from a text file, I’d love to know how you do it, as that would remove the need to update the table from time to time.

Maxima and IMaxima on Emacs

Sometimes (actually less frequently than I should) I try to do some maths. I tend to do more of it in teaching than in my research, as my research is mostly experimental and usually I’m using someone else’s maths. Most of my research calculations are numerical, and J works really well and efficiently for these sorts of things. However sometimes you need to use analytical maths, and when that happens and you forget the calculus you learned as an undergraduate, then computational algebra systems (CAS) can be very useful.

Probably the most well known CAS is mathematica, which became popular at the time when Macintosh GUI-based applications were coming out, but mathematica was at least in part the outgrowth of existing primarily text-based CASs such as Macsyma, Derive and others. Many of these started their lives as commercial packages, but were either discontinued because of mathematica’s dominance of the commercial space (like MathCad, which I used during my PhD) or were open-sourced. There are several open-source CASs around now, with the most common being maxima, SageMath and Fricas. Here I’m going to be talking about maxima.

Maxima is a descendent of Macsyma, a lisp-based CAS first written in the 1960s and released into the public domain. I have chosen this CAS, as it has good hooks into emacs, in particular a nice plugin that is part of the maxima package called imaxima.

There is another package called wxmaxima that contains a GUI interface to maxima, and many people use this because it has lots of nice icons and is easy to use. However, as an emacs user, I prefer to use the maxima REPL within an emacs buffer, and fortunately maxima comes with a file imaxima.el that allows all the power of maxima to be available from an emacs buffer.


First, install maxima itself. On linux systems, use your favourite package manager (eg apt on Debian systems). In my case, I use Manjaro, so I use pacman as the package manager. You can see the maxima-based packages using

pacman -Ss maxima

This produces

extra/maxima 5.46.0-2 [installed]
    A sophisticated computer algebra system
extra/maxima-ecl 5.46.0-2 [installed]
    ECL backend for Maxima
extra/maxima-fas 5.46.0-2
    Maxima FAS module for ECL
extra/maxima-sbcl 5.46.0-2 [installed]
   SBCL backend for Maxima

on my system. As I use sbcl, maxima-sbcl provides a back-end for that Lisp interpreter. Installation on my system is done typing

pacman -S maxima maxima-ecl maxima-sbcl

on the command line.

If you have the locate command installed (it can also be installed via pacman on Manjaro) then after typing

sudo updatedb

on the command line, followed by

locate imaxima.el



as an output. Then, to install imaxima, I put the following lines in my init.el file

; imaxima - interactive computational algebra system
(push "/usr/share/emacs/site-lisp" load-path)
(autoload 'imaxima "imaxima" "Maxima frontend" t)
(autoload 'imath "imath" "Interactive Math mode" t)
(setq imaxima-fnt-size "Large")

You may need to change the path based upon your locate command. These commands run the imaxima and imath functions that allow interactive use of imaxima and inline insertion of latex-quality equations in the output.


Once installed, typing M-x imaxima starts the interactive shell for imaxima. I’m not even going to try to make a maxima tutorial here, as there are already several available. Figure 1 contains an example of calculation of normal unit vectors for a vector function at a given location. Note that the equations are nicely typeset.


Figure 1: Maxima session showing vector calculations

And of course no demo would be complete without the arbitrary plotting of a pretty 3D function, in this case a hyperboloid.


Figure 2: Maxima with pretty plot

The M-x maxima command also works if the maxima package is installed (see next section), but the output is in text form, which is not as aesthetically appealing. The text below is the equivalent maxima output to that in Figure 1 for imaxima.

(%i92) r(t) := [t, cos(t), sin(t)];
(%o92)                    r(t) := [t, cos(t), sin(t)]
(%i93) limit(r(t), t, 2, plus);
(%o93)                        [2, cos(2), sin(2)]
(%i94) limit(r(t), t, 3, minus);
(%o94)                        [3, cos(3), sin(3)]
(%i95) diff(r(t), t);
(%o95)                       [1, - sin(t), cos(t)]
(%i96) define(rp(t), diff(r(t), t));
(%o96)                  rp(t) := [1, - sin(t), cos(t)]
(%i97) load(eigen);
(%o97)          /usr/share/maxima/5.46.0/share/matrix/eigen.mac
(%i98) uvect(rp(t));
                     1                           sin(t)
(%o98) [---------------------------, - ---------------------------, 
                2         2                    2         2
        sqrt(sin (t) + cos (t) + 1)    sqrt(sin (t) + cos (t) + 1)
                                                           2         2
                                                   sqrt(sin (t) + cos (t) + 1)
(%i99) trigsimp(%);
                             1       sin(t)   cos(t)
(%o99)                   [-------, - -------, -------]
                          sqrt(2)    sqrt(2)  sqrt(2)

It does the job, but the text is harder to read, at least for me, than the nicely typeset output.

Other Computer Algebra Systems

In addition to imaxima, there is a package on ELPA called maxima, which installs a major mode for editing maxima files. Also there is an interface to the Fricas CAS, called Frimacs, which I have not used, but which is worth investigating. Fricas is descended from Axiom, another commercial CAS, and is apparently strong at automatic integration of functions.


There is no question, these CASs can be very useful, either for learning algebra and calculus, or for using them in mathematics, physics and engineering applications. They can be used for anything from a simple calculator alternative (although M-x calc is much better for this) to a solver of integrals, derivatives and linear algebra problems, and for plotting of functions. Often the output of calculations is not simplified in the way one might expect when doing it by hand, but there are simplifying commands that can get around those limitations. It’s also a great tool when I can’t remember my integration techniques. I’ll be trying to use it more in my maths calculations in future now that it’s set up.

New Daily Driver: the Odroid N2+

I have always had a soft spot for fanless ARM single-board computers (SBCs) because they are quiet, portable and consume very little power compared to a typical laptop or desktop machine. A typical desktop computer will consume from 100 to 500 Watts of power, while a typical laptop consumes 60 to 90 Watts. An ARM SBC can consume anything from 6 to 30 Watts, which is considerably less than either of the more common formats. They also have less in the way of hardware monitoring than intel-based CPUs, and can run linux, which is my preferred operating system.

Until fairly recently, however, these machines have been too slow to operate well as standard work computers because package availability was sloppy and memory and CPU availability were at the low end of what you would typically need to get the job done. Also graphic processor support, the bane of linux, is particularly bad for these devices, as they tend to have commercial GPU drivers (as phones are their main application market).

As I mostly use open-source software for my research, and my graphical needs are fairly simple, I have proved to my own satisfaction that I can use these boards to do real work, albeit with a performance penalty compared to a modern i5 or i7 intel chip laptop. My first arm machine was an Odroid XU4 which I brought with me on sabbatical and used for writing papers and reports over a 5-month period. The only problem I had with that machine was that it would get into funny states after updating the OS, and it required a fan. Subsequent to this, I purchased a Pinebook Pro, which I could use as a laptop but which was a slower than the XU4, making the experience a little too frustrating to persevere with in the longer term, though I still use it from time to time.

Now Hardkernal, the maker of the odroid machines, has a new ARM64 SBC which is more powerful than the XU4, the Odroid N2+. This device is marketed as an alternative to the Raspberry pi 4 (which I have not used), being more powerful and more expensive. I purchased mine for USD86 with a plastic case, wireless dongle, and 128 GB emmc card (note that if you are going to use a computer seriously, having as much solid state storage as possible is very helpful). The device comes with 4GB of memory which, as the old Rolls-Royce acceleration specs used to say, is `adequate’.

This device uses more power than the XU4, requiring 12V and 2 A, rather than 5V at 3A. But this is not so surprising given the extra speed of the newer device. It also is by default fanless, although a fan is available for high load applications. So far in using it for my work, the heatsink has not got much more than warm. Although the device can apparently be overclocked to 2.4 GHz, I have not attempted overclocking it.

Initially if you purchase the device from the hardkernel web site, the emmc chip comes with ubuntu mate installed as the recommended operating system. As I like manjaro better than ubuntu, after playing around a bit with the default I used etcher to implement a manjaro sway windowing environment that has been compiled specifically for this computer. After a successful install I noticed that the screen I was using with the N2+ (a QHD 32″ lenovo monitor) would flicker randomly, which was very irritating. In case it was a problem with the Wayland system, I installed manjaro XFCE, which uses X11 rather than Wayland, but when I tried the XFCE version of manjaro, the flickering still occurred. So after a couple of unsuccessful installs, I went back to the original Ubuntu mate installation, which does not cause the flicker problem on my monitor, presumably because hardkernel installed the correct graphics drivers.

I really like tiling window managers that you can control via the keyboard (hence my initial desire to use sway), so once I had mate installed and the default user account removed I installed i3. The i3 window manager seems to work really well under ubuntu, and I was able to set things up just the way I like them. One of the things I don’t like about ubuntu and other Debian-based distributions is the slow turnaround time, as several applications require very up-to-date versions to operate properly (like my University’s owncloud server). However I was pleasantly surprised this time that most programs installable by apt were able to work without causing me problems because of their age.

Here is a picture of what the configuration looks like. The image shows emacs, a translucent shell window (terminator, using powershell) and a web browser all open on the same workspace. The little icons on the bottom right show the other three workspaces that can be used.

Figure 1: i3 configuration

Because i3 is pretty lightweight compared to many window managers, the transition between workspaces and switching between applications is very fast. Using the emacs daemon makes editing very fast too. Once you get used to it, the keyboard-driven workflow associated with i3 and emacs is pretty hard to beat.

In total, with the blind alleys caused by trying the other distros, it took about 10 hours to completely set up the N2 the way I want it. Now I can use all the tools that I use on the intel laptop for my research work, and apart from taking a little longer to load programs, I don’t experience much lag at all compared to my i7 laptop. I was able to connect to my cloud service and to run all the codes I need to, either using snaps, apt, or in a couple of cases compiling from source. The experience is no worse than my usual linux installation experience (best described as me trying lots of permutations of random things based upon internet searches until something works). My existing i3 and emacs configurations were basically able to be transferred directly to the new computer with very few changes necessary. Because all my work in progress is on the cloud, this means that I can work on my project either on my laptop or on this SBC with seamless results, as I have the same applications installed on both machines.

In summary, I’m impressed. This blog post was written on it using emacs org2blog.  It’s possible that I might get bored or frustrated and stop using this machine for work, in which case it will be used as a lab device for transferring data from instruments, or as a connected diary device.  But at the moment I can’t much tell the difference between working on this machine and working on my laptop, and that’s a very good sign.  It’s really impressive how far ARM64 support has come in linux.  For a total outlay of less than AUD200, this is a really fun-to-use laptop replacement, provided you have access to a HDMI monitor and as long as you don’t mind shutting it down between moves (because it does not have a battery).  4GB is not a lot of memory, but in linux I have yet to reach a limit that affects my work in terms of available memory.

The small size of this machine means that, with a big rechargeable battery, it could be made into a very nice portable computer provided you have access to a TV or monitor with HDMI support, which is pretty much everything these days.  I have some ideas about form factors that I hope to have time to try out one day…

Getting Gnuplot Working in J

I have been trying to get Gnuplot working in J under linux. It should be easy, as there are at least two packages in J for using Gnuplot. However, it appears both of these packages presume that the user is operating the language under windows.

The motivator for this is having a way of visualising data in J. The plot package is what I would usually use, but the plot package seems to be incompatible with the i3 window manager in linux. In JQt when I try to plot something the plot window seems to be generated and then immediately vanishes. I have not been able to work out why. At least in gnuplot the window is generated and stays in place until dismissed. Using, gnuplot, although more work, also generates better-looking plots.

My particular interest is in loading, processing and plotting data files. These are usually in tab-delimited or in comma-delimited form, so let’s make an example file using the following data that gnuplot can easily read.

For this example we will look at comparing two models for the drag on a spherical object travelling at supersonic speed, such as a meteor. If you want to know more about drag on objects travelling at high speed you might want to look at this paper. The drag on a sphere is usually characterised in terms of a non-dimensional quantity known as the drag coefficient, which is the quantity we have tabulated in our data file. The definition of the drag coefficient is

C_D = \frac{D}{0.5 \pi \rho u^2 r^2}

where \(C_D\) indicates the drag coefficient, \(\rho\) is the gas density, \(u\) is the gas velocity and \(r\) is the diameter of the sphere.

This data is saved in the comma-delimited text file ‘DragDataSpheres.csv’. The first column contains the Mach number of the flow, and the second column contains the drag coefficient.

0.22703679008802968, 0.4754504865676933
0.4289222926495784, 0.5029886144003781
0.612450731328545, 0.52777191664627
0.8164462412329196, 0.693557725580885
0.8726990369926486, 0.779240903758345
0.984149624840694, 0.8814834195623001
1.1685642666036478, 0.9643307478710026
1.4622350885781084, 1.0056430036376527
1.7918182355274597, 0.9999307917591553
2.084687254711646, 0.9887089287070719
2.651941628756867, 0.9552155161499962
2.890034857321303, 0.9550838516918039
3.0728458935037084, 0.9328629423630395
3.201345340681785, 0.9521467214705908
3.347653249833309, 0.9382409290784333
3.494045559278546, 0.9298650439303529
3.6039347416929006, 0.9298042757188796
3.7687685153144335, 0.9297131234016695
3.842365571432188, 0.951792240236996
3.97018981626056, 0.9268367613919297
4.097971860942075, 0.8991163289248247
4.372905817712245, 0.9127891765063343
4.83077741110539, 0.9125359756251952
5.05042917549353, 0.9041195783361327
5.251766076145945, 0.8957133090823157
5.892955107483774, 0.9064186423368754
6.5339753382341765, 0.9060641611032807
6.9369445405670005, 0.908606297949917
7.339913742899824, 0.9111484347965532
7.852476726619007, 0.8942751280774458
8.786872378315877, 0.9158782272562309
9.556138855363217, 0.918217803397956
9.775537418870218, 0.8932116843766618

There is an empirical formula to fit this data, and it is given by a sum of two exponentials:

C_D (M) = 2.1 e^{-1.2(M+0.35)}-8.9e^{-2.2(M+0.35)} + 0.92

In hypersonic theory there is an approximate method for calculating drag, due to the great Lester Lees, called the modified Newtonian method. There’s an interesting story to Newtonian methods. As the name suggests, the model was first postulated by Isaac Newton as a way of measuring drag forces at subsonic speeds. The fluid was considered as a series of particles that hit the object like bullets fired from a gun. When the particles collide with a surface they slip along the surface, only imparting the perpendicular component of their momentum to the object. This model never really took off because it was very inaccurate at subsonic speeds. However the approximation becomes pretty good at hypersonic speeds, and so is often used as a simple way of determining the drag force, because it has the advantage of being simple to determine analytically. So I guess that proves that Newton really was ahead of his time. Lees took Newton’s idea and made a modifying calibration factor that removes an offset inherent to the model by fitting the relationship to the pressure coefficient at the stagnation point of the object, which makes it quite accurate. For a sphere, the drag can be computed using the Modified Newtonian method using the equation

C_D = \frac{1}{2} \left( \frac{2}{\gamma M^2} \left( \left[ \frac{(\gamma + 1)^2 M^2}{4 \gamma M^2 -2(\gamma-1)} \right]^{\frac{\gamma}{\gamma-1}} \left[ \frac{1-\gamma + 2 \gamma M^2}{\gamma+1} \right] -1 \right) \right)

as explained in this paper.

We would like to prepare the data in J, then plot it and save it to a graphical file using Gnuplot. First, let’s just plot the first set of data from the file. The following J code sets the directory and then loads the CSV file columns into the variables M_data and CD_data.

1!:44 '/home/sean/ownCloud/Org/blog/Gnuplot/'
load 'trig plot files csv'

NB. Read the data
a =: 5 }. readcsv 'DragDataSpheres.csv'
'M_data CD_data' =: 0 ". each |:(0,1){"0 1 1 a

text =: 0 : 0
set term qt
set style data points
set datafile separator ","
set pointsize 2
plot "DragDataSpheres.csv" using 1:2 pt 6
set ylabel 'Drag coefficient'
set xlabel 'Mach number'
set nokey
pause 5
set term png
set out 'testplot.png'
set out ''
set term qt

text fwrites 'testplot.gnu'
2!:0 'gnuplot < testplot.gnu'


Figure 1: Drag coefficient plot

We put all of our gnuplot instructions in an adjective definition called ‘text’. The ‘pause 5’ instruction pauses for 5 seconds to give us enough time to examine the plot before it goes away. Now that we have this, we can plot our two functions

NB. C_D (M) = 2.1 e^{-1.2(M+0.35)}-8.9e^{-2.2(M+0.35)} + 0.92
CD_Fit =: (2.1 *  (^_1.2*(M_data+0.35))) - (8.9* (^_2.2*(M_data+0.35)))) + 0.92

gamma =: 1.4
term1 =. 2%gamma * *: M_data
term2 =. (((*: M_data)*(*:gamma+1))%((4*gamma* *: M_data)-2*gamma-1))^gamma%gamma-1
term3 =. ((1-gamma) + 2*gamma* *: M_data)%(gamma+1)

CD_Mod_Newt =: 0.5*term1*(term2*term3)-1

(|:M_data,CD_data,CD_Fit,:CD_Mod_Newt) writecsv 'Compar.csv'

Note that I split the more complex modified Newtonian form into three separate terms before combining them. This eases debugging and also makes the probability of incorrectly parsing the right-to-left evaluation of the J code lower.

The three plots can be plotted together using the following gnuplot command…

text =: 0 : 0
set term qt
set style data points
set datafile separator ","
set pointsize 2
plot "Compar.csv" using 1:2 pt 6 t 'data', '' using 1:3 with lines lw 2 t 'empirical fit', '' using 1:4 with lines lt 4 lw 2 dashtype 2 t 'modified Newtonian'
set ylabel 'Drag coefficient'
set xlabel 'Mach number'
set yrange [0:1.2]
pause 10
set term pngcairo
set out 'Compar.png'
set out ''
set term qt


Figure 2: Drag coefficient plot

Note that it’s better to use the pngcairo terminal type than png. For some reason the latter will not do dashed lines, whereas the former will.

Physically, for those interested, it shows that the modified Newtonian theory is not as good a fit as the empirical curve fit at low Mach number, but it fits the behaviour well at Mach number of 5 or greater. The curve fit won’t help you at all if the ratio of specific heats \(\gamma\) is not 1.4, but the modified Newtonian method can account for that, which makes it very useful in predicting drag.

To summarise: by loading data, doing the appropriate calculations and saving the data to a text file, we can plot the data from J using a system call to gnuplot, which will plot the raw data and any manipulations we choose to perform, combining the power of J for data analysis with the rich plotting functionality of gnuplot.

NB. Here is the full J source code

Org-babel for J

As part of my emacs org-mode work flow, I have been using org-babel for a while. This allows you to insert code blocks into org buffers and have those blocks be executed when your file is compiled. This is a really handy method for doing reproduceable research. For example, you can call the source code function in R to do the statistical calculations for data in a table. If the data in the table changes, so will the calculation of the output data change. This prevents the perennial problem of having data in one file (typically a spreadsheet) and not knowing whether the document you generated for a paper used the 12th of September or the 15th of September version of the spreadsheet. By having explicit links to data and to the algorithm that manipulates that data, you can explicitly record the calculations you used to produce your data. And so can anyone else if they want to. This is very important for producing believeable data.

Org-babel is built into emacs org-mode, and supports an amazing array of programming languages, from compiled languages like C to interpreted languages like python or MATLAB, to specialised scripting languages like awk or gnuplot.

The best feature for me is that org-mode can read from or write to org tables, allowing a seamless integration between code and document. However, this capability differs between programming languages. Some languages, like python and common lisp, seem to be very well catered for in this regard. However my favourite programming language, J, is rather less well catered for. In particular, there does not seem to be a built-in way to pass variables to and from the code block. Instead, you can run your code as if it were a script, and the source block will provide the last calculated value as an output. For example,

#+BEGIN_SRC j :exports both
 NB. The square root of the sum of the squares of the numbers
 NB. between 1 and 10
	    [a =: %: +/ *: 1 + i.10
: 19.6214

The output, as stated by the comment, produces the Euclidean norm of the integers between 1 and 10 inclusive, which is 19.6214, and displays it as the result from evaluating the source block. However, for other programming languages one could supply a variable argument using the :var command in the header, to pass a variable argument to a function. So, for example, the 10 in the example above could be replaced by each of the values in the column of a table.

Like most things in emacs, the code for executing commands in code blocks is available as elisp. So, in theory, it should be possible to modify the existing elisp export code to pass variables, including rows and columns in tables, to a J function. At the moment though, my understanding of elisp is not sufficiently good to be able to work out how to do this, but it sounds like a very useful thing to do, and necessary if J is to be seriously used from within org-mode. If anyone has managed to do this, I’d be very interested to know how it’s done. If not, I’ll need to learn some more elisp and try to reverse engineer how it’s already been done for MATLAB code to see if I can do something equivalent for J.

Oh, and happy new year for 2021. I wanted to get one more blog entry done before the end of 2020, as an old-year’s resolution…