HyperScience

Org-mode Signature Code

I’ve been trying to learn a little elisp lately, and have written a useful routine that I thought someone might like. It’s interesting because it uses both elisp and org-mode tables.

Lately I have been setting up mu4e to read my personal email. As part of using that I wanted to have a signature that would generate a wise saying from a country, but to make it a bit more interesting I wanted the signature to choose a randomly selected saying from a different country each time I generated the signature.

I have an org-mode file that contains a two-column table. The first column contains the country and the second contains the quote. 

#+NAME: SayingsTab
| Afghanistan           | Don't show me the palm tree, show me the dates                                      |
| Afghanistan           | No one says his own buttermilk is sour                                              |
| Afghanistan           | What you see in yourself is what you see in the world                               |
| Africa                | The tree that has been axed will never forget                                       |
| Albania               | Don't put gold buttons on a torn coat                                               |
| Albania               | Patience is the key to paradise                                                     |
| Albania               | When you have no companion, consult your walking stick                              |
| Algeria               | Do bad and remember, do good and forget                                             |
| America               | A pessimist is a person who has lived with an optimist                              |
| Armenia               | You cannot start a fast with baklava in your hand                                   |
| Australia             | There are none so deaf as those who will not hear                                   |
|-------------------------------------------------------------------------------------------------------------|
#+name: quote
#+begin_src elisp  :exports results :var sayings=SayingsTab
	(defun genquote()
      "Generate a random quote from an org-mode table.
  The table contains the country name in col 0 and the quote in col 1.
  Add more quotes to the table as needed, the function
 selects from the current number of rows."
	(interactive)
   (let ((selection  (nth (random (length sayings)) sayings)))
	  (message "As they say in %s, \"%s\"." (car selection) (cadr selection))))
   (genquote)
#+end_src

Running the source block here generates a random quote from the list in the table as a message. Hitting C-c C-c in the code will run it and generate a quote like 

As they say in America, "A pessimist is a person who has lived with an optimist".

I like using the org-table because the new items can be sorted alphabetically, and its easy to add new sayings when I find them.

What I’d like to be able to do is to load the org table using an elisp function and then do the same thing as part of a signature-generating function. Currently I can’ do that, but I can read the text from a tab-delimited text file. Fortunately, I can generate the tabular data that can be read by elisp by putting the point at the table and using the M-x org-table-export command. I save the table to a tab-delimited file called sayings.txt whenever I add to the org-table.

Then in my init.el file, I define the following elisp function to generate an email signature using the data in the text file. 

(defun signature()
   "Generates a random quote from a tab-delimited file for use as an email signature quote. 
	       Table of quotes is in ~/ownCloud/Org/sayings.txt
	       The tab-delimited table contains the country name in col 0 and the quote in col 1.
	       Add more quotes to the table as needed, the function
	       selects from the current number of rows."
   (interactive)
   (setq sayings (read-lines "~/ownCloud/Org/sayings.txt"))
   (setq sel (nth (random (length sayings)) sayings))
   (insert "\n-- \n" "John Doe \n \n" "john.doe@google.com \n" "Ph: 867 5309 \n")
   (insert "As they say in ")
   (insert (remove-trailing (car (split-string sel "\t" t))) ", '")
   (insert (remove-trailing (cadr (split-string sel "\t" t))) "'. \n")
   )

This should produce something like 

-- 
John Doe 
john.doe@google.com 
Ph: 867 5309 
As they say in Malaysia, 'One buffalo brings mud and all the herd are smeared with it'.

where the last line is the randomly generated quote. So then, when I’m editing my email, I just need to call the signature function when editing an email in mu4e and I get my signature with a new quote every time.

If you know how to read direct from the org-table, rather than from a text file, I’d love to know how you do it, as that would remove the need to update the table from time to time.

New Daily Driver: the Odroid N2+

I have always had a soft spot for fanless ARM single-board computers (SBCs) because they are quiet, portable and consume very little power compared to a typical laptop or desktop machine. A typical desktop computer will consume from 100 to 500 Watts of power, while a typical laptop consumes 60 to 90 Watts. An ARM SBC can consume anything from 6 to 30 Watts, which is considerably less than either of the more common formats. They also have less in the way of hardware monitoring than intel-based CPUs, and can run linux, which is my preferred operating system.

Until fairly recently, however, these machines have been too slow to operate well as standard work computers because package availability was sloppy and memory and CPU availability were at the low end of what you would typically need to get the job done. Also graphic processor support, the bane of linux, is particularly bad for these devices, as they tend to have commercial GPU drivers (as phones are their main application market).

As I mostly use open-source software for my research, and my graphical needs are fairly simple, I have proved to my own satisfaction that I can use these boards to do real work, albeit with a performance penalty compared to a modern i5 or i7 intel chip laptop. My first arm machine was an Odroid XU4 which I brought with me on sabbatical and used for writing papers and reports over a 5-month period. The only problem I had with that machine was that it would get into funny states after updating the OS, and it required a fan. Subsequent to this, I purchased a Pinebook Pro, which I could use as a laptop but which was a slower than the XU4, making the experience a little too frustrating to persevere with in the longer term, though I still use it from time to time.

Now Hardkernal, the maker of the odroid machines, has a new ARM64 SBC which is more powerful than the XU4, the Odroid N2+. This device is marketed as an alternative to the Raspberry pi 4 (which I have not used), being more powerful and more expensive. I purchased mine for USD86 with a plastic case, wireless dongle, and 128 GB emmc card (note that if you are going to use a computer seriously, having as much solid state storage as possible is very helpful). The device comes with 4GB of memory which, as the old Rolls-Royce acceleration specs used to say, is `adequate’.

This device uses more power than the XU4, requiring 12V and 2 A, rather than 5V at 3A. But this is not so surprising given the extra speed of the newer device. It also is by default fanless, although a fan is available for high load applications. So far in using it for my work, the heatsink has not got much more than warm. Although the device can apparently be overclocked to 2.4 GHz, I have not attempted overclocking it.

Initially if you purchase the device from the hardkernel web site, the emmc chip comes with ubuntu mate installed as the recommended operating system. As I like manjaro better than ubuntu, after playing around a bit with the default I used etcher to implement a manjaro sway windowing environment that has been compiled specifically for this computer. After a successful install I noticed that the screen I was using with the N2+ (a QHD 32″ lenovo monitor) would flicker randomly, which was very irritating. In case it was a problem with the Wayland system, I installed manjaro XFCE, which uses X11 rather than Wayland, but when I tried the XFCE version of manjaro, the flickering still occurred. So after a couple of unsuccessful installs, I went back to the original Ubuntu mate installation, which does not cause the flicker problem on my monitor, presumably because hardkernel installed the correct graphics drivers.

I really like tiling window managers that you can control via the keyboard (hence my initial desire to use sway), so once I had mate installed and the default user account removed I installed i3. The i3 window manager seems to work really well under ubuntu, and I was able to set things up just the way I like them. One of the things I don’t like about ubuntu and other Debian-based distributions is the slow turnaround time, as several applications require very up-to-date versions to operate properly (like my University’s owncloud server). However I was pleasantly surprised this time that most programs installable by apt were able to work without causing me problems because of their age.

Here is a picture of what the configuration looks like. The image shows emacs, a translucent shell window (terminator, using powershell) and a web browser all open on the same workspace. The little icons on the bottom right show the other three workspaces that can be used.

Figure 1: i3 configuration

Because i3 is pretty lightweight compared to many window managers, the transition between workspaces and switching between applications is very fast. Using the emacs daemon makes editing very fast too. Once you get used to it, the keyboard-driven workflow associated with i3 and emacs is pretty hard to beat.

In total, with the blind alleys caused by trying the other distros, it took about 10 hours to completely set up the N2 the way I want it. Now I can use all the tools that I use on the intel laptop for my research work, and apart from taking a little longer to load programs, I don’t experience much lag at all compared to my i7 laptop. I was able to connect to my cloud service and to run all the codes I need to, either using snaps, apt, or in a couple of cases compiling from source. The experience is no worse than my usual linux installation experience (best described as me trying lots of permutations of random things based upon internet searches until something works). My existing i3 and emacs configurations were basically able to be transferred directly to the new computer with very few changes necessary. Because all my work in progress is on the cloud, this means that I can work on my project either on my laptop or on this SBC with seamless results, as I have the same applications installed on both machines.

In summary, I’m impressed. This blog post was written on it using emacs org2blog.  It’s possible that I might get bored or frustrated and stop using this machine for work, in which case it will be used as a lab device for transferring data from instruments, or as a connected diary device.  But at the moment I can’t much tell the difference between working on this machine and working on my laptop, and that’s a very good sign.  It’s really impressive how far ARM64 support has come in linux.  For a total outlay of less than AUD200, this is a really fun-to-use laptop replacement, provided you have access to a HDMI monitor and as long as you don’t mind shutting it down between moves (because it does not have a battery).  4GB is not a lot of memory, but in linux I have yet to reach a limit that affects my work in terms of available memory.

The small size of this machine means that, with a big rechargeable battery, it could be made into a very nice portable computer provided you have access to a TV or monitor with HDMI support, which is pretty much everything these days.  I have some ideas about form factors that I hope to have time to try out one day…

Org-babel for J

As part of my emacs org-mode work flow, I have been using org-babel for a while. This allows you to insert code blocks into org buffers and have those blocks be executed when your file is compiled. This is a really handy method for doing reproduceable research. For example, you can call the source code function in R to do the statistical calculations for data in a table. If the data in the table changes, so will the calculation of the output data change. This prevents the perennial problem of having data in one file (typically a spreadsheet) and not knowing whether the document you generated for a paper used the 12th of September or the 15th of September version of the spreadsheet. By having explicit links to data and to the algorithm that manipulates that data, you can explicitly record the calculations you used to produce your data. And so can anyone else if they want to. This is very important for producing believeable data.

Org-babel is built into emacs org-mode, and supports an amazing array of programming languages, from compiled languages like C to interpreted languages like python or MATLAB, to specialised scripting languages like awk or gnuplot.

The best feature for me is that org-mode can read from or write to org tables, allowing a seamless integration between code and document. However, this capability differs between programming languages. Some languages, like python and common lisp, seem to be very well catered for in this regard. However my favourite programming language, J, is rather less well catered for. In particular, there does not seem to be a built-in way to pass variables to and from the code block. Instead, you can run your code as if it were a script, and the source block will provide the last calculated value as an output. For example, 

#+BEGIN_SRC j :exports both
 NB. The square root of the sum of the squares of the numbers
 NB. between 1 and 10
	    [a =: %: +/ *: 1 + i.10
#+END_SRC

#+RESULTS:
: 19.6214

The output, as stated by the comment, produces the Euclidean norm of the integers between 1 and 10 inclusive, which is 19.6214, and displays it as the result from evaluating the source block. However, for other programming languages one could supply a variable argument using the :var command in the header, to pass a variable argument to a function. So, for example, the 10 in the example above could be replaced by each of the values in the column of a table.

Like most things in emacs, the code for executing commands in code blocks is available as elisp. So, in theory, it should be possible to modify the existing elisp export code to pass variables, including rows and columns in tables, to a J function. At the moment though, my understanding of elisp is not sufficiently good to be able to work out how to do this, but it sounds like a very useful thing to do, and necessary if J is to be seriously used from within org-mode. If anyone has managed to do this, I’d be very interested to know how it’s done. If not, I’ll need to learn some more elisp and try to reverse engineer how it’s already been done for MATLAB code to see if I can do something equivalent for J.

Oh, and happy new year for 2021. I wanted to get one more blog entry done before the end of 2020, as an old-year’s resolution…

Hello, Org

This is a test blog post for writing wordpress blog items using org-mode. I’m hoping this will let me populate my blog in a more seamless manner than editing it via the WordPress web interface.

Introduction

When Ι started this blog, I got an account with an ISP and installed WordPress using some kind of automatic template. The good thing about this is that I could start a blog without knowing what I was doing. The unfortunate part about it is that when I need to configure the blog, I have no idea whatsoever about how things are set up. Also, I would need to log in to the WordPress site and use their editor to build the pages.

But what I really want to do is use org-mode directly to upload posts to my blog. As I do most of my work documentation it would be awesome to convert some of that to my web site. Fortunately some awesome person has already done the work to make this possible. It’s called Org2Blog, created by Puneeth Chaganti, and currently maintained by Grant Rettke. It’s an org-mode package that allows you to edit posts using org-mode markup and extensions. You can edit and upload the post without leaving emacs, which is great!

The remainder of this post contains some fairly useless examples of this packages capabilities.

Examples

Figure 1 is an example of html import for images. One of the nice things about org-mode is you can specify different output parameters for images in pdf and html export formats. This can be handy for making things work in HTML.

avatar image

Here is an equation, in case you ever need to know the solution to a quadratic equation:

\[ x = -b \pm \frac{\sqrt{b^2-4 a c}}{2a} \]

  • Here are some points
    • \(\LaTeX\) is great to put inline into your blog, like \(\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}\)
    • Centered equations use two dollar signs, as opposed to inline equations like that above, which use only one

    \[\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}\]

I have not yet done much to see what can be done with org-mode and wordpress. There are several things I don’t know how to do. For example, to center the image above I had to set margins, and if I were to caption it, the caption would be left aligned. I also don’t know how to number equations. References would be the next interesting thing to try hruschka2010two. Well, it seems that citations from a bibtex file will export to WordPress, but the citation style, based on the key, does not seem to change (ie I can’t get numbered or superscript citations). Nonetheless, I’m declaring the export sufficiently capable to be useful for most blog posting purposes.

Bibliography

  • [hruschka2010two] Hruschka, O’Byrne & Kleine, Two-component Doppler-shift fluorescence velocimetry applied to a generic planetary entry probe model, Experiments in Fluids, 48(6), 1109-1120 (2010).

Marking pdf Assignments in Emacs

Most of the undergraduate student assignments I need to mark are numerical or analytical in nature.  The assignments are for small classes, and are often written on paper, and even if submitted electronically they are relatively easy to mark up on a tablet, because the answer is either correct or it isn’t.  But I recently had to mark a large number of assignments, for a course on the social context of engineering, where answers to questions are in paragraph form, and are fairly open-ended.  This requires explanation on my part for why marks were not allocated and suggestions on what would have made the answers better.

First time I did this I used a tablet and pen, and although it worked, it was very time consuming and messy to add comments.  Using Adobe Acrobat under Windows was not better, as I find the comment interface unwieldy when I have to use it a lot.  The most tedious aspect of this was that many of the comments were the same for large numbers of students, requiring you to type the same thing over and over again: this year I had around 75 students, with 6 pages and on average around 6 individual comments per page – a total of 2700 comments! Many of the mistakes were common to several students in the class.

In an attempt to improve the experience of marking these assignments, I looked at doing the marking in emacs, using two very simple tools: the pdf-tools package maintained by Andreas Politz, and the built-in registers of the editor.  If you do much marking up of pdfs, this may be of interest to you.

If you use emacs, and don’t currently use pdf-tools, you should.  It’s an easy install from the melpa archive, and it is a lot faster than the default pdf viewer DocView.  Most importantly, it has commands that are useful for marking up and annotating pdf files.  I have added the following commands to my init.el file to allow single-key commands in pdf-tools such as ‘h’ for highlight text, ‘s’ for strikeout and ‘t’ for text comment, as well as ‘d’ to delete existing comments.  I also have changed the default strikeout line from red to green, because red is so judgemental…

configuration of pdf-tools

Configuration file settings for pdf-tools

When this is loaded, opening the pdf file provides a view just like that of a normal pdf-viewer, scaled to fit in the emacs window.  If I select text and hit ‘h’ I highlight that text in the pdf and get to edit a comment on the highlighted text in the minibuffer.  Pressing C-c C-c exits from the minibuffer and the comment can be seen when the cursor rolls over the highlighted text in the pdf.  Pressing ‘t’ does the same, but produces an icon in the pdf instead of the highlighted text.

Annotating in pdf-tools

Annotating in pdf-tools

Once mastered, this is a very quick way of adding comments or deleting text is very fast, as only a few keystrokes are required to add annotations, which are automatically timestamped and user stamped.  Note that I changed my user designation to anonymous here, as sometimes (eg if using for annotating review papers) it’s not desirable to identify.  By default it’s your username.

As I hinted previously, what makes this even more useful is the effectiveness in combination with emacs’ registers.  I had not really used these much previously, as I had no idea what one was supposed to do with them, but populating many pdfs with identical comments is one example of where they can be useful.

While editing a comment in the minibuffer, you can select it and type C-x-r-s-1 and it will save that text to register 1.  Then the next time you want to insert that text in a comment, you type C-x-r-i-1 and it will insert the text.  The box below gives the general idea.

Insert comment from register

Using emacs registers to recycle comments

The process can then be repeated as many times as necessary when needed in a comment.  The usual C-x C-s will save the pdf for you.  Another handy command is C-c C-a l, which lists the annotations.

This procedure is not flawless: there is a bug that comes up every so often (but not always) when editing existing annotations with pdf-tools, where the link to the animation is lost and the updated animation cannot be edited.  If this happens, kill the annotation minibuffer and make and delete another comment, then try again.  This works for me.  The only other thing I’d like to see from the excellent pdf-tools package for marking up documents is a carat symbol for insertion of text.  But I can use it perfectly well in its current form.

You can find out more about registers at https://www.emacswiki.org/emacs/Registers.

The source for pdf-tools is currently at https://github.com/politza/pdf-tools.