Sean's Research

Org-babel for J

Dec
31

As part of my emacs org-mode work flow, I have been using org-babel for a while. This allows you to insert code blocks into org buffers and have those blocks be executed when your file is compiled. This is a really handy method for doing reproduceable research. For example, you can call the source code function in R to do the statistical calculations for data in a table. If the data in the table changes, so will the calculation of the output data change. This prevents the perennial problem of having data in one file (typically a spreadsheet) and not knowing whether the document you generated for a paper used the 12th of September or the 15th of September version of the spreadsheet. By having explicit links to data and to the algorithm that manipulates that data, you can explicitly record the calculations you used to produce your data. And so can anyone else if they want to. This is very important for producing believeable data.

Org-babel is built into emacs org-mode, and supports an amazing array of programming languages, from compiled languages like C to interpreted languages like python or MATLAB, to specialised scripting languages like awk or gnuplot.

The best feature for me is that org-mode can read from or write to org tables, allowing a seamless integration between code and document. However, this capability differs between programming languages. Some languages, like python and common lisp, seem to be very well catered for in this regard. However my favourite programming language, J, is rather less well catered for. In particular, there does not seem to be a built-in way to pass variables to and from the code block. Instead, you can run your code as if it were a script, and the source block will provide the last calculated value as an output. For example,

#+BEGIN_SRC j :exports both
 NB. The square root of the sum of the squares of the numbers
 NB. between 1 and 10
	    [a =: %: +/ *: 1 + i.10
#+END_SRC
#+RESULTS:
: 19.6214

The output, as stated by the comment, produces the Euclidean norm of the integers between 1 and 10 inclusive, which is 19.6214, and displays it as the result from evaluating the source block. However, for other programming languages one could supply a variable argument using the :var command in the header, to pass a variable argument to a function. So, for example, the 10 in the example above could be replaced by each of the values in the column of a table.

Like most things in emacs, the code for executing commands in code blocks is available as elisp. So, in theory, it should be possible to modify the existing elisp export code to pass variables, including rows and columns in tables, to a J function. At the moment though, my understanding of elisp is not sufficiently good to be able to work out how to do this, but it sounds like a very useful thing to do, and necessary if J is to be seriously used from within org-mode. If anyone has managed to do this, I’d be very interested to know how it’s done. If not, I’ll need to learn some more elisp and try to reverse engineer how it’s already been done for MATLAB code to see if I can do something equivalent for J.

Oh, and happy new year for 2021. I wanted to get one more blog entry done before the end of 2020, as an old-year’s resolution…

Hello, Org

Nov
27

This is a test blog post for writing wordpress blog items using org-mode. I’m hoping this will let me populate my blog in a more seamless manner than editing it via the WordPress web interface.

Introduction

When Ι started this blog, I got an account with an ISP and installed WordPress using some kind of automatic template. The good thing about this is that I could start a blog without knowing what I was doing. The unfortunate part about it is that when I need to configure the blog, I have no idea whatsoever about how things are set up. Also, I would need to log in to the WordPress site and use their editor to build the pages.

But what I really want to do is use org-mode directly to upload posts to my blog. As I do most of my work documentation it would be awesome to convert some of that to my web site. Fortunately some awesome person has already done the work to make this possible. It’s called Org2Blog, created by Puneeth Chaganti, and currently maintained by Grant Rettke. It’s an org-mode package that allows you to edit posts using org-mode markup and extensions. You can edit and upload the post without leaving emacs, which is great!

The remainder of this post contains some fairly useless examples of this packages capabilities.

Examples

Figure 1 is an example of html import for images. One of the nice things about org-mode is you can specify different output parameters for images in pdf and html export formats. This can be handy for making things work in HTML.

avatar image

Here is an equation, in case you ever need to know the solution to a quadratic equation:

\[ x = -b \pm \frac{\sqrt{b^2-4 a c}}{2a} \]

  • Here are some points
    • \(\LaTeX\) is great to put inline into your blog, like \(\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}\)
    • Centered equations use two dollar signs, as opposed to inline equations like that above, which use only one

    \[\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}\]

I have not yet done much to see what can be done with org-mode and wordpress. There are several things I don’t know how to do. For example, to center the image above I had to set margins, and if I were to caption it, the caption would be left aligned. I also don’t know how to number equations. References would be the next interesting thing to try hruschka2010two. Well, it seems that citations from a bibtex file will export to WordPress, but the citation style, based on the key, does not seem to change (ie I can’t get numbered or superscript citations). Nonetheless, I’m declaring the export sufficiently capable to be useful for most blog posting purposes.

Bibliography

  • [hruschka2010two] Hruschka, O’Byrne & Kleine, Two-component Doppler-shift fluorescence velocimetry applied to a generic planetary entry probe model, Experiments in Fluids, 48(6), 1109-1120 (2010).

Marking pdf Assignments in Emacs

Sep
01

Most of the undergraduate student assignments I need to mark are numerical or analytical in nature.  The assignments are for small classes, and are often written on paper, and even if submitted electronically they are relatively easy to mark up on a tablet, because the answer is either correct or it isn’t.  But I recently had to mark a large number of assignments, for a course on the social context of engineering, where answers to questions are in paragraph form, and are fairly open-ended.  This requires explanation on my part for why marks were not allocated and suggestions on what would have made the answers better.

First time I did this I used a tablet and pen, and although it worked, it was very time consuming and messy to add comments.  Using Adobe Acrobat under Windows was not better, as I find the comment interface unwieldy when I have to use it a lot.  The most tedious aspect of this was that many of the comments were the same for large numbers of students, requiring you to type the same thing over and over again: this year I had around 75 students, with 6 pages and on average around 6 individual comments per page – a total of 2700 comments! Many of the mistakes were common to several students in the class.

In an attempt to improve the experience of marking these assignments, I looked at doing the marking in emacs, using two very simple tools: the pdf-tools package maintained by Andreas Politz, and the built-in registers of the editor.  If you do much marking up of pdfs, this may be of interest to you.

If you use emacs, and don’t currently use pdf-tools, you should.  It’s an easy install from the melpa archive, and it is a lot faster than the default pdf viewer DocView.  Most importantly, it has commands that are useful for marking up and annotating pdf files.  I have added the following commands to my init.el file to allow single-key commands in pdf-tools such as ‘h’ for highlight text, ‘s’ for strikeout and ‘t’ for text comment, as well as ‘d’ to delete existing comments.  I also have changed the default strikeout line from red to green, because red is so judgemental…

configuration of pdf-tools

Configuration file settings for pdf-tools

When this is loaded, opening the pdf file provides a view just like that of a normal pdf-viewer, scaled to fit in the emacs window.  If I select text and hit ‘h’ I highlight that text in the pdf and get to edit a comment on the highlighted text in the minibuffer.  Pressing C-c C-c exits from the minibuffer and the comment can be seen when the cursor rolls over the highlighted text in the pdf.  Pressing ‘t’ does the same, but produces an icon in the pdf instead of the highlighted text.

Annotating in pdf-tools

Annotating in pdf-tools

Once mastered, this is a very quick way of adding comments or deleting text is very fast, as only a few keystrokes are required to add annotations, which are automatically timestamped and user stamped.  Note that I changed my user designation to anonymous here, as sometimes (eg if using for annotating review papers) it’s not desirable to identify.  By default it’s your username.

As I hinted previously, what makes this even more useful is the effectiveness in combination with emacs’ registers.  I had not really used these much previously, as I had no idea what one was supposed to do with them, but populating many pdfs with identical comments is one example of where they can be useful.

While editing a comment in the minibuffer, you can select it and type C-x-r-s-1 and it will save that text to register 1.  Then the next time you want to insert that text in a comment, you type C-x-r-i-1 and it will insert the text.  The box below gives the general idea.

Insert comment from register

Using emacs registers to recycle comments

The process can then be repeated as many times as necessary when needed in a comment.  The usual C-x C-s will save the pdf for you.  Another handy command is C-c C-a l, which lists the annotations.

This procedure is not flawless: there is a bug that comes up every so often (but not always) when editing existing annotations with pdf-tools, where the link to the animation is lost and the updated animation cannot be edited.  If this happens, kill the annotation minibuffer and make and delete another comment, then try again.  This works for me.  The only other thing I’d like to see from the excellent pdf-tools package for marking up documents is a carat symbol for insertion of text.  But I can use it perfectly well in its current form.

You can find out more about registers at https://www.emacswiki.org/emacs/Registers.

The source for pdf-tools is currently at https://github.com/politza/pdf-tools.