kyle kneitinger

Perplexing Config Lexeme Conflicts in the i3 Window Manager

Sat, 04 Mar 2017 00:00:00 +0000

The Issue

In February, 2016, Github user @BobuSumisu filed [a bug report about variable expansion collisions] (https://github.com/i3/i3/issues/2235) when one variable starts with the name of another variable.

For example, say you had the following variable declarations, and an exec line to autolaunch a program at boot:

set $shoopaloop HOWDY
set $shoop      i heard you have scrumptious seafood

exec $shoopaloop

One would expect this to evaluate to exec HOWDY, however it becomes exec i heard you have scrumptious seafoodaloop. Furthermore, given the configuration below:

set $shoop      i heard you have scrumptious seafood
set $shoopaloop HOWDY

exec $shoopaloop

The correct result of exec HOWDY is achieved.

Why is it happening?

How Variables Are Parsed

To understand this bug, lets look at the parse_file function of src/config_parser.c from the commit right before I started working on this bug. This function encapsulates a lot of logic, but the only code that is applicable to this issue is where variables are read in and stored in a list (src/config_parser.c:861887), and when that list is iterated over, replacing all occurrences of the variable with its intended value (src/config_parser.c:914-944).

The grammar for a variable declaration is essentially:

vardecl -> "set" [ \t]+ \$[\S]+ [ \t]+ [\S]+ [ \t]*

where \$ matches a literal $, \S matches any non-whitespace character, all tokens are space or tab separated, and all configuration lines are newline separated.

When a variable is found in a config (indicated by the set keyword), a few things happen in parse_file to verify it is syntactically correct, and then the key and value are copied to a new Variable struct which is added to the front of a list called variables (this is important in diagnosing the issue)

    struct Variable *new = scalloc(1, sizeof(struct Variable));
    new->key = sstrdup(v_key);
    new->value = sstrdup(v_value);
    SLIST_INSERT_HEAD(&variables, new, variables);
    DLOG("Got new variable %s = %s\n", v_key, v_value);
    continue;

The code for substituting the variable key occurrences with their respective values is shown here:

/* Then, allocate a new buffer and copy the file over to the new one,
 * but replace occurrences of our variables */
char *walk = buf, *destwalk;
char *new = smalloc(stbuf.st_size + extra_bytes + 1);
destwalk = new;
while (walk < (buf + stbuf.st_size)) {
    /* Find the next variable */
    SLIST_FOREACH(current, &variables, variables)
    current->next_match = strcasestr(walk, current->key);
    nearest = NULL;
    int distance = stbuf.st_size;
    SLIST_FOREACH(current, &variables, variables) {
        if (current->next_match == NULL)
            continue;
        if ((current->next_match - walk) < distance) {
            distance = (current->next_match - walk);
            nearest = current;
        }
    }
    if (nearest == NULL) {
        /* If there are no more variables, we just copy the rest */
        strncpy(destwalk, walk, (buf + stbuf.st_size) - walk);
        destwalk += (buf + stbuf.st_size) - walk;
        *destwalk = '\0';
        break;
    } else {
        /* Copy until the next variable, then copy its value */
        strncpy(destwalk, walk, distance);
        strncpy(destwalk + distance, nearest->value, strlen(nearest->value));
        walk += distance + strlen(nearest->key);
        destwalk += distance + strlen(nearest->value);
    }
}

There is a lot of logistical stuff going on here that makes this code a little hard to read, but the gist of this algorithm is to iterate through the list of variables, hence the SLIST_FOREACH macro, from front to back, replacing the nearest variables with their values.

So What Went Wrong?

It should be clear now that the variables are substituted in an order reversed from the order in which they were declared. With that in mind, let’s look at an example configuration that generates a happy dialog of a fish connesior visiting Los Angeles, and see how it is processed.

set $foo  hey la
set $foodies i heard you have scrumptious seafood

exec echo $foo > ~/pete
exec echo $foodies >> ~/pete

After the variables are read in, we have a list ( $foodies->i heard you have scrumptious seafood, $foo->hey la ). When the replacement starts, we find the instances of the first variable, $foodies and essentially rewrite them in-place (note: this doesn’t actually happen “in-place”, but for the sake of simplicity in this article, it is a safe abstraction to make).

At this stage, we have

exec echo $foo >  ~/pete
exec echo i heard you have scrumptious seafood >> ~/pete

Then, we move on to the next variable in the list, $foo, and replace its occurences, resulting in the final configuration:

exec echo hey la >  ~/pete
exec echo i heard you have scrumptious seafood >> ~/pete

Checking the output of cat pete verifies this:

That seemed to work well, but what if the variables were declared the other way around?

set $foodies i heard you have scrumptious seafood
set $foo  hey la

exec echo $foo > ~/pete
exec echo $foodies >> ~/pete

This gives us the list of variables containing ( $foo->hey la, $foodies->i heard you have scrumptious seafood ). Let’s see what happens when we start to replace the keys with their values, again, starting with the first variable in the list, $foo. There is only one instance of $foo, so there should be no issue.

exec echo hey la > ~/pete
exec echo hey ladies >> ~/pete

Sigh, what does cat pete have to say?

Woah woah woah…what?. Our previously sweet Ol’ Pete fled, and became a deviously effete meat head! So yeah, turns out there were two instances of “$foo”, but one was contained in “$foodies”. At this point there’s nothing left to even do, we’ve overwritten all of the $foo substrings in the occurrences of $foodies, leaving just the skeezy “hey ladies” behind.

Fixing the Issue

So it seems that there are two ways we could possibly fix this, one based on whitespace, and another more subtle way. If we looked for variable occurrences that were complete tokens separated by whitespace, this would certainly solve the issue…at least in the above example. However, it is extremely common, and sometimes necessary to have variable references that are not separated by whitespace, such as:

bindsym $mod+1 workspace 1

Therefore, this option is not feasible in the current context. So what does that leave us?

Longest Lexeme Based Parsing

The concept of matching on longest lexeme means that given a variable reference in which more than one substring starting at position 0 of the name is equal to a defined variable, then it should be replaced by the one with the longest length. For example, in our running example, a reference to $foodies should not be matched with $foo, because there is a longer declared variable that (exactly) matches it $foodies. This is a pretty simple concept, and seems perfect for this scenario, but how exactly do we implement it in the context of this codebase?

Implementing Longest Lexeme Matching

Recall the example configuration:

set $foo  hey la
set $foodies i heard you have scrumptious seafood

exec echo $foo > ~/pete
exec echo $foodies >> ~/pete

The variables in this configuration were correctly replaced, so what is there to learn from it? Well, remember that the variable key/value pairs are added to the Variables list at the front, which means that after all variable declarations were parsed, the list had the longer lexeme at the head. When variable replacement started, it substituted all instances of the longest variable, $foodies leaving none for $foo to eagerly match on.

In essence, if we can ensure that our complete list of Variables is sorted with the longest key at the head and the shortest key at the tail, then we can ensure that if variables are initial substrings of another, the longer variable reference will have already been replaced, avoiding any conflict.

The i3 codebase is very elegantly designed, and a lot of decisions are made to maintain a cohesive style, convention, and design. The initial implementation of the Variables list uses the <sys/queue.h> SLIST singly-linked list macros as its structure. Obviously it seemed like a perfect choice to insert in O(1) on a tried and true data structure, but unfortunately this would have to be changed in one of two way: either change the data structure and its many interactions to a better suited sorted list, or use the SLIST and sacrifice O(1) in exchange for minimal codebase changes. Since variable parsing only ever happens once, is relatively lightweight, and generally will not have a sufficiently large amount of variables to work with, I opted for the latter approach, wherein I compared the length of each new variable key to those of the current values; inserting before any keys that are shorter, or, simply sorting the list by length upon insertion. If the new variable key is longer than all others it becomes the new front of the list. The new code is shown below, if you compare it to the original Variables insertion code, you will find that consists of actually very few changes.

struct Variable *new = scalloc(1, sizeof(struct Variable));
struct Variable *test = NULL, *loc = NULL;
new->key = sstrdup(v_key);
new->value = sstrdup(v_value);
/* ensure that the correct variable is matched in case of one being
 * the prefix of another */
SLIST_FOREACH(test, &variables, variables) {
    if (strlen(new->key) >= strlen(test->key))
        break;
    loc = test;
}

if (loc == NULL) {
    SLIST_INSERT_HEAD(&variables, new, variables);
} else {
    SLIST_INSERT_AFTER(loc, new, variables);
}

DLOG("Got new variable %s = %s\n", v_key, v_value);
continue;

feel free to contact me with any comments or corrections via the options listed at the bottom of the page.

Ensuring Beautiful Commits with rustfmt and Travis-CI

Sat, 26 Nov 2016 00:00:00 +0000

When Style is Standardized, Style Can Be Standard

The rust-lang-nursery GitHub organization is a fantastic group of folks building tools for both working in, and working with, the Rust language. One of these tools, rustfmt, is quite helpful for maintaining a consistent code style throughout a project.

rustfmt is pretty easy to work with. Its default behavior, when executed on a file, is to check it for anything that violates the style guide, and if anything needs to be changed, backup the file with a .bk suffix and silently replace the original with the new formatting. If passed a valid rust file, it returns 0 regardless of any changes being made. rustfmt only operates on a single file, so if the current directory is a cargo project, a handy cargo fmt wrapper is provided that checks the entire project.

There are a few nice ways to integrate rustfmt into your workflow, such as vim plugins, and git commit hooks. Unfortunately, these depend on the individual developer to use them, and thus do not enforce any project-wide policy.

Enter Travis

A lot of the themes brought up in the previous paragraph are pretty reminiscent of testing: consistent when used, but largely reliant on the developer to use them. Since Travis-CI was already set up to run our tests and report upon failure, why not leverage that build process to handle formatting?

The default Rust .travis.yml provided by Travis CI is pretty straightforward:

language: rust
rust:
  - stable
  - beta
  - nightly
matrix:
  allow_failures:
    - rust: nightly

It relies on cargo to run cargo build --verbose and cargo test --verbose during the implicit “script” phase of the build. With just a few slight modifications, we can have format checking be a part of this process.

Installing the Tool

Travis CI can has an optional “install” step (see Travis-CI: The Build Lifecycle) where we can have cargo install rustfmt and then add it to our path. This step can take a very long time to finish due to compilation of the env_logger crate. Thankfully Travis CI provides a mechanism for caching dependencies with the cache key. Putting what we have so far together, we get:

language: rust

cache: cargo

rust:
  - stable
  - beta
  - nightly

matrix:
  allow_failures:
    - rust: nightly

install:
  - (cargo install rustfmt || true)
  - PATH=$PATH:/home/travis/.cargo/bin

Recall how I said that any non-zero return status breaks the build? Well the cargo install command returns non-zero if the desired package is already installed (which means caching worked!). In order to get around this, we massage it into returning true upon “failure” and thus continuing the build.

Checking the Style

Now that our build context has rustfmt, we can have it run along with the builds and test. It would be nice if we could have Travis-CI automatically and politely run rustfmt and fix any format-offensive code, but that would have to rewrite git history and is out of the question. The best we can do is break the build upon a push or pull request, so that the developer knows to run the format command and recommit.

Since any command with a non-zero return status breaks the build, we just need cargo fmt to break at some point. Again, the default behavior of rustfmt, even when wrapped into cargo, is to silently backup and reformat and offending code. We need a way to detect when code is malformatted. If we check the usage with rustfmt -h, we see the --write-mode [replace|overwrite|display|diff|coverage|checkstyle] option. The default setting is ‘replace’, but after trying them out, it seems the most helpful mode for this purpose is ‘diff’. It provides two things that work well in this context: a non-zero return code upon code that doesn’t meet the style guide, and output that makes it clear why the build broke. Since we’re using the project-aware cargo fmt wrapper, we cannot use rustfmt flags directly until we insert a -- argument.

At this point, we could just put the cargo fmt line in the “install step”, it would certainly work as desired. However, for the sake of clarity and further modularity, let’s make the “script” step explicit now. This means inserting the cargo fmt command as well as the default build and test commands as elements of the “script” key:

language: rust

cache: cargo

rust:
  - stable
  - beta
  - nightly

matrix:
  allow_failures:
    - rust: nightly

install:
  - (cargo install rustfmt || true)
  - PATH=$PATH:/home/travis/.cargo/bin

script:
  - cargo fmt -- --write-mode=diff
  - cargo build --verbose
  - cargo test --verbose

And there you have it! I hope this helps you make the world a cleaner, more stylish place! If you have any feedback or questions, feel free to comment below or reach out on twitter.

SysEx-ploration

Mon, 27 Jun 2016 00:00:00 +0000

Motivation

I recently purchased an Akai MPD32 MIDI controller from a swell person on craiglist. I wanted a relatively small controller that had enough knobs and sliders to map to various parameters when sketching out ideas in ChucK and Faust, and this seemed like a perfect solution…plus 4 banks of 16 pads with independent pressure tracking!

I plugged it in, did a little pad tapping, a little knob turning, and a little fader fading. It was quite nice. However, on every preset, each pad was set to channel aftertouch and not polyphonic pressure sensing, and the pads were arranged chromatically always starting at C. No worries I thought, I could just spend a few minutes changing those to my liking. I was horribly mistaken.

After entering edit mode, setting the pressure requires 6 button presses for each pad. With 64 pads to set, that’s 384 presses plus the bank change presses. And don’t even get me started on changing the pads to diatonic from chromatic! There had to be a better way. I noticed that the Akai website had a download for the Vyzex preset editor, so I decided to give it a shot. I installed it in my Windows 10 VM, and redirected the MPD32 USB to it no problem. I fired it up and was greeted with a nice splash screen letting me know that the copyright date was 2007…and it showed. Apparently in 2007, nobody could fathom that screens would be above 1024x768, as the window is tiny and nonadjustable on a relatively modern screen. But, if it would make editing easier, I could live with those. I was hoping for bulk edit of pads, increment/decrement buttons for values, and toggle switches for options. I got text boxes. Click a pad, then click the note field then enter value in a text box. Womp womp, this wasn’t a solution either.

So when faced with only two systems that are difficult to interact with, what should one chose? A third!

Idea

I noticed that in the GLOBAL menu on the MPD32, there is a SysEx Tx option to transmit the MIDI System Exclusive representation of a single preset to another device. I had encountered SysEx in the past when I wanted to back up my Yamaha DX-7 patches to a computer, but knowing nothing about its data representation, ‘twas a pure mystery to me back then. Now, equipped with 3 years of CS knowledge, I figured I could certainly grok the format enough to write my own editor…and that is just what I did!

The System Exclusive Specification

Since I was going to be extracting meaning from a SysEx dump, it would probably be a good idea to see exactly what the SysEx specification is. I referred to a couple of sites, and learned the entire specification in about 20 seconds. I’m not saying that to sound boastful, its just that the entire specification, for better or for worse is:

0xF0	0xXX, 0xYY, ...	0xF7
SysEx Begin	Any amount of data arranged in any way	SysEx End

Luckily, the MIDI Manufacturer’s Association adopted the following convention to specify which device a message is intended for.

0xF0	0xMM	0xMM	0xMM	0xPP	0xXX, 0xYY, ...	0xF7
SysEx Begin	Manufacturer ID			Product ID	Any amount of data arranged in any way	SysEx End

It also seems that usually the first data byte will provide some sort of opcode-like purpose, declaring the intent of the message. With all this in mind, lets look at the MPD32 data.

Dumping the Data

I used the amidi from the alsa-utils suite of programs to accomplish most of this task. First, I needed to find the device ID, which is easily found with the list (-l) flag

$ amidi -l
Dir Device    Name
IO  hw:1,0,0  Akai MPD32 MIDI 1
IO  hw:1,0,1  Akai MPD32 MIDI 2
I   hw:1,0,2  Akai MPD32 MIDI 3

The MPD32 is quite flexible, so it provides 3 virtual devices, yet as it turns out (from trial and error), only the first one transmits and receives SysEx messages, so I used that one from there on out. Note the “Device” field lists it as “hw:1,0,0”, which is the value to be used in the upcoming steps when interacting with the device. Next, I used the port and receive flags for amidi to store the data to a file called mpd_dump.syx. First, I navigated to the SysEx Tx page so as to not clutter the dump with realtime button press info. Then after running the following command, I pressed the enter knob to initiate the transfer, and pressed Ctrl+C to stop receiving:

$ amidi -p hw:1,0,0 -r mpd_dump.syx
^C
1033 bytes read

Wow. So 1033 bytes. That is a fairly hefty amount of data to sift through, so first I need to verify that it was actually the preset dump. First, a simple hexdump on the SysEx file, confirms that it starts with an 0xF0, and ends with an 0xF7 (remember, little endian!).

$ hexdump mpd_dump.syx
47f0 6c00 0810 1e01 6547 656e 6972 2063
7800 0401 3201 013a 0204 323c 0300 0101
0002 0000 0000 0000 0000 0000 0003 0024
0001 0000 0003 0025 0001 0000 0003 0026
# many lines omitted for space!
6f00 0001 0000 0000 0170 0000 0000 7100
0001 0000 0100 7f00 0000 0b01 7f00 0100
0040 0000 4001 0000 00f7

Furthermore, the preset was called “Generic”. Assuming that the name is stored in an ASCII-like format, the strings command would find it.

$ strings mpd_dump.syx
Generic

Bingo! So I knew I was definitely dealing with the correct data, but how could I know which of the 1033 bytes did what?

Deciphering the Data

The most obvious way to determine which byte was associated with which parameter, was to change some parameters and track the deltas between the Generic dump and the new ones. But again, I’m doing this all to save work not make more and that sounds like quite an undertaking! Since I was tracking changes or differences, perhaps there was a standard tool I could use to, such as…python! Oh yeah, it would probably be helpful to use diff too. I wrote a script that encapsulated the following flow:

Write a program that takes a raw binary SysEx message, and outputs it as hex values, one per line.
1. Run it on the Generic mpd_dump.syx file
2. Run it on the modified mpd_edit.syx file
diff the two outputs to calculate any lines that have changed. Because there is only one byte per line, the line number - 1 is the byte offset in the SysEx dump
Prompt for a comment on what I changed
Append the byte number, the generic value, the new value, and the comment to a .csv file
Assess if a coffee refill is needed
Repeat steps 1.2-5 until all desired parameters are mapped.

It became clear pretty quickly that the data was arranged in a very intuitive manner! For example, I began with the 1st pad, and changed it’s menu options one by one, and found that they are arranged in the dump in the same order they appear on the screen. Furthermore, the biggest relief was to find that the data for all pads is stored identically and sequentially! After mapping the first pad (8 bytes), and the second pad, my prediction of where the 64th pad should begin was spot on: PAD_BEGIN + (PAD_WIDTH * padNum), where the respective values are 0x2C, 0x08, and the pad number desired (zero-indexed). I ended up with the following map of how pad parameters are stored:

Offset from 0x2C + (0x8 * Pad #)	Parameter	Values
0x00	Mode	3:Note, 4:Program Change
0x01	Channel	0:Common. 1-3
0x02	Note Number	0-127
0x03	Trigger mode	0:Momentary 1, Toggle
0x04	Pressure Mode	0:Off, 1:Channel Press, 2: Polyphonic
0x05	Program Num	0-127
0x06	Bank MSB	0-127
0x07	Bank LSB	0-127

The layout was similar for the knobs, switches and faders, each of which have 8 physical * 3 banks = 24 virtual units. They all varied in the different ways they can be configured, some controls requiring 7 bytes, some only 5. Additionally there are a few one-off settings such as the note-repeat (drum-roll) button, tempo, playback control, etc. The final mapping of all parameters that were physically modifiable is:

Byte	Field	Valid Values
0x0000	SysEx Begin	(0xF0)
0x0001	Mfg Id	(0x47)
0x0002	Mfg Id	(0x00)
0x0003	Mfg Id	(0x6C)
0x0004	Prod Id	(0x10)
0x0007	Preset number	1-30
0x0008	Preset Name Char 0	Pretty much ASCII
…	…	…
0x000F	Preset Name Char 7	Pretty much ASCII
0x0010	(Presumed null char for string)	(0x00)
0x0011	Tempo	30-300
0x0012	Time Divide Mode	0:Momentary 1, Toggle
0x0013	Time Divide Amount	0:1/4-Note-7:1/32T-Note
0x0014	Note Repeat Mode	0:Momentary 1, Toggle
0x0015	Note Repeat Gate	0-99
0x0016	Note Repeat Swing	50-75
0x0020	Transport Mode	0:MMC, 1:MIDI. 2:MMC/MIDI, 3:CTRL
0x002C	Pad Mode	3:Note, 4:Program Change
+0x01	Channel	0:Common. 1-3
+0x02	Note Number	0-127
+0x03	Trigger mode	0:Momentary 1, Toggle
+0x04	Pressure Mode	0:Off, 1:Channel Press, 2: Polyphonic
+0x05	Program Num	0-127
+0x06	Bank MSB	0-127
+0x07	Bank LSB	0-127
0x022C	Knob Mode	0:Control Change, 1:Aftertouch. 2: Inc/Dec
+0x01	Channel	0:Common. 1-3
+0x02	Control Change Number	0-127
+0x03	CC/Aftertouch Min	0-127
+0x04	CC/Aftertouch Max	0-127
+0x05	NRPN Left	0-127
+0x06	NRPN Right	0-127
0x02D4	Fader Mode	0:Control Change, 1:Aftertouch
+0x01	Channel	0:Common. 1-3
+0x02	Control Change Number	0-127
+0x03	CC/Aftertouch Min	0-127
+0x04	CC/Aftertouch Max	0-127
0x034C	Switch Mode	0:Control Change, 1:Program Change
+0x01	Channel	0:Common. 1-3
+0x02	Control Change Number	0-127
+0x03	Trigger mode	0:Momentary 1, Toggle
+0x04	Program Number	0-127
+0x05	Bank MSB	0-127
+0x06	Bank LSB	0-127
0x0408	SysEx End	(0xF7)

Color key:

Pad data block

Knob data block

Fader data block

Switch data block

Working with the Data

So there it was, my very own preset map! Now I had the key to whatever configuration I could dream up! I did a quick test to verify that uploading a .syx works, by changing the mpd_edit.syx’s name bytes from “Generic” to “Doop” (with spaces filling in the remaining 4 bytes) in the bless hex editor

and then uploaded it with our good pal amidi

$ amidi -p hw:1,0,0 -s mpd_edit.syx

and what do you know…it worked!

With basic editing and uploading confirmed to be working, I could finally code any patch I wanted: polyphonic pressure, diatonic scales, momentary buttons…anything! I have plans to develop this into a GTK app, ideally one that can edit many different MIDI controllers, but below is a snippet demonstrating low-level manipulation of the .syx file. Then I can use aplay, just like above, to upload this patch to the MPD32.

from struct import *

PAD_BASE  = 0x2C
PAD_WIDTH = 0x8
PAD_MAX   = 0x40

SWITCH_BASE  = 0x34C
SWITCH_WIDTH = 0x7
SWITCH_MAX   = 0x18

WHOLE     = 2
HALF      = 1
MAJOR  = [ WHOLE, WHOLE, HALF, WHOLE, WHOLE, WHOLE, HALF ]

# Load base file into list of bytes
with open("mpd_dump.syx", mode='rb') as file:
    rawSysex = file.read()
sysex = list(unpack("B" * len(rawSysex), rawSysex))

# Set all pads to polyphonic aftertouch!
for i in range(PAD_MAX):
    sysex[PAD_BASE + (PAD_WIDTH * i) + 0x4] = 2

# Arrange the pads to the E major scale!  (MIDI note 28 = E1)
step = 0
base = 28
for i in range(PAD_MAX):
    sysex[PAD_BASE + (PAD_WIDTH * i) + 0x2] = base
    base += MAJOR[step]
    step = (step + 1) % 7

# Set all of the buttons to momentary!
for i in range(SWITCH_MAX):
    sysex[SWITCH_BASE + (SWITCH_WIDTH * i) + 0x3] = 0


# Write modified values to new file
with open("mpd_edit.syx", mode='wb') as file:
    file.write(bytes(sysex))

Moving Forward

Like I said, my next step is turn this into an extensible GUI app, so I would have to take care to design it in such a way that facilitates easy community creation of other controllers. Furthermore, the very offline workflow of receive .syx, modify .syx, send .syx could be replaced by dynamic, realtime editing, including synchronized editing in the device and the app. I know that the MPD transmits and receives these events, using much smaller 10-byte realtime messages reflecting just the edited parameter. This workflow is a bit nicer in that changes are instantly synced to the device, and can be experimented with on the fly, while the device is in use even.

This exercise taught me a lot, and I look forward to seeing where this goes…not to mention making some quite expressive instruments and grooving with them!

feel free to contact me with any comments or corrections via the options listed at the bottom of the page.

Picnote

Fri, 18 Sep 2015 00:00:00 +0000

A Tool is Born from Horrible UI

Over the summer I was enrolled in a introductory level chemistry class to fulfill a miscellaneous lab science credit. Overall, I had a great time in this class with the exception of the web app created by one of the major textbook corporations that was required to complete the homework assignments. Though there were certain questions that greatly benefited from an interactive style, by and large, the app was a hindrance more than an aid. The app would reset and lose all of your progress if you were to use the browser navigation buttons, and only one tab of the app could be open at a time. These factors become extremely frustrating when certain questions depended on the diagrams featured in earlier ones’. Because one habitually, and thus frequently uses the browser buttons to go back, I lost my progress well into an assignment.

To prevent this from happening again (as well as take a break from reaction balancing) I whipped up a little solution to take a snapshot of an area of the screen and display it in a small window on top of all others. I call this little utility picnote and it has already proven to be very helpful for carrying little bits of reference around my window manager.

Here is a little gif of it in action:

Components of picnote

There are three programs that come together to make picnote:

scrot: a screenshot utility with a command line interface for easy scripting.
feh: a lightweight, minimal image viewer.
i3: my window manager of choice. Easy to configure and very powerful.

How it works

picnote is actually quite simple. First, the following shell script handles capturing the image and opening in feh.

1
2
3
4
5
6
#!/bin/bash

PIC="/tmp/$RANDOM.png"

scrot -s $PIC && feh --title feh:$PIC $PIC &
rm $PIC

Line number 3 sets up a variable to store the name and path of image to be a randomly generated .png residing in /tmp/. Line 5 begins by executing the image capture with scrot -s. The “-s” flag indicates that upon execution, the next mouse drag will select the box to be captured. The image is saved as the file from line 3. Upon successful execution, feh sets its title as “feh:<image name from line 3>” in order to be easily identifiable as a picnote window for the i3 rule I’ll show soon, and then opens the image. Line 6 simply deletes the image which is fine to do once feh has opened it.

Because I primarily use i3 as a tiling window manager, and such have set that as its default behavior, I needed to create a configuration rule to set all windows titled feh:/tmp/… to be floating (which have the additional benefit of displaying on top of all tiled windows).

The .i3/config line to accomplish this is

for_window [title="feh:/tmp/\*"] floating enable

which simply expresses that all windows whose title matches that regular expression should be put into floating mode upon creation.

Usage

It would be a pain to call picnote from the command line every time it is needed, so I simply mapped it to an unused media key on my ThinkPad for easy use. Once it is launched, you can close the image with Esc, Del, q, or your window managers keybinding for closing windows.

Thank you for reading this and I hope you enjoy picnote if it is a solution for a problem you are having!

Btrfs on dm-crypt

Thu, 17 Sep 2015 00:00:00 +0000

Two of the most important and exciting tools for data storage that Linux has to offer are disk encryption via dm-crypt, and built-in RAID, copy-on-write, and other modern storage features via the Btrfs file system. Both dm-crypt and Btrfs have volumes of information written on them, however, when I bought a shiny (on second thought, matte) new ThinkPad 450s with 3 SSDs and set out to implement a system utilizing both features, I had trouble connecting the information into a cohesive process. I learned a lot from this installation and hope this tutorial will be helpful in providing a clear guide to a system with Btrfs RAID on top of dm-crypt.

Who This Tutorial is For

This article is written with a few assumptions. First, I assume that you have a general comfortability with the command line, as these steps have to be executed in a manual or “guided” CLI installation. Second, this tutorial is written for Arch Linux, however, the majority of the article will remain distro-agnostic as the main steps only concern disk setup and the boot process.

Brief Description of the Setup

To effectively describe how data is organized, processed, and stored, we will follow a file as it traverses down the data hierarchy to the disk. First, when we write a file, it will be stored inside of a directory, such as /etc/ or /home/sarah/ (indicated in the figure below as orange). That directory resides in a subvolume (kind of like a virtual partition, or LVM volume), such as “home”, “var”, or “etc” (magenta). Each subvolume is a member of the Btrfs pool (lavender), which consists of one or more device mappings (blue). These devices are dm-crypt containers that act as an intermediary step, encrypting the data before finally writing it to a particular hard drive (green).

Attribution

Much of the information from this article was stitched together from the Arch Wiki, primarily the page on Btrfs, and the series on dm-crypt. The use of the encrypt2 hook comes from a comment by AUR user benke, Btrfs subvolume organization and mounting the Btrfs top level subvolume to /var/lib/btrfs_root comes from the Arch Forums user WorMzy’s excellent post. Additional Btrfs insight and clarification was gained from Marc Merlin’s Linuxcon 2014 talk “Why you should consider using Btrfs, real COW snapshots and file level incremental server OS upgrades.”

Preparation
- Securely Wiping the Disks
- Planning Your Disk Layout
Partitioning the Disks
Setting Up dm-crypt
- Creating the Containers
- Opening the Containers
Creating the Filesystems
Mounting the Btrfs Filesystem and Creating Subvolumes
Assembling the / Filesystem
- A Bit about Btrfs Mount Options
- Mounting the Subvolumes and /boot Partition
Proceeding with Installation
- Generating the fstab file
Setting Up initramfs
- Creating Additional encrypt Hooks
- Editing /etc/mkinicpio.conf
Setting the Boot Options
Post Installation Considerations and Niceties
Conclusion

Preparation

Once you’re booted into a live environment, you’ll want to run lsblk to find and verify the disks you will be working with based on their size and any partitions on them. Since the naming of devices can change, you should double check the devices if you reboot. If there are any partitions or disks that have important data on them, back them up now.

Securely Wiping the Disks

To ensure that no sensitive data remains on hard drive, the disk will be overwritten with random data. To accomplish this, we will open temporary encrypted containers on each disk and write data to the entire disks.

First, use cryptsetup to set up the containers on each device to be used. The following command creates a dm-crypt plain container for each partition (specified as dev/sdXY) or disk (specified as /dev/sdX), named container_0, container_1, …, container_n.

# cryptsetup open --type plain /dev/sdXY container_n

Now that an encrypted container has been opened on each disk, any data written to it will appear to be suitably random. Therefore, you can use /dev/zero to generate data as it is much faster than /dev/urandom, which on my machine is ~100 times slower to read from. To overwrite to disk, simply execute the following dd command for each dm-crypt container you created.

# dd if=/dev/zero of=/dev/mapper/container_XY

Since writing a whole disk can take a long time, you can speed the process up by opening a new virtual terminal by pressing Ctrl+Alt+F2 and then executing the command on another disk. Note, it would not be effective to do this on two partitions of the same disk, as the bus of one device would be handling both loads.

If you are curious as to how the operation is progressing, simply open a new virtual terminal and execute

# watch -n 20 kill -USR1 $(pidof dd)

Then switch back to the virtual console of the dd command for updates every 20 seconds. When the operation is done, switch back to the terminal you executed watch in and press Ctrl+c.

Once every dd operation has completed, you can proceed on to the actual installation. I find that the easiest way to clean up the installation environment of the containers is to simply reboot.

Planning Your Disk Layout

Now you have to decide how exactly to layout the partitions and drives. In every case you will need a boot partition. Depending on your needs and the number of disks in your system, there a few considerations to be had. With one disk, the only partitioning considerations are whether or not you would like a swap partition (note: hibernate will not be possible with btrfs and dm-crypt) and/or additional free space for other operating systems. With two or more disks, you can utilize the built in RAID features of Btrfs. There are three options for btrfs raid:

RAID 0 - Data is striped, or distributed across all disks.
- Pros:
  - Write/Read speed increased because disks are accessed in parallel
  - Total usable space is equal to that of the disk combined
- Cons:
  - No redundancy. If one disk fails, entire filesystem is unusable.
RAID 1 - Data is mirrored, or copied to both devices.
- Pros:
  - Resiliency. All but one drive can fail without harming the filesystem
- Cons:
  - Total usable space is equal to that of the smallest disk
  - Read/Write speed is equal to that of the slowest disk.
RAID 1+0 - Data is striped and mirrored, as if a two or more RAID 1 pools were RAID 0’ed together.
- Pros:
  - Resiliency. All but one drive from each RAID 1 pool can fail without harming the filesystem
  - Write/Read speed increased because disks are accessed in parallel
  - Total usable space is equal to that of each RAID 1 pool combined
- Cons:
  - Requires 4 or more disks
  - Could have way more redundancy with 4 disks in RAID 1
  - Could have way more data throughput with 4 disks in RAID 0

Clearly there is no correct choice for every situation. In a file server where data integrity is key, go with RAID1 or RAID10. On a desktop or laptop workstation, RAID0 can greatly benefit one’s user experience. If you chose RAID 0, be sure to keep ample off-machine backups!

Partitioning the Disks

For remainder of the tutorial, I will write as though you are using two disks of different sizes in RAID0, a pretty common setup for one’s first foray into RAID :)

/dev/sda: 128G
/dev/sdb: 120G

For /dev/sda, create a 120G partition for one half of the btrfs pool, a 7.5G swap partition and a 500M boot partition.

# fdisk /dev/sda
Command (m for help): n
Partition number: <enter>
First sector: <enter>
Last Sector: +120G

Command (m for help): n
Partition number: <enter>
First sector: <enter>
Last Sector: +7.5G

Command (m for help): t
Partition number: <enter>
Hex code: 14

Command (m for help): n
Partition number: <enter>
First sector: <enter>
Last Sector: <enter>

Command (m for help): t
Partition number: <enter>
Hex code: 1

Command (m for help): w

Since btrfs can reside on either a partition or an entire disk, you do not have to partition /dev/sdb since you will be using the whole disk.

Setting up dm-crypt

With the disks partitioned, you can now setup the dm-crypt containers before installing the filesystems. You will have to set up two containers, one for each of the disks in the btrfs pool: /dev/sda1 and /dev/sdb.

dm-crypt has two types of containers with different benefits, “plain” and “luks”. The plain encryption sets up a “dumb” container. It has no knowledge that it is even encrypted until you open it, specifying every relevant parameter each time (keysize, cipher, hash). LUKS on the other hand, places a header on the disk that is aware of the parameters it was encrypted with and allows from multiple decryption keys. While LUKS seems like the obvious choice, one may decide to use plain if plausible deniability is a concern, as the disk will appear as simply random data to an adversary. For this tutorial, I will proceed using LUKS.

Creating the Containers

Different machines have different hardware support for the various crypto hashes and ciphers, so to find what settings will work best for you, check the results of the cryptsetup benchmark command.

# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1      1178175 iterations per second
PBKDF2-sha256     755458 iterations per second
PBKDF2-sha512     613920 iterations per second
PBKDF2-ripemd160  678250 iterations per second
PBKDF2-whirlpool  252061 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   611.1 MiB/s  2632.0 MiB/s
 serpent-cbc   128b    81.7 MiB/s   521.5 MiB/s
 twofish-cbc   128b   181.6 MiB/s   333.4 MiB/s
     aes-cbc   256b   453.3 MiB/s  2011.5 MiB/s
 serpent-cbc   256b    82.9 MiB/s   523.0 MiB/s
 twofish-cbc   256b   183.9 MiB/s   334.6 MiB/s
     aes-xts   256b  2240.3 MiB/s  2236.9 MiB/s
 serpent-xts   256b   520.6 MiB/s   506.5 MiB/s
 twofish-xts   256b   325.5 MiB/s   330.1 MiB/s
     aes-xts   512b  1732.0 MiB/s  1714.2 MiB/s
 serpent-xts   512b   520.0 MiB/s   506.3 MiB/s
 twofish-xts   512b   325.7 MiB/s   329.8 MiB/s

What one wants is the best balance of strength to performance. In terms of the hashes, less iterations per second is stronger, as it increases the amount of time needed to decrypt, which decreases the ability to brute force. Though by that metric whirlpool seems best, sha256 is considered suitable secure and quite standard. For the cipher, aes-xts clearly has the best performance on this computer. As a compromise of strength and speed, I will go with 256 bit aes-xts.

Based on those choices, the resulting cryptsetup command is

# cryptsetup --key-size 512 --hash sha256 --iter-time 5000 --use-random luksFormat /dev/sda1

The reason that the key size is 512 although we decided on 256 is explained on the Arch Wiki article on dm-crypt encryption:

XTS splits the supplied key in half. So to use AES-256 instead of AES-128 you would have to set the XTS key-size to 512.
The --iter-time option sets how long we spend hashing the passphrase. At 755458 iterations per second, the number of iterations is 755458*5=3777290.
By default cryptsetup 1.6.2 uses aes-xts, so the –cipher option is unnecessary.
The --use-random specifies to use /dev/random as an entropy (randomness) source as opposed to the default of /dev/urandom. The reason for this is that on a fresh, live environment, /dev/urandom may not have suitable entropy.

Upon executing the command, you will be prompted to enter a passphrase, pick one that is sufficiently strong.

Repeat the command for any remaining partitions of the btrfs pool, in this case just /dev/sdb. When prompted for a passphrase, it is best to enter one different from the other disk’s, as both passphrases are necessary to decrypt the btrfs pool. If using RAID1, the same passphrase could perhaps be used, as knowing either passphrase can access the pool.

Opening the Containers

Now, the LUKS containers have to be opened in order to use them as block devices. Once opened, the containers reside in the filesystem as /dev/mapper/container_name. The syntax for the cryptsetup open command is

# cryptsetup open --typeluks /dev/sdXY <container_name>

Where container_name is an arbitrary name for the container. We will use “btrfs_pool0” and “btrfs_pool1” for organizational purposes, so the commands are

# cryptsetup open --typeluks /dev/sda1 btrfs_pool0
# cryptsetup open --typeluks /dev/sdb btrfs_pool1

Run ls /dev/mapper and verify that both containers are there.

Creating the Filesystems

There are three filesystems to create in this setup: boot, swap and the btrfs pool. First create the boot partition.

If using a legacy BIOS motherboard, you can simply run mkfs.ext3 /dev/sda3. If you are on a computer with UEFI, you will need a EFI System Partition with a FAT32 filesystem, which can be created by running mkfs.fat -F32 /dev/sda3.

To create the swap partition, simply run mkswap /dev/sda2 and then swapon /dev/sda2.

To create the btrfs filesystem use the mkfs.btrfs command. You will need to specify the RAID level for both the data (-d) and the metadata (-m). We are using RAID0 for the data, but for the metadata we will use RAID1 so that it resides on both disks. Since metadata is both small and important, this is a good measure that results in virtually no performance loss.

# mkfs.btrfs -m raid1 -d raid0 /dev/mapper/btrfs_pool0 /dev/mapper/btrfs_pool1

Mounting the Btrfs Filesystem and Creating Subvolumes

Normally in an Arch Linux install, you would mount the root device to /mnt and begin constructing the filesystem hierarchy, but first we are going to utilize one of the coolest features of btrfs, subvolumes!

Subvolumes are kind of like partitions, except much more flexible. You do not have to specify a fixed size when creating a subvolume, although you can set disk quotas for each subvolume (for example, limit the size of each home folder). Arguably, the best feature of subvolumes is snapshots. A snapshot is like a backup of the subvolume at some point in time that is extremely fast and space efficient as it only stores changes from one point to another, also known as delta backups. This is made possible by the copy-on-write mechanism of btrfs.

To begin creating the various subvolumes, first make a directory to mount the btrfs pool: mkdir /mnt/btrfs. Now, mount either of the btrfs_pool containers to this directory (it does not matter which one you choose).

# mount /dev/mapper/btrfs_pool0 /mnt/btrfs

There are two common layouts of subvolumes: the parent method, and the sibling method. In the parent method, one would create a subvolume called “root” and inside of it place others, such as “home” (which could contain the subvolumes “alice” and “bob”), “etc” and “var”. This may seem more traditionally intuitive, but other than some ease in initial mounting during installation, it actually less ideal in terms of snapshotting, as you’ll see later. The sibling method places all snapshots at the same level of the core btrfs filesystem. The “root” subvolume, which will later be mounted as /, is at the same level of the subvolume hierarchy as “bob”, “alice”, “etc”, “var”, and so on. This makes snapshotting nicer in terms of scripting as all subvolumes live in the same directory.

Change your working directory to the btrfs mountpoint with cd /mnt/btrfs so you can easily create the subvolumes.

You create subvolumes using the btrfs subvolume create command with the syntax

# btrfs subvolume create <subvolume name> <location>

You will need to make a “root” subvolume, but other than that, it is up to you to decide what subvolumes you would like to create. For a desktop use-case, you may want a “home” subvolume if you are the only user, or, if not, then a subvolume for each user. A “var” subvolume can be handy for package management purposes, and perhaps you will want a “data” subvolume for shared files, such as music and movies. In a server scenario, you may want an “opt” subvolume for various services. I will demonstrate with the subvolumes I used (again, in the /mnt/btrfs/ directory):

# btrfs subvolume create root .
# btrfs subvolume create home .
# btrfs subvolume create var .
# btrfs subvolume create data .

Now run btrfs subvolume list . and verify that they are all there. You should see output similar to:

# btrfs subvolume list .
    ID 259 gen 6244 top level 5 path home
    ID 260 gen 4778 top level 5 path data
    ID 271 gen 5976 top level 5 path root
    ID 272 gen 6219 top level 5 path var

Now that the subvolumes are created, we can move on to assembling the filesystem hierarchy, just like in a traditional multi-partition install.

Assembling the / Filesystem

Begin by changing your working directory to /mnt/

# cd /mnt

Now create a directory that will be the mount point for the root filesystem (just as you would mount you / partition to /mnt or /mnt/arch in a normal installation) with mkdir /mnt/arch

A Bit about Btrfs Mount Options

You may have used some options with the mount command before such as noatime or defaults. Btrfs has some additional options to specify some of its features, most notably:

ssd: enables various optimizations for solid state disks
compress= : enables on the fly compression of the data before writing it, which uses less space and often, on modern hardware, is actually faster than non-compressed as hard drives (even SSDs) remain the bottleneck in data storage. The available choices are zlib (better compression) or lzo (faster compression)
subvol= : specify which subvolume is to be mounted

Mounting the Subvolumes and /boot Partition

The first step is to mount the “root” subvolume to the /mnt/arch directory, with any options you may want.

# mount -t btrfs -o defaults,noatime,ssd,compress=lzo,subvol=root /dev/mapper/btrfs_pool0 /mnt/arch

Now make the necessary directories that will be mount points for the other subvolumes and partitions:

# mkdir /mnt/arch/{boot,data,var,home}

And finish mounting the devices

# mount -t btrfs -o defaults,noatime,ssd,compress=lzo,subvol=home /dev/mapper/btrfs_pool0 /mnt/arch/home
# mount -t btrfs -o defaults,noatime,ssd,compress=lzo,subvol=data /dev/mapper/btrfs_pool0 /mnt/arch/data
# mount -t btrfs -o defaults,noatime,ssd,compress=lzo,subvol=var /dev/mapper/btrfs_pool0 /mnt/arch/var
# mount -o defaults,noatime /dev/sdc3 /mnt/arch/boot

There are two modifications we will now make to improve filesystem cleanliness and ease of btrfs administration. First, since log files are written to extremely frequently, you will want to disable copy-on-write for them to avoid fragmentation of thousands of unnecessary log file revisions.

# mkdir /mnt/arch/var/log/
# chattr +C /mnt/arch/var/log/

The second modification you’ll make will help with later snapshotting and other btrfs operations. Remember how easy creating subvolumes was when you mounted the top level btrfs pool to /mnt/btrfs? You can retain this functionality by mounting the btrfs pool with no “subvol” option somewhere in your filesystem. I will do this at /var/lib/btrfs_root, but you are welcome to place this where ever you like, such as /data/btrfs_pool.

# mkdir /mnt/arch/var/lib/btrfs_root
# mount -t btrfs -o defaults,noatime,ssd,compress=lzo /dev/mapper/btrfs_pool0 /mnt/arch/var/lib/btrfs_root

Proceeding with Installation

Now that the filesystem is assembled, you can essentially follow the rest of the Arch Installation guide with only a few slight differences.

Executing `pacstrap`

When running the pacstrap command, you need to include an additional package in order to have the btrfs tools you have been using. Also, since you mounted your filesystem at /mnt/arch instead of /mnt/ as used in the guide, you need to account for that:

# pacstrap /mnt/arch base btrfs-progs

Generating the fstab File

In order for the system to properly locate and mount all of the partitions and subvolumes at boot, a filesystem table needs to exist. Arch Linux has a great tool for simplifying the process, genfstab. Although the Arch installation guide does not specify this option, the ‘-U’ flag is quite important, since the order the devices are recognized could change (switching what is /dev/sda and /dev/sdb for example). This is crucial if you decide to have an encrypted swap partition, which we’ll cover later.

Before writing the fstab to new installation, first run it without the output redirection (“»”) and verify it against ls -l /dev/disk/by-uuid

# genfstab -p -U /mnt/arch
# ls -l /dev/disk/by-uuid

If everything looks good, write it to the new installation

# genfstab -p -U /mnt/arch >> /mnt/arch/etc/fstab

Further Configuration

Run arch-chroot /mnt/arch to chroot into the new system and follow the steps in Configure the System until Configure /etc/mkinitcpio.conf, where you will need to make some slight modifications.

Setting Up initramfs

In order to decrypt devices when starting the system, you need to include the “encrypt” hook in your initramfs. There is a slight problem though, in that the encrypt hook only allows you to specify 1 device to decrypt. Since your btrfs pool consists of 2 or more devices this will need some modifications. The “encrypt” hook is really just a shell script that lives at /lib/initcpio/hooks/encrypt, with a corresponding install script that lives at /lib/initcpio/install/encrypt.

All you need to do is create copies of the two scripts make some slight modifications.

Creating Additional encrypt hooks

# cp /lib/initcpio/hooks/encrypt /lib/initcpio/hooks/encrypt2
# cp /lib/initcpio/install/encrypt /lib/initcpio/install/encrypt2

The install/encrypt2 script is fine as-is, but you need to make some slight adjustments to /lib/initcpio/hooks/encrypt2. Using your favorite text editor, change every occurrence of “$cryptkey” to “$cryptkey2” (there should be 2 occurrences), and “$cryptdevice” to “$cryptdevice2” (2 occurrences). Additionally, find the line “mkdir /ckey” and remove it to avoid an annoying but harmless error message upon startup.

If you are using more than 2 encrypted disks for your pool, simply repeat the process for encrypt3, encrypt4, etc.

Editing /etc/mkinitcpio.conf

Now it is time to tell the initramfs to include the necessary components upon being built. Open up /etc/mkinitcpio.conf in your favorite text editor. Find the MODULES="" line and change it to MODULES="nls_cp437 crc32c" note: if you would like to use a keyfile to automatically unlock your devices on boot when in a safe place, include the kernel module for the unencrypted filesystem it will reside on, such as vfat or ext3. Setting this up will be covered later in the tutorial. Now find the uncommented line beginning with HOOKS= and change it to HOOKS="base udev autodetect encrypt encrypt2 modconf block filesystems keyboard fsck btrfs" in that order.

Now simply run mkinicpio -p linux to generate the initramfs.

Setting the Boot Options

In order to decrypt the disks and mount the “root” subvolume to /, you need to set a few kernel options in your boot loader configuration. The specifics may be slightly different depending on your choice of bootloader, but generally there is an “options” line in the configuration for each boot entry.

To unlock the encrypted LUKS containers, use the cryptdevice= and cryptdevice2= (and any additional) options from your encrypt hooks. The format for this option is

# cryptdevice=UUID=<uuid-of-disk-or-partition>:<name-of-encryption-container>:allow-discards

Again, because device names of the form /dev/sdXY can change, UUIDs should be used. I find it helpful to append all of the disk UUIDs to the end of the boot entry for easy copy and pasting: ls -l /dev/disk/by-uuid/ >> /path/to/boot/entry, just be sure to delete all of the lines appended by ls before saving the file!

To indicate that you want to mount your Btrfs pool’s “root” subvolume as /, use the standard root= parameter but also include a rootflags= parameter like so:

# root=/dev/mapper/btrfs_pool0 rootflags=subvol=root

Putting everything together you get something like:

title       Arch Linux
linux       /vmlinuz-linux
initrd      /initramfs-linux.img
options      cryptdevice=UUID=72347efa-f59b-de3a-42fe-02849feacc72:btrfs_pool0:allow-discards cryptdevice2=UUID=987aefcc-f119-56aa-43c1-48cffeaa891e:btrfs_pool1:allow-discards root=/dev/mapper/btrfs_pool0 rootflags=subvol=root quiet rw

Now the base system is completely installed an ready to boot! Feel free to add some users, install some other simple packages you prefer, or any other basic administrative tasks before you reboot into the installation. When you’re all set, type exit to leave the chroot environment, unmount the subvolumes and disks with umount -R /mnt/arch and then reboot. If all goes well, you will boot into your new installation, asking you for the passphrase for each disk. If there are any problems, boot back into the live disk, mount your filesystems and compare the configuration files to the ones listed in this article. The most likely culprits are your boot configuration, /etc/fstab, and /etc/mkinitcpio.conf.

Post Installation Considerations and Niceties

A Warning About Kernel Updates

In the past, some kernel updates have been known to cause corruption to the filesystem, most recently in the 3.19.1 kernel where a deadlock prevented access. The issue was easily repairable, and is now fixed, however instability in your data storage can be frightening. To prevent this type of surprise, there are a few things that can be done. The simplest, if your hardware is supported, is to switch to a Long Term Support (LTS) kernel. These are released far less frequently and geared to provide the utmost stability. They are generally heavily tested before deployment to ensure that no bugs of this sort can creep up. If you must run a newer kernel, I recommend a reasonable external back up system of your most important files and waiting a few days to upgrade after a new kernel version is released, in hopes that any such bugs can be discovered and promptly patched.

Snapshots

Snapshotting, again, is a wonderful feature of Btrfs that allows us to take fast, lightweight backups of a subvolume. Each snapshot essentially freezes a file in in the state is it at that moment. If a file hasn’t changed since a year ago, that file in each snapshot is the same exact location on the disks, a file in a snapshot only takes up space if it were to change. Because of Btrfs’ copy-on-write mechanisms, when you change a file, it is written to a new location and the previous snapshots are unaffected, and if you were to delete a file that was previously snapshotted, the file tells Btrfs “Hey, I’m still wanted in this one snapshot…so don’t forget about me!” and those blocks are not freed. If something were to happen to a file or directory, we can simply roll back to an appropriate snapshot and retrieve that copy.

To make a snapshot, first change directories into the /var/lib/btrfs_root directory you made earlier. Run ls and you should see all of the subvolumes you created. Feel free to poke around inside of them and see exactly how the system is assembled. When you’re ready to make a snapshot, run mkdir snapshots to create a directory to store them in. This is just a standard POSIX directory at the top most level of your Btrfs system.

I like to organize my snapshots by date, so I usually create a new subdirectory in /var/lib/btrfs_root/snapshots named after the date. For example, mkdir snapshots/$(date +"%F_%H-%M") would create the directory snapshots/2015-09-02_14-56/ because I ran it at 2:56pm on September 2nd, 2015. This way of organizing snapshots works for me, but by all means, use something that is intuitive to you! To actually create the snapshot use your good pal, the btrfs command like so:

# btrfs subvolume snapshot <subvolume name> snapshots/2015-09-02_14-56/

So for the system installed in this article:

# btrfs subvolume snapshot root snapshots/2015-09-02_14-56/
# btrfs subvolume snapshot home snapshots/2015-09-02_14-56/
# btrfs subvolume snapshot var snapshots/2015-09-02_14-56/
# btrfs subvolume snapshot data snapshots/2015-09-02_14-56/

To verify, run ls in the snapshot/2015-09-02_14-56/ directory. You don’t have to snapshot every subvolume at the same time, for instance, your personal data in home is likely to change more and you may want to snapshot more often to be able to revert to a very recent state.

Btrfs snapshots are very powerful, and can be scripted by hand or with various tools such as openSUSE’s Snapper. These operations are outside of the scope of this tutorial, but are by no means difficult to do, so have fun checking them out!

Keyfiles and Additional LUKS Keys

One of the most appealing aspects of the LUKS containers is the ability to have up to 8 keys that are able to unlock the container. The keys can be text entered interactively at a prompt, like a standard password, or keyfiles, any standard file, such as a picture, a song, a pdf…anything you could store on a hard drive! Additional text keys are helpful to have an unlocking mechanism in case you forget or damage another key. Keyfiles are nice as a quick, automatic way to decrypt devices when in a safe place, or to have a key much longer than a memorable text one.

For example, my ThinkPad has an SD card slot. When I am at home, a keyfile on the card is used to automatically decrypt the drives in my Btrfs pool at boot. If I leave the house, I simply eject the SD card, and when I startup the computer, I am asked for the passphrases for both of the involved disks.

To add new key, use the cryptsetup luksAddKey command. To add a key to the LUKS container at /dev/sda1 for example:

# cryptsetup luksAddKey /dev/sda1

The command will prompt you to type in an existing key and then have you type the new passphrase twice.

If you would like to add a keyfile instead, simply add the path of the file to your command:

# cryptsetup luksAddKey /dev/sda1 /path/to/file

Once you enter in an existing key, the keyfile will be added. Revoking keys is not difficult, but not in the scope of this article. For more info on LUKS management, check out the Arch Wiki page dm-crypt/Device encryption.

Assuming that you added a keyfile to each partition’s LUKS header, you may now setup your computer to use those upon boot thanks to the encrypt hook. Change your boot loader configuration entry’s options line to include cryptkey and cryptkey2 options in the format cryptkey=/dev/disk/by-uuid/xxxx-xxxx:<filesystem type>:/path/from/toplevel/of/device For example, to use a keyfile on my SD card I would have the following options line:

options     cryptdevice=UUID=6e3e4026-5b53-4bc0-8980-ffe59765f85d:btrfs_pool0:allow-discards cryptdevice2=UUID=16d8e0bf-5384-4f5d-a785-9c7eaf775fa4:btrfs_pool1:allow-discards root=/dev/mapper/btrfs_pool0 rootflags=subvol=root cryptkey=/dev/disk/by-uuid/F224-CEA1:vfat:/cats.gif cryptkey2=/dev/disk/by-uuid/F224-CEA1:vfat:/cats.gif quiet rw

Though absurdly long, it is important (at least for my bootloader) that it all to be on one line.

Now reboot, and if your keyfile is present on the specified removable media, your LUKS containers should automatically unlock themselves. If your removable media is not present, you will be brought to the standard prompt to type a key in by hand.

Encrypted Swap

If you chose to have a swap partition on your system, it is a good idea to encrypt it. The process is quite simple and unobtrusive, requiring just a slight change to both /etc/fstab and /etc/crypttab, with no additional passwords or anything at boot since they are encrypted with a one-time random “throwaway” passphrase.

First, in /etc/crypttab, find the line with the swap entry:

# swap         /dev/sdaX        /dev/urandom    swap,cipher=aes-cbc-essiv:sha256,size=256

and uncomment it. Then replace /dev/sdaX with the UUID of your swap partition in the same way you did for /etc/fstab and our boot options. It is extremely important to use the UUID here!: in the boot options or /etc/fstab, if the wrong partition was referenced, the system would simply not boot until it is fixed. Here, however, the partition would be rendered completely corrupted after mkswap gets called and a dm-crypt container is created.

Now edit /etc/fstab. Find the line for the swap partition and change the partition to /dev/mapper/swap like so:

/dev/mapper/swap    none        swap        defaults    0 0

this indicates that swap partition will be the “swap” container that you just told /etc/crypttab to create at boot time.

Conclusion

I hope that you have found this guide to be informative and helpful. While it is quite thorough in explaining the set up process, there is much more to learn and use now that your system is up and running. From automated snapshot management, to LUKS header backup and removal, from Btrfs filesystem hygiene practices, to further securing your system; there is a plethora of further fun, security and data integrity topics to explore with these tools. Have fun and enjoy.

feel free to contact me with any comments or corrections via the options listed at the bottom of the page.

kyle kneitinger

Perplexing Config Lexeme Conflicts in the i3 Window Manager

The Issue

Why is it happening?

How Variables Are Parsed

So What Went Wrong?

Fixing the Issue

Longest Lexeme Based Parsing

Implementing Longest Lexeme Matching

Ensuring Beautiful Commits with rustfmt and Travis-CI

When Style is Standardized, Style Can Be Standard

Enter Travis

Installing the Tool

Checking the Style

SysEx-ploration

Motivation

Idea

The System Exclusive Specification

Dumping the Data

Deciphering the Data

Working with the Data

Moving Forward

Picnote

A Tool is Born from Horrible UI

Components of picnote

How it works

Usage

Btrfs on dm-crypt

Who This Tutorial is For

Brief Description of the Setup

Attribution

Contents

Preparation

Securely Wiping the Disks

Planning Your Disk Layout

Partitioning the Disks

Setting up dm-crypt

Creating the Containers

Opening the Containers

Creating the Filesystems

Mounting the Btrfs Filesystem and Creating Subvolumes

Assembling the / Filesystem

A Bit about Btrfs Mount Options

Mounting the Subvolumes and /boot Partition

Proceeding with Installation

Executing pacstrap

Generating the fstab File

Further Configuration

Setting Up initramfs

Creating Additional encrypt hooks

Editing /etc/mkinitcpio.conf

Setting the Boot Options

Post Installation Considerations and Niceties

A Warning About Kernel Updates

Snapshots

Keyfiles and Additional LUKS Keys

Encrypted Swap

Conclusion

Executing `pacstrap`