Musescore is a wonderful tool. It has made a huge impact on my musical development over the past couple decades. Sheet music is the primary way I communicate and record musical ideas, and Musescore the tool and musecore.com have combined to make a process that works for me and my musical needs.

I have also spent a few years writing software, and the methods that we have learned to use in software development have evolved due to needs of scale and flexibility. I would like to apply some of those lessons to how I manage sheet music. But there are disconnects.

The biggest issue I have is that I want the same core song in multiple different but related formats. The second biggest issue is that I want to be able to make changes to a song, and to collaborate with other composers in making those changes.

The Sheet music I work with is based on the western notation system. I use a fairly small subset of the overall notation system, as what I am almost always working towards is a musical arrangement for small groups. The primary use case I have is for lead sheets: a melody line and a series of chords for a single instrument. Often, this is for a horn player. I need to be able to transpose the melody line into the appropriate key and range for that particular horn: E flat for a Baritone sax, C for a Flute, B flat for a saxophone, C but in Bass clef for a Trombone.

The second use case I have is to be able to arrange a section harmonically for a set of instruments. This means reusing the melody from a lead sheet, but then selecting notes for the other instruments to play as accompaniment. Often, this can be done on piano first, and then allocation the notes the piano plays out to the single-note horns.

I also wish to make play-along recordings, using ireal pro. This uses a very small subset of the lead sheet information: really just the chords and the repeats. A more complex version could be used to build MMA versions.

When I work with a more complex arrangement, I often find that I change my mind on some aspect: I wish to alter a repeat, a chord symbol, or the set of notes in the melody line.ideally that would be reflected through all the various aspects of the score.

The musescore files are stored in an xml format developed for the program. This XML file is compressed, and the the file extension reflects this: mscz. I have a song that is 40 measures long (32 with 8 bars repeated) and no more than 2 chords per measure, no more than 8 notes per measure. The underlying document used to store this song, when transposed for 6 different instruments is 22037 lines long. This is not a document format meant to be consumed by humans.

Here is a sample of how a single note is represented:

        <Articulation name="marcatoStaccato">
          <velocity>120</velocity>
          <gateTime>50</gateTime>
          </Articulation>

Here is an example for a chord

 <Chord>
            <linkedMain/>
            <durationType>quarter</durationType>
            <Note>
              <linkedMain/>
              <pitch>74</pitch>
              <tpc>16</tpc>
              </Note>
            </Chord>

This is generated code. When software is responsible for creating more software, it will often product the output in a format that can then be consumed by another tool designed to read human readable source and convert it to binary. XML is a notoriously chatty format, and the number of lines in the Musescore file reflects that.

The “document” that we humans interface with is based on pen and paper. If I were to do this by hand, I would start with a blank page of pre-printed staff paper, and annotate on it such details as the clef, the key signature, the bar lines, the notes, and the chord symbols. This system is optimized for hand-written production, and human reading-on-the-stage consumption. This is how it is displayed on the screen as well. Why, then is the underlying file format so unfriendly?

Because file formats are based on text, and we have restricted that to very specific rules as well: Characters are represented as numeric values (ascii now unicode) and anything implying the layout on the page needs to be embedded in the document as well.

There are options for turn text into sheet music: ABC and Lilypond are both formats that do this. Why don’t I use those? I have tried in the past. But when I am writing music, I am thinking in terms of notation, not in terms of text, and they end up preventing me from doing what I need. They don’t solve the transposition or other problems as is. Perhaps the issue is that we need more tooling around one of these format, but then we return to the problem of generated code.

Once the sheet music is printed out, the options for annotating a revision is to write on the sheet music, or to edit the original source and reprinting it. In practice, both of these are done often. IN front of me on my desk is a copy of a tune where both chords and individual notes have been changed in pencil. After living with these revisions for quite some time, I eventually got them recorded back into the source file and reprinted it.

Carrying around reams of sheet music quickly becomes unfeasible. If you are in multiple groups, each of which have a large repertoire, the need to reduce the physical burden will likely lead you to an electronic solution: sheet music displayed on a tablet. However, the way that you distribute sheet music here will determine what options you have allowed for the artist to annotate corrections and instructions on the music in front of them: most PDFs don’t let you edit them, and you cannot write on a screen with a ball point pen.

As a band director, I would like to be able to note changes on a score for a particular group and automate the distribution of that change to the group.

As a band member I would like to be able to make changes to a score and have them show up when I view the score electronically. Ideally, these changes would be reflected in the ireal/MMA version that I use to practice as well as the sheet music I read.

As a collaborator, I would like to be able to submit a suggested change to a score as played by my group and have the music director be able to incorporate my suggestion into the score.

As a collaborator, I would like to be able to receive suggestions from another collaborator, and provide feedback on the suggestion if I decide not to include it as-is.

As an arranger, I would like to be able to come up with a score on piano and automate the process of converting it to a set of instruments.

As a band leader, I would like to be able to come up with a lead sheet in Musescore and automate the process of converting it to MMA and irealpro formats. I would like to be able to import the accompaniment from an irealpro or MMA format into a musescore document as a first pass for the percussion or other rhythm section arrangement.

I had long thought about an application to make it possible to work on music together with other people. At West Point we used to say Collaborate and Graduate. For this, I say:

Collaborate and Orchestrate.

Any new technology requires a mental effort to understand. When trying to automate the boring stuff, one decision I have to make is whether to use straight shell scripting or whether to perform that operation using Ansible. What I want to do is look at a simple Ansible playbook I have written, and then compare what the comparable shell script would look like to determine if it would help my team to use Ansible or not in this situation.

The activity is building a Linux Kernel that comes from a series of topic branches applied on top of a specific upstream version. The majority of the work is done by a pre-existing shell script, so what we mostly need to do is git work.
Here is an annotated playbook. After each play, I note what it would take to do that operation in shell.

---
- name: Build a kernel out of supporting branches
  hosts: servers
  remote_user: root
  vars:
    #defined in an external vars file so we can move ahead
    #kernel_version: 
    #soc_id: 
    test_dir: /root/testing
    linux_dir: "{{ test_dir }}/linux"
    tools_dir: "{{ test_dir }}/packaging"

  tasks:

  - name:  ssh key forwarding for gitlab
    ansible.builtin.copy:
      src: files/ssh.config
      dest: /root/.ssh/config
      owner: root
      group: root
      mode: '0600'
      backup: no
  #scp $SRCDIR/files/ssh.config $SSH_USER@SSH_HOST:/root/.ssh/ssh.config
  #ssh $SSH_USER@SSH_HOST chmod 600/root/.ssh
  #ssh $SSH_USER@SSH_HOST chown root:root /root/.ssh/ssh.config

  - name: create testing dir
    ansible.builtin.file:
      path: /root/testing
      state: directory
      #ssh $SSH_USER@SSH_HOST mkdir -p /root/testing


  - name: Install Build Tools
    ansible.builtin.yum:
      name: make, gcc, git, dwarves, openssl, grubby, rpm-build, perl
      state: latest
    #ssh $SSH_USER@SSH_HOST yum -y install make, gcc, git, dwarves, openssl, grubby, rpm-build, perl

  - name: git checkout Linux Kernel
    ansible.builtin.git:
      repo: git@gitlab.com:{{ GIT_REPO_URL }}/linux.git
      dest: /root/testing/linux
      version: v6.5
#This one is a bit more complex, as it needs to check if the repo already 
#exists, and, if so, do a pull, otherwise do a clone.

  - name: add stable stable Linux Kernel repo
    ansible.builtin.command: git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
    changed_when: false
    args:
      chdir: /root/testing/linux
    ignore_errors: true
#there should be and Ansible git command for adding an additional remote.  
#I could not find it, so I resported to command.  
#This is identical to running via ssh

  - name: fetch stable stable Linux Kernel repo
    ansible.builtin.command: git fetch stable
    args:
      chdir: /root/testing/linux
#Same issue as above.  This shows that, when an Ansible command is
#well crafted, it can link multiple steps into a single command, reduce the #need for an additional ssh-based command.

  - name: git checkout gen-early-patches
    ansible.builtin.git:
      repo: git@gitlab.com:{{ GIT_REPO_URL }}/packaging.git
      dest: "{{ tools_dir }}"
      version: main
#same issue as with the clone for  the Linux Kernel repository

  - name: generate early kernel patches
    ansible.builtin.shell:
      cmd: "{{tools_dir }}/git-gen-early-patches.sh {{ tools_dir }}/gen-{{ soc_id }}-{{ kernel_version }}-general.conf"
      chdir: /root/testing/linux
#One benefit to running with Ansible is that it will automatically
#wrap a shell call like this with an error check.  This cuts dowm
#on boilerplate code and the potential to miss one.

  - name: determine current patch subdir
    ansible.builtin.find:
      paths: /root/testing/linux
      use_regex: true
      #TODO build this pattern from the linux kernel version
      pattern: ".*-v6.7.6-.*"
      file_type: directory
    register: patch_directory
#This would probably be easier to do in shell:
#BUILD_CMD=$( find . -name ".*-v6.7.6-.*/build.sh" | sort | tail -1 )
#

  - ansible.builtin.debug:
      msg: "{{ patch_directory.files | last }}"

  - set_fact:
      patch_dir: "{{ patch_directory.files | last }}"

  - ansible.builtin.debug:
      msg: "{{ patch_dir.path }}/build.sh"


  - name: build kernel
    ansible.builtin.shell:
      cmd: "{{ patch_dir.path }}/build.sh"
      chdir: /root/testing/linux
#Just execute the  value of BUILD_CMD
#ssh $SSH_USER@SSH_HOST /root/testing/linux/$BUILD_CMD

So, should this one be in Ansible or shell? It is a close call. Ansible makes it hard to do shell things, and this needs a bunch of shell things. But Ansible is cleaner in doing stuff on a remote machine from a known starting machine, which is how this is run: I keep the Ansible playbook on my Laptop, connect via VPN, and run the playbook on a newly provisioned machine, or rerun it on a machine while we are in the process of updating the Kernel version, etc.

This use case does not make use of one of the primary things that Ansible is very good at doing: running the same thing on a bunch of machines at the same time. Still, it shows that Ansible is at least worth evaluating if you are running a workflow that spans two machines, and has to synchronize state between them. For most tasks, the Ansible play will be sufficient, and falling back to Shell is not difficult for most other tasks.

After you get something working, you find you might have missed a step in documenting how you got that working. You might have installed a package that you didn’t remember. Or maybe you set up a network connection. In my case, I find I have often brute-forced the SSH setup for later provisioning. Since this is done once, and then forgotten, often in the push to “just get work done” I have had to go back and redo this (again usually manually) when I get to a new machine.

To avoid this, I am documenting what I can do to get a new machine up and running in a state where SSH connections (and forwarding) can be reliably run. This process should be automatable, but at a minimum, it should be understood.

To start with, I want to PXE boot the machine, and reinstall the OS. Unless you are using a provisioning system like Ironic or Cobbler, this is probably a manual process. But you still can automate a good bit of it. The first step is to tell the IPMI based (we are not on Red Fish yet) BMC to boot to PXE upon the next reboot.

This is the first place to introduce some automation. All of our ipmitool commands are going to take the majority of the same parameters. So we can take the easy step of creating a variable for this command, and use environmental variables to fill in the repeated values.

export CMD_IPMITOOL="ipmitool -H $DEV_BMCIP -U $IPMI_USER -I lanplus -P $IPMI_PASS"

One benefit of this is that you can now extract the variables into an environment variable file that you source separate from the function. That makes this command reusable for other machines.

To require PXE booting on the next pass we also make use of a function that can be used to power cycle the system. Note that I include a little bit of feedback in the commands I use so that the user does not get impatient.

dev_power_on(){
        CMD_="$CMD_IPMITOOL power on"
        echo $CMD_
        $CMD_
}

dev_power_off(){
        CMD_="$CMD_IPMITOOL power off"
        echo $CMD_
        $CMD_
}
dev_power_cycle(){
        dev_power_off
        dev_power_on
        dev_power_status
}

dev_pxe(){
       CMD_="$CMD_IPMITOOL  chassis bootdev pxe" 
        echo $CMD_
        $CMD_
        dev_power_cycle
}

Once this function is executed, the machine will boot to PXE mode. What this looks like is very dependent on your setup. There are two things that tend to vary. One is how you connect to the machine in order to handle the PXE setup. If you are lucky, you have a simple interface. We have a serial console concentrator here, so I can connect to the machine using a telnet command: I get this command from our lab manager. IN other stages of life, I have had to use minicom to connect to a physical UART (serial port) to handle PXE boot configuration. I highly recommend the serial concentrator route if you can swing it.

But usually you have an IPMI based option to open the serial console. Just be ware that this might conflict with, and thus disable, a UART based way of connecting. For me, I can do this using:

$CMD_IPMITOOL sol activate

The other thing that varies is your PXE set up. We have a PXE menu that allows us to select between many different Linux distributions with various configurations. My usual preference is to do a minimal install, just enough to get the machine up and on the network accessible via SSH. This is because I will almost always do an upgrade of all packages (.deb/.rpm) on the system once it is booted. I also try to make sure I don’t have any major restrictions on disk space. Some of the automated provisioning approaches make the Root filesystem or the home filesystem arbitrarily small. For development, I need to be able to build a Linux Kernel and often a lot of other software. I don’t want to run out of disk space. A partitioning scheme that is logical for a production system may not work for me. My Ops team provides and option that has Fedora 39 Server GA + Updates, Minimal, Big Root. This serves my needs.

I tend to reuse the same machines, and thus have ssh information in the files under ~/.ssh/known_hosts. After a reprovision, this information is no longer accurate, and needs to be replaced. In addition, the newly provisioned machine will not have an ssh public key on it that corresponds with my private Key. If only they used FreeIPA…but I digress…

If I try to connect to the reprovisioned machine:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:EcPh9oazsjRaC9q8fJqJc8OjHPoF4vtXQljrHJhKDZ8.
Please contact your system administrator.
Add correct host key in /home/ayoung/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/ayoung/.ssh/known_hosts:186
  remove with:
  ssh-keygen -f "/home/ayoung/.ssh/known_hosts" -R "10.76.111.73"
Host key for 10.76.111.73 has changed and you have requested strict checking.
Host key verification failed.

The easiest way to wipe the information is:

ssh keygen -R $DEV_SYSTEMIP

Coupling this with provisioning the public key makes sense. And, as I wrote in the past, I need to set up ssh-key forwarding for gitlab access. Thus, this is my current ssh prep function:

#Only run this once per provisioning
dev_prep_ssh(){
        ssh-keygen -R $DEV_SYSTEMIP
        ssh-copy-id -o StrictHostKeyChecking=no root@$DEV_SYSTEMIP
        ssh-keyscan gitlab.com 2>&1  | grep -v \# | ssh root@$DEV_SYSTEMIP "cat >> .ssh/known_hosts"
}

The first two could be done via Ansible as well. I need to find a better way to do the last step via Ansible (line_in_file seems to be broken by this), or to bash script it so that it is idempotent.

Much of my work is done on machines that are only on load to me, not permanently assigned. Thus, I need to be able to provision them quickly and with a minimum of fuss. One action I routinely need to do is to check code out of a git server, such as gitlab.com. We use ssh keys to authenticate to gitlab. I need a way to do this securely when working on a remote machine. Here’s what I have found

Key Forwarding

While it is possible to create an ssh key for every server I use, that leads to a mess. As important, it leads to an insecure situation where my ssh keys are sitting on machines that are likely to be reassigned to another user. To perform operations on git over ssh, I prefer to use key forwarding. That involves setting up on the development host a .ssh/config file that loos like this:

Host *.gitlab.com
   ForwardAgent yes

Depending on your setup, you might find it makes sense to just copy this file over as is, which is what I do. A more flexible scheme using something that appends these entries if they are non-existent may make sense if you are using Ansible and the ssh_config module or a comparable tool.

known_host seeding

When you first ssh to a development host, there is a likelihood that it will not know about the git server host. In order to make connections without warning or errors, you need to add the remote hosts fingerprints into the ~/.ssh/know_hosts files. This one-liner can do that:

ssh-keyscan gitlab.com 2>&1  | grep -v \# | ssh $DEV_USER@$DEV_HOST  "cat >> .ssh/known_hosts"

ssh-keyscan will produce output like this:

# gitlab.com:22 SSH-2.0-GitLab-SSHD
gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9
# gitlab.com:22 SSH-2.0-GitLab-SSHD
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
# gitlab.com:22 SSH-2.0-GitLab-SSHD
gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
# gitlab.com:22 SSH-2.0-GitLab-SSHD
# gitlab.com:22 SSH-2.0-GitLab-SSHD

So I remove the comments and just add the fingerprints.

I tried to get this to work using Ansible and the lineinfile module, but I got an error 127…not sure why.

EDIT: I have corrected it. I should have used with_items, not with_lines, and ssh_keyscan_output.stdout_lines

---
- name: Set up ssh forwarding for gitlab 
  hosts: servers
  remote_user: root

  tasks:

  - name: keyscan gitlab.com
    command: ssh-keyscan gitlab.com
    register: ssh_keyscan_output

  - name: Save key fingerprints
    ansible.builtin.lineinfile:
      path: /root/.ssh/known_hosts
      line: "{{ item }}"
    with_items: " {{ ssh_keyscan_output.stdout_lines }}"

But something like that should be possible. When I did not first pre-seed the fingerprint, and I tried to do a git checkout over ssh, I would get this error:

 ssh root@10.76.111.74 "cd testing ;  git clone git@gitlab.com:$REMOTE_REPO "
bash: line 1: cd: testing: No such file or directory
Cloning into 'kernel-tools'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I saw a comparable error over Ansible. The solution was to run the one liner I posted above.

EDIT: One thing I did not make expliciti is that you need to enable ssh forwarding in your ssh command:

        ssh $DEV_USER@$DEV_SYSTEMIP -A

My Band, The Standard Deviants, had a rehearsal this weekend. As usual, I tried to record some of it. As so often happens, our best performance was our warm up tune. This time, we performed a tune called “The SMF Blues” by my good friend Paul Campagna.

What is going on here? Quite a bit. People talk about Jazz as improvisation, but that does not mean that “Everything” is made up on the spot. It takes a lot of preparation in order for it to be spontaneous. Here are some of the things that make it possible to “just jam.”

Blues Funk

While no one in the band besides me knew the tune before this session, the format of the tune is a 12 bar blues. This is probably the most common format for a jam tune. A lot has been written on the format elsewhere, so I won’t detail here. All of the members of this group know a 12 bar blues, and can jump in to one after hearing the main melody once or twice.

The beat is a rock/funk beat set by Dave, our Drummer. This is a beat that we all know. This is also a beat that Dave has probably put 1000 hours into playing and mastering in different scenarios. Steve, on Bass, has played a beat like this his whole career. We all know the feel and do not have to think about it.

This song has a really strong turn around on the last two bars of the form. It is high pitched, repeated, and lets everyone know where we are anchored in the song. It also tells people when a verse is over, and we reset.

Signals

The saxophone plays a lead role in this music, and I give directions to the members of the band. This is not to “boss” people around, but rather to reduce the number of options at any given point so that we as a unit know what to do. Since we can’t really talk in this scenario, the directions have to be simple. There are three main ways I give signals to the band.

The simplest is to step up an play. As a lead instrument, the saxophone communicates via sound to the rest of the band one of two things; either we are playing the head again, or I am taking a solo. The only person that really has to majorly change their behavior is Adam 12 on Trombone. Either he plays the melody with me or he moves into a supporting role. The rest of the band just adjusts their energy accordingly. We play louder for a repetition of the head, and they back off a bit for a solo.

The second way I give signals to the band is by direct eye contact. All I have to do is look back at Dave in the middle of a verse to let him know that the next verse is his to solo. We have played together long enough that he knows what that means. I reinforce the message by stepping to the side and letting the focus shift to him.

The third way is to use hand gestures. As a sax player, I have to make these brief, as I need two hands to play many of the notes. However, there are alternative fingerings, so for short periods, I can play with just my left hand, and use my right to signal. The most obvious signal here is the 4 finger hand gesture I gave to Adam 12 that we are going to trade 4s. This means that each of us play for four measures and then switch. If I gave this signal to all of the band, it would mean that we would be trading 4s with the drums, which we do on longer form songs. Another variation of this is 2s, which is just a short spot for each person.

One of the wonderful thing about playing with this band is how even “mistakes” such as when i tried to end the tune, and no one caught the signal…and it became a bass solo that ended up perfectly placed. Steve realized we all quieted out, and that is an indication for the Bass to step up, and he did.

Practice Practice Practice

Everyone here knows their instrument. We have all been playing for many decades, and know what to do. But the horn is a harsh mistress, and she demands attention. As someone once said:

Skip a day and you know. Skip two days and your friends know. Skip three days and everyone knows.

Musical Wisdom from the ages

Adam 12 is the newest member of the band. It had been several years since he played regularly before he joined us. His first jam sessions were fairly short. We have since heard the results of the hard work that he has put in.

I try to play every day. It completes with many other responsibilities and activities in modern life. But my days seems less complete if I did not at least blow a few notes through the horn.

Listening

Music is timing. Our ears are super sensitive to changes in timing, whether at the micro level, translating to differences in pitch, or the macro level, with changes in tempo…which is just another word for time. Dave is the master of listening. He catches on to a pattern one of us it playing and he works it into his drumming constantly. Steve is the backbone of the band. Listening to the bass line tells us what we need to know about speed and location in the song. The more we play together, the more we pick up on each others cues through playing. The end effect is that we are jointly contributing to an event, an experience, a performance.

I have been getting in the pool for the past couple of months, with a goal of completing a triathalon this summer. Today I got some pointers on things I need to improve on in my freestyle technique.

Kick

My kick needs some serious work. I am pulling almost exclusively with my arms. As Jimmy (the lifeguard and a long time swim coach) said, “You’re killing yourself.”

He had me do a drill with just a kickboard. “If your thighs are not hurting you are doing it wrong.” I focused on barely breaking the surface with my heels, pointing my toes, and keeping my legs relatively straight…only a slight bend.

Next he had me do a lap with small flippers. “You shouldn’t fell like you are fighting them.” The force you to point your toe. It felt almost too easy. We ran out of time for me to try integrating it into a regular stroke.

For weight lifting, he recommended squats.

For a sprint he recommended 3 kicks per stroke/6 per pair. For longer courses, two kicks per stroke. I think I will shoot for 2/ stroke, as I am going for 1/2 mile total.

Breathing

Improving my kick will improve my whole body position, including my breathing. It seems I am pulling my head too far out of the water, mainly because my legs are dropping. Although the opposite is true, too: pulling my head too far out of the water is causing my legs to drop. The two should be fixed together.

Arm Entry

One other swimmer at the pool that I asked for advice told me to “lead with my elbows” and then to think about entering the water “like a knife through butter”. Jimmy added that I should be reaching “long and lean.” Like a fast sailboat.

After that, the stroke should go out, and then finish in an S.

I think I need to glide more during the initial entry of the arm into the water.

Jimmy recommended a drill, either using a kickboard or a ring, and holding that out in front, and passing it from hand to hand.

Head position

I should be looking down and to the front,while the the top of my head breaks the surface.

The basic tool for neural networks is: vector times matrix equals vector. The first vector is your input pattern, the second is your output pattern. Stack these in a series and you have a deep neural network. The absolute simplest implementation I could find for this is in C++ using boost.

Here it is:

#include <boost/numeric/ublas/matrix.hpp>
#include <boost/numeric/ublas/io.hpp>

int main () {
    using namespace boost::numeric::ublas;
    matrix<double> m (3, 3);
    vector<double> v (3);
    for (unsigned i = 0; i < std::min (m.size1 (), v.size ()); ++ i) {
        for (unsigned j = 0; j < m.size2 (); ++ j)
            m (i, j) = 3 * i + j;
        v (i) = i;
    }


     vector<double> out1 = prod (m, v);
     vector<double> out2 = prod (v, m);

    std::cout << out1 << std::endl;
    std::cout << out2 << std::endl;


}

This is almost verbatim the sample code from Boost BLAS. Scroll down to the prod example. So now the question is, how does one go from that to a neural network? More to come.

Another question that comes to my mind is how would you optimize this if you have a vector based co-processor on your machine?

For my birthday last year, my family got me an accordion. Yes, I am one of those people that collect musical instruments, not for their monetary value but because I enjoy trying them out. I spent some time trying to wrap my mind and my hands around the one button chord set up on the side. I know there are masters out there that have developed a set of techniques, but for those of use from other instruments, the arraignment of notes and chords in a cycle of fifth is challenging.

It got me thinking about how else you could arrange them. This might also be somewhat inspired by guitar and bass setups, but here is what I came up with:

Start with a chromatic arrangement like this:

Assume that this is a keyboard with the bottom left nearest to you, pointing toward the ground. The Top right is pointing away and skyward. You put your pinky on A, ring finger on C, middle finger on D#, and index finger on F#. If you play those four notes, you get a diminished 7th chord.

What should be apparent from this set up is that you can play all forms of 7th chords with your pinky on the root note. To play A Major seventh, pinky on A, ring on C#, middle on E, Index on G#. If each button is a centimeter apart, Your index finger reaches at most 4 cm from lowest to highest.

To play an A augmented pinky on A. Ring on C#, Middle on F. Your index could play the root an octave up.

This same pattern would work up to F. We’d need to extend up four notes in each column to get something that would work for each chord.

Lets move our index finger to the A four notes from the bottom. From here, an Adom7 chord in second inversion could be played with pinky (P) on C#, ring(R) on E, middle(M) on G. A Maj 7moves the ring one more space up, which should still be comfortable to reach.

Lets say you have to play a I-vi-ii-V7 progression. We’ll do this in C major, both because it is well known and because it is in the middle of our keyboard. Lets say we start with the Cmaj7 in root position. Using the first letter of each finger as short hand: P-C, R-E, M-G, I-B.

(I realize now that all the images are 180 flipped from how I described it above. Just treat it like guitar tab….)

Now to Amin7, we’ll play it as I noted above: Index stays on the A. All other fingers stay in position: P-C, R-E, M-G, I-A.

Now, to move to a Dmin7, we leave the Pinky and the Index in place (P-C, I-A) Ring finger and Middle fingers move down two keys to D and F.  Seems pretty efficient P-C, R-F, M-F I-A.

That leaves the G7. For this we want to use a third inversion, with the middle finger staying on the G. The Ring finger extends up to the B. The Pinky extends up to the D. The Ring finger extends up to the F. Again, seems pretty efficient.

I thik I ma going to do some paper prototyping here.

And with gratitude I think this is a more efficient G7 position.

We have an internal system for allocating hardware to developers on a short term basis. While the software does have a web API, it is not enabled by default, nor in our deployment. Thus, we end up caching a local copy of the data about the machine. The machine names are a glom of architecture, and location. So I make a file with the name of the machine, and a symlink to the one I am currently using.

One of the pieces of information I need out of this file is the IP address for ssh access. The format is two lines in the file, like this:

BMCIP
10.76.244.85
SYSTEMIP
10.76.244.65

While the SYSTEMIP is an easy regular expression match, what I need is the line after the match, and that is a little less trivial. Here is what I have ended up with (using this post as a reference).

awk 'f{print; f=0};/SYSTEMIP/{f=1}' current_dev 

Which produces the expected output:

10.76.244.65

Linux is only one of the Operating Systems that our chips need to support. While some features need a complete OS for testing, we want to make sure that features are as tested as possible before we try to debug them through Linux.

My current code implements has to talk to the firmware. Thus we need to have a responder in the firmware. Testing this from Linux has to go through the entire Linux networking stack.

The developers of the firmware already have to deal with UEFI. UEFI is installed with the firmware. Since writing UEFI apps is basically like writing DOS apps (according to Ron Minnich) it is not a huge demand on our Firmware team to have them use UEFI as the basis for writing thei test code. Thus, they produced a simple app called RespTest.efi That checks that their code does what they expect it to do.

The normal process is that they boot the UEFI shell, switch to the file system, and copy up the Executable via tftp, Once it is on the system, they run it manually. After the first time the executable is leaded, the executable is persisted on the system. Thus, to run it a second or third time, after, say, a change to the firmware, can be automated from the operating system.

ls /boot/efi/
EFI  ResponderTest.efi  spinor.img

The above was executed from a Linux system running that has the efi partition mounted in /boot/efi. The RespTest.efi binary was copied up using tftp.

To see the set of UEFI boot options, we can use efibootmgr. These are mostly various flavors of PXE

efibootmgr
BootCurrent: 0001
Timeout: 10 seconds
BootOrder: 0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000A
Boot0000* UiApp	FvVol(5c60f367-a505-419a-859e-2a4ff6ca6fe5)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* Fedora (Samsung SSD 970 EVO Plus 250GB)	PcieRoot(0x60000)/Pci(0x1,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-25-38-5A-91-51-38-0A)/HD(1,GPT,4c1510a2-110f-492f-8c64-50127bc2e552,0x800,0x200000)/File(\EFI\fedora\shimaa64.efi) File(.????????)
Boot0002* UEFI PXEv4 (MAC:0C42A15A9B28)	PcieRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(0c42a15a9b28,1)/IPv4(0.0.0.00.0.0.0,0,0){auto_created_boot_option}
...
Boot0009* UEFI HTTPv6 (MAC:0C42A15A9B29)	PcieRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(0c42a15a9b29,1)/IPv6([::]:<->[::]:,0,0)/Uri(){auto_created_boot_option}
Boot000A* UEFI Shell	FvVol(5c60f367-a505-419a-859e-2a4ff6ca6fe5)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)

Lets say I want to boot to the UEFI shell next time. Since that has an index of 000A, I can run the following command to tell UEFI to boot to the shell once and only once. After that it will return to the default order:

efibootmgr –bootnext 000A

Here are the first few lines of output from that command;

# efibootmgr --bootnext 000A
BootNext: 000A
BootCurrent: 0001
Timeout: 10 seconds

If I want to add an addition boot option, I can use the –create option. To tell it to boot the ResponderTest,efi executable:

efibootmgr efibootmgr --create -d /dev/nvme0n1p1 --label RespTest -l RespTest.efi

The last line of output from that command is:

Boot000B* RespTest	HD(1,GPT,4c1510a2-110f-492f-8c64-50127bc2e552,0x800,0x200000)/File(RespTest.efi)efibootmgr

If I then want to make UEFI select this option upon next boot…

# efibootmgr --bootnext 000B
BootNext: 000B
BootCurrent: 0001
Timeout: 10 seconds

Note that the executable is in /boot/efi mounted on the Linux filesystem, but that will be the EF0: in UEFI. If you wish to put something further down the tree, you can, but remember that UEFI uses reverse slashes. I think it would look something like this…but I have not tried it yet:

efibootmgr efibootmgr –create -d /dev/nvme0n1p1 –label RespTest -l EFI\\RespTest.efi

Note the double slash to avoid Linux escaping….I think this is necessary.

Something I did in a commit broke my code. I have a git bisect that shows me the last good commit and the first bad one.

The code in question is a device driver for MCTP over PCC. MCTP is a network protocol used to have different components on a single system talk to each other. PCC is a shared buffer mechanism with a “doorbell” or register-that-triggers-an-interrupt. Another team needs to use this driver.

In looking through the output, a co-worker stated “it looks like you are sending the wrong buffer.” I suspected he was correct.

The buffers are pulled from ACPI information in a table specific to PCC, called the PCCT.

This is from the PCCT, which is common to both the good and bad run, and shows the machine (physical) addresses of the memory.

03 [Extended PCC Master Subspace]
               Base Address       = 00004000000D4004


04 [Extended PCC Slave Subspace]
               Base Address       = 00004000000D4404

Here is output data I get from a run of code from tip of tree.

[  644.338469] remap_pcc_comm_addresses outbox pcc_chan->shmem_base_addr = 4000000d4004 mapped to  ffff800080621004
[  644.348632] remap_pcc_comm_addresses inbox pcc_chan->shmem_base_addr = 4000000d4404 mapped to ffff80008061f404

[  644.338469] remap_pcc_comm_addresses outbox pcc_chan->shmem_base_addr = 4000000d4004 mapped to  ffff800080621004
[  644.348632] remap_pcc_comm_addresses inbox pcc_chan->shmem_base_addr = 4000000d4404 mapped to ffff80008061f404
 



 
[  828.014307] _mctp_get_buffer pcc_chan->shmem_base_addr = 1  ptr = 000000007f3ab582 
[  828.021950] mctp_pcc_tx buffer = ffff80008061f404

What I see is that I am sending data on the inbox address, not the outbox. So where did I swap it?

541392a553a2c2b7f741d89cff0ab8014a367299 is the first bad commit.

Lets take a look at the code in that commit. The diff shows a lot of changes in _mctp_get_buffer. Here’s what it looks like in non-diff form

static unsigned char * _mctp_get_buffer (struct mbox_chan * chan){
        void __iomem * pcc_comm_addr;
         
        struct pcc_mbox_chan * pchan = (struct pcc_mbox_chan *) chan;

        
        if (pchan == mctp_pcc_dev->out_chan){
                pcc_comm_addr =  mctp_pcc_dev->pcc_comm_outbox_addr;
        }else{
                pcc_comm_addr = mctp_pcc_dev->pcc_comm_inbox_addr;
        }
        pr_info("%s pcc_chan->shmem_base_addr = %llx  ptr = %p \n",
                        __func__, pchan->shmem_base_addr, pcc_comm_addr );
        return (unsigned char *  )pcc_comm_addr;
} 

How could this fail? Well, what if the cast from chan to pchan is bogus? If that happens, the if block will not match, and we will end up in the else block.

Why would I cast like that? Assume for a moment (as I did) that the definitions looked like this:

struct   pcc_mbox_chan {
 struct mbox_chan  chan;
}

Ignore anything after the member variable chan. This is a common pattern in C when you need to specialize a general case.

This is not what the code looks like. Instead, it looks like this:

struct   pcc_mbox_chan {
 struct mbox_chan*  chan;
}

The structure that chan points to was allocated outside of this code, and thus cannot be cast.

The else block always matched.

The reason the old code ran without crashing is it was getting a valid buffer, just not the right one. I was trying to send the “outbox” buffer, but instead sent the “inbox” buffer.

Here is the corrected version of the if block.

if (chan == mctp_pcc_dev->out_chan->mchan){
        pcc_comm_addr = mctp_pcc_dev->pcc_comm_outbox_addr;
}else if (chan == mctp_pcc_dev->in_chan->mchan){
        pcc_comm_addr = mctp_pcc_dev->pcc_comm_inbox_addr;
}
if (pcc_comm_addr){
        pr_info("%s buffer ptr = %px \n",__func__, pcc_comm_addr );
        return (unsigned char * )pcc_comm_addr;
}

What made this much harder to debug is the fact that the “real” system I was testing against had changed and I was not able to test my code against it for a week or so, and by then I had added additional commits on top of the original one. This shows the importance of testing early, testing often, and testing accurately.

To give a sense of how things got more complicated, here is the current version of the get_buffer function:

static unsigned char * _mctp_get_buffer (struct mbox_chan * chan){
        struct mctp_pcc_ndev * mctp_pcc_dev = NULL;
        struct list_head *ptr;
        void __iomem * pcc_comm_addr = NULL;
        list_for_each(ptr, &mctp_pcc_ndevs) {
                mctp_pcc_dev = list_entry(ptr,struct mctp_pcc_ndev, head);
        
                if (chan == mctp_pcc_dev->out_chan->mchan){
                        pcc_comm_addr =  mctp_pcc_dev->pcc_comm_outbox_addr;
                }else if (chan == mctp_pcc_dev->in_chan->mchan){
                        pcc_comm_addr = mctp_pcc_dev->pcc_comm_inbox_addr;
                }
                if (pcc_comm_addr){
                        pr_info("%s buffer ptr = %px \n",
                                __func__, pcc_comm_addr );
                        return (unsigned char *  )pcc_comm_addr;
                }
        }
        //TODO this should never happen, but if it does, it will
        //Oops.  Handle the error better
        pr_err("%s unable to match mctp_pcc_ndev", __func__);
        return 0;
}

Why is it now looking through a list? One of the primary changes I had to make was to move the driver from only supporting a single device to supporting multiple. Finding which buffer to get from struct mbox_chan is again going from general to specific.

This code is still in flight. The list lookup may well not survive the final cut, as there is possibly a better way to get the buffer from the original structure, but it is not clear cut. This works, and lets the unit tests continue to pass. This allows the team to make progress. It might be technical debt, but that is necessary part of a complicated development integration process.

With a socket created, and a message sent, the next step in our networking journey is to receive data.

Note that I used this post to get the parameter right. Without using the ctypes pointer function I got an address error.

print("calling recvfrom")
recv_sockaddr_mctp = struct_sockaddr_mctp(AF_MCTP, 0, mctp_addr , 0, 0)
len = 2000;
rxbuf = create_string_buffer(len)
sock_addr_sz = c_int(sizeof(recv_sockaddr_mctp))
MSG_TRUNC = 0x20
rc = libc.recvfrom(sock, rxbuf, len, MSG_TRUNC, byref(recv_sockaddr_mctp), pointer(sock_addr_sz));
errno = get_errno_loc()[0]

if (rc < 0 ):
    print("rc = %d errno =  %d" % (rc, errno))
    exit(rc)

print(rxbuf.value)

From previous articles you might be wondering about these values.

AF_MCTP = 45
mctp_addr = struct_mctp_addr(0x32)

However, the real magic is the structures defined using ctypesgen. I recommend reading the previous article for the details.

While the previous two article are based on the code connect example that I adjusted and posted here, the recvfrom code is an additional call. The c code it is based on looks like this:

    addrlen = sizeof(addr);

    rc = recvfrom(sd, rxbuf, len, MSG_TRUNC,
                (struct sockaddr *)&addr, &addrlen);

Once we open a socket, we probably want to send and receive data across it.Here is the system call we need to make in order to send data as I wrote about in my last post:

c = sendto(sd, buf, sizeof(buf), 0,
                (struct sockaddr *)&addr, sizeof(addr));

Note the cast of the addr parameter. This is a structure with another structure embedded in it. I tried hand-jamming the structure, but the data sent across the system call interface was not what I expected. In order to make it work ,I uses a utility called ctypesgen with the header I separated out in my last post. Runing this:

ctypesgen ./spdmcli.h

generates a lot of code, much of which I don’t need. The part I do need looks like this:

__uint8_t = c_ubyte# /usr/include/bits/types.h: 38
__uint16_t = c_ushort# /usr/include/bits/types.h: 40
__uint32_t = c_uint# /usr/include/bits/types.h: 42
uint8_t = __uint8_t# /usr/include/bits/stdint-uintn.h: 24
uint16_t = __uint16_t# /usr/include/bits/stdint-uintn.h: 25
uint32_t = __uint32_t# /usr/include/bits/stdint-uintn.h: 26

# /root/spdmcli/spdmcli.h: 5
class struct_mctp_addr(Structure):
    pass

struct_mctp_addr.__slots__ = [
    's_addr',
]
struct_mctp_addr._fields_ = [
    ('s_addr', uint8_t),
]

# /root/spdmcli/spdmcli.h: 9
class struct_sockaddr_mctp(Structure):
    pass

struct_sockaddr_mctp.__slots__ = [
    'smctp_family',
    'smctp_network',
    'smctp_addr',
    'smctp_type',
    'smctp_tag',
]
struct_sockaddr_mctp._fields_ = [
    ('smctp_family', uint16_t),
    ('smctp_network', uint32_t),
    ('smctp_addr', struct_mctp_addr),
    ('smctp_type', uint8_t),
    ('smctp_tag', uint8_t),
]

# /root/spdmcli/spdmcli.h: 3
try:
    MCTP_TAG_OWNER = 0x08
except:
    pass

mctp_addr = struct_mctp_addr# /root/spdmcli/spdmcli.h: 5

sockaddr_mctp = struct_sockaddr_mctp# /root/spdmcli/spdmcli.h: 9

With those definition, the following code makes the system call with the parameter data in the right order:

mctp_addr = struct_mctp_addr(0x32) # /root/spdmcli/spdmcli.h: 16
sockaddr_mctp = struct_sockaddr_mctp(AF_MCTP, 1, mctp_addr , 5, 0x08)

p = create_string_buffer(b"Held") 
p[0] = 0x10 
p[1] = 0x82

sz = sizeof(sockaddr_mctp)
print("calling sendto")


rc = libc.sendto(sock, p, sizeof(p), 0, byref(sockaddr_mctp), sz) 
errno = get_errno_loc()[0]
print("rc = %d errno =  %d" % (rc, errno)   )

if (rc < 0 ):
    exit(rc)

print("OK")

With this Next step is to read the response.

While the existing documentation is good, there are a couple things that have changed since it was originally written, and I had to make a couple minor adjustments to get it to work. Here’s the code to send a message. The receive part should work as originally published; what is important is the set of headers. I built and ran this on an AARCH64 platform running Fedora 38.

I split the code into a header and a .c file for reasons I will address in another article.

spdmcli.h

#include <stdint.h>

#define MCTP_TAG_OWNER 0x08

struct mctp_addr {
    uint8_t             s_addr;
};

struct sockaddr_mctp {
    uint16_t            smctp_family;
    uint32_t            smctp_network;
    struct mctp_addr    smctp_addr;
    uint8_t             smctp_type;
    uint8_t             smctp_tag;
}

I also coded this for spdm, not MCTP control messages. The difference is the value for addr.smctp_type

EDIT: I also added the recvfrom call to make this complete.

spdmcli.c

#include <linux/mctp.h>
#endif
#include <linux/if_link.h>
#include <linux/rtnetlink.h>

#include "stdio.h"
#include "spdmcli.h"

int main(void)
{
    struct sockaddr_mctp addr = { 0 };
    char buf[] = "hello, world!";
    int sd, rc;

    /* create the MCTP socket */
    sd = socket(AF_MCTP, SOCK_DGRAM, 0);
    if (sd < 0)
        err(EXIT_FAILURE, "socket() failed");

    /* populate the remote address information */
    addr.smctp_family = AF_MCTP;  /* we're using the MCTP family */
    addr.smctp_addr.s_addr = 0x32;   /* send to remote endpoint ID 32 */
    addr.smctp_type = 5;          /* encapsulated protocol type iSPDM = 5 */
    addr.smctp_tag = MCTP_TAG_OWNER; /* we own the tag, and so the kernel
                                        will allocate one for us */

    addr.smctp_network = 1 ;
    /* send the MCTP message */
    rc = sendto(sd, buf, sizeof(buf), 0,
                (struct sockaddr *)&addr, sizeof(addr));

    if (rc != sizeof(buf))
        err(EXIT_FAILURE, "sendto() failed");

   //EDIT: The following code recieves a message from the network and prints it to the console.
        if (rc < 0){
        err(EXIT_FAILURE, "recvfrom() failed");
    }

    printf("recv rc = %d\n",rc);        

    for (int i =0; i< rc; i++){
        printf("%x", rxbuf[i]); 

    }



    return EXIT_SUCCESS;
}

While the Python socket API is mature, it does not yet support MCTP. I thought I would thus try to make a call from python into native code. The first step is to create a socket. Here is my code to do that.

Note that this is not the entire C code needed to make network call, just the very first step.I did include the code to read errno if the call fails.

#!/usr/bin/python3
from ctypes import *
libc = CDLL("/lib64/libc.so.6")
AF_MCTP = 45
SOCK_DGRAM  = 2
rc = libc.socket (AF_MCTP, SOCK_DGRAM, 0)
#print("rc = %d " % rc)
get_errno_loc = libc.__errno_location
get_errno_loc.restype = POINTER(c_int)
errno = get_errno_loc()[0]
print("rc = %d errno =  %d" % (rc, errno)   )
print("OK")

Running this code on my machine shows success

# ./spdm.py 
rc = 3 errno =  0
OK

There is a pretty common pattern in C coding that uses a Key, often an integer, to select a function to call. However, code may not start there, and there is a pretty common set of interim steps you might go to get there.

The growth is often like this:

If the logic for each case is similar and/or small, the switch statement may well be the final state. However, if the logic inside each of the case blocks gets too complex, then an common step that can be injected as soon as step 2 is to execute the refactoring of extract method except that we will call it extract function, we don’t have methods in C.

To give a concrete example, I am working with MCTP, a protocol where a message is delivered in a buffer. We take the payload of the buffer and select a specific offset to see the “type” of the message, and based on that type code, we need to perform a different operation. The Type code is the first byte of the payload, and it states what the higher level protocol is. In its current state of evolution, that can be one of three things: MCTP control messages, PLDM, or SPDM. Thus, the code evolution looks like this.

Handle the first case in the main method.

static void mctpserv_deliver(struct mbox_client * cl, void * data){
        struct mctp_pcc_packet * mpp;
        mpp  = (struct mctp_pcc_packet *)data;
        //do Simple MCTP stuff
        //send back status
}

Use an If statement to handle the error case

static void mctpserv_deliver(struct mbox_client * cl, void * data){
        struct port_handler mapping;
        struct mctp_pcc_packet * mpp;
        mpp  = (struct mctp_pcc_packet *)data;
        pr_info("%s signature = %x \n", __func__,
        (u32)mpp->pcc_header.signature);
        if (mpp->payload[0]  == MCTP_CONTROL){
                  //do Simple MCTP stuff
                  //send back success status  
        }else{
                  //send back error code
        }
 
}

Use an extended if/else clause to handle the second case

static void mctpserv_deliver(struct mbox_client * cl, void * data){
struct port_handler mapping;
struct mctp_pcc_packet * mpp;
mpp = (struct mctp_pcc_packet *)data;
pr_info(“%s signature = %x \n”, __func__,
(u32)mpp->pcc_header.signature);
if (mpp->payload[0] == MCTP_CONTROL_CODE){
//do Simple MCTP stuff
//send back success status
}else if((mpp->payload[0] == SPDM_CODE)) {
//do SPDM stuff
}else{
//send back error code
}
}

Use a switch statement to handle more than one case

static void mctpserv_deliver(struct mbox_client * cl, void * data){
        struct port_handler mapping;
        struct mctp_pcc_packet * mpp;
        mpp  = (struct mctp_pcc_packet *)data;
        pr_info("%s signature = %x \n", __func__,
        (u32)mpp->pcc_header.signature);
        switch (mpp->payload[0]){
             case MCTP_CONTROL_CODE: {
                  //do Simple MCTP stuff
                  //send back success status
             };  
             break;
             case SPDM_CODE: {
                  //do SPDM stuff        
              };
              break;
              default: 
                  //send back error code
        }
}

Extract function

void handle_mctp(struct mctp_pcc_packet * mpp){
//...
}

void handle_spdm(struct mctp_pcc_packet * mpp){
//...
}

void handle_pldm(struct mctp_pcc_packet * mpp){
//...
}


static void mctpserv_deliver(struct mbox_client * cl, void * data){
        struct port_handler mapping;
        struct mctp_pcc_packet * mpp;
        mpp  = (struct mctp_pcc_packet *)data;
        pr_info("%s signature = %x \n", __func__,
        (u32)mpp->pcc_header.signature);
        switch (mpp->payload[0]){
             case MCTP_CONTROL_CODE:  
               handle_mctp(mpp);
               break;
             case SPDM_CODE:
               handle_spdm(mpp);
               break;
            case PLDM_CODE:
               handle_pldm(mpp);
               break;
              default: 
                  //send back error code
        }
}

Note that when we extract the function, we use the same function signature. This is deliberate so we can use a function pointer in the next step without having to change the parameters passed to it. In order to define the function signature and the key used to select it, we put both inside a structure. for added type safety, use an enum to limit the values to the acceptable range.

enum MCTP_MSG_TYPE {
        MCTP_MSG_TYPE_CTRL = 0,
        MCTP_MSG_TYPE_PLDM = 1,
        MCTP_MSG_TYPE_SPDM = 5,
};
struct port_handler {
        enum MCTP_MSG_TYPE code;
        void (* handler )(struct mctp_pcc_packet * mpp);
};
struct port_handler handlers[]  = {

        {MCTP_MSG_TYPE_CTRL, handle_mctp },
        {MCTP_MSG_TYPE_PLDM, handle_pldm},
        {MCTP_MSG_TYPE_SPDM, handle_spdm}
};

One thing this example shows is why not to use the index of the array as the key; a sparse and non contiguous integer sequence means you would have a lot of wholes in that array. We need 0, 4, and 5 only.

Since our sequence is short, we can iterate through it using a linear search:

        for (int i = 0;  i <  sizeof(handlers)  / sizeof(struct port_handler )  ; i++){
                if (handlers[i].code == mpp-> payload[0]){
                        handlers[i].handler(mpp);
                        break;
                }
        }

For longer lists, it might be necessary to use a binary search. I am not going to show an example of that here, as I have not written it, and my example would certainly be buggy.

Handle a missing function

This example does not show how to handle a missing function. There are two ways I have done this: either set a boolean when the handler is called, or have the default case be an error handler. Here is the default case as a function.

void default_handler(struct mctp_pcc_packet * mpp){
        pr_info("%s unsupported protocol \n", __func__);
}
struct port_handler default_mapping = {
        MCTP_MSG_TYPE_MAX,
        default_handler,
};
static void mctpserv_deliver(struct mbox_client * cl, void * data){
        struct port_handler * mapping = &default_mapping;
        struct mctp_pcc_packet * mpp;

        pr_info("%s\n", __func__);

        mpp  = (struct mctp_pcc_packet *)data;
        pr_info("%s signature = %x \n", __func__,
        (u32)mpp->pcc_header.signature);
        pr_info("%s type code = %d \n", __func__, mpp-> payload[0] );


        for (int i = 0;  i <  sizeof(handlers)  / sizeof(struct port_handler )  ; i++){
                if (handlers[i].code == mpp-> payload[0]){
                        mapping = &handlers[i];
                        break;
                }
        }
        mapping->handler(mpp);
}

The future Ampere System-on-a-chip that is the basis for my work has a couple of processors that are for providing system wide services, not for end users to run things on directly. One of these is called the Management Processor, or Mpro. We have to talk to one of these services from the operating system using the Management Control Transport Protocol, or MCTP, over a Platform Communication Channel (PCC).

I just sent ran a user-land program that sends a message via a socket into the operating system, via PCC, to the MPro, and got a response back.

It took a while for it to sink in. And then I raised my hands in the air and proclaimed in my Best Dr. Frankenstein voice,

“It’s Alive!”

There were so many lessons I had to learn to get this this point. Here are a short list:

1. How to implement a Linux Kernel network device driver
2. How to register with the Linux Kernel to get an Advanced Configuration and Power Interface (ACPI) based device from data in the Differentiated System Description Table (DSDT).
3. How to convert physical memory offsets to virtual
4. How to read and write data from device memory
5. How to lookup data in the ACPI tables (a work around for a mechanism not yet written)
6. How to work with our firmware team to get their view of the debug data
7. How to build a firmware image and deploy it to one of our lab machines
8. How to build and run a unit test against mock objects inside the Linux Kernel
9. How to read and implement the MCTP and related level protocol specs.

I work with really smart people. I would probably have been able to do this all myself…in another 15 years. The amount of time saved by working with the members of my team is probably longer than the life of the product I am working on.

Someone once said, “If you are the smartest person in the room, you are probably in the wrong room.” I suspect that I am absolutely in the right room for me.

The code is by no means done. In some ways, you might say that the real work begins now: getting it production ready, However I have always used the principal of “build from success” when working in software. First get something working, then make incremental improvements while keeping it working. This is my first real success and it feels good.

If you don’t want someone to be able to delete the database, do not give them authority to delete the database.

Azure recently announced that they were implementing Negative access rules. A rule of this nature is such that you/ can say “User U cannot perform action A on resource R.”

This is an anti-pattern. It is a panic response to breaches caused by principals (users or agents ) that have far more power than they should have to perform operations than they need. What is the alternative?

Create a agents with minimal access to get done exactly what they need an no more.

Lets break this on down. I am going to use a web application to explain what this should look like. This application takes orders on line and send them to a back-end processing system to fulfill. As such, there are roughly four agents for different parts of the overall system.

• The Web server
• The Web servers Database
• The messaging broker
• The back-end fulfillment system.

Ulysses is the end user. Ulysses wants to buy a thing. Ulysses opens his browser and clicks through the pages. We’ll assume that the web-server is not running as root on the remote server, as that would be bad and we’ve all grown beyond that point. It is running as, say, the STORE user. This user can render pages to send back to Ulysses, and can query the database to get categories, products, SKUs, and few more things. For write access, this user can write to the order database, and can write order messages to the message queuing system. The STORE user can also read from the order update queue to change the order management database, although this is likely done by a different agent in practice.

OK, I think you can see how this same kind of pattern would be implemented by the order fulfillment system. Lets go back to our cloud provider and discuss a little further.

Odette is our owner. She is the one that pays the Cloud provider to set up her store, to sell things. Odette has contracted to Carlita to modify her web application to customize it for her needs. Dabney is her database administrator. In addition, she uses a service from her cloud provider to set up the application in the first place.

Which of these user should have authority to delete the database? None, obviously. But how does the database get created in the first place? Automated processes set up by the cloud provider. Modifications performed on the database should happen via git, with code review required by at least one other person to accept a merge request. This merge request then goes into a staging branch, staged, and again requires manual approval to be pushed to the live server.

THE actual pushing of changes from one stage to another are performed by agents with only the ability to perform the operations they need. Create and drop tables are handled by the database management agent. And yes, this one, if compromised, could destroy the database. However, another agent performs backups prior to any operations that alter the tables. A deleted table can be created by rolling aback to a previous version, acknowledging that all data that came in post corruption will be lost.

It ain’t easy. It ain’t simple, and it ain’t free. But it is all necessary.

So why are negative rules such a big deal? Surely adding an explicit deny rule on top of all the explicit allow rules provides an added layer of security? There are a few problems with this approach, First, it only actively denies actions you have actually foreseen as problematic. For instance, it would not catch that the Store user could delete the order export queue, or that the order update agent could alter the order update table to a format that was broken. You want to deny everything by default. Then, to test to see if a user can perform a specific action, you can query the role assignments to see if a user has the dangerous permission. If they do, you probably need to create a new, more fine grained role, that has only the permissions required.

A good example of a system that works this way is SELinux. IN an SELinux hardened system, everything is forbidden until explicitly allowed. The critical tool that makes SELinux workable is permissive mode. When you run a NON_PRODUCTION system in permissive mode, you see what would have been denied by the existing set of rules, and you make new rules that would allow those actions. Then you go back into enforcing mode and run, and make sure you can get through the required workflow with all required actions allowed and all forbidden actions denied. This workflow would replace the explicit Deny rules above.

Deny rules add an additional complication. Now you need to explicitly order your rules. First, you have the default Rule of Deny All. Then you have the explicit allow rules, and finally the deny rules. But what happens if someone comes by later and wants an exception to the deny rule. Maybe this account should not be able to delete production databases, but staging ones are just fine? If your blanket deny stops them from deleting all databases, you might be stuck.

And that is what the Deny rules should be: unit tests that show an action is denied. Not an actual rule itself. Because all actions should be denied by default, and if you can accidentally assign one action, you probably accidentally allowed a bunch of other actions as well. So, do not make Deny actions part of your enforcement rule set. Make them part of your explicit tests after the fact.

Deny rules have been a part of many access control systems over the years. This is not the first time this discussion has come up, nor will it be the last. This is one of those “must have” features that keep popping up. Its an anti pattern and should be avoided. The rational developer can only say no so many times before someone does an end run around them and someone puts negative rules in the system.

One of my current engineering efforts is in support of the concept of hardware attestation. What is that? Here’s my attempt at a summary:

Attestation is a way of showing something is true. In this case, it is proving that the software you are talking to is from a trusted vendor. In my case, that software is the Firmware for the System-on-A-Chip. While there are many levels of Attestation, they all resolve down to the lowest level software. It doesn’t matter if the Operating System is good if the Firmware is bad, there is the potential for a violation. This is no guarantee that there is not a violation. Attestation is necessary-but-not-sufficient.

How do we attest? Well, we have to talk to a server. There are two different types of servers I have been exposed to in my current effort: BMC and Operating System. They do roughly the same thing. Some external program makes a connection to either the BMC or the OS, and and makes an attestation request. “Show me the firmware you are running and a cryptographic signature for it.” Basically, I am back in the world of X509 certificates. Although attestation does not need to be done with X509, it needs all the stuff from X509 and so it is easiest to explain in terms of X509.

The building blocks:

The main building block of most computer security is the concept of asymmetric cryptography. This means that you have a computer algorithm that can encry0pt and decrypt messages. The keys used by this algorithm are related to each other but different from each other. One is going to be labeled the public key. The other is the private key. Which is which is arbitrary, but once you decide, you need to treat them consistently. If you use the public key to encrypt a message, you can only use the private key to decrypt it, But if you use the private key to encrypt a message, you can only use the public key to decrypt it. There are many ways that these are used, but I am going to focus on one of the; signatures. However, before we can explain signatures, we need one more concept: hashes.

A cryptographic hash algorithms takes a long document and produces a short string that represents that document. What is special about cryptographic hashes is that a very small change in the docuemtn produces a different hash. If you change a period to a comma, you get a different hash.

I can take the text above, starting with the phrase “Attestation is a way…” up to the phrase ” you get a Different Hash” and put and put it in a text document. I will also add an addtional line and put a period on it. I can use the program sha256sum to product the hash like this:

sha256sum attest.txt 
396f4edd98ddf09d44a2842e9495db25b7e7b5a0b5a747a4fd29b8e36c5abfab  attest.txt

However, if I change the final character from a period to a comma,

diff attest.txt attest2.txt 
10c10
< .
---
> ,

I get this:

sha256sum attest2.txt
9282e247e2b21a49731b2a631ed2fc2afb3184fc5908ee893111ad6cfb142f9e attest2.txt

A cryptographic signature for a document takes a sha256 hash and encrypts it with a private key. Now If someone wants to confirm a document, they 1. Use the public key to decrypt the hash and 2, hash the document themselves to see that the hashes match.

Attestation is based on requesting a signature for the firmware running on a system. The public key used to validate that signature comes from a cryptographic certificate (that is where X509 comes in) so you can establish a chain of trust. X509 is a fairly huge area, but the only part of it necessary to understand here is that one public key can be used to sign a document containing another public key…and so on in chains. This is how your computer set up an https session to read this blog, and also used in many of the algorithms used to identify yourself on the internet.

OK, that is a pretty big nutshell. Maybe a coconut. But I think that is a succinct as I can make it. This skips a lot of the context.

This post is a novelty post merely to say hello to a friend of mine that gave me a shout out else-where.

Hi Stephen.

We have a few different ways we build a Kernel for internal consumption. One is very minimal, to be used when space is at a premium. Another is very closely based on the Fedora configuration to be used as a replacement Kernel for a Fedora install. While we do provide our team with RPMs to install, development often requires building from source. I’ve done this (and forgotten steps) enough times to want to have a quick reference. The following should work on a recent Fedora install.

yum groupinstall "Development Tools"
yum install dwarves openssl grubby
git clone git@$(GITSERVER):$(LINUX_REPO).git
cd linux
cp $( KERNEL_CONFIG_SOURCE) ./.config
make olddefconfig
make -j `nproc`  && make -j `nproc` modules && make -j `nproc` modules_install && make install
grubby --set-default-index=0
reboot