Overview

This is a personal project, the target is to help me generate and maintain synthetic songbooks, that I can use when playing with my amateur pop band.

Standard music notation looks something like this :

In pop music, this can be quite hard to follow, you may have 96 times the same chord, which ends up in a music sheet of 5 pages with the same thing. When you play, you cannot turn the page, so you want a synthetic view, with easy to visualize patterns ( ie verse, pre-chorus, chorus, bridge, ...), with bar number, and annotations.

The goal is also for the band, not just for the guitar player, to have a short way to communicate about the song articulations, the breaks, and so on. Of course this does not replace the need to learn the song by other means ( real score, learn by ear, or any other way), and it does not replace having to practice your instrument with a full score support, or by ear.

The targeted goal is to have pdf files that you can either print or see on a tablet, such as :

one A4 page

easy to follow grid notation, one cell per bar

a music sheet snippet, for the bridge

lyrics are in sync with the score

This document has the following sections, that hopefully help to understand the :

• what do we want ( the specifications )
• the detailed design ( how is that written, how to maintain it )
• the user manual ( how to use it )
• the installation manual ( how to install it )

other stuff

Other than providing songbooks, this project is also for me a fun stuff :

• a fun way to practice rust
• learn leptos to have a web application, with client side only : no dev on server side
• play with rust mdbook
• deploy doc with github gh-pages

inputs and outputs requirements

export requirements

dependencies requirements

interface requirements : two user modes

dependencies requirements

rendering requirements :

what to find in the pdf output

wav files

whenever a piece of music sheet is present in the song, it will be possible to generate a wav output for that piece.

input

all inputs are readable text files. It will therefore be possible to put them in a git repo and manage the life of these files

two edit modes

there will be two modes : the local and the web mode

output export

provided correct configuration, it will be possible to export the pdf outputs to a google drive

web mode

in web mode, you edit the remote files via a web interface, you trigger the generate of the pdf file by clicking a button on the web interface

Operating System

the software will run on standard ubuntu. There is no requirement that it runs on windows.

Delivery Directory

All output files will be available in one directory.

google drive

A command will allow to upload all pdf files to a google drive location, if correct credentials are provided

pdf output

the tool output will be pdf files

we want to be able to print a songbook, and also to download it on a tablet, and be available without internet connection

wav files

whenever a piece of music sheet is present in the song, it will be possible to generate a wav output for that piece.

input

all inputs are readable text files. It will therefore be possible to put them in a git repo and manage the life of these files

two edit modes

there will be two modes : the local and the web mode

local mode

in local mode, you have access to a machine where you cloned the repo that has the songs, and you also cloned the repo that has the code of the tool. (currently they are in the same repo)

it requires that you have some computer science knowledge, as you will have to install a few things, edit the data files and run the tool

OS

the software will run on ubuntu. There is no requirement that it runs on windows.

output export

provided correct configuration, it will be possible to export the pdf outputs to a google drive

web mode

in web mode, you edit the remote files via a web interface, you trigger the generate of the pdf file by clicking a button on the web interface

requirements on rendering the pdf output

tempo

the tempo will be shown, in BPM. There is one tempo for the whole song, if the real song has different tempos, this will not be rendered

sections

what we call a section is what you naturally see in a song : verse, chorus, pre-chorus, bridge, intro, ...

1. it will be possible to define new sections
2. sections will be associated with colors
3. each section has associated lyrics

time signature

time signature will be shown in the output. There will be one time signature for the whole song, except if a bar is marked with another time signature

music sheet snippet

it will be possible to have music sheet snippets in the rendered output pdf file. This way we can show the solos, riffs, gimmicks....

coherence

the sections in the grid rendering will be in coherence with the sections in the lyrics. This mean that if we have, for instance, a song with verse-1, verse-2, pre-chorus-1, chorus-1, we will find these sections in both rendering

grid

the chords will be shown as a grid, with 4 columns, one grid per section

chords symbol

the chords will be shown with the A...G symbols ( not Do Ré Mi...).

• the minus symbol will show the minor
• 7 for 7 chords
• flat and sharp
• m7
• sus chords
• 5 chords
• 7M will be show as a triangle
• diminished chords as small circle

the size of a cell for a chord will not change, this means that a C chord rendering will take as much width as a C#m7, so all cells have same size

chords per bar

there will be one or two chords per bar, for visibility reason. if a bar has more than 2 chords, this will be in annotation

line repeat

it will be possible to specify a line repeat. In this case, we will have a symbol that shows the line has to be played this number of repeats.

the line repeat value will be either 2, 3 or 4.

the bar count numbering and time will have to be coherent with the time repeat

text rendering

text rendering will have these features :

1. foreground color
2. background color
3. internal and external links
4. italic, bold, and other fonts
5. foot notes

local mode

in local mode, you have access to a machine where you cloned the repo that has the songs, and you also cloned the repo that has the code of the tool. (currently they are in the same repo)

it requires that you have some computer science knowledge, as you will have to install a few things, edit the data files and run the tool

look at installation procedure for configuring the machine.

web mode

Generate the outputs

Installation

Overview

There are many ways to achive that. We could use a word processing tool, such as Microsoft Word, latex, lilypond, musescore or even sketch with a pencil on a paper and take a picture.

but none of these respect our requirements, so this is our design :

text : latex

textual information, such as lyrics, comments, annotations,... are captured in latex files. This way we have the formating flexibility we want.

We could have done everything with latex, except from the music sheet, by writing a new latex class. But choose to limit latex, for the user, to the capture of text.

music sheet : lilypond

Music sheet snippets are captured in lilypond files. We will be able to output the partitions of solos, bridges, chord diagrams, ... and insert them in the pdf output file. This will also allow to export wav files, because lilypond has a midi export.

there are other tools to do that, but lilypond is way superior, at the cost of some complexity. We think it is worth the cost.

master file

A master file will contain the description of the sections, the tempo, and all other informations. This master file will have a fixed name : song.json.

master tex file

A master tex file will contain the latex information. This file is named body.tex. This is not the main.tex file, that contains the document class, this file is generated.

book

A book is only a list of songs, with a book title. It is defined in a json file

file organization

Each song is contained in one directory. It will have :

1. song.json : the master file
2. body.tex : the master tex file
3. lyrics directory : contains the lyrics of the song, one tex file per section
4. additional regular tex files. You just import them as you would do
5. additional lilypond files (.ly extension).

The information to name the pdf file are inside the song.json master file, but, for sake of readability, we will use :

songdir root
   +--- artist 1
           +--- song 1
                  +--- song.json
                  +--- body.tex
                  +--- ... other tex files if any...
                  +--- ... lilypond files if any ....
                  +--- lyrics
                         +--- lyrics tex fils
           +--- song i
           +--- song N
                  .... subtree of song 2
   +--- artist i
   +--- artist N
           +--- a song
                  ... files
           +--- another song
                  ... files



bookdir root
    +--- book1.json
    +--- ...
    +--- bookN.json

file tree

all songs will be under a root directory : the songdir directory.

all books will be under a root directory : the bookdir directory. Each json file in this directory defines a book.

omake

the omake build tool omake, not to be confused with IBM omake tool, is a make tool that looks like make, but is actually much superior :

1. it does not have the dependency issue, see recursive make considered harmful, which prevents from efficiently use make in big projects
2. dependencies are built on checksum of files, not on dates, so it does not matter if you regenerate files before running a build
3. a lot of other stuffs... not discussed here

we want a build system because building a pdf or a wav file can be long, and we are going to iterate a lot when capturing a new song, especially with the lilypond files.

the tools

we have three tools, written in rust, in this project :

1. songbook : this is a code generator
2. for web only : songbook-server see server
3. for web only : songbook-client see client

workflow

this is :

1. run songbook <songdir> <bookdir> <builddir>
2. ( cd builddir && omake )

the songbook tool will

1. read all the songs from the songdir tree
2. read all the books from the bookdir tree,
3. generate the OMakeroot and OMakefile in the builddir tree
4. genere some ressource files at the root of the builddir tree

you can then run omake.

code generation

see code generation section

lilypond files

input

master json

We use serde_json to automate reading from json file to a rust struct, this is documented here :

rust doc for master file

available section types

The list of valid sections is built at compile time, it is read with serde json from this file : others/texfiles/sections.json

To modify the available section types, just change the file and rebuild.

lilypond

book

rust doc for master file

code generation

the songbook tool is a code generator. Syntax is :

songbook <songdir> <docdir> <builddir>

it takes inputs :

• songdir : the root dir of songs
• bookdir : the root dir of books

and will generate code in builddir

when and why do we need to generate ?

we could have used omake scanner facility to replace some features, but it would have been tricky.

We need to regenerate whenever a master song.json has changed, or a song was added or removed, so that the makefiles are up-to-date. The action of generating is a few ms so in our workflow there is no point in trying to optimize that, compared to the generation of pdfs that is 100 times longer.

So the answer is : generate each time you build, it is harmless.

thanks omake

Omake uses digest of files to know if a target needs to be rebuild. So when you regenerate you will have source files that are more recent than the output pdf file, but omake will not rebuild the target.

tree structure

all files are generated in the builddir tree. No file is generated in the source directory : code generate leaves the sources unchanged.

<builddir>
    OMakeroot
    OMakefile
    +--- delivery
    |        +--- .... all the delivery files : pdfs and wav
    +--- songs
    |        +--- artist 1
    |        |        +--- song 1
    |        |        |       +--- OMakefile
    |        |        |            --> ... generated files : main.tex, ...
    |        |        |            --> ... all vmount files...
    |        |        |            --> ... files generated by the build
    |        |        +--- song N
    |        |               ....
    |        +--- artist N
    |                 +--- songs....
    +--- books
             +--- book 1
             |        +--- OMakefile
             |              --> ... generated files : main.tex
             |              --> ... files generated by the build
             +--- book N

vmount files

omake has the vmount feature : any file used as a dependency in an OMakefile is copied from the source directory to the build directory, and this is done each time you run the build. So it looks as if the srcdir is mounted in the builddir.

For that to work, source files have to be declared in the OMakefile, and this is done because they are declared in the master json file. Other mounted files added automatically are :

1. body.tex ( we have to have it, so it does not need to be declared in the master json file )
2. the tex lyrics ( one per section ) : this list is generated

the code generator will generate :

files for running omake :

1. OMakeroot
2. root OMakefile
3. song OMakefile
4. book OMakefile

latex files

1. main.tex
2. preamble.tex
3. sections.tex
4. struct.tex
5. chords.tex

OMakeroot

omake requires to have a OMakeroot file, (see omake doc) the code is a handle bar template, others/makefiles/omakeroot

root OMakefile

this is the root makefile. Here we define some phony targets, the code is a handle bar template, others/makefiles/root_omakefile

song OMakefile

this is the makefile you find in every song directory, the code is a handle bar template, others/makefiles/omakefile

book OMakefile

this is the makefile you find in every book directory, the code is a handle bar template, others/makefiles/omakefile_book

main.tex

this is the main.tex file, where the latex document is declared. This is file is the input of the lualatex command. the code is a handle bar template, others/texfiles/main.tex

We have here the input of other generated files, and the input of the user's code : body.tex

preamble.tex

this is where we define the latex preamble, that is the require of all latex packages we need. others/texfiles/preamble.tex

sections.tex

This is where, for each section type xxx, we define the latex macro \songbooksectionxxx and \\songbookcolor that can be reused in latex code others/texfiles/sections.tex

data.tex

This is where, using the data from master file song.json, latex macros are defined : others/texfiles/data.tex

1. \songtitle
2. \songauthor
3. \songtempo
4. \makesongtitle : will format the string
5. \songlastupdate
6. \songbooksongstruct : this one has the sections of the song, it is the one you want to put in your body.tex file

chords.tex

see the fonts section

Our design is to define our fonts to display chords, so there will be one glyph per chord, for instance D flat 7 will be one glyph.

In this file we define marcros that map chords to glyphs : others/texfiles/chords.tex. For instance, we have for D flat 7 :

\newcommand\chordDfsept{{\songbookfontflat\fontsize{18pt}{18pt}\selectfont R ~ }}

fonts

we have our own fonts to display chords, each chord is a glyph

birdfont

we use the birdfont tool to edit the fonts. This tool has a commercial and a SIL open font license, that you can use for free.

capture fonts

use birdfont to load and modify a font.

all glyphs cannot be in one font, so we split them in three fonts :

1. software/fonts/songbook.birdfont : the fonts for chords
2. sofware/fonts/songbook_flat.birdfont : the fonts for flat chords
3. software/fonts/songbook_sharp.birdfont : the fonts for chords

For each font, we have, from letter A :

• major chords (A => G)
• minor chords (H => N)
• 7 chords ( O => U )
• minor 7 chords ( V => b )
• 7 major chords ( c => i )
• diminished chords ( j => p )
• suspended chords ( q => w )

export fonts

use (Ctrl-e) menu (export fonts), it will export the fonts to ttf files.

make them available

the pdf generator lualatex will look for additional fonts in $HOME/.fonts, so copy the ttf files there. Other tools like LibreOffice may look for the fonts in other locations, read the doc for these tools

code generator

the mapping between chords and fonts is done in software/others/texfiles/chords.tex. if you change the organization of chords/glyphs/fonts, don't forget to update this file

For instance, we have for D flat 7 :

\newcommand\chordDfsept{{\songbookfontflat\fontsize{18pt}{18pt}\selectfont R ~ }}

This is the songbook_flat font, so using letter R in our example we will have the wanted glyph :

@todo : finish writing the fonts

We use the same mechanism for the pause and repeat glyphs.

web

We will have web pages delivered, but with no web server.

no web server ?

Well, no developement of a web server, we will only have nginx running, and using a nice feature of nginx : the ability to trigger sh scripts. This will allow us to have post actions, the two main ones are :

1. trigger a build
2. write a source file

this feature is configured in the /etc/nginx/sites-enabled/songbook.conf config file of nginx, in which we have :

   location ~ (\.sh)$ {
       types  { application/json sh; }
       gzip off;
       root /var/www/songbook/scripts;
       autoindex on;
       fastcgi_pass unix:/var/run/fcgiwrap.socket;
       include /etc/nginx/fastcgi_params;
       fastcgi_param DOCUMENT_ROOT /var/www/$server_name/scripts ;
       fastcgi_param SCRIPT_FILENAME /var/www/$server_name$fastcgi_script_name;
       fastcgi_param QUERY_STRING    $query_string;
   }

web rendering

We will do that with rust-leptos create, that allows to write code in rust, that is translated to javascript and runs in the browser : there is no server side code.

why ?

just because in our workflow, we don't really need a web server. We just need to trigger omake to build the outputs, and

songbook user and www-data

Let's assume the tool is installed under songbook user. the web server runs under www-data user, has no home directory, and has no right to change the songbook user files. The only thing www-data can do is run the request.sh script in the /var/www/songbook/scripts directory

client web browser
  |
  +-->  action --> http transfer --> nginx server
                                            |
request.sh  <-------------------------------|
    |
    +---> push request ( songbook-client running under www-data )
                 |
               ZMQ messaging service
                 |
                 +---> songbook-server running under songbook user
                             polls the request
                                    +---> action taken

ZeroMQ

this is free, easy to use and performant messaging library. There is a rust crate for that, and no server to configure.

server

client

web frontend

rust leptos

ace.js

setup

mdbook / cargo doc

mdbook

we use mdbook to generate the document you are reading

cargo doc

we use cargo doc to generate the documentation of the rust code source

deploy to github.io

we use github actions to deploy this documentation

mdbook / cargo doc

it does not seem that these tools were designed to work together, so, in order to have cargo doc generated files inside the mdbook doc, you will see this file in .github/workflows/main.yml action :

  - name: Build Book
    run: |
      cd doc/srs
      mdbook build


  - name: Build cargo doc
    run: |
      ( cd software && cargo doc )
      src=software/target/doc
      target=doc/srs/book
      mkdir -p $target
      cp -R $src/static.files $target/.
      cp -R $src/songbook $target/.
      cp -R software/others $target/.
      find  $target/others -type f  | while read f ; do echo $f ; cp $f $f.txt ; done

basically we :

1. copy the rust doc generated html files in the mdbook generated tree.
2. copy other files we want to see
3. copy some files with the extension .ext, so that they can be rendered