This document was last modified

Nuggets: System File Management

In order to keep namespace control files intact and homogeneous across hosts, we have developed a mechanism currently known as "nuggets". This name may change in the future :) . Each system file that would normally be hand-edited on all hosts can now be managed via the mount located at /usr/nikola/nugget. Either triggered manually or automatically, each host will determine if a newer version exists, and if so, compile the nugget into the desired system file(s). This allows us to quickly distribute system namespace changes to multiple hosts with a minimum amount of duplicated effort.

Typographical conventions

Normal prose is in this times font. Computer files are underlined, fixed width; while the text that is output is just fixed width. Where a substitution should be made, the substitution text is italicized, fixed width. Text in square brackets [] is optional.

The syntax diagrams are really meant for tool writers, and they are modified Bachus-Nauer format (BNF). Keywords and required punctuation are bolded. Text enclosed in {} is repeated between the lower and upper bounds, given as subscript and superscript.

Some major facets of nuggets worth discussing:

1. Prerequisite Host-level Variables
2. Nugget Organization
3. The Nugget Control File
4. Configuration Parameters

Prerequisite Host-level Variables

A nugget should not begin to compile if the following variables are not either declared or guessed for the host:

• nugget: name of the nugget (usually coinciding with the rendered system file or a superset thereof)
• arch: This combines a host's OS, CPU, distribuation and version. Acceptable values are defined in nikola.conf.
• bunch: Defines grouping of machines, either by cluster, lab, etc. Acceptable values are defined in nikola.conf.
• duty: Specifies services the host provides. Acceptable values are defined in nikola.conf.

These variables will be used throughout this document.

Nugget Organization

To facilitate the fact that Nikola will run on multiple system architectures, nuggets will be broken up into a format much like this:

The nugget control file

The nugget control file is a single file that controls which of the possible variations get compiled into the rendered system file. The file is organized into brace-separated stanzas, like this:

nugget chicken {
        masterfile=chicken.master       # A test comment.
        postproc=chicken.postproc
        generatedby=justin
        bunch ssli {
                duty webserver {
                        postfile=chicken.webserver.postfile
                        postproc=chicken.webserver.postproc
                        arch rh71-ia32 {
                                postproc+chicken.rh71-ia32.postproc
                        }
                        arch sol8-sparc {
                                postfile=chicken.sol8-sparc.postfile
                                postproc+chicken.rh71-ia32.postproc
                        }
                }
                duty fileserver {
                        postfile=chicken.fileserver.postfile
                        postproc=chicken.fileserver.postproc
                        arch rh71-ia32 {
                                postproc+chicken.rh71-ia32.postproc.fileserver
                        }
                        arch sol8-sparc {
                                postfile=chicken.sol8-sparc.postfile.fileserver
                                postproc+chicken.rh71-ia32.postproc     # test comment
                        }
                }       # end of duty fileserver
        }
        bunch vlsi {
                duty webserver {
                        postfile=chicken.webserver.postfile
                        postproc=chicken.webserver.postproc
                        arch rh71-ia32 {
                                postproc+chicken.rh71-ia32.postproc
                        }
                        arch sol8-sparc {
                                postfile=chicken.sol8-sparc.postfile
                                postproc+chicken.rh71-ia32.postproc
                        }
                }
                duty fileserver {
                        postfile=chicken.vlsi.testserver.postfile
                        postproc=chicken.vlsi.testserver.postproc
                        arch rh71-ia32 {
                                postproc+chicken.rh71-ia32.postproc.fileserver
                        }
                        arch sol8-sparc {
                                postfile=chicken.sol8-sparc.postfile.fileserver
                                postproc+chicken.rh71-ia32.postproc     # test comment
                        }
                }       # end of duty fileserver
        }
        bunch rcs {
                duty webserver {
                        postfile=chicken.webserver.postfile
                        postproc=chicken.webserver.postproc
                        arch rh71-ia32 {
                                postproc+chicken.rh71-ia32.postproc
                        }
                        arch sol8-sparc {
                                postfile=chicken.sol8-sparc.postfile
                                postproc+chicken.rh71-ia32.postproc
                        }
                }
                duty fileserver {
                        postfile=chicken.rcs.testserver.postfile
                        postproc=chicken.rcs.testserver.postproc
                        arch rh71-ia32 {
                                postproc+chicken.rh71-ia32.postproc.fileserver
                        }
                        arch sol8-sparc {
                                postfile=chicken.sol8-sparc.postfile.fileserver
                                postproc+chicken.rh71-ia32.postproc     # test comment
                        }
                }       # end of duty fileserver
        }
        bunch rcs.gnl {
                duty webserver {
                        postfile=chicken.webserver.postfile
                        postproc=chicken.webserver.postproc
                }
                duty fileserver {
                        postfile=chicken.rcsgnl.testserver.postfile
                        postproc=chicken.rcsgnl.testserver.postproc
                }       # end of duty fileserver for GNL
        }
# Radar lab is special in its requirements
        bunch rcs.rrsl {
                duty computeserver {
                        postfile=chicken.rcsrrsl.computeserver.postfile
                        postproc=chicken.rcsrrsl.computeserver.postproc
                }       # end of duty computeserver for RRSL
        }


        duty computeserver {
                postfile=chicken.computeserver.postfile
                postproc=chicken.computeserver.postproc
                arch rh71-ia32 {
                        postproc+chicken.rh71-ia32.postproc
                }
                arch sol8-sparc {
                        postfile=chicken.sol8-sparc.postfile
                        postproc+chicken.rh71-ia32.postproc
                }
        }
# Here is a comment that spans the entire line.
}

The format of the nugget control file

This is the modified BNF of the nugget control file.

nugget_conf_file ::== {stanza}+

stanza ::== stanza_type stanza_name { {parameter_assignment}* stanza } 
stanza_type ::== nugget | arch | bunch | duty
stanza_name ::== see discussion of stanza names below.
parameter_assignment ::== assignment | append
assignment ::== parameter=value


append ::== parameter+value
value ::== see discussion of datatypes below.
parameter ::== letter{letter_digit_underscore}*

Comments are introduced by a pound sign (#) and terminated by the end of a line. Parameters are case-insensitive.

Each of these control files is required to include at least the one nugget stanza, and can then be followed by any number of arch, bunch, or duty stanzas to provide information specific to that category. Allowed values for bunch or duty are listed in the Nikola FAQ. Stanzas will be read in order, so to keep stanza heirarchy intact, all key/value pairs must occur before nesting of another stanza begins. Pairs can use either the = operator to assign a value or the + operator to append/concatonate.

In version 1, the format of the control file is restricted. The stanza introductions must be keyword value {. Each stanza ends with }. There can be only a single assignment or concatenation per line. Each line may have as much white space as desired. In version 2, the format of the control file is unrestricted. Comments begin with a # and end at the end of the line.

The inner meaning of stanzas

The operation of the master controlfile parser can be understood in terms of set theory. The universe of discourse is the set of all computers that a nikola installation supports. The MCP extracts subsets. The different stanza names are different ways to slice and dice the subsets. For example, if we run
mcp -b rcs -d nameserver,fileserver,workstation
then the stanza

bunch rcs {
   ...
   duty nameserver {
        ...
   }
   duty fileserver {
        ...
   
}

would be accepted. The stanza

bunch rcs.dcl {
   ...
   duty nameserver {
        ...
   }
   duty fileserver {
        ...
   }
}

would also be accepted, because bunch rcs.dcl is a subbunch of rcs.

By way of contrast, the stanza

bunch rcs.dcl {
   foo=seven
   duty computeserver {
        foo+fourteen
   }
}

foo+fourteen would be rejected because computeserver is not a duty we're handling in this run, but foo=seven would be accepted.

Note that the rules are associative. The stanza

bunch rcs.dcl {
   duty timeserver {
        foo=bar
   }
   duty nisserver {
        foo=pool
   }
}

is exactly the same as the stanzas

duty timeserver {
    bunch rcs.dcl {
        foo=bar
    }
}
duty nisserver {
    bunch rcs.dcl {
        foo=pool
    }
}

For example, here are 4 stanza beginnings and 4 different bunches specified with the -b switch.

Note that the only stanza that is required is the nugget stanza although this requirement is not enforced in the current version of the MCP. In particular, the arch stanza is not required, and if a given subset was always on a single architecture, then it could be left out. For example,

nugget wisdom {
   arch rh71-intel {
      bunch rcs {
        duty nameserver {
             foo+rachel
        }
      }
    }
}

is completely equivalent to

nugget wisdom {
   bunch rcs {
      duty nameserver {
         foo+rachel
      }
   }
}

because all of the RCS nameservers are on RedHat 7.1 based machines.

The differences in the subsetting rules for the stanzas are

• nugget
  Always accepted.
• duty
  If the duty named in the stanza is a member of the set of duties specified on the command line, then the stanza is accepted
• bunch
  If the bunch.subbunch.subsubbunch specified on the command line is within the bunch named in the stanza, then the stanza is accepted.
• arch
  If the architecture named on the command matches precisely with the named architecture in the stanza, then the stanza is accepted

Note that it is possible to create impossible stanzas, e.g.:

nugget dumb {
	arch rh71-intel {
	bunch rcs {
		duty nameserver {
			foo+rachel
			bunch ssli.nameservers {
				foo+leah
			}
		}
	}

}

Since bunch ssli.nameservers is not a subbunch of rcs, leah will never be appended to foo. This might occur if you were sloppy cutting and pasting.

Configuration Parameters

• filename
• masterfile
• prefile
• postfile
• preproc
• postproc

Stanzas

This is the current list of allowed values for the stanza names. The Nikola Team will add new values as the need arises. So, for example, when RedHat software releases Red Hat 8.0 and if we decide that we will support it, then we will add the values rh80_ia32, rh80_ia64, rh80_alpha.

Nugget control file parameter list

Data types understood by the nikola database include:

Master Controlfile Processor (MCP) Program

The Master Controlfile Processor (MCP) program accepts a control file and some switches and emits key/value pairs appropriate for further processing.

The usage of the MCP is:
mcp -c controlfile -b bunch -d duties -n narch [-v]
mcp -h | --help | -V
mcp -H hostname | --host hostname -c controlfile
where:

controlfile is the name of the control file as described above.

bunch is the name of a group of computers. Subbunches are appended to the bunch with a period, e.g. rcs.gnl

duties is a comma separated list of duties.

narch is an architecture from the list above.

-v run verbosely. This is most commonly used for debugging

hostname is the name of the host for which we wish to run this controlfile. The mcp will lookup the parameters in a database and use them instead of the command line switches. This will be implemented in a future version.

EXAMPLE

mcp -c ted -b vlsi.mtml -d workstation,time -n sol8-sparc -v
would read the control file ted nugget and process it for the vlsi bunch and mtml subbunch; generating all of the outputs required for a workstation and a timeserver running on a SPARCstation^{TM, Sun Microsystems} running Solaris version 8.

mcp -H truk -c genetic In version 2, this will lookup truk in the nikola host database and generate all of the outputs required based on the input genetic nugget.

mcp -c $NUGGET -b `cat /var/nugget/bunch` -d `cat /var/nugget/duties` -n `cat /var/nugget/arch` -v
This would read the control file in the shell variable NUGGET and process it according the bunch and duties in the files in /var/nugget. This idiom would probably go in a script and would be run on many different machines without changes.

Copyright (c) 2001, University of Washington, UW/EE