------------------------------------------------------------------------------------- Document.....: FOSDEM_SLACK.txt Purpose......: Framework for FOSDEM 2005 Slackware talk Resulting stuff will be turned into Open Office slides & accompanying discussion Author.......: Stuart Winter Date.........: 07-Jan-2005 Version..... : 1.01-final ------------------------------------------------------------------------------------- Key: [*] (*) Stuff I don't like - needs changing /*/ An insertion of something such as a screen shot, logo, icon. Anything else is slide material. ------------------------------------------------------------------------------------- Introduction ------------ Good afternoon! I'll just briefly introduce myself. - Have been a contributor to the project for about 5 years - wrote slacktrack: a tool to help produce Slackware packages from simple build scripts. I will briefly talk about slacktrack during the course of the presentation. - Develop ARMedslack: an unofficial port of Slackware to the ARM architecture This talk will cover the areas of the Slackware package system with a view to building packages. There is no official documentation describing the official build process, so this afternoon, I will lift the lid on some of the processes Patrick Volkerding uses to build the packages you see in Slackware. Topics covered -------------- - Introduce the basic structure of a Slackware package - what would you expect to find in a typical package? - Talk about the package management tools - Then get more in depth: package naming & version numbering and we'll look more specifically at the contents of the package. - Cover the packaging guidelines & policies - Move onto build scripts - introduce the two methods used in Slackware today - For those of you who may be interested in building your own packages, we'll finish with a few pointers to help produce good packages. 1.0 Overview of the Slackware packaging system ------------------------------------------ 1.1 Top level overview of a typical Slackware .tgz package: ``````````````````````````````````````````````````````` [ * ] To give you a quick overview/reminder of the basic nature of Slackware packages before discussing the Slackware package management tools. [ * ] - gzipped tar archives containing the directory structure & files [ * The files & dirs have the permissions already set * ] [ * People ask why not bzip2? uses more memory! However, it can run in a low memory mode but is considerably slower than gzip * ] [ * Cannot get uncompressed size easily with bzip2 which pkgtools uses for display purposes * ] - contains install scripts - contains package description - typically monolithic (all-in-one) packages [ * Slackware typically does not have 'dev' packages has its good and bad points + Good in that it helps ease dependencies - only one package is needed; easier maintenance. [ * Personally I prefer this: I remember when I first started using Linux and began building stuff from source on Red Hat Linux 4. I'd find the RPM I needed that provided the headers or libraries required, go to a friend's house to download it (I had no Internet); try to install the new RPM only to find that it needed another.. rinse & repeat. With Slackware 3.5 I didn't need to download anything! * ] - Bad in that it makes it harder to install a stripped down installation of Slackware. - You may wish to take this into consideration if producing packages, but it's up to the individual to decide what is included within their package * ] 1.2 Introducing the Slackware package management tools: ``````````````````````````````````````````````````` [ * Once you have initially installed Slackware from the installer you have facilities to install, remove and upgrade packages using the package management tools * ] [ * The package tools are shell scripts and use rudimentary tools such as tar, find, sed, cat -- not reliant on Python/Perl * ] Command line tools: - installpkg: used to install new packages - removepkg : used to remove installed packages - upgradepkg: used to upgrade already installed packages with new .tgz's [* but can also be used to install new pkgs - useful for scripting *] - explodepkg: used to extract Slackware .tgz packages into the current directory. Mainly used by package build scripts to extract 'package framework'. [ * which I will talk about later * ] Curses based tools: - pkgtool: Front end to the above utilities Provides information about packages in a convenient list format Provides ability to run/re-run interactive package setup scripts [ * These scripts are those that run during the OS installation; I will be talking about these in the next section * ] [ * However, these scripts can be run from the shell * ] [ * The simplicity of the package system is a deliberate design choice. A small shell script can be constructed to examine & manipulate the package database -a collection of text files- * ] 1.3 In depth look at package anatomy: a typical Slackware package [ MAJOR HEADING ] ````````````````````````````````````````````````````````````` 1.3.0 Package naming scheme ````````````````````` [ * From Slackware 8.1 and up, each package .tgz is named as follows: * ] packagename-version-architecture-buildnumber.tgz * Package names are either the name of a single program or of a collection/bundle of utilities: autoconf-2.59-noarch-1.tgz: single program [ * 'autoconf' is the name of the the entire collection of binaries and associated documents that are extracted from the autoconf source distribution archive * ] [ * However, if we consider another example * ] tcpip-0.17-i486-29.tgz: a bundle of utilities [ * There is no single piece of software called 'tcpip'. This package contains a number of associated utilities written by different authors but bundled into one single 'package' * ] 1.3.0.1 Package version numbers ``````````````````````` [ * If the package is the name of a particular piece of software such as 'autoconf' from the example above, then the version number represents the version number that its authors distribute * ] Single programs: version number is that of the released software 'Bundle' packages: version number *usually* relates to the most major piece of software contained within the package. [ * If the package is a 'bundle' such as 'tcpip' then the version number increases as and when you add a new piece of software to the package, or upgrade a particular piece of major software contained within the package. For example, with 'tcpip' above, the 0.17 refers to the version of Linux Netkit. However, there are other programs included within the Slackware tcpip package that are not part of 'Linux netkit' * ] 1.3.0.2 Package architectures ````````````````````` The architecture name tells you which architecture the package was built for. The current official values in use are: ----- [ Official Slackware architecures ] noarch - Architecture independent files such as config files i386 - packages for the i386 or newer (Slackware v8.1 & 9) i486 - packages for the i486 or newer (Slackware 9.1+) i586 - packages for the i586 or newer s390 - packages for the IBM s/390 mainframe [ * Note: Whilst Slackware v10 is primarily built for i486, you may find that there are some packages whose architecture versions are higher than i486. This is for two reasons: [a] There is no source for the package - it is a repackaged binary distribution (SUN's j2sdk is an example). [b] The package is not required or otherwise not suitable for earlier revisions of the architecture (this is especially the true with ARM and SPARC) * ] 1.3.0.3 Build numbers ````````````` [ * Build numbers supplement the version number and are changed when the *package maintainer* makes a change to the package but the original source code and version number remains the same * ] - Supplement the major version number - Changed only when the *package maintainer* makes a change [ * Example: I building a new package - version 1.56 of 'foo' Slackware package name is: foo-1.56-i486-1.tgz I haven't configured the 'bar' parameter correctly in /etc/foo.conf; I fix it and re-build the package. The new .tgz package is named foo-1.56-i486-2.tgz * ] 1.3.1 Exploding a package ``````````````````` /*/ tar ztvvf of a typical .tgz containing: install/, /usr/bin, /usr/doc, /usr/man, /usr/info, /usr/X11R6/ /var/log/setup/ /*/ Figure 1: Cross section of the contents of a typical Slackware package [ * As you can see, this is the sudo package, but trimmed for brevity. I've emboldened a few of the lines here as they are of importance. Take note of the owner and group names of the binary directories. These directories and permissions form part of the Slackware packaging policy and standards, which I will be discussing in the next few slides * ] 1.3.1.1 install/slack-desc ``````````````````` [ * The /install/slack-desc file is a plain text file containing the package's description; it is displayed when the package is installed * ] /*/ Show a small package description along with the usual handy ruler stuff /*/ Figure 2: the package description file, slack-desc This file is *mandatory*. 1.3.1.2 install scripts: install/doinst.sh ``````````````````````````````````` Three types of optional post installation ash-compatible shell scripts: install/doinst.sh : Typically used to recreate symlinks and configure the newly installed package; executed by installpkg - not interactive [ * Often these scripts contain code to create symlinks but may also be used to do other stuff such as: - move new configuration files into place; - used by glibc to move the libraries around to allow for on-the-fly upgrading - add in users into passwd, group, shadow (openssh) * ] /var/log/setup/setup.foo: Interactive configuration script - run from 'setup' configure option and from the OS installer (once all selected packages have been installed) [ * (for Apache, modssl etc, x11 fonts) * ] /var/log/setup/setup.onlyonce.foo: Run once from the OS installer These are deprecated and were last used in the 1993/94 era. Everything in these have been moved to either doinst.sh or setup.foo [ * can also be run from pkgtools but I don't think it should ;) * ] [ * The typical package configuration stuff is the installing/renaming of config files -- some Slackware packages use a .new extension to avoid trampling over existing config files. If no original config file is found, then the .new copy is renamed * ] [ * The reason that the shell script must be compatible with the ash shell is because this is what the OS installer uses. However, it's unlikely that your package installation script would get past Patrick if it were to be in the main package series in Slackware :-) * ] 1.3.1.3 Package policies & general guidelines ````````````````````````````````````` * Major binary directories are chown root:bin /sbin, /bin, /usr/sbin, /usr/bin, /usr/local/sbin, /usr/local/bin /usr/X11R6/bin [ * Following an old standard, now mainly cosmetic * ] * gzipped info pages, man pages and Kernel modules * FHS compliance: In particular, the following locations are not compliant: /usr/share/{man,doc} - these are /usr/{man,doc} [ * This is one of the key areas that you should be aware of when producing packages. Slackware maintains symlinks in /usr/share * ] * Hardlinks converted to symlinks [ *although in packages with many hardlinks such as glibc, this tends to slide * ] * Symlinks converted into shell script code; placed into install/doinst.sh [ * This allows the packaging system to keep track of which symlinks belong to which package, and where any symlinks overlap between packages * removepkg does not handle symlinks if they are not converted into shell script code ! * ] * stripping of: static archives, shared objects and binaries [ * usually -g for .a ; --strip-unneeded for .so & binaries * ] * Deletion of any perlocal.pod files. * HTML man pages are removed from any doc directories if they have counterparts in /usr/man. 1.4 Dealing with dependencies: `````````````````````````` - The recommended installation of Slackware is to install all packages, therefore the problem of dependencies does not exist - Official Slackware tools do not handle package dependencies - Dependencies may be listed in the package description or in some config files - 3rd party tools can help, but these are unsupported and may use additional non-standard files in install/ Unsupported by Slackware's official pkgtools - a/aaa_elflibs package: satisfy some core dependencies [ * contains shared objects from a variety of core packages against which a number of required packages are linked. Stuff like zlib, bzip2, gpm, ncurses, termcap are included within here. This allows your system to boot and work if you do not install the full packages. * ] [ * As an aside: An important note about this package: You should NEVER upgrade this package if you are using Slackware-current as it is only updated prior to a release of Slackware; as such it contains old versions of libraries - often much older than the versions in the packages from which the .so came * ] 2.0 Building Slackware packages: ---------------------------- [ * Now we get onto the exciting bit! ;-) I'm now going to explain the methods used to build the official Slackware packages and give a few pointers to help you produce your own packages * ] [ * I'm assuming that you have some rudimentary experience of building open source software * ] 2.1 What is a build script? ----------------------- [ * Just to make sure we understand the basic concepts of what we consider to be a package build script * ] [ * In essence, a build script is a set of instructions to: * ] - prepare a build location/directory - extract source archives - configure the source - compile software - prepare a packaing location [ * a directory into which the package will be installed * ] - install the components into a location ready for packaging - apply the packaging/distribution policies to the package contents - create a binary package from the packaging location [ * So, that's an overview of what a build script is. Now let's look at how we do it in Slackware * ] 2.2 Overview of the package building system: introducing makepkg ------------------------------------------------------------ [ * Before moving on with build scripts, I first need to introduce makepkg! * ] - belongs to the a/pkgtools package - produces Slackware .tgz packages from the contents of the pwd - converts symlinks to shell script code - does some basic sanity checks -- zero length files [ * Useful to highlight any files that shouldn't be 0 bytes -- but remember that some 0 byte files are meant to be there as place holders * ] Basic syntax: # makepkg -l y -c n /tmp/foo-1.0-i486-1.tgz [ * Note on the slide that I say it converts symlinks to shell script code. You should ALWAYS do this because removepkg does not remove symlinks! * ] [ * the -l y tells makepkg to convert symlinks into shell script code the -c n tells makepkg not to change permissions and ownerships of directories. You should always have these worked out before building a package * ] 2.2 Overview of the package building system: Build types ---------------------------------------------------- [ * In Slackware we have what I call clean & dirty builds. I could have called them clean and unclean, but I wanted to make sure the distinction was clear :-) * ] Two types of build script: - Clean package building scripts: - Compiles software - Installs into a special build location [ * rather than the file system * ] Package is created by running makepkg within the build location - Dirty build scripts: - Compiles software - The package is installed onto the file system. [ * We'll talk about how we make packages from dirty build scripts BUILD NOTES ----------- * Always built as root [ * No chroot or fakeroot * ] * Unless explicitly stated in build scripts, all packages are built on a full Slackware system. [ * This can mean that certain dependencies are created * ] Almost all build scripts are available in the source directories of each named package. 2.3 Overview of the package building system: SlackBuild scripts ------------------------------------------------------------ [ * Gentoo has ebuilds, Debian has 'rules', RPM has 'specs', we have SlackBuilds * ] [] SlackBuilds: the clean package building scripts - Reproducible package builds - Basic, simple 'bash' shell scripts - no macros/functions - Understandable at a glance - well commented - Installs package into a pseudo root directory [ * DESTDIR, TOPDIR, INSTALL_PREFIX or similar * ] - Sets Slackware packaging policy - The preferred method. [ * Let's look at the very basic pseudo example now * ] Example of a SlackBuild script: # [ Setup build location & extract source ] # VERSION=2.4 # ARCH=${ARCH:-i486} # BUILD=${BUILD:-1} # ./configure --prefix=/usr # make # make install DESTDIR=/tmp/package-foo # [ Install docs into package ] # [ Set Slackware packaging policies ] # makepkg -l y -c n /tmp/foo-$VERISION-$ARCH-$BUILD.tgz To compile the software and build the package: # ./foo.SlackBuild [ * Note the ARCH, VERSION and BUILD variables. Most SlackBuilds allow you to override these parameters from the shell * ] [ * With most SlackBuilds, the .tgz will be dropped into /tmp * ] [ * There is much more to know about SlackBuild scripts than I can fit into this presentation: check the OVERVIEW document included with slacktrack. If you're interested then look in the extra directory. * ] 2.4 Overview of the package building system: .build scripts ------------------------------------------------------- [ * This is the other way! * ] [] .build scripts: the dirty build & install scripts - Very basic shell scripts - Do not set Slackware packaging policies - Installs onto the file system - Do NOT produce packages! - Usually used when the specific software's build system has no provision for installing into a separate psuedo root location - Gradually being phased out of Slackware Example of a .build script: # [ setup build location & extract source ] # ./configure --prefix=/usr # make # make install # [ Install docs onto file system ] To compile the software and install onto the file system: # ./foo.build [ * The thing you may have noticed is that these build scripts do NOT produce packages! * ] [ * So, you may be wondering how we can produce packages these build scripts? * ] [ * Some of you may remember, or even use protopkg? This was written by David Cantrell when he worked for Slackware. protopkg's modus operandi was based on scripts written by Patrick, and protopkg was used to build the packages for the SPARC and Alpha ports of Slackware and is used as Slack390's build system * ] Anybody worked it out yet? :-) * ] 2.5 Overview of the package building system: creating packages from .build scripts ------------------------------------------------------------------------------ [ * This is how Patrick produces packages from .build scripts * ] - Find all files on the file system & log - Launch the .build script (build & install software) - Find all files on the file system & log - Determine the differences between the two logs - More or less, these are the contents of your package! - Copy the contents to a psuedo root directory - Set the Slackware packaging policies - Run makepkg [ * Quite an involved process don't you think? Patrick uses an unreleased script called "bp" to help with some of these steps, but the process is still somewhat involved. Actually, bp does a few more things than those listed on the slide but I don't want to frighten you with those details ;-) * ] 2.6 Overview of the package building system: creating packages from .build scripts ------------------------------------------------------------------------------ Introducing slacktrack ---------------------- [ * So, you'd hope that there's an easier, safer and more efficient way of producing packages from .build scripts? Yes there is! * ] [ * I originally wrote slacktrack so that I could rebuild replica Slackware packages from the .build scripts, but also to just build packages from simple build scripts I'm sure many of you are aware of checkinstall. It's a user friendly utility that can track a 'make install' and produce packages in a number of formats - Slackware, Debian, RPM etc. I started using checkinstall and found it great to use, but unfortunately contained a number of bugs at the time. Since I was starting work on ARMedslack, I thought that it'd be easier to write my own installation tracking and package building script that was aimed purely at Slackware and did exactly what I wanted. So, I took some great ideas from checkinstall, took its version of installwatch and wrote the slacktrack utility! * ] - Orginally written to be ARMedslack's primary build system [ * my port of Slackware Linux to the ARM architecture; so that I could build the packages that have .build scripts * ] - Inspired by "bp" (buildpackage), checkinstall and 'protopkg' - Specifically aimed at Slackware - only builds .tgz's - Can apply all Slackware packaging policies to your package [ * I set about finding out what these were by examining package tar balls and build scripts and asking Patrick * ] - Can be used to build replica packages from .build scripts - Includes 'altertrack' which does not use installwatch - Can be found in slackware-current/extra/slacktrack Sample usage: # slacktrack -Qp bar-5.2-i486-1.tgz ./bar.build slacktrack will launch the dirty build script, track the installation, apply all Slackware packaging policies and produce you a package in /tmp. [ * slacktrack has whole host of options - the above is just the recommended way of calling slacktrack. slacktrack is not JUST for use with Slackware's .build scripts; there are a number of helpful features to help you build packages with your own home brewed scripts. slacktrack also includes an alternate utility called altertrack which is almost identical to slacktrack but it does not use installwatch. altertrack uses the method I touched on earlier which is to compare file systems before and after the build. This is actually the method I tend to favour when building ARMedslack packages because it means that static binaries will be tracked (something which installwatch cannot presently do). But for most users, the standard slacktrack utility will be the best. One thing worth noting is that you can't always produce 100% replica packages even with slacktrack. Patrick often makes tweaks by hand before running makepkg. I always compare ARMedslack's packages to Slackware's after a build, and I rarely see any differences that shouldn't be there so it's not too much of a deal. I thought I'd just tell you incase somebody sends me a bug report ;-) * ] 2.7 Package building checklist -------------------------- * Directory locations ------------------- [ * FHS non compliance: /usr/info, /usr/man, /usr/doc Ensure your package does not put such data into the wrong place! * ] * install/slack-desc ------------------ [ * - Best to always copy a template from Slackware's source tree - The description doesn't display when installed with installpkg? does it match the package name? typos? hyphens* ] install/doinst.sh ----------------- [ * A basic ash compatible shell script although you could use bash-only functionality if you wish, as long as your package doesn't end up in Slackware ;-) * ] [ * Always use relative pathnames! cd into local root and scripts executed. * ] Permissions ----------- [ * root:bin binary directories and contents? The rest are owned by root:root? - packages produced by checkinstall in the wild -- uid problems * Check your package does not have any globally writeable files that should not be there. PHP used to be a classic example - it'd install some files globally writeable. * root directory is chmod 755! Users may not be able to login otherwise! * Ensure man pages are gzipped and that you have fixed any broken symlinks * ] Stripping --------- [ * Most of Slackware's packages are stripped to reduce the on-disk size of the package. Optional * ] Comparison with official .tgz ----------------------------- If you are modifying a Slackware build script and rebuilding a package then the easiest way is to compare the contents of the official .tgz with your own. This way you can see whether your build has missed any files, whether the permissions are the same, and so on. 3.0 Closing talk: ------------ I have tried to keep this presentation at a fairly high level in order not to bog you down with too much detail, but I hope that I've given you enough of an introduction for you to begin building your own packages if you wish, or just to have explained the Slackware package managemenet and build system. Thanks to the Slackware release team for suggestions and Thank you for listening! I think it's now time for the Q&A session. This is not relevant:: - Usually 'virgin' sources, but ocasionally patched: - for security - due to Slackware policy (file hierarchy locations, for example) - in order to build with the latest toolchains - Most packges compiled for i486 and optimised for i686 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< How to write a typical slackbuild ---------------------------------- Write the script to be as portable as possible. - ARCH - BUILD - VERSION [ ******* Is this stuff relevant? I don't think so ** ] Although the setup scripts are: [] What happens when a package is installed Basic outline: - tar archive is extracted into the root directory (either / or a user-defined location) - an entry is made in /var/log/packages - Any scripts are placed into /var/log/scripts - The package description is displayed - the installation script is executed - /var/log/setup/setup{.onlyonce,}.foo interactive setup scripts are executed if the package is installed via pkgtool or from the installer. I don't think this is relevant either -- not to building packages. This stuff isn't on my mind when I'm building packages: [] Getting information on packages - pkgtool - /var/log/packages MANIFEST.BZ2 file - In the slackware/ tree on the installation media [] Packaging system directories: /var/log/packages: lists package contents & description /var/log/scripts: contains the package's original non-interactive installation scripts /var/log/setup: contains the package's interactive installation scripts For reference: /var/log/removed_packages /var/log/removed_scripts Commented notes ----------------