Planet Exherbo

It’s been a while since Ciaranm McCreesh started working on SUMMER, but now it finally found a proper place on Exherbo infrastructure: http://summer.exherbo.org/.

SUMMER, also known as Statically Updated Metadata Manifestation for Exherbo Repositories is an online browser for Exherbo packages.
It’s written in Ruby, using Paludis’ Ruby bindings and ERB, a Ruby templating engine.

Currently, packages’ metadata from all official repositories, plus all repositories included in the set of unofficial repositories known as ::unavailable-unofficial is available. Especially the integration of unofficial repositories,as an important part of Exherbo, makes SUMMER different from its counterparts in other distributions.

The code is available on our gitweb. Suggestions, especially accompanied by git-format-patches, are highly welcomed! There’s a short wishlist in the README, which is hopefully going to start shrinking soon.

sydbox-0.1_beta5 is released.

• Implement /dev/sydbox/{un,}ban_exec. paludis-sydbox branch makes use of these magic commands to ban execve() calls in metadata phase.
• When shell_expand() returns empty string, it was added to the pathlist resulting every path prefix check to be allowed, this is now fixed.

tarball: sydbox-0.1_beta5.tar.bz2
sha1sum: 2b02bc204148f94bb79b7a5b190f1d2069394ecb

Running external commands in the metadata phase of exheres/ebuild is obviously
a bad idea because this phase is used to generate caches.

Ciaranm has come up with an idea to generate sydbox access violations when execve() family functions are called in the metadata phase. This was rather easy to
implement.

I’ve added two sydbox magic commands, namely /dev/sydbox/ban_exec and /dev/sydbox/unban_exec. Writing to the former file sets the flag to ban all execve() calls and writing to the latter unsets the flag.

A small example looks like:

#!/bin/sh

/bin/true # This call succeeds.
:>/dev/sydbox/ban_exec
/bin/true # This call fails with EACCES.
:>/dev/sydbox/unban_exec
/bin/true # This call succeeds.

sydbox-0.1_beta4 is released.

• Use an lstat(2) wrapper which tries hard to avoid ENAMETOOLONG issues.
• Handle /proc/self correctly when resolving paths.

tarball: sydbox-0.1_beta4.tar.bz2
sha1sum: ebc650689267539e22da1c1dc2aec818b29382c6

sydbox-0.1_beta3 is released.

• Fixed event handling and inheritance of sandbox data. Sydbox behaves correctly now when a child calls fork(), vfork() or clone().

tarball: sydbox-0.1_beta3.tar.bz2
sha1sum: 7ace8ee1463e3b76543c401334e7f6666547b97b

sydbox-0.1_beta2 has been released.

• Canonicalize filenames by default. This was previously only done in paranoid mode. This makes sydbox stricter (expect more testsuites to fail).

tarball: sydbox-0.1_beta2.tar.bz2
sha1sum: 34cef23db6d81a34b27617c07e5c3f67128ca99d

sydbox-0.1_beta has been released.

• Fix bogus access violations. As an example sydbox would throw an
  access violation in a simple case like

So MSDNAA lets university students get some Microsoft products for free for educational purposes. That’s all fine and dandy except that you can’t directly download stuff, you have to do it using a shitty download manager.

It is hilarious that you have to run a Windows executable to download a copy of Windows XP.

Chicken and egg?… Yeah, that’s clever.

One point for you Suckrosoft.

As some of you may be aware, there exist a few dark heretics who have yet to embrace the One True Editor and instead go around peddling their evil ways (via m-x peddle-evil-ways) or their primitive superstitions (yes, ^O to save makes perfect sense!).

Although Exherbo would ordinarily go out of its way to smite those wicked sinners, unfortunately it seems that some of them are so set in their ways that even removing their satanic instruments from the basic install will not quell their iniquity. Thus, in a rare and display of compromise that risks turning Exherbo into… uh, no, wait, I can’t think of any distributions capable of sensible compromises. Anyway:

e4r is a small script that turns Vim into a very crude, minimally functional non-modal editor that approximately resembles Nano. It has no dependencies other than Vim, and is extremely tiny, making it practical to include it in stages for people whose time isn’t valuable enough for them to learn how to use an effective text editor.

Some screenshots:

Editing a file using e4r

Loading a file with e4r

e4r Help Menu

Git format-patches welcome.

For SpaceFish I wanted a quick way to load a 3D model and be able to rotate it easily. This, along with an artificial horizon is a nice way to see the attitude of the plane while it is far to see it. It is also quite good to replay recorded data during experiments to study it further in the lab.

Since I haven’t really done anything in OpenGL beyond stupid examples and helloworld-like programs and given that I don’t have much time to devote to this task I decided to use an existing framework. Among Irrlicht, OpenSceneGraph and Ogre. I decided to give OpenSceneGraph a try, for no particular reason.

Unfortunately, it took a while to get it working under OSX… it needs something like this. I use MacPorts for this kind of stuff under OSX:

fixOpenSceneGraph()
{
	local lib_dir=/opt/local/lib
	pushd ${lib_dir} > /dev/null
	echo pushd ${lib_dir}
	for i in libosg*.dylib* libOpenThreads*.dylib* ; do
		echo install_name_tool -id ${lib_dir}/${i} ${i}
		otool -L ${i} | sed -e 1d | while read f ; do
			if [[ ${f} =~ ^libosg ]] || [[ ${f} =~ ^libOpenThreads ]] ; then
				l=${f%% *}
				echo install_name_tool -change ${l} ${lib_dir}/${l} ${i}
			fi
		done
	done
	popd > /dev/null
	echo popd
	port contents OpenSceneGraph |
		sed -n -e '/^[[:space:]]*\/opt\/local\/bin\//p' -e '/\.so/p' |
		while read f ; do
		otool -L ${f} | sed -e 1d | while read d ; do
			if [[ ${d} =~ ^libosg ]] || [[ ${d} =~ ^libOpenThreads ]] ; then
				l=${d%% *}
				echo install_name_tool -change ${l} ${lib_dir}/${l} ${f}
			fi
		done
	done
}

It wasn’t fun to figure that out (thank you CMake)…

Anyway, I think I’ll stick to OpenSceneGraph until it annoys me enough. For now, I managed to do what I wanted in less than 2 hours, which is pretty good when you have a stupidly tight schedule.

— ferdy

envtag-0.4 has been released.

• Modular architecture, exporting tag functions to Lua.
• Proper exit codes
• Update alt_getopt to 0.6

tarball: envtag-0.4.tar.gz
sha1sum: a74c84286e1b61dfe9522635b2f7970589793fe0

As many of you know paludis has a –resume-command-template option to save the resume command to a file. In addition to that there’s the pretend-resume.hook which displays the resume command or the file saved at the end of –pretend –install output.
The only missing thing is a way to quickly execute this command prepending with sudo. I wrote a tiny zsh function, which i called plast, to do the job. To use it, save the file somewhere in your $fpath and put

export PALUDIS_RESUME_DIR=/path/to/the/directory/where/you/save/your/paludis/resume/command/files
autoload -U plast

in your .zshrc.
You can use it like:

paludis -ip bla bla bla
plast # executes the command in the newest resume file prepending with sudo

I’ve already covered everything that made it into EAPI 3; for those interested, I’ll briefly discuss some of the things that were voted out or didn’t make it.

First, things that made it into the draft proposal, but were voted out:

• A ban on || ( use? ( foo ) ). This construct is impossible to use correctly (despite misinformation to the contrary from certain people who keep trying very hard to come up with a perverse justification for it), is used incorrectly everywhere it is currently used and has lots of special wording for it in PMS. Unfortunately, the Council weaselled out of outright banning it in favour of maybe introducing a repoman check and maybe banning it later.
• doexample and doinclude. Nobody cared enough to make a convincing defence of their necessity, so they were voted out.
• More econf option changes. --enable-fast-install was snuck into the --disable-dependency-tracking feature request with no explanation. The Council voted against it.
• unpack --if-compressed, which would make it otherwise fatal for unpack to be called with an unsupported extension, was voted out. This means that when you explicitly call unpack doesnotexist or unpack foo.tar.bz3, unpack will continue to do nothing silently and not indicate any error.

Then, things that were getting ready before the Council decided that in the interests of getting things done, anything not already on the list would be postponed for consideration for a future EAPI:

• Merger mtime preservation. This one, as worded, received much objection from the secret cabal directing Gentoo’s future from the shadows since it would mandate emulating Portage’s broken behaviour that results in files being merged with ridiculous timestamps.
• Prefix variables. The Gentoo prefix people wanted to sneak a few prefix variables into PMS by the back door, presumably in a way of getting prefix accepted without them having to discuss or address design, goals or implementation — this discussion getting postponed is probably good for Gentoo, since it’s well known that anyone who dares suggest that prefix is anything less than perfect ends up mysteriously no longer being a Gentoo developer.

This is part of a series on EAPI 3.

First, two removals: AA and KV. AA has been deprecated for ages, and isn’t useful for ebuilds — it’s like A, but contains every component, even those in unselected conditional branches. KV, meanwhile, contains “the kernel version”. Ebuilds that need that kind of thing should use one of the eclasses that provides similar but more useful functionality instead.

Now the more interesting part: two additions. In the good old days, you could use convoluted has_version magic in pkg_ phases to work out whether we’re in an upgrade, downgrade, reinstall or fresh install. The devmanual used to contain examples. Unfortunately, Zac went and broke all this when he changed phase ordering without an EAPI bump, leaving all the widely used and documented techniques for detecting upgrades unusable. Thus, EAPI 3 adds two new variables to fix this mess.

The REPLACING_VERSIONS variable contains a space separated list of all the versions of the current package name that will be replaced. It is only valid in pkg_preinst and pkg_postinst; if set during pkg_pretend and pkg_setup, it will contain valid values, but it may be misleading in the case of binary package building.

Note that REPLACING_VERSIONS is a list, not a single value. This is to deal with weird cases like having foo-1:2 and foo-2:1 installed, and installing foo-1:1.

The REPLACED_BY_VERSION variable contains the version of the current package name that has been replaced. It is only valid in pkg_prerm and pkg_postrm. If it is unset, it indicates a simple uninstall.

This is part of a series on EAPI 3.

With older EAPIs, econf (being a shell function) would die on failure, as do most eclass-provided functions, but emake (being an external, since people call it via xargs) and most package-manager-provided utilities (external or not) merely return non-zero.

In EAPI 3, most utilities (consult PMS for exact details) will now die when an error is encountered. This in turn means the package manager has to be able to die from an external utility.

Occasionally, this isn’t desired behaviour. You might need to call a function, but do something else if it fails rather than aborting. In this case, use the nonfatal utility with the command to run as its arguments. Mostly this is just used to give better error messages, but there are sometimes other cases:

nonfatal emake plugins || die "Making plugins failed. Do you need to frozbinate your zorgons?"

# these usually exist, but sometimes don't, and we're too lazy to check manually
nonfatal dodoc foo bar baz

This feature originated with exheres-0.

This is part of a series on EAPI 3.

With EAPI 3, the much-hated and overly complicated RDEPEND from DEPEND automagic assignment (along with its highly weird rules for eclasses, which no-one ever remembers) is dead. Developers should set DEPEND and RDEPEND explicitly, and if any sharing is required, a (non-package-manager-handled) COMMON_DEPEND variable of some kind can be used.

I’ve just created a repository to publish my paludis hooks. Right now there are two simple hooks in the repository. The first one is to generate locales after glibc updates. You can think this as a locale-gen equivalent for Exherbo. The second one updates /etc/localtime when timezone-data is updated.

This is part of a series on EAPI 3.

The introduction of a default src_install has one weird side effect that probably wouldn’t have been noticed before EAPI 3’s release without extensive use in Exherbo: if S is incorrect, developers will often end up accidentally installing an empty package rather than seeing the expected error. This is because the S to WORKDIR fallback happens even when it probably shouldn’t; with developers often no longer coding a src_install that will die if the expected Makefile isn’t found, this gets through unnoticed until it’s too late.

EAPI 3 therefore removes the S to WORKDIR fallback, except under certain carefully chosen circumstances: if the ebuild has empty A, and it defines none of src_unpack, src_prepare, src_configure, src_compile and src_install, the fallback will occur as normal.

Thus, ebuilds that install nothing don’t have worry; other ebuilds must set S=${WORKDIR} in global scope if necessary.

This is part of a series on EAPI 3.

EAPI 3 makes some fairly substantial changes to how use flags are handled inside the package manager. For developers, this requires a certain amount of attention; for users, there’s probably not much visible difference.

Historically, core Portage didn’t really do much with USE flags. They were used by conditional dependency specifications, but that was it — things like the use function were just bits of shell script that looked at the USE variable, which mostly just came directly from the user’s configuration. Similarly, IUSE was originally only used for cosmetic purposes — Portage used it to decide which flags to display in output, but didn’t use it to restrict USE in any way. Thus, packages could query use flags that weren’t listed in IUSE and still expect things to work.

USE_EXPAND gets even more complicated. Developers often didn’t bother even trying to list such flags in IUSE, leading to much confusion and strange behaviour.

This is a bit of a nuisance. It makes later user-facing functionality like “has this USE flag changed?” queries much harder to define, and it severely cripples use dependency defaults (was a flag not present for a given version, and thus should the default be used, or was it present but just not listed?). EAPI 3 fixes this.

In EAPI 3, IUSE is strictly enforced. If a flag isn’t listed in IUSE (and isn’t covered by the exceptions detailed later), the package manager will filter it from USE and any relevant USE_EXPAND variable, and make the use functions die if the flag is used in a query.

The problem is, Gentoo developers don’t tend to like the idea of having to explicitly list ARCH flags like x86 or certain USE_EXPAND flags like userland_GNU in IUSE. There are two reasons given for this:

• The flags shouldn’t be user visible.
• It’s lots of effort.

The first argument is mostly invalid; we already have USE_EXPAND_HIDDEN, and EAPI 3 is tweaking the definition of USE_EXPAND to let us treat ARCH as a USE_EXPAND value that isn’t prefixed with arch_ like it otherwise would be, allowing us to hide those too.

The second argument may or may not have merit; Gentoo’s gone for the “at least keep the possibility that we’ll avoid having to make people explicitly list things open” approach (at the expense of use dependency defaults not being usable on anything that’s treated that way), whilst Exherbo’s gone for “list everything”.

Here’s (roughly — see PMS for the gory details) how the compromise works:

• The package manager works out something we’ll call IUSE_EFFECTIVE (which may or may not be an ebuild-visible variable, is not an ebuild-modifiable variable and which will end up in VDB). It contains everything in IUSE, along with profile-injected extra values.
• Any value in the profile IUSE_IMPLICIT variable is treated as being in IUSE_EFFECTIVE. This would let the much-abused build flag carry on working; it has not yet been decided whether any values will be set this way in the profile.
• Any USE_EXPAND or USE_EXPAND_UNPREFIXED variable whose name appears in the profile USE_EXPAND_IMPLICIT variable is treated as having its values implicitly listed. This will probably contain USERLAND, KERNEL, ARCH etc, but not LINGUAS et al. For any USE_EXPAND_IMPLICIT variable FOO, the profile USE_EXPAND_VALUES_FOO variable lists its (unprefixed) implicitly injected values.
• IUSE_EFFECTIVE is thus a fixed, well defined set rather than an infinite set as it is in older EAPIs. Any value that doesn’t end up in IUSE_EFFECTIVE cannot be queried in any way, and is treated as ‘missing’ for use dependency defaults. Any value in IUSE_EFFECTIVE, whether or not it is explicitly in IUSE, is ‘present’ for use dependency defaults.

For developers, all you really need to know is that you need to list things in IUSE if you want to use them, except if they’re in a small group of special values that will likely contain ARCH, USERLAND and KERNEL flags and possibly things like build.

This is part of a series on EAPI 3.

With EAPI 3, the requirement that a package be installed before the package manager calls its pkg_info function has been dropped. If developers find they usually need to ask for certain information when a package build fails, they can make a pkg_info that will try to display that information, rather than having to ask the user to run commands by hand.

Some points to note:

• You can’t rely upon anything in particular being installed. We don’t force the package manager to ensure that a package’s dependencies are met. Thus, write code that only tries to display information, and doesn’t mind if it can’t.
• You could use has_version to determine whether something’s installed, but it’s better to just try to do whatever you want.
• There is no variable telling you whether your package is installed. This is because it’s not a simple boolean choice — do you care whether a package is installed, whether our exact version is installed, whether our slot is installed or something else? Again, has_version is available, but using it is probably a sign you’re writing code that’s trying to be too clever.

This feature was originally in kdebuild-1.

This is part of a series on EAPI 3.

In EAPI 3, econf will pass --disable-dependency-tracking. This might marginally speed up a few packages’ builds, and breaks a few others that have funky half-hand-written makefiles.

What it won’t do, despite what you may of heard from certain less well informed Council members, is break packages that are using custom configure scripts. Those packages, if they have configure scripts that die on unrecognised options, aren’t using econf anyway since econf already passes all kinds of weird nonsense through.

In summary, pretty much no-one has any reason to care about this particular feature, and it’s only being discussed here for completeness.

This is part of a series on EAPI 3.

In another disappointing defeat for the shadowy cabal controlling Gentoo’s every move (no, not that cabal, the other one), EAPI 3 will see unpack support xz format archives.

This is icky.

Support for unpacking lzma files was retroactively shoved into existing EAPIs because Mike Frysinger decided to just commit support to Portage without an EAPI change and start relying upon it for coreutils, making it impossible for everyone else to do anything except copy the change. This, it turns out, was a waste of time, since the lzma format is effectively deprecated in favour of xz.

Now, despite it not having any unmasked extractor on Gentoo, unpack in EAPI 3 is required to know how to unpack xz files. Whilst this is a step better than doing the change without an EAPI bump (no thanks to Mike, who tried to put it in without one), it’s still silly.

Of course, soon we’ll have to add in support for whatever the cool kids are using next week into EAPI 4.

This is somehow surprising the first time you see it and it is interesting to remind it to people from time to time. The question is easy, does this program always return 0? That is, are those summations the same?

If there’s a case when it doesn’t, provide a sample input and explain why it happens.

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
	float f[argc - 1];
	for (int i = 1; i < argc; i++)
		f[i - 1] = atof(argv[i]);
	float r1 = 0;
	for (int i = 0; i < argc - 1; i++)
		r1 += f[i];
	float r2 = 0;
	for (int i = argc - 2; i >= 0; i--)
		r2 += f[i];
	return r1 != r2;
}

As usual, I’ll post the solution in a couple of days or when someone gets it right (which is always the case :))

— fpereda

Update: correct typo spotted by Snaury.

This is part of a series on EAPI 3.

The dohard utility is used to create a hard link between two files. Except it can’t necessarily do that, and even when it does it probably won’t end up being merged as a hard link; up until recently, Portage only supported merging hard links by fluke if it happened to take a particular fast-path in the code, and few people even noticed. Even in ideal circumstances, dealing with hard links is fraught with mess, so we’ve decided to ban dohard in EAPI 3 as a way of discouraging their use.

As for dosed… It doesn’t really do what you think, it’s been considered deprecated for ages and there are better alternatives. EAPI 3 bans it too.

Both of these were originally banned in kdebuild-1. Unfortunately, kdebuild-1’s banning of the abomination that is dohtml hasn’t made it into EAPI 3.

This is part of a series on EAPI 3.

Currently, to install a directory full of documentation, you have to either copy the directory yourself (after making the parent directory), or play around with into and doins -r. EAPI 3 adds a -r switch to dodoc for recursively installing a directory.

On the subject of doins, currently doins -r’s behaviour is undefined for anything except files or directories. With EAPI 3, doins -r will also be guaranteed to install symlinks correctly.

Incidentally, there’re probably a whole bunch more small improvements like these that we could make to the utilities. Suggestions (along with demonstrations of use cases) for EAPI 4 will be being taken in the not too distant future; something like the above is easy to get in.

This is part of a series on EAPI 3.

Gentoo is, as everyone knows, about choice. This means it is considered vitally important that users get to choose such earth-shatteringly important matters as which program is used to compress their documentation files. It doesn’t matter that blockers repeatedly confuse people, that the tree is an unmanageable mess or that there’s no way of handling USE flag requirements — ensuring that the user’s choice of compression is applied to a small number of documentation files in every case is critical to Gentoo’s future.

As such, EAPI 3 introduces controllable compression. It works like this:

• The package manager maintains a list of things that may be compressed in some arbitrary way. By default, this list includes /usr/share/doc/, /usr/share/info and /usr/share/man.
• The package manager also maintains a list of things to be excluded from the above compression. By default, this list includes /usr/share/doc/${PF}/html.
• Inside src_install, packages can add things to the inclusion list using docompress blah, and to the exclusion list using docompress -x blah.
• After src_install has finished, the package manager may apply some kind of compression to anything in the inclusion list that is not also in the exclusion list.

PMS is carefully worded to allow all of this nonsense to be a no-op.

Controllable compression is a Gentoo-original feature, and clear proof that there is no magic external cabal dictating Gentoo’s future, since they’d never let people get away with this if there were.

This is part of a series on EAPI 3.

EAPI 3 will feature a default src_install that isn’t just a no-op. While it won’t eliminate the need for some packages to write their own implementation, it will help in a good number of cases.

The exact default is still subject to bikeshedding and could be changed at any point right up until PMS is tagged for EAPI 3. Right now we’re going with the following, and it’s very likely the final choice will look more or less like this:

src_install() {
    if [[ -f Makefile ]] || [[ -f GNUmakefile ]] || [[ -f makefile ]]; then
        emake DESTDIR="${D}" install
    fi

    if ! declare -p DOCS >/dev/null 2>&1 ; then
        local d
        for d in README* ChangeLog AUTHORS NEWS TODO CHANGES \
                THANKS BUGS FAQ CREDITS CHANGELOG ; do
            [[ -s "${d}" ]] && dodoc "${d}"
        done
    elif declare -p DOCS | grep -q '^declare -a ' ; then
        dodoc "${DOCS[@]}"
    else
        dodoc ${DOCS}
    fi
}

Note the following:

• There will be a DOCS variable. This was decided by a Council vote, since not everyone sees the obvious sense of the idea.
• If something is explicitly listed in DOCS, it will be an error for it not to exist. But if we’re using the default values, things are only installed as documentation if they’re there and not empty.
• We don’t use -j1 for the install. This will break some packages built using older autotools. This is in line with Gentoo policy.

The first practical non-empty src_install was in exheres-0. The exheres-0 version remains considerably more flexible and parametrisable than the above.

This is part of a series on EAPI 3.

DEFINED_PHASES and PROPERTIES were retroactively introduced into earlier EAPIs. We could get away with this because it’s legal for a package manager to do absolutely nothing with them and pretend they don’t exist.

PROPERTIES isn’t very interesting; it’s only mandatory in EAPI 3 because we like to tidy things up as we go along. DEFINED_PHASES, however, is now mandatory for a very good reason: it’s necessary to make pkg_pretend usable.

Invoking an ebuild process to run pkg_pretend, even if it does nothing, takes something like a tenth of a second. If you’re doing a world update, that could mean thirty seconds of sitting around waiting, and if you’re doing a full reinstall, several minutes. In the grand scheme of things, it’s not a huge penalty, but it’s one that occurs when the user is sitting waiting rather than something added to build time — and if we add too much to pretend time, users will just stop waiting for it, which defeats the whole point of it. By requiring DEFINED_PHASES support, a package manager only has to pay the penalty for packages where it will be useful.

This one’s more something that would be annoying if it were not there than a revolutionary new feature; users and developers shouldn’t have to care about it, except if they’re interested in the thought and planning that the shadowy cabal directing Gentoo’s future from their secret moon base put into all of this.

DEFINED_PHASES first appeared in exheres-0, where it was always mandatory.

This is part of a series on EAPI 3.

In EAPI 2, use dependencies are only allowed to reference USE flags that are listed in IUSE for every package that matches the associated specification. Unfortunately, this isn’t enforced, and developers often forget to pay special attention to it.

Use dependency defaults provide a way of saying “if this flag isn’t listed in IUSE, pretend that it had this value instead”. The syntax is to append (+) to the use flag name (the parentheses aren’t entirely pretty, but they’re necessary to avoid ambiguity) if the package manager should pretend that it is on if not present, and (-) if the package manager should pretend that it is off. So, one can do things like foo/bar[baz(+)] or foo/bar[!baz(+)?].

Because of EAPIs before 3 not being strict about what ends up in IUSE, there are still limitations. This can only be used on flags that would be listed in IUSE were they supported; it is not usable for USE_EXPAND values.

Use dependency defaults still require a degree of care from the developer. In particular, it’s not possible to write ‘pre-emptive’ defaults to handle future possible use flag removals; one does not know whether a use flag removal will be because it is always on or because a feature will be removed.

Use dependency defaults first appeared in exheres-0.

This is part of a series on EAPI 3.

Traditional dependencies and slots don’t really work very well together. If a package depends upon foo, and foo has multiple slots, the package manager doesn’t know what that means at runtime. Some packages just need any foo, and don’t mind if a completely different version in a different slot is swapped in later. Others need the slot that was around when they were installed.

The original theory was that all dependencies would be runtime switchable, and that any that weren’t would be controlled via use flags. But that’s extremely impractical, and largely pointless for heavily slotted things like db. Slot operator dependencies are a more pragmatic solution for the common case; thus, as of EAPI 3, any package that has both a DEPEND and an RDEPEND upon anything that can match more than one slot should specify either :* or := after the spec.

A :* dependency means “I can use anything that matches the specification, regardless of what was there at build time”. This is often the case for packages using non-compiled dynamic language libraries. Any dependency that is just an RDEPEND and not a DEPEND is implicitly of this type.

In the simple case, a := dependency means “I can be built against anything that matches the specification, but I need the same slot at runtime”. This is the case for most C-style linkage libraries that have a more or less stable API but not a stable ABI or shared object version (db is the canonical example), along with some sort-of-compiled dynamic languages that don’t maintain ABI stability.

The definition of := is slightly trickier if there are multiple slots already installed at build time. Since slots are in no way orderable, it is taken to mean “the slot of the best version that was installed at build time”. We considered letting ebuilds specify which slot they used, but real world use has shown that this is unnecessary and impractical — in reality, packages either just go for the best version of something automatically, or require complex special case handling.

Ebuild authors won’t see this, but anyone poking around in VDB may notice :=blah slot operator dependencies. This is how the package manager handles := dependencies — at install time, it rewrites the dependency specification to have the slot name straight after the equals.

We decided against having a form meaning “I need all the slots that were there at compile time”. Although at first glance this might sound useful, a closer look reveals that it doesn’t solve the whole ABI problem (it just makes it even messier), and we’ve yet to run across a legitimate use case for it that isn’t merely a horrible hack to work around lack of multi-ABI support.

It’s possible to combine either slot operator dependency form with normal dependencies just like EAPI 1 slot dependencies; >=cat/foo-1.3:= is fine. Slot operator dependencies do not make sense when combined with conventional slot dependencies.

Widespread use of slot operator dependencies will allow a package manager to know whether older slots of slotted packages are really required or not, rather than having to assume that they are for safety reasons, or assume that they aren’t and risk causing horrible breakage.

Slot operator dependencies were originally in exheres-0, and were used in kdebuild-1.

This is part of a series on EAPI 3.

EAPI 3 has a new phase function known as pkg_pretend. This can be used to carry out checks at pretend-install time, allowing many things that would otherwise abort in pkg_setup midway through the build process to be caught whilst the user is still at the terminal.

The default is to do nothing. Custom implementations might output warnings (which should be prefixed with the program name, to avoid confusion) or call die.

Care must be taken when writing pkg_pretend, since there is no guarantee that conditions that hold at that stage will also hold later on. Unless developers are absolutely certain they know what they are doing, all checks that aren’t merely checking USE flags of the active package should be repeated in pkg_setup later on — PMS deliberately doesn’t do much to restrict what could have changed between when pkg_pretend was run and when the main build starts, so it’s easiest to assume that pkg_pretend might not have been run at all.

For example, one might do something like this:

pkg_pretend() {
    if use foo && ! use bar ; then
        ewarn "${PF}: Support for foo requires bar enabled, but you have USE=\"foo -bar\"."
        ewarn "${PF}: foo support will not be enabled."
    fi

    check_foo_not_strict
}

pkg_setup() {
    check_foo_not_strict
}

check_foo_not_strict() {
    if grep --quiet '^STRICT' /etc/foo.conf 2>/dev/null ; then
        die "/etc/foo.conf sets foo to STRICT mode, which this build cannot support."
    fi
}

Note the check for foo not being strict being repeated. It’s possible that foo won’t be installed when pkg_pretend is run, but that it will have been installed and incorrectly configured by the time we reach our build.

Also note that there’s no environment saving for pkg_pretend. You can’t set variables in pkg_pretend and have them still around by pkg_setup. Nor are you allowed to modify the root filesystem or perform any action requiring privileges that would not be available in a normal phase.

pkg_pretend was originally an exheres-0 feature.

Paludis 0.36.1 has been released:

• Various Ruby 1.9 fixes.
• Hashes are now memoised, to speed up --pretend --install for certain cases.
• We now use pcrecpp rather than pcre++ to avoid the latter’s lack of thread sanity.

Gentoo is slowly shuffling towards switching from CVS to Git. This is a good thing, because CVS stinks. Using Git will reduce the amount of time developers need to waste to get something committed, make it easier to apply patches from third parties and make tree-wide changes merely a lot of work rather than practically impossible. What it will not do is make Gentoo in any way more ‘distributed’, ‘decentralised’ or ‘democratic’.

Some of the Git work has already been done, in a reduced manner (no history and no mirroring), by Daniel Robbins’ Funtoo, which is purported to be more distributed than Gentoo. The problem is, there’s nothing there to back up the distributed claim.

Distributed development, in the sense for which Git was designed (and ignoring the intervening BitKeeper stage), meant moving away from having a single central repository off of which everyone worked to having everyone work off their own, publishable repositories and providing easy ways of merging changes from one to another. ‘Good’ changes would tend to find their way from the authors up the food chain to the main repository whence official releases are made. Users requiring things that hadn’t made their way to the top would maintain their own repository, and merge in changes from elsewhere that they needed.

Typical Git Workflow Model

For a conventional codebase, this model works. But it’s not particularly nice, and it’s driven by necessity. You’ll note the big red dots in the diagrams. These represent places where people (assisted to some highly variable degree by Git) have to do merges. I chose big red dots rather than soft fluffy clouds because merges can be a lot of work (and because drawing clouds takes effort).

If you’ve got a conventional codebase, you have to do merges to make use of things from multiple sources — the compiler takes a single codebase and produces a program from it. You can do the same thing with a distribution. Funtoo, for example has had the Sunrise repository merged in to the main repository. Such a change would likely not be possible with Gentoo’s current CVS architecture.

It’s not entirely clear whether Funtoo intends to have users who want to use other overlays merge those overlays into their own tree. Doing so would be more Gitish.

Apparent Funtoo Workflow Model

But why bother? There’s no need to have a single codebase — there’s no compiler that has to take every input at once and turn it into a single monolithic product. Those big red dots are unnecessary.

A lot of fashionable programs are moving away from the big monolithic binary model and towards a plugin-assisted architecture. If you want Firefox to do a few things it doesn’t, you don’t hunt around for people who have already written them and then try to merge their source trees together. You install plugins. Only for more severe changes do you have to dive into the source, and the severity of change requiring a trip to the source is gradually increasing.

There’s a reason for this — whilst the merge model is a lot better than a single authoritative codebase and a bunch of patches, it’s a lot more work than providing limited composable extensibility at a higher level.

What, then, would a plugin-based model look like for a Gentoo-like distribution?

Presumably, one would have a centralised ‘main’ codebase. One could then add additional small extras to that main codebase to obtain new functionality (packages, in this case); these extras would rely upon parts of the main codebase and wouldn’t be able to operate on their own. Sound familiar? Yup, overlays are plugins.

This whole “merging overlays into the main tree” thing is starting to look like a step in the wrong direction. What would be some steps in a better direction?

One thing that comes instantly to mind is improving overlay handling. Portage’s overlay handling currently (at least in stable) looks like this:

Portage Overlay Model

Portage takes the main Gentoo repository, and then merges it with each overlay in turn, creating one ‘final’ overlay that ends up being used. I’ve used an orange dot here rather than a red one because it’s a different kind of merge. Rather than doing a source-level merge, the orange dot merge more or less (sort of) works like this:

• If there’s a package with the same name and version in the origin and the overlay we’re merging in, take the overlay version.
• If there’s an eclass with the same name and version in the origin and the overlay we’re merging in, sometimes take the overlay version.
• Do some horrid hackery to merge together any colliding profile things in an uncontrolled manner that doesn’t work for more than one merge.
• Pass everything else through.

Now, to be fair, the orange dot merge usually works. Most overlays don’t try to override eclasses, don’t have eclasses that conflict with each other an