release_managers_guide - Releasing a new version of perl 5.x
As of August 2009, this file is mostly complete, although it is missing some detail on doing a major release (e.g. 5.10.0 -> 5.12.0). Note that things change at each release, so there may be new things not covered here, or tools may need updating.
This document describes the series of tasks required - some automatic, some manual - to produce a perl release of some description, be that a snaphot, release candidate, or final, numbered release of maint or blead.
The release process has traditionally been executed by the current pumpking. Blead releases from 5.11.0 forward are made each month on the 20th by a non-pumpking release engineer. The release engineer roster and schedule can be found in Porting/release_schedule.pod.
This document both helps as a check-list for the release engineer and is a base for ideas on how the various tasks could be automated or distributed.
The outline of a typical release cycle is as follows:
(5.10.1 is released, and post-release actions have been done)
...time passes...
an occasional snapshot is released, that still identifies itself as
5.10.1
...time passes...
a few weeks before the release, a number of steps are performed,
including bumping the version to 5.10.2
...a few weeks passes...
perl-5.10.2-RC1 is released
perl-5.10.2 is released
post-release actions are performed, including creating new
perl5103delta.pod
... the cycle continues ...
Some of the tasks described below apply to all four types of release of Perl. (snapshot, RC, final release of maint, final release of blead). Some of these tasks apply only to a subset of these release types. If a step does not apply to a given type of release, you will see a notation to that effect at the beginning of the step.
A snapshot is intended to encourage in-depth testing from time-to-time, for example after a key point in the stabilisation of a branch. It requires fewer steps than a full release, and the version number of perl in the tarball will usually be the same as that of the previous release.
A release candidate is an attempt to produce a tarball that is a close as possible to the final release. Indeed, unless critical faults are found during the RC testing, the final release will be identical to the RC barring a few minor fixups (updating the release date in perlhist.pod, removing the RC status from patchlevel.h, etc). If faults are found, then the fixes should be put into a new release candidate, never directly into a final release.
At this point you should have a working release candidate with few or no changes since.
It's essentially the same procedure as for making a release candidate, but with a whole bunch of extra post-release steps.
It's essentially the same procedure as for making a release candidate, but with a whole bunch of extra post-release steps.
Before you can make an official release of perl, there are a few hoops you need to jump through:
SKIP this step for SNAPSHOT
Make sure you have a PAUSE account suitable for uploading a perl release. If you don't have a PAUSE account, then request one:
https://pause.perl.org/pause/query?ACTION=request_id
Check that your account is allowed to upload perl distros: goto https://pause.perl.org/, login, then select 'upload file to CPAN'; there should be a "For pumpkings only: Send a CC" tickbox. If not, ask Andreas König to add your ID to the list of people allowed to upload something called perl. You can find Andreas' email address at:
https://pause.perl.org/pause/query?ACTION=pause_04imprint
Make sure that search.cpan.org knows that you're allowed to upload perl distros. Contact Graham Barr to make sure that you're on the right list.
Some release engineering steps require a full mirror of the CPAN. Work to fall back to using a remote mirror via HTTP is incomplete but ongoing. (No, a minicpan mirror is not sufficient)
You will need a working git installation, checkout of the perl git repository and perl commit bit. For information about working with perl and git, see pod/perlrepository.pod.
If you are not yet a perl committer, you won't be able to make a release. Have a chat with whichever evil perl porter tried to talk you into the idea in the first place to figure out the best way to resolve the issue.
SKIP this step for SNAPSHOT and RC
For a numbered blead or maint release of perl, you will need a quotation to use as an epigraph to your release announcement. (There's no harm in having one for a snapshot, but it's not required).
The work of building a release candidate for a numbered release of perl generally starts several weeks before the first release candidate. Some of the following steps should be done regularly, but all must be done in the run up to a release.
Ensure that dual-life CPAN modules are synchronised with CPAN. Basically, run the following:
$ ./perl -Ilib Porting/core-cpan-diff -a -o /tmp/corediffs
to see any inconsistencies between the core and CPAN versions of distros, then fix the core, or cajole CPAN authors as appropriate. See also the -d and -v options for more detail. You'll probably want to use the -c cachedir option to avoid repeated CPAN downloads.
To see which core distro versions differ from the current CPAN versions:
$ ./perl -Ilib Porting/core-cpan-diff -x -a
If you are making a maint release, run core-cpan-diff on both blead and maint, then diff the two outputs. Compare this with what you expect, and if necessary, fix things up. For example, you might think that both blead and maint are synchronised with a particular CPAN module, but one might have some extra changes.
Ensure dual-life CPAN modules are stable, which comes down to:
for each module that fails its regression tests on $current
did it fail identically on $previous?
if yes, "SEP" (Somebody Else's Problem)
else work out why it failed (a bisect is useful for this)
attempt to group failure causes
for each failure cause
is that a regression?
if yes, figure out how to fix it
(more code? revert the code that broke it)
else
(presumably) it's relying on something un-or-under-documented
should the existing behaviour stay?
yes - goto "regression"
no - note it in perldelta as a significant bugfix
(also, try to inform the module's author)
Similarly, monitor the smoking of core tests, and try to fix. See http://doc.procura.nl/smoke/index.html for a summary.
Similarly, monitor the smoking of perl for compiler warnings, and try to fix.
Run Porting/cmpVERSION.pl to compare the current source tree with the previous version to check for for modules that have identical version numbers but different contents, e.g.:
$ cd ~/some-perl-root
$ ./perl -Ilib Porting/cmpVERSION.pl -xd . v5.10.0
then bump the version numbers of any non-dual-life modules that have changed since the previous release, but which still have the old version number. If there is more than one maintenance branch (e.g. 5.8.x, 5.10.x), then compare against both.
Note that some of the files listed may be generated (e.g. copied from ext/ to lib/, or a script like lib/lib_pm.PL is run to produce lib/lib.pm); make sure you edit the correct file!
Once all version numbers have been bumped, re-run the checks.
Then run again without the -x option, to check that dual-life modules are also sensible.
$ ./perl -Ilib Porting/cmpVERSION.pl -d . v5.10.0
Get perldelta in a mostly finished state.
Read Porting/how_to_write_a_perldelta.pod, and try to make sure that every section it lists is, if necessary, populated and complete. Copy edit the whole document.
A week or two before the first release candidate, bump the perl version number (e.g. from 5.10.0 to 5.10.1), to allow sufficient time for testing and smoking with the target version built into the perl executable. For subsequent release candidates and the final release, it it not necessary to bump the version further.
There is a tool to semi-automate this process. It works in two stages. First, it generates a list of suggested changes, which you review and edit; then you feed this list back and it applies the edits. So, first scan the source directory looking for likely candidates. The command line arguments are the old and new version numbers, and -s means scan:
$ Porting/bump-perl-version -s 5.10.0 5.10.1 > /tmp/scan
This produces a file containing a list of suggested edits, e.g.:
NetWare/Makefile
89: -MODULE_DESC = "Perl 5.10.0 for NetWare"
+MODULE_DESC = "Perl 5.10.1 for NetWare"
i.e. in the file NetWare/Makefile, line 89 would be changed as shown. Review the file carefully, and delete any -/+ line pairs that you don't want changing. You can also edit just the + line to change the suggested replacement text. Remember that this tool is largely just grepping for '5.10.0' or whatever, so it will generate false positives. Be careful not change text like "this was fixed in 5.10.0"! Then run:
$ Porting/bump-perl-version -u < /tmp/scan
which will update all the files shown.
Be particularly careful with INSTALL, which contains a mixture of 5.10.0-type strings, some of which need bumping on every release, and some of which need to be left unchanged. Also note that this tool currently only detects a single substitution per line: so in particular, this line in README.vms needs special handling:
rename perl-5^.10^.1.dir perl-5_10_1.dir
Commit your changes:
$ git st
$ git diff
B<review the delta carefully>
$ git commit -a -m 'Bump the perl version in various places for 5.x.y'
When the version number is bumped, you should also update Module::CoreList (as described below in "Building a release - on the day") to reflect the new version number.
Review and update INSTALL to account for the change in version number; in particular, the "Coexistence with earlier versions of perl 5" section.
Update the Changes file to contain the git log command which would show all the changes in this release. You will need assume the existence of a not-yet created tag for the forthcoming release; e.g.
git log ... perl-5.10.0..perl-5.12.0
Due to warts in the perforce-to-git migration, some branches require extra exclusions to avoid other branches being pulled in. Make sure you have the correct incantation: replace the not-yet-created tag with HEAD and see if git log produces roughly the right number of commits across roughly the right time period (you may find git log --pretty=oneline | wc useful).
$ sh Configure -Dprefix=/tmp/perl-5.x.y -Uinstallusrbinperl \
-Duseshrplib -Dd_dosuid
$ make
$ LD_LIBRARY_PATH=`pwd` make test # or similar for useshrplib
$ make suidperl
$ su -c 'make install'
$ ls -l .../bin/sperl
-rws--x--x 1 root root 69974 2009-08-22 21:55 .../bin/sperl
(Then delete the installation directory.)
XXX think of other configurations that need testing.
perlport has a section currently named Supported Platforms that indicates which platforms are known to build in the current release. If necessary update the list and the indicated version number.
This section describes the actions required to make a release (or snapshot etc) that are performed on the actual day.
Re-read the perldelta to try to find any embarrassing typos and thinkos; remove any TODO or XXX flags; update the "Known Problems" section with any serious issues for which fixes are not going to happen now; and run through pod and spell checkers, e.g.
$ podchecker -warnings -warnings pod/perl5101delta.pod
$ spell pod/perl5101delta.pod
Also, you may want to generate and view an HTML version of it to check formatting, e.g.
$ perl pod/pod2html pod/perl5101delta.pod > /tmp/perl5101delta.html
$ git status
$ ./Configure -Dusedevel -des && make
$ git status
$ make regen_all
$ make regen_perly
$ git status
If any of the files managed by regen.pl have changed, then you should re-make perl to check that it's okay, then commit the updated versions:
$ git commit -a -m 'make regen; make regen_perly'
Update Module::Corelist with module version data for the new release.
Note that if this is a maint release, you should run the following actions from the maint directory, but commit the Corelist.pm changes in blead and subsequently cherry-pick it.
corelist.pl uses ftp.funet.fi to verify information about dual-lived modules on CPAN. It can use a full, local CPAN mirror or fall back to wget or curl to fetch only package metadata remotely. (If you're on Win32, then installing Cygwin is one way to have commands like wget and curl available.)
(If you'd prefer to have a full CPAN mirror, see http://www.cpan.org/misc/cpan-faq.html#How_mirror_CPAN)
Then change to your perl checkout, and if necessary,
$ make perl
If this not the first update for this version (e.g. if it was updated when the version number was originally bumped), first edit dist/Module-CoreList/lib/Module/CoreList.pm to delete the existing entries for this version from the %released and %version hashes: they will have a key like 5.010001 for 5.10.1.
XXX the edit-in-place functionality of Porting/corelist.pl should be fixed to handle this automatically.
Then, If you have a local CPAN mirror, run:
$ ./perl -Ilib Porting/corelist.pl ~/my-cpan-mirror
Otherwise, run:
$ ./perl -Ilib Porting/corelist.pl cpan
This will chug for a while, possibly reporting various warnings about badly-indexed CPAN modules unrelated to the modules actually in core. Assuming all goes well, it will update dist/Module-CoreList/lib/Module/CoreList.pm.
Check that file over carefully:
$ git diff dist/Module-CoreList/lib/Module/CoreList.pm
If necessary, bump $VERSION (there's no need to do this for every RC; in RC1, bump the version to a new clean number that will appear in the final release, and leave as-is for the later RCs and final).
Edit the version number in the new 'Module::CoreList' => 'X.YZ' entry, as that is likely to reflect the previous version number.
Also edit Module::CoreList's new version number in its Changes file and in its META.yml file.
In addition, if this is a final release (rather than a release candidate):
%released hash with today's date.CAVEATS sectionFinally, commit the new version of Module::CoreList: (unless this is for maint; in which case commit it blead first, then cherry-pick it back).
$ git commit -m 'Update Module::CoreList for 5.x.y' dist/Module-CoreList/lib/Module/CoreList.pm
$ make manisort
$ make distclean
$ git clean -xdf # This shouldn't be necessary if distclean is correct
$ perl Porting/manicheck
$ git status
XXX manifest _sorting_ is now checked with make test_porting
Commit MANIFEST if it has changed:
$ git commit -m 'Update MANIFEST' MANIFEST
Add an entry to pod/perlhist.pod with the current date, e.g.:
David 5.10.1-RC1 2009-Aug-06
Make sure that the correct pumpking is listed in the left-hand column, and if this is the first release under the stewardship of a new pumpking, make sure that his or her name is listed in the section entitled THE KEEPERS OF THE PUMPKIN.
Be sure to commit your changes:
$ git commit -m 'add new release to perlhist' pod/perlhist.pod
Update patchlevel.h to add a -RC1-or-whatever string; or, if this is a final release, remove it. For example:
static const char * const local_patches[] = {
NULL
+ ,"RC1"
PERL_GIT_UNPUSHED_COMMITS /* do not remove this line */
Be sure to commit your change:
$ git commit -m 'bump version to RCnnn' patchlevel.h
$ git clean -xdf
$ ./Configure -des -Dprefix=/tmp/perl-5.x.y-pretest
# or if it's an odd-numbered version:
$ ./Configure -des -Dusedevel -Dprefix=/tmp/perl-5.x.y-pretest
$ make test install
/tmp/perl-5.x.y-pretest/bin/perl -v and /tmp/perl-5.x.y-pretest/bin/perl -V are as expected, especially as regards version numbers, patch and/or RC levels, and @INC paths. Note that as they have been been built from a git working directory, they will still identify themselves using git tags and commits.
Then delete the temporary installation.
$ git push origin ....
Tag the release (e.g.):
$ git tag v5.11.0 -m'First release of the v5.11 series!'
(Adjust the syntax appropriately if you're working on Win32, i.e. use -m "..." rather than -m'...'.)
It is VERY important that from this point forward, you not push your git changes to the Perl master repository. If anything goes wrong before you publish your newly-created tag, you can delete and recreate it. Once you push your tag, we're stuck with it and you'll need to use a new version number for your release.
-s option to specify a suitable suffix for the tarball and directory name:
$ cd root/of/perl/tree
$ make distclean
$ git clean -xdf # make sure perl and git agree on files
$ git status # and there's nothing lying around
$ perl Porting/makerel -b -s `git describe` # for a snapshot
$ perl Porting/makerel -b -s RC1 # for a release candidate
$ perl Porting/makerel -b # for a final release
This creates the directory ../perl-x.y.z-RC1 or similar, copies all the MANIFEST files into it, sets the correct permissions on them, adds DOS line endings to some, then tars it up as ../perl-x.y.z-RC1.tar.gz. With -b, it also creates a tar.bz2 file.
If you're getting your tarball suffixed with -uncommitted and you're sure your changes were all committed, you can override the suffix with:
$ perl Porting/makerel -b -s ''
XXX if we go for extra tags and branches stuff, then add the extra details here
$ rm -rf ../perl-x.y.z-RC1
$ ./Configure -des && make all test
$ make distclean
$ ./Configure -des -Dprefix=/install/path && make all test_harness install
$ cd /install/path
perl -v and perl -V are as expected, especially as regards version numbers, patch and/or RC levels, and @INC paths.
Note that the results may be different without a .git/ directory, which is why you should test from the tarball.
$ bin/perlivp
...
All tests successful.
$
cd installdir-5.10.0/
find . -type f | perl -pe's/5\.10\.0/5.10.1/g' | sort > /tmp/f1
cd installdir-5.10.1/
find . -type f | sort > /tmp/f2
diff -u /tmp/f[12]
$ bin/perl -MCPAN -e'shell'
(Use ... -e "shell" instead on Win32. You probably also need a set of Unix command-line tools available for CPAN to function correctly without Perl alternatives like LWP installed. Cygwin is an obvious choice.)
CPAN> install Inline
CPAN> quit
Check that your perl can run this:
$ bin/perl -lwe 'use Inline C => "int f() { return 42;} "; print f'
42
$
(Use ... -lwe "use ..." instead on Win32.)
$ bin/cpanp
(Again, on Win32 you'll need something like Cygwin installed, but make sure that you don't end up with its various bin/cpan* programs being found on the PATH before those of the Perl that you're trying to test.)
CPAN Terminal> i DBI
CPAN Terminal> quit
$ bin/perl -MDBI -e 1
$
perlbug utility works. Try the following:
$ bin/perlbug
...
Subject: test bug report
Local perl administrator [yourself]:
Editor [vi]:
Module:
Category [core]:
Severity [low]:
(edit report)
Action (Send/Display/Edit/Subject/Save to File): f
Name of file to save message in [perlbug.rep]:
Action (Send/Display/Edit/Subject/Save to File): q
and carefully examine the output (in perlbug.rep]), especially the "Locally applied patches" section. If everything appears okay, then delete the file, and try it again, this time actually submitting the bug report. Check that it shows up, then remember to close it!
Then check that the smoke tests pass (particularly on Win32). If not, go back and fix things.
https://pause.perl.org/
(Login, then select 'Upload a file to CPAN')
If your workstation is not connected to a high-bandwidth, high-reliability connection to the Internet, you should probably use the "GET URL" feature (rather than "HTTP UPLOAD") to have PAUSE retrieve the new release from wherever you put it for testers to find it. This will eliminate anxious gnashing of teeth while you wait to see if your 15 megabyte HTTP upload successfully completes across your slow, twitchy cable modem. You can make use of your home directory on dromedary for this purpose: http://users.perl5.git.perl.org/~USERNAME maps to /home/USERNAME/public_html, where USERNAME is your login account on dromedary. Remember: if your upload is partially successful, you may need to contact a PAUSE administrator or even bump the version of perl.
Upload both the .gz and .bz2 versions of the tarball.
Wait until you receive notification emails from the PAUSE indexer confirming that your uploads have been successfully indexed. Do not proceed any further until you are sure that the indexing of your uploads has been successful.
$ git push origin tag v5.11.0
static const char * const local_patches[] = {
NULL
- ,"RC1"
PERL_GIT_UNPUSHED_COMMITS /* do not remove this line */
Be sure to commit your change:
$ git commit -m 'disarm RCnnn bump' patchlevel.h
$ git push origin ....
Ask Jarkko to update the descriptions of which tarballs are current in http://www.cpan.org/src/README.html, and Rafael to update http://dev.perl.org/perl5/
Remind the current maintainer of Module::CoreList to push a new release to CPAN.
Bump the perlXYZdelta version number.
First, create a new empty perlNNNdelta.pod file for the current release + 1; see Porting/how_to_write_a_perldelta.pod.
You should be able to do this by just copying in a skeleton template and then doing a quick fix up of the version numbers, e.g.
Then commit this file. (If this new perldelta needs to get cherrypicked anywhere, this provides a clean base to merge from.)
For example, assuming 5.10.2:
$ cp -i Porting/perldelta_template.pod pod/perl5102delta.pod
$ (edit it)
$ git add pod/perl5102delta.pod
$ git commit -m 'create perl5102delta'
Edit the previous delta file to change the NAME from perldelta to perlNNNdelta and commit it. For example:
$ git add pod/perl5101delta.pod
$ git commit -m 'retitle perl5101delta'
Now you need to update various tables of contents, most of which can be generated automatically.
Edit pod.lst: add the new entry, flagged as 'D', and unflag the previous entry from being 'D'; for example:
-D perl5101delta Perl changes in version 5.10.1
+D perl5102delta Perl changes in version 5.10.2
+ perl5101delta Perl changes in version 5.10.1
Run perl pod/buildtoc --build-all to update the perldelta version in the following files:
MANIFEST
Makefile.SH
pod.lst
pod/perl.pod
vms/descrip_mms.template
win32/Makefile
win32/makefile.mk
win32/pod.mak
Then manually edit (vms/descrip_mms.template to bump the version in the following entry:
[.pod]perldelta.pod : [.pod]perl5101delta.pod
XXX this previous step needs to fixed to automate it in pod/buildtoc.
Manually update references to the perlNNNdelta version in these files:
INSTALL
README
These two lists of files probably aren't exhaustive; do a recursive grep on the previous filename to look for suitable candidates that may have been missed.
Finally, commit:
$ git commit -a -m 'update TOC and perldelta references'
At this point you may want to compare the commit with a previous bump to see if they look similar. See commit ca8de22071 for an example of a previous version bump.
If this was a maint release, then edit Porting/mergelog to change all the d (deferred) flags to . (needs review).
If this was the first release of a new maint series, (5.x.0 where x is even), then create a new maint branch based on the commit tagged as the current release and bump the version in the blead branch in git, e.g. 5.12.0 to 5.13.0.
[ XXX probably lots more stuff to do, including perldelta, lib/feature.pm ]
Assuming you're using git 1.7.x or newer:
$ git checkout -b maint-5.12
$ git push origin -u maint-5.12
Copy the perlNNNdelta.pod for this release into the other branches; for example:
$ cp -i ../5.10.x/pod/perl5101delta.pod pod/ # for example
$ git add pod/perl5101delta.pod
Edit pod.lst to add an entry for the file, e.g.:
perl5101delta Perl changes in version 5.10.1
Then rebuild various files:
$ perl pod/buildtoc --build-all
Finally, commit:
$ git commit -a -m 'add perlXXXdelta'
5.8.9-RC1 2008-Nov-10
5.8.9-RC2 2008-Dec-06
5.8.9 2008-Dec-14
perlbug-admin at perl.org requesting that new version numbers be added to the RT fields Perl Version and Fixed In.Thanks for releasing perl!
Based on http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-05/msg00608.html, plus a whole bunch of other sources, including private correspondence.
