== Testing the code ==

"make" here in the test directoy runs a complete set of regression
tests. It's what "make check" in the main directory redirects to.

If you get a failure on the repotool-initialize-cvs-git test, but
nowgere else, try updating your versions of CVS and cvs-fast-export
and trying again.

There are tools and makefile productions for finer-grained testing.  
Their behavior is controlled by these variables:

REPOSURGEON: defaults to 'reposurgeon '

TESTOPT: Use this to pass in a command or option setting, such
as "set experimental",

STOPOUT: 1 (the default) produces a bailout after the first failed
test.  Set it to 0 to continue after failed tests - you have to do
this on the make command line, e.g. "cd test; make STOPOUT=0
fi-regress".  Note, the output from this may be voluminous and
unuseful.

QUIET: 0 (the default) allows display of a diff against the checkfile
when a test fails. When QUIET=1 the duff is suppressed and te word
FAILED appears instead.  This is usedul wuth STOPOUT to see a summary
of success/fail status.

To run a single test:

   cd test; ./singletest foo

will check the actual result of the script in foo.tst against its expected
output in foo.chk.  No output other than the test notice is good.

The "liftcheck" script is useful for testing the conversion correctness
of a Subversion stream:

   cd test; ./liftcheck foo.svn
   
No output means that a compare-all showed no errors or warnings.

To allow the code to emit a stack trace, set anny log bit.

== Notes on the test machinery ==

A .fi extension means it's a git fast-import stream.  
A .svn extension means it's a Subversion repo dump.
A .chk extension means it's expected output from a test.
A .tst extension means it's a test driver script.

The "singletest" script takes the stem of a .tst file name, captures stdout
and stderr, and diffs the result against the corresponding check file.

The "singlelift" script takes the stem of a .svn file name, captures stdout
and stderr, and diffs the resultihg Subversion and git repos.

The "liftcheck" script takes a Subversion stream file, builds a Git
repository from it, and checks for differences using repotool's
compare-all function.

== The scripts ==

=== fi-to-fi ===

Builds a git repo from a fast-import stream on stdin, or streams an
existing repo to stdout.

=== hg-to-fi ===

Builds an hg repo from a fast-import stream on stdin, or streams an
existing repo to stdout.

=== svn-to-svn ===

Build a Subversion repo from a Subversion stream dump on stdin, or
dump an existing repo as a stream to stdout.  Can also be used to edit
a Subversion test load, regenerating its checksums.

=== svn-to-git ===

Build a Subversion repo from a Subversion stream dump on stdin,
convert it to a local git repository, and stream the result to
standard output. Mainly a wrapper around git-svn, to be used
for comparison tests.  Can also be used to test against agito
and svn2git.

To add a new tool for comparison, do something like:

   git submodule add GIT-URL/foobar externals/foobar
   git update submodules

from the repo root.

=== liftcheck ===

Takes a named Subversion dump and compare a checkout of it with
its git conversion at all tags and branches.

== The tests ==

Keeping live Subversion repos under version control doesn't work very well; 
in particular, things like the entries file change too often.  So what we 
do is keep Subversion dump files and rebuild the repo before each test.

In general, a file named foo.chk is the expected output from a test involving
either some operation test on an input stream in foo.fi or a repo conversion
from foo.svn.

To see summary lines from all tests, 'make testlist'.  

The Subversion dumpfiles have summary comments in them that abuse a loophole
in Subversion's parser - header lines that begin with whitespace are ignored.
When you create a new one, insert a second line that begins with a space and
two pound signs.

=== Making new Subversion test loads ===

1. To create a new Subversion test repo, 'make svn-flat' or 'make svn-branchy'.

2. To clone an existing test load foo.svn into the test repo,
'svn-to-svn -n <foo.svn'

3. To copy trunk, creating a tag named 'foo'

   svn copy file://${PWD}/test-repo/trunk file://${PWD}/test-repo/tags/foo

4. To delete a branch foo

   svn delete file://${PWD}/test-repo/branches/foo

5. 'make svn-dump' will dump the new test load to stdout.

== The agito test case ==

Samuel Howard had this to say:

git-svn's handling of tags is broken.

In this demonstration repository, a trunk directory of /trunk/proj exists,
containing some code.  This is improperly tagged by doing:

	svn cp trunk tags/proj-1.0

Where as what should have been done is this:

	svn cp trunk/proj tags/proj-1.0

This is significant because this is exactly what the CVS to SVN conversion
script (cvs2svn) does, to handle the fact that a CVS repository can contain
multiple modules.  Fixing a "mistake" like this is therefore necessary when
converting to SVN, to get tags stored properly.

In the SVN repository, this is fixed by deleting the branch and recreating
it properly (ie. the second command above).  To verify that this has been
done successfully, try this:

	svn log file://$PWD/myrepo/tags/proj-1.0

outputs:

	-----------------------------------------------------------------------
	r4 | fraggle | 2009-10-02 23:37:42 +0100 (Fri, 02 Oct 2009) | 2 lines

	Recreating the tag properly.

	-----------------------------------------------------------------------
	r1 | fraggle | 2009-10-02 23:36:41 +0100 (Fri, 02 Oct 2009) | 2 lines

	Initial import.

	-----------------------------------------------------------------------

Only the history of the directory being tagged and the commit that created the
tag are shown.  The "mistake" is kept in the history of /tags, but not in
the history of the tag itself.

The repository is then converted to git, using git-svn (see the shell script).
Two tags are created (proj-1.0@1 is the older, broken tag).  However, the
newer tag retains the history of the broken tag:

	git log tags/proj-1.0

outputs:

	Author: fraggle <fraggle@f01c4a58-e860-4891-ae86-76464917f484>
	Date:   Fri Oct 2 22:37:42 2009 +0000

	    Recreating the tag properly.

	commit 4aeb0a415e5be12d28a8af1128315e44d44a10d7
	Author: fraggle <fraggle@f01c4a58-e860-4891-ae86-76464917f484>
	Date:   Fri Oct 2 22:37:07 2009 +0000

	    Creating a tag in a BROKEN way, like how cvs2svn does it.

	commit 866f94c91de7628d7251098efcc133e6b5900f88
	Author: fraggle <fraggle@f01c4a58-e860-4891-ae86-76464917f484>
	Date:   Fri Oct 2 22:36:41 2009 +0000

	    Initial import.

	commit e8a2ee18774e319d33cb5bd418e03a5281b75268
	Author: fraggle <fraggle@f01c4a58-e860-4891-ae86-76464917f484>
	Date:   Fri Oct 2 22:36:41 2009 +0000

	    Initial import.

We now handle this case properly by renaming the earlier creation of
the tag.

== The tagretract test case ==

According to Mike Fleetwood, fleetwood.svn was created with the following
sequence of operations:

------------------------------------------------------
svn commit -m 'commit one'
svn copy $REPO/trunk $REPO/tags/1.0 -m 'Release 1.0'
svn mv $REPO/tags/1.0 $REPO/tags/1.0rc1 -m 'No release ready yet'
svn commit -m 'commit two'
svn copy $REPO/trunk $REPO/tags/1.0 -m 'Fixed release 1.0'
------------------------------------------------------

He then converted it with these commands:

------------------------------------------------------
branchify_map :tags/(.*)/:tags/\1:
read </tmp/repo.svndump
prefer git
write >/tmp/repo.fi
------------------------------------------------------

This sequence is captured in tagretract.tst.

Before the simplification of permission calculation:
After conversion the tag named '1.0' referred to the first commit with
the first tagging message 'Release 1.0', rather than the second commit
with the second tagging message 'Fixed release 1.0'.

== A note about references.svn ==

This was produced from the NUT Subversion repo as follows:

svncutter -q -r 0:4,6,8,56,58,61,65,350,355,356,543:546 select <nut.svn \
    | svncutter -q expunge 'tags/' \
        'branches/regex_branch' \
        'branches/Tripp_Lite_Omni' \
        'branches/r2.0.1' \
        'branches/v2.1.0' \
        'branches/Stable' \
        '.*/ChangeLog' \
        '.*/CHANGES' \
        '.*/README' \
        '.*/INSTALL' \
        '.*/UPGRADING' \
        '.*/NEWS' \
        '.*/CREDITS' \
        '.*/COPYING' \
        '.*/man/' \
        '.*/docs/' \
        '.*/.*txt' \
        '.*/.cvsignore' \
        '.*/MAINTAINERS' \
        '.*/configure' \
        '.*/config.sub' \
        '.*/config.guess' \
        '.*\.conf' \
        '.*mge.*' \
        '.*hid.*' \
        '.*libupsclient.*' \
        '.*missing.*' \
        '.*aclocal.*' \
        '.*/ltmain.sh' \
        '.*/Makefile.in' \
        '.*/Makefile.am' \
        '.*/stamp.h.in' \
        '.*/depcomp' \
        '.*\.h' \
        'trunk/drivers/megatec.c' \
        'trunk/drivers/optiups.c' \
        'trunk/drivers/tripplite-hid.c' \
        'branches/reportbuf/server/conf.c' \
        'branches/Development/drivers/apc-hid.c' \
        'branches/reportbuf/Makefile' \
        'branches/Testing/drivers/apc-hid.c' \
        'branches/Testing/drivers/tripplite_usb.c' \
        'trunk/drivers/al175.c' \
        'trunk/drivers/apc-hid.c' \
        'trunk/drivers/belkin-hid.c' \
        'branches/reportbuf/packaging/debian' \
        'branches/automake/packaging/debian' \
        'branches/JD-NewConf/common/data_types.c' \
        'branches/JD-NewConf/lib/libupsconfig.c' \
        'branches/JD-NewConf/utils/Makefile.in' \
        'branches/JD-NewConf/utils/migratetool.c' \
        'branches/JD-NewConf/utils/upsconfig.c' \
        'branches/JD-NewConf/clients/upsmon.c' \
        'branches/JD-NewConf/common/nutparser.c' \
        'branches/JD-NewConf/common/tree.c' \
        'branches/JD-NewConf/drivers/main.c' \
        'branches/JD-NewConf/drivers/upsdrvctl.c' \
        'branches/JD-NewConf/server/conf.c' \
        'branches/JD-NewConf/server/user.c' \
        'branches/JD-NewConf/utils/migrateconfig.c' \
        'branches/HAL/Makefile.in' \
        'branches/HAL/drivers' \
        'branches/HAL/utils' \
        'branches/JD-NewConf/conf' \
        'branches/JD-NewConf/drivers' \
        'branches/JD-NewConf/Makefile.in' \
        'branches/reportbuf/drivers' \
        'trunk/drivers/rhino.c' \
        'branches/Testing/drivers/rhino.c' \
        'branches/automake/AUTHORS' \
        'branches/reportbuf/scripts' \
        'trunk/scripts' \
        'trunk/clients' \
        'trunk/server' \
        'branches/Testing/scripts' \
        'branches/Testing/clients' \
        'branches/Testing/server' \
        'branches/Development/scripts' \
        'branches/Development/clients' \
        'branches/Development/server' \
        'branches/Development/drivers/tripplite_usb.c' \
        'branches/Development/drivers/dummy-ups.c' \
        'branches/automake/m4' \
        'branches/automake/clients' \
        'branches/automake/server' \
        'branches/automake/scripts' \
        'branches/Development/data' \
        'branches/automake/data' \
        'trunk/common' \
        'branches/automake/common' \
        'branches/Development/debian' \
        'trunk/packaging' \
        'branches/Development/packaging' \
        'branches/Testing/packaging' \
        'branches/automake/packaging' \
        'branches/automake/.*\.[ch]' \
        'trunk/drivers/belkin.c' \
        'trunk/drivers/belkinunv.c' \
        'trunk/drivers/bestfcom.c' \
        'trunk/drivers/bestuferrups.c' \
        'trunk/drivers/bestups.c' \
        'trunk/drivers/blazer.c' \
        'trunk/drivers/dstate.c' \
        'trunk/drivers/dummycons.c' \
        'trunk/drivers/esupssmart.c' \
        'trunk/drivers/everups.c' \
        'trunk/drivers/genericups.c' \
        'trunk/drivers/liebert.c' \
        'trunk/drivers/masterguard.c' \
        'trunk/drivers/metasys.c' \
        'trunk/drivers/mustek.c' \
        'trunk/drivers/oneac.c' \
        'trunk/drivers/powercom.c' \
        'trunk/drivers/safenet.c' \
        'trunk/drivers/skel.c' \
        'trunk/drivers/sms.c' \
        'trunk/drivers/tripplitesu.c' \
        'trunk/drivers/victronups.c' \
        '.*/apcsmart.c' \
        '.*/ippon.c' \
        '.*/isbmex.c' \
        '.*/powermust.c' \
        '.*/cyberpower.c' \
        '.*/solis.c' \
        '.*/fentonups.c' \
        '.*/tripplite.c' \
        'trunk/Makefile' \
        '.*/energizerups.c' \
        '.*/etapro.c' \
        '.*/upsdrvctl.c' \
        '.*/snmp-ups.c' \
        '.*/cpsups.c' \
        '.*/main.c' \
        '.*/upscode2.c' \
        '.*/bcmxcp.c' \
        'branches/automake/Makefile.dist' \
        '.*/gendb' \
        '.*/version' \
        '.*/install-sh' \
        '.*/upsd.users' \
    | svncutter -q renumber \
>reduced.svn

Later it was stripped and renumbered.  Later still, a 0 revision and
an 89 revision were added to make the revision sequence 0-origin
and continuous, which the Go implementation requires.

It captures the most serious pathologies in the NUT repo.

It's too complex for correctness to be audited in detail by eyeball,
but it makes a good stability test. Any change that breaks the
Subversion analyzer even subtly is likely to produce a diff on this
test.

== Troubleshooting incorrect conversions ==

Test loads that convert incorrectly arge tagged with "BUG:" in the 
list above.  These seem to reveal bugs in reposurgeon's interpretation
of dump stream semantics. It is a safe bet that these have to do with 
odd combinations of branch-copy operations.

Start by reading this:

https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt

Then read the section on working with Subversion in the manual.

Then read a simple dumpfile - like, say, samplebranch.svn - to see how
dump streams look in practice. Next run

-----------------------------------------------------------
repocutter -q see <samplebranch.svn
-----------------------------------------------------------

for a condensed version of the structure that leaves out the blobs.
The "M-N" at the left margin expands to "Revision M, node N"; copy 
is just add with a copy source.

You can use liftcheck to see exactly how a conversion goes wrong;
you'll get a diff. Try this on a member of the awkward squad, like
emptyfrom.svn.  By default liftcheck examines the head revision;
with the -r option you can check correctness at a specified 
esrtlier one.

You can use singlelift -o to dump a fast import stream made
from a named Subversion dump.  In this more you can also use
-l to set reposurgeon log flags.

The objective is to fix reposurgeon so that all those diffs go away.

== Sporadic tests ==

We have some scripts intended to check Subversion's behavior.

svncheck0.sh:
   General test of dump stream operations.

svncheck1.sh:
   Simple check that a file copy propagates the executable bit.

svncheck2.sh:
   A directory copy propagates it too.

svncheck3.sh
   Variant of previous test with different op order.

== Tuning for speed ==

The goal of moving the code to Golang is to improve conversion
performance on large repositories by an order of magnitude or more,
with the horrible example being the GCC subversion history.  Trial
runs with the Python version were taking 9-10 hours!

If you are already a Go expert, you can help by tuning for speed.  The
most important single operations to speed up are fast-import stream
reads and Subversion dump stream reads.

First thing to do is make a test load.  The reposurgeon history itself is
large enough to be a useful one.  So:

  $ reposurgeon "read ." "write >rs.fi"

The ability to dump profile data is built into reposurgeon itself:

  $ reposurgeon "verbose 1" "profile start all reposurgeon" "read <rs.fi"

Once you have the profile data you can sic the profile viewer on it.
Have graphviz installed and do

  $ go tool pprof -http=":" ./reposurgeon reposurgeon.cpu.prof

There are lots of ways to explore the data but the single most
interesting one to start with is the graph view. The size of each box
is proportional to the number of profiler samples it appears in, and
the arrows are sized in proportion to the time spent calling them.
The 'top' view gives you the same data in tabular form:

     Flat  Flat%    Sum%      Cum    Cum%  Name
  203.36s  9.51%   9.51%  224.91s  10.52%  runtime.findObject
  132.43s  6.20%  15.71%  197.87s   9.26%  syscall.Syscall
   99.48s  4.65%  20.36%  107.34s   5.02%  syscall.Syscall6
   93.39s  4.37%  24.73%  395.22s  18.49%  runtime.mapassign_faststr
   84.39s  3.95%  28.68%  346.39s  16.20%  runtime.scanobject

This is telling us that (a) disk I/O (syscall.Syscall,
syscall.Syscall6) is slow, (b) assigning things to maps is a little
slower still (runtiem.mapassign_faststr), and (c) that garbage
collection is worse (runtime.scanobject, runtime.findObject).

The relative balance of these things does depend a lot on your
hardware. Most of the syscalls are for reading or writing files, so if
your disks are slower then that will be higher in the list. Reading
and/or writing less data would help (though it might be
impractical). Exercising the allocator less will be (and has been) a
good source of improvements. In a way these are good news - it
suggests we don't have a big-O/algorithmic problem. On the other hand,
it could just mean that they will only show up on larger repositories.

The obvious thing to do first is a search-and-destroy for heap escapes.
We can't avoid doing a lot of allocation; what we can do is avoid creating
lots of short-lived heap objects that will churn heap storage and trigger GC.

You can view the allocation profile with this command:

  $ go tool pprof -http=":" -sample_index=alloc_space ./reposurgeon reposurgeon.cpu.prof

This shows all allocations over the entire run of the program, not
just what was still live at a particular point in time. Using the
option -sample_index=alloc_space instead will show the count of all
objects allocated instead of the space that they occupied.

Finally, a trace of the execution is also recorded. This tracks which
threads are started, and which goroutines are running on them, at a
very high resolution. This trace is particularly useful for those
parts of Reposurgeon which are parallelized, as it is possible to see
where cpus are left idle. View the trace with this command:

  $ go tool trace reposurgeon.trace.prof

It will automatically open a page in your default web browser, but
unfortunately the trace viewer itself only works in Chrome or
Chromium; you'll need to open it there yourself if you prefer a
different browser.

In addition to the trace, this also provides a list of tasks, regions,
and associated log messages. Reposurgeon does not use this logging
capability very extensively, but it is used enough that you can see a
timeline of what Reposurgeon was doing:

  When                 Elapsed  Goroutine ID  Events
  0.000000000s  1m7.920970625s                Task 1 (incomplete)
  0.000125812      .                       1  region logfile started (duration: 15.964µs)
  0.000220496      .    94684              1  region readlimit started (duration: 3.431µs)
  0.000239855      .    19359              1  region set started (duration: 4.551µs)
  0.000281811      .    41956              1  region script started (duration: 260.3µs)
  0.000425845      .   144034              1  region branchify started (duration: 11.982µs)
  0.000472494      .    46649              1  region branchmap started (duration: 48.515µs)
  0.000557453      .    84959              1  region read started (duration: 33.919414851s)
  33.958593745   33.958036292              1  region authors started (duration: 3.646869ms)

Reposurgeon surrounds each command executed by the user with a region,
and each region adds a log entry; the duration of the region is also
calcuated. The 'elapsed' column shows the time in nanoseconds that has
elapsed since the previous log message. These regions can be nested,
although this is not shown very clearly. In particular, the 'script'
regions will contain regions of all the commands that were in the
script that was run.

It may be useful to add additional logging of this type. The Go trace
library is used to add new regions to the trace file. Its first
argument is a Context object from the Go context library, which needs
to be passed down the call tree. The Context objects can have metadata
associated with them which is supposed to show up in the trace viewer,
but I haven't yet explored this feature.

=== Some references ===

https://blog.golang.org/profiling-go-programs

https://artem.krylysov.com/blog/2017/03/13/profiling-and-optimizing-go-web-applications/

https://github.com/google/pprof/blob/master/doc/README.md

https://www.signalfx.com/blog/a-pattern-for-optimizing-go-2/

http://www.agardner.me/golang/garbage/collection/gc/escape/analysis/2015/10/18/go-escape-analysis.html

https://github.com/golang/go/wiki/Performance

https://groups.google.com/forum/#!msg/golang-nuts/pxfhKGqHNv0/If4Gz09r_2gJ

// end
