Sparse Directories, Now With Exclusion

“cmpilato ❤ sparse directories”

If I had a dollar for every time I’ve typed that… well, you and I could at least spring for some Fazoli’s fast-food Italian. Okay, I admit that the emotion doesn’t always drive me to public expression, but that in no way diminishes my fondness for this feature.

Introduced in Subversion 1.5, sparse directory support is one of a few features from that release (besides merge tracking and foreign repository merges) that I’ve fully integrated into my day-to-day activities. I dig organization. I tend to keep a pretty neat home directory. But I routinely work on several different pieces of software, and at any given time, I’m tracking several different development branches in each of those pieces of software. Were I not using sparse directories, my “projects” directory would look something like this:

$ ls ~/projects
subversion/		      svnbook/		 viewvc-1.0.7/
subversion-1.5.x/	      thotkeeper/	 viewvc-1.0.x/
subversion-1.6.x/	      thotkeeper-0.3.0/  viewvc-1.1.0-beta1/
subversion-http-protocol-v2/  viewvc/		 viewvc-1.1.x/
$

On the positive side of things, I could quickly update all my working copies by simply running svn update ~/projects/*.

But those of you who have command-line completion as hard-wired into your habits as I do will immediately notice that so many common directory prefixes does a useless completion environment make. And not using common prefixes? Well that’s just barbaric.

Fortunately, sparse directories has given me a whole new perspective on working copy organization. Now, my projects directory contains (gasp!) only projects, and looks like this:

$ ls ~/projects
subversion/  svnbook/  thotkeeper/  viewvc/
$

Those directories are sparse checkouts of the root directories of their respective project repositories. Beneath them are (at least) “trunk” directories, probably “branch” and some of its children, and maybe even “tags” with some of its children in certain cases. svn up ~/projects/* still works, and my working copy topology matches that of the repositories.

I won’t go into the details of how sparse checkouts works here — I’ve already documented the Subversion 1.5 implementation of them in the second edition of Version Control With Subversion (which you can read at http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html). The point of this blog post is to tell you about how Subversion 1.6 improves this feature in a key way.

In Subversion 1.6 (slated for release Any Day Now), the --set-depth parameter to svn update has grown a new value — exclude. This value tells Subversion to exclude the target from the working copy, immediately and until further notice. Prior to Subversion 1.6, if a branch I was working on was no longer of interest to me, I couldn’t easily remove it from my working copy. If I simply deleted it, it would return the next time I updated the working copy. If I svn delete‘d it, the branch remained as a local modification forever. (Unless, of course, I accidentally committed it, which brought a whole different sort of trouble to my doorstep. Angry peers. Pitchforks and torches. It was a mess — you don’t want to go there.) The new exclusion mechanism in Subversion 1.6 is the Right Way To Do It.

Say I no longer care about what’s going on some directory of one my project working copies. Maybe I don’t care about the Subversion project’s website any more. Well, with this new exclusion feature, I can tell Subversion to remove that directory:

$ cd ~/projects/subversion/trunk
$ svn update --set-depth=exclude www
D         www
$ ls www
ls: cannot access www: No such file or directory
$

Done deal. When I update my working copy in the future, I will not receive any changes aimed at that www directory. If I later decide that I once again care about that directory, I can “resubscribe” to it again:

$ svn update --set-depth=infinity www
A    www
A    www/links.html
A    www/testing-goals.html
…
A    www/tigris-permissions.html
A    www/webdav-usage.html
Updated to revision 36292.
$

Note that if you exclude a versioned directory that has some unversioned files in it, or some files with local modifications, Subversion handles this situation gracefully. All the files that aren’t safe to delete, Subversion leaves around, and of course leaves any intermediate directories required to reach those files, too.

I hope this enhancement serves you as well as it has served me. What do you think? How are you using sparse directories to better organize your life?

C. Michael Pilato

C. Michael Pilato is a core Subversion developer, co-author of Version Control With Subversion (O'Reilly Media), and the primary maintainer of ViewVC. He works remotely from his home state of North Carolina as a software engineer for CollabNet, and has been an active open source developer since early 2001. Mike is a proud husband and father who loves traveling, soccer, spending quality time with his family, and any combination of those things. He also enjoys composing and performing music, and harbors not-so-secret fantasies of rock stardom. Mike has a degree in computer science and mathematics from the University of North Carolina at Charlotte.

Posted in Subversion
23 comments on “Sparse Directories, Now With Exclusion
  1. Nice feature!
    What is the syntax to checkout with several excludes so you don’t have to download unwanted subdirectories in the first place?

  2. David Aldrich says:

    Thanks, that’s a great explanation and a very useful feature. Will it work with a 1.6 client and 1.5 server?

  3. C. Michael Pilato says:

    @René: Checking out without unwanted items in the first place is a process more than any particular syntax. :-) The gist of it is that you build out the tree of things you want, using extremely shallow depths for intermediate placeholder directories and such. There’s a bit of an example of this at the end of the svnbook section I referred to: http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html
    @David: Yep. Works great. In fact, I’m pretty sure that it works with a 1.6 client and *any* server version.

  4. Does the new syntax only allow you to exclude something, or does it also allow you to reduce the depth? In other words, could have allowed the “www” folder to remain in your WC but now be empty? As if you had checked out its parent with –depth=immediates ?
    Mark

  5. To answer my own question, having just tried it, Yes! This is pretty sweet. So now if you no longer need a folder you can reduce it’s depth and then put it back again later.
    Mark

  6. C. Michael Pilato says:

    @Mark: Exclusion *is* depth reduction, of course. But yes, you can do arbitrary depth reduction.

  7. Graham Nash says:

    Completely off topic but cannot seem to find any information on this… Any word on whether or not #3128 will be fixed in 1.6?
    http://subversion.tigris.org/issues/show_bug.cgi?id=3128

  8. Ben says:

    Adding the ability to exclude directories now, put simply, tops off the wonderful sparse checkouts feature.
    Before, I had to be extra careful when doing a sparse checkout; our repository layout isn’t changing anytime soon and a full checkout is currently about 1GB since we (for better or worse) have some large binary data mixed in with our source. A checkout of source code only is about 3.4MB so a sparse checkout is really a nice thing.
    Not that this is the place, but I always wonder if there is a way to do an overlay – that is have items (files) from different repositories in the same directory – so that source code can be kept in one repo, and designated (large) binary files are somehow kept in another repository or other storage method. Rsync is an option I guess, has anyone else had this problem?

  9. Kevin says:

    Excellent! I’ve wanted something like this for a long time!
    Now, if svn could support a “p4 client view” like interface for adding/removing/remapping the client files… :)
    [1] http://www.perforce.com/perforce/doc.current/manuals/cmdref/o.views.html#1040759

  10. @Graham. I just updated issue 3128. IMO, it was fixed in SVN 1.5.5

  11. Jack Bates says:

    We manage several domains, hosted on at least two servers, in one Subversion repository. Before sparse directories, we either had to checkout all domains on each server, or checkout each domain to a separate working copy.
    Now we use sparse directories to checkout to one working copy on each server, only the domains which are hosted on that server.

  12. KB says:

    When excluding a directory with unversioned or modified files, svn does leave those files behind. That’s a good thing. Unfortunately, svn deletes the .svn directory; so we can longer run update with –set-depth=infinity on that directory. For example, assume that directory foodir contains unversioned or modified files:
    $ svn update –set-depth=exclude foodir
    D foodir
    $ svn update –set-depth=infinity foodir
    svn: Failed to add directory ‘foodir’: an unversioned directory of the same name already exists
    I was hoping that svn would preserve foodir/.svn so we can undo the exclude later. It would be even better if svn preserves foodir/.svn and also do some cleaning of the foodir/.svn/text-base directory to remove files that are safe to be deleted.

  13. C. Michael Pilato says:

    @KB: Subversion doesn’t need to preserve the .svn/ directory, because you can use the –force option to ‘svn update’ (so, ‘svn update –set-depth=infinity –force foodir’) to accomplish what you want. Give it a shot — see what you think.

  14. KB says:

    Mike: That works great. Thanks for the tip.

  15. is this akin to “cloaking” used in some of the more centralized systems out there?… or is it not exactly synonymous to cloaking directories?

  16. Stas says:

    Hi,
    Thanks for a good explanation !
    But what about the problem of tagging a sparse dir ?
    Suppose I have this structure:
    myproj
    dA1

    dB1

    I want user A to work on dA1, and user B – on dB1.
    But both are contributing to the same release, let’s call it rel_X.
    So, user A builds sparse working copy of myproj
    % svn co $SVNROOT/myproj -–depth empty myproj
    % svn update -–depth infinity myproj/dA1
    .. possibly modifies it in some way and tries to tag:
    % svn mkdir $SVNROOT/tags/rel_X
    % svn copy myproj $SVNROOT/tags/rel_X/ -m “tagging my part”
    The problem that we see is SVN for some reason also copies dB1 under $SVNROOT/branches/rel_X !!
    % svn ls $SVNROOT/tags/rel_X/myproj
    dA1
    dB1 <- this should not be there !
    Any ideas ?
    thanks !
    Stas

  17. C. Michael Pilato says:

    Stas, the very behavior you deem buggy is one that many folks call a feature. In fact, one of the values that sparse directories brings is that you can do large tree re-organization (copies, movies, deletes, etc.) in a working copy without needing to have all those trees present in full. Imagine the simple case of a tags/ directory that’s gotten too big to manage, so an admin wants to shard the tags across many subdirectories (tags/nightly, tags/releases, …) and do it all in one shot. Sparse checkouts of the tags, a little ‘svn mkdir’ and ‘svn copy’ action, and a commit, and it’s done.
    That said, I can easily see the value of making a particular sparse configuration “stick” when it is copied, so perhaps there’s room for optional behavior here. Maybe a flag to ‘svn copy’. You should raise the suggestion over at dev@subversion.apache.org.

  18. Stas says:

    Michael,
    thanks for the answer, I sent the bug report/feature request as you suggested.
    It is interesting how this problem is usually solved until now?
    How each developer tells an integrator which version of each part should be taken for next release?
    My solution was to use common tag (such as: rel_candidate). This would help to automate the problem and avoid the need for the integrator to know specific versions of all components. This solution was very naturally implemented in CVS or Perforce (the systems I worked a lot until now).
    It would only seem natural that “svn copy WC -> URL” can take whatever I have in my WC,
    and ONLY that, similar to unix copy.
    thanks,
    Stas

  19. nice job mike, i will add your blog to my website, ill be back for more updates

  20. sun says:

    Hi,My return message is
    “svn: Request depth ‘exclude’ not supported” when I execute “svn up –set-depth=exclude switch”.
    My svn enviroment is 1.6.12 .
    Any ideas?
    thanks.
    sun

  21. C. Michael Pilato says:

    Sun, the error “Request depth ‘exclude’ not supported” comes from the server when the client asks it to generate an update report with depth “exclude”. Subversion shouldn’t ever be doing that, though, as exclusion is handled by the client alone. Can you provide a reliable recipe for reproducing this problem, starting with a fresh checkout from your repository? If so, that would be extremely helpful.

  22. peter says:

    I got the same message as Sun did whenever I try to update a file rather than a directory. That is, if I do
    svn update –set-depth=exclude adir/
    I get
    D adir/
    as expected. But if I do
    svn update –set-depth=exclude adir/afile.txt
    I get “svn: Request depth ‘exclude’ not supported”.
    Does the “exclude” feature only work on directories but not files?
    peter

  23. C. Michael Pilato says:

    Exclusion of files works fine for me:
    $ cd projects/subversion/site/publish
    $ svn up –set-depth exclude TEMPLATE
    D TEMPLATE
    $ svn up TEMPLATE
    At revision 1021365.
    $
    If someone could fill my request for a “reliable recipe for reproducing this problem, starting with a fresh checkout from your repository”, that could really help diagnose this issue.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

CAPTCHA Image

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

connect with CollabNet
   Contact Us
sign up for emails
looking for something
conversations

CollabNet: Join the webinar: A Proven Path for Migrating from ClearCase to #Git or #Subversion http://t.co/IzQQRsV6cX
Date: 29 September 2014 | 4:46 pm

CollabNet: Join the webinar: A Proven Path for Migrating from ClearCase to #Git or #Subversion http://t.co/IzQQRsV6cX
Date: 26 September 2014 | 8:30 pm

CollabNet: Join @tech_mahindra: and @CollabNet: for this webinar: Achieving #DevOps Using Open Source Tools in the Enterprise http://t.co/sjUJewOojf
Date: 26 September 2014 | 5:26 pm