Sparse Directories, Now With Exclusion

About C. Michael Pilato

C. Michael Pilato is a core Subversion developer, co-author of Version Control With Subversion (O'Reilly Media), and the primary maintainer of ViewVC. He works remotely from his home state of North Carolina as a software engineer for CollabNet, and has been an active open source developer since early 2001. Mike is a proud husband and father who loves traveling, soccer, spending quality time with his family, and any combination of those things. He also enjoys composing and performing music, and harbors not-so-secret fantasies of rock stardom. Mike has a degree in computer science and mathematics from the University of North Carolina at Charlotte.

  23 comments for “Sparse Directories, Now With Exclusion

  1. March 4, 2009 at 11:16 pm

    Nice feature!
    What is the syntax to checkout with several excludes so you don’t have to download unwanted subdirectories in the first place?

  2. David Aldrich
    March 5, 2009 at 1:08 am

    Thanks, that’s a great explanation and a very useful feature. Will it work with a 1.6 client and 1.5 server?

  3. C. Michael Pilato
    March 5, 2009 at 6:35 am

    @René: Checking out without unwanted items in the first place is a process more than any particular syntax. :-) The gist of it is that you build out the tree of things you want, using extremely shallow depths for intermediate placeholder directories and such. There’s a bit of an example of this at the end of the svnbook section I referred to: http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html
    @David: Yep. Works great. In fact, I’m pretty sure that it works with a 1.6 client and *any* server version.

  4. March 5, 2009 at 7:32 am

    Does the new syntax only allow you to exclude something, or does it also allow you to reduce the depth? In other words, could have allowed the “www” folder to remain in your WC but now be empty? As if you had checked out its parent with –depth=immediates ?
    Mark

  5. March 5, 2009 at 7:38 am

    To answer my own question, having just tried it, Yes! This is pretty sweet. So now if you no longer need a folder you can reduce it’s depth and then put it back again later.
    Mark

  6. C. Michael Pilato
    March 5, 2009 at 7:43 am

    @Mark: Exclusion *is* depth reduction, of course. But yes, you can do arbitrary depth reduction.

  7. Graham Nash
    March 5, 2009 at 7:45 am

    Completely off topic but cannot seem to find any information on this… Any word on whether or not #3128 will be fixed in 1.6?
    http://subversion.tigris.org/issues/show_bug.cgi?id=3128

  8. Ben
    March 5, 2009 at 11:34 am

    Adding the ability to exclude directories now, put simply, tops off the wonderful sparse checkouts feature.
    Before, I had to be extra careful when doing a sparse checkout; our repository layout isn’t changing anytime soon and a full checkout is currently about 1GB since we (for better or worse) have some large binary data mixed in with our source. A checkout of source code only is about 3.4MB so a sparse checkout is really a nice thing.
    Not that this is the place, but I always wonder if there is a way to do an overlay – that is have items (files) from different repositories in the same directory – so that source code can be kept in one repo, and designated (large) binary files are somehow kept in another repository or other storage method. Rsync is an option I guess, has anyone else had this problem?

  9. Kevin
    March 5, 2009 at 4:15 pm

    Excellent! I’ve wanted something like this for a long time!
    Now, if svn could support a “p4 client view” like interface for adding/removing/remapping the client files… :)
    [1] http://www.perforce.com/perforce/doc.current/manuals/cmdref/o.views.html#1040759

  10. March 6, 2009 at 1:42 pm

    @Graham. I just updated issue 3128. IMO, it was fixed in SVN 1.5.5

  11. March 6, 2009 at 3:35 pm

    We manage several domains, hosted on at least two servers, in one Subversion repository. Before sparse directories, we either had to checkout all domains on each server, or checkout each domain to a separate working copy.
    Now we use sparse directories to checkout to one working copy on each server, only the domains which are hosted on that server.

  12. KB
    March 23, 2009 at 9:40 am

    When excluding a directory with unversioned or modified files, svn does leave those files behind. That’s a good thing. Unfortunately, svn deletes the .svn directory; so we can longer run update with –set-depth=infinity on that directory. For example, assume that directory foodir contains unversioned or modified files:
    $ svn update –set-depth=exclude foodir
    D foodir
    $ svn update –set-depth=infinity foodir
    svn: Failed to add directory ‘foodir’: an unversioned directory of the same name already exists
    I was hoping that svn would preserve foodir/.svn so we can undo the exclude later. It would be even better if svn preserves foodir/.svn and also do some cleaning of the foodir/.svn/text-base directory to remove files that are safe to be deleted.

  13. C. Michael Pilato
    March 23, 2009 at 10:26 am

    @KB: Subversion doesn’t need to preserve the .svn/ directory, because you can use the –force option to ‘svn update’ (so, ‘svn update –set-depth=infinity –force foodir’) to accomplish what you want. Give it a shot — see what you think.

  14. KB
    March 23, 2009 at 11:21 am

    Mike: That works great. Thanks for the tip.

  15. April 17, 2009 at 8:34 am

    is this akin to “cloaking” used in some of the more centralized systems out there?… or is it not exactly synonymous to cloaking directories?

  16. January 6, 2010 at 7:56 am

    Hi,
    Thanks for a good explanation !
    But what about the problem of tagging a sparse dir ?
    Suppose I have this structure:
    myproj
    dA1

    dB1

    I want user A to work on dA1, and user B – on dB1.
    But both are contributing to the same release, let’s call it rel_X.
    So, user A builds sparse working copy of myproj
    % svn co $SVNROOT/myproj -–depth empty myproj
    % svn update -–depth infinity myproj/dA1
    .. possibly modifies it in some way and tries to tag:
    % svn mkdir $SVNROOT/tags/rel_X
    % svn copy myproj $SVNROOT/tags/rel_X/ -m “tagging my part”
    The problem that we see is SVN for some reason also copies dB1 under $SVNROOT/branches/rel_X !!
    % svn ls $SVNROOT/tags/rel_X/myproj
    dA1
    dB1 <- this should not be there !
    Any ideas ?
    thanks !
    Stas

  17. C. Michael Pilato
    January 6, 2010 at 8:08 am

    Stas, the very behavior you deem buggy is one that many folks call a feature. In fact, one of the values that sparse directories brings is that you can do large tree re-organization (copies, movies, deletes, etc.) in a working copy without needing to have all those trees present in full. Imagine the simple case of a tags/ directory that’s gotten too big to manage, so an admin wants to shard the tags across many subdirectories (tags/nightly, tags/releases, …) and do it all in one shot. Sparse checkouts of the tags, a little ‘svn mkdir’ and ‘svn copy’ action, and a commit, and it’s done.
    That said, I can easily see the value of making a particular sparse configuration “stick” when it is copied, so perhaps there’s room for optional behavior here. Maybe a flag to ‘svn copy’. You should raise the suggestion over at dev@subversion.apache.org.

  18. January 7, 2010 at 1:23 am

    Michael,
    thanks for the answer, I sent the bug report/feature request as you suggested.
    It is interesting how this problem is usually solved until now?
    How each developer tells an integrator which version of each part should be taken for next release?
    My solution was to use common tag (such as: rel_candidate). This would help to automate the problem and avoid the need for the integrator to know specific versions of all components. This solution was very naturally implemented in CVS or Perforce (the systems I worked a lot until now).
    It would only seem natural that “svn copy WC -> URL” can take whatever I have in my WC,
    and ONLY that, similar to unix copy.
    thanks,
    Stas

  19. June 30, 2010 at 3:54 pm

    nice job mike, i will add your blog to my website, ill be back for more updates

  20. sun
    August 30, 2010 at 11:09 pm

    Hi,My return message is
    “svn: Request depth ‘exclude’ not supported” when I execute “svn up –set-depth=exclude switch”.
    My svn enviroment is 1.6.12 .
    Any ideas?
    thanks.
    sun

  21. C. Michael Pilato
    August 31, 2010 at 6:04 am

    Sun, the error “Request depth ‘exclude’ not supported” comes from the server when the client asks it to generate an update report with depth “exclude”. Subversion shouldn’t ever be doing that, though, as exclusion is handled by the client alone. Can you provide a reliable recipe for reproducing this problem, starting with a fresh checkout from your repository? If so, that would be extremely helpful.

  22. October 10, 2010 at 12:34 pm

    I got the same message as Sun did whenever I try to update a file rather than a directory. That is, if I do
    svn update –set-depth=exclude adir/
    I get
    D adir/
    as expected. But if I do
    svn update –set-depth=exclude adir/afile.txt
    I get “svn: Request depth ‘exclude’ not supported”.
    Does the “exclude” feature only work on directories but not files?
    peter

  23. C. Michael Pilato
    October 11, 2010 at 8:02 am

    Exclusion of files works fine for me:
    $ cd projects/subversion/site/publish
    $ svn up –set-depth exclude TEMPLATE
    D TEMPLATE
    $ svn up TEMPLATE
    At revision 1021365.
    $
    If someone could fill my request for a “reliable recipe for reproducing this problem, starting with a fresh checkout from your repository”, that could really help diagnose this issue.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

CAPTCHA Image

*