On April 18^th, I co-presented the webinar, “Continuous Delivery: The Last Mile,” with my colleague Brian Dawson, Sr. Solutions Consultant.  It was the last webinar in our Closing the Agile Loop: CI&CD for the Enterprise Series. In the session, we talked about how automating integration and load tests in production-like environments can help dev teams achieve the highest quality standards. Specifically, we covered:

• best practices to industrialize release processes,
• lessons on how to enforce production-ready software quality, and
• considerations for provisioning of build, test and run clouds.

During the webinar, we encouraged our attendees to interact with us by participating in polls and by entering questions for the live Q&A session that we had at the end of the presentation. As a result, we received many great questions that we just couldn’t get to during the live event so we have provided answers to the audience’s questions below.

If you missed the session live, you can watch the recording and download my slides. For more information about this topic, please visit www.collab.net/getCI here, you can access additional webinars, informative assets, and free product downloads.

Don’t forget to register for our upcoming webinars. CollabNet offers multiple webinars a month that cover a variety of topics related to software development.

As promised, here’s the follow up to the live audience questions.

Q: An area that can take a long time to complete and go under-appreciated is the training materials that need to be developed / released before production deployment.  How does that figure in?
A: Create story(s) for the training materials to be created as part of the sprint.  Develop those in the TeamForge document manager (or in the projects folder in Subversion (SVN)),  and “associate” the materials to the story(s).  The training materials must be developed, reviewed, approved and closed during the sprint, just the same as other storys.

Q: Do you have a more detailed case study of using TeamForge for “Making DevOps Real?”
A:  In addition the examples we have shared in our CI webinars about implementation details around deploying components of DevOps, we can provide one-on-one sessions to discuss the case studies and answer specific questions.  Documented white-papers will be forthcoming. Contact us at: info@collab.net for more info.

Q: How does DevOps differ from continuous delivery?
A: Continuous Delivery is about the tactical approach to implementation of best practices to support a delivery pipeline from coding to production.  Dev-Ops is a concept or philosophy that emphasizes a synergy between Development and Operations in delivery of software product.  You could say that Dev-Ops is the theory, Continuous Delivery is the application.

Q: I would love to move to automation. My team of developers is quite reluctant to move to automation (because this is an unknown territory). How I can help them engage in this process? What are the initial steps for the team to take to move towards automation?
A: To help them engage in the process I would recommend:

1. Impress upon them that the individual and team benefits of automation. Such as;

• Less time fixing issues, doing tedious manual builds, deployments etc.
• More time to develop, innovate and implement.

2. Reassure them then automation does not obsolete them, but rather allows them to do what they do best.  Especially for QA. Domain experts will always be needed and there will always be a need for some level of automated testing.

3. Highlight that this is the adoption of cutting edge technology and processes.  This should be more exciting, and it increases their value to the company and the market.

4. Training.  It is often the unknown cause for resistance.  People are more comfortable with change when they understand it, and have a stake in it.  Help facilitate this by getting training on CI/CD, allotting and assigning specific time to do research.  They are likely to feel more comfortable and even get excited about the possibilities.

Some initial recommended steps are:

1. As said above, training and research.

2. Conduct a brainstorming/review session where you evaluate the trouble spots in your process (without finger pointing). Write down the short list, prioritize, and then identify how automation may help address the highest priority issues.

3. Start by automating your builds and move to a centralized infrastructure such as a Jenkins build farm.  A CI server like Jenkins integrated with TeamForge, will serve as an expandable framework where you can grow further automation on.

Q: I have some programming experience but it goes back to quite a few years ago. How I can best empower my team of developers to improve towards the best practices you mentioned here. One difficulty is that I am working with an offshore team in India. Any suggestions on how to best work with offshore teams?
A: To address your first question about empowering your offshore team to improve towards the best practices we shared, I recommend:

1. Get them exposure in the form of training, presentations and research.

2. Conduct a brainstorming/review session where you identify areas of improvement and then identify how these processes may help address those areas.

In regards to your second question, working with an offshore team in India, I suggest that you follow the basic tenants expressed in the talk:

1. Automation of process stops

2. Integration between tools

3. Visibility into what is being done by all members of the process, on and off shore.

4. Communication.  Implement a solution like TeamForge Trackers, Discussions, and overall monitoring/notifications to allow for asynchronous communications between time zones, and to overcome language and cultural variables.

5. Conduct short meetings (“Standups”) at mutually convenient (or mutually inconvenient times ;^) at least on weekly basis, to facilitate personal interactions and help continually evolve the development process.

6. Recognize that the centralization and automation at the core of Continuous Delivery inherently eases some of the coordination between team members in different time zones.

Q: We are a small non-profit, and I am working only with 3 developers, and time to time external pro-bono teams that volunteer for us. What tools would you suggest for us to use?
A: Most of the tools mentioned in the presentation have free and open source versions and are relatively low effort to start up and use.  In a lot of ways,  you are at an advantage with such a small team.  The team can communicate more effectively then a large team, and should encounter less political, cultural and technical obstacles to implementing Continuous Delivery.  In order, I recommend:

1. TeamForge

2. Subversion (Git has less value with such a small team but can be used as well)

3. Jenkins

4. Maven/Nexus (Primarily for Java)

5. Sonar

6. Cucumber

7. Puppet

Q: Are the blocks you described, modular?
A: Anywhere in the presentation where “blocks” were shown, they need to have some level of interaction.  Per the definition of Continuous Delivery, all participants, tools, and activities should have some level of interaction.

Q: When implementing CI for multiple sub projects is it better to have same build process/Configuration management or it can vary depending on project preferences?
A: This is very much dependent on the relationship between the sub-projects. For example, if this was a Java project the relationship could be libraries that are imported to support an application and hence based on that that would have common:

• scripting tool, i.e. Maven, ANT etc..
• release management process
• common architecture and application platform

So yes,  it would make absolute sense to have a standard build process and configuration. However, if the sub-projects have distinct  technologies, (i.e. .NET versus Java), then the build process will be different. However, its good practice to commonly align the configuration and release management processes.

Q: Using Jenkins, which artifact repository to use. Team doesn’t want/like Maven
A: This really depends on the reason for not liking Maven. Some don’t like the structure it imposes; others don’t have strong support for their technology, etc.  Ivy is another popular alternative.   The TeamForge file release system can store binary artifacts and meta-data (albeit less meta-data and fewer tools support.)  Jenkins also can store build artifacts and “front” as an artifact repository.

Q: What are the levels of integration of CollabNet product with TFS 2010, especially for Work Items and Requirement Gathering?
A: TeamForge and other CollabNet products are technology agnostic.  Integration with TeamFoundation Server is provided directly through theCollabNet Desktop – Visual Studio Edition.

Q: What tools do you use for your Artifact Repository? Or what you recommend?
A: Artifactory and Nexus are two very popular Maven/Ivy artifact repositories.

Q: Brian touched on source code branching, but are there general practices which can help avoid the overhead of maintaining multiple code branches?
A: Branching and merging is anti-pattern to continuous integration as it typically defers the integration and testing of software in its complete form. It’s recommended that branching and merging is only used when it is completely necessary. For example, disruptive hot fixes that will be integrated very quickly back into the mainline (trunk).  There are new design patterns that help with limiting the use of branching such as “branching by abstraction” and “feature toggles”.

Q: Would you recommend the version control to include the requirement change request as well?
A: In general yes, especially if the change request comes after development on related requirements have already begun.  The change request should be implemented as artifact in the Tracker and associated to the original requirements.   Code commits to implement the change should be associated to both the change and the change request.

Q: How can we be flexible but still “time boxed” during the development stage, especially when we recognize that the original design will not work as the business formerly requested?
A: There could be a few answers to this question. It implies that this was not caught at the Sprint planning stage where there is universal agreement that the story would be completed. The reason why it was not identified should be noted during the review and documented to ensure it does not happen again. There could be numerous reasons of why it happened, such as inexperience or poor communication, all of which can be addressed to ensure it does not repeat.

Also, some people have commented that “time boxing” is not completely effective for developing software. Lots of stops and go’s can be very disruptive. For example, when a defect is found, this profoundly affects the rhythm of the group. Some prefer Kanban which is “time boxless” and uses a pull method for work.

Q: At work I have applied the following rules:

• Commit message must start with a ref to the issue# that the commit is connected to
• The commit message must then contain at least X number of words
• The committer MUST be the assignee of the referenced issue#
• The referenced issue MUST be in one of some defined statuses (ie. In Progress)

I feel this helps us avoid

• Anonymous commits which are hard to trackCode conflicts since no two developers can work and commit to the same issue# at once. (This does not make you immune to conflicts though)
• Helps with change management since we know that commits won’t happen to anything that is not in the “”In Progress”" status

But, how do you handle lazy developers that complain on the fact that they must be the assignee of an issue to be able to commit. Or that they must have more than three words in a commit message. Personally, I find it hard to explain the changes and implications of these changes in that few words
A: All processes return measurable value and in your case, you have instilled a process that brings traceability, visibility, and the ability to report at various levels and to different stakeholders. Without a doubt, your process is valuable, and in fact, it is something that I would also promote as this is good constructive governance. In my opinion, justifying the process is not a problem. I would probably recommend a tool that allows developers to only see or filter for their own work; this would allow them to be more effective at seeing work that is more of a priority and not to accidentally work on items that are not assigned to them.