We recently conducted a webinar called Test Management Using TeamForge . The audience engagement was fantastic and we received many insightful questions from our listeners that we were not able to get to during the live webinar. Here are additional questions from the audience and answer from our Test Management expert CollabNet Director of engineering, Venkat Janardhanam.
Doug: I would like to know how Selenium is communicating with TestLink to update the status of the Testcase, API ? Is there an available API to contact TestLink?
Venkat: The automation works through integration of three different tools. 1. Jenkins 2. Selenium and 3. TestLink plug-in for Jenkins. The Jenkins TestLink plug-in communicates to TestLink through an open source Java API. After the build is complete the plug-in will execute all test cases that are marked Automated by invoking the test file that is specified in the Test Case. The user defined field in Test Case will have the test file that Jenkins will pull and execute. There should be one to one relationship between at TestCase in TestLink to automated test file. The test file is an automation script that represents a TestCase and the script can be a plain Java class, JUNIT class or Selenium Java class. Once the test files are executed the result pass or fail will be updating the test case with the right status through the plug-in. The test file in our reference automation is Java Selenium file generated through Selenium RC or through manual Java coding of test case using the Selenium API. You can use other CI tools and test files from different automation tool to make the whole automation work.
Doug: Do you have automation reference with JMeter?
Venkat: Not at the moment. We are in process of investigating the JMeter for our reference automation.
Doug: Please show defects generated automatically with selenium.
Venkat: This feature will be available in our interim release and currently it is disabled due to feedback received from pilot customers. The reason is there are many scenarios where the automation will generate duplicate defects. For example, if a CI tool is used for the build and test runs daily automatically then the defects will be created. If there are failures and the team is not able to fix the defects before the test runs again then duplicate defects will be created. So we are looking at all the scenarios before turning on this feature in next release and also trying to make the defect creation more intelligent to update an existing defect instead of creating new defects.
Doug: How is selenium defects added?
Venkat: The selenium defects are added into TestLink through Jenkins TestLink plug-in. In your Selenium Java test case there is a one line code snippet that needs to be added for a success scenario and a failure scenario in the Selenium Java script. The assert snippet will be recognized by plug-in to update the corresponding TestCase in TestLink with the right status.
Doug: My test results are stored in XML. Is there a way to import these test results into the TestLink. I am using the C# to create the XML is one of the participants response.
Venkat: There is an import feature in TestLink where you can import XML results generated from other automation tool into TestLink.
Doug: We currently use both TeamForge and TestLink. Is there a way we can glue users?
Venkat: The integration has to be installed to glue existing TeamForge and TestLink systems to work as one. After that there are migration scripts to migrate data between TeamForge and TestLink. The users who are in TestLink will be migrated to TeamForge and users in TeamForge will be migrated to TestLink. The script has to be run only once after installing the integration after that any projects, users, permissions and roles are automatically synchronized between the two systems. If the same user name in TestLink exists in TeamForge then there is a user conflict that the migration script will report and this conflict user will not be migrated. The scenario below will help us understand how to resolve the conflict.
Doug: For example, John is a user who exists in TeamForge and TestLink. The TeamForge already had Venkat: Project A and TestLink had Project B which is in two separate systems. The one time migration script moves Project B from TestLink to TeamForge. The user John has access to Project A in TeamForge only. To fix this the issue the TeamForge admin has to be check if the conflict user John and TeamForge user John is the same person. If it is same person then in TeamForge John already has access to Project A and John has to be given permission to Project B so that the conflict gets resolved. Now John will have access to both Project A and Project B.
Doug: We a test result is set at TestLink as Failed is it possible to automatically fill some of the fields at the created Defect? Specially the Assing To field.
Venkat: Currently, we are pushing Time of Execution, Test Plan Name, Build and User Comments to TeamForge. The feedback received so far is automating AssignedTo is difficult because there are cases where the person who is going to work on a defect may need specialized skills that only one person in a team can fix it and it is hard for system to identify AssignedTo. In case, the system identifies the AssignedTo there may be cases where the person to whom the defect is assigned may be too busy then we have to work on work load distribution to see who is free and then assign.
I am open to receive more feedback from users on what other field that will make sense to be added to the auto generated defect so that we can plan to incorporate the request in the future roadmap.
Doug: Is it possible to configure the TestLink project to generate a specific Defect tracker type? For instance we have functional defects and performance Defects
Venkat: During initial configuration of TestLink integration with TeamForge by project admin the TestLink expects the TeamForge user to provide all the tracker ids of all requirement trackers. For example, if you have Epic, Story and Task tracker then you need to provide tracker id for all three trackers with comma separation during configuration. Similarly for defect tracker the integration accept only one defect tracker id and you cannot have multiple defect tracker.
Doug: What TestLink version support synchr with TeamForce?(1.7.?)
Venkat: The integration works on TeamForge 7.0, 7.1 and 7.2 with TestLink version 1.9.11.
Doug: Can you automatic fetch test result from TestLink to dashboard webpage? (and from multiple project in TestLink?)
Venkat: At the moment the integration maps one TeamForge project with one TestLink project. Currently, you cannot report out of multiple TestLink project into one TeamForge project.
Doug: Any input for how to ensure quality test cases? (and not only x number of test cases per requirement) e.g. Checkmark to indicate review of test cases? Yes my questions was to understand if there can be set a checkmark to indicate test cases have been reviewed or any similar way to ensure good quality test cases.
Venkat: The TestLink provide custom fields where multiple custom fields can be created that can be used as a checklist. The reviewer of the test case has to review test case and update the checklist one by one. The check list can be a single text area or multiple check boxes. If all the items in the check list are complete then the user can mark the test case as approved by clicking another custom field check box.
Doug: Can we shut off auto defect generation?
Venkat: Sure, you can turn on/off auto defect generation at a project level.
Doug: I don’t want a test suite created for each requirement.
Venkat: You can select the TestSuite field to ‘NONE’ instead of ‘Create’ while creating requirement and this will not create Test Suite in TestLink. This is controlled at every requirement artifact level.
Read more on Venkat’s blog about test management in TeamForge.
Listen to the complete webinar to understand how test management can be tied into continuous integration using TeamForge, allowing your agile teams to collaborate and get early feedback. Also, see how traceability can be easily maintained from requirement to test cases to defects and builds.