By Python

Testing is a fundamental skill that a developer should have when it comes to programming. There are many blog articles on how to best test your code, however, what makes this article special is that it shows the lessons learned by Yogesh Chaudhary, a Python Developer Intern at VersionBay. Yogesh is a final year student of Software Engineering in the Netherlands who joined VersionBay with limited exposure to testing methodologies and Python. During Yogesh’s course, he was exposed to C# and Java. This is common across many Software Engineering courses and as such this article can help learning Python when coming from C#.

What makes Python attractive?

Having started his programming career with C#, he found Python more interesting because of the following reasons:

  • Less overhead when coding
  • A substantial number of additional packages (PyPi.org has over 290,000 packages)
  • Widely used in many different application areas from web development, numeric/scientific calculations to data visualization
  • Function decorators are elegant
  • Encourages the use of a separate development environment for each project (e.g conda environment or virtual environment)
  • List comprehensions are very useful for looping through data
  • with statement is appealing to minimize boiler code
  • An important skill for job/career development in 2021 and beyond
  • Easy to work with JSON
  • Democracy at Python Steering Committee
  • Easier to understand due to indentation and less braces

Gotchas when going from C# to Python 

It is also important to callout some of the things that caught Yogesh off-guard:

  • Managing Python environments did not come naturally and it took a while to understand the difference between using conda and virtual environments.
  • If you are coming from an OOP background, it is a little strange that Python does not require Object-Oriented Programming (OOP).
  • Given a large number of packages, it is actually really hard to know which one is best. Yogesh spent more time than he anticipated in exploring different packages. For example, to visualize data, Yogesh looked at Matplotlib, Plotly, Seaborn, Bokeh, and several others and found it hard to decide which is the best one for his graduation project.

Why unit testing?

Testing is a fundamental skill when it comes to programming and is generally a best practice that is often neglected in education. Here are some key reasons to embrace unit testing:

  • code robustness
  • enables CI/CD workflows
  • migration
  • improving code quality
  • test-driven development is a common practice in industry

to make this more concrete, here is Yogesh’s first example of using the unittest module in Python.

Example: Testing a Calculator 

There are several frameworks available in Python for unit testing having their own pros and cons and ease of use. As a starting point, Python comes integrated with a unit test framework called Unit Test which means, no need to install any additional packages.

Given the Python class named Calculator :

class Calculator(object): 
    """A simple calculator class""" 
     
    #constructor 
    def __init__(self, name): 
        self.name = name 
     
    # method to print the name of calculator 
    def info(self): 
        print(f"{self.name}'s calulator") 
 
    # a method which parse parameter to integers and returns the sum of two integers, otherwise throws error 
    def add(self, num1, num2): 
        try: 
            first_number = int(num1) 
            second_number = int(num2) 
            # return first_number + second_number 
        except IOError: 
            raise IOError 
        return first_number + second_number 
 
    # a method which parse parameter to integers and returns the difference of two integers, otherwise throws error 
    def subtract(self, num1, num2): 
        try: 
            first_number = int(num1) 
            second_number = int(num2) 
        except IOError: 
            raise IOError 
        return first_number - second_number

In this Calculator class, there is a constructor, __init__ which takes a string (name as parameter) and assigns it to the name of Calculator Object. In addition, the method info prints the name of Calculator object and the add method takes two parameters, tries to parse them to integers and if successful, returns the sum of the parameters, otherwise throws IOError. Similarly the subtract method is implemented to subtract two numbers.

In order to test this class, here is the test class TestCalculator:

import unittest 
from calculator import * 
 
 
class TestCalculator(unittest.TestCase): 
    def setUp(self): 
        self.cal = Calculator("Your String") 
 
    def test_add(self): 
        self.cal.add(10, 5) 
        self.assertEqual(self.cal.add(10, 5), 15) 
 
    def test_negative(self): 
        result = self.cal.add(10, -2) 
        self.assertEqual(result, (10 + (-2))) 
 
    def test_sub(self):
        self.assertEqual(self.cal.subtract(10, 20), -10) 
 
    def test_error(self): 
        with self.assertRaises(Exception): self.cal.add("Hello",2) 
 
    def test_error_for_add(self): 
        self.assertRaises(Exception, lambda:self.cal.add('Hello', 3), 'failureException') 
 
   def tearDown(self): 
       super(TestCalculator, self).tearDown() 
 
 
if __name__ == '__main__': 
    unittest.main()

To run this simply type at the terminal:

python -m unittest

Getting Started with GitLab CI/CD 

Yogesh did not appreciate the value of unit tests until they were used with pipelines. The idea is that tests are automatically triggered after changes are made to a branch and development errors can be found earlier in the process and code quality increases. There are really 3 steps to making this possible in GitLab:

  1. Freezing environment: create a requirements.yml
  2. Adding .gitlab-ci.yml to GitLab repository
  3. Updating/maintaining .gitlab-ci.yml based on needs

Freezing environment

To set up a CI/CD pipeline, it is a common practice to setup an environment. When using conda, an environment can be specified with a requirement file. This file has all the packages needed to run the project. With conda this file can be created from an environment with the following command:

conda env export > requirement.yml

This is particularly useful when more packages are used in larger projects.

Adding .gitlab-ci.yml to GitLab repository

GitLab needs to have a “.gitlab-ci.yml” file in the repository to define the CI/CD pipeline. GitLab triggers Gitlab Runners based on the stages and commands specified in the configuration file. GitLab makes it really easy to visualize and configure pipelines.

image: continuumio/miniconda3:latest
build-job:
  stage: build
  before_script:
    - conda env create -f requirement.yml
    - echo "environment created"
    - source activate “Name of your environment”
  script:
    - echo "PyLint"
    - pylint efficiencyplot.py (#checks efficiencyplot.py module with PEP standard)
  allow_failure: true

test-job1:
  stage: test
  needs: [build-job]
  script:
    - python -m unittest (#command to run unittest)

test-job2:
  stage: test
  script:
    - “more parallel test execution can be scripted here”

deploy-prod:
  stage: deploy
  script:
    - echo "This job deploys something from the $CI_COMMIT_BRANCH branch."

In the above example, there are 3 stages defined in a pipeline that is running in a Docker container from continuumio/miniconda3. The first stage is the build stage, where a conda environment for the project is created. All the dependencies are installed during environment creation. After the environment has been created, the script part is executed. In this example, PyLint is checking the “efficiencyplot.py” module with PEP8.

Allow_failure: true allows to continue to test the module even if the code is not fully PEP8 compliant.

After the build job is successfully completed, the second stage can start, which is the test stage. Here, two tests; test-job1 and test-job2 run in parallel. The final stage is the deploy-prod stage. Normally, the target branch for a CI/CD pipeline is the develop branch which means, whenever the develop branch changes, then the CI/CD pipeline is triggered.

Lessons Learnt:

  • Python also has Unit Testing and that is very powerful when combined with a CI/CD workflow.
  • There are many unit testing frameworks in Python.
  • GitLab makes it easy to get started with DevOps (the workflow with GitLab is similar on other platforms such as GitHub).
  • DevOps is a reality and it’s easy to get started with but hard to master.
  • PyLint helps with writing understandable code.
  • Python and C# have different coding standards.
  • There is a bit of repetitive code writing while writing unit tests with unittest.