Development Guide¶

Overview¶

In this project we use a bunch of extra tools that simplify the drudgery of manual maintenance tasks so we can get more coding done. Its also probably not how your used to.

This “middleware” includes:

invoke: for creating the runnable endpoints or targets.
jubeo: for importing and updating a standard set of project independent invoke targets and endpoints.

invoke is common but jubeo is a creation of my own and may still have some rough edges.

Furthermore, this project is made from a cookiecutter template to bootstrap it. You may find some odd stubs around and that is why. Feel free to get rid of them. If you ever want them back you can transcribe from the source.

Ideally you won’t have to do much outside of running the invoke targets (i.e. any command that starts with inv).

To see all of the commands run:

inv -l

You should read and understand the jubeo documentation so that you know how to add your own project-specific targets via invoke.

Getting Set Up¶

Obtaining the source code¶

For the source code:

git clone {{ cookiecutter.dev_url }}
cd {{ cookiecutter.project_name }}

Tooling¶

The project comes configured for use with jubeo (see the .jubeo directory) but without the tasks imported.

To get started you will need to install jubeo and then run this from the project directory:

jubeo init .

This will download the appropriate invoke tasks organized into a special structure along with the necessary dependency specs to use them.

To be able to run the tasks you should install these dependencies:

pip install -r .jubeo/requirements.in

If you add to the libraries needed (through plugins discussed later) you will need to edit the .jubeo/requirements.txt file and recompile the .jubeo/requirements.in file by either manually running pip-compile or using the inv core.pin-tool-deps target.

Configuring¶

We typically manage configuration values in the tasks/config.py and tasks/sysconfig.py files as opposed to global system environment variables.

See the jubeo documentation on how to use these configuration files.

For this python project template we do need to set some values before all of the features will work. We also avoid setting these in shell configuration variables and as of now it is just up to the user to customize these. To the tasks/config.py file add the following:

PROJECT_SLUG = "{{cookiecutter.project_slug}}"
VERSION="{{cookiecutter.initial_version}}"

Just make sure to update the version string here when you do releases (included in the checklist for releases).

Virtual Environments¶

There are helpers for pinning and (re)generating python virtual environments which are helpful in developing and testing this project, and not necessarily just for running it as a user. See Managing Dependencies for details on managing dependencies of the installable project.

If an environment has been already been written and compiled you need only create it locally and then activate it.

To create an env called dev just run the env (env.make) target from invoke:

inv env -n dev

If it fails double check that all the dependencies have been compiled.

If it still fails, likely the environment is meant to be used for simultaneous development of multiple projects. You can check which installable source repos are expected in which locations by looking at the self.requirements.txt file. If there are simultaneous dev requirements make sure these source repos can be found at those locations.

Then follow the activation instructions that are printed as different projects might use different backends.

For pure python projects the default venv tool should be used, but conda is also an option.

For venv envs they will be stored in a directory called _venvs and for conda _conda_envs (this is customizable however). Simply:

source _venvs/dev/bin/activate_

or

conda activate _conda_envs/dev

In any case the environments are not stored with other user-level environments, what we call ambient environments, and are instead stored in the project directory.

If you ever have problems with an environment just rerun the env.make target to get a clean one. A practice we encourage to do frequently so that developers don’t diverge in their envs with local modifications. So while you can make your env, try to use this one unless you have problems.

We maintain a number of preconfigured environments in the envs directory which are used for different purposes. Calling inv env -n dev is the same as inv dev since it is the default, but any other environment can be created by passing the matching name. For instance there is an environment that mimics the user’s installation environment so that we can test experiences upon install, to make sure we haven’t accidentally depended on something in the dev env:

inv env -n test_install

Maintenance Tasks¶

Managing Dependencies¶

Quick Reference

To initially pin an environment or when you add requirements run this target:
```
inv env.deps-pin -n dev
```
To update it (should be accompanied by a reason why):
```
inv env.deps-pin-update -n dev
```
The best practice here is to make initial pinning and updating a single commit so that it can easily be rolled back or patched e.g.:
```
git add envs/*
git commit -m "Updates dev environment"
```
Explanation

Reminder that there are two separate goals of managing dependencies and where they are managed:

Python Libraries
These dependencies are managed in setup.py and in PyPI or other indices.

Python Applications/Deployments
These are dependencies managed in requirements.in and requirements.txt and used for developer environments and deployment environments.

In this template project there are a number of different places dependencies are managed according to both of these purposes. As far as the python library specs are concerned it is simpler and well documented elsewhere. In this template we introduce a few other mechanisms for managing development environments. They are as follows with the specific purpose of them:

setup.py
specifying high level requirements for installation of a released version from an index by a user or system integrator.

tools.requirements.txt
A bare minimum high-level listing of dependencies necessary to bootstrap the creation of development environments from the project tooling itself. You are free to install these in any ambient environment you see fit. We suggest using something like pyenv-virtualenv.

envs/env_name dirs
a directory with a set of files that are used to reproduce development environments the full structure will be discussed separately. There can be any number of these but they shouldn’t start with a double-underscore ‘__’ which is used for temporary utility environments.

requirements.in
An optional high-level specification of install dependencies readable from other projects for simultaneous development. Should be the same as setup.py install dependencies.

The biggest concern for developers is writing env specs in the envs dir. These add a few features a simple requirements.in/requirements.txt file can’t solve alone. Here is the full listing of possible files that can be authored by the developer in this directory:

requirements.in
(required) abstract specification of packages

self.requirements.txt
(required) how to install packages actively being worked on

dev.requirements.list
A list of paths to other requirements.in files that will be included in dependency compilation with this env.

pyversion.txt
the python version specified (if supported)

This also supports the use of conda for managing environments, although this isn’t recommended for python packages which are not intended to be distributed via conda. This is however, useful for projects like the analytics-cookiecutter project which won’t actually be distributed to others as general purpose. For this you need only add another file for the abstract conda dependencies:
- env.yaml (required for conda managed envs) an abstract specification for dependencies. Compiled to env.pinned.yaml
All the other files are still valid for conda environments still.
1. Abstract Requirements
```
requirements.in
```
  The basic part of this spec is the requirements.in and self.requirements.txt files.
  
  The requirements.in file is as described in the pip-tools documentation (i.e. pip-compile requirements.in).
  
  Running inv env.deps-pin will compile this file to a requirements.txt file, which can then be used to create an environment via inv env (i.e. pip install -r requirements.txt).
  
  It should look something like this:
```
requests
networkx >= 2
```
  There should be no entries like -e . for installing the package or any local file paths. This should be portable between machines and developers.
2. Development Project Installation Spec
```
self.requirements.txt
```
  The self.requirements.txt file instead is where these kinds of specifications should be.
  
  At its simplest it may look like this:
```
-e .
```
  Which means just to install the package of this current repo.
  
  However, it is often that you are working on multiple separate projects at once in different version control repos and want to develop simultaneously without either releasing them every time you want to make changes or even push them to a git repo. You can then write a self.requirements.txt file that looks like this:
```
-e .

-e ../other_project
-e $HOME/dev/util_project
```
3. Simultaneous Project Development
```
dev.requirements.list
```
  During simultaneous development however, the dependencies of these other repos won’t be included in the compilation of the requirements.txt file.
  
  Your options are to:
  1. manually transcribe their dependencies into the env’s requirements.in file (not recommended)
  2. write top-level requirements.in files for each project and include paths to these files in the envs/env_name/dev.requirements.list file.
  The tooling here provides support for the second one. For this you must write a list text file (see rfc:salotz/016trivial-plaintext-formats for a discussion of the format), where each line should be a path to a requirements.in file, e.g.:
```
../other_project/requirements.in
$HOME/dev/util_project/requirements.in
```
  This will include each of these files in the dependency compilation step. Note that the requirements.in can come from any location and is not a specification other projects must support.
4. Meta-Tools Installation Spec
```
tools.requirements.txt
```
  Use this to “pin” or constrain versions of tools which won’t be or can’t be managed by the pinning tool (i.e. pip-tools, meaning pip, setuptools etc.).
  
  The main use of this to pin the version of pip in case it breaks some other tools.
5. Specifying Python Version
```
pyversion.txt
```
  This file should only contain the text that specifies the version of python to use that is understood by the env method (e.g. conda).
  
  E.g.:
```
3.7.6
```
  Only the conda method supports this as of now.
  
  For the venv method it is still encouraged to write this file though, as a warning will be generated to remind you.
  
  For managing different python versions we recommend using something like pyenv and we may integrate with this or manually specifiying interpreter paths in the future.

Documentation and Website¶

Writing Documentation

The primary source of the documentation is in the info folder and is written in Emacs org mode.

Because of the powerful wiki-like capabilities of org mode it can serve as a primary source for reading docs. This obviously serves the devs more than end user’s expecting an HTML website it is a good first measure for writing docs.

Org-mode documents can be converted to RestructuredText files for use in generators like Sphinx (for documentation) or Nikola (for static sites) using the pandoc tool which we expect to be installed.

Furthermore, org-mode & Emacs provides excellent facilities for writing foreign source code blocks which allow for literate documents which can easily be tangled into code files that can then be tested automatically.

The documentation can roughly be broken down into three major parts:

pages
Documents the actual project this repo is about. Should always be tested with the same version of the software it is released with. Should not include extra dependencies.

examples & tutorials
Extended documentation of the project, however this may include extra dependencies of the project. These are tested separately from the pages documentation.

meta
General information about the project management itself. Will not be tested and should only contain source code in extremely small doses.

If you write code blocks in your documentation (which is highly recommended) you must at least write tests which run the code to make sure it at least runs.

When you write code blocks you should use this format:
```
#+begin_src python :tangle ex0.py
  print("Hello!!!")
#+end_src
```
Notice there is no extra paths to get the tangling right. The tooling for running the tests will take care of setting up an environment for tangling scripts as the docs shouldn’t really be tangled in place in the info tree.
Writing Examples & Tutorials

For our purposes as devs examples and tutorials are almost the same in structure. The distinction is mostly for end users that have different expecations from examples over tutorials.

Examples should be provide less explanation whereas tutorials are likely to be long form prose documents with literate coding and may even provide media like graphs and pictures.

Examples can also be literate but they are restricted to formats like org mode, whereas the tutorials may also be in formats like Jupyter, which integrate well with Sphinx docs.
1. Initializing a Tutorial/Example
  
  To write examples and tutorials that play nice with testing and the basic rules of the examples (described in the usersguide) there are some templates available in the templates directory for templates/examples, templates/tutorials, and environment specs templates/envs.
  
  You can either just copy these templates over or use the targets:
```
inv docs.new-example --name='myexample' --template='org'
cp -r -u templates/envs/conda_blank info/examples/myexample/env
inv docs.new-tutorial  --name='mytutorial --template=jupyter'
cp -r -u templates/envs/conda_blank info/examples/mytutorial/env
```
  After you have your directory set up there are some things to keep in mind while you are constructing your tutorial.
2. Managing Dependencies and Envs
  
  First, write source either in the literate document (README.org) or in the source file. Not both, unless you intend to test both separately. For tutorials you should prefer to write them directly in the literate doc, but particularly long and uninteresting pieces of code can be put into the source.
  
  As you write the code pay attention to your dependencies and virtual environment. If you add new dependencies, add them to the requirements.in or env.yaml file and compile:
```
cd $PROJECT_DIR
inv docs.pin-example -n 'myexample'
```
  You can then make the env 2 ways (the latter is intended to be run by users who don’t want to be overwhelmed by all the dev options):
```
cd $PROJECT_DIR
inv docs.env-example -n 'myexample'
```
  or
```
inv env
```
3. Writing Code Examples
  
  When writing examples and tutorials you should manually write the tangle targets to be the _tangle_source folder:
```
Here is some code I am explaining that you will run:

#+begin_src python :tangle _tangle_source/tutorial.py
  print("Hello!")
#+end_src
```
  As stated in the user’s guide if you don’t follow this rule (or any others) then it is wrong and an issue should be filed.
  
  When using input files, please copy them to the input dir and reference them relative to the example dir. So that when you execute a script:
```
python source/script.py
```
  The code for reading a file would look like:
```
with open("input/data.csv", 'r') as rf:
    table = rf.read()
```
  and not:
```
with open("../data.csv", 'r') as rf:
    table = rf.read()
```
  Similarly writing and creating files should be done into the _output dir:
```
with open("_output/test.txt", 'w') as wf:
    wf.write("Hello!!")
```
4. Adding to the built documentation
  
  The tutorial README files will be automatically converted to ReStructuredText and built into the documentation, but in order to have links to them from the Tutorials page you will need to manually add them to the table of contents section in the sphinx/tutorials_index.rst file, e.g.:
```
.. toctree::
   :maxdepth: 1

   tut0/README
   tut1/README
```
Testing Documentation

There is a folder just for tests that target the docs tests/test_docs. You should be able to run them after tangling:
```
inv docs.tangle
inv docs.test-example
inv docs.test-tutorial
inv docs.test-pages
```
See these targets for more fine-grained tests or to run them using nox for the python version matrix or just to have a more minimal and reproducible test environment.
```
inv -l | grep docs.test
```
Building Documentation

To compile and build the docs just run:
```
inv py.docs-build
```
Which will output them to a temporary build directory _build/html.

You can clean this build with:
```
inv py.clean-docs
```
To view how the docs would look as a website you can point your browser at the _build/html folder or run a python http web server with this target:
```
inv py.docs-serve
```
Building and testing the website

For now we only support deploying the sphinx docs as a website and on github pages (via the gh-pages branch, see Website Admin).

So to view your undeployed docs just run:
```
inv py.docs-serve
```
And open the local URL.

Once you are happy with the result, you must commit all changes and have a clean working tree then you can push to github pages:
```
inv py.website-deploy
```
Basically this checks out the gh-pages branch merges the changes from master builds the docs, commits them (normally these files are ignored), and then pushes to github which will render them.

We may also support other common use cases in the future as well like Gitlab pages or a web server (via rsync or scp).

We also will support a more traditional static site generator workflow instead of relying in addition to the sphinx docs.
Deploying the website

We are using github pages. To avoid having to keep the entire built website in the main tree we use the alternate gh-pages branch. To make this process easy to deploy we have a script sphinx/deploy.sh that checks the gh-pages branch out, does some necessary cleaning up, and copies the built website to the necesary folder (which is the toplevel), commits the changes and pushes to github, and then returns to your working branch.

The invoke target is:
```
inv docs.website-deploy
```

Testing¶

This is about testing the actual source tree (see Testing Documentation for testing the docs).

Testing in the Dev Cycle

You can either test in the dev (or test) environment while working:
```
inv py.tests-all
```
There are specific commands for each section of tests, primarily:
```
inv py.tests-integration
inv py.tests-unit
```
If you use the -t option you can specify a tag. The tag will be used as an identifying string for output to reports etc. Currently it will generate test results into the reports
Automated Test Matrix

We use nox as the runner for parametrizing and running tests in isolated environments for the test matrix. See the noxfile.py on how this is configured.

You can run the “session” directly since there are other session definitions for docs etc.:
```
nox -s test
```
There is also a target for this:
```
inv py.tests-nox
```
Auxiliary “tests”

We also have two other “testing” targets for the benchmarks and the “interactive” tests.

Benchmarks have a special toolchain for recording and publishing them as metrics.

The ‘interactive’ tests are just tests which have something like a breakpoint() in them. This is kind of an experimental thing, and probably more useful for you to write and call them individually for different purposes. The idea is that you can write “tests” that generate something like realistic live environment (kind of like integration tests) that you can drop into a debugger with and poke around in.

Code Quality Metrics¶

Just run the end target:

inv quality

This will write files to metrics.

Releases¶

The typical pre-requisites for a release are that:

the documentation has been updated and tested
the tests have been run and results are recorded
the quality metrics have been run and are recorded
the changelog has been written

Making a release then follows these steps:

test the build
make a pre-release build and publish
make the release build and publish
build and publish documentation, website, etc.
Writing and/or Generating the Changelog and Announcement

Simply go into the info/changelog.org file and write it.

There are conventions here per-project. Follow them.
Choosing a version number

There are some semantics around changing the version number beyond the ‘semver’ sense of the ‘v.X.Y.Z’ meanings.

To make a release do some changes and make sure they are fully tested and functional and commit them in version control. At this point you will also want to do any rebasing or cleaning up the actual commits if this wasn’t already done in the feature branch.

If this is a ‘dev’ release and you just want to run a version control tag triggered CI pipeline go ahead and change the version numbers and commit. Then tag the ‘dev’ release.

If you intend to make a non-dev release you will first want to test it out a little bit with a release-candidate prerelease.
Changing the version number

You can check the current version number with this command:
```
inv py.version-which
```
The places where an actual version are needed are:
- setup.py
- sphinx/conf.py
- src/package/__init__.py
- tasks/config.py
- conda/conda-forge/meta.yaml (optional)
- the git tag
The setup.py and src/package/__init__.py version is handled by versioneer using the git tag for the release. This allows for fine-grained versions using git hashes on “dirty” releases.

The sphinx/conf.py just gets the current version from __init__.py so it is also downstream of versioneer.

So currently only the tasks/config.py and conda versions need to be updated manually.

In this project we never like to initiate configuration tasks at the REPL/shell so we never actually run git tag under normal circumstances.

Instead we configure the desired version “bump” in one place tasks/config.py and then generate the rest downstream through invoke endpoints.

So simply edit the tasks/config.py VERSION variable and then run:
```
inv git.release
```
Which will write the git tag in the correct format. versioneer takes over from there.

Here then is the checklist of manually edited versions (currently the conda packaging stuff is not supported):
- [ ] edit tasks/config.py
- [ ] commit
- [ ] run inv git.release
Changing the version may happen a few times through the release process in order to debug wrinkles in the process so its useful to have this workflow in mind.
Release Process
1. Testing the build
  
  To test a build with whatever work you have go ahead and run:
```
inv py.build
```
  And then try to install it from an empty environment:
```
inv env -n test_install
```
  Activate the environment e.g.:
```
source _venv/test_install/bin/activate
```
  or
```
conda activate _conda_envs/test_install
```
  then run it for each build, e.g.:
```
pip install dist/BUILD.tar.gz
```
  They should all succeed. You should also test the installation somehow so that we know that we got the dependencies correct.
2. Making the Pre-Release
  
  All releases should be preceded by a release candidate just to make sure the process is working as intended.
  
  So after this testing of your potentially “dirty” tree (which is anything that is not equal to a ‘vX.Y.Z.etc’ git tag) change the versions to have ‘rc0’ at the end of the new intended (semantic) number, e.g. v1.0.0.rc0.
  
  Then go ahead and commit the changes with a message like this:
```
1.0.0rc0 release preparation
```
  Then do the git release (just tags it doesn’t ‘publish’ it) and rebuild before doing the next steps:
```
inv git.release
inv py.build
```
  Once you have built it and nothing is wrong go ahead and publish it to the test indexes (if available):
```
inv py.publish-test
```
  You can test that it works from the index using the same test_install environment above.
  
  And install the package from the test repo with no dependencies:
```
pip install --index-url https://test.pypi.org/simple/ --no-deps package
```
  Then you can publish this pre-release build.
  
  Publishing the results will vary but you can start with publishing the package to PyPI and the VCS hosts with the real publish target:
```
inv git.publish
inv py.publish
```
3. The final public release
  
  Edit the version number to something clean that won’t be hidden on PyPI etc.
  
  Then:
```
inv git.release
inv py.build
inv py.publish
inv git.publish
```

Initializing this repository¶

These are tasks that should only be done once at the inception of the project but are described for posterity and completeness.

Version Control¶

First we need to initialize the version control system (git):

inv git.init

If you want to go ahead and add the remote repositories for this project. We don’t manage this explicitly since git is treated mostly as first class for these kinds of tasks and is better left to special purpose tools which are well integrated and developed.

Python Packaging¶

There is a target to initialize python specific packaging things. This is because some tools (like versioneer) need to be generated after project instantiation.

Make sure you have a clean tree so you can see the changes then:

inv py.init

then inspect and commit.

Compiling Dependencies¶

Then add any extra dependencies you want to the development environment requirements.in file and then compile and pin them:

inv env.deps-pin -n dev env.deps-pin -n test_install

Then commit this.

Creating Environments¶

Then just create the virtual environment. For portability we use the builin venv package, but this is customizable.

inv env

Then you can activate it with the instructions printed to the screen.

Website Admin¶

We use Github Pages by default since it is pretty easy. Because we don’t want to clutter up the master branch with website build artifacts we use the gh-pages branch approach.

If you just run the inv py.website-deploy target this will idempotently take care of setting this up for you.

However, you will need to create it and push it before you can set this in the github settings for the page.