.. _developer_guide: Developer Guide =============== Design ------ For **basic users**, **nwb2bids** is intended to be a simple-to-use tool for converting NWB datasets to near-BIDS format with minimal user configuration. For **advanced users**, **nwb2bids** is designed to be easily extensible to support new NWB data types, BIDS extensions, and custom configurable behavior. Whenever working on a new feature, keep in mind how to make it easy to understand and use for the **basic users**, while still being flexible enough for the **advanced users**. **nwb2bids** makes every effort to collect and return all errors, warnings, and informational messages encountered during conversion to the user at the end of the process, rather than stopping at the first error. This provides a comprehensive overview of all issues that need to be addressed. Philosophy ---------- **nwb2bids** is also designed with the following principles in mind: - **Modularity**: The codebase is organized into clear, modular components that encapsulate specific functionality. This makes it easier to maintain, test, and extend the code. - **Extensibility**: The architecture allows for easy addition of new features, data types, and configurations without requiring major changes to the existing codebase. - **Readability**: Code should be clean, well-documented, and follow consistent style guidelines to ensure that it is easy to understand and collaborate on. - **Performance**: While prioritizing usability and extensibility, the code should also be efficient and performant, especially when handling large datasets. - **Collect and report all notifications**: During conversion, all encountered issues (including internal runtime errors) should be collected and reported to the user at the end, rather than stopping at the first error. This provides a comprehensive overview of any problems that need to be addressed. Testing ------- This project uses ``pytest`` for testing with comprehensive coverage across multiple platforms and Python versions. Tests are organized into three categories: - **Unit tests** (``tests/unit/``): Test individual components in isolation. - **Integration tests** (``tests/integration/``): Test interactions between components. - **CLI tests** (``tests/convert_nwb_dataset/``): Test command-line interface behavior. Assertion Style ~~~~~~~~~~~~~~~ Always place the **actual** (test) value on the **left** and the **expected** value on the **right** in assertions: .. code-block:: python # Convention assert actual_value == expected_value # Not conventional assert expected_value == actual_value Some tests are marked as ``remote`` when they require downloading data from remote sources (e.g., DANDI Archive). Running Tests Locally ~~~~~~~~~~~~~~~~~~~~~ .. note:: Installing with ``[all]`` extra includes the ``dandi`` optional dependencies needed only for remote tests. First, install the package with test dependencies: .. code-block:: bash pip install -e ".[all]" --group test For coverage reporting, also add the ``coverage`` group: .. code-block:: bash pip install -e ".[all]" --group test --group coverage Run non-remote tests (does not require network): .. code-block:: bash pytest -m "not remote" -vv Run only remote tests: .. code-block:: bash pytest -m "remote" -vv Run tests with coverage: .. code-block:: bash pytest -m "not remote" -vv --cov=nwb2bids --cov-report html The coverage report will be generated in ``htmlcov/index.html``. Code Quality (Pre-commit Hooks) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This project uses `pre-commit `_ to run code quality checks automatically before each commit. The hooks include **black** (formatting), **ruff** (linting), **mypy** (type checking), and **codespell** (spell checking). Install pre-commit and the hooks once: .. code-block:: bash pip install pre-commit pre-commit install Run all hooks manually against all files (recommended before opening a PR): .. code-block:: bash pre-commit run --all-files Pre-commit will attempt to auto-fix most formatting and lint issues. After it modifies files, re-run the command to confirm everything passes: .. code-block:: bash pre-commit run --all-files # re-run to verify all hooks pass after auto-fixes Resolving MyPy errors ^^^^^^^^^^^^^^^^^^^^^ MyPy errors must be fixed manually. Common patterns: - **Missing return type annotation**: add a return type to the function signature (e.g., ``-> None``, ``-> str``). - **Unexpected type**: narrow the type with a guard clause or cast (e.g., ``if value is None: return``). - **Missing stub package**: install the missing stubs (e.g., ``pip install types-PyYAML``) or add ``# type: ignore[import-untyped]``. Run mypy on its own to iterate quickly: .. code-block:: bash mypy src/ tests/ Building Docs Locally ~~~~~~~~~~~~~~~~~~~~~ .. include:: README.md :parser: myst_parser.sphinx_ :start-line: 2 Documentation Tests ~~~~~~~~~~~~~~~~~~~ Tutorial code blocks are tested using `sybil `_ to ensure examples stay in sync with the codebase. Run all doc tests: .. code-block:: bash pytest docs/ -v For debugging, you must specify each code block you want to run by line number: .. code-block:: bash # List available doc tests with their line numbers pytest docs/tutorials.rst --collect-only # Run specific code blocks. Always use column:1. pytest "docs/tutorials.rst::line:30,column:1" \ "docs/tutorials.rst::line:158,column:1" \ "docs/tutorials.rst::line:183,column:1" -v Hidden assertions use ``.. invisible-code-block: python`` directives which run during testing but don't render in the documentation. CI Troubleshooting ~~~~~~~~~~~~~~~~~~ For debugging CI failures interactively, use the **Custom dispatch tests** workflow which supports `tmate `_ debugging sessions. 1. Go to `Custom dispatch tests workflow `_. 2. Click **"Run workflow"**. 3. Select the desired OS and Python version from the dropdowns. 4. Check **"Enable tmate debugging session"**. 5. Click **"Run workflow"** to start. 6. Monitor the workflow run. When it reaches the "Setup tmate session" step, it will display an SSH command like:: ssh randomstring@nyc1.tmate.io 7. Use this command to connect to the CI environment. The session runs under `tmux `_. Quick reference: - ``Ctrl-b ?`` - show help with all keybindings - ``Ctrl-b d`` - detach from session (workflow continues) - ``Ctrl-b c`` - create new window - ``Ctrl-b n`` / ``Ctrl-b p`` - next/previous window When you exit tmux (``exit`` or ``Ctrl-d``), the workflow continues to completion. Container CLI Testing ~~~~~~~~~~~~~~~~~~~~~ CLI tests can be run against a Docker container to verify the packaged application works correctly. First, build the dev container: .. code-block:: bash docker build -f containers/Dockerfile.dev -t nwb2bids:dev . Run CLI tests against the container: .. code-block:: bash pytest -m container_cli_test -v --container-image=nwb2bids:dev This runs all tests marked with ``@pytest.mark.container_cli_test`` inside the specified container. The test fixture automatically handles volume mounts and environment setup. Releasing --------- This repo uses Auto (https://github.com/intuit/auto) with ``hatch-vcs`` to cut releases from PR labels. The workflow is as follows: - One-time setup: run the GitHub Action "Setup Release Labels" to create Auto's full default label set in this repo (includes the semver labels and other labels used by Auto and enabled plugins). - For a release PR: add both of the following labels to the PR before merging: - One semver label: ``major``, ``minor``, or ``patch`` (controls version bump). - The ``release`` label (authorizes publishing). - After merge to ``main``, the **"Release with Auto"** workflow will: - Compute the next version from labels on merged PRs. - Create and push a Git tag (e.g., ``v1.2.3``) and a GitHub Release with a changelog. - When the GitHub Release is published, the existing **"Upload Package to PyPI"** workflow builds from that tag and uploads to PyPI. The version is derived from the Git tag via ``hatch-vcs``. .. note:: Only PRs with the ``release`` label will trigger a release; other PRs are collected until a release-labeled PR is merged. You can trigger a release manually via the ``Release with Auto`` workflow dispatch if needed. If labels ever get out of sync, restore and rerun the ``Setup Release Labels`` workflow to re-seed the full set. Changelog ~~~~~~~~~ The `CHANGELOG.md `_ file is auto-generated from merged PRs via the same Auto release process described above. Each entry is created from the PR title and categorized by the labels applied to the PR. Custom entries can be created by using the PR description with the following body: .. code-block:: markdown # What Changed ### Release Notes [ Enter your custom changelog entry here. ] Refer to `PR #244 `_ and corresponding `v0.9.0 Release Notes `_ for an example of this behavior. **For stylistic consistency**, please name all PR titles using the past tense (e.g., "Fix bug about..." -> "Fixed bug about...") since these titles become changelog entries. The label interactions leading to changelog sections are roughly as follows: .. container:: table-wrapper .. list-table:: :header-rows: 1 * - PR labels - Changelog section - Assign semantic label - Considerations * - ``enhancement`` - 🚀 Enhancement - ``minor`` - A new feature. * - ``bug`` - 🐛 Bug Fix - ``patch`` - Fixed a problem. * - ``documentation`` - 📝 Documentation - None [#documentation-note]_ - Only changed something about the documentation. * - ``internal`` - 🏠 Internal - None [#internal-note]_ - Only changed something about the testing infrastructure or CI (including docs), but no changes to source code. * - ``dependencies`` - 🔩 Dependency Updates - None - Only changed the dependencies of internal workings, such as the CI or docs. If altering the dependencies of the source code, consider it a ``patch`` without a related ``bug``. .. [#documentation-note] If given a semantic version such as ``patch`` (*e.g.*, `PR #355 `_ had both ``documentation`` and ``patch``, but was automatically placed under `"Bug Fix" `_) .. [#internal-note] If given a semantic version such as ``patch`` (*e.g.*, `PR #357 `_ had both ``internal`` and ``patch``, but was automatically placed under `"Bug Fix" `_)