• 3 min read

devenv 1.3: Instant developer environments with Nix caching

Hot on the heels of the previous release of tasks, we're releasing devenv 1.3! 🎉

This release brings precise caching to Nix evaluation, significantly speeding up developer environments.

Once cached, the results of a Nix eval or build can be recalled in single-digit milliseconds.

If any of the automatically-detected inputs change, the cache is invalidated and the build is performed.

Caching comparison

Note

If you run into any issues, run devenv with --refresh-eval-cache and report an issue.

How does it work?

Behind the scenes, devenv now parses Nix's internal logs to determine which files and directories were accessed during evaluation.

This approach is very much inspired by lorri, but doesn't require a daemon running in the background.

The caching process works as follows:

  1. During Nix evaluation, devenv parses the Nix logs for any files and directories that are accessed.
  2. For each accessed path, we store:
  3. the full path
  4. a hash of the file contents
  5. the last modification timestamp

This metadata is then saved to a SQLite database for quick retrieval.

When you run a devenv command, we:

  1. Check the database for all previously accessed paths
  2. Compare the current file hashes and timestamps to the stored values
  3. If any differences are detected, we invalidate the cache and perform a full re-evaluation
  4. If no differences are found, we use the cached results, significantly speeding up the process

This approach allows us to efficiently detect changes in your project, including:

  • Direct modifications to Nix files
  • Changes to imported files or directories
  • Updates to files read using Nix built-ins, like readFile or readDir

Comparison with Nix's built-in flake evaluation cache

Nix's built-in flake evaluation caches outputs based on the lock of the inputs, ignoring changes to Nix evaluation that often happen during development workflow.

Comparison with existing tools

Let's take a closer look at how devenv's new caching system compares to other popular tools in the Nix ecosystem. Running our own cache gives us more control and visibility over the caching process, and allows us to improve our integration with other tools, like direnv.

lorri

While lorri pioneered the approach of parsing Nix's internal logs for caching, devenv builds on this concept, integrating caching as a built-in feature that works automatically without additional setup.

direnv and nix-direnv

These tools excel at caching evaluated Nix environments, but have limitations in change detection:

  • Manual file watching: Users often need to manually specify which files to watch for changes.
  • Limited scope: They typically can't detect changes in deeply nested imports or files read by Nix built-ins.

To leverage devenv's caching capabilities with direnv, we've updated the .envrc file to utilize devenv's new caching logic.

If you currently enjoy the convenience of our direnv integration to reload your development environment, make sure to update your .envrc to:

source_url "https://raw.githubusercontent.com/cachix/devenv/82c0147677e510b247d8b9165c54f73d32dfd899/direnvrc" "sha256-7u4iDd1nZpxL4tCzmPG0dQgC5V+/44Ba+tHkPob1v2k="

use devenv

to benefit from the new caching system.

What's next?

nix develop currently remains the last bit that's rather slow and uncacheable, particularly on macOS. We're working on bringing its functionality in-house to further bring down the overhead of launching a cached shell to under 100ms.

Join us on Discord if you have any questions,

Domen & Sander

  • 2 min read

devenv 1.2: Tasks for convergent configuration with Nix

For devenv, our mission is to make Nix the ultimate tool for managing developer environments. Nix excels at congruent configuration, where the system state is fully described by declarative code.

However, the real world often throws curveballs. Side-effects like database migrations, one-off tasks such as data imports, or external API calls don't always fit neatly into this paradigm. In these cases, we often resort to convergent configuration, where we define the desired end-state and let the system figure out how to get there.

To bridge this gap and make Nix more versatile, we're introducing tasks. These allow you to handle those pesky real-world scenarios while still leveraging Nix's powerful ecosystem.

Tasks interactive example

Usage

For example if you'd like to execute python code after virtualenv has been created:

devenv.nix
{ pkgs, lib, config, ... }: {
  languages.python.enable = true;
  languages.python.venv.enable = true;

  tasks = {
    "python:setup" = {
      exec = "python ${pkgs.writeText "setup.py" ''
          print("hello world")
      ''}";
      after = [ "devenv:python:virtualenv" ];
    };
    "devenv:enterShell".after = [ "python:setup" ];
  };
}

python:setup task executes before devenv:enterShell but after python:virtualenv task:

For all supported use cases see tasks documentation.

Task Server Protocol for SDKs

We've talked to many teams that dropped Nix after a while and they usually fit into two categories:

  • 1) Maintaining Nix was too complex and the team didn't fully onboard, creating friction inside the teams.
  • 2) Went all-in Nix and it took a big toll on the team productivity.

While devenv already addresses (1), bridging the gap between Nix provided developer environments and existing devops tooling written in your favorite language is still an unsolved problem until now.

We've designed Task Server Protocol so that you can write tasks using your existing automation by providing an executable that exposes the tasks to devenv:

devenv.nix
{ pkgs, ... }:
let
  myexecutable = pkgs.rustPlatform.buildRustPackage rec {
    pname = "foo-bar";
    version = "0.1";
    cargoLock.lockFile = ./myexecutable/Cargo.lock;
    src = pkgs.lib.cleanSource ./myexecutable;
  }
in {
  task.serverProtocol = [ "${myexecutable}/bin/myexecutable" ];
}

In a few weeks we're planning to provide Rust TSP SDK with a full test suite so you can implement your own abstraction in your language of choice.

You can now use your preferred language for automation, running tasks with a simple devenv tasks run <names> command. This flexibility allows for more intuitive and maintainable scripts, tailored to your team's familiarity.

For devenv itself, we'll slowly transition from bash to Rust for internal glue code, enhancing performance and reliability. This change will make devenv more robust and easier to extend, ultimately providing you with a smoother development experience.

Upgrading

If you run devenv update on your existing repository you should already be using tasks, without needing to upgrade to devenv 1.2.

Domen

  • 2 min read

devenv 1.1: Nested Nix outputs using the module system

devenv 1.1 brings support for Nix outputs, matching the last missing piece of functionality with Flakes.

It was designed to make outputs extensible, nested, and buildable as a whole by default.

This allows exposing Nix packages for installation/consumption by other tools.

Nested Nix outputs

If you have a devenv with outputs like this:

devenv.nix
{ pkgs, ... }: {
  outputs = {
    myproject.myapp = import ./myapp { inherit pkgs; };
    git = pkgs.git;
  };
}

You can build all outputs by running:

$ devenv build
/nix/store/mzq5bpi49h26cy2mfj5a2r0q69fh3a9k-git-2.44.0
/nix/store/mzq5bpi49h26cy2mfj5a2r0q71fh3a9k-myapp-1.0

Or build specific attribute(s) by listing them explicitly:

$ devenv build outputs.git
/nix/store/mzq5bpi49h26cy2mfj5a2r0q69fh3a9k-git-2.44.0

This is useful for tools that need to find and install specific outputs.

Defining outputs as module options

By default, any derivation specified in outputs nested attributes set is recognized as an output.

You can define custom options as output types in devenv. These will be automatically detected and built:

devenv.nix
{ pkgs, lib, config, ... }: {
  options = {
    myapp.package = lib.mkOption {
        type = config.lib.types.outputOf lib.types.package;
        description = "The package for myapp";
        default = import ./myapp { inherit pkgs; };
        defaultText = "myapp-1.0";
    };
  };

  config = {
    outputs.git = pkgs.git;
  }
}

Building will pick up all outputs, in this case myapp.package and outputs.git:

$ devenv build
/nix/store/mzq5bpi49h26cy2mfj5a2r0q69fh3a9k-myapp-1.0
/nix/store/mzq5bpi49h26cy2mfj5a2r0q69fh3a9k-git-2.44.0

If you don't want to specify the output type, you can just use config.lib.types.output.

Referencing outputs from another devenv

If you import another devenv.nix file, the outputs will be merged together, allowing you to compose a developer environment and outputs in one logical unit.

You could also import outputs from other applications as inputs instead of composing them.

Leave a thumbs on the issue if you'd like to see it happen.

Documentation

See Outputs section in documentation for the latest comprehensive guide to outputs.

We're on Discord if you need help, Domen

  • 5 min read

devenv 1.0: Rewrite in Rust

We have just released devenv 1.0! 🎉

This is a rewrite of the CLI to Python Rust, which brings with it many new features and improvements.

I would like to thank mightyiam for a week-long, Rust pair-programming session at Thaiger Sprint.

Note: Read the migration guide at the end of this post, as 1.0 is not entirely backwards compatible.

Why rewrite twice?

When I started to write this blog post for the Python rewrite, I came up with only excuses as to why it is not fast and realized that we were simply breaking our promise to you.

The second reason is that in the Nix community there has been a lot of controversy surrounding flakes (that's for another blog post); two years ago, the tvix developers decided to do something about it and started a rewrite of Nix in Rust. This leaves us with the opportunity in the future to use the same Rust libraries and tooling.

What's new?

There are many contributions in this release, spanning over a year, but here are some of the highlights:

process-compose is now the default process manager

devenv up is now using process-compose, as it handles dependencies between processes and provides a nice ncurses interface to view the processes and their logs.

Testing infrastructure

Testing has been a major focus of this release, and a number of features have been added to make it easier to write and run tests.

The new enterTest attribute in devenv.nix allows you to define testing logic:

{ pkgs, ... }: {
  packages = [ pkgs.ncdu ];

  services.postgres = {
    enable = true;
    listen_addresses = "127.0.0.1";
    initialDatabases = [{ name = "mydb"; }];
  };

  enterTest = ''
    wait_for_port 5432
    ncdu --version | grep "ncdu 2.2"
  '';
}

When you run devenv test, it will run the enterTest command and report the results.

If you have any processes defined, they will be started and stopped.

Read more about this in the testing documentation.

This allows for executing tests with all of your tooling and processes running—extremely convenient for integration and functional tests.

devenv-nixpkgs

Since nixpkgs-unstable has fairly few tests, we have created devenv-nixpkgs to run tests on top of nixpkgs-unstable—applying patches we are upstreaming to address any issues.

We run around 300 tests across different languages and processes to ensure all regressions are caught.

Non-root containers

Generated containers now run as a plain user—improving security and unlocking the ability to run software that forbids root.

DEVENV_RUNTIME

Due to socket path limits, the DEVENV_RUNTIME environment variable has been introduced: pointing to $XDG_RUNTIME_DIR by default and falling back to /tmp.

First-class support for Python native libraries

This one was the hardest nut to crack.

Nix is known to provide a poor experience when using tools like pip.

A lot of work has been put in here, finally making it possible to use native libraries in Python without any extra effort:

{ pkgs, lib, ... }: {
  languages.python = {
    enable = true;
    venv.enable = true;
    venv.requirements = ''
      pillow
    '';
    libraries = [ pkgs.cairo ];
  };
}

CLI improvements

If you need to add an input to devenv.yaml, you can now do:

devenv inputs add <name> <url>

To update a single input:

devenv update <input>

To build any attribute in devenv.nix:

devenv build languages.rust.package

To run the environment as cleanly as possible while keeping specific variables:

devenv shell --clean EDITOR,PAGER

The default number of cores has been tweaked to 2, and max-jobs to half of the number of CPUs. It is impossible to find an ideal default, but we have found that too much parallelism hurts performance—running out of memory is a common issue.

... plus a number of other additions:

https://devenv.sh 1.0.0: Fast, Declarative, Reproducible, and Composable Developer Environments

Usage: devenv [OPTIONS] <COMMAND>

Commands:
  init       Scaffold devenv.yaml, devenv.nix, .gitignore and .envrc.
  shell      Activate the developer environment. https://devenv.sh/basics/
  update     Update devenv.lock from devenv.yaml inputs. http://devenv.sh/inputs/
  search     Search for packages and options in nixpkgs. https://devenv.sh/packages/#searching-for-a-file
  info       Print information about this developer environment.
  up         Start processes in the foreground. https://devenv.sh/processes/
  processes  Start or stop processes.
  test       Run tests. http://devenv.sh/tests/
  container  Build, copy, or run a container. https://devenv.sh/containers/
  inputs     Add an input to devenv.yaml. https://devenv.sh/inputs/
  gc         Deletes previous shell generations. See http://devenv.sh/garbage-collection
  build      Build any attribute in devenv.nix.
  version    Print the version of devenv.
  help       Print this message or the help of the given subcommand(s)

Options:
  -v, --verbose
          Enable debug log level.
  -j, --max-jobs <MAX_JOBS>
          Maximum number of Nix builds at any time. [default: 8]
  -j, --cores <CORES>
          Maximum number CPU cores being used by a single build.. [default: 2]
  -s, --system <SYSTEM>
          [default: x86_64-linux]
  -i, --impure
          Relax the hermeticity of the environment.
  -c, --clean [<CLEAN>...]
          Ignore existing environment variables when entering the shell. Pass a list of comma-separated environment variables to let through.
  -d, --nix-debugger
          Enter Nix debugger on failure.
  -n, --nix-option <NIX_OPTION> <NIX_OPTION>
          Pass additional options to nix commands, see `man nix.conf` for full list.
  -o, --override-input <OVERRIDE_INPUT> <OVERRIDE_INPUT>
          Override inputs in devenv.yaml.
  -h, --help
          Print help

Migration guide

Deprecations

  • devenv container --copy <name> has been renamed to devenv container copy <name>.
  • devenv container --docker-run <name> has been renamed to devenv container run <name>.
  • devenv ci has been renamed to devenv test with a broader scope.

Breaking changes

  • .env files must start with the .env prefix.
  • The need for the --impure flag has finally been removed, meaning that devenv is now fully hermetic by default.

Things like builtins.currentSystem no longer work—you will have to use pkgs.stdenv.system.

If you need to relax the hermeticity of the environment you can use devenv shell --impure.

  • Since the format of devenv.lock has changed, newly-generated lockfiles cannot be used with older versions of devenv.

Looking ahead

There are a number of features that we are looking to add in the future—please vote on the issues:

Running devenv in a container

While devenv is designed to be run on your local machine, we are looking to add support for running devenv inside a container.

Something like:

devenv shell --in-container
devenv test --in-container

This would be convenient when the environment is too complex to set up on your local machine; for example, when running two databases or when you want to run tests in a clean environment.

Generating containers with full environment

Currently, enterShell is executed only once the container has started. If we want to execute it as part of the container generation, we have to execute it inside a container to generate a layer.

macOS support for generating containers

Building containers on macOS is not currently supported, but it should be possible.

Native mapping of dependencies

Wouldn't it be cool if devenv could map language-specific dependencies to your local system? In this example, devenv should be able to determine that pillow requires pkgs.cairo:

{ pkgs, lib, ... }: {
  languages.python = {
    enable = true;
    venv.enable = true;
    venv.requirements = ''
      pillow
    '';
  };
}

Voilà

Give devenv a try, and hop on to our discord to let us know how it goes!

Domen

  • 3 min read

devenv 0.6: Generating containers and instant shell activation

After about two months of active development, I'm happy to announce devenv 0.6 is ready.

This release comes with the most notable improvements based on the feedback from existing users:

Generating containers

While devenv shell provides a simple native developer environment experience, devenv container <name> allows you to generate and copy OCI container into a registry.

Containers are a great way to distribute ready-made applications, leveraging platforms like fly.io to deploy them into production.

An example for Ruby:

devenv.nix
{
  name = "simple-ruby-app";

  languages.ruby.enable = true;
  languages.ruby.version = "3.2.1";
}

We can generate a container called shell that enters the environment, copy it to the local Docker daemon and run it:

$ devenv container shell --docker-run
...
(devenv) bash-5.2# ruby --version
ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux]

You can read more in the new Containers section of the documentation, specifically:

Instant shell activation

Especially monorepo developer environments can sometimes be even a few gigabytes of size, taking a few seconds for the environment to be activated.

A developer environment should only be built when something changes and if not, the environment can be used instantly using a cached snapshot.

With the latest direnv.net integration, we've finally reached that goal by making caching work properly (it will even watch each of your imports for changes!).

Migrating from an older devenv

Make sure to use the latest .envrc from devenv init and for everyone on the team to upgrade to devenv 0.6.

In the near future we'll experiment to improve devenv shell experience.

Hosts and certificates

Hosts and certificates can now be specified declaratively:

{ pkgs, config, ... }:

{
  certificates = [
    "example.com"
  ];

  hosts."example.com" = "127.0.0.1";

  services.caddy.enable = true;
  services.caddy.virtualHosts."example.com" = {
    extraConfig = ''
      tls ${config.env.DEVENV_STATE}/mkcert/example.com.pem ${config.env.DEVENV_STATE}/mkcert/example.com-key.pem
      respond "Hello, world!"
    '';
  };
}

And when you run devenv up to start the processes, these hosts and certificates will be provisioned locally.

allowUnfree and overlays

For example in devenv.yaml:

allowUnfree: true
inputs:
  nixpkgs:
    url: github:NixOS/nixpkgs/nixpkgs-unstable
  rust-overlay:
    url: github:oxalica/rust-overlay
    overlays:
      - default

Will allow building unfree software and wire up default overlay into pkgs from rust-overlay.

Migrating from an older devenv

Make sure everyone on the team upgrades to devenv 0.6.

Languages changelog

  • Python: Added support for virtualenv creation and poetry by bobvanderlinden.
  • Ruby: First-class support for setting version or versionFile by bobvanderlinden.
  • Go: Received significant improvements by shyim.
  • PHP: Added first-class support for setting version to make it easier to set extensions by shyim.
  • Scala: Now allows changing the package and offers scala-cli as an option if the JDK is too old by domenkozar.
  • R: Added an option to specify the package by adfaure.
  • Rust: Can now find headers for darwin frameworks by domenkozar.
  • OCaml: Allowed using a different version of OCaml by ankhers.
  • Tex Live: Added support by BurNiinTRee.
  • Swift: Added support by domenkozar.
  • Raku: Added support by 0pointerexception.
  • Gawk: Added support by 0pointerexception.
  • Racket: Added support by totoroot.
  • Dart: Added support by domenkozar.
  • Julia: Added support by domenkozar.
  • Crystal: Added support by bcardiff.
  • Unison: Added support by ereslibre.
  • Zig: Added support by ereslibre.
  • Deno: Added support by janathandion.

Services changelog

  • Cassandra: Added by ankhers.

  • CouchDB: Added by MSBarbieri.

  • MariaDB: Corrected user and database handling by jochenmanz.

  • MinIO: Now allows specifying what buckets to provision by shyim.

Fixed issues and other improvements

  • process-compose: Faster shutdown, restart on failure by default, escape env variables properly by thenonameguy.

  • Support assertions in modules by bobvanderlinden.

  • Fix overmind root by domenkozar.

  • Make devenv info output pluggable from devenv modules by domenkozar.

  • Expand the flake guide by sandydoo.

  • Set LOCALE_ARCHIVE when missing by sandydoo.

  • Numerous option documentation fixes by sandydoo.

  • Fix starship integration with a custom config by domenkozar.

  • Test direnv integration with strict bash mode by stephank.

  • Add a shim devenv for flakes integration by rgonzalez.

  • 2 min read

devenv 0.5

Highlights

  • devenv search now shows results from the options that can be set in devenv.nix:

devenv search results

Languages

  • Rust language support now integrates with fenix to provide stable/nightly/unstable toolchain for cargo, rustc, rust-src, rust-fmt, rust-analyzer and clippy.

  • Python language now sets $PYTHONPATH to point to any installed packages in packages attribute.

  • Ruby langauge support now defaults to the latest version 3.1.x, ships with an example running rails, sets $GEM_HOME and $GEM_PATH environment variables. Next release will support picking any version of Ruby - please leave a thumbs up.

  • jpetrucciani contributed Nim, V and HCL/Terraform languages support.

Services

Integrations

Bug fixes

  • 1 min read

devenv 0.4

Highlights

  • New command devenv info shows locked inputs, environment variables, scripts, processes and packages exposed in the environment.

  • Tracebacks are now printed with most relevent information at the bottom.

  • New option process.implementation allows you to choose how processes are run. New supported options are overmind and process-compose.

  • Instead of passing each input separately in devenv.nix, the new prefered and documented way is via inputs argument, for example inputs.pre-commit-hooks.

  • samjwillis97 contributed support for MongoDB.

  • shyim contributed MySQL/MariaDB support.

  • shyim made PHP configuration more configurable, for example you can now set extensions.

  • JanLikar improved PostgreSQL support to expose psql-devenv script for connecting to the cluster.

  • datakurre added robotframework support.

Bug fixes

  • Composing using inputs has been fixed.

  • It's now possible to use devenv on directories with spaces.

  • Update checker is no longer using environment variables to avoid some corner cases.

  • 1 min read

devenv 0.3

It has been 3 days since 0.2 release, so it's time for 0.3:

Highlights

Bug fixes

Domen

  • 1 min read

devenv 0.2

After an intense weekend and lots of incoming contributions, v0.2 is out!

Highlights

  • All the devenv.nix options you can define now come as an input (instead of being packaged with each devenv release). To update the options you can run devenv update and it will match devenv.nix reference.

  • New devenv search command:

$ devenv search ncdu
name         version  description
pkgs.ncdu    2.1.2    Disk usage analyzer with an ncurses interface
pkgs.ncdu_1  1.17     Disk usage analyzer with an ncurses interface
pkgs.ncdu_2  2.1.2    Disk usage analyzer with an ncurses interface

Found 3 results.

  • shyim contributed Redis support and is working on MySQL.

  • Languages: raymens contributed dotnet, ankhers contributed Elixir and Erlang support.

  • If devenv.local.nix exists it's now also loaded, allowing you to override git committed devenv.nix with local changes. Hurrah composability!

Bug fixes

  • Variables like env.DEVENV_ROOT, env.DEVENV_STATE and env.DEVENV_DOTFILE are now absolute paths paths
  • shyim fixed /dev/stderr that is in some environments not available.
  • domen fixed shell exiting on non-zero exit status code.

Domen

  • 1 min read

Hello world: devenv 0.1

After lengthy conversations at NixCon 2022 about Developer Experience and current painpoints around documentation, I've started hacking and experimenting.

The goal is to bring the strengths of Nix to the world with what we have best to offer, and I'm happy to announce:

devenv: Fast, Declarative, Reproducible, and Composable Developer Environments

Local containerless environments

One of the reasons why developer environments are moving into the cloud are the lack of good tooling how to make those environments reproducible.

In the last decade we've doubled down on shipping binary blobs in containerized environments.

Just as we went from virtual machines to containers, we can make one step further and create guarantees at the package level and treat those as a building block.

devenv 0.1 release brings the basic building blocks for many possibilities of what can be built in the future.

I invite you to explore the documentation and give it a try.

Summary

I'm looking forward in what ways the developer community uses devenv and stay tuned for roadmap updates by subscribing at our newsletter at the bottom of the page.

Domen