To continually enhance the AI’s recommendations, we collect anonymous data on the environments generated. This feedback helps us train better models and improve accuracy.
Of course, your privacy matters—if you prefer not to participate, just add the --disable-telemetry flag when generating environments.
We also adhere to the donottrack standard.
In February 2020, I went on a 16-day, 1200km moped trip across northern Thailand with a couple of friends.
Somewhere in northern Thailand near Pai.
As we drove for hours on end, I was listening to an audiobook fittingly called Crossing the Chasm.
The book explores the challenges faced by nacent technologies on their way to mainstream adoption.
In the years that followed, I couldn't help noticing the disconnect between Nix's devoted user base
and its apparent lack of widespread adoption in the broader tech community.
Over 2021 and 2022, I focused my efforts and started nix.dev, a resource for practical and accessible Nix tutorials, thinking that good documentation was the thing holding Nix back.
Eventually, I came to the realization that improving documentation alone will only get us so far.
We needed to fix the foundations.
We needed to remove things to reduce the cognitive load when using Nix.
Over the years at Cachix we've talked to team after team
abandoning Nix and have observed a surprisingly consistent pattern.
Nix is initially introduced by someone enthusiastic about the technology. Then, faced with
a steep adoption curve, it is abandoned after backlash from the rest of the team.
Making it trivial for a project to adopt and maintain a development environment
is crucial for other team members to see the benefits of Nix.
For example, Shopify was vocal about Nix way back in 2020,
but eventually went quiet. Having companies like Shopify adopt Nix would be a major step forward for the whole ecosystem.
With the introduction of tasks in the 1.2 release and
Nix caching in 1.3, we're pretty happy with
the devenv command-line interface and the extensible nature of the module system.
The modular architecture of the module system allows for seamless addition, modification, and removal of configuration options.
This flexibility extends to defining your own options for the software you're writing.
We've been using the Nix command-line interface under the hood as a low-level API to
the evaluator and Nix store. We would've preferred to use something akin to an SDK instead, however
the command-line interface was the most sensible interface two years ago out of the available options.
The new C FFI (Foreign Function Interface) could potentially grow into a viable solution, but
it would necessitate substantial development effort and still leave us vulnerable to memory-safety
issues. Moreover, the architecture of the Nix codebase is structured more as a monolithic framework rather than a modular library.
Ideally, if we're committing to fixing the developer experience over the next years, we'd want to have Nix implemented as a library in Rust.
Fortunately, such a project already exists and it's called Tvix. Started by flokli and tazjin in "Rewriting Nix",
Tvix is a re-implementation of Nix in Rust, offering both memory-safety and a library-oriented architecture with independently usable components. Leveraging Rust's
abstractions and ecosystem (e.g. tracing.rs), Tvix is positioned to significantly enhance the developer experience for devenv developers and users.
There are many architectural differences besides the obvious "Rewrite In Rust" cliche,
so we'll talk about them as we start replacing our Nix command-line calls with Tvix libraries, starting with the evaluator.
The Nix evaluator directly traverses the abstract syntax tree (AST) during evaluation, while Tvix uses a bytecode virtual machine crafted
according to the Crafting Interpreters book.
Tvix compiles Nix code into compact bytecode, then executes it in a virtual machine. This two-step approach offers potential performance benefits and optimization opportunities,
like many other interpreted languages.
When you re-evaluate devenv.nix, you're most likely changing devenv.nix and not one of the few dozen Nix files that come from the devenv repository,
or even the few thousand Nix files from the nixpkgs repository that could all be cached as bytecode.
In order to integrate the Tvix evaluator with devenv we'll need to:
Finish implementing builtins.fetchTree, where we have some ideas on how to simplify the caching layer
and get rid of the annoying dependency on GitHub's rate-limited api.github.com endpoint.
Implement an evaluation debugger that will allow inspecting a program's state in case of errors.
Finish implementing tvix-eval-jobs that will be used for regression tests against nixpkgs
to make sure that the evaluator behaves correctly.
Create debug tooling for when we discover regressions in the evaluator.
Once we've integrated the evaluator, we can finally generalize building languages using Nix reproducible builds
by running the underlying build system to generate Nix expressions:
graph TD
A[devenv];
A -->|Rust| C[Cargo];
A -->|JavaScript| D[npm];
A -->|PHP| E[Composer];
C -->|Cargo.lock| F{Nix};
D -->|package.json| F{Nix};
E -->|composer.lock| F{Nix};
In the general case, the dependency graph is computed statically, but a dependency can
declare its dependencies dynamically as part of the build by returning more Nix code.
That's when evaluation and build phases start to mix, with evaluation depending on the result of a build,
which is typically called import from derivation (as the naming comes from the implementation).
sequenceDiagram
autonumber
participant NixEvaluator as Nix evaluator
participant NixStore as Nix store
NixEvaluator->>NixEvaluator: evaluate
NixEvaluator->>NixStore: write derivation
NixStore->>NixStore: build
NixStore->>NixEvaluator: read derivation output
NixEvaluator->>NixEvaluator: evaluate
Since evaluation in Nix is single-threaded, the process described above gets blocked on each build requested during evaluation.
Implementing parallel evaluation in Tvix, after we figure out the architectural details of how it should work, will unlock the ability to support automatic conversion
of language-specific build systems into Nix without sacrificing neither the developer experience, nor memory safety.
As we embark on this new chapter with Tvix, I'm reminded of the journey that brought us here.
It's been a decade since I wrote the we can do better blog post,
highlighting the potential for improvement in configuration management and development environments, and I'm glad to see it all finally coming together.
Keep an eye out for updates and join the discussion:
Nix's built-in flake evaluation caches outputs based on the lock of the inputs,
ignoring changes to Nix evaluation that often happen during development workflow.
Let's take a closer look at how devenv's new caching system compares to other popular tools in the Nix ecosystem.
Running our own cache gives us more control and visibility over the caching process, and allows us to improve our integration with other tools, like direnv.
While lorri pioneered the approach of parsing Nix's internal logs for caching,
devenv builds on this concept, integrating caching as a built-in feature that works automatically without additional setup.
These tools excel at caching evaluated Nix environments, but have limitations in change detection:
Manual file watching: Users often need to manually specify which files to watch for changes.
Limited scope: They typically can't detect changes in deeply nested imports or files read by Nix built-ins.
To leverage devenv's caching capabilities with direnv, we've updated the .envrc file to utilize devenv's new caching logic.
If you currently enjoy the convenience of our direnv integration to reload your development environment, make sure to update your .envrc to:
source_url "https://raw.githubusercontent.com/cachix/devenv/82c0147677e510b247d8b9165c54f73d32dfd899/direnvrc" "sha256-7u4iDd1nZpxL4tCzmPG0dQgC5V+/44Ba+tHkPob1v2k="
use devenv
nix develop currently remains the last bit that's rather slow and uncacheable, particularly on macOS.
We're working on bringing its functionality in-house to further bring down the overhead of launching a cached shell to under 100ms.
For devenv, our mission is to make Nix the ultimate tool for managing developer environments. Nix
excels at congruent configuration,
where the system state is fully described by declarative code.
However, the real world often throws curveballs. Side-effects like database migrations, one-off
tasks such as data imports, or external API calls don't always fit neatly into this paradigm.
In these cases, we often resort to convergent configuration,
where we define the desired end-state and let the system figure out how to get there.
To bridge this gap and make Nix more versatile, we're introducing tasks. These allow you to
handle those pesky real-world scenarios while still leveraging Nix's powerful ecosystem.
We've talked to many teams that dropped Nix after a while and they usually fit into two categories:
1) Maintaining Nix was too complex and the team didn't fully onboard, creating friction inside the teams.
2) Went all-in Nix and it took a big toll on the team productivity.
While devenv already addresses (1), bridging the gap between Nix provided developer environments
and existing devops tooling written in your favorite language is still an unsolved problem until now.
We've designed Task Server Protocol so that you can write tasks
using your existing automation by providing an executable that exposes the tasks to devenv:
In a few weeks we're planning to provide Rust TSP SDK
with a full test suite so you can implement your own abstraction in your language of choice.
You can now use your preferred language for automation, running tasks with a simple devenv tasks run <names> command. This
flexibility allows for more intuitive and maintainable scripts, tailored to your team's familiarity.
For devenv itself, we'll slowly transition from bash to Rust for
internal glue code, enhancing performance and reliability. This change will make devenv more
robust and easier to extend, ultimately providing you with a smoother development experience.
If you import another devenv.nix file, the outputs will be merged together,
allowing you to compose a developer environment and outputs in one logical unit.
You could also import outputs from other applications as inputs instead of composing them.
When I started to write this blog post for the Python rewrite, I came up with only excuses as to why it is not fast and realized that we were simply breaking our promise to you.
The second reason is that in the Nix community there has been a lot of controversy surrounding flakes (that's for another blog post); two years ago, the tvix developers decided to do something about it and started a rewrite of Nix in Rust. This leaves us with the opportunity in the future to use the same Rust libraries and tooling.
devenv up is now using process-compose,
as it handles dependencies between processes and provides a nice ncurses interface to view the processes
and their logs.
Since nixpkgs-unstable has fairly few tests,
we have created devenv-nixpkgs to run tests on top of nixpkgs-unstable—applying patches we are upstreaming to address any issues.
We run around 300 tests across different languages and processes to ensure all regressions are caught.
Due to socket path limits, the DEVENV_RUNTIME environment variable has been introduced: pointing to $XDG_RUNTIME_DIR by default and falling back to /tmp.
If you need to add an input to devenv.yaml, you can now do:
devenv inputs add <name> <url>
To update a single input:
devenv update <input>
To build any attribute in devenv.nix:
devenv build languages.rust.package
To run the environment as cleanly as possible while keeping specific variables:
devenv shell --clean EDITOR,PAGER
The default number of cores has been tweaked to 2, and max-jobs to half of the number of CPUs.
It is impossible to find an ideal default, but we have found that too much parallelism hurts performance—running out of memory is a common issue.
... plus a number of other additions:
https://devenv.sh 1.0.0: Fast, Declarative, Reproducible, and Composable Developer Environments
Usage: devenv [OPTIONS] <COMMAND>
Commands:
init Scaffold devenv.yaml, devenv.nix, .gitignore and .envrc.
shell Activate the developer environment. https://devenv.sh/basics/
update Update devenv.lock from devenv.yaml inputs. http://devenv.sh/inputs/
search Search for packages and options in nixpkgs. https://devenv.sh/packages/#searching-for-a-file
info Print information about this developer environment.
up Start processes in the foreground. https://devenv.sh/processes/
processes Start or stop processes.
test Run tests. http://devenv.sh/tests/
container Build, copy, or run a container. https://devenv.sh/containers/
inputs Add an input to devenv.yaml. https://devenv.sh/inputs/
gc Deletes previous shell generations. See http://devenv.sh/garbage-collection
build Build any attribute in devenv.nix.
version Print the version of devenv.
help Print this message or the help of the given subcommand(s)
Options:
-v, --verbose
Enable debug log level.
-j, --max-jobs <MAX_JOBS>
Maximum number of Nix builds at any time. [default: 8]
-j, --cores <CORES>
Maximum number CPU cores being used by a single build.. [default: 2]
-s, --system <SYSTEM>
[default: x86_64-linux]
-i, --impure
Relax the hermeticity of the environment.
-c, --clean [<CLEAN>...]
Ignore existing environment variables when entering the shell. Pass a list of comma-separated environment variables to let through.
-d, --nix-debugger
Enter Nix debugger on failure.
-n, --nix-option <NIX_OPTION> <NIX_OPTION>
Pass additional options to nix commands, see `man nix.conf` for full list.
-o, --override-input <OVERRIDE_INPUT> <OVERRIDE_INPUT>
Override inputs in devenv.yaml.
-h, --help
Print help
devenv shell --in-container
devenv test --in-container
This would be convenient when the environment is too complex to set up on your local machine; for example, when running two databases or when you want to run tests in a clean environment.
Wouldn't it be cool if devenv could map language-specific dependencies to your local system? In this example, devenv should be able to determine that pillow requires pkgs.cairo:
Especially monorepo developer environments can sometimes be even a few gigabytes of size, taking a few seconds for the environment to be activated.
A developer environment should only be built
when something changes and if not, the environment
can be used instantly using a cached snapshot.
With the latest direnv.net integration,
we've finally reached that goal by making caching work properly (it will even watch each of your imports for changes!).
Migrating from an older devenv
Make sure to use the latest .envrc from devenv init and for everyone on the team to upgrade to devenv 0.6.
In the near future we'll experiment to improve devenv shell experience.
Rust language support now integrates with fenix to provide stable/nightly/unstable toolchain for cargo, rustc, rust-src, rust-fmt, rust-analyzer and clippy.
Python language now sets $PYTHONPATH to point to any installed packages in packages attribute.
Ruby langauge support now defaults to the latest version 3.1.x, ships with an example running rails, sets $GEM_HOME and $GEM_PATH environment variables. Next release will support picking any version of Ruby - please leave a thumbs up.
New option process.implementation allows you to choose how processes are run. New supported options are overmind and process-compose.
Instead of passing each input separately in
devenv.nix, the new prefered and documented way is via inputs argument, for example inputs.pre-commit-hooks.