To continually enhance the AI’s recommendations, we collect anonymous data on the environments generated. This feedback helps us train better models and improve accuracy.
Of course, your privacy matters—if you prefer not to participate, just add the --disable-telemetry flag when generating environments.
We also adhere to the donottrack standard.
In February 2020, I went on a 16-day, 1200km moped trip across northern Thailand with a couple of friends.
Somewhere in northern Thailand near Pai.
As we drove for hours on end, I was listening to an audiobook fittingly called Crossing the Chasm.
The book explores the challenges faced by nacent technologies on their way to mainstream adoption.
In the years that followed, I couldn’t help noticing the disconnect between Nix’s devoted user base
and its apparent lack of widespread adoption in the broader tech community.
Over 2021 and 2022, I focused my efforts and started nix.dev, a resource for practical and accessible Nix tutorials, thinking that good documentation was the thing holding Nix back.
Eventually, I came to the realization that improving documentation alone will only get us so far.
We needed to fix the foundations.
We needed to remove things to reduce the cognitive load when using Nix.
Over the years at Cachix we’ve talked to team after team
abandoning Nix and have observed a surprisingly consistent pattern.
Nix is initially introduced by someone enthusiastic about the technology. Then, faced with
a steep adoption curve, it is abandoned after backlash from the rest of the team.
Making it trivial for a project to adopt and maintain a development environment
is crucial for other team members to see the benefits of Nix.
For example, Shopify was vocal about Nix way back in 2020,
but eventually went quiet. Having companies like Shopify adopt Nix would be a major step forward for the whole ecosystem.
An interface as the heart of the Developer Experience
With the introduction of tasks in the 1.2 release and
Nix caching in 1.3, we’re pretty happy with
the devenv command-line interface and the extensible nature of the module system.
The modular architecture of the module system allows for seamless addition, modification, and removal of configuration options.
This flexibility extends to defining your own options for the software you’re writing.
We’ve been using the Nix command-line interface under the hood as a low-level API to
the evaluator and Nix store. We would’ve preferred to use something akin to an SDK instead, however
the command-line interface was the most sensible interface two years ago out of the available options.
The new C FFI (Foreign Function Interface) could potentially grow into a viable solution, but
it would necessitate substantial development effort and still leave us vulnerable to memory-safety
issues. Moreover, the architecture of the Nix codebase is structured more as a monolithic framework rather than a modular library.
Ideally, if we’re committing to fixing the developer experience over the next years, we’d want to have Nix implemented as a library in Rust.
Fortunately, such a project already exists and it’s called Tvix. Started by flokli and tazjin in “Rewriting Nix”,
Tvix is a re-implementation of Nix in Rust, offering both memory-safety and a library-oriented architecture with independently usable components. Leveraging Rust’s
abstractions and ecosystem (e.g. tracing.rs), Tvix is positioned to significantly enhance the developer experience for devenv developers and users.
There are many architectural differences besides the obvious “Rewrite In Rust” cliche,
so we’ll talk about them as we start replacing our Nix command-line calls with Tvix libraries, starting with the evaluator.
The Nix evaluator directly traverses the abstract syntax tree (AST) during evaluation, while Tvix uses a bytecode virtual machine crafted
according to the Crafting Interpreters book.
Tvix compiles Nix code into compact bytecode, then executes it in a virtual machine. This two-step approach offers potential performance benefits and optimization opportunities,
like many other interpreted languages.
When you re-evaluate devenv.nix, you’re most likely changing devenv.nix and not one of the few dozen Nix files that come from the devenv repository,
or even the few thousand Nix files from the nixpkgs repository that could all be cached as bytecode.
In order to integrate the Tvix evaluator with devenv we’ll need to:
Finish implementing builtins.fetchTree, where we have some ideas on how to simplify the caching layer
and get rid of the annoying dependency on GitHub’s rate-limited api.github.com endpoint.
Implement an evaluation debugger that will allow inspecting a program’s state in case of errors.
Finish implementing tvix-eval-jobs that will be used for regression tests against nixpkgs
to make sure that the evaluator behaves correctly.
Create debug tooling for when we discover regressions in the evaluator.
Once we’ve integrated the evaluator, we can finally generalize building languages using Nix reproducible builds
by running the underlying build system to generate Nix expressions:
In the general case, the dependency graph is computed statically, but a dependency can
declare its dependencies dynamically as part of the build by returning more Nix code.
That’s when evaluation and build phases start to mix, with evaluation depending on the result of a build,
which is typically called import from derivation (as the naming comes from the implementation).
sequenceDiagram
autonumber
participant NixEvaluator as Nix evaluator
participant NixStore as Nix store
NixEvaluator->>NixEvaluator: evaluate
NixEvaluator->>NixStore: write derivation
NixStore->>NixStore: build
NixStore->>NixEvaluator: read derivation output
NixEvaluator->>NixEvaluator: evaluate
Since evaluation in Nix is single-threaded, the process described above gets blocked on each build requested during evaluation.
Implementing parallel evaluation in Tvix, after we figure out the architectural details of how it should work, will unlock the ability to support automatic conversion
of language-specific build systems into Nix without sacrificing neither the developer experience, nor memory safety.
As we embark on this new chapter with Tvix, I’m reminded of the journey that brought us here.
It’s been a decade since I wrote the we can do better blog post,
highlighting the potential for improvement in configuration management and development environments, and I’m glad to see it all finally coming together.
Keep an eye out for updates and join the discussion:
Nix’s built-in flake evaluation caches outputs based on the lock of the inputs,
ignoring changes to Nix evaluation that often happen during development workflow.
Let’s take a closer look at how devenv’s new caching system compares to other popular tools in the Nix ecosystem.
Running our own cache gives us more control and visibility over the caching process, and allows us to improve our integration with other tools, like direnv.
While lorri pioneered the approach of parsing Nix’s internal logs for caching,
devenv builds on this concept, integrating caching as a built-in feature that works automatically without additional setup.
nix develop currently remains the last bit that’s rather slow and uncacheable, particularly on macOS.
We’re working on bringing its functionality in-house to further bring down the overhead of launching a cached shell to under 100ms.
For devenv, our mission is to make Nix the ultimate tool for managing developer environments. Nix
excels at congruent configuration,
where the system state is fully described by declarative code.
However, the real world often throws curveballs. Side-effects like database migrations, one-off
tasks such as data imports, or external API calls don’t always fit neatly into this paradigm.
In these cases, we often resort to convergent configuration,
where we define the desired end-state and let the system figure out how to get there.
To bridge this gap and make Nix more versatile, we’re introducing tasks. These allow you to
handle those pesky real-world scenarios while still leveraging Nix’s powerful ecosystem.
We’ve talked to many teams that dropped Nix after a while and they usually fit into two categories:
Maintaining Nix was too complex and the team didn’t fully onboard, creating friction inside the teams.
Went all-in Nix and it took a big toll on the team productivity.
While devenv already addresses (1), bridging the gap between Nix provided developer environments
and existing devops tooling written in your favorite language is still an unsolved problem until now.
We’ve designed Task Server Protocol so that you can write tasks
using your existing automation by providing an executable that exposes the tasks to devenv:
In a few weeks we’re planning to provide Rust TSP SDK
with a full test suite so you can implement your own abstraction in your language of choice.
You can now use your preferred language for automation, running tasks with a simple devenv tasks run <names> command. This
flexibility allows for more intuitive and maintainable scripts, tailored to your team’s familiarity.
For devenv itself, we’ll slowly transition from bash to Rust for
internal glue code, enhancing performance and reliability. This change will make devenv more
robust and easier to extend, ultimately providing you with a smoother development experience.
If you import another devenv.nix file, the outputs will be merged together,
allowing you to compose a developer environment and outputs in one logical unit.
You could also import outputs from other applications as inputs instead of composing them.
