Under the hood, process-compose now runs processes through devenv-tasks run --mode all devenv:processes:<name> instead of executing them directly. This preserves all existing process functionality while adding task capabilities.
The --mode all flag ensures that both before and after tasks are executed, maintaining the expected lifecycle behavior.
Future work on process dependencies (#2037) will also address native health check support (process-compose#371), eliminating the need for manual polling scripts.
We've rewritten our tracing integration to improve reporting on what devenv is doing.
More importantly, devenv is now fully asynchronous under the hood, enabling parallel execution of operations. This means faster performance in scenarios where multiple independent tasks can run simultaneously.
The new progress interface provides real-time feedback on what devenv is doing:
We're continuing to improve visibility into Nix operations to give you even better insights into the build process.
We've integrated SecretSpec, a new standard for declarative secrets management that separates secret declaration from provisioning.
This allows teams to define what secrets applications need while letting each developer, CI system, and production environment provide them from their preferred secure provider.
You can now run tasks at any level in the hierarchy. By default, tasks run in single mode (only the specified task):
# Run only frontend:build (default single mode)$devenvtasksrunfrontend:build
Runningtasksfrontend:build
Succeededfrontend:build5ms
1Succeeded5.75ms
# Run frontend:build with all its dependencies (before mode)$devenvtasksrunfrontend:build--modebefore
Runningtasksfrontend:build
Succeededfrontend:lint4ms
Succeededfrontend:test10ms
Succeededfrontend:build4ms
3Succeeded20.36ms
# Run frontend:build and all tasks that depend on it (after mode)$devenvtasksrunfrontend:build--modeafter
Runningtasksfrontend:build
Succeededfrontend:build5ms
Succeededdeploy:production5ms
2Succeeded11.44ms
The CLI now supports specifying single packages via the --option flag (#1988). This allows for more flexible package configuration directly from the command line:
The CI container ghcr.io/cachix/devenv/devenv:v1.8 has been reduced (uncompressed) from 1,278 MB in v1.7 to 414 MB in v1.8—that's a reduction of over 860 MB (67% smaller!).
This makes devenv container much faster to pull and more efficient in CI/CD pipelines.
Single key management - requires distributing and managing a master key
Trust requirements - everyone with the key can decrypt all secrets
Rotation complexity - departing team members require key rotation and re-encrypting all secrets
Larger teams often adopt solutions like OpenBao (the open source fork of HashiCorp Vault), requiring significant infrastructure and operational overhead. Smaller teams face a gap between simple .env files and complex enterprise solutions.
What if instead of choosing one tool, we declared secrets uniformly and let each environment use its best provider?
Imagine you commit a secretspec.toml file that declares:
# secretspec.toml - committed to your repo[project]name="my-app"revision="1.0"[profiles.default]DATABASE_URL={description="PostgreSQL connection string",required=true}REDIS_URL={description="Redis connection string",required=false}STRIPE_API_KEY={description="Stripe API key",required=true}[profiles.development]# Inherits from default profile - only override what changesDATABASE_URL={default="postgresql://localhost/myapp_dev"}REDIS_URL={default="redis://localhost:6379"}STRIPE_API_KEY={description="Stripe API key (test mode)"}[profiles.production]# Production keeps strict requirements from default profile
Now, here's the magic:
You (on macOS): Store it in Keychain, retrieve with secretspec --provider keyring run -- cmd args
Your teammate (on Linux): Store it in GNOME Keyring, same command works
That one developer: Still uses a .env file locally (we don't judge, we've been there)
CI/CD: Reads from environment variables in GitHub Actions secretspec --provider env run -- cmd args
Production: Secrets get provisioned using AWS Secret Manager
Same specification. Different providers. Zero code changes.
# .github/workflows/test.yml-name:Run testsenv:DATABASE_URL:{{secrets.TEST_DATABASE_URL}}STRIPE_API_KEY:{{secrets.STRIPE_TEST_KEY}}run:|secretspec run --provider env --profile production -- npm test
# fly.toml[processes]web="secretspec run --provider env --profile production -- npm start"# Set secrets using fly CLI:# fly secrets set DATABASE_URL=postgresql://... STRIPE_API_KEY=sk_live_...# SecretSpec will read these from environment variables
Notice what didn't change? Your secretspec.toml. Same specification, different providers, zero code changes.
While secretspec run provides secrets as environment variables, your application remains disconnected from knowing which secrets it requires. The Rust SDK bridges this gap by providing type-safe access to your declared secrets.
The Rust SDK provides compile-time guarantees:
// Generate typed structs from secretspec.tomlsecretspec_derive::declare_secrets!("secretspec.toml");fnmain()->Result<(),Box<dynstd::error::Error>>{// Load secrets using the builder patternletsecretspec=SecretSpec::builder().with_provider("keyring")// Can use provider name or URI like "dotenv:/path/to/.env".with_profile(Profile::Production)// Can use string or Profile enum.load()?;// Access secrets (field names are lowercased)println!("Database: {}",secretspec.secrets.database_url);// DATABASE_URL → database_urlprintln!("Stripe: {}",secretspec.secrets.stripe_api_key);// STRIPE_API_KEY → stripe_api_key// Optional secrets are Option<String>ifletSome(redis)=&secretspec.secrets.redis_url{println!("Redis: {}",redis);}// Access profile and provider informationprintln!("Using profile: {}",secretspec.profile);println!("Using provider: {}",secretspec.provider);// For backwards compatibility, export as environment variablessecretspec.secrets.set_as_env_vars();Ok(())}
The application code never specifies where to get secrets - only what it needs through the TOML file. This keeps your application logic clean and portable.
We've started work on supporting multiple Nix implementations in devenv. The codebase now includes a backend abstraction layer that will allow users to choose between different Nix implementations.
This architectural change paves the way for integrating Snix (our development fork). While the Snix backend isn't functional yet, the groundwork is in place for building out this Rust-based reimplementation to the C++ Nix implementation. See PR #1950 for implementation details.
Tasks now skip execution when their input files haven't changed, using the new execIfModified option:
{tasks={"frontend:build"={exec="npm run build";execIfModified=["src/**/*.tsx""src/**/*.css""package.json"];};"backend:compile"={exec="cargo build --release";execIfModified=["src/**/*.rs""Cargo.toml""Cargo.lock"];};};}
This dramatically speeds up incremental builds by skipping unnecessary work.
Join our Discord community to share your experiences and help shape devenv's future.
We're particularly interested in feedback on the standardized language tooling configuration coming in 1.8 - let us know if this approach works for your use cases!
One of the most powerful applications of ad-hoc environments is in CI pipelines, where you can easily implement testing matrices across different configurations:
This approach lets you validate your code across multiple language versions or dependency combinations without maintaining separate configuration files for each scenario.
When used with an existing devenv.nix file, --option values override the configuration settings in the file, making it easy to temporarily modify your environment.
We're excited to see how you'll use ad-hoc environments to streamline your development workflow. Share your feedback on GitHub or join our Discord community!
In this release, we're introducing a powerful Nix concept: overlays for modifying and extending the nixpkgs package set, along with significant performance and TLS certificate improvements.
Overlays allow you to modify or extend the default package set (pkgs) that devenv uses. This is particularly useful when you need to:
Apply patches to existing packages
Use different versions of packages than what's provided by default
Add custom packages not available in nixpkgs
Use packages from older nixpkgs versions
Here's an example of using overlays in your devenv.nix file to apply a patch to the hello package:
{ pkgs,...}:{# Define overlays to modify the package setoverlays=[# Override an existing package with a patch(final: prev:{hello= prev.hello.overrideAttrs (oldAttrs:{patches=(oldAttrs.patches or[])++[./hello-fix.patch];});})];# Use the modified packagespackages=[ pkgs.hello pkgs.my-tool ];}
{ pkgs, inputs,...}:{overlays=[(final: prev:{nodejs=(import inputs.nixpkgs-unstable {system= prev.stdenv.system;}).nodejs;})];# Now you can use the unstable version of Node.js languages.javascript.enable=true;}
We've heard from ZScaler how they are using devenv and we've fixed their major annoyance
by ensuring devenv now respects system certificates that many enterprises rely on.
For macOS developers, we've added the ability to customize which Apple SDK is used for development:
{ pkgs,...}:{ apple.sdk= pkgs.apple-sdk_15;}
This allows you to:
- Control exactly which version of the SDK to use
- Ensure consistency across development environments
- Avoid incompatibilities between different macOS versions
To continually enhance the AI’s recommendations, we collect anonymous data on the environments generated. This feedback helps us train better models and improve accuracy.
Of course, your privacy matters—if you prefer not to participate, just add the --disable-telemetry flag when generating environments.
We also adhere to the donottrack standard.
In February 2020, I went on a 16-day, 1200km moped trip across northern Thailand with a couple of friends.
Somewhere in northern Thailand near Pai.
As we drove for hours on end, I was listening to an audiobook fittingly called Crossing the Chasm.
The book explores the challenges faced by nacent technologies on their way to mainstream adoption.
In the years that followed, I couldn't help noticing the disconnect between Nix's devoted user base
and its apparent lack of widespread adoption in the broader tech community.
Over 2021 and 2022, I focused my efforts and started nix.dev, a resource for practical and accessible Nix tutorials, thinking that good documentation was the thing holding Nix back.
Eventually, I came to the realization that improving documentation alone will only get us so far.
We needed to fix the foundations.
We needed to remove things to reduce the cognitive load when using Nix.
Over the years at Cachix we've talked to team after team
abandoning Nix and have observed a surprisingly consistent pattern.
Nix is initially introduced by someone enthusiastic about the technology. Then, faced with
a steep adoption curve, it is abandoned after backlash from the rest of the team.
Making it trivial for a project to adopt and maintain a development environment
is crucial for other team members to see the benefits of Nix.
For example, Shopify was vocal about Nix way back in 2020,
but eventually went quiet. Having companies like Shopify adopt Nix would be a major step forward for the whole ecosystem.
With the introduction of tasks in the 1.2 release and
Nix caching in 1.3, we're pretty happy with
the devenv command-line interface and the extensible nature of the module system.
The modular architecture of the module system allows for seamless addition, modification, and removal of configuration options.
This flexibility extends to defining your own options for the software you're writing.
We've been using the Nix command-line interface under the hood as a low-level API to
the evaluator and Nix store. We would've preferred to use something akin to an SDK instead, however
the command-line interface was the most sensible interface two years ago out of the available options.
The new C FFI (Foreign Function Interface) could potentially grow into a viable solution, but
it would necessitate substantial development effort and still leave us vulnerable to memory-safety
issues. Moreover, the architecture of the Nix codebase is structured more as a monolithic framework rather than a modular library.
Ideally, if we're committing to fixing the developer experience over the next years, we'd want to have Nix implemented as a library in Rust.
Fortunately, such a project already exists and it's called Tvix. Started by flokli and tazjin in "Rewriting Nix",
Tvix is a re-implementation of Nix in Rust, offering both memory-safety and a library-oriented architecture with independently usable components. Leveraging Rust's
abstractions and ecosystem (e.g. tracing.rs), Tvix is positioned to significantly enhance the developer experience for devenv developers and users.
There are many architectural differences besides the obvious "Rewrite In Rust" cliche,
so we'll talk about them as we start replacing our Nix command-line calls with Tvix libraries, starting with the evaluator.
The Nix evaluator directly traverses the abstract syntax tree (AST) during evaluation, while Tvix uses a bytecode virtual machine crafted
according to the Crafting Interpreters book.
Tvix compiles Nix code into compact bytecode, then executes it in a virtual machine. This two-step approach offers potential performance benefits and optimization opportunities,
like many other interpreted languages.
When you re-evaluate devenv.nix, you're most likely changing devenv.nix and not one of the few dozen Nix files that come from the devenv repository,
or even the few thousand Nix files from the nixpkgs repository that could all be cached as bytecode.
In order to integrate the Tvix evaluator with devenv we'll need to:
Finish implementing builtins.fetchTree, where we have some ideas on how to simplify the caching layer
and get rid of the annoying dependency on GitHub's rate-limited api.github.com endpoint.
Implement an evaluation debugger that will allow inspecting a program's state in case of errors.
Finish implementing tvix-eval-jobs that will be used for regression tests against nixpkgs
to make sure that the evaluator behaves correctly.
Create debug tooling for when we discover regressions in the evaluator.
Once we've integrated the evaluator, we can finally generalize building languages using Nix reproducible builds
by running the underlying build system to generate Nix expressions:
graph TD
A[devenv];
A -->|Rust| C[Cargo];
A -->|JavaScript| D[npm];
A -->|PHP| E[Composer];
C -->|Cargo.lock| F{Nix};
D -->|package.json| F{Nix};
E -->|composer.lock| F{Nix};
In the general case, the dependency graph is computed statically, but a dependency can
declare its dependencies dynamically as part of the build by returning more Nix code.
That's when evaluation and build phases start to mix, with evaluation depending on the result of a build,
which is typically called import from derivation (as the naming comes from the implementation).
sequenceDiagram
autonumber
participant NixEvaluator as Nix evaluator
participant NixStore as Nix store
NixEvaluator->>NixEvaluator: evaluate
NixEvaluator->>NixStore: write derivation
NixStore->>NixStore: build
NixStore->>NixEvaluator: read derivation output
NixEvaluator->>NixEvaluator: evaluate
Since evaluation in Nix is single-threaded, the process described above gets blocked on each build requested during evaluation.
Implementing parallel evaluation in Tvix, after we figure out the architectural details of how it should work, will unlock the ability to support automatic conversion
of language-specific build systems into Nix without sacrificing neither the developer experience, nor memory safety.
As we embark on this new chapter with Tvix, I'm reminded of the journey that brought us here.
It's been a decade since I wrote the we can do better blog post,
highlighting the potential for improvement in configuration management and development environments, and I'm glad to see it all finally coming together.
Keep an eye out for updates and join the discussion:
Nix's built-in flake evaluation caches outputs based on the lock of the inputs,
ignoring changes to Nix evaluation that often happen during development workflow.
Let's take a closer look at how devenv's new caching system compares to other popular tools in the Nix ecosystem.
Running our own cache gives us more control and visibility over the caching process, and allows us to improve our integration with other tools, like direnv.
While lorri pioneered the approach of parsing Nix's internal logs for caching,
devenv builds on this concept, integrating caching as a built-in feature that works automatically without additional setup.
These tools excel at caching evaluated Nix environments, but have limitations in change detection:
Manual file watching: Users often need to manually specify which files to watch for changes.
Limited scope: They typically can't detect changes in deeply nested imports or files read by Nix built-ins.
To leverage devenv's caching capabilities with direnv, we've updated the .envrc file to utilize devenv's new caching logic.
If you currently enjoy the convenience of our direnv integration to reload your development environment, make sure to update your .envrc to:
source_url "https://raw.githubusercontent.com/cachix/devenv/82c0147677e510b247d8b9165c54f73d32dfd899/direnvrc" "sha256-7u4iDd1nZpxL4tCzmPG0dQgC5V+/44Ba+tHkPob1v2k="
use devenv
nix develop currently remains the last bit that's rather slow and uncacheable, particularly on macOS.
We're working on bringing its functionality in-house to further bring down the overhead of launching a cached shell to under 100ms.
For devenv, our mission is to make Nix the ultimate tool for managing developer environments. Nix
excels at congruent configuration,
where the system state is fully described by declarative code.
However, the real world often throws curveballs. Side-effects like database migrations, one-off
tasks such as data imports, or external API calls don't always fit neatly into this paradigm.
In these cases, we often resort to convergent configuration,
where we define the desired end-state and let the system figure out how to get there.
To bridge this gap and make Nix more versatile, we're introducing tasks. These allow you to
handle those pesky real-world scenarios while still leveraging Nix's powerful ecosystem.
We've talked to many teams that dropped Nix after a while and they usually fit into two categories:
1) Maintaining Nix was too complex and the team didn't fully onboard, creating friction inside the teams.
2) Went all-in Nix and it took a big toll on the team productivity.
While devenv already addresses (1), bridging the gap between Nix provided developer environments
and existing devops tooling written in your favorite language is still an unsolved problem until now.
We've designed Task Server Protocol so that you can write tasks
using your existing automation by providing an executable that exposes the tasks to devenv:
In a few weeks we're planning to provide Rust TSP SDK
with a full test suite so you can implement your own abstraction in your language of choice.
You can now use your preferred language for automation, running tasks with a simple devenv tasks run <names> command. This
flexibility allows for more intuitive and maintainable scripts, tailored to your team's familiarity.
For devenv itself, we'll slowly transition from bash to Rust for
internal glue code, enhancing performance and reliability. This change will make devenv more
robust and easier to extend, ultimately providing you with a smoother development experience.
