Anton Babenko

Deliberation Dashboard: see and analyze your consensus results

Thu, 08 Oct 2026 18:53:08 GMT

Note: I use Deliberation heavily every day, in the harnesses I picked for my own work: Claude Code, Codex and Antigravity, running in T3 Code and Claude Code on the web at the same time. It has been tested in real work much harder than the version I wrote about in June.

In June I wrote that every model on a panel has to prove it earns its seat, and I shipped a debug log and an analyze tool to do the proving. Both worked. The log on my laptop is 2.4 MB of JSONL today. Measuring was the easy part. A number I have to go and dig for is a number I do not act on.

So most of the work since the June post went into making the panel visible: a local dashboard that draws every run while it happens, an analyzer that says which model to drop and why, and a record of which config each run used. Around that, a lot of smaller fixes for the ways a multi-model run goes wrong when nobody is watching. Deliberation went from v3.5.2 to v3.28.0 in that time.

If you are new here: Deliberation is an open-source MCP server that lets the agent you already use (Claude Code, Codex, Cursor, Kiro, Antigravity, OpenCode) ask GPT, Gemini, Grok and any OpenRouter model for a second opinion, or make them argue until they agree. The June post explains the consensus loop. This one is about watching it.

Install in Claude Code is still three lines:

/plugin marketplace add antonbabenko/agent-plugins
/plugin install deliberation@antonbabenko
/deliberation:setup

Other hosts have their own guides in docs/hosts.

You cannot fix a panel you cannot see

Turn it on with one line in ~/.config/deliberation/config.json:

"dashboard": { "enabled": true }

Then run /deliberation:dashboard (or deliberation-mcp dashboard in a terminal). It prints a URL on 127.0.0.1 and opens it. There are five tabs: Live, Runs, Config, Stats and Analyzer. I generated most of the dashboard UI in a single shot with Impeccable.

The Runs tab is the history. Every /consensus, /ask-all and /ask-* call is one row with its status, the project it came from, the providers, rounds, tokens and wall time. You can filter by project, tool, provider, status and date. The yellow UNRESOLVED rows are the panel saying it could not agree in five rounds.

Click a run and you get the part I wanted from the start:

This is a real consensus-step run on the Deliberation repo itself, with Claude as the arbiter and five reviewers. Each model gets a lane, and the bars show when it was thinking and what it voted. The trajectory below tells the story of the run in three lines. Round 1: Codex and Grok asked for changes. Round 2: only Codex still objected. Round 3: all five approved, 21 minutes 46 seconds and 330k tokens after the start.

The latency table under it is where the panel stops being abstract. Grok spent 14 minutes 04 seconds in total over three calls. Kimi K3 spent 1 minute 20 seconds and voted the same way every round. In a parallel fan-out the slowest member sets the clock, so that table is effectively a bill.

Live shows the same graph while a run is still going, so you can see which lane a slow run is waiting on.

The dashboard is read-only and it is built to stay on the machine. It listens on loopback only, accepts GET requests only, and the URL carries a token that becomes a cookie for that one browser. By default it records metadata only, without prompts or responses. If you switch dashboard.capture to content, secrets are scrubbed before anything is written and PII is redacted when a page is served, unless you turn on dashboard.showPII.

The analyzer tells me which model to drop

The June version of analyze was a command that printed a report. It is still there (/deliberation:analyze, now with a since window and only the models you have configured), but the useful version is the Analyzer tab.

The first table answers the question from the June post: which model agrees more than it adds. It reads only consensus rounds, since those are the ones with verdicts. "Adds nothing" means every issue category the model raised in a round was also raised by another voice. A model becomes a drop candidate only when all of these hold: at least 10 rounds, adds nothing in 80% or more of them, is the lone objector in 10% or fewer, rarely raises an issue the arbiter accepts, and is either slow or errors a lot.

My own numbers over the last 30 days, 388 runs: Gemini sat in 273 rounds, added nothing in 95% of them and was slower than the other responders' median every time. Grok added nothing in 87%. Codex looks worse on paper at 85%, but it has the highest lone-dissent rate on the panel (15%), and the arbiter accepted its issue in all 6 rounds where a decision was recorded. The lone objector who is usually right is exactly the model you keep, and the table says "keep".

Gemini and Grok are marked UNCONFIRMED, with the config line that would turn them off. They are still on in my config. Six rounds with arbiter decisions is too few for me to trust the verdict, and the analyzer says the same thing by not calling them drop candidates yet.

The second table groups runs by project (by git remote, so every clone and worktree of a repo lands on one row) with runs, calls, timeouts, errors, p50 and p95. The third one plots request size against latency per model. For Codex on text-only prompts the fit is about 5.5 seconds per thousand characters, and the advice column refuses to suggest a timeout until a bucket has 20 calls. For Kimi K3 with files, 1 of 9 calls timed out, and it already points at models.kimi-k3.timeout as probably too low.

To get these numbers, every call now records its request size, the attached file bytes, the timeout it was actually granted and which setting granted it, and every run records the project it came from.

Every run remembers the config it ran with

In June, runs made under different configs all landed in the same log, with nothing saying which config produced which number. Now each run keeps a snapshot of the config it ran with, and the Stats tab compares configs side by side.

Two rows from my own history: config 8e33794253ec had 68 runs, 23 errors and a p95 of 25 minutes 37 seconds. Config dd82d2de6943, the one the run above used, has 20 runs, 1 error and a p95 of 7 minutes 14 seconds. I would not call that a controlled experiment, but it is the first time I could see that a config change did something.

This works because more of the config is pinned in one file. GPT's model and reasoning effort (providers.codex.model, providers.codex.reasoningEffort), the Gemini and Grok models, a default OpenRouter model, per-provider timeouts and a separate reasoning effort for consensus rounds (providers.<name>.consensusReasoningEffort) all live in config.json. The Config tab shows the effective values and where each one came from.

Slow is fine, unbounded is not

The June post said a full consensus loop can take minutes and that is fine for the right job. That held until one /consensus run took more than 45 minutes, because a single Codex call inside it ran for about 38. Nothing was broken. It was just waiting.

Since then a run has limits:

consensus.maxWallMs is a time budget for the whole run (default 30 minutes), and a peer that times out twice in a row is dropped from the remaining rounds.
consensus.quorumFloor (default 2) means a run cannot converge when too few reviewers answered. Two models dropping out no longer leaves one model "agreeing" with the arbiter.
Real disagreement resets the error streak, so a model that keeps objecting is never cut by the circuit breaker that is meant for broken providers.
On rounds with dissent, the arbiter's adjudication and revision overlap instead of running one after another.
Later rounds get a digest of earlier ones instead of the full history, and static files go first in the prompt so OpenRouter can cache them.

The other half is about calls that look like answers and are not. A 429 is retried once, honoring Retry-After. A reply like "I will begin by reviewing the files..." counts as a failure instead of a vote. One broken provider no longer costs the whole round. And timeouts are per provider now, after I watched a DeepSeek call succeed at 250 seconds while its three siblings were killed at 180.

Models argue about today, not their training cutoff

Every delegate prompt now starts with today's UTC date and one rule: when a model does not recognize something (a new model name, a recent release, a CLI flag), it marks the claim [unverified] instead of saying it does not exist.

Smaller things that removed friction

Claude Code on the web works: every call fits under the host's 60-second tool-call cap, and the plugin declares a 30-minute per-server timeout where the host allows it.
Codex login is a device login (/deliberation:codex-login): it shows a link and a code in about a second, prefers your ChatGPT login and never uses OPENAI_API_KEY.
Antigravity is a native host, and Codex and Gemini CLIs start on Windows.
/deliberation:help shows real prompts you can paste, /deliberation:doctor checks config, CLIs and paths without changing anything, and /deliberation:reload-mcp cycles the server after an update.
Read-only is enforced for advisory calls: Codex runs with --sandbox read-only, Gemini under a read-only sandbox.
Secret scrubbing understands syntax now: private key blocks, JWTs, API key formats and assignments in JSON, shell and YAML are removed before anything is written.
Grok and OpenRouter cannot read your repo, so Deliberation can attach a small orientation bundle (orientation.enabled). It has a byte budget now (orientation.maxBytes, 16 KB by default), which cut that part of each call from about 26K tokens to about 4K in my runs.

What I would still not trust it with

The dashboard is for the machine it runs on. There is no hosted version and no team view.

The analyzer works on coarse issue categories and needs volume. With a few dozen consensus rounds it will mostly say UNCONFIRMED, and that is the correct answer. Treat it as advice, as the page itself says.

And agreement is still not truth. The dashboard makes it much easier to see when five models agree. It does not make them right.

Try it

Update the plugin, add "dashboard": { "enabled": true } to your config, run a couple of /ask-all or /consensus calls, then /deliberation:dashboard. The Analyzer tab gets useful after a week of real runs.

Source, issues and the star button are at github.com/antonbabenko/deliberation. I want to hear the feedback (good and bad), especially from anyone running a panel I have not tried.

Also published on dev.to.

Meet Deliberation: 400+ models is easy, knowing which ones earn a place is hard.

Wed, 03 Jun 2026 08:54:04 GMT

It is open source and ready to use today: github.com/antonbabenko/deliberation. If you only do one thing, star the repo and install it in your own agent - it takes about two minutes, and the install section below has the exact command for Claude Code, Codex, Cursor, Kiro, and OpenCode.

Here is the part that surprises people: you are probably already paying to access a few models. A ChatGPT subscription includes Codex which allows you to use models like gpt-5.5 and gpt-5.3-codex from Codex CLI. A Claude subscription includes Claude Code. So you can wire GPT and Claude to review each other right now, at no extra cost, and add Gemini, Grok, or any OpenRouter model when you want a third opinion.

A few weeks ago I wrote that one model is a guess and three that agree is a plan. I still believe it. But it skipped the next problem, the one I hit the moment the trick worked: once you can ask three models, you can ask thirty seven. And more voices is not the same as more signal. A slow model that always agrees with a faster one is not a second opinion. It is a bill.

So the project grew up, and it got a name: Deliberation.

The word fits. Deliberation is the slow, careful weighing of a decision - the thing a jury or a council does before it returns a verdict. That is exactly the job here: a panel of models that weighs one artifact and argues its way to a call, instead of one model blurting the first thing that sounds right. It answers a different question than the first post did. The first post asked should you make models argue? This one is about which models, under what rules, and how do you know any of it was worth the wait?

It still runs the everyday work of compliance.tf, same as before. Most of what follows came from that: not from theory, from watching real runs and asking why one took four minutes to tell me what another said in twenty seconds.

A few words first

If you are new to this, here are the only terms you need. Plain version, with examples.

Agent host. The tool you already code in: Claude Code, the Codex CLI, Cursor, Kiro, OpenCode. Deliberation plugs into all of them and behaves the same way in each.
MCP. A standard plug that lets your agent host talk to outside tools. Deliberation is one MCP server, so any host that speaks MCP can use it. You install it once; you do not write glue code.
Provider / model. A provider is a company (OpenAI, Google, xAI, OpenRouter). A model is one brain from that provider (GPT, Gemini, Grok, or any of OpenRouter's 400-plus, like Qwen, DeepSeek, or Kimi).
Panel. The set of models you ask at once. You choose who is on it. Example: a panel of GPT + Gemini + Grok.
Expert (or persona). A hat a model wears for one job: Architect, Security Analyst, Code Reviewer, Plan Reviewer, and three more. The same model reviews differently depending on the hat.
Arbiter. The one who reads everybody's answers and makes the call. It can be a model, or it can be your own agent (you).

That is the whole vocabulary. The rest is just rules for how the panel talks.

Install it now

Deliberation is one MCP server with native packaging for each host. Set only the providers you use - a missing key just turns that one provider off. Full per-host guides: public-docs/hosts.

Claude Code: add the marketplace, install the plugin, run setup.

/plugin marketplace add antonbabenko/agent-plugins
/plugin install deliberation
/deliberation:setup

Codex CLI: codex plugin marketplace add antonbabenko/deliberation, then install deliberation from /plugins.
Cursor: drop in the rule file and use the one-click MCP deeplink (guide).
Kiro: "Import power from GitHub" (guide).
OpenCode: add the opencode.json MCP snippet (guide).

Credentials come from your environment: GPT uses your Codex/OpenAI login (run codex login), Gemini signs in once through its new Antigravity CLI (run agy), Grok reads XAI_API_KEY, OpenRouter reads OPENROUTER_API_KEY. Start with GPT and Claude, since you likely already pay for both.

What actually changed

Three things.

It is one MCP server now, not a Claude-only plugin. It works in Claude Code, Cursor, Codex, Kiro, OpenCode - anything that speaks MCP. Your primary agent stays in charge and calls the panel when a decision is worth a second look.

The panel opened up. It used to be three fixed externals: GPT, Gemini, Grok. Those are still the built-in voices, but you can now add any of OpenRouter's 400-plus models (including Qwen, DeepSeek, Kimi, and others) as named records in a config file. You pick them. You say which ones join a quick fan-out, which ones sit on the consensus panel, and which expert hat each one is allowed to wear - not all are equally good for all tasks.

And the whole thing learned to measure itself. That last part is the real subject of this post.

I will walk through it as five categories. Each one has a switch you flip in config, and each one votes - or refuses to - in a specific way.

Quick one-offs: just ask

Before the heavy machinery, the everyday move. Most of the time you do not want a five-round debate - you want a fast second opinion from someone who is not the model you have been talking to all session.

So you just ask, in plain words: "Ask Grok and Gemini whether this retry loop can deadlock." Your agent routes that to the right voices and brings back the answers. Or use the explicit commands when you know who you want:

/ask-gpt does this IAM policy grant more than it should?
/ask-grok poke holes in this rollback plan
/ask-gemini summarize the risk in this migration in 5 bullets

These are single-shot and advisory: one question, one answer, no loop, no context contamination from your session. The exact same commands work in Claude Code, Codex, Cursor, Kiro, and OpenCode - the server is the same everywhere, so a prompt you learn in one host transfers to the rest unchanged.

And none of this is a separate ritual you have to schedule. Call /ask-all or /consensus at any point in the work - while you are still scoping a feature, halfway through writing a plan, or in the middle of a /grill-me session when you want a real outside voice in the room instead of arguing with only yourself. The panel is available the whole time, not just at the review at the end. The earlier you pull in a dissenting model, the cheaper the disagreement is to act on.

1. Fan-out with no cross-talk

The next cheapest move: ask several models the same question at once and read the answers side by side. Nobody sees anyone else's reply, so you get real independence instead of an echo.

Two examples where this works the best:

/ask-all can this Terraform state migration be rolled back safely, and what breaks if it cannot?
/ask-all what are the failure modes of switching this table's primary key on a live database?

/ask-all runs the parallel version. panel tells you exactly who would answer for your current config before you spend a token. ask-one lets you fire each member yourself, so each answer lands on screen as it finishes instead of waiting on the slowest one.

Switches: routing.maxFanout caps how many OpenRouter models join (default 3), and each model record has an askAll flag to opt in or out.

There is no voting here. It is parallel sampling. You and your agent (Claude Code, Codex, or anything else) get all the answers and decide.

2. The consensus loop, where the voting lives

This is the heart of it, and it is a state machine, not a vibe. One round goes:

Blind pre-commit. The arbiter writes down its own verdict - approve, request changes, or reject - before any other model sees the work. In writing, first. So its judgment cannot quietly drift to match the crowd later.
Parallel peer review. The panel reviews the same artifact, each in a fresh thread, none seeing another's review.
Blind cross-review. Each model then rates the others' answers with names stripped off (to avoid bias and deference). A "not viable" vote becomes a candidate problem the arbiter has to deal with. This catches the case where everyone looked like they agreed but were each walking past the same hole. (Pattern borrowed from karpathy/llm-council.)
Adjudication. The arbiter goes through every objection and accepts it, dismisses it with a written reason, or defers it. Then it revises the artifact and the round runs again.

It converges only when every responding model approves and the arbiter's pre-committed verdict agrees with them. If they cannot get there inside the round cap, it returns UNRESOLVED and says so. It does not fake a number to look finished.

Click for the detailed diagram with the bias guards and per-model flow.

The answers are not free text the arbiter has to guess at. Every voice returns a structured opinion, and the engine parses it into fixed fields:

a recommendation (the actual call),
a confidence label, so a weak "maybe" does not count the same as a firm "yes",
dissent points, assumptions, and tradeoffs as separate lists, so a disagreement is attached to a reason instead of buried in prose.

Reviews add a verdict (approve / request changes / reject) and a list of critical issues, each sorted into a closed six-category taxonomy - so "three reviewers, nine objections" collapses into a clean, deduplicated set the arbiter can act on. Models are asked to emit this as JSON; the parser is best-effort and never throws, so if a model returns slightly malformed JSON or plain text, the content is salvaged and tagged with how it was read (clean parse vs recovered). Structured where it can be, never brittle.

Switches: consensus.arbiter (auto, the host model, a named provider, or a dedicated model record), consensus.blindVote (add the blind pre-vote on the synthesis path), consensus.maxRounds (default 5, capped at 50), and a per-model consensus flag for who sits on the panel.

So there are four distinct votes in play: the arbiter's blind pre-commit, each peer's independent verdict, the anonymized cross-review rating, and the final convergence check. They are doing different jobs on purpose.

One honest note, since a reviewer pushed me on it: that convergence rule is a heuristic, not a proof. Unanimous approval means nobody in the room objected. It does not mean the answer is right. More on that below.

3. Synthesis, when there is no verdict to give

Not everything is approve-or-reject. "Which of these two designs should we pick?" has no yes or no. Flip synthesizeAlways: true and instead of the loop you get a single arbiter pass that reads every opinion and writes one combined answer - free text, no verdict, no rounds. Use the loop for go/no-go calls. Use synthesis for open questions. Same panel, two shapes.

4. Two drivers, one rulebook

The loop logic lives in one place - a single state machine - and two things drive it.

consensus runs the whole loop server-side with a model as the arbiter and hands you back the result in one call. consensus-step lets your own agent be the arbiter and drive the loop one step at a time, so every move shows up in your transcript where you can see it.

Two entry points is more surface area than one, and they do not behave identically - one is visible step by step, one is a single call. The win is that the rules (how rounds count, when it converges) are written once and shared, so the two paths cannot drift apart on the part that matters. That was worth the extra work.

5. Make the panel stay accountable

In a parallel fan-out, your wall-clock time is the slowest model, not the average. One slow member that rarely says anything new sets the clock for everyone. You will not notice by feel. You need numbers.

So there is an opt-in debug log. Turn on debug.enabled and it writes one line per model call and per round: latency (p50, p95, max), token counts, error rate, the reasoning effort used. It never records your prompts, the responses, or the issue text - only the timing and the outcome of each vote.

Turn on sessions.persist and runs are saved too, so you can ask how often a given model's verdict matched the final call - and, just as useful, how often it was the lone voice that caught something the others missed, or the lone voice that was simply wrong. A model that always agrees adds cost; a model that disagrees and is usually right is the one you keep.

Then an analyze tool reads both back and tells you, in plain terms, who is slow, who is redundant, and which config line to change. From my own panel last week: one model sat at a 200-second p95 while another finished in 15 (Grok 4.3 is often the fastest). analyze flagged it and suggested the one-line switch to drop it from the default fan-out. I had been waiting on that model for almost a week without realizing it.

There is also a small dedup cache, so an identical advisory question inside one session returns instantly instead of paying twice. Well, in tool-heavy work the prompts vary, so it hits less than you would hope.

This is the whole shift from the first post. Consensus was the idea. Making the panel prove it earns its seat is the engineering!

The parts I will not oversell

Multi-model review has real failure modes, and a post that hides them deserves the distrust it gets. So:

Agreement is not truth. Three models can be confidently, unanimously wrong - especially when they were trained on overlapping data and share the same blind spot. Consensus lowers the odds of a lone model's odd mistake. It does not discover facts. Treat it as risk reduction, not a truth oracle.
Pick models that actually differ. A panel of three close cousins is one model agreeing with itself in three accents. The value comes from genuinely different models arguing, which is exactly why the config lets you choose them by hand.
UNRESOLVED is a feature. When the room cannot agree, the honest output is "we could not." That is a signal to slow down and look, not a bug to smooth out. If you wire this into CI, decide on purpose what a deadlock means there - it should probably stop the line, not wave it through.
It is slow, and that is fine for the right job. A full consensus loop can take minutes. Run it on plans, designs, and reviews in the background. For a fast second opinion, use single-shot fan-out or synthesis. Do not put a five-round loop in an interactive path.
Mind where the text goes. Fanning a compliance artifact out to 400 third-party models is a data-boundary decision, not a free upgrade. The config defaults the long tail to off for exactly this reason. Turn models on deliberately.

None of this is specific to Terraform, AWS, or even to code. The loop runs on anything you can put in text - a plan, a runbook, a decision memo. That generality is the point. It just happens to have been built running a compliance product, which is a good place to learn that "looks done" and "is done" are different claims.

The takeaway

The first lesson was: do not trust one confident model, make a few argue. The second one is harder and more useful: adding voices is easy, and most of them will not earn their seat. Measure the panel like you would measure any other service or people, and cut what does not pay.

Deliberation is open source, it works in the agent you already use, and most of the models are ones you already pay for. It ships in the agent-plugins marketplace and as a standalone MCP server on npm (@antonbabenko/deliberation-mcp). Source, and a star button I would genuinely appreciate, are at github.com/antonbabenko/deliberation. Install it, point GPT and Claude at each other, and let me know where it breaks. The cheapest review still happens before you execute - just make sure every reviewer in the room is worth the wait and the cost.

Also published on dev.to.

One model is a guess. Three that agree is a plan.

Mon, 18 May 2026 08:04:46 GMT

Updated (28.5.2026): /consensus is now a 3-stage loop. Stage 2 adds a blind cross-review where each external rates the others' answers, anonymized, before Claude adjudicates. Pattern from karpathy/llm-council.

Updated (26.5.2026): Grok (xAI) joins GPT and Gemini as the third external provider, Gemini 3 is the default via Google's Antigravity CLI, two new experts (Researcher and Debugger) bring the count to seven, reviews are severity-graded, and /consensus now forces Claude to commit a blind verdict in the transcript before dispatching - arbiter-mediated, not pure democracy.

Ask one model to plan something hard - a migration, a refactor, a cutover - and you get a fluent, confident answer. Fluency is not correctness. A single model is articulate and alone, and being alone is the problem: nothing in the loop disagrees with it, so it rationalizes its first guess into a plan.

The expensive failures with coding agents are almost never syntax. They are plans that read well and were wrong: the wrong abstraction, the missed blast radius, the migration step that bricks state. You find out three hours into execution, not at review time.

I have been running the fix for the last few months - on Terraform modules, and on the everyday work of running compliance.tf, not only on code. It is not a bigger model. It is making models disagree on purpose and then forcing them to resolve it. That is what consensus does, and it is the reason the agent-plugins repo exists. Two other tools ship with it; they are narrower, and I will get to them.

One model is a guess

A single model samples one distribution with no adversary in the room. Two independent models rarely make the same mistake on a plan. Where they diverge is, almost exactly, the risky part of the plan - the assumption nobody checked. A consensus loop turns that disagreement from noise into a signal you can act on.

What `/consensus` actually does

/consensus runs GPT (via Codex), Gemini 3 (via Antigravity), and Grok (via the xAI API) against the same artifact, with Claude as the arbiter. Each round has three stages.

Stage 1 - parallel verdicts. Claude posts its own verdict (APPROVE / REQUEST CHANGES / REJECT) into the transcript first, blind, before any external sees the work. The pre-commitment sits there in writing, so Claude's judgment cannot drift later to match what the others say. Then GPT, Gemini, and Grok review the artifact in parallel, each in a fresh thread, single-shot. None sees another's review.

Stage 2 - blind cross-review. Each external then rates the OTHER externals' answers, identity stripped best-effort. Votes of "not viable" become candidate critical issues the arbiter has to weigh. This catches the case where Stage 1 looks like agreement but is really three reviewers each rationalizing past the same hole. Pattern adapted from karpathy/llm-council. Stage 2 fires every round 1, and after that only when Stage 1 disagreed or the previous Stage 2 surfaced an accepted issue.

Stage 3 - arbiter adjudication. Claude reconciles the Stage 1 verdicts, the Stage 2 candidate issues, and its own blind verdict. Every objection is accepted, dismissed with a recorded reason, or deferred. Claude revises the artifact and the loop runs again, up to five rounds. It converges only when every responding external approves and Claude's pre-committed verdict agrees, with reasons on both sides where it walked back. If the group cannot agree, it says so plainly instead of faking it.

Click for the detailed diagram with bias guards and per-model flow.

Independence is not only about which model. It is sharper when each reviewer wears a different hat. claude-delegator ships seven expert profiles - Architect, Plan Reviewer, Scope Analyst, Code Reviewer, Security Analyst, Researcher, and Debugger.

Combine the axes. A Security Analyst on Gemini and an Architect on GPT fight about different things than one model reviewing twice. Different profiles catch different categories of mistake.

For a migration plan I run the Plan Reviewer broadly and add a Security Analyst pass on top. "Is this safe to run" and "is this the right shape" get argued by separate reviewers, not averaged into one bland verdict.

There is a second kind of contamination worth naming: me. In consensus, every round sends the reviewers the same artifact text, cold, in a fresh thread. They never see my triage, my running verdict, or how I framed the previous round - only the artifact and bounded round metadata. My judgment is applied after they report, not baked into what they receive.

The ask-* commands are the opposite by design: the expert gets exactly the prompt I assembled from the conversation - what I chose to hand it. Fast for a second opinion, but the input is mine, not independent. consensus keeps the input independent and pays for it in extra rounds.

It does not have to be a plan. The loop runs on anything you can put in text - a design, a runbook, a decision memo, a spec. Plans are simply where I reach for it most, and where it converges fastest. The looser and fuzzier the input, the more rounds it takes to agree, so non-plan runs tend to run longer - worth it when the answer matters, overkill for a quick lookup. For that, single-shot ask-gpt, ask-gemini, ask-grok, and ask-all (which fans out to all three in parallel) are right there. consensus is for when it has to be right.

Nothing about this is Terraform, or even code. That generality is why it is the headline.

The quieter two

Same release, narrower scope:

code-intelligence - a language-agnostic skill. Agents grab text grep when they should ask the language server, and silently swap tools when one is missing, then report "found all references." This encodes search precedence: the language server (LSP) for symbols, rg/ripgrep for exact text, an embedding/semantic grep (such as mgrep) for fuzzy discovery - and a hard rule to disclose any substitution on the first line of the reply. You learn the moment coverage drops.
terraform-skill - routes a Terraform/OpenTofu request to its real failure mode (identity churn, blast radius, state corruption) before emitting HCL. It is terraform-ls aware: it knows the language server has no rename provider, so it runs the safe manual reference workflow instead of a blind find-replace. It is approaching 2,000 GitHub stars - the part of this post I am quietly proud of.

Those two are discipline for the agent's hands. Consensus is discipline for its judgment, and judgment generalizes further.

A note on claude-delegator

I did not invent the delegation layer. claude-delegator is a fork of Jarrod Watts' original (MIT, upstream currently quiet), fully based on his design - I kept the structure and the license.

What I added is what months of daily use exposed: a Gemini bridge that wraps Google's Antigravity CLI (agy) with auto-gemini-3 as the routing default and recovers an answer the CLI flushed to disk after a soft timeout instead of failing the call; a fresh Grok bridge over the xAI API that is advisory-only but reads attached files via the xAI Files API (with TTL-based cleanup); two more experts (Researcher and Debugger) on top of the original five; severity-graded reviews so three parallel reports merge cleanly; and a hardened /consensus loop where Claude pre-commits a blind verdict before any external sees the artifact, with a Stage 2 blind cross-review on top (adapted from karpathy/llm-council). Plus the bundled ask-gpt / ask-gemini / ask-grok / ask-all / consensus commands, so the workflow ships with the plugin instead of living in my dotfiles. The seven expert prompts borrow from oh-my-openagent and PAL; both credited in the README.

Credit for the foundation is his; the bug-fixing scar tissue is mine.

Why a plugin, not a blog post

One honest caveat, because I have hit it myself: skills are model-triggered, which makes them soft. Packaging this as a plugin improves reuse and discoverability. It does not guarantee the agent obeys every time - hard enforcement (a real pre-tool gate) is a separate, still-open problem.

I keep finding and fixing bugs in all three. That constant repair is the only reason I trust them enough to write this - and I would rather say so than oversell the fix.

The takeaway

Stop accepting the first confident plan from one model. Make them argue, and only move when they stop. The release is at github.com/antonbabenko/agent-plugins; consensus ships in claude-delegator from the same marketplace. The cheapest review is the one that happens before you execute.

Also published on dev.to.

Your Weekly Dose of Terraform — Live Streams

Sat, 26 Sep 2020 17:35:00 GMT

I do live streams talking about things I enjoy working with — IaC, DevOps, Terraform, and AWS.

The primary focus is on Terraform — what’s new, tools reviews, live-coding, do interviews with creators and community members, answer questions live, and so on.

Here you can see previous streams — https://www.youtube.com/channel/UCGH0yYPvlCN1VjSFMGVmFgQ

Or, if you want to see just Terraform tools reviews:

checkov —Static code analysis for Terraform/CloudFormation/etc — https://www.youtube.com/watch?v=KdH-7_vpFag
tflint — Linting Terraform configurations for best-practices — https://www.youtube.com/watch?v=6w_f3MOVxBI
Terragrunt — https://www.youtube.com/watch?v=ILKJ3IuKCAo — to be presented on the 2nd of October 2020.

If you are learning Terraform or want to stay informed on what is going on in the Terraform world — subscribe and join live-streams on YouTube, Twitch, and Periscope (Twitter).

YouTube: https://www.youtube.com/channel/UCGH0yYPvlCN1VjSFMGVmFgQ

Twitch: https://www.twitch.tv/antonbabenko

Periscope: https://periscope.tv/antonbabenko

Anonymized, Secure, and Free Terraform Cost Estimation

Thu, 14 May 2020 14:42:00 GMT

There are some things you should know about infrastructure costs:

How much does my infrastructure is going to cost before create?
How much does my infrastructure cost after it is created (based on Terraform state)?
What is the difference in the price comparing to the current infrastructure (based on Terraform plan)?
Can I have cost estimation based on Terraform 0.7 state files? Yes, any version of Terraform state files is supported!

This is exactly why I started terraform-cost-estimation project while working on bigger modules.tf!

Anonymized, secure and free Terraform cost estimation based on Terraform plan (0.12+) or Terraform state (any version) – https://github.com/antonbabenko/terraform-cost-estimation

In a nutshell, infrastructure developer posts some input data to cost.modules.tf endpoint and receives a response containing costs summary:

$ terraform state pull | \
curl -s -X POST -H "Content-Type: application/json" -d @- https://cost.modules.tf/

{"hourly": "0.01", "monthly": "9.07"}

These are the supported input types:

Terraform state file (as JSON)
Terraform plan (as JSON)
List of cost keys extracted from Terraform state or plan locally

I don't want to send my Terraform data, because I have secrets there!

I know you have them there, and there is a solution for this which I call client-side filtering using JQ.

It is the most secure way and it works like this – pass Terraform state or plan file through terraform.jq filter to extract so-called cost keys, and then submit them to get cost estimation without need to reveal any of your Terraform files.

$ curl -sLO https://raw.githubusercontent.com/antonbabenko/terraform-cost-estimation/master/terraform.jq

$ terraform state pull | jq -cf terraform.jq | \
curl -s -X POST -H "Content-Type: application/json" -d @- https://cost.modules.tf/

Cost keys do not contain any customer secrets or sensitive information (like AWS accounts, AWS keys, tags, names, AMI ID, IP addresses, etc). Cost keys are used only to get cost information.

In the following example cost keys for single EC2 instance, NAT Gateway and an Application Load Balancer in eu-west-1 region were extracted:

{"keys":["ec2#eu-west-1#t3.nano#shared#linux", "ec2#eu-west-1#nat", "ec2#eu-west-1#alb"]}

You can see complete examples and list of supported services here - https://github.com/antonbabenko/terraform-cost-estimation

Bonus: Run Terraform Cost Estimation from the browser

Terraform Cost Estimation - Anonymized, secure, and free Terraform cost estimation based on Terraform plan (0.12+) or Terraform state (any version)

Remember to star&watch this project on GitHub, and share this project with your connections!

antonbabenko/terraform-cost-estimation - Anonymized, secure and free Terraform cost estimation based on Terraform plan (0.12+) or Terraform state (any version) - antonbabenko/terraform-cost-estimation

"Terraform AWS modules" as a code

Fri, 20 Mar 2020 08:52:46 GMT

After several years of maintaining Terraform AWS modules on GitHub and making constant improvements in processes I decided to spend more time and improve things even further.

I want other people to be involved and recognized also while I spend more time on other projects (eg, modules.tf).

The main motivation for doing changes is that I want to:

Recognize individuals who help triaging and fixing issues. This is by far the most time-consuming activity in this project.
Recognize contribution of everyone involved in the process (write code, answer issues, update documentation, etc).
Standardize releasing, updating, testing, documenting and some other procedures across all Terraform modules.
Create and manage many new Terraform AWS modules which satisfy the same quality requirements as other modules.
Integrate more Terraform modules into modules.tf

From now on, the configuration of all terraform-aws-modules repositories and common files (not related to the module itself, but more related to the management procedures) is available in this repository - https://github.com/terraform-aws-modules/meta

Feedback and questions are welcome!

My Terraform AWS journey — HashiTimes Interview

Thu, 13 Jun 2019 08:00:00 GMT

I’d like to share an extended interview which I gave to HashiTimes (newsletter curated by the community and not affiliated with HashiCorp) in June 2019. I added some more additional links and information.

In this week’s HashiTimes Interview, we sat down with Anton Babenko, a Developer, Public Speaker, and Evangelist based in Oslo, Norway.

Tell us a little about yourself, your background around HashiCorp, and what do you do professionally?

I started working with Terraform and Packer in 2015. In December 2015, I gave a talk about how to use Terraform with AWS at a local AWS User Group meetup in Oslo because I wanted to share how awesome Terraform was (and still is). You can see a video recording here.

Around the same time, I requested access to push my first Terraform module and got access to manage all other repositories under terraform-community-modules organization. This was the starting point for my open-source participation within Terraform. I got full access, and soon I started answering questions about other modules even without using them myself. I was a little uncertain if I was doing it right by assuming positive intent, and soon, I realized that other contributors liked the way I act and joined the organization and submitted their modules.

Fast-forward to 2017, I cleaned up existing modules, added many missing features and rewrote the most popular AWS modules including VPC, RDS, Autoscaling, ELB, EC2 instance, and EC2 security group. These modules went live as verified Terraform AWS modules during HashiConf 2017 in Austin together with the announcement of the Terraform Registry. Right after the announcement at HashiConf during a break, some people were joking “why the VPC module is such a monster and creates so much…”. Well, because this is a powerful infrastructure module…

Since then I maintain terraform-aws-modules used by hundreds of companies worldwide, reviewing code, answering questions, adding new features, collecting use-cases from users, giving talks and workshops. There were more than 1K of pull-requests and issues open & closed in all modules I maintain, and more than 4 million downloads since the release of the Terraform Registry.Terraform AWS modules are now 0.12 ready!

In June 2019 Amazon recognized my work in the community and called me an AWS Community Hero. It took me several years of doing the things I enjoy.

Today, for the living I provide consulting services and workshops for companies who want to use Terraform and AWS. Most of my customers are using open-source modules and tools, as well as contribute back to open-source (few Terraform modules and projects were released like that, few more — in progress).

Terraform AWS modules were the starting point. Almost immediately after I started maintaining a couple of them (back in 2016) I have been looking into tools which could simplify the review, update, test, release process, and allow to scale up (in quality and in complexity). My ultimate goal is to make a system which will handle hundreds of high-quality modules with fewer efforts on the developer’s side. The first prototype called “terrapin” (see source code) I made in 2017. It was good and working as I wanted, but a bit too over-engineered according to myself after I came back from vacation and looked at it again soberly.

I like to make things which can impact developers or just fun to work on, learn from it, and go on to the next.

At some point, when I already realized that modules in Terraform are the building blocks of infrastructure, I wanted to figure out how to simplify infrastructure as code more and how to make it available for a wider group of people. I bought a start-up ticket to Web Summit 2017 in Lisbon, made a website to illustrate that infrastructure code can be generated by completing easy forms and stayed there for one day pitching my ideas and asking potential users. It turned out that many people didn’t understand the point, they didn’t have the problem I anticipated. They liked nice diagrams created using cloudcraft.co for AWS infrastructure. They liked visual, and I wanted to push Terraform. This was the beginning of modules.tf in the way it is now. Modules.tf — is an open-source project which allows converting visual diagrams created with cloudcraft.co into infrastructure configurations using Terraform.

As a developer myself I wanted users of modules.tf to think about it as an opinionated (yet, powerful) starting point for AWS projects. The project is still early in the development phase. The source code is available here. It is using Serverless Framework with Python 3 and running on AWS Lambda.

While working on modules.tf I needed to connect values in tfvars defined in different directories/stacks/layers dynamically so that outputs of one terraform module can be passed to another terraform module transparently. “Update values in terraform.tfvars using annotations” — this is tfvars-annotations is about. It is currently not complete but already usable for my cases in modules.tf. I also wanted to learn Go language better by doing something small and real.

In 2016, I released pre-commit-terraform — a collection of pre-commit git hooks to take care of Terraform configurations. Those hooks can automatically format, validate, generate documentation for Terraform code. I am glad to see more and more developers are using it in their work.

During my work, I develop various solutions and tools which I prefer to share with the hope that someone will find it useful and begin to cooperate (open issue, give feedback, write code, update documentation, etc). Often people find them useful because it saves them time and makes them more productive. I think this is the reason why some people appreciate “Terraform docs as pdf” I made after I didn’t have Terraform documentation during the cross-Atlantic flight.

After being in the community for some time I’ve gathered enough do and don’ts from real projects and use-cases, so I published them as Terraform Best Practices (source code). Some things are incomplete there, and people still ask me “how to structure my infrastructure as code?” even after I pointed them to this website. For me, this means that I need to find time to update it significantly.

Another piece of educational content I have created recently is — Terraform best practices workshop materials. I run various Terraform AWS workshops on-site for companies and at community events like DevOpsDays and AWS Community Summits.

In May 2019, I decided to share my knowledge regarding Terragrunt and published real code with some of the important points highlighted as Terragrunt Reference Architecture. I believe this will be relevant to people who want to write clean code and use Terraform modules in the most efficient ways.

There are more than 30 HashiCorp user groups in EMEA with several thousands of active members. The number of meetups and quality of talks is increasing. Few organizers are trying out different formats for their events and aiming meetups for different audiences.

The HUG in Hamburg, for example, there was a workshop with a dedicated group of people per product. Some HUGs are mixing advanced and beginner talks during one meetup to satisfy a wider audience. HUG in Zürich is doing a great job by having at least 2–4 talks during a meetup, while most meetups have one talk because it is hard to find good speakers.

When I started the HUG in Oslo, I wanted to have meetups about HashiCorp products and have speakers share their real stories and challenges. I’ve been co-organizing AWS User Group in Oslo since 2015 and already knew that it was very helpful and interesting to learn & share experiences with like-minded individuals. To make it more interesting, I decided to have meetups where attendees can learn about real and advanced use-cases. I don’t want us to repeat official documentation and well-known information which can be found easily online. In general, I want to avoid impractical talks.

Being a smaller community is actually good. The amount of generic DevOps meetups is decreasing, and attendees prefer to have specialization by product and by technologies (CNCF, K8S, Serverless) where they can dive deeper. Additionally, HUG meetups are often relevant to AWS or Google Cloud user groups as long as they touch those services. Win-win, and cross-links :)

In 2018, I proposed to HashiCorp a new format for the event where anyone in the HashiCorp community around the globe can participate and share whatever is interesting for the HUG community worldwide. Each speaker had 30 minutes; the whole event was virtual and lasted 24 hours to cover all time zones. This way HashiCorp All-Day HashiTalks was organized in February 2019.

Finally, If you are still reading this and you want to organize HashiCorp User Group in your area, please contact Katie Reese to help you get started.

Finally, how do you believe the HashiCorp Community plays a part in the overall success of HashiCorp as a Company?

The HashiCorp community is definitely playing a significant role in company success. Open-source versions of their products are driving most of the roadmaps of the individual projects. This is why I always recommend people to open issues if they found bugs or want some feature requests to be considered.

The community is also growing pretty nicely with lots of tools being created to integrate and extend the functionality provided by HashiCorp tools, as well as lots of Slack groups (eg, SweetOps and hangops) where a couple of thousand users are online daily. Some of the tools are being updated to support Terraform 0.12 and HCL2 syntax, while some people in the community discover short-term hacks to adopt Terraform 0.12 and its pretty long list of features as soon as possible. The community is certainly busier now more a month ago :)

PS: I’ve recently studied Amazon Leadership Principles and realized that I apply these principles on a daily basis: “Invent And Simplify”, “Customer Obsession”, “Ownership”, “Are Right, A Lot”, “Learn and Be Curious”, “Insist on the Highest Standards”, “Think Big”, “Bias for Action”, “Frugality”, “Earn Trust”, “Dive Deep”, “Have Backbone; Disagree and Commit”, “Deliver Results”. It didn’t give me a job at AWS if you wonder. :)

https://github.com/antonbabenko

https://twitter.com/antonbabenko

https://www.linkedin.com/in/antonbabenko

https://medium.com/@anton.babenko

Terraform documentation as PDF files

Tue, 08 Jan 2019 23:00:00 GMT

I like to search in it and access it offline (during a flight, for example).

I’ve made a repository where documentation for Terraform core and all official Terraform providers automatically updates every night — https://github.com/antonbabenko/terraform-docs-as-pdf

Hope you find it useful.

Follow me on GitHub to see what I am working on next— https://github.com/antonbabenko

AWS Console — Browse public S3 bucket (without asking for listing permission)

Sat, 22 Dec 2018 12:00:00 GMT

If you want to browse public S3 bucket to list the content of it and download files. You can do so by just logging in to your AWS account and putting a bucket name after https://s3.console.aws.amazon.com/s3/buckets/, to be like this: https://s3.console.aws.amazon.com/s3/buckets/ryft-public-sample-data/ (random bucket name I found online)

This is pretty useful if you don’t know exact filename you want to download from the bucket.

You won’t be able to see bucket policy or change anything sensitive there, which is good. (“You don’t have access to view this configuration. Contact your account administrator to request access.”)

Merry Christmas :)

modules.tf — Convert visual AWS diagram into Terraform configuration

Mon, 19 Nov 2018 09:00:00 GMT

I am excited to announce the public-release of modules.tf, which allows a conversion from visual AWS diagrams using cloudcraft.co to infrastructure code using Terraform.

See this in action (added 21.3.2019):

modules.tf demo — from visual diagrams (cloudcraft.co) to infrastructure as code using Terraform

Features

Resources with arguments once specified during creation of diagrams are present in infrastructure code.
Most popular EC2 types of resources are supported, more will be added in the future. VPC, Security group, Autoscaling Group, RDS, ELB, ALB, SNS, SQS, EC2 instances are implemented already.
Many of relations between resources and their dependencies are respected in infrastructure code. For example, ELB will be created before Autoscaling Group, VPC will be created before Autoscaling Group and ELB.
Incorrect relations are ignored. For example, ELB can’t depend on RDS even if a user draws a connection in a diagram.
The orchestration of infrastructure modules is supported. Thanks to built-in features of Terragrunt.
Dynamic values in terraform.tfvars are supported! Dependent arguments are updated by a link. (Hack or know-how — you decide)
The generated code is good for bootstrap.
The generated code is not a boilerplate which you should rewrite entirely — it is a working solution for small and medium infrastructures.

Excited? Let’s walk through some components to see how to convert your AWS diagram to Terraform infrastructure code.

First, Cloudcraft.co

Here are visual diagrams for 2-tier web-applications hosted on AWS:

Typical 2-tier web-application hosted on AWS may use a couple of auto-scaling groups, a couple of ELBs, RDS instances, CloudFront (CDN) and Route53 (DNS).

Same 2-tier web-application with VPC and security groups added to the picture.

Terraform

Terraform is a well-known (and my favorite) tool to write, plan and create infrastructure as code.

NB: modules.tf generates infrastructure code and relies on Terragrunt for heavy-lifting. Terragrunt is used as a wrapper for Terraform modules, allows orchestration of dependencies, updates of arguments dynamically and keeps infrastructure configurations DRY.

Terraform AWS modules

https://github.com/terraform-aws-modules — these building blocks of AWS infrastructure are used by thousands of people around the globe. As a core maintainer, it was natural for me to bring them as first-class citizens to modules.tf.

Let’s try it!

Once you draw AWS infrastructure, which can be a single resource or several interconnected resources (as shown above), you should click “Export” and then “Terraform code export” in top-right corner.
Zip-archive with all configurations and extra files will be generated dynamically.
Extract content of this archive into your working directory and read README.md which is inside. It contains all information about dependencies, and how to create the desired infrastructure.

Notes

modules.tf is open-source, MIT licensed. Source code: https://github.com/antonbabenko/modules.tf-lambda
modules.tf is using AWS Lambda and running Serverless framework with Python 3.6
modules.tf is free for all. It is available for all types of users on cloudcraft.co (including free users)
modules.tf has support for a limited amount of AWS resources, because we don’t have Terraform AWS modules for all of them (yet).
This is the first iteration of public release, and definitely contain bugs. I couldn’t imagine what kind of scenarios people were building, so PLEASE report problems on GitHub issues. I am especially interested in learning about any usability issues or problems with the developer experience. What makes you stuck? What is not logical? What is simply missing? What is useless? What would be AMAZING?
modules.tf — is also a static web-site, but I have not updated it yet :)

Summary

Go to cloudcraft.co, open an account (free is fine, too), draw your AWS infrastructure, click “Export” then “Terraform code export”, download and unzip archive, read README.md.

Atlantis + Terragrunt = ?

Wed, 21 Feb 2018 19:00:00 GMT

TLDR; Does Atlantis work with Terragrunt? The short answer — pretty much, yes (updated 18.9.2018).

UPD (7.5.2019): Published new repository Terragrunt Reference Architecture where the very real code is published. Check it out if you want to see Terragrunt in action.

UPD (18.9.2018): Updated blog post, added a link to Terragrunt use-case in Atlantis documentation.

UPD (25.5.2018): I have just published Terraform module for running Atlantis on AWS Fargate — https://github.com/terraform-aws-modules/terraform-aws-atlantis

When I wrote this blog post in February 2018, Atlantis didn’t have proper integration with Terragrunt and a lot of things have changed since that time. Please check Terragrunt use-case in Atlantis documentation because some of the following may be outdated.

I go through my experience of integrating these two tools:

Terragrunt — Thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules.
Atlantis — A unified workflow for collaborating on Terraform through GitHub and GitLab.

Both of these projects are open-source and have decent community (1000 stars and 39 contributors — Terragrunt; 600 stars and 13 contributors — Atlantis).

Terragrunt

I use Terragrunt because I like the way my Terraform code is structured and I’d like to keep it DRY. I use Terraform AWS modules and promote reusability whenever possible (internally — with private infrastructure modules shared between teams; externally — with resource modules published in terraform-aws-modules).

Most of experienced users I know tend to not mix Terraform configurations (*.tf) with the actual values (*.tfvars) when working on medium/large setups. I like to hide Terraform code from users by just giving them access to values they should control (of course, they contribute to Terraform configurations, but it happens not in the same place they play with values). Terragrunt helps with this quite a lot.

Atlantis

Once code is pushed, developer opens a pull-request and pull-request reviewers can see Terraform plan before merge. This is the simplest functionality I wanted to achieve with Atlantis.

Now, let me explain a project structure and see whether Atlantis can help.

Project setup:

Different projects are located in directories like project-env/region/layer. For example: user-stage/eu-west-1/vpc — has code for VPC layer deployed in eu-west-1 region used by user project in staging environment.
Each project has its own set of AWS IAM roles which should be assumed to be able to change infrastructure (readonly role — to do terragrunt plan, power-user role — to do terragrunt apply)

Let’s get it working together…

Following setup instructions described on Atlantis project page was pretty easy to get a Terraform project of supported structure to work. My project structure was not supported out of the box, and in order to make Atlantis to work with my structure I used to come up with some hacks.

Which binary to run?

It is possible to specify version of Terraform using terraform_version parameter in configuration file (atlantis.yaml), but not complete path to a binary or shell wrapper, for my case.

Solution #1— make terraform an alias to terragrunt wrapper script and tell wrapper script where correct terraform actually is:alias terraform=/usr/local/bin/terragrunt.sh

# terragrunt.sh
export TERRAGRUNT_TFPATH=/usr/local/bin/terraform

Solution #2 — change PATH environment variable to resolve a terragrunt shell wrapper instead of the correct terraform binary and rename shell wrapper to terraform.mv /usr/local/bin/terragrunt.sh /usr/local/my-bin/terraform
export PATH=/usr/local/my-bin:$PATH

# terragrunt.sh
export TERRAGRUNT_TFPATH=/usr/local/bin/terraform

I leave to you to decide which solution is less ugly :)

Ideally, there should be a good way to specify path to a binary through configuration.

Workspaces? No, please!

I don’t use Terraform workspaces and don’t want any tool to decide this for me. This should be controllable and should be disabled by default on my opinion. In Atlantis it is not possible to disable this.

Arguments are appended

Similar to workspaces, Atlantis assumes I use terraform and appends some arguments to commands it run (eg, -no-color). In some cases it works well, but it expects a single string as an output of terraform version so I used to make another hack in my shell wrapper to make Atlantis happy:# terraform version should return single string, but terragrunt outputs a bit more
if [[ "$1" == "version" ]]; then
echo "Terraform v0.11.3"
exit 0
fi

Configuration file

This is the main show stopper. Atlantis expects atlantis.yaml to be placed in a project root. In a single repository we have dozens of projects at the same time, so there should be at least the same amount of configuration files, which is simply not acceptable for us. 100% of my projects could be served with the same atlantis.yaml file. Remember, Terragrunt helps to make infrastructure configurations DRY.

It would be good to support some inheritance of configuration file, or have an environment variable, or command line argument to specify the configuration file to use.

Project structure

Atlantis supports project structures described here. I have opened an issue 523 which describes desired project structure and will be glad to elaborate as time allows.

I think Atlantis should have a list of whitelisted and blacklisted directories where Atlantis has to check for changes and run commands.

Wouldn’t it be great to be able to write atlantis apply user-stage/eu-west-1 it a pull-request comment and it understands what is a project and make things happen. I think it is an awesome goal.

More points and ideas for another time:

Expose terraform execution via API to be able to trigger same commands using other webhooks (eg, in Slack messages — /atlantis apply user-stage/eu-west-1 or /atlantis output user-stage/eu-west-1)
Assume role support is tightly connected to Terraform at the moment and does not work like this with Terragrunt which support CLI argument and environment variables.
Assume role with MFA is impossible at the moment. Add integration with Slack or Twilio to send request to a person who has requested atlantis apply.
Reformat output of terraform plan which is pasted as a comment into collapsable list (using “<details><summary>Summary</summary>Details…</details>” tags)

Summary

I want to say that Atlantis is a great young project which is very much needed in the Terraform community. Let’s contribute our time to it (by trying it at least) and make it even better— https://github.com/runatlantis/atlantis.

Stay in touch: https://twitter.com/antonbabenkoat

Make Terraform faster — skip unnecessary checks (AWS and S3)

Sat, 27 Jan 2018 10:00:00 GMT

Working with Terraform on a daily basis I feel it can be a good idea to share some small tips I have. Let me start with this one.

Initializing remote state S3 backend is pretty fast operation, but it can be even faster if you disable all unnecessary checks:

terraform {
  backend "s3" {
    encrypt        = true
    region         = "..."
    key            = "..."
    bucket         = "..."
    dynamodb_table = "..."

    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_credentials_validation = true
  }
}

provider "aws" {
  region = "..."
  skip_requesting_account_id  = true # this can be tricky
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_credentials_validation = true
}

Notes:

Skipping region validation on S3 backends became available in Terraform 0.11.2 (see PR): skip_region_validation = true

I didn’t spend too much time measuring time difference, but it certainly reduces time required for initializing S3 backend by 20%. This is not a big deal if you have just a few remote states and you run terraform init seldom.

I usually use Terragrunt with lots of isolated state files and run terragrunt validate-all on CI. It saves second here, second there.

More importantly skipping these checks will allow you to run terraform plan without having internet access (on the plane, for eg).

Follow me on Twitter — @antonbabenko

Terrapin: making good Terraform modules

Sat, 15 Jul 2017 08:00:00 GMT

You are reading the second blog post in the series about using Terraform continuously. In the previous one I outlined common traits in Terraform modules.

Once more, the list of keywords describing common traits is: not clean code, inextensible module features, tests, defaults, examples, documentation, lifecycle, versioning, security.

Today, let me introduce a new open-source project called Terrapin. In a nutshell Terrapin is a Terraform modules generator.

The ultimate goal of Terrapin is to automatically generate structured, tested, easy-extendable and well-documented Terraform modules with predictable versioning strategy.

Being a new project now it can only generate structured, extendable, documented and testable code (all with varying level of success).

There is already a documentation and sample module available.

Open an issue on github, if there is anything you want to ask or leave a feedback.

Thank you!

Using Terraform continuously — Common traits in modules

Thu, 08 Jun 2017 18:00:00 GMT

This is the first blog post in the series about using Terraform continuously, which I am going to publish over the next few weeks.

UPD (15.07.2017): I wrote another blog post in the series — Terrapin: making good Terraform modules

I am Anton Babenko (github.com/antonbabenko) one of the active maintainers and contributors of several terraform-community-modules. Since I started using Terraform back in mid-2015, I have talked to many people (online and during conferences) who are using Terraform to build their infrastructures, so it feels natural to write some blog posts about what I am experiencing.

I want to hear feedback, so please leave a comment below if you have something to say (or agree, or disagree). You can also send me an email and I will reply to you.

Let’s start with important things first — Terraform modules.

What is Terraform module?

Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group. Modules are used to create reusable components, improve organization, and to treat pieces of infrastructure as a black box.

(via https://www.terraform.io/intro/getting-started/modules.html)

Overall there are around 15 actively used Terraform community modules and lots of custom built modules (often niche one’s). Those modules were created to solve common scenarios — create network components (VPC, subnets, NAT gateways, etc) and various types of resources (RDS, ECS, Elasticache, etc).

Common traits in Terraform modules

Not clean code

Quote by Nick Dellamaggiore, Infrastructure Lead at Coursera, taken from the book «Terraform: Up & Running» by Yevgeniy Brikman:

If I look at a single file and it’s written by 10 different engineers, it should be almost indistinguishable which part was written by which person. To me, that is clean code.

In collaborative environment following coding style, naming conventions and agreed best-practices is important, because it lets others contribute faster.

Clean code is also aggregative term and is related to other traits listed below.

There is no single set of Terraform configurations best-practices yet, which most of Terraform users know and follow regularly. List of traits below may initiate discussions, which may turn into statements, which may turn into “Terraform modules best-practices”. Who knows?

Module features

Modules should be extensible. Often developers are taking single-purpose design principle too literally, which leads to not-extensible set of features which makes it impossible to extend without forking it (for example, there are 128 forks of very popular terraform-community-modules/tf_aws_vpc module). Good modules are as flexible as Terraform allows us to do without complicated hacks. For example, if there is a security group in the module, then it should be created as two resources — aws_security_group and aws_security_group_rule, so that user can append some rules, if necessary. One of the limitation which is still hard to do nicely is related to resource tags.

Tests

It is good practice in software development to include tests for your code and run them often. In my opinion it is still not well-covered area when it comes to Terraform configurations even though there are several tools which can help doing this. At the very least you should have a CI system run “terraform plan” with sane defaults. Apply code to create resources and then destroy them is more tricky, and it is fine to do it less often, but still automatically.

Sometimes there are syntax errors in completed code and in pull requests, which basic “terraform plan” can catch.

Defaults

Modules should have sane defaults. By sane I mean that it should be possible to use the module providing just essentially important arguments and use module’s defaults for the rest.

Examples

It is helpful for the users of the module to be able to see complete examples of the module and be able to run plan and apply on it effortlessly.

Documentation

…and deficit of it. Documentation should outline the purpose of a module, which resources it creates, how to use the module, required version of Terraform, how to report issues, etc.

Lifecycle

Module resources lifecycle should implement create, read, update and destroy functions. Resources creation is easy to handle in code, and this is often the main reason why the module was created. Read and destroy functions are also easy, because once created you should be able to read and destroy resources (bear with me “easy” is subjective). Update function is the most complicated, because of variety of resources and ways how they can be updated (replace in place, replace gradually, replace all at once, blue-green, etc). Modules are almost never implemented with a clear path through updates.

Versioning

Infrastructure as code should always be versioned. Versioning strategy and available releases should be documented. It is handy to be able to use specific release instead of using “master” branch for your critical infrastructure.

Security

You should always attempt to use the principle of least privilege when working with infrastructure. It is especially important when multiple operators handle infrastructure in self-service way. Module should specify set of least privileges it uses.

For now this is it, and in the next blog post I want to propose some ideas to the points listed above.

Thanks to Tim Hartmann, Steve Huff and Alex Matsuk for initial review and proof-reading.

Running "terraform fmt" as pre-commit git hook [archived]

Tue, 27 Sep 2016 15:10:00 GMT

This is my second blog post related to Terraform. Previous one was about how I structure Terraform configuration, which got some interesting feedback on reddit.

This time I want to share pre-commit hook which runs "terraform fmt" on your Terraform configuration files. Nothing more, nothing less.

Code is here - antonbabenko/pre-commit-terraform

How I structure Terraform configurations? [archived]

Wed, 21 Sep 2016 17:05:00 GMT

Note that this blog post was originally written in September 2016, and many things has been changed since then.

In this blog post I want to describe one of the ways I like to structure
Terraform configurations which can be used to manage medium and large
infrastructures with resources span multiple AWS accounts (for eg,
development, staging, production) and multiple regions.

Here I follow so called "layered infrastructure principle".

There was very interesting talk about running Terraform At Scale by
Calvin French-Owen (Co-Founder and CTO at Segment) at HashiConf 2016,
where he was describing same types of challenges I was solving in my setup.

Infrastructure review:

Resources are grouped into layers, where each layer consists only of:

global resources (IAM, Cloudfront, Cloudtrail, Config, Route53 zones)
or regional resources (EC2, VPC, Codedeploy, Cloudformation)

Developer can interact with a single layer, which is identifiable by combination of arguments:

AWS account alias (company-dev, for example)
Region name (eu-west-1, for example)
Layer name (global, shared, application, for example)

Directory structure:

.
├── accounts
│   ├── company-dev
│   │   ├── application.eu-west-1.tfvars
│   │   ├── global.tfvars
│   │   └── shared.eu-west-1.tfvars
│   ├── company-cicd
│   │   ├── cicd.eu-west-1.tfvars
│   │   └── global.tfvars
│   ├── company-legacy
│   │   └── legacy.tfvars
│   ├── company-prod
│   │   ├── application.eu-west-1.tfvars
│   │   ├── global.tfvars
│   │   └── shared.eu-west-1.tfvars
│   └── company-staging
│       ├── application.eu-west-1.tfvars
│       ├── global.tfvars
│       └── shared.eu-west-1.tfvars
├── layers
│   ├── company-dev
│   │   ├── application
│   │   │   ├── main.tf
│   │   │   ├── output.tf
│   │   │   └── variables.tf
│   │   ├── global
│   │   │   ├── main.tf
│   │   │   ├── output.tf
│   │   │   └── variables.tf
│   │   └── shared
│   │       ├── main.tf
│   │       ├── output.tf
│   │       └── variables.tf
│   ├── company-infra
│   │   ├── cicd
│   │   │   ├── iam.tf
│   │   │   ├── main.tf
│   │   │   ├── output.tf
│   │   │   └── variables.tf
│   │   └── global -> ../../company-dev/global
│   ├── company-static
│   │   └── static
│   │       ├── main.tf
│   │       ├── output.tf
│   │       └── variables.tf
│   ├── company-prod
│   │   ├── application -> ../../company-staging/application
│   │   ├── global -> ../../company-staging/global
│   │   └── shared -> ../../company-staging/shared
│   └── company-staging
│       ├── application
│       │   ├── main.tf
│       │   ├── output.tf
│       │   └── variables.tf
│       ├── global
│       │   ├── main.tf
│       │   ├── output.tf
│       │   └── variables.tf
│       └── shared
│           ├── main.tf
│           ├── output.tf
│           └── variables.tf
├── modules
│   ├── application
│   │   ├── codedeploy.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── global
│   │   ├── cloudtrail.tf
│   │   ├── iam.tf
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── shared
│   │   ├── bastion.tf
│   │   ├── networking.tf
│   │   ├── output.tf
│   │   ├── route53.tf
│   │   ├── variables.tf
│   │   └── vpc_peering.tf
│   └── typical_service
│       ├── codedeploy.tf
│       ├── output.tf
│       ├── service.tf
│       └── variables.tf
└── terraform.sh

Notes:

accounts/{aws_alias}/*.tfvars - values
layers/{aws_alias}/{layer_name}/*.tf - Terraform configurations. Normally it contains module invocations and terraform_remote_state configurations to higher layers (global and shared).
layers/company-prod/* - is a symlink to layers/company-staging/* to reduce chance of configuration drift
modules - Terraform modules (typically invoked from multiple AWS accounts with various values)
terraform.sh - Terraform wrapper script (or Terragrunt), which sets correct working directory and load correct tfvars file

Pros:

Easy to navigate infrastructure as code
Outputs of higher level layers are available as read-only values to layers below
Full set of Terraform features (commands and arguments) is supported
Versioned layers based on same tfvars can be implemented (A/B testing, blue-green deployment)
Very easy to integrate with CI pipeline
Locking is implemented using Terragrunt

Cons:

May be hard to limit permissions for Terraform users between state files inside one AWS account (all states files are readable by users in that account, by default)

Lessons learned from AWS and Varnish [archived]

Mon, 09 Mar 2015 09:18:00 GMT

In this post I want to share a few thoughts I figured out during my latest assignment where I used to design and then implement architecture for e-commerce web-site for pet owners.

The project was built using Symfony2 PHP framework with rather high demands on code quality as well as performance for end-users.

There was multi-tier architecture consisting of several Amazon web-services like:

Route 53 (DNS)
Cloudfront (CDN)
Elastic load-balancer
Front-caching layer (EC2 instances with Varnish)
Internal elastic load-balancer
Application layer (auto-scalling group of stateless web-servers)
RDS, Elasticache, etc

The architecture looks rather common and there is a lot of information already online, so I won't go in details here. An interesting and challenging part in the implementation of that architecture was related to the ELB service and how exactly TCP connections handled by AWS services (ELB and Cloudfront).

Error "502 Bad Gateway"

The error I was struggling was "502 Bad Gateway - CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection".

Even after reading such detailed error message I could not find the right combination of values for:

ELB timeouts and draining connection
timeout_idle for Varnish
nginx keep-alive setting

After some time AWS support gave a clear explanation for the problem:

Varnish apparently sitting with a timeout_idle of 20 seconds, means that Varnish will at times close the TCP connection that the ELB has established to it, while the ELB will continue to think that this connection is still available. If the ELB then tries to send a client request down this connection at a later stage, it will receive an exception, and this will cause the ELB to close the client-side connection with no response.

Sounds clear, I modified timeouts according to their suggestions (most critical piece was to increase timeout_idle in Varnish config from 20 to 60 seconds and it all just works).

BUT! I would not write this post, if it would be just one line fix... :)

Question: Why there is even a connection between ELB and the backend, if I don't make any requests ?

Answer:

ELB pre-opens TCP connections to the back-ends, even when it does not have a request to dispatch on them. It does this as a performance optimization to reduce latency when a request is eventually available to be dispatched along the TCP connection. The ELB assumed that any TCP connection it has between itself and your back-ends can be left idle for the idle timeout and that the back-end will not close it before this time."

This answer put everything together and it explained why I didn't see an issue while debugging it with ELB excluded.

Also, worth mentioning, Cloudfront on some requests may return cached data completely from the cache, and then ELB is not contacted and therefore backend connection was not established in advance. So, part of our sites were never affected. No ELB = no problem, hehe.

Cloudfront tips

There are few behavior settings (like headers, cookies and query string forwarding) which can dramatically decrease amount of requests to the backend. If misconfigured then you may serve wrong content to users. Enabled "Query string forwarding" together with Varnish's sub vcl_hash { hash_data(std.querysort(req.http.x-url)); } keeps single cached copy in Varnish cache, while bypassing Cloudfront.

Since most of end-users were located in Sweden, where AWS has edge location, by just adding Cloudfront after Route 53 we got average time to connect decreased 80-100ms comparing to time when Route 53 was pointing to ELB. Minor improvement, and for free.

Varnish tips

The biggest performance improvement happened after I revised parts of Varnish configs regarding cookies and ESI, so that average page load time decreased from 1 second per page to 150ms.

Few points on that:

Try to make an application to serve as much content as possible independently of cookies. Criticize your work!
Whitelist just accepted cookie names (should be always in-sync with developers and marketing people who like to add something with Google Tag Manager)
When your application depends on some cookies then consider to put that part of your application info light-weight ESI call, which will do just a little cookie-dependent-magic, while the rest of the page is cookie agnostic (see point 1)
When your application is setting cookies you ideally should know which urls to allow to set cookies and whitelist them (also require developers awareness)
If performance is very important, then use libvmob-cookie instead of modifying headers with regexps.

Use something what provides an integration with caching proxy, so that developers can control large part of ESI and cookies magic themselves - for PHP there is FOSHttpCache, which is awesome and it can save a lot of time.

Anton Babenko

Deliberation Dashboard: see and analyze your consensus results

You cannot fix a panel you cannot see

The analyzer tells me which model to drop

Every run remembers the config it ran with

Slow is fine, unbounded is not

Models argue about today, not their training cutoff

Smaller things that removed friction

What I would still not trust it with

Try it

Meet Deliberation: 400+ models is easy, knowing which ones earn a place is hard.

A few words first

Install it now

What actually changed

Quick one-offs: just ask

1. Fan-out with no cross-talk

2. The consensus loop, where the voting lives

3. Synthesis, when there is no verdict to give

4. Two drivers, one rulebook

5. Make the panel stay accountable

The parts I will not oversell

The takeaway

One model is a guess. Three that agree is a plan.

One model is a guess

What /consensus actually does

The quieter two

A note on claude-delegator

Why a plugin, not a blog post

The takeaway

Your Weekly Dose of Terraform — Live Streams

I do live streams talking about things I enjoy working with — IaC, DevOps, Terraform, and AWS.

Anonymized, Secure, and Free Terraform Cost Estimation

I don't want to send my Terraform data, because I have secrets there!

Bonus: Run Terraform Cost Estimation from the browser

"Terraform AWS modules" as a code

My Terraform AWS journey — HashiTimes Interview

Tell us a little about yourself, your background around HashiCorp, and what do you do professionally?

Anton, you have been a very active member in the HashiCorp Dev Community with some of your open source projects, Terraform Modules, and other tools available today. Can you share with the readers what modules.tf does, terraform docs as pdf and how these tools can enable them as Developers?

You were instrumental in launching the HashiCorp User Groups program along with Kate Reese @ HashiCorp. For those in the community who have not attended an EMEA region HUG before, could you share the success you have seen in the EMEA region growing the HUG community?

Finally, how do you believe the HashiCorp Community plays a part in the overall success of HashiCorp as a Company?

Please share this article on your social networks and feel free to connect with me:

Terraform documentation as PDF files

AWS Console — Browse public S3 bucket (without asking for listing permission)

Read more

modules.tf — Convert visual AWS diagram into Terraform configuration

Features

First, Cloudcraft.co

Terraform

Terraform AWS modules

Let’s try it!

Notes

Summary

Atlantis + Terragrunt = ?

Terragrunt

Atlantis

Project setup:

Let’s get it working together…

Which binary to run?

Workspaces? No, please!

Arguments are appended

Configuration file

Project structure

More points and ideas for another time:

Summary

Make Terraform faster — skip unnecessary checks (AWS and S3)

Terrapin: making good Terraform modules

Using Terraform continuously — Common traits in modules

What is Terraform module?

Common traits in Terraform modules

Not clean code

Module features

Tests

Defaults

Examples

Documentation

Lifecycle

Versioning

Security

Running "terraform fmt" as pre-commit git hook [archived]

How I structure Terraform configurations? [archived]

What `/consensus` actually does