BIG-bench vs Terracotta

BIG-bench measures how well large language models handle reasoning, math, bias, and multilingual tasks across more than 200 community-written evaluation challenges. Google hosts the open source repository on GitHub, where researchers contributed tasks through pull requests and published comparative model scores on the leaderboard. Each task scores models through text generation or log-probability queries, using metrics like BLEU, BLEURT, and exact string match.

Unlike fixed benchmarks such as GLUE or SuperGLUE, BIG-bench grew through community pull requests, so task authors could submit challenges designed to exceed what existing models could solve. The suite also ships BIG-bench Lite, a 24-task subset that gives a cheaper canonical score across the full collection of 200+ tasks. Programmatic tasks support multi-turn model interaction, while JSON tasks work through a simpler task.json format with built-in scoring rules.

ML researchers use BIG-bench to compare model scaling trends and publish leaderboard results. Model developers run evaluations locally with HuggingFace models or through Docker scripts, then submit score files via pull request. The benchmark is archived and read-only as of April 2026, but the tasks, code, and published TMLR 2023 analysis paper remain available for reproducible research.

Terracotta is an Infrastructure as Code governance tool that audits every pull request before merge, checking Terraform and OpenTofu changes against live cloud resources, remote state, and your team's governance policies. It installs as a GitHub or GitLab app, posts findings in the PR thread, and builds a tamper-evident audit trail regulators can export. The product targets platform engineering teams that need security, drift, cost, and compliance checks without rewriting CI pipelines.

Static scanners like Checkov or tfsec lint HCL syntax and known misconfigurations, but they never compare a plan to what is actually running in AWS. Terracotta closes that gap by correlating code, Terraform state, and live resources so drift, cross-PR conflicts, and hidden blast radius show up before anyone clicks merge. Its guardrails are written in plain English rather than Rego or Sentinel, which lowers the bar for teams that lack a dedicated policy-as-code engineer.

DevOps leads and platform engineers at regulated shops use Terracotta to block public S3 buckets, open SSH rules, and unapproved cost spikes at review time instead of in production. Security and compliance teams get a fleet-wide dashboard with drift posture, policy compliance rates, and exportable records for SOC 2 or HIPAA audits. Developers keep working inside GitHub or GitLab because findings arrive as PR comments, not another portal to check.

Beacon, Terracotta's in-PR chat assistant, answers questions about specific findings using context from the repo, plan output, and drift reports. The Platform tier adds unlimited drift repos, IAM and blast-radius analysis, Slack notifications, and a command center dashboard for $49 per engineer per month. Enterprise customers can run Terracotta self-hosted with SSO, SAML, and custom integrations for HCP Terraform or CircleCI.