HashiCorp Nomad is a distributed workload orchestrator designed to deploy and manage containers, batch jobs, long-running services, legacy applications, and other workloads across clusters of machines.
If Docker answers:
“How do I run this container?”
and Kubernetes answers:
“How do I orchestrate containers across a cluster?”
Nomad takes a somewhat broader and simpler approach:
“How do I schedule and operate different kinds of workloads across my infrastructure?”
Nomad can run Docker containers, binaries, Java applications, QEMU workloads, and other task-driver-based workloads. HashiCorp describes it as a highly available, distributed, datacenter-aware scheduler designed for services, batch jobs, and more.
1. What Is HashiCorp Nomad?
Nomad is a cluster scheduler and workload orchestrator developed by HashiCorp.
Its core responsibilities include:
- Scheduling workloads
- Allocating CPU and memory
- Placing workloads on appropriate nodes
- Running long-lived services
- Running batch jobs
- Running jobs on every node
- Handling workload failures
- Rolling out updates
- Managing task lifecycle
- Service registration
- Integrating with Consul
- Integrating with Vault
- Supporting multiple datacenters
- Supporting heterogeneous workloads
A simplified architecture is:
NOMAD │ ┌────────┴────────┐ │ │ Nomad Servers Nomad Clients │ │ Scheduling Run workloads Cluster State Containers Coordination Binaries │ Batch jobs │ Services ▼ Allocations
Nomad's servers handle scheduling and cluster management, while clients execute the workloads assigned to them.
2. Why Nomad?
Traditional application deployment might look like:
Server 1 → Application A Server 2 → Application B Server 3 → Application C Server 4 → Application D
As infrastructure grows, you need to answer:
- Where should an application run?
- Does the server have enough CPU?
- Does it have enough memory?
- Does it have the required runtime?
- What happens if the server fails?
- How do we deploy five copies?
- How do we update them?
- How do we run a job every night?
- How do we deploy something to every node?
- How do services discover each other?
Nomad provides a scheduling and reconciliation layer.
Desired State │ ▼ Nomad Job │ ▼ Scheduler │ ▼ Allocation Plan │ ▼ Nomad Clients │ ▼ Workloads
3. Nomad vs Kubernetes
The obvious comparison is Kubernetes.
They overlap substantially, but their architecture and operational models differ.
| Area | Nomad | Kubernetes |
|---|---|---|
| Primary role | Workload orchestrator | Container orchestration platform |
| Configuration | HCL jobspec | YAML/JSON APIs |
| Control plane | Servers | Control-plane components |
| Worker | Client | Worker node |
| Basic workload abstraction | Job / task group / allocation | Pod |
| Containers | Yes | Yes |
| Docker | First-class task driver | Usually through CRI-compatible runtimes |
| Batch workloads | Strong | Jobs/CronJobs |
| System-wide workloads | System jobs | DaemonSets |
| Service discovery | Native + Consul | Kubernetes Services/DNS |
| Service mesh | Consul integration | Ecosystem-dependent |
| Secrets | Vault integration | Kubernetes Secrets/external systems |
| Web UI | Yes | Kubernetes ecosystem |
| Operators | Not central to core model | Major Kubernetes ecosystem pattern |
| Complexity | Generally smaller operational footprint | Broader, more extensive platform |
| Legacy binaries | First-class workload possibility | Usually containerized |
| Windows workloads | Supported | Supported with limitations/architecture considerations |
Nomad's documentation emphasizes simplicity: it can run as a single process without external dependencies and supports containers, legacy workloads, microservices, and batch workloads.
That does not mean Kubernetes is inherently inappropriate or Nomad is inherently preferable. The appropriate platform depends on workload requirements, ecosystem integration, operational model, and organizational needs.
4. Nomad's Architecture
A Nomad cluster consists primarily of:
Nomad Cluster │ ┌────────────┴────────────┐ │ │ Servers Clients │ │ │ ┌──────┼──────┐ │ │ │ │ │ Node1 Node2 Node3 │ ├── API ├── Scheduler ├── Raft └── Cluster State
Nomad servers are responsible for accepting jobs, managing clients, and computing workload placement. Clients register their resources and execute allocations.
5. Nomad Servers
Servers form the control plane.
They are responsible for:
- Receiving jobs
- Maintaining cluster state
- Scheduling workloads
- Tracking clients
- Creating allocations
- Handling evaluations
- Maintaining consensus
Nomad uses Raft consensus between servers.
A typical production deployment uses:
3 Servers
or:
5 Servers
HashiCorp's reference architecture recommends three or five Nomad servers for a production cluster, with clients providing the execution capacity.
6. Nomad Clients
Clients are the machines that actually run workloads.
A client reports information such as:
CPU Memory Operating System Architecture Drivers Node attributes Available resources
For example:
Nomad Client │ ├── CPU: 16 cores ├── RAM: 64 GB ├── Docker ├── Java ├── QEMU └── Nomad Agent
The scheduler uses this information when deciding where to place allocations.
7. Server vs Client
Think of it this way:
NOMAD SERVER │ "Where should this run?" │ ▼ Scheduler │ ▼ Allocation │ ▼ NOMAD CLIENT │ "Run this workload." │ ▼ Task Driver │ ▼ Application
This separation is fundamental.
8. Nomad Regions
Nomad supports the concept of regions.
A region contains:
Nomad Servers + Nomad Clients
For example:
Region: India │ ├── Server 1 ├── Server 2 ├── Server 3 │ ├── Client 1 ├── Client 2 └── Client 3
A Nomad region can include clients from multiple datacenters.
9. Multiple Regions
You can have:
Nomad │ ┌─────────────┴─────────────┐ ▼ ▼ Region A Region B India Europe │ │ Clients Clients
Nomad supports federating multiple regions.
However, regions are independently managed from a state perspective; they do not simply replicate all workload state between one another.
10. Datacenters
Nomad jobs can target datacenters.
Example:
job "web" { datacenters = ["dc1", "dc2"] ... }
This allows workload placement to be controlled geographically or architecturally.
For example:
Region │ ├── dc1 │ ├── client │ └── client │ └── dc2 ├── client └── client
11. The Four Most Important Nomad Concepts
HashiCorp identifies four primary concepts:
Jobs Nodes Allocations Evaluations
The relationship is:
Job │ ▼ Evaluation │ ▼ Scheduler │ ▼ Allocation │ ▼ Node / Client │ ▼ Tasks
Understanding this flow makes Nomad significantly easier to learn.
12. Jobs
A Job describes the desired state of an application or workload.
For example:
Run nginx 3 instances 1 CPU each 512 MB RAM Only on Linux Expose port 80
The job does not normally say:
“Run this on server-02.”
Instead it describes:
“This is what I need.”
Nomad determines placement.
HashiCorp describes a job as a user-specified desired state containing tasks organized into task groups and associated resource requirements.
13. Jobspec
Nomad job specifications are written in HCL.
Example:
job "web" { datacenters = ["dc1"] group "web" { count = 3 task "nginx" { driver = "docker" config { image = "nginx:1.28" } resources { cpu = 500 memory = 256 } } } }
This is one of the biggest differences from Kubernetes.
Instead of:
apiVersion: apps/v1 kind: Deployment
Nomad uses:
job "web" { ... }
14. HCL
HCL stands for:
HashiCorp Configuration Language
It is used across HashiCorp products.
For example:
Terraform → HCL Nomad → HCL Vault → HCL Consul → HCL
That makes Nomad particularly familiar to organizations already using HashiCorp tooling.
15. Task Groups
A Job contains one or more task groups.
Example:
job "application" { group "frontend" { task "web" { ... } task "proxy" { ... } } }
A task group represents tasks that should be scheduled together.
Conceptually:
Job │ ├── Group: frontend │ ├── Task: web │ └── Task: proxy │ └── Group: worker └── Task: worker
16. Tasks
Tasks are the actual workload definitions.
Examples:
Docker container Java application Binary QEMU VM Podman workload
The task driver determines how the task runs.
17. Allocations
An allocation is the placement of a task group on a specific client.
For example:
Job │ └── Group │ ├── Allocation A → Client 1 ├── Allocation B → Client 2 └── Allocation C → Client 3
This is one of the most important Nomad concepts.
An allocation answers:
Which workload instance is running on which client?
18. Evaluations
An evaluation occurs when Nomad needs to reconsider scheduling.
For example:
New job submitted ↓ Evaluation ↓ Scheduler ↓ Placement
Or:
Client fails ↓ Evaluation ↓ Rescheduling ↓ New allocation
HashiCorp describes evaluations as being generated when desired or external state changes.
19. Scheduling
Nomad scheduling consists broadly of:
Feasibility ↓ Ranking ↓ Placement
First, Nomad filters nodes that cannot run the workload.
Then it ranks feasible nodes.
Finally, it creates an allocation plan.
20. Constraints
Constraints are hard requirements.
Example:
constraint { attribute = "${attr.kernel.name}" operator = "=" value = "linux" }
This means:
Don't run the workload on a node that doesn't satisfy this requirement.
Other possible constraints can involve:
- OS
- Architecture
- Datacenter
- Node pool
- Metadata
- Hardware
- GPU availability
- Custom attributes
21. Affinity
Affinity is different.
A constraint says:
Must run here.
Affinity says:
Prefer running here.
For example:
Constraint ────────── GPU required Affinity ──────── Prefer GPU with A100
Nomad supports both hard constraints and soft preferences through affinities.
22. Resource Scheduling
Nomad schedules workloads according to requested resources.
Example:
resources { cpu = 1000 memory = 512 }
Conceptually:
Task requires: CPU = 1 CPU Memory = 512 MB
Nomad evaluates whether a client has enough available capacity.
23. Bin Packing
Nomad's scheduler can use bin packing to efficiently place workloads.
Imagine:
Node 1 CPU: 16 RAM: 64 GB
You have workloads:
A → 4 CPU B → 4 CPU C → 2 CPU D → 2 CPU
Nomad can attempt to place workloads efficiently rather than unnecessarily spreading every workload across machines.
HashiCorp describes its placement algorithm as considering resource utilization and constraints, with bin packing used to maximize utilization while avoiding resource exhaustion.
24. Task Drivers
Task drivers determine how Nomad executes tasks.
Bundled drivers include:
Docker Java QEMU Raw Fork/Exec Isolated Fork/Exec
Additional plugins include technologies such as:
Podman Exec2 Virt
and community-developed drivers.
This is a major Nomad differentiator.
25. Docker Driver
The Docker driver provides first-class Docker support.
Example:
task "web" { driver = "docker" config { image = "nginx:1.28" } }
Nomad can handle:
- Pulling images
- Starting containers
- Monitoring containers
- Cleaning up containers
- Port mapping
- Volume configuration
26. Raw Execution
Nomad can also execute host binaries through appropriate task drivers.
This means you aren't forced to containerize every workload.
For example:
Legacy Application ↓ Binary ↓ Nomad Client
This can be useful for organizations migrating legacy applications gradually.
27. Java Workloads
Nomad has a Java task driver.
Conceptually:
Nomad ↓ Java Driver ↓ JAR ↓ JVM ↓ Application
This makes Nomad useful in environments that have both containerized and traditional application workloads.
28. QEMU
Nomad can also manage QEMU workloads.
Conceptually:
Nomad ↓ QEMU Driver ↓ Virtual Machine
This broadens Nomad beyond pure container orchestration.
29. Job Types
Nomad provides several schedulers/job types:
service batch system sysbatch
30. Service Jobs
Service jobs are intended for long-running applications.
Example:
Web API Database Proxy Frontend Backend
Example:
job "web" { type = "service" group "web" { count = 3 task "nginx" { driver = "docker" config { image = "nginx:1.28" } } } }
Service is the default job type.
31. Batch Jobs
Batch jobs run until completion.
Examples:
Data processing Report generation Database migration ETL Backup Machine learning training
Example:
job "backup" { type = "batch" group "backup" { task "backup" { driver = "docker" config { image = "backup:1.0" } } } }
32. System Jobs
System jobs are designed to run on all eligible clients.
Example:
Nomad Cluster │ ├── Client 1 → Monitoring Agent ├── Client 2 → Monitoring Agent ├── Client 3 → Monitoring Agent └── Client 4 → Monitoring Agent
System jobs are useful for:
- Monitoring agents
- Log collectors
- Security agents
- Node-level utilities
33. System Batch Jobs
sysbatch is useful when you want a batch task to execute across all eligible nodes.
For example:
Every node ↓ Collect diagnostics ↓ Finish
Unlike a persistent system job, a successful system-batch task isn't continually restarted on that node.
34. Periodic Jobs
Nomad supports periodic scheduling.
For example:
Every night ↓ Database backup
or:
Every Sunday ↓ Cleanup
A periodic job can use a schedule expressed using cron-like syntax.
35. Parameterized Jobs
Parameterized jobs allow runtime parameters to be supplied when dispatching a job.
This is useful for:
Data processing Customer-specific jobs Ad-hoc tasks Migration operations Batch processing
Conceptually:
Generic Job + Input Parameter ↓ Specific Execution
36. Job Lifecycle
A simplified lifecycle:
Jobspec │ ▼ Submit │ ▼ Evaluation │ ▼ Scheduling │ ▼ Allocation │ ▼ Task Driver │ ▼ Running │ ▼ Complete / Failed
37. Installing Nomad
Nomad is distributed as a binary and packages are available for supported operating systems.
A typical Linux installation involves:
Download Nomad ↓ Install binary ↓ Create configuration ↓ Create systemd service ↓ Start Nomad
HashiCorp provides official installation and deployment documentation for supported platforms.
For production, use the version and installation method appropriate to your operating system and HashiCorp's current support matrix.
38. Nomad Agent
Nomad runs an agent process.
A server:
nomad agent -server
A client:
nomad agent -client
In real deployments, the configuration is normally placed in files rather than relying on a long command line.
39. Example Server Configuration
Conceptually:
datacenter = "dc1" data_dir = "/opt/nomad/data" server { enabled = true bootstrap_expect = 3 } bind_addr = "0.0.0.0" advertise { http = "10.0.0.10" rpc = "10.0.0.10" serf = "10.0.0.10" }
For a real production cluster, networking, encryption, ACLs, TLS, retry joins, and server/client configuration should be designed according to the deployment topology.
40. Example Client Configuration
datacenter = "dc1" data_dir = "/opt/nomad/data" client { enabled = true } server { enabled = false } advertise { http = "10.0.0.20" rpc = "10.0.0.20" serf = "10.0.0.20" }
The client registers with the Nomad servers.
41. Nomad CLI
The primary CLI is:
nomad
Check the version:
nomad version
Cluster status:
nomad server members
Node list:
nomad node status
Jobs:
nomad job status
42. Running Your First Job
Create:
nginx.nomad
Contents:
job "nginx" { datacenters = ["dc1"] group "web" { count = 2 task "nginx" { driver = "docker" config { image = "nginx:1.28" } resources { cpu = 500 memory = 256 } } } }
Run:
nomad job run nginx.nomad
Check:
nomad job status nginx
43. Allocations
Check allocations:
nomad job allocations nginx
You'll see allocations assigned to clients.
For example:
Allocation ID Node abc123 client-01 def456 client-02
Inspect one:
nomad alloc status abc123
44. Logs
Nomad stores task logs within allocation directories on clients.
You can retrieve logs with:
nomad alloc logs <allocation-id>
For a specific task:
nomad alloc logs <allocation-id> nginx
Allocation directories also contain task logs, templates, artifacts, and other task-related data.
45. Executing Commands
For supported drivers, Nomad provides allocation execution capabilities.
For example:
nomad alloc exec -i -t <allocation-id> /bin/sh
The exact behavior depends on the task driver.
The Docker driver explicitly supports nomad alloc exec.
46. Nomad Networking
Nomad supports several networking approaches.
A simplified model:
Allocation │ ├── Network ├── Ports └── Service registration
With Docker:
Nomad ↓ Docker ↓ Container
Nomad can allocate ports dynamically and expose them to tasks through environment variables such as:
NOMAD_PORT_http
47. Dynamic Ports
Instead of hard-coding:
Port 8080
Nomad can allocate a port.
Example:
network { port "http" {} }
The resulting port can be exposed to the task.
Inside the workload:
NOMAD_PORT_http
This is particularly useful when multiple allocations run on the same client.
48. Service Discovery
Nomad includes service discovery capabilities.
However, for production workloads with more complex service-discovery requirements, HashiCorp recommends integrating Consul.
Architecture:
Application │ ▼ Nomad │ ▼ Consul │ ▼ Service Catalog
49. Nomad + Consul
This is a very important HashiCorp ecosystem combination.
HashiCorp Stack Terraform │ ▼ Infrastructure ──────────────┐ │ ▼ Nomad │ ┌────────────┴────────────┐ ▼ ▼ Consul Vault Discovery Secrets
Nomad handles workload scheduling.
Consul handles service discovery and related networking/service-mesh capabilities.
Vault handles secrets.
Terraform can provision the infrastructure.
50. Consul Service Registration
A Nomad service block can register a workload with Nomad's service discovery or Consul.
Example:
service { name = "web" provider = "consul" port = "http" check { type = "http" path = "/health" interval = "10s" timeout = "2s" } }
Nomad can register the service and health check with Consul.
51. Nomad + Consul Connect
Consul can also provide service-mesh capabilities.
Conceptually:
Service A │ Envoy │ ▼ Encrypted Service Network │ Envoy │ ▼ Service B
This can provide:
- mTLS
- Service identity
- Traffic control
- Service-to-service authorization
- Observability
Nomad integrates with Consul for service registration and sidecar-based service-mesh deployments.
52. Nomad + Vault
Secrets are another major part of the HashiCorp ecosystem.
Instead of:
PASSWORD=supersecret
inside a job file, use Vault.
Architecture:
Nomad │ ▼ Vault │ ▼ Secret │ ▼ Application
Nomad can obtain Vault tokens for tasks and handle token renewal.
53. Vault Example
Conceptually:
vault { policies = ["database-read"] }
The task can then access the required secret according to the configured Vault policy.
Nomad's Vault integration is designed to avoid requiring application developers to manually manage long-lived Vault tokens.
54. Workload Identity
Modern Nomad supports workload identity.
Each workload can receive an identity represented as a signed JWT.
Conceptually:
Task │ ▼ Nomad Workload Identity │ ▼ JWT │ ├── Consul └── Vault
This enables identity-based access rather than relying entirely on static tokens.
55. Nomad Variables
Nomad also provides its own variable functionality for configuration/secrets use cases.
A common enterprise architecture might therefore distinguish:
Nomad Variables │ └── Nomad-native configuration/secrets Vault │ └── Centralized secret management
For highly sensitive, centralized secret management, Vault remains the HashiCorp product specifically designed for secrets.
56. Storage
Nomad can use different storage mechanisms depending on workload requirements.
For example:
Local filesystem Host volumes CSI volumes Ephemeral storage
For stateful applications, CSI integration can provide external persistent storage.
Conceptually:
Nomad Task │ ▼ CSI Volume │ ▼ Storage Backend
57. CSI
The Container Storage Interface enables Nomad to integrate with storage systems through CSI plugins.
Architecture:
Nomad │ ▼ CSI Plugin │ ▼ Storage Platform
This is useful for:
- Databases
- Stateful applications
- Persistent queues
- Enterprise storage
58. Update Strategies
Nomad supports controlled deployment updates.
For example:
update { max_parallel = 1 min_healthy_time = "10s" healthy_deadline = "5m" progress_deadline = "10m" auto_revert = true }
This can help implement rolling deployments.
Conceptually:
Version 1 │ ├── Instance A ├── Instance B └── Instance C ↓ Version 2 │ ├── Instance A → updated ├── Instance B └── Instance C ↓ Version 2 │ ├── Instance A ├── Instance B → updated └── Instance C
59. Canary Deployments
Nomad supports deployment strategies involving canaries.
For example:
10 instances │ ├── 9 × version 1 └── 1 × version 2
Validate version 2.
Then:
Version 2 healthy ↓ Continue rollout
If the deployment fails:
Rollback
60. Health Checks
Services can define health checks.
For example:
check { type = "http" path = "/health" interval = "10s" timeout = "2s" }
Possible checks include:
HTTP TCP Script
When integrated with Consul, health information can participate in service discovery and traffic decisions.
61. Namespaces
Nomad supports namespaces for workload segmentation.
For example:
Nomad Cluster │ ├── production ├── staging ├── development └── data
Jobs, allocations, deployments, and evaluations are associated with namespaces.
Create:
nomad namespace apply production
Run a job:
nomad job run -namespace=production app.nomad
62. ACLs
Nomad supports ACLs for controlling access.
An ACL policy can define permissions for:
Namespaces Jobs Nodes Volumes Variables Quotas
Example conceptual policy:
namespace "production" { policy = "read" }
ACL policies are written in HCL.
63. Nomad Enterprise
Nomad Enterprise adds features intended for larger organizations.
HashiCorp currently documents capabilities including:
- Resource quotas
- Sentinel policies
- Audit logging
- Multi-region deployments
- Advanced server management through Autopilot
64. Resource Quotas
In a multi-team environment:
Cluster │ ├── Team A ├── Team B └── Team C
One team shouldn't necessarily consume the entire cluster.
Nomad Enterprise quotas can limit aggregate resource consumption by namespace.
For example:
Production Team CPU: 20,000 MHz Memory: 64 GB
When a namespace reaches its quota, new allocations can remain queued until resources become available or the quota changes.
65. Sentinel
Nomad Enterprise can use Sentinel policies for policy-as-code.
For example:
Policy: Production workloads ↓ Must use approved images
or:
Production deployment ↓ Allowed only under certain conditions
HashiCorp documents Sentinel as an enterprise policy mechanism layered on top of Nomad ACLs.
66. Node Pools
Node pools allow workloads to target logical groups of clients.
For example:
Nomad Cluster │ ├── general │ ├── node1 │ └── node2 │ ├── gpu │ ├── gpu1 │ └── gpu2 │ └── high-memory ├── mem1 └── mem2
Then workloads can specify where they should run.
This is useful for:
- GPU workloads
- Big Data
- AI
- High-memory applications
- Compliance isolation
67. GPU Scheduling
Nomad can schedule device resources.
For example:
Node │ ├── CPU ├── RAM └── GPU
A workload can request a device resource.
Conceptually:
resources { device "gpu" { count = 1 } }
The exact device configuration depends on the Nomad client/plugin environment.
68. Nomad for AI/ML
Nomad can be useful for:
Model Training Inference Batch Processing GPU Workloads Data Pipelines
Architecture:
Nomad │ ┌──────────┼──────────┐ ▼ ▼ ▼ Training Inference Batch │ │ │ GPU GPU CPU
This is particularly interesting when the organization doesn't need the broader Kubernetes ecosystem around its ML workloads.
69. Nomad for Big Data
Nomad can also schedule:
Spark Kafka Airflow ETL Data Processing Batch Analytics
For example:
Nomad │ ├── Kafka ├── Spark ├── Airflow ├── ETL workers └── Batch jobs
But stateful distributed systems require careful planning around persistent storage, replication, networking, failure recovery, and data locality.
Nomad is the scheduler; it doesn't automatically solve the distributed-system design of Kafka, databases, Hadoop, or Spark.
70. Nomad for Legacy Applications
One of Nomad's interesting characteristics is that workloads don't have to be containers.
For example:
Legacy Java Application ↓ Nomad ↓ Java Driver
or:
Legacy Binary ↓ Nomad ↓ Execution Driver
This allows gradual modernization.
Instead of:
Legacy ↓ Rewrite everything ↓ Kubernetes
you can use:
Legacy ↓ Nomad ↓ Gradual modernization
71. Nomad for Batch Processing
This is one of Nomad's strongest use cases.
Imagine:
Daily ETL │ ▼ Extract │ ▼ Transform │ ▼ Load │ ▼ Complete
Nomad can schedule the workloads and provide resource constraints and placement.
72. Nomad for Microservices
A microservice architecture could look like:
Load Balancer │ ▼ Frontend │ ┌────────────┼────────────┐ ▼ ▼ ▼ User API Order API Payment API │ │ │ └────────────┼────────────┘ ▼ Consul │ ▼ Vault
Nomad handles workload placement.
Consul handles service discovery.
Vault handles secrets.
73. Nomad + Terraform
A HashiCorp-heavy architecture often looks like:
Terraform │ ▼ Infrastructure │ ▼ Nomad │ ├── Applications │ └── Workloads
Terraform can provision:
- VMs
- Networks
- Load balancers
- Storage
- Nomad infrastructure
Nomad then schedules the applications.
This provides a clean separation:
Terraform = Infrastructure provisioning Nomad = Workload scheduling
74. Nomad + Consul + Vault + Terraform
The classic HashiCorp ecosystem architecture:
Terraform │ Infrastructure as Code │ ▼ ┌───────────┐ │ Nomad │ │ Scheduler │ └─────┬─────┘ │ ┌───────────┼───────────┐ ▼ ▼ Consul Vault Discovery Secrets │ │ ▼ ▼ Service Catalog Credentials
Each component has a distinct responsibility.
75. Nomad API
Nomad exposes an HTTP API.
This enables automation beyond the CLI.
Conceptually:
Automation │ ▼ Nomad HTTP API │ ├── Jobs ├── Nodes ├── Allocations ├── Evaluations └── Cluster State
This is useful for:
- CI/CD
- Custom tooling
- Monitoring
- Automation
- Platform engineering
76. Nomad UI
Nomad provides a web interface for viewing cluster state.
You can inspect:
Jobs Allocations Clients Evaluations Deployments Namespaces
This provides operational visibility without requiring every task to be performed from the CLI.
77. Deployment Architecture
A production Nomad deployment might look like:
Users │ ▼ Load Balancer │ ▼ Nomad / Proxy │ ┌─────────┴─────────┐ ▼ ▼ Service A Service B │ │ ▼ ▼ Client 1 Client 2 │ │ └─────────┬─────────┘ ▼ Consul │ ▼ Service Mesh Nomad Servers ┌─────┬─────┐ │ S1 │ S2 │ S3 └─────┴─────┘ │ Raft + Vault
HashiCorp's reference architecture uses three or five Nomad servers and a set of clients, commonly integrated with Consul for service discovery and health checking.
78. High Availability
A production Nomad control plane should not rely on one server.
Instead:
┌──────────────┐ │ Nomad Server │ └──────┬───────┘ │ ┌──────┴───────┐ ▼ ▼ Server 2 Server 3
Raft maintains consensus.
With three servers:
1 failure ↓ Cluster can maintain quorum
With five:
2 failures ↓ Cluster can maintain quorum
This is one reason HashiCorp recommends three or five servers.
79. Nomad Failure Handling
Suppose:
Client 1 │ └── Web allocation
Client 1 fails.
Nomad detects the failure and can create a new allocation on another eligible client.
Conceptually:
Client 1 X │ ▼ Evaluation │ ▼ Scheduler │ ▼ Client 2 │ ▼ New Allocation
This is the basic self-healing scheduling model.
80. Nomad Deployment Strategies
Common strategies include:
Rolling update Canary deployment Automatic rollback Blue/green-style architectures
The exact implementation depends on the jobspec and surrounding traffic-management architecture.
81. Blue-Green Deployment
You can architect:
Load Balancer │ ┌──────┴──────┐ ▼ ▼ Blue Green Version 1 Version 2
Initially:
100% → Blue
Then switch:
100% → Green
This is generally implemented using Nomad plus service discovery/load-balancing mechanisms.
82. Canary Deployment
A canary approach:
10 allocations │ ├── 9 × Version 1 └── 1 × Version 2
Monitor:
Errors Latency Health Business metrics
Then progressively increase Version 2.
83. Nomad Security Architecture
A secure deployment should consider:
Security │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ ACL TLS Workload Identity │ │ │ ▼ ▼ ▼ API Access Encryption Vault/Consul
Additional controls include:
- Network segmentation
- Server/client encryption
- ACLs
- Least privilege
- Secure task drivers
- Container security
- Host hardening
- Secret management
84. TLS
Production Nomad deployments should secure communication between:
CLI → Server Server → Server Client → Server
This protects:
- API traffic
- RPC
- Cluster communication
TLS configuration should be designed consistently across the cluster.
85. ACL Bootstrapping
Nomad ACLs should be enabled for production environments where multiple users or teams interact with the cluster.
A typical architecture:
User ↓ Authentication ↓ Nomad ACL ↓ Policy ↓ Namespace / Job
Policies can restrict what users can see and modify.
86. Nomad Variables vs Vault
Use the appropriate tool for the requirement.
Application configuration ↓ Nomad variables / configuration Sensitive centralized secrets ↓ Vault
Vault provides richer capabilities around:
- Dynamic credentials
- Secret rotation
- PKI
- Leasing
- Revocation
- Auditing
Nomad's Vault integration can automatically obtain and renew task tokens.
87. Monitoring Nomad
Monitor:
Nomad Servers Nomad Clients Jobs Allocations Evaluations CPU Memory Scheduling latency Failed allocations Deployment health
A monitoring stack could look like:
Nomad │ ▼ Metrics │ ▼ Prometheus │ ▼ Grafana
Logs can flow through:
Nomad / Application │ ▼ Log Collector │ ▼ Central Logging
88. Troubleshooting Workflow
When something goes wrong:
Step 1
Check cluster servers:
nomad server members
Step 2
Check clients:
nomad node status
Step 3
Check job:
nomad job status <job>
Step 4
Check allocations:
nomad job allocations <job>
Step 5
Inspect allocation:
nomad alloc status <alloc>
Step 6
Check logs:
nomad alloc logs <alloc>
Step 7
Inspect events and evaluations.
This gives you:
Cluster ↓ Node ↓ Job ↓ Allocation ↓ Task ↓ Logs
89. Common Nomad Problems
Allocation Pending
Possible causes:
Insufficient CPU Insufficient memory Constraint mismatch No eligible datacenter Missing driver Node unavailable Quota exhausted
Inspect:
nomad alloc status <allocation>
90. Missing Task Driver
Suppose the job uses:
driver = "docker"
but the client doesn't have the Docker driver available.
The allocation won't be placed there.
Check client capabilities:
nomad node status
Then inspect the node:
nomad node status <node-id>
91. Docker Problems
For Docker workloads:
docker ps
on the client.
Also check:
systemctl status docker
and Nomad client logs.
Nomad's Docker driver requires Docker to be installed and running on the client.
92. Networking Problems
Check:
Allocation ↓ Network namespace ↓ Port allocation ↓ Service registration ↓ Service discovery ↓ Load balancer
Verify the allocated port:
nomad alloc status <alloc>
If using Consul:
Nomad ↓ Consul ↓ Service ↓ Health Check
93. Constraint Problems
If a workload cannot be placed, examine its constraints.
Example:
constraint { attribute = "${attr.cpu.arch}" operator = "=" value = "arm64" }
If all available clients are:
amd64
the workload remains unscheduled.
94. Resource Problems
Suppose:
Job requests: CPU = 4000 MHz RAM = 8 GB
but every client has:
Available CPU = 2 cores
The scheduler can't place the allocation.
Check:
nomad node status
and resource information.
95. Nomad vs Kubernetes: Operational Philosophy
A useful high-level distinction:
Kubernetes
Very broad platform ↓ Large ecosystem ↓ Many controllers/operators ↓ Extensive extensibility
Nomad
Focused scheduler ↓ Simpler core architecture ↓ Multiple workload types ↓ HashiCorp ecosystem integration
Nomad's official documentation explicitly emphasizes its simplicity and ability to support heterogeneous workloads.
96. When Nomad Can Be Attractive
Nomad can be particularly relevant when an organization needs:
- A simpler orchestrator
- Container workloads
- Batch processing
- Legacy workloads
- Mixed workload types
- Strong HashiCorp integration
- Multi-datacenter scheduling
- Straightforward HCL configuration
- Consul-based service discovery
- Vault-based secrets management
97. When Kubernetes May Be More Appropriate
Kubernetes may be more appropriate when the organization requires a very broad ecosystem around:
- Kubernetes-native operators
- Extensive CRDs
- Large cloud-provider ecosystem
- Kubernetes-native platform engineering
- Broad third-party integrations
- Existing Kubernetes operational expertise
- Kubernetes-specific commercial platforms
This is an architectural consideration rather than a universal rule.
98. Nomad Architecture for a Production Environment
A practical enterprise architecture might look like:
Users │ ▼ Load Balancer │ ▼ Consul / LB │ ┌─────────────┴─────────────┐ ▼ ▼ Service A Service B │ │ ┌─────┴─────┐ ┌─────┴─────┐ ▼ ▼ ▼ ▼ Client 1 Client 2 Client 3 Client 4 │ │ │ │ └───────────┴───────────────┴───────────┘ │ Nomad Servers ┌────┬────┬────┐ │ S1 │ S2 │ S3 │ └────┴────┴────┘ │ Raft │ ┌──────────┴──────────┐ ▼ ▼ Consul Vault Service Discovery Secrets
99. CI/CD with Nomad
A modern pipeline:
Developer │ ▼ Git │ ▼ CI │ ├── Build ├── Test ├── Security Scan └── Publish Image │ ▼ Registry │ ▼ Nomad Job Spec │ ▼ nomad job run │ ▼ Scheduler │ ▼ Allocations
Nomad doesn't need to be the build system.
It is primarily responsible for running the resulting workload.
HashiCorp's application workflow similarly separates artifact creation from Nomad scheduling.
100. GitOps with Nomad
You can store jobspecs in Git:
Git │ ├── services/ │ ├── frontend.nomad │ ├── backend.nomad │ └── api.nomad │ └── batch/ ├── backup.nomad └── cleanup.nomad
Pipeline:
Git Commit ↓ Validation ↓ Security Checks ↓ Nomad Plan ↓ Approval ↓ Nomad Run
This gives you:
- Version control
- Audit trail
- Reproducibility
- Rollback
- Peer review
101. Nomad Job Example — Production-Oriented
A more complete example:
job "web" { datacenters = ["dc1"] type = "service" group "web" { count = 3 network { port "http" {} } update { max_parallel = 1 min_healthy_time = "10s" healthy_deadline = "5m" progress_deadline = "10m" auto_revert = true auto_promote = true canary = 1 } service { name = "web" provider = "consul" port = "http" check { type = "http" path = "/health" interval = "10s" timeout = "2s" } } task "web" { driver = "docker" config { image = "nginx:1.28" ports = ["http"] } resources { cpu = 500 memory = 256 } } } }
This demonstrates several important concepts together:
Job ├── Datacenter ├── Service scheduler ├── 3 replicas ├── Network port ├── Rolling update ├── Canary ├── Consul service registration ├── Health check ├── Docker └── Resource limits
102. Nomad Mental Model
The easiest way to remember Nomad is:
NOMAD Job │ ▼ Evaluation │ ▼ Scheduler │ ▼ Allocation │ ▼ Client │ ▼ Task │ ┌─────────┼─────────┐ ▼ ▼ ▼ Docker Java QEMU
Then add:
Consul ↓ Service Discovery Vault ↓ Secrets Terraform ↓ Infrastructure Nomad ↓ Workload Scheduling
103. Docker → Kubernetes → Nomad
A useful comparison of the three:
DOCKER ────── Build Package Run containers KUBERNETES ─────────── Pods Deployments Services Scaling Controllers Operators Cluster orchestration NOMAD ───── Jobs Tasks Allocations Scheduling Services Batch Legacy workloads Multi-workload orchestration
And Nomad's ecosystem:
Terraform │ Infrastructure │ ▼ Nomad │ Workloads │ ├──────────────┐ ▼ ▼ Consul Vault Discovery Secrets
104. Nomad Learning Path
If you're coming from Docker and Kubernetes, learn Nomad in this sequence:
1. Nomad Architecture ↓ 2. Servers & Clients ↓ 3. HCL ↓ 4. Jobs ↓ 5. Task Groups ↓ 6. Tasks ↓ 7. Allocations ↓ 8. Evaluations ↓ 9. Scheduling ↓ 10. Constraints & Affinity ↓ 11. Resource Management ↓ 12. Docker Driver ↓ 13. Networking ↓ 14. Service Discovery ↓ 15. Consul ↓ 16. Vault ↓ 17. ACLs ↓ 18. Workload Identity ↓ 19. CSI / Storage ↓ 20. Deployments & Canaries ↓ 21. Monitoring ↓ 22. HA / Raft ↓ 23. Multi-region ↓ 24. Enterprise Governance
105. Interview Questions You Should Know
If you're preparing for a DevOps/SRE/Platform Engineer role, understand these:
Architecture
What are Nomad servers and clients?
Servers schedule and manage cluster state; clients execute allocations.
Workloads
What is a Job?
A declarative description of desired workload state.
Allocation
What is an allocation?
A placement of a task group on a client.
Evaluation
What triggers an evaluation?
Changes in desired or external cluster state can trigger scheduling evaluations.
Task Driver
What is a task driver?
The component responsible for executing and isolating a task.
Scheduler
Difference between service and batch?
Service jobs are intended for long-running workloads; batch jobs are intended to run to completion.
System Job
What is a system job?
A workload intended to run on all eligible clients.
Consul
Why use Consul with Nomad?
For production-grade service discovery, health checking, and service-mesh capabilities.
Vault
Why integrate Vault?
For centralized secret management and task-specific secret access.
106. Production Checklist
Before deploying Nomad in production:
Cluster
- 3 or 5 Nomad servers
- Proper Raft quorum
- Separate server and client roles where appropriate
- TLS enabled
- ACLs enabled
- Backup strategy
- Monitoring configured
Clients
- Resource capacity planned
- Required task drivers installed
- Node metadata configured
- Networking configured
- Storage configured
- OS hardened
Workloads
- CPU requirements defined
- Memory requirements defined
- Constraints reviewed
- Health checks configured
- Restart policy configured
- Reschedule policy configured
- Update strategy configured
- Canary considered
Security
- ACLs
- TLS
- Vault integration
- Workload identity
- Least privilege
- Network segmentation
- Secure container images
Operations
- Metrics
- Logs
- Alerts
- Disaster recovery
- Upgrade strategy
- Capacity planning
107. Nomad's Biggest Strength
The core idea behind Nomad is simplicity without restricting the workload model to containers alone.
You can have:
Nomad │ ┌───────────────┼───────────────┐ ▼ ▼ ▼ Containers Legacy Batch │ Apps │ ▼ ▼ ▼ Docker Java ETL │ ▼ QEMU
The same scheduler manages these different workload types.
That's a fundamentally different emphasis from platforms built primarily around the Kubernetes API and container ecosystem.
108. Final Takeaway
HashiCorp Nomad is best understood as a distributed workload scheduler, not merely a Docker alternative.
Its fundamental architecture is:
Nomad │ ┌────────┴────────┐ ▼ ▼ Servers Clients │ │ Scheduling Execution │ │ ▼ ▼ Jobs Allocations │ │ └────────┬────────┘ ▼ Tasks │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ Docker Java QEMU
And when combined with the HashiCorp ecosystem:
Terraform │ ▼ Infrastructure │ ▼ Nomad │ ├───────────────┐ ▼ ▼ Consul Vault │ │ Service Secrets Discovery
The concepts worth mastering are:
Jobs → Task Groups → Tasks → Evaluations → Allocations → Clients → Scheduling → Constraints → Resources → Networking → Consul → Vault → ACLs → Workload Identity → Storage → Deployments → HA → Multi-region operations.
For someone coming from a Docker/Kubernetes/SRE/Big Data infrastructure background, Nomad is particularly interesting because it exposes the underlying scheduling concepts very clearly while also allowing workloads that don't fit neatly into a container-only model. HashiCorp's current documentation explicitly positions Nomad around containers, legacy workloads, services, and batch processing, with integrations into Consul and Vault for production environments.
Current lifecycle note
HashiCorp's current documentation states that Nomad 1.10.x is the last LTS release, with support through April 2027; the 1.11.x line is supported through October 2026, while Nomad 2.0.x has a longer support lifecycle. HashiCorp also changed its self-managed product versioning/support model beginning in April 2026. For a production installation, select the version based on the current support matrix rather than blindly installing latest.
No comments:
Post a Comment
Thank you for Commenting Will reply soon ......