Introduction
Newcomers to data science often assume that statistical analysis requires Python, R or specialized software. In fact, the command line is already a capable statistics workbench: standard Unix tools can process large datasets faster than loading them into memory-heavy applications, they script and automate easily, and they work on any Unix-like system with nothing to install. This tutorial covers essential statistical operations performed directly in the terminal with built-in tools only. A Unix-like environment is required (Linux, macOS, or Windows with WSL); typing the commands along with the article is the best way to absorb the concepts.
Setting up sample data
Analysis needs a dataset. Create a simple CSV representing daily website traffic by running the following command:
cat > traffic.csv << EOF
date,visitors,page_views,bounce_rate
2024-01-01,1250,4500,45.2
2024-01-02,1180,4200,47.1
2024-01-03,1520,5800,42.3
2024-01-04,1430,5200,43.8
2024-01-05,980,3400,51.2
2024-01-06,1100,3900,48.5
2024-01-07,1680,6100,40.1
2024-01-08,1550,5600,41.9
2024-01-09,1420,5100,44.2
2024-01-10,1290,4700,46.3
EOFThis creates a file called traffic.csv with a header line and ten rows of sample data.
Exploring the data
Counting rows
A natural first question is how many records a dataset contains. The wc (word count) command with the -l flag counts lines: wc -l traffic.csv outputs 11 traffic.csv — 11 lines total, minus 1 header, equals 10 data rows.
Viewing the data
Before calculating anything, verify the structure. The head command displays the first few lines of the file for a quick preview.
date,visitors,page_views,bounce_rate
2024-01-01,1250,4500,45.2
2024-01-02,1180,4200,47.1
2024-01-03,1520,5800,42.3
2024-01-04,1430,5200,43.8Extracting a single column
Working with specific CSV columns calls for the cut command with a delimiter and field number. The following extracts the visitors column:
cut -d',' -f2 traffic.csv | tail -n +2Field 2 (the visitors column) is extracted with cut, while tail -n +2 skips the header row.
Measures of central tendency
Finding the mean (average)
The mean is the sum of all values divided by their count. Extract the target column, then let awk accumulate:
cut -d',' -f2 traffic.csv | tail -n +2 | awk '{sum+=$1; count++} END {print "Mean:", sum/count}'The awk program accumulates the sum and count while processing each row, then divides in the END block.
Finding the median
The median is the middle value of the sorted dataset — or the average of the two middle values when the count is even. Sort first, then locate the middle:
cut -d',' -f2 traffic.csv | tail -n +2 | sort -n | awk '{arr(NR)=$1; count=NR} END {if(count%2==1) print "Median:", arr((count+1)/2); else print "Median:", (arr(count/2)+arr(count/2+1))/2}'sort -n sorts numerically; the script stores values in an array and picks the middle element (or averages the two middle elements).
Finding the mode
The mode is the most frequent value. Sort, count duplicates, and take the top result:
cut -d',' -f2 traffic.csv | tail -n +2 | sort -n | uniq -c | sort -rn | head -n 1 | awk '{print "Mode:", $2, "(appears", $1, "times)"}'uniq -c counts duplicates after sorting; a reverse sort by frequency puts the mode first.
Measures of spread
Maximum value
awk -F',' 'NR>1 {if($2>max) max=$2} END {print "Maximum:", max}' traffic.csvThis skips the header with NR>1, compares each value against the running maximum, and updates when a larger value appears.
Minimum value
Symmetrically, initialize the minimum from the first data row and update as smaller values appear:
awk -F',' 'NR==2 {min=$2} NR>2 {if($2Both minimum and maximum in one pass
awk -F',' 'NR==2 {min=$2; max=$2} NR>2 {if($2max) max=$2} END {print "Min:", min, "Max:", max}' traffic.csv The single-pass version initializes both variables from the first row and updates each independently — a pattern worth remembering for large files.
Population standard deviation
Standard deviation measures how far values spread from the mean. For an entire population:
awk -F',' 'NR>1 {sum+=$2; sumsq+=$2*$2; count++} END {mean=sum/count; print "Std Dev:", sqrt((sumsq/count)-(mean*mean))}' traffic.csvThe script collects the sum and the sum of squares, then applies \( \sqrt{\frac{\sum x^2}{N} – \mu^2} \).
Sample standard deviation
When the data is a sample rather than the full population, Bessel’s correction applies — dividing by \( n-1 \) for an unbiased estimate:
awk -F',' 'NR>1 {sum+=$2; sumsq+=$2*$2; count++} END {mean=sum/count; print "Sample Std Dev:", sqrt((sumsq-(sum*sum/count))/(count-1))}' traffic.csvVariance
Variance is simply the square of the standard deviation — the same computation without the square root:
awk -F',' 'NR>1 {sum+=$2; sumsq+=$2*$2; count++} END {mean=sum/count; var=(sumsq/count)-(mean*mean); print "Variance:", var}' traffic.csvPercentiles and quartiles
Quartiles
Quartiles split sorted data into four equal parts and are especially useful for understanding a distribution:
cut -d',' -f2 traffic.csv | tail -n +2 | sort -n | awk '
{arr(NR)=$1; count=NR}
END {
q1_pos = (count+1)/4
q2_pos = (count+1)/2
q3_pos = 3*(count+1)/4
print "Q1 (25th percentile):", arr(int(q1_pos))
print "Q2 (Median):", (count%2==1) ? arr(int(q2_pos)) : (arr(count/2)+arr(count/2+1))/2
print "Q3 (75th percentile):", arr(int(q3_pos))
}'The script stores sorted values in an array, computes quartile positions with the \((n+1)/4\) formula, and extracts the values at those positions.
Q1 (25th percentile): 1100
Q2 (Median): 1355
Q3 (75th percentile): 1520Any percentile
Any percentile follows by adjusting the position calculation; this flexible version uses linear interpolation:
PERCENTILE=90
cut -d',' -f2 traffic.csv | tail -n +2 | sort -n | awk -v p=$PERCENTILE '
{arr(NR)=$1; count=NR}
END {
pos = (count+1) * p/100
idx = int(pos)
frac = pos - idx
if(idx >= count) print p "th percentile:", arr(count)
else print p "th percentile:", arr(idx) + frac * (arr(idx+1) - arr(idx))
}'Position is computed as \( (n+1) \times (percentile/100) \), with linear interpolation between array indices for fractional positions.
Working with multiple columns
Statistics for several columns can be computed at once — here, the averages of visitors, page views and bounce rate together:
awk -F',' '
NR>1 {
v_sum += $2
pv_sum += $3
br_sum += $4
count++
}
END {
print "Average visitors:", v_sum/count
print "Average page views:", pv_sum/count
print "Average bounce rate:", br_sum/count
}' traffic.csvSeparate accumulators per column share one pass over the data:
Average visitors: 1340
Average page views: 4850
Average bounce rate: 45.06Calculating correlation
Correlation measures the relationship between two variables. The Pearson correlation coefficient ranges from -1 (perfect negative) to 1 (perfect positive):
awk -F', *' '
NR>1 {
x(NR-1) = $2
y(NR-1) = $3
sum_x += $2
sum_y += $3
count++
}
END {
if (count < 2) exit
mean_x = sum_x / count
mean_y = sum_y / count
for (i = 1; i <= count; i++) {
dx = x(i) - mean_x
dy = y(i) - mean_y
cov += dx * dy
var_x += dx * dx
var_y += dy * dy
}
sd_x = sqrt(var_x / count)
sd_y = sqrt(var_y / count)
correlation = (cov / count) / (sd_x * sd_y)
print "Correlation:", correlation
}' traffic.csvThe script computes Pearson correlation as covariance divided by the product of the standard deviations.
Conclusion
The command line is a legitimate statistical tool: it processes large volumes of data, computes non-trivial statistics and automates reporting without installing anything. These skills complement rather than replace Python and R — command-line tools shine for quick exploration and data validation on messy datasets, while specialized environments remain better for complex modeling and visualization. Two honest caveats: hand-rolled awk statistics are easy to get subtly wrong (floating-point behavior, empty fields, malformed rows), so results on real data deserve a spot-check against a known-good library; and quoting/escaping differs slightly across shells and awk variants, so scripts should be tested on the target system. Within those limits, these tools are available on virtually every system a data scientist will ever touch.
