podcast traffic time lapse (24 hrs in 30 mins)

all web servers log requests to a file. each request is a single line in the logfile that looks something like this

127.0.0.1 - - [04/Jun/2014:21:00:48 -0700] "GET /ARCHIVE_-_0000-00-00_-_Show_Name_With_DJ_Name_From_Location.mp3 HTTP/1.1" 200 140398275 "-" "iTunes/11.2.1 (Macintosh; OS X 10.9.3) AppleWebKit/537.75.14"

there is a log entry for every whole or partial content request. every time you visit a website, for every image you see, for every js, css, etc file that loads is an entry. on busy sites this log file can become huge. the log file i used for this demonstration has 22,577 lines. each line (see above) can tell you a few things:

the ip address of the person making the request
time stamp
request type and request
http response code
bytes transferred
user agent

and now, with a wonderful tool called logstalgia combined with ffmpeg and some command line-fu

cat /var/log/apache2/podcast.access.log | logstalgia -1920x1080 -g Archives,ARCHIVE,99 --paddle-mode pid --update-rate 1 --output-framerate 60 --output-ppm-stream - - | ffmpeg -f image2pipe -r 60 -c:v ppm -s 1920x1080 -pix_fmt yuv420p -i - -crf 1 -c:v h264 -pix_fmt yuv420p -f mp4 outfile.mp4

and if you want real time…

tail -F -q /var/log/apache2/podcast.access.log | logstalgia -1920x1080 -g Archives,ARCHIVE,99 --paddle-mode pid --update-rate 1 --output-framerate 60 --output-ppm-stream - - | ffmpeg -f image2pipe -r 60 -c:v ppm -s 1920x1080 -pix_fmt yuv420p -i - -crf 1 -c:v h264 -pix_fmt yuv420p -f flv rtmp://videoservice.com/live/secretkey

and.. the timescale is non-linear! (because i like to fiddle)

T: 2 | 32,736 | 3894.13 GB (3.8TB)

and heres one using logs from a more well established instance of apache.