Lomont.org

(Also lomonster.com and clomont.com)

Chris Lomont's lines of code over time

Published

Over decades I’ve written a lot of personal code. I made a bar chart race showing how many lines I’ve written in different languages over time.

Here is a bar chart race for code I’ve written over the decades.

Method

I took every source file on my hard drive, did cleanup to remove duplicates and those not mine and so on, counted lines in each file and noted the creation date, and visualized the data in a bar chart race.

I wrote a script (below) in F# to analyze the data and put it in a format suitable for an online bar chart race generator at https://app.flourish.studio/@flourish/bar-chart-race.

Details

Here are the steps my program took to gather statistics

  1. I selected all filenames with specific extensions that were sub to certain directories.
  2. To try and count legitimate lines of code, I did several passes of cleanup. I removed expected duplicates via MD5 hashing, and I did several visual scans through all files and added patterns to exclude. This could be strengthened to count unique lines to help detect derivative files.
  3. Sorted by date.
  4. Created running tallies by file extension.
  5. Export as CSV in format suitable for the bar chart race generator.

Sources of error

As a kid I wrote a lot of assembly code on a TRS-80 Color Computer, but those files are offline (I still have them on disk and cassette, if I ever get around to reading them out). So this animation starts after that era.

I possibly missed other directories - in the process of doing this I found a few places I had tucked away old code.

I may have some eras in zip files.

I originally added Mathematica *.nb files, but they were too large, since they have code and generated data intermixed. I have written a significant amount of them, but I threw them out.

I did add *.bat files since I wrote a lot of them, but there are also many I didn’t counted.

I ignored Powershell scripts *.ps since I have too many postscript files around.

I suspect there is a lot of overcounting from derived code files.

This also doesn’t have most if any of my professional code, since I don’t have it to analyze.

Description of visualization

Around 1988 or so I got a PC, and I started writing code in Turbo Pascal. My senior project in undergrad in 1991 was a chess program, which I still have. I also wrote a lot of assembly code for fun and learning.

In 1992 I worked in Chicago, and I bought Turbo C++ to learn C and C++, so the code turns to them along with the assembly code.

I wrote games and did game consulting in the mid 90s, and then went to grad school. All my side work was still C/C++/asm, with a tiny bit of Python.

In the mid 90s through early 2000s, I was in grad school, and I wrote a lot of code for fun.

In the mid-late 2000s, I started doing my side code in C#, which catches up pretty quickly.

In the mid 2010s, I worked fulltime on my own projects, so my professional code count shows up as well.

Code

Here is the F# I used. To use it, scan the top and replace the file types, the exclusion lists, and whatever else needs cleaned for your needs. I’m posting it in a messy state, but it worked so far.

I do plan on cleaning it up and this post to get more accurate counts.

// Learn more about F# at http://fsharp.org

open System
open System.Text
open System.IO
open System.Collections.Generic
// open Opulos.Core.IO
open System.Diagnostics
open System.Security.Cryptography

(* Todo
- Exclusions by list
- Detect dups and dont count
- 

*)
    

(*
Things to gather
- list of all
    (filename, extension, year, month, day, day of week, linecount)
    or possibly (if not then recomputing fields many times)
    (filename,Date,linecount)

- all extensions and counts
- exclude things TODO
- all list of
   (filename, extension, year,month, day, day of week, linecount)
   -- for each month, count total lines and lines by filetype
   -- store as map[year*100+month,map[filetype,int]]

Nice graphs:
- code lines by month broken down into type and colored
- code total lines in pie by type
- code by day of week
- # code files of type lines per file over time
- # projects - each in own dir?
- make bar chart race
- total pic: 
     use SkiaSharp, Bar chart race in UL corner, total growth LL with progress, 
     Per month (or week) in UR, and about in LR
*)

// code extensions we care about
let sourceExtensions = [
        ".cxx";".c";".cpp";".h";".hpp";
        ".cs";".fs";".xaml"; 
        ".asm";".s";".py";".js";".go"; ".pas";
        ".rc";
        ".bat"; // too much noise
        ".m"; 
        // ".ma"; ".nb"; // far too many lines, generated
        "bas"
        ]

// sequence to lowercase
// replace '\' with '/'
let normalize (str:string list) = 
    [ for s in str do 
        let s1 = s.Replace("\\","/")
        yield s1.ToLower() 
    ]

// exclusions from above
let excludeSuffixes = ["g.i.cs"]
let excludeContains = normalize [
    "/boost/";
    "/boost_1_28_0/";
    "/boost_old/";
    "/buildprocesstemplates/";
    "/libjpeg/";
    "/libpng/";
    "/loki/";
    "/opengl/";
    "/zlib/";
    "/extfilesystem/";
    "/lemmings.net/";
    "/properties/";
    "/xsharp/";
    "/wolfsrc/src/";
    "/antigraingeometry/";
    "/keccakreferenceandoptimized/";
    "/golly-1.1-src/";
    "/poker-eval-129.0/";
    "/id3lib/";
    "/lua/";
    "/borlandc/";
    "/at2g3r38sourcecode/";
    "/fmt-master/";
    "/stockfish/";
    "/relacy/";
    "/stdmarkdown/";
    "/opengl line rendering/code/";
    "/pmars-0.9.2/";
    "/obj/";
    "/debug/";
    "/release/";
    "/properties/";
    "/prettycomments_integrationtests/";
    "/zlibnet/";
    // new
    "/zlib.net/";
    "/sourcesafedata/";
    "/winwolf3d/";
    "/fontmaker/bitmapfonts/";
    "/usbgadgetdriver/usb/";
    "/libf2c/";
    "/rfvdemo/";
    "/ez_lcd_wrapper/";
    "/liblz4/";
    "/google job interview/reservations_files/";
    "/lemc/arm9/";
    "/lemmingsds/lemmingsvc/";
    "/golly/";
    "/skein/nist/";
    "/externals/";
    ".acceptancetests/";
    "/adplug";
    "/dosbox";
    "/libxmp";
    "/relacy";
    "/gumstix/";
    "/spiffs/";
    "/meyers/";
    "/applewin/";
    "/tundra python";
    "/massivelyparallelvectorgraphics/";
    "/turbo55/oop_demo/";
    "/antlr";
    "/visualgdbcache/";
    "/packages/";
    "/include/config/";
    "/c/wolfsrc/";
    "/icsharpcode.texteditor/";
    "/amber";
    "/dr dobbs";
    "/myers/";
    "/cstuff/gems";
    "/cstuff/wolfsrc/";
    "/me/src/";
    "/sound/modplay/";
    "/sound/sbdox/";
    "/sound/voc/";
    "/gadgetdriverold/";
    "/e_dec96/graphics/src/";
    "/zork123/";
    "/tume/";
    "/fof/";
    "/png/";
    "/plytool/";
    "/shellcrypt/";
    "/adplay/";
    "/turboprg/";
    ]

// file paths to check
//let dirs =  [ "D:\Programming\Code2"]
let dirs = normalize [ 
    "D:\Programming\Code2"; 
    "D:\Programming\CSTUFF2"; 
    "C:\Users\Chris\OneDrive\Code"; 
    "C:\Users\Chris\Desktop"; 
    "D:\Hypnocube\Development\Code";
    "D:\CCLText"
    ]
//let dirs =  [ "D:\Programming\Code2\FlashAnalyzer" ; "C:\Users\Chris\OneDrive\Flash Code" ]

//let dirs = ["C:\Users\clomont\Documents\Code\DataScience";"C:\Users\clomont\Documents\Code\Math3D"]
//let dirs = ["C:\Users\clomont\Documents\Code\Qt"]
//let dirs = ["C:\Users\clomont\Desktop\CSTUFF2"]


// file sequence to extension
let extension (files:string list) = [ for f in files do yield Path.GetExtension f ]

// see if item has a suffix from a list
let hasSuffix (item:string) (suffixes: string list) = 
    List.exists (fun (suffix:string) -> item.EndsWith(suffix)) suffixes

let containsAny (item:string) (matches:string list) = 
    matches |> List.exists (fun m -> item.Contains(m))

// let only code through this filter that has no matched patterns
let codeOnly (files : string list) = 
    files 
        |> normalize 
        |> Seq.filter(fun fn -> List.contains (Path.GetExtension(fn)) sourceExtensions )
        |> Seq.filter(fun fn -> not (hasSuffix fn excludeSuffixes))
        |> Seq.filter(fun fn -> not (containsAny fn excludeContains))
        |> Seq.toList

// turn a sequence of directory names into files
let rec allFiles (dirs : string list) =
    if List.isEmpty dirs then List.empty else
        [ yield! dirs |> Seq.collect Directory.EnumerateFiles
          yield! dirs |> Seq.collect Directory.EnumerateDirectories |> Seq.toList |> allFiles ]


// DateTime GetLastFileModifiedFast(string dir)
// {
//     DateTime retval = DateTime.MinValue;
//     
//     FileData [] files = FastDirectoryEnumerator.GetFiles(dir);
//     for (int i=0; i<files.Length; i++)
//     {
//         if (files[i].LastWriteTime > retval)
//         {
//             retval = lastWriteTime;
//         }
//     }
//     
//     return retval;
// }

//let allFilesFast dirs = 
//    FastFileInfo.EnumerateFiles(Seq.head dirs, "*", SearchOption.AllDirectories)
//    if Seq.isEmpty dirs then Seq.empty else
//        seq { 
//            for d in dirs do
//                yield! FastFileInfo.EnumerateFiles(d, "*", SearchOption.AllDirectories)
//            }

    //FastFileInfo.EnumerateFiles(path, "*.*", SearchOption.AllDirectories)
    //FastDirectoryEnumerator.GetFiles


// get lowercase extensions and count them
// return sequence of string*int
let tallyLowercaseExtensions (files : string list) = files |> List.countBy (fun fn->Path.GetExtension fn) 

// length of file
let countLines filename = File.ReadAllLines(filename).Length

// print all items in sequence
let printSeq s = 
    let rec phelp s c =
        if Seq.isEmpty s then c
        else
            printfn "%A" (Seq.head s)
            phelp (Seq.tail s) (c+1)
    phelp s 0


(*********************************** various code sampling things **********************)

// list all files to output, return count
let listFiles files = 
    let count = printSeq files
    printfn "Count %A" count
    count

// tally items and output
let listTally s  = 
    let fastTally s : ((string*int) list) = 
        let d = new Dictionary<string,int>()
        s |> Seq.iter(fun t -> 
                                            if d.ContainsKey(t) then ignore()
                                            else d.Add(t,0)
                                            d.[t] <- d.[t] + 1
                                            )
        [ for (KeyValue(k,v)) in d do yield (k,v) ] |> List.sortBy (fun (k,v)-> (-v))

    // let t = s |> Seq.countBy (fun f->f) 
    let t = fastTally s
    let count = printSeq t
    printfn "Count %A" count
    count

// one record per file
type Record = 
    {
    filename : string
    extension : string
    date: DateTime
    hash : string

    //year : int
    //month : int
    //day : int
    //hour : int
    //weekday : DayOfWeek
    linecount : int
    }

type TimeStep = 
    | Daily
    | Weekly
    | Biweekly
    | Monthly
    | Bimonthly
    | Quarterly

let hashFile (md5:MD5) f = 
    let bytes = File.ReadAllBytes f
    (StringBuilder(), md5.ComputeHash(bytes)) ||> Array.fold (fun sb b -> sb.Append(b.ToString("x2"))) |> string

// convert files to seq of records
let toRecords s = 
    let md5 = MD5.Create()
    s |> List.map(fun f ->
        let date = File.GetLastWriteTime f
        let ext = Path.GetExtension f
        let hash = hashFile md5 f
        {filename = f; extension = ext; hash = hash; date = date; linecount = (countLines f)  }
        )

// allow date time iteration
type Span = Span of TimeSpan with
    static member (+) (d:DateTime, Span wrapper) = d + wrapper
    static member Zero = Span(new TimeSpan(0L))

// count lines by month, return count of all lines
// takes a sequence of all records
let computeLines recs timestep = 

//    // year and month encoded as 100*y+m
//    let encode (r:Record) = 
//        match timestep with
//        | Month -> 100*(100*r.year + r.month) + 15 // all on mid month
//        | Week  -> 100*(r.year + r.month) + (r.day/7)*7
//        | Day   -> 100*(r.year + r.month) + r.day
//        
//    let decode key = 
//        let y = key/10000
//        let m = (key%10000)/100
//        let d = key%100
//        (y,m,d)
//
//    printfn "Do min max"

    // get min and max
    //let rec minmax r (min,max) = 
    //    if Seq.isEmpty r then (min,max)
    //    else
    //        let v = encode (Seq.head r)
    //        minmax (Seq.tail r) (Math.Min(min,v), Math.Max(max,v))
    //let (min,max) = minmax recs (Int32.MaxValue, Int32.MinValue)

    // sorted file types that occur
    let types = recs |> List.map (fun r->r.extension) |> Set.ofList |> Set.toList |> List.sort
    let typeCount = List.length types
    printfn "Types %A" types

    // map from code type to name index
    let typemap = types |> List.mapi (fun i x -> x,i) |> Map.ofList

    // snap date to nearby value
    let snapDate (date:DateTime) = 
        let y,m,d = date.Year, date.Month, date.Day
        match timestep with 
        | Daily ->  new DateTime(y,m,d)
        | Weekly -> 
            // 1,8,15,22 
            let d2 = ((d-1)/7) * 7 + 1
            let d3 = if d2 > 22 then 22 else d2
            new DateTime(y,m,d3)
        | Biweekly ->
            // 1,15
            let d2 = ((d-1)/14) * 14 + 1
            let d3 = if d2 > 15 then 15 else d2
            new DateTime(y,m,d3)
        | Monthly -> 
            new DateTime(y,m,1)
        | Bimonthly ->
            // m = 1,3,5,7,11
            let m2 = ((m-1)/2)*2+1
            new DateTime(y,m2,1)
        | Quarterly ->
            // m = 1,4,7,10
            let m2 = ((m-1)/3)*3+1
            new DateTime(y,m2,1)

    // find first date
    let dates = recs |> Seq.map(fun r->r.date)
    let startDate = dates |> Seq.min
    let endDate   = dates |> Seq.max

    printfn "Date range %A to %A" startDate endDate

    printfn "Do array"

    let dayData = new Dictionary<DateTime,(int array)>()

    // populate
    let days = 
        seq {
            let mutable d = snapDate startDate
            let e = snapDate endDate
            while d < e do    
                yield d
                d <- d + TimeSpan.FromDays(1.0)
            yield e
        } |> Seq.toList

    printfn "Column count %A" days.Length

    for d in days do
        let snap = snapDate d
        if not (dayData.ContainsKey(snap)) then
            dayData.Add(snap, Array.create (types.Length) 0)

    printfn "Num entries %A" dayData.Keys.Count


    // add record to data
    let addTo (r:Record) = 
        let date = snapDate r.date
        let m = dayData.[date]
        let index = typemap.[r.extension]
        m.[index] <- m.[index] + r.linecount

    printfn "Do fill"
    for item in recs do
        addTo item

    // sorted data
    printfn "Do sort"
    let array = dayData |> Seq.sortBy(fun (KeyValue(k,v))->k) |> Seq.map(fun (KeyValue(k,v))->(k,v))

    printfn "Do output"

    // now output 
    use file = File.CreateText("Data.csv")
    
    // header and dates
    file.Write("Name,")
    array |> Seq.iter (fun (date,_) -> 
        let y,m,d = date.Year,date.Month,date.Day
        file.Write(String.Format("{0}/{1}/{2},",y,m,d))
        )
    file.WriteLine()

    for i in [0..types.Length-1] do
        let t = types.[i]
        file.Write(t+",")
        let mutable sum = 0
        array |> Seq.iter (fun (date,data) -> 
            let v = data.[i]
            sum <- sum + v
            file.Write(String.Format("{0},",sum))
            )
        file.WriteLine()

//    array |> Array.iteri (fun i (total,arr) -> 
//        let y,m,d = decode (i+min)
//
//        printf "%A-%A-%A %A: " y m d total
//        arr |> Array.iteri (fun j count ->
//            printf "%A=%A" types.[j] count
//            )
//        printfn ","
//        )

    file.Close()
    printfn "File written"

let timer = new Stopwatch();
let timeMsg msg = 
    printf msg
    printfn ": %A" timer.Elapsed

let time f = 
    let proc = Process.GetCurrentProcess()
    let cpu_time_stamp = proc.TotalProcessorTime
    let timer = new Stopwatch()
    timer.Start()
    try
        f()
    finally
        let cpu_time = (proc.TotalProcessorTime-cpu_time_stamp).TotalMilliseconds
        printfn "CPU time = %dms" (int64 cpu_time)
        printfn "Absolute time = %dms" timer.ElapsedMilliseconds

let rec loop n f x =
    if n > 0 then
        f x |> ignore
        loop (n-1) f x

let runit() = 

    timer.Start()

    printfn "Do files"
    let files = (allFiles dirs) |> Seq.toList
    timeMsg "files done"

    //printfn "Do files 2"
    //let f2 = allFilesFast dirs |> Seq.toList
    //timeMsg "files 2 done"

    //printfn "Sizes %A %A" files.Length f2.Length

    printfn "Do derived"
    let lowercaseFiles = files |> normalize   
    let lowercaseCodeFiles = lowercaseFiles |> codeOnly
    printfn "%A code files " lowercaseCodeFiles.Length

    let removed = lowercaseCodeFiles |> List.filter (fun fn -> containsAny fn excludeContains)
    for r in removed do 
        printfn "%A" r
    
    //ignore <| listFiles files 
    //ignore <| listFiles lowercaseFiles
    //ignore <| listFiles lowercaseCodeFiles

    // extension count of all 
    //ignore <| listTally (lowercaseFiles |> extension)

    // extension count of code
    //ignore <| listTally (lowercaseCodeFiles |> extension)


    printfn "Do recs"

    // (filename, extension, year, month, day, day of week, linecount)
    let mutable recs = lowercaseCodeFiles |> toRecords |> List.filter(fun r -> r.date.Year > 1989) |>  List.sortBy(fun r -> r.date)

    printfn "Rec from %A to %A " recs.[0].date recs.[recs.Length-1].date

    //for i in [0..10] do
    //    printfn "%A" recs.[i]

    let linesOfCode = recs |> List.sumBy(fun r-> r.linecount)
    printfn "Lines of code %A" linesOfCode

    // remove dups, dump again
    printfn "Removing dups"
    let mutable s = Set.empty
    let crecs = seq {
        for r in recs do
            if not (s.Contains r.hash) then
                yield r
                s <- s.Add(r.hash)
        } 
    
    recs <- crecs |> Seq.toList
    printfn "Lines of code %A" (recs |> List.sumBy(fun r->r.linecount))

    // dump files
    use f = File.CreateText("Files.txt")
    for c in recs do
        f.WriteLine(c.filename)
    f.Close()
    printfn "Files dumped"



    // ignore <| printSeq dat

    printfn "Do lines"

    // lines by month or day
    computeLines recs Bimonthly
    

    0 // return an integer exit code

let testit() = 
    ignore()
    let mutable c = 0
    let files = 
        seq {
        for f in File.ReadAllLines("exclude1Files.txt") do
            yield f
        } |> Seq.filter (fun f -> not (String.IsNullOrEmpty f))
        |> Seq.toList
        |> codeOnly
    printfn "Test files %A" files.Length

    for c in files do
        printfn "%A" c

//    let test = 
//        [
//        //"d:/programming/code2/buildprocesstemplates/defaulttemplate.11.1.xaml";
//        "d:/programming/cstuff2/boost_1_28_0/boost/any.hpp"
//        ] |> codeOnly 
//    ignore <| listFiles test


[<EntryPoint>]
let main argv =
    //testit()
    runit()
    0 // return an integer exit code

Categories: Software

Tags: Software F# C# C++ Languages

Comments

Comment is disabled to avoid unwanted discussions from 'localhost:1313' on your Disqus account...