Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Friday, May 20, 2022

Fiddler + EXE + localhost server

(a friend was asking about this one, so here's a repost / rewrite of a previous blog from 2014-12-03, back when Visual Studio 2012 was state of the art)

Capturing fiddler traffic between a local EXE and a web server running on localhost turned out to be a little more difficult than I originally imagined. I tried to set my server to localhost or 127.0.0.1 while my local EXE had a proxy through fiddler (localhost:8888), but some/most/many/mine web requests appear to skip proxies when going to a local address.

To get this done:

    1. Run fiddler (Download Fiddler Web Debugging Tool for Free by Telerik), make sure "File / Capture Traffic" is checked.
    2. Make sure fiddler's filters show traffic to localhost.fiddler

      I also like to Hide if URL contains /arterySignalR, which is a special endpoint that IIS express/Visual Studio will use on localhost for debugging. It can generate a lot of noise in fiddler.

      See below for a screen cap of my fiddler filter settings.
    3. Start your web server on localhost
    4. Reconfigure your EXE's target server to point to http://localhost.fiddler:>PORT< 
      This will be EXE specific.
      (note, >PORT< depends on the server you're running and what port it binds to)
    5. Start your EXE
    6. Make sure your EXE's proxy is pointing to localhost:8888, so all traffic goes through fiddler.

    Here's my current fiddler filter settings





    Thursday, March 25, 2021

    power(shell), corruption & lies: Building command line arguments

    I hate typing long command lines to command line utilities. Invariably I will get one obscure path wrong and spend an hour with a copy / pasted command line in notepad to split out the components and figure out what I screwed up this time.

    A lot of those long typing jobs have to do with feeding a bunch of file or directory names to an .exe

        foo_the_bar.exe --blort first\dir second\dir third\dir hey\heres\a\filename.txt

    If I only had a tool that I could use to generate that list of directories and filenames.

    powershell and Invoke-Expression (iex to it's friends) to the frickin' rescue.

    1. Write a powershell expression using Get-ChildItem (gci), Where-Object (?) and Select-Object (select) to build your list of things. Use -join ' ' to space separate the list of things

      PS c:\users\kpk> (gci c:\my\dir -recurse | ?{ $_.FullName -match 'some.*criteria$' } | select -expand FullName) -join ' '

    2. Use string interpolation to embed that expression with the name of the .exe you want to run. In my case, I was using cloc to (duh) Count Lines Of Code.

      PS c:\users\kpk> "cloc $((gci c:\my\dir -recurse | ?{ $_.FullName -match 'some.*criteria$' } | select -expand FullName) -join ' ')"

    3. Run that expression a few times to make sure you're happy with it. Add some command line arguments as needed.
    4. Use iex (Invoke-Expression) to run that beautiful mess.

    Let's unpack the expression a bit

        PS c:\users\kpk> (gci c:\my\dir -recurse | ?{ $_.FullName -match 'some.*criteria$' } | select -expand FullName) -join ' '

    1. gci c:\my\dir -recurse |

      gci (Get-ChildItem) recurses through a directory structure. It's more complicated than that, but things often are.

    2. ?{ $_.FullName -match 'some.*criteria$' } |

      ? (Where-Object) will test each produced file & directory for a criteria. In this case, a regular expression. Only the worthy shall pass.

    3. select -expand FullName

      select (Select-Object) extracts the FullName property from the produced collection of objects and expands it to a list

    4. -join ' '

      I need that list turned into a single flattened string with space separation, because cloc wants space separation. Other command line utilities want other kind of formatting (prefixed by a magic argument, comma separated, etc.). -join's your friend here.
    Once all of this is looking good, and you've prefixed it with the name of the command you want to run, iex takes care of the rest.

    Never let a human do a job that a robot can do better. Ok, maybe not 'Never', but mostly. Sometimes.... Whatever.

    Saturday, June 20, 2020

    power(shell), corruption & lies: Tee-Object and the clipboard

    Really useful powershell for me this morning. I wanted to run some commands, look at the output, and capture the output to the clipboard so I could paste it elsewhere.

    Rather than run my command and then use the mouse in the comand window to copy the output to my paste buffer, like some kind of animal, I wanted to have a full powershell solution.

    FOOBAR | Tee-Object -Variable c; Set-Clipboard $c

    How it works:
    1. FOOBAR is the name of the command (or series of piped commands) that I want to run that write to the console.
    2. Tee-Object -Variable c takes the output from FOOBAR and passes it forward (in this case to the console output) and to the variable $c.  

      Tee-Object is kind of like a plumbing tee where data is sent in two directions. It's normally used to show output on the console and to redirect to a file. 

      Note that you say -Variable c not -Variable $c. 
    3. Set-Clipboard $c sets the contents of the paste buffer with the value in $c.

    power(shell), corruption & lies: Directory Info Aggregation

    To get a recursive summary of files, folders and sizes from a root directory:

    $len=0; Get-ChildItem -Recurse | 
        %{ $_.GetType().Name; $len+=$_.Length } | 
        Group-Object -NoElement | ft -auto; "Total size= $($len / 1Gb) GB"

    Output looks like this
    Count Name
    ----- ----
    18386 DirectoryInfo
    57784 FileInfo
    Total size= 25.7907013082877 GB
     
    How it works:

    1. $len=0 sets the variable len to 0. No duh here.
    2. Get-ChildItem -Recurse recurses through a directory tree from the current directory. You can use the -Path option for a different root directory.
    3. %{} is the alias of ForEach-Object and is a shorthand way of looping over results that are piped from the previous command
    4. $_.GetType().Name is emitting the type of the directory entry (DirectoryInfo or FileInfo)
    5. $len+=$_.Length is summing up directory entry sizes
    6. Group-Object -NoElement is counting directory entry types from step 4 and printing them out.
    7. ft -auto automatically formats the output so that large numbers are not truncated to ellipsis (...).
    8. "Total size= $($len / 1Gb) GB" prints out the total accumulated size from the $len variable in Gigabytes.

    Saturday, May 2, 2020

    Little Metrics: Providing Observability in our Projects


    Everybody reading this should be familiar with the concept of taking your temperature. When you don't feel well, you (or Mom or Dad) pop a thermometer into your mouth, wait a few seconds, and then take a look. If you're above 98.6° F (37° C, 310.15 K, ...), you may have an infection. Note: I'm not a Doctor (but I play one on this blog).

    Recording your temperature is a "little metric" around making a quick judgement around your health, and it's effortless to collect. Well, possibly not effortless for parents of toddlers (I remember those dark days), but you get the idea.

    Like taking your body's temperature, we should be able to find and record some effortless "little metrics" around our software projects to make judgements about their health. This is especially true around any project that is:

    • taking more than some tiny number of iterations. 
    • split up into many tasks.
    • worked on by multiple developers.

    We demand observability (one of the core -ilities) for the execution of our software, using logging libraries and monitoring services. We should also demand observability on the health of our development efforts, and present that observability in a way that everybody, from individual contributors to C-suite executives, can easily understand.

    I'm deliberately wishy-washy about when to apply "little metrics". You'll know where you need it.

    My "little metrics rules":
    1. The little metrics collected should be an easy concept for everybody to understand.

      A common example is number of work items (tickets) total for a project, number that are closed, and number that are active. Collecting these metrics requires everybody to be good work item citizens and follow the rules on work sizing and not reopening closed work.

      Other good examples of little metrics are cost, performance (how fast to do the thing(s)), and scalability (how many things are supported simultaneously before the system goes pear shaped).
    2. The little metrics should be small in number.

      If you start collecting and reporting 10 numbers for a project, my eyes are going to start glazing over and thinking about potato chips before you're done. 3-5 numbers sound good.
    3. Collection of the little metrics should be effortless.

      Clicking two urls is good. Clicking one is double plus good.

      The urls should show results in less than 5s. Any more, and you've lost my attention.
    4. Collection of the little metrics should use your source of truth.

      Think about where your source of truth is for tracking the work to be done. Use that. Any use of a denormalized copy of that data (like in an external spreadsheet) will be outdated as soon as you hit File > Save.
    5. Collection of the little metrics should be done in a ceremonial fashion.

      While it's great to collect these metrics via robot, incorporate their collection in your meetings, so that everybody can witness them getting collected. Playing music with their collection is optional.

      Collect and show their graphing over time (the next step) to everybody in your ceremonies simultaneously, and take a short amount of time to speak to what they mean.
    6. The little metrics should be recorded and graphed over time.

      Work tracking systems are great at showing the current state of things. What you want to see is how project health is progressing over time. Every stakeholder, from individual contributor to executive, should be able to easily see these graphs, and understand what they mean.

      If you can record these metrics and graph them over time in your work tracking system, great. If you need to copy them into a spreadsheet, that's fine too. As you're capturing the health state at a point in time, you don't risk outdated data listed in rule 4 above, because changes to the future state don't affect what is happening right now.

      In the example of total, closed, and active, it's very interesting to see a graph like this.

      but you can slice it or dice it however you want: percentages, lines towards 0 (project completion), break out active work as a separate graph, etc. Whatever is most useful to the stakeholders to gauge the "temperature" of a project.

      If total work is growing over time, then the project isn't properly scoped yet (not a pejorative - growth is bound to happen, as the unknown unknowns become known). If closed work stays flat over time, then you can see if you've got enough resources on the project. There's a ton more to be written here, and way out of scope for this blog post.
    7. The little metrics should NOT BE WEAPONIZED.

      I'm 100% serious here. This observability on the state of our projects should be collected and presented in a judgement free manner. This is using data to determine project health. Projects that need love and attention should get love and attention to get them back on track.
    I see a lot of metrics in my day to day that can be hard to digest what they really mean - think "can't see the forest for the trees". I'm a firm believer that having a system like these 7 rules in place around project observability will help tremendously in building a shared understanding project health, which further fosters a collaborative development culture.

    Saturday, March 14, 2020

    On Prioritization: Your Team Structure Articulates Your Goals

    Of the three legged stool that defines the controllable outputs of development of a software system
    • Features
    • Performance
    • Stability
    it's self evident that everything comes at a cost; you have to employ prioritization to determine what gets the most focus.

    More time spent on features equals less time spent on performance and stability, and vice-versa.

    Imagine having a development team that has 1,000 hours of capacity. How do you split the time? 333 1/3 hours per output? Are you feature driven: 600 hours spent on features, 350 on performance and 50 on stability?

    Everything is a choice.

    Note, I'm specifically avoiding what might be called Engineering as a controlled output. Mostly, customers don't directly notice Engineering - they can't tell if my interfaces are clean, if I'm employing a single responsibility principle, or my code is a spaghetti mess of variables named x1, x2, x3, ... x999 looking like it was produced by a poorly written minifier.

    Certainly Engineering is indirectly noticed, as poorly Engineered software requires more resources to add Features, is less likely to be Stable, etc. That frequently manifests over the software system's lifetime.

    So, we have our software system, and we've made our choices about who is working on what, and for how much time.

    Easy peasy, right?

    Maybe. Maybe not. There are other inputs to this problem. 

    Regardless of the time allocation, how have you segmented your development teams? 

    If your teams are aligned around a Feature, they're going to have a Feature mindset. They will communicate with other teammates about Feature implmentation. Regardless of direction around resource allocation, Features will be artifically weighted.

    I'm being delibarately provocotive here. Many of us, myself included, are used to teams with Feature segmentation. For developing a software system of any complexity, it just makes sense - separate into feature based bounded contexts, and throw a team on each context.

    I challenge myself to not default to that muscle memory way of team creation, and to be considerate to organizational structure when laying out goals. 

    Thursday, April 4, 2019

    Tidying up a project - safe refactoring rules

    With 6 kids (!) that I love with every fiber of my being, I'm used to rooms being a bit of a mess when I enter. Therefore, I try to lead by example and:
    "Make the space better for me having been there"
    Which means, depending on where I am, I wipe down the sink in the bathroom, put a dish or two in the dishwasher, etc. Of course, any child within earshot, when appropriate, gets a "everything on pause, come help me with this..."

    And, like all rules, there is nuance. When we're out the door for a thing in 2 minutes, I don't do this. Everybody needs to stay on task. I'm not going to distract from the shoe tying & coat buttoning to get "the dang toys off of Bruce (our table, don't ask) and into a bin!"

    Likewise, when I open up a C# programming project - .sln in Visual Studio - to do a thing, I will accompany that thing with some refactorings that feel generally safe.

    Nuance: I calculate the days to when this code will hit my production environment (the bake time). If we're below my comfort zone, I do not do any of these steps. I don't have a hard cutoff for comfort zone, and much of it depends on the project. When we're talking hours to prod and not days or weeks, my comfort zone is a little on edge, so I say skip these. From one of my favorite writings
    "vi. Break any of these rules sooner than say anything outright barbarous." - George Orwell
    Ok, enough with the boring. Onto my rules
    NOTE: I'm nowhere near as good of a C# developer as I am C++. These rules are going to change for me over time. Also, the point of this is to not proscribe a set of rules for the reader, but to get you thinking in terms of what are YOUR safe refactoring rules.

    My Safe Refactoring Rules


    Definitions

    • TEST projects are C# projects (.csproj) that contain the unit tests, integration tests, smoke tests, whatever tests. The build artifacts produced do NOT run on production. The build artifacts produced by a TEST project are used to exercise the artifacts that DO run on production. 
    • PRODUCTION projects are the C# projects that produce build artifacts that DO run on production.

    Rules

    1. Validate that all TEST projects are on the latest department accepted version of the .NET Framework
    2. Make sure that all projects are configured so that StyleCop style violations are errors, not warnings.
    3. Update all NuGet packages in TEST projects, except for the department documented "don't update" packages.
    4. For each .cs file that is modified, make sure that unnecessary using statements are removed.
      • Visual Studio has a function under Edit / IntelliSense / Organize Usings / Remove Unnecessary Usings. I have muscle memory around alt e i o r.
    5. For each .cs file that is modified, sort your using statements.
      • Visual Studio has a function under Edit / IntelliSense / Organize Usings / Sort Usings. I have muscle memory around alt e i o s.
    6. If your PRODUCTION projects are developing NuGet packages, avoid updating dependent NuGet packages unless absolutely necessary.
      • Final choice of the proper version consumed should be up to the application, not the NuGet package.
      • NuGet package dependencies here are an expression of "what interface do I expect."

    Deets

    I want to go into a bit of detail about each of the choices.

    1. Update .NET Framework

    Developers should be familiar with how to update the target .NET Framework and not allow code to languish on an old framework version, like some kind of an animal. Tools like the Target Framework Migrator can help tremendously. 

    2. StyleCop

    While I can be annoyed at StyleCop, it's an "eat your broccoli" to me, as I think projects are generally better with StyleCop than without. 

    I like StyleCop.Error.MSBuild to turn StyleCop warnings into errors, although it needs a hand modification to your packages.config file to turn it into a development dependency

    <package id="StyleCop.MSBuild" version="5.0.0" targetFramework="net461" developmentDependency="true" />
    lest a build choice on a package be injected into a downstream consumer that is not quite ready for it.

    Sometimes this choice can cause me to have to fix more warnings-turned-errors than I'm comfortable with, so it may turn into scheduled work for the future.

    3. Update all NuGet packages in TEST projects

    Developers should be "watching the skies" for updates to NuGet packages, as stale packages can cause bugs (been there, done that). 

    By doing this in a safe manner, where the only thing you may break are your tests (which would reject your build because you gate, right, RIGHT?) gets you in a good mindset.

    If your department does not have a documented list of "don't update these" packages, then demand one. Better yet, create & champion one.

    4. & 5. Fix yer usings

    Using statements are happier when they are organized this way; I've spoken to them.

    6. Be careful with package to package dependencies

    While this feels like a weird one, as it's not a "do this" rule, but an "avoid this", I thought it was important to call out the difference between what a package dependency represents when developing a NuGet package and what a package dependency represents when developing an application.

    While developing a NuGet package, you're selecting an expected interface by the version choices of your package dependencies.

    If you find bugs with particular releases of your dependencies, then by all means, express that in your dependency ("Anybody that uses package FOOBAR version 3.1.5 or below is a FOOL and will get the BAZ race condition bug - Minimum expected version is 3.1.6").

    Other than bugs, you should let the application be the final arbiter of what version of package they are dependent upon. When wearing the NuGet package developer hat, you can't determine the full context in which your package will be used, and the application may be combining your package with other packages written by other developers in interesting ways that require more control over dependencies.

    Being as flexible as possible over dependent versions helps here.

    Conclusion

    Again, these are my rules, not yours. My rules will change over time as I get more seasoned in C#. The purpose of this is to get you thinking about what your "tidying up" rules might be so your projects stay fresh and moving forward.

    Wednesday, April 3, 2019

    FILE_FLAG_DELETE_ON_CLOSE, temporary files and tidying up your room

    I'm sure most of you have a favorite CreateFile flag (I kid, I kid). Mine is FILE_FLAG_DELETE_ON_CLOSE and I've been using it in my software development career for many years, to great effect, IMHO.

    I'd like to write a few articles to describe what it does, why it's awesome, and to present some frustrations with its usage as a challenge to library writers.

    OVERVIEW

    At the lowest API level, when opening up a file for reading or writing in a Windows process, your call will wind up in the CreateFile function. Actually, CreateFile does a ton more than just reading and writing - check out that MSDN link above - but for this article, let's just focus on regular disk files.

    NOTE: The above link takes you to CreateFileW, which is different from CreateFileA. The W and A are Microsoft's way of distinguishing Wide (2 byte) characters from nArrow (1 byte) characters. Macros hide this from you in your C++ code, so your calls are all done using CreateFile without the W or A suffix.

    Resolving to CreateFile is done whether the call originates from a higher level API like C++'s std::fstream, .Net's File.Open, whatever.

    CreateFile has a number of flags that tweak behavior that can be bitwise OR'ed together in the 6th paramter called dwFlagsAndAttributes.

    When you use the flag FILE_FLAG_DELETE_ON_CLOSE, this tells the Windows operating system to delete the file when all handles to it are closed.

    In case you don't see it yet, this is super effective for temporary files that are created during process execution, where data has to live outside of the process and on disk for whatever reason (too big for process memory, needs to be accessed by an external process, library requires filesystem access to do it's thang, ...).

    When you're all done with the temporary file, you need to tidy up your room, by getting rid of the junk piles on the floor.  To delete those files after usage (Mom says "clean your room... NOW!") instead of executing DeleteFile, you can just CloseHandle and let the operating system take care of it.

    Now, here's the awesomeness: Think about what happens if your process is torn down by an exception, taskkill, whatever. You've never reached the line of code that calls DeleteFile, so you've left a mess.

    <IMPORTANT>
    If you use FILE_FLAG_DELETE_ON_CLOSE, the operating system will delete the file for you as it tears down the process.
    </IMPORTANT>

    As the operating system tears down a process for whatever reason, it iterates through all open handles and, effectively, calls CloseHandle one by one. Side effects of the CloseHandle, like honoring FILE_FLAG_DELETE_ON_CLOSE, are done.

    Combining this with RAII semantics in code can make the resource of the temporary file automatically go away as soon as it is no longer needed, where the destructor of your temporary file management class closes the handle before the object dies, with the added safeguard that if something bad happens during your process execution, your file will still be removed.

    NOTE: FILE_FLAG_DELETE_ON_CLOSE can't protect you against all events: a power outage or a well placed hammer strike on the CPU will not go through the operating system's process teardown routine. YMMV.

    Next up, FILE_FLAG_DELETE_ON_CLOSE all the things! Where using this flag doesn't work well. Soon.