Thanks for all the thoughtful replies and ideas. The last two days I've had my head buried in the computer, and all 5 of my monitors cluttered with chrome, visual studio, explorer, putty, winscp, etc windows...
My first thought was running a light webserver, then just having a browser load the page which would be generated via asp.net, then push whatever I needed to it via the code on the server. Having done quite a lot of server side web work, it seemed pretty natural. However....
Over the last two days, this is what I've done and it's working nicely with a test that just displays current time updating every second:
Installed Stretch Lite
Installed GPU drivers
installed x server
Jacked with Chromium config to enable GPU rendering
Installed .net core 2.1 (since I'm pretty proficient in .net technologies) - pulled in the selenium references via nuget
wrote a simple html page that has my base structure and loads appropriate js & css
wrote a .net core console app
config'd x to autorun my console app
console app uses selenium API to fire up chromium and load my html page
This seems very clean and very light weight. I could do this with a webserver and websockets using something like SignalR, but it seems like a lot of extra work vs what I had in my head and the way I accomplished this same setup on Win10IOT.
So with this setup, the client is the web browser, the "server" is my console app using Selenium as an API to then control the browser. I know Selenium was created for test automation, but I've found that if thought of as a browser API it's useful for many more applications.
With this configuration, I can re-use a TON of code and libraries I've already written - only now on Stretch instead of Win10 IOT. Much faster boots, much more flexibility, and probably, much more horsepower available since now I'm able to render the web page and any animations with the GPU.
There was some pain and suffering to be had along the way - probably mostly due to my limited and far between experience with Linux. My main hangup with Linux was how to fire up a "GUI" app (i.e chromium) using a console only OS. The answer to that was to run the lightweight Stretch lite, then install a bare bones x-server and use that to fire up a "kiosk" mode chromium window. Loving it! Thinking this will be the first of many projects of this type.