Art Project: Berlin Blues

This is my second digital art project, with some images, words and music… About ten years ago I commuted to Berlin to manage a team there. I really fell in love with Berlin and I learned a lot there and I miss not being able to visit it. I’m getting better at the tools for doing the audio. I especially appreciate the auto-ducking feature that smoothly fades out the music to allow the spoken word track to come through.

Enjoy…

This piece was inspired by my time working in Berlin ten years ago. Just based on how much I think about it and respond emotionally to the pictures, it had a big impact on me. I wanted to mash up the electronica vibe of Berlin with the blues to try to express the feeling of being an alien there but also weirdly feeling at home.

Art Project: I remember every day

I’ve been working on combining images, words and some improvised music into a digital presentation… The whole process is a WIP but I thought I’d just share what I come up with as I go. I’m still figuring out the tools, the workflow, quality, etc…

This piece was inspired by a theme that I think about a lot, memory. I was looking through images I captured while I was commuting by train. The words came to me while I was looking at them. The music is a set of changes that I heard and then modified and a melody that I played while I was watching the sequence of images.

UPDATE: I thought it needed a bass line so I added one in, I like this version better:

Management: Things I won't miss...

I’ve been writing a series of posts on management based on my experiences. If you’re starting with this one, you may want to have a look at: https://www.jlgoodwin.com/words-pictures/2020/9/17/management-the-job-of-a-manager

Introduction

I’ve been retired now for ten months after a thirty-nine year career in the software industry. I’ve had pretty much every role from software engineer to vice president of engineering. Overall, it was an enormously rewarding career and I’m honestly not bitter or angry about any of it now that it is over. I’ve certainly had periods of bitterness and anger over the years, and there are definitely things that I think are systematically wrong with the industry that I found myself struggling against (especially as a manager). It can be a great career for anyone as long as you know what you’re getting into. Here’s the list…

Being between the team and upper management

I will not miss: 

• Waking up every day thinking about some executive’s expectations of me and my team, the problems I should solve, the deadlines I should pay attention to, the people I need to wrangle, my performance measured against an invisible changeable ruler.

• Having to get people to do things that they don’t want to do for reasons I don’t entirely agree with (or in some cases completely understand). Negotiating the “merely goofy” vs. “the actually stupid and evil” to figure out how much political capital to spend on any given act of inquiry, redirection, or resistance.

• Trying to deduce, from a cloud of barely differentiated requests for my time and my team's time, which actions will actually be valuable to the business if we do them. This is where actual management happens, and actual decisions, and actual risk… Some folks completely avoid this, but they tend to be crap managers.

• Facing my email with dread, waiting for the next capricious turn that will invalidate all of the planning and understanding and work me and my team had done for weeks or months before. Our decisions, supported by a lot of work and thought, undone by the sudden “pivot” supported by a rambling semi-coherent e-mail. 

Human frailty

I will not miss:

• Having to listen to other folks’ emotional outpourings, worries, freak-outs, paranoia, sadness, grief, anger, you name it. Having to absorb and understand that emotion and not respond in kind, and then talk them around to some productive direction that keeps them getting paid and us all doing what we get paid for. Hard to do and still have some personal integrity…

• Changing my entire mental focus an average of ten or twelve times a day… and not being able to focus afterward on anything that I want to pay attention to.

• Having people who pay me a lot of money for my experience and knowledge ignore or contradict my advice. The worst version of this is when they come back and tell me “You know, you were right…” after the debacle/or resisting me tooth and nail.

• Having to fire people either for something stupid that they did or failed to do or because the company did something stupid or failed to do something. Fortunately, if you are breathing in and out and have a good story about being a software engineer you can get a job in this industry. Believe me, I’ve reviewed a LOT of resumes and interviewed a LOT of folks, so, generally I don’t worry about people I’ve fired, It’s more the emotional impact of the moment that I won’t miss.

Male dominance behavior

I will not miss having to step on the neck of male bullies in the tech leadership because it's the only way they’ll “respect” me. This is one of the worst things about the industry, and it stems from overwhelming and systematic male dominance, and by this I mean, only hiring men into leadership positions, only listening to men, only valuing particular male styles of communication, having to dominate other people to show how “good” you are in a male recognized sort of way. Even in the very best of companies I was frequently confronted with men telling me about how women engineers were “too junior,” “didn’t communicate directly enough,” “didn’t communicate clearly,” or “weren’t organized.” What they actually meant was “They’re not aggressively suppressing all other ideas in favor of their reductionist linear version of what to do.”

All, and I do mean all, of the monumental software project failures that I’ve witnessed in my career were led by aggressive, opinionated, and actually under-qualified, men. They all went off the rails because these folks couldn’t allow consideration of alternatives or even whether the project had useful results. Their egos wouldn’t allow it; their entitlement wouldn’t allow it.

I never fit into male culture very well even as a child and male culture knew it. I was frequently the focus of a lot of aggression and bullying. I figured out how to survive by using my intelligence (and low animal cunning), and I never developed any sympathy for my abusers. In my personal life, I avoid folks who behave like that and now that I’m an older, bigger, more affluent man, they tend to avoid me too. 

I did my best to be an ally for folks who weren’t aggressive men during my career. I used my power and influence to counteract the default narrative and get more women into higher positions. It was a fucking struggle, even when overtly these same companies had “diversity” goals. I have to say that announcing “diversity goals” is mostly talk, because I did the awkward thing and actually hired the women and others. Most of my peers didn’t. I would provide them with excellent candidates who I vouched for; they would invent a reason not to hire them. They would be happy about my team improving the stats but they weren’t interested in helping.

Meaningless technical debate

I won’t miss having to explain why I don’t want to include all of GraphQL for one tiny feature because someone thinks it’s “cool.”

I also won’t miss Java, JavaScript, the C++ Standard Template Library, browsers as the worst runtime environments of all time, vim, emacs, the Windows API from any century, distributed object systems of any kind, micro-services and other overused flavor-of-the-year one-size-fits-all architecture patterns that substitute for actual architecture design, Big Company drive-by-do-nothing-take-no-responsibility software architects, the new language to end all languages (Golang, Rust, etc.)… I could go on.

The fact of the matter is that good engineering is good engineering and you can do good engineering in Assembly language if you have the skills and the discipline. And good engineering means that what you are engineering is appropriate to the problem you’re solving, the economics of the business you’re in, and is reasonable for the talent you have access to. You can even build good things out of imperfect pieces (and you often do for economic, social, and political reasons).

All failed software projects fail due to one or more of: poor goals, poor leadership, a bad business model, a lack of customer value, poor analysis of requirements, outrageous scope vs. capacity, and human frailty. Of course, the least valued skill in the software industry is management, and it is the skill that is required for success. 

Business travel

I love to travel for pleasure; business travel is the opposite of that. I won’t miss being forced to fly to places I’d never go on a bet, to talk to people I don’t know, about stuff none of us are interested in, followed by tiresome forced socializing. All of this happens in a dreary business hotel or some kind of function hall or closed down public venue or worst of all a fucking golf resort. Business trips often are scheduled so that you are “ON” for twenty-four hours a day and constantly in large groups of people you only slightly know. This is the very vision of hell for an introvert.

Even in the best case, where I got to go someplace I really wanted to and had some time to myself, the logistics of traveling efficiently so you don’t waste your own time and also have time to do the work ruin a lot of it. When I managed folks in Berlin, I visited about every five weeks. To assure I still had time for myself and my family, I would fly out late Sunday night, arrive on Monday morning, work all day, go out in the evening until my normal bed time. That first “day” was thirty-six hours long and by three o’clock in the afternoon local time I was almost hallucinatory. I made some great friends there and in the end it was worth it, but the travel was a huge drag.

I really hope for everyone’s (and the planet’s) sake that, when the pandemic is over, people realize that most business travel is just posturing and a meaningless, inefficient display of wealth and entitlement. It’d be great if folks would recognize that business travel isn’t a perk for an introvert; if you’d like to give us a perk, we’d prefer some free time on our own.

Finally

I sincerely hope that the other allies out there will keep working to reform the industry. It was a great career and I loved a lot of it. I’m glad to be done with these particular things.

Robot Project: Treat dispenser installed on robot

Huge progress on the robot and treat dispenser! They are connected and working together! Thanks to a very helpful suggestion from my wife I was able to implement a simple fix for the clogging problem. I cut some thin flexible plastic from the top of a takeout sushi container and cut one end into fingers. I attached it so that it reaches about one “treat height” above the belt. The fingers brush back extra treats that would get carried forward as a stack due to their stickiness but they don’t allow them to get jammed by the fingers because they are so weak and flexible. You can just see the fingers in this picture:

Detail of anti-clogging solution…

The fingers also make the kicker servo and armature more effective. I added a zip tie around the kicker servo to make it more stable and to transfer more of the energy to the treats. This works good and having run multiple full hopper tests I can confirm that it successfully clears all the treats without jamming.

The next thing I needed to do was to mount the dispenser on the robot and as part of that mount the servo controller and the servo batteries somewhere. I got some nice 1.5cm x 4mm standoffs and drilled and countersunk some holes in the treat dispenser base plate and mounted the controller behind the left side servo. I was able to cut a couple of slots and mount the battery pack using a velcro strap next to the controller.

Controller board mounted using standoffs and the battery pack for the servos mounted using velcro strap.

Two new tools made a huge difference in this process, I got a set of small metric wrenches and a set of coarse and fine files. The wrenches allow the tightening of these tiny fasteners so much better and more easily and the files allowed me to drill out the slots and then remove the rest of the material by filing to create nice clean slots without cracking/burning/destroying the acrylic.

File set, not an expensive one but it works great, the small files on the right are perfect for making slots or fitted holes in acrylic. Just use the brush regularly to clear the file when working with plastic. Also, go slow to not melt the plastic with friction.

Tiny wrenches apparently usually used for working on carburetors. The metric ones are wonderful for tightening the tiny nuts on the robot without mangling them.

I was able to use some of the existing holes in the upper deck of the GoPiGo robot to finally mount the treat dispenser on top of it just behind the camera. Handily it also centers most of the weight of the treat dispenser over the drive axles and it doesn’t seem to make the robot more prone to tipping. I was worried because this particular robot only has a single roller caster in the back.

Using the existing holes in the upper deck of the GoPiGo robot to mount the treat dispenser.

I ran a test with everything integrated this morning and the robot was able to move, turn, use it’s sensors and dispense treats with no problems. Here is the video of the run:

I’m really happy with the progress so far, the next step is to make the software part of this project more usable. After messing around with trying to make the robot autonomous I have decided that it just isn’t up to it in terms of sensors, steering and maneuverability. So, I’m going to implement a dashboard and keyboard controls so I can directly control the robot and treat dispenser remotely via an ssh tunnel.

Robot Project: Treat dispenser acrylic version powered test

I got the servos installed after some back and forth. During powered testing to find the right position for the servos I managed to have the servo lever arm push on one of the axles. This popped the entire feeder body off of the base breaking the CA glue free. I had suspected that it would have a hard time holding on that long narrow seam on the bottom with torque and a big lever arm. So you’ll see later that I fabricated a couple of aluminum mounting brackets and screwed the dispenser to the base. This is much stronger and I realized it may be useful to disassemble this to clean it at some point.

I figured out the range for each of the servos to power it’s part of the driving sequence. I wrote a little python program to just run through four cycles. I tested it with real treats. For the most part it works fine and the ramp inside performs as expected, it can clear all the treats now. However, the narrowed area where the ramp is causes the treats to bind up there. They are a little moist and stick together enough to get suspended in that gap. I’m going to have to put another small servo with a kicker somewhere in that area to twitch the food to keep it moving. But, overall it has been very sucessful.

Here is the video of the test run:

Next up is solving the clogging problem then mounting it on the robot somehow.

Robot Project: Acrylic treat dispenser

Over the holidays, after evaluating alternatives for cutting the acrylic and aluminum, I got a bench top band saw.

It has allowed me to make all the parts for the feeder quickly and reasonably straight. I even figured out how to make the treat ramp inside to keep them from getting stuck. I used CA glue to bond everything. That was a little learning curve. My new clamping surface plus some masking tape to avoid gluing the parts to the bench worked great. So, here is the assembled unit:

I'm very happy with how it came out. Next is installing the servos and getting it running under power. The placement is going to be tricky.

Robot Project: Treat dispenser cardboard mockup

I’ve been working on a mock up of the servo driven treat dispenser for my robot. I wanted to build a mock up to look for any design problems before I commit to cutting aluminum and acrylic sheet. Here is a quick design walk-around video:

I also did a test with real treats that went well, here are the videos of that:

So far, so good, I’m a little worried about the size and weight of the treat dispenser but I think my next step is to try to get to a powered test of the mechanism.

Jazz: What would I recommend...

I made this list about 9 years ago and I just came across it again. I realized that I should put it here so it’ll be more accessible than in my FB feed. I don’t mean it as any judgement on things I’ve omitted, this is just a starting point to find the path through the huge world of Jazz that you belong on.

Relatively frequently I get into a conversation where I mention that I am a Jazz fan, and that I play Jazz piano, and someone asks me: "So tell me what to listen to in Jazz...", at which point my brain locks up and I stammer, forget everyone's name... so I made a list and I'm sharing it here FWIW:

(Artist * means important -- Suggested recordings -- not in any order, not really grouped at all, just my opinion from my listening so far )

*Ahmad Jamal -- The Legendary Okeh & Epic Recordings

*Art Blakey & The Jazz Messengers, Art Blakey Quintet --

*Art Tatum -- Piano Grand Master 1-3, The Tatum Group Masterpieces

Anita O'Day -- Anita O'Day Live in Tokyo

*Bill Evans -- Portrait in Jazz, Sunday at the Villiage Vanguard

Bix Beiderbecke -- Bixology

Bobby Timmons -- This Here is Bobby Timmons

*Bud Powell -- Paris Sessions

*Chick Corea -- Piano Standards

Chucho Valdes -- Fantasia Cubana Variations on Classical Themes

*Clifford Brown & Max Roach -- At Basin Street

*Dave Brubeck Quartet -- Time Out, Time Further Out

*Freddie Hubbard -- Red Clay

Gonzalo Rubalcaba -- Inner Voyage

*Herbie Hancock -- Corea Hancock (Live), Maiden Voyage, Speak Like a Child

Hiromi -- Spiral

*Horace Silver -- Song for My Father

*Jelly Roll Morton -- Greatest Hits

*John Coltrane -- Afro Blue Impressions

*Keith Jarrett -- The Koln Concert, Standards Vol.1, Standards Vol.2

Louis Armstrong -- Hot Fives and Sevens

Mary Lou Williams -- Live At The Cookery (Live), Marian McPartland's Piano Jazz with Mary Lou Williams esp "What's Your Story, Morning Glory"

Les McCann & Eddie Harris -- Swiss Movement

*McCoy Tyner -- Soliloquy, McCoy Tyner and the Latin All-Stars

Mose Allison -- I Don't Worry About A Thing

Nina Simone -- Feeling Good - The Very Best of Nina Simone

*Oscar Peterson -- Night Train, Oscar Peterson Et Joe Pass A La Salle Pleyel

Ramsey Lewis -- Ramsey Lewis's Finest Hour

*Sonny Rollins -- A Night At The Village Vanguard

*Stan Getz -- Captain Marvel, Sweet Rain

*Thelonious Monk -- Thelonius Monk Quartet with John Coltrane - At Carnegie Hall

Tony Bennett -- Tony Bennett and Bill Evans (two albums)

Vince Guaraldi Trio -- Jazz Impressions of Black Orpheus

*Wayne Shorter -- Adam's Apple

Robot Project: Audio In/Out, Cat Treat Feeder, Refining the Patrol code

Whenever we’re away for a long time we worry about our cats. There are people who come in to feed them, play with them, and change the cat box, but still we worry. Over the years I’ve developed more and more technological solutions to allow us to check in on them remotely. Now, before we go I set up multiple wifi cameras around the house, each of which is in a likely cat activity space. I can reposition the cameras remotely and they capture based on motion and save up a nice movie of all the activity broken down by hour. They have infrared cameras and so we get twenty four hour coverage. This works pretty well and reduce worry a lot.

There was only one gap in this system, sometimes the cats decide to be completely outside of the view of all of the cameras for many hours. Being an engineer I felt compelled to try and build a solution to fill this gap. I decided that I would build a robot rover that could do the following:

1. Autonomously roam one or more rooms of the house

2. Take pictures during it’s patrol and upload them

3. Record sounds

4. Play our voices calling for the cats and offering treats

5. Dispense treats

6. Allow me to take over and control it’s functions manually over the internet

The theory would be that while a robot would likely make the cats wary, something that sounded like us and dispensed tasty treats would make them get over that and come close to be seen on camera somewhat on demand.

Now that I have the robot assembled and I’ve figured out how to program it, I’ve been working on the patrol code. It is still a work in progress but I’ve managed to get ten probes into a room and not get stuck, I have some ideas on how to get to the next level. Along the way I’ve discovered some interesting edge conditions in the python api code, for example in some locations where they convert from degrees to a number of intervals from the wheel sensor they use integer math. Not in general a problem except when the division comes back as zero in which case the onboard controller takes that as meaning “turn on the motors until you’re told to stop” instead of “stop after some number of wheel sensor signals" So, every so often everything would be going well and then the robot would need to turn a small amount and it would just start spinning in place until it hit the code to go forward which at that point was in a random direction. There was a similar issue with the code to go forward a certain number of centimeters resulting in running right into whatever wall was in front of it.

I’ve got all of the safeguards in my code now to prevent those conditions now, and it has been interesting debugging both in the code and physical world. One of the mechanical things that I ended up having to fix was that I had originally mounted the servo on the front and the ultrasound sensor on the lower deck of the robot ( they didn’t say you shouldn’t and it was out of the way of the camera .) Turns out, that you don’t get full range of motion of the sensor on the servo because the circuit board hits the upper deck. More importantly for something that rolls around the floor it was easy for the low mounted sensor to miss things that would interfere with the top half of the robot. So, I moved it up to the upper deck. I just pivot it out of the frame when I use the camera.

I also learned that you shouldn’t operate the motors and take a picture at the same time, the ribbon cable ( not shielded ) for the camera runs right past the power circuit for the motors. There is a lot of RFI when they’re running and so you get some pretty funky images as a result.

Robot showing amplified speaker, microphone, and new placement of the ultrasonic sensor

I got a very cheap USB microphone dongle to use for audio input… bad idea… the gain on the device was terrible, it worked but the audio was barely audible over the noise. The noise was because all of it’s circuits were yes, you guessed it, right near the motor controller board. I ended up using a nice USB microphone via a USB cable clipped to the side of the robot, that worked like a champ and had enough gain to pick up a cat vocalizing.

I also got a tiny rechargeable amplified speaker to plug into the Raspberry Pi’s headphone jack and that worked fine. I got a 90 degree adapter to keep the wire away from the wheels and mounted it to the back with some double stick tape. Using the pygame module it is easy to play any audio file so we should be able to record ourselves calling to the cats and use that.

Design for cat treat feeder…

The final item is the treat dispenser, this is an entire project in itself. I have a design that I’m working on and I’ve created a prototype of the dispenser mechanism. I have been using components from the Lego Technic collection of parts. The great thing is that they have all kinds of shafts, bearings, gears, wheels, beams, etc… that are all compatible with each other. I’m not qualified to machine anything or even do a competent job designing something to be machined. So, these are a godsend and have allowed me to build the core mechanism of the treat dispenser and get it working. The power for the treat dispenser will come from two additional servos one on each side which will push a peg on a drive wheel, they are set 180 degrees apart. (see video for demonstration in the prototype) Fortunately the folks who make the robot provide a kit with an add on servo controller board that can control multiple servos and it comes with a couple of big servos that should work. It also has it’s own batteries and so it won’t be dependent on the robot’s power.

You can see the up and down action of the servo being translated to rotation to drive the belt.

I’m going to make the case for the treat dispenser out of 1/8” acrylic sheet and I’m going to have to cut these parts a lot more cleanly that the last one because I intend to glue them together. My Dremel has a straight cutting bit and a guide collar and I think that would work with the panel clamped firmly to a sacrificial piece of wood and pressing the Dremel against a guide fence. I’m going to be experimenting soon.

That’s where I’m at for the moment.

Robot Project: Adding pi camera and making a mounting bracket

I’ve started hacking away on a prototype program to have the robot drive around the room and investigate the objects it finds with the ultrasound sensor. Part of that was to take a picture, so I needed to get a camera. Raspberry Pi makes a nice add on camera which just comes with a ribbon cable and the camera card. You can just let the ribbon cable anchor the camera to the upper deck of the GoPiGo robot, but it seemed to be too low so that you’d always take pictures of the ultrasound sensor and also not well anchored so when the robot drove around it would flop back and forth. So I needed to fabricate a bracket.

I decided to make it in two parts, a lower right angle made out of .032” aluminum and an upper mounting plate made of the same 1/8” acrylic sheet that the rest of the body parts are made of. I made the cuts in both the acrylic and the aluminum using my Dremel tool and a cutting disk. It worked pretty well just needing to take my time and not heat the acrylic up too much. The only downside is that there isn’t a way to use a guide ( or shall I say I haven’t figured that out yet ) so the cut edges are a bit erratic. The resulting bracket is charmingly imperfect, but completely functional. I secured it to the upper deck with 8x3mm screws, nuts and a compression washer, the mounting plate is attached to the aluminum angle in a similar manner. I almost had an issue because unlike everything else on the GoPiGo the camera board has 2mm holes. Fortunately I had a couple of 8x2mm screws and nuts and I was able to use them to attach the board to the mounting plate. Because of the inaccuracies of my fabrication I had resigned myself to crooked pictures, but due to a happy accident, my inaccuracies canceled out and the camera ends up quite level.

Now back to debugging my patrol code and adding in the image captures.

Robot Project: Assembly and programming

Back in July of 2016 I guess I thought I’d have some time on my hands to play around with robots… that didn’t happen… but I did buy a GoPiGo kit from Dexter Industries along with some accessories. It has been in my office drawer for four years until this week when I got it out and put it together. It is a nice kit and targets kids 10 years and older and is really quick and easy to assemble. I would say that the youtube videos are the documents of record on how to assemble and get started with the GoPiGo. The robot is a simple rover powered by a Raspberry Pi and a Debian Linux distribution called Raspbian for Robots. It also has servos, cameras, and ultrasonic sensors that you can add on. There is a simple python api for programming the robot and interacting with the servos and sensors.

The version that I have is the GoPiGo 2 and the current one is GoPiGo 3, they have an archive of all of the software and documentation for old versions and the YouTube videos for the old ones are also still there. I got it assembled with only one mishap where I cracked a small Plexiglas part by over-tightening a fastener, I was able to mend it and continue. The software setup was equally easy, just plugged in an Ethernet cable, pointed a browser at it and they have a vnc based desktop where you can configure WiFi. Once it is on the wifi network you unplug the cable and you can just log on to the robot where ever it is via WiFi.

I created a github project for my programs and using git I can just pull them onto the robot. That way I can code on my desktop and just move things over via git. You can power the robot for programming purposes using a USB power adapter so that you don’t deplete the batteries while you’re debugging. I’ve successfully shipped over a couple of test programs and tested moving the robot and also manipulating the servo and reading the ultrasonic sensor. Now I’ve just got to think about what my first substantial robot project is going to be.

Digital archaeology...

I was going through a drawer in my file cabinet and came across a bunch of 3.5 inch floppies. These are now approximately 28 years old and approaching their expected use life of around 30 years. I thought “Why don’t I get a USB floppy drive and see if I can back up any documents and such that are still on these disks?” So, I did.

I was going to just attach the USB floppy drive to my Ubuntu machine and do it all from there, but for some reason Ubuntu really didn’t like the floppy drive and it was making bad noises and not working at all… So, I dug out the emergency Acer Windows 10 micro laptop that I keep for just these “I need an actual hardware device running Windows to do this thing…” sorta moments. Windows 10 was very happy with the floppy drive and I started to inspect the disks. Before a single file showed up I was transported back 30 years, the sound of the disk slotting into the drive, the shingggg noise as the metal shield pulled back from the media, the whirring and clacking of the disk spinning and the head seeking all sounds that I haven’t heard on a regular basis in decades.

Many of the disks are backups of Dana’s archaeology work, papers, databases, and her dissertation from grad school and beyond. So, digital archaeology of actual archaeology… I immediately ran into some interesting file format issues. At the time in the 1990’s I was working for Lotus Development Corporation (best known for the popular and best spreadsheet program 1-2-3) and so Dana was using Lotus software on DOS to do her work. The first word processor was Lotus Manuscript ( gone into the depths of time, only a Wikipedia entry to mark it’s passing ) and then Lotus AmiPro followed by Lotus WordPro ( similarly gone ) when she moved to Windows later. There are no import filters to read those file formats because Lotus lost the office suite wars with Microsoft and IBM had no interest in that market post acquisition.

Thank goodness for retro-gamers, they have built great emulators for both DOS and early Windows so that they can run vintage games. Because games tend to max out all kinds of capabilities of these environments, the emulators are more than adequate to run original office software if you can get disk images of it. I happen to have original Lotus SmartSuite 97 CD-ROMs (I worked on 1-2-3 for Windows 97 so I got a box of product when we shipped it, back when software came in boxes). I installed it under PlayOnLinux (a wrapper around the WINE windows API emulator for Linux) and very shortly had the applications running. They are VERY FAST on my 16 core 3.60 GHz desktop, the fastest computer of the time was around 50 MHz, and they were written to be fast on those, so even in an emulator which must have a lot of overhead they scream.

Because Lotus was always a very customer oriented company, we actually shipped a high fidelity file import filter for Lotus Manuscript 2.0 in Lotus Wordpro 97 and it works a treat to read Dana’s oldest documents.

Next problem was a database that Dana used for cataloging all of her artifacts and creating the tables and appendices for her dissertation. It was called Q&A from Symantec and we chose it at the time because it had a really simple form builder that let us model the same recording sheets she was using in the field while excavating. We were on the cutting edge at the time… So again, you’ve never heard of Q&A and that’s because it was crushed by many other more popular databases including one from Microsoft. Fortunately we still had the disks… I just copied them all into a folder on Linux and used Dosbox ( one of many MSDOS emulators ) to run the program… and it just works… it came up and read the original files and could query them and generate tables etc… very cool.

There were some disks that contained files created by the MSDOS backup tool… These I can’t get to restore, I may figure it out eventually, but even running actual MSDOS 6.22 in a virtual machine I can’t seem to get the restore program to read them. I’ll keep digging, there are hints out there on the interwebs that folks have figured this out.

I found a couple of disks that belonged to me and that contained backups of source code for a program I had been writing from 1991-1993 in my own time. It was written originally in Borland Turbo C and some assembly language. So strange to look at code 28 years later, I immediately had all kinds of memories come back of where I was and what I was doing then. The program was building a .COM executable which modern Windows won’t even load. These programs were essentially a single 64k code segment and a 64k data segment, no fixups, no dynamic linking, just a code image that got plopped into memory and executed. Most software nowadays doesn’t even draw breath in under 64k… And the program still runs nicely in Dosbox and does it’s thing like it was yesterday.

Now these files are getting safely backed up for the next 30 years along with the software to read them. I think there are some more floppies lurking in the basement, I may dig them out and do some more digging…

terminal-dashboard v0.0.5

I’ve made a lot of progress on the testing for the dashboard tool, I’m at about 80% coverage across the whole codebase. It was interesting getting the fixtures written for each of the different data sources so that they could be tested. I also added some features for testing and also some refactoring to enable testing. I moved most of the code for loading and processing the configuration file out of the main script and into a separate module. This allows those functions to be tested in isolation but also they now represent a usable API in their own right. I didn’t want to include the config code directly in the dashboard module since it is coupled to all of the different types of data sources and their dependencies. This way the dashboard code could be used with totally alternative graph objects or data table objects without any direct coupling to those specific implementations or their dependencies.

I used a model of snapshot comparison to verify the graphics rendering code, this involved creating static data sets and controlling the screen size carefully. It also requires some honesty on the part of the tester to carefully verify the renderings and not accept bugs. I have at least one bug in my example dashboard that I so-far haven’t been able to reproduce in the test automation… but it is on my list to track down.

My plan is to finish getting through the CLI and config testing and then on to the next round of features and enhancements.

Breaks...

Part of the fun in being retired is that I get to decide when to take a break and it is now free from second thoughts about “What if <name of executive> has a freak out when I’m taking a break?” I realized that I’d been pretty focused for a bunch of weeks in a row on various projects and I needed to step back and get outside and look at some distance and give my brain a rest. So I have been… I’ve been reading some Chester Himes (recommended), getting back to my walks which had been interrupted by having contractors in the house every day, and even just taking a drive and looking at stuff on the rest of Cape Ann. Also, giving in to the urge to nap is also good… the cats are reaching an age where they’ll nap with me instead of just trying to pull my nose off…

I’ve been getting better at resting and recharging over the last three decades, I reached a bad spot in my twenties where after having an episode where I was sent home for yelling at a colleague in a build lab at three in the morning… I realized that I had gotten too wrapped up in work and was just teetering on the edge of burn out… Shortly after that I started taking longer and more restful vacations… one of the best was to a small island next to Tahiti called Moorea… very remote, very beautiful, and our accommodations were a hut over a reef… no clock, no computers, no air-conditioning, no TV… the first few days were hard and then we broke through and gave in to island time and pace and it was an amazing fifteen days… After that we tried to add something like that into our year as a time to just disconnect ( and islands for whatever reason are great for disconnecting ).

I’m not worried about finishing my projects, I honestly don’t care if I do or not… I probably will given my track record… I mostly worry about compulsively turning them into a job because that’s what I’m used to… Ok, back to my break, it’s Friday, never start anything new on a Friday…

Cooking: Fermenting and glass followers...

Since we’ve been getting a farm share from our local CSA at Appleton Farms ( https://thetrustees.org/place/appleton-farms/ ) , I’ve been learning how to ferment things to preserve them ( you can only eat so many beets in one week ). I’ve gotten pretty good at various varieties of pickles and sauerkraut variants based on this very good book Fermented Vegetables: Creative Recipes for Fermenting 64 Vegetables & Herbs in Krauts, Kimchis, Brined Pickles, Chutneys, Relishes & Pastes by Amazon.com Services LLC ( https://www.amazon.com/dp/B00KLNAJ0C/ref=cm_sw_em_r_mt_dp_HS-lFb6ZBKYVZ )

I had initially gotten a ceramic crock for fermenting, but I found that after a while the salt from the brine would leech into the body of the ceramic because it had un-glazed edges and would crack the glaze from inside. This made it so that you couldn’t really sanitize the crock and mold would get into it and would infect your product. I ended up having to discard the crock.

Now I use glass jars, quart wide mouth jars, and this works great and additionally everything is already in a jar and you can just apply lids and refrigerate when the fermentation is done. The only challenge is having a proper seal and keeping the fermenting vegetables submerged in the brine. I had been using plastic bags filled with water which provided both functions, however it wastes plastic bags… So I found two options for glass “followers” (that is what these weights are called) one is a set of solid glass disks made to fit inside wide mouth jars, and the other is using the jam size ball jars which also fit inside filled with water. Both can be adequately cleaned and are reusable. I use a primary follower of parchment paper ( not having fresh grape leaves handy ) then the weight on top, cover everything with a towel to keep the light out.

I have a batch of beet, carrot, onion, cabbage sauerkraut and some dill pickles going now using both methods. I have high hopes, they were both a lot easier to use than water bags…. I’ll report back…

Software Project: terminal-dashboard, progress...

I’ve made a lot of progress on the terminal based dashboard tool that I’ve been working on. It is out there on github now ( https://github.com/jpfxgood/dashboard ) and I’ve made some pre-releases. It is on pypi as well as terminal-dashboard.

The feature set has increased a lot, it handles getting data for graphs from Elasticsearch, ODBC databases, CSV files, JSON files, generic log files, remote systems via SSH. It is integrated with the system keyring so no credentials need to be stored in the config. I’ve improved the rendering and scaling of the graphs a lot, smart shortening labels, shifting stuff slightly to make axis and labels clear, reducing the number of pie slices for clarity, etc… It has a plugin interface for both graphs and data sources, as yet untested… but I’ll get it working…

Next up is writing a test suite for the system and improving the API and user documentation. There are also some minor features like better y-axis scale choices, better format trade offs as the graphs get compressed, performance improvements, check-pointing some of the log readers so that they don’t have to completely reread everything on startup if shut down for a little while, and JSON record format logs…

In the meantime, I’m running endurance tests as well to see if it’s leaking or has any long running bugs…. I still really like the model and architecture, I haven’t had any major refactors and the number of additions and tweaks to the API are rapidly declining… I think the sub-modules for character graphics, graphs, and data tables and data sources are all going to be useful by themselves and I’ve been careful to keep them very decoupled…

I’ll probably do another pre-release next week…

Software Project: terminal-dashboard v0.0.1

Back in 2017 when I was working at Elastic and leading the Kibana team ( Kibana if you don’t know is a web based tool for querying, analyzing, and visualizing data that is stored in elasticsearch, which has as one of it’s core use cases rendering dashboards for collections of operational monitoring data in data centers ) and being the contrarian that I am, I thought: “What about the green screen folks who manage lots of systems, only use the command line and a terminal, and who would balk at installing a whole server just to run Kibana? Why can’t we have a dashboard tool that presents graphs of the same data in a terminal window?” Probably an insane idea, but you’re reading the words of someone who wrote graphical games for the TRS-80 Model I so it didn’t seem insane to me… So I started out using curses and block graphic characters to create a drawing package with all the essential graphics primitives. Then I built a data table abstraction to allow flexible access to the data without too much coupling… Then things got really busy at work and my attention span disappeared and I put the project aside.

Three intense years later I’m retired and I’m spending some time converting old projects to Python 3, moving old svn repos to git, and I come across this project again… I have the time now so I decided to push the project forward a bit more. I’ve managed to implement some basic chart types, line, bar, pie and table. I’ve created data sources for reading the syslog and summarizing it and for getting the machine’s process, memory, and disk statistics. I’ve written a driver script that loads a JSON configuration file that sets up the pages, panels and graphs of the dashboard and displays them and allows interaction as well as an unattended cycling kiosk mode. At this point I think this thing could actually be useful. Here’s a short demo of it running:

There’s a few things that I want to add in the near future:

• Ability to monitor other systems via ssh

• A flexible data source for querying elasticsearch so folks could use it with an existing Elastic Stack deployment

• Management of credentials using the system keyring so it can auto connect and self deploy through ssh to multiple systems

The sub-modules I think might be useful for other applications since they are pretty general and not coupled to this particular application.

It’ll be fun to show the next steps…

Cooking...

I’ve been getting back into cooking now that I feel like I have more time and mental bandwidth. I used to experiment a lot more and try new recipes and ingredients, but when my work life got to a certain level of intensity I fell back to more of a rotation of greatest hits and familiar things. Without a work life anymore, I find that it’s easier to take my time and look at a bunch of the cook books I’ve accumulated or search the interwebs for something interesting to cook.

It is also farm share season and the constant influx of a box of vegetables from Appleton Farm helps to drive cooking activities and planning for the week. Making dishes or food products that can be frozen or preserved is a part of this too, we can’t eat all those vegetables in one week, but over the rest of the year we do. Pickles and sauerkraut are also great things to make and enjoy later. I also have a fish share from Cape Ann Fresh Catch and this means that I'll be cooking at least one fish dish every week.

I also like classic dishes done in a classic way, they became classics for a reason, they became cliches due to overuse and losing focus on what was great about them. For example I made a classic Caesar Salad a little while ago from scratch, it was amazing especially with ultra fresh romaine lettuce from the farm. It was nothing like the watery bland stuff that you get in most restaurants nowadays… The umami from the anchovies and egg yolks against the bite of the garlic and the acid from the lemon is just amazing.

Cooking, when you can do it in a relaxed way, is extremely satisfying, the process is engaging, the result happens in a finite time, and it is something you can share and so it is also gratifying beyond just consuming the food. When I was much younger I cooked in restaurants as a short order cook, turning and burning as it were, and I learned how to relate to food, how to prep, how to be confident about the process, and how to multi-task to get things out at the same time. I remember closing the restaurant and feeling completely drained and yet calm after an especially intense shift. It also formed bonds with the rest of the team in the kitchen even if you were very different people, the intensity that you shared drove you together for that time and when you were flowing together and it worked it was a great experience.

Software Project: slides-sound v1.0.0 shipped...

So, the final project that I’m going to convert to Python 3 has shipped. I restructured it to a standard python project, converted it to Python 3, fixed a bunch of bugs and things that just weren’t good… added packaging and did some manual testing. It is here on github (https://github.com/jpfxgood/slides-sound) and from pypi by doing python3 -m pip install slides-sound.

This is really an experiment if anything and I intend to keep iterating on it, so possibly massive changes will come. I’m not doing formal tests for this one because honestly it doesn’t warrant it yet. If it ever stabilizes then I’ll dig in and do that.

There were more changes than I expected because some of the packages I used had changed in unexpected ways, some of the way I was doing things was completely inefficient, and stuff was broken because it was just hacked together ;-). So now it is a lot cleaner and less hacked…

You can check out the readme and the doc for further information…

This is the end of the Python 2 to Python 3 saga for now, there are a few other things that I haven’t published that I may convert in the future, but these are the ones I use most often and they’re published.

Here’s a parting video generated using the slides and music scripts:

Pictures from my several years commuting to Berlin.

Software Project: ped-editor v1.2.0 shipped!

A new feature release of the editor. The feature is that either when you pipe content into the editor like “find ~ | ped” or you use the F10 shell command dialog the buffer will update in the background for as long as the command produces output. You can press Ctrl-F in that buffer and toggle following, i.e. the screen will automatically show you the end of the stream and the latest content. If you want to follow a file on disk you can use “tail -f filename.log” as the command and the resulting buffer can be used to view this information. All of the regular read only buffer commands are available, copying content, searching with regular expressions, etc…

There are new tests since the underlying StreamEditor class is used in the StreamSelect component and so the existing tests caught a lot of side-effects that I had to clean up as I did this. I have to say having sensitive tests has made this whole thing a lot safer, I probably would have pushed my first cut on this in the past and then discovered the problems later. Now, it all works and is covered with regression tests.

I’ve also updated the wiki usage documentation with up to date screen shots and more information about how the editor works.