Software Project: bkp-sync v1.0.0 shipped!

The suite of backup, restore and sync tools that I use to back up and sync all of our home systems is now officially a v1.0.0. The repository is here https://github.com/jpfxgood/bkp and the package is in pypy as bkp-sync so you can just do python3 -m pip install bkp-sync. The scripts will get installed on your ~/.local/bin…

The transition from the original code base to this one was a massive change, I converted to Python 3, restructured the project to be a standard Python project format, wrote pytest automation for all of the modules and the command line scripts.

I just converted my main system over to using the packaged version in “production.”

In the process of doing this I cleaned up the code a lot, the original had lots of globals and module level state which made the modules sorta useless as an API. I fixed all that, encapsulating everything into classes so now the modules form a very powerful API for moving files between file systems, sftp sites, and Amazon s3. In the process the command line tools gained new capabilities that they didn’t have before, “sync” for example was never intended to work with s3 and now it does, so you could set up a sync between a local folder or folders and an s3 bucket and changes would just get automatically pushed and pulled.

The backup api creates versioned sets of changed files which is very cool as a concept to build other things besides backup software. The controls over what gets backed up are very flexible and powerful.

The whole process went faster start to finish than the editor project partly because it was a smaller codebase with less “features” but also because I had already climbed the learning curve of the tools and such.

So, I think I’ll get https://github.com/jpfxgood/slides-sound converted to Python 3 but without the rigorous testing since I really don’t use those tools for anything but experimenting. The https://github.com/jpfxgood/my_radio project is probably dead, it wouldn’t be worth converting it to Python 3 so I’ll just leave that there as an archive.

Software Project: ped v1.1.2, pypi and packaging

I finally built a standard python package for ped and released it to pypi which means it can be installed now by doing: pip install ped-editor ( just make sure that ~/.local/bin is on your path ). It is a sign of my brain recovering from years of being a manager ( and by definition very distracted all the time ) that I figured out the packaging stuff in a few hours. I have to say I looked at it many times over the years and I just didn’t get it so I put it off.

I’ve even started following a software development process, running and adding tests, building documentation, pushing releases. I’ve been the one making other folks do those things, but I haven’t had to do them myself for a while. There was considerable grinding and screeching as those old wheels started to turn again in my head.

Now that I have this project in shape I’m going to shift my attention to the backup software ( https://github.com/jpfxgood/bkp ) and do all these things there. Hopefully it’ll go a bit quicker since it won’t include the learning/relearning that I did on this project.

Also, I need to rehab a bike for my wife, but that is a different thread…

Software Project: ped the editor, v1.0.0

After all the work getting the editor ( https://github.com/jpfxgood/ped ) to Python 3 and getting the test coverage to 78% and fixing a lot of bugs, I also wanted to do some improvements. I worked hard on getting the optimal redraw to work and the terminal cursor was still being a problem so I turned it off and now render my own cursor. I also had to redo a lot of the code that managed multiple views so that they would render efficiently and also keep their cursor positions when you changed frames or current files and updates in other windows are properly reflected in all the windows. I also made it easier to see which frame is current when there are multiple frames and cleaned up a bunch of over refreshing in the dialog code as well.

I made it so that editing a new file doesn’t create the file until you save it and also the editor now warns you if the file you are editing has been modified on disk.

I optimized the code for recoloring the current file and now it runs very fast reducing or eliminating any flashes when it has to catch up.

So, given all the improvements and the testing I decided to mark that moment with a v1.0.0 release. I’ll try to do patch releases relatively frequently as I find and fix other things ( two seconds after the v1.0.0 release I found several things and fixed them…)

It’s been a lot of fun getting this code cleaned up and improving the testing, I’ve spend hours at a time in a kind of flow state just coding and debugging. I even forgot to get lunch a few times, which I never do…

Software Project Python 2to3 continued... 66% coverage, >75% in the core editor...

This has been a very healthy process for the editor code, as I’ve been writing tests I’ve also rewritten code to make it much better than when I originally hacked it up. That is the thing when you’re in a hurry, often you’ll stay on a bad implementation track because you’re “close” to getting it to work. It is nice to go back and have the time and focus to reconsider a bunch of things. The tests themselves are interesting, for example I had to figure out how to resize the terminal window to test the code that resizes all the editor panes in response. That code always annoyed me and had a lot of bugs, now it is rewritten and it is very stable.

I’ve been using the python coverage tool to measure the coverage and it has also helped point out a few modules that aren’t used anymore so I’ve been improving coverage through deletion as well. I’m going to try and get the overall coverage over 70% and I need to make another pass over the minimal redraw logic as well… It still has some odd behaviors…

Then on to the backup and sync scripts that I wrote, it’s going to be interesting to create valid tests for those…

Software Project Python 2to3 continued... 47% coverage, 70% on the core editor...

Whoo… a lot of bug fixing, a lot of test cases, and getting all the test cases to work in both non-wrapping and line-wrapped mode… The cool part is that now when I’ve made some minor improvements, the tests are showing me areas that are impacted by the changes and catching side effects. I should have done this a long time ago. I’m done with the core editor module and I’m going to do the module that manages windows and switches between files and such at a high level next. I’m hoping to get the overall coverage over 60% with this one… we’ll see…

Here’s the latest video capture with the coverage report at the end:

This gets me up to about 47% coverage but 70% coverage of the core editor.

Bike Errands / Bike Video

I found a cool mini clamp and ball head rig that is just right for attaching my Olympus Stylus camera to my bike handlebars. I did a test today and it worked great, taking a video of most of my trip to and from an errand this morning. I edited it down from 20 minutes. I’m glad that because it is far enough ahead of me you can’t hear me breathing, just road noise and wind from the bike moving.

Here it is:

This is a test of the bike mount I got for my Olympus Stylus camera. I edited it down from about 20 minutes. I was heading out to get some scissors sharpened.

Software Project Python 2to3 continued... Wow, such bugs...

The process of converting the editor ( https://github.com/jpfxgood/ped ) to Python3 and writing tests is going well. I am fixing a LOT of minor bugs and edge conditions that don’t really come up a lot in my normal use of the editor but are very evident doing detailed test cases. I came up with some cool ways to test features that are only available interactively by using the keyboard macro feature of the editor to feed the required interaction to the editor main processing loop. It reminds me of testing 1-2-3 for DOS where virtually all of the testing was done with keyboard macros and screen comparisons. It also tests the macro feature, and it already helped me fix a hidden bug in the macro feature where it was accumulating null characters when no key was being pressed. If one left macro record on it would have eventually sucked up all of the heap memory, though I guess it would have taken a while on modern machines. I may have to refactor the main test function since it is getting pretty long, I’ll see where it’s at when I finish.

Oh and running pytest inside VSCode is wonderful for debugging both the code under test and the test automation at the same time.

Here’s the simple launch config that I use:

{
    // Use IntelliSense to learn about possible attributes.    
    // Hover to view descriptions of existing attributes.    
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "externalTerminal"
        },
        {
            "name": "Python: Pytest",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "console": "externalTerminal"
        }
    ]
}

If you want to watch the tests run I have this handy screen recording:

Bike rehab project... finale, and next...

I’ve been doing rides and errands on the bike for the last couple of weeks and making minor adjustments. I got the derailleurs dialed in so that there is no skipping or slipping. The bike is great for errands, the cargo net that I got for the back rack works perfectly keeping even odd shaped things safely in place. After a series of rides last week I realized due to pain in my rear end that my old body needed a better seat than the stock Trek mountain bike seat. I got a Terry ( http://www.terrybicycles.com/Saddles ) gel saddle for men, the one made for recreational riding. If you ever need a bike saddle for either a woman or a man, Terry are the best. I just installed it today and the test ride was a huge improvement. Also reinstalling the seat gave me a chance to adjust it’s position front to back and that also helped, I had noticed that I was pushing way back on the seat while pedaling; now I can pedal comfortably in a normal seated position. In any case here is the today “final” picture:

View fullsize

My wife likes what I’ve done with my bike and she’d like me to rehab her bike as well… so that is what is next. It “should” go quicker, I have ordered all of the parts at the same time and I already have the required tools. Her bike is a Specialized Rockhopper probably from around 1995 or so. It has slightly nicer drive components on it than mine but otherwise very similar. Here is the before picture:

View fullsize

I’ll post about progress once all the parts get in.

Software Project Python 2 to 3 continued... so many bugs...

Well, it has been good writing tests for the converted editor code ( wip is here: https://github.com/jpfxgood/ped/tree/master/2to3out ) as it has also shown up a bunch of existing bugs and nits and stuff that was just appearing to work but wasn’t… Plus I got to figure out a nice way to test both the internal editor logic and the rendering at the same time. I’ve been using Microsoft’s VSCode to debug things because it can do it in a detached terminal, very handy for a curses based application like the editor.

I’ve also made some decisions about removing some features that I don’t use anymore ( and I don’t think anyone else would use either, an AIM IM integration, nntp client, svn browser… see… you don’t want those ) and I don’t really want to test them or get them working. So those came out of the code as well, the editor will be much lighter.

pytest is pretty robust I must say, it is handling test code with nested functions without batting an eye, it even shows the nesting in the test failure output… very nice. Anyway onward to hopefully finish the core editor classes in the next couple of days.

Software Project Python 2 to 3 conversion for my scripts...

I’ve got a bunch of python scripts that I have written over the last eleven years or so. I have published some of them on github here: https://github.com/jpfxgood Probably the most sophisticated is the editor ped (python editor) that I wrote originally back in 2009 and have been working on ever since. Over those years I was pretty busy as a professional programmer and managing software engineers in ever larger groups. I snuck in the work on these scripts when I had the time to focus. A couple of things just didn’t get done as part of that, I never wrote proper tests, and I ignored the looming end of life for Python 2 and the migration to Python 3.

Well, now I have the time so I’ve started on both projects. First I brought the github projects up to date, truth be told I use svn locally because it is what I was used to. After this project github will be the canonical version of these scripts. Then I used the 2to3 script to do the rough conversion to Python 3 into a new temporary subdirectory “2to3out”. The script does a decent job, but it was still a surprise that division suddenly works differently… Also some of the code around reading and writing “utf-8” encoded files need to be tweaked. The differences between byte streams and character streams came up in several places. A few miscellaneous changes to the signatures of imported modules and functions had to be corrected. In general the changes beyond what the script converted were small.

The testing was the real problem. As I went through and manually tested each feature of each script I found a number of bugs, most of those bugs were unrelated to the 2to3 conversion, they were simply things I’d never really tested before. So as I got to the point that I was fixing more and more of these I decided that now was the time to bite the bullet and do penance for my past sins and write a regression test suite for these projects.

I liked pytest in the past, it is simple, doesn’t require any intrusive changes, is flexible, and I’ve used it before. So, after a few minutes reviving the brain cells that used to know about pytest I started on a tests folder and a set of test suites for each module in the scripts. I started with the editor first since it is the most complex. I’ll probably do the backup script after that since the fixtures to test it are going to be interesting to implement.

I’ve got a few cases written already and I do thank my past self for refactoring the editor multiple times over the years to make the internals pretty modular and independent. Apparently they screwed with the regular expression module as well as I’m getting Deprecation warnings for many of my uses of it, sigh, I’ll have to go through and clear those up too.

Here’s the test output so far:

View fullsize

It is a meditative process writing tests for eleven year old code, I get to appreciate the good parts of the code and laugh at the crappy parts again.

More updates as it continues…

Bike rehab project... shift and brake cables...

The new shift and brake cables and housings (along with the cutter/crimper tool for making cables and housings) finally arrived. I was able to build new cables to fit the position of the shifters/brake levers on the new handlebars. It went pretty smooth, I got to use my new bench vice at one point to hold the shifter assembly while I pushed on the cable with some vice grips to get it to come out of the shifter. Other than that everything came apart and went back together easily.

I may still have some fine adjustments to make to the brakes so that the front and rear engage at the same handle angle and the front derailleur may need tweaking. I raised the neck on the handlebars before I started to make the fit comfortable while I was sitting more upright. The test ride went well, shifting and braking work acceptably, the riding position is a lot better, no pressure on my hands and arms and my back is relaxed.

Here is the most recent picture:

Software Project Timelapse / Slides / Music

It has been overcast here so mostly taking pictures of the clouds illuminated by the light pollution… In any case I switched to a 28mm prime wide angle lens that is F 2.8 and that has improved a number of things including making the focus easier and getting a wider aperture. I tweaked the music script and also created additional sections in the music so the sound track is better. The ISO was 1600 and the exposure was 10 seconds, I think I can double the ISO so I’m going to just adjust that for the next experiment.

In any case here is the latest WiP:

Software Project Timelapse / Slides / Music

Well here is the first effort that produced images:

It is a bit overexposed due there being a lot of ambient light, this was 28 mm, F 2.8, 20 second exposure, ISO 3200. The focus isn’t good either so I’m going to work on that too. I’m thinking for the next attempt I will try cutting the exposure time and the ISO in half. And I’ll spend some more time in the setup phase on getting the focus right, just have to get used to working with the camera in the dark.

Software Project Timelapse / Slides / Music ...

I’ve been experimenting with controlling my Canon 6D using Linux and using it to capture sequences of photographs in a Timelapse. A while ago I wrote a script to stitch images together into a video slideshow and also generate an improvised soundtrack to play along with them. You can see a version of this project here: ( https://github.com/jpfxgood/slides-sound ). I’m using that script to make the images into a video with a soundtrack.

The camera control software comes from the gphoto2 package ( thank you Linux) and some helpful instructions here: https://www.moreno.marzolla.name/software/linux-time-lapse/

Here’s a sample of the output so far:

I’m working on improvements to the generated music and I’m going to try some night sky time lapse video… more to come… ( also I need to update my github project… )

Bike rehab project... contiued...

After a few rides to reacquaint my body with the bike I realized that for a utility bike I should convert the handlebars from straight bars to swept bars that would allow a more upright riding position. I ordered some bars and they arrived. I was hoping that the existing cables would have enough slack to used on the new bars, but no it was not to be. So now I await cable kits to build new brake and shift cables with the appropriate slack for the new bars. Should be here mid-week. Here’s a preview of the new bars below:

I also realized that I may need to raise the neck as well so I’ll fit the bars and the neck again before I build the cables…

Bike rehab project... Running driving bike...

Update below at 2:23 PM EST…

A flock of parts came in yesterday and so I made some rapid progress. The replacement hose and nozzle for my floor pump came in and after some violence I was able to remove the old hose and using a hose clamp fashion a replacement fitting and now I have a nice floor pump with a gauge and both presta and schrader fittings on the nozzle. My new bike lock came in and I installed the holder on the frame and programmed the new combination on it. The new tires ( very nice road/trail combination tread ) and tubes ( heavy duty puncture resistant ) came in so I was able to assemble the front and rear wheels. I managed to remember all the steps without resorting to youtube. I even replaced the rim tapes with very nice new ones, the old ones were dried out, rotten and crumpled up.

The front and rear fenders came in and fortunately since it is an old steel frame it has mounting points already on the frame for fenders and racks so it was pretty easy to get them on. I only had to modify the front one a tiny bit to make it sit high enough and still fit between the front forks and clear the tire. I was a little worried that they weren’t going to clear the tires since these are slightly larger than the original tires on the bike, 2.25 as opposed to 2.0, but not to worry they fit just fine. I had to adjust the front brake calipers so that they would clear the front tires and seat on the rim, but that was pretty simple.

I have a rack for the back of the bike but I may not get to that until later in the day or tomorrow. I took the bike out for a test run after modifying my bike helmet’s straps to fit my apparently fatter head. Apparently it is true that you don’t forget how to ride a bike, after a bit of panic when I was up high zooming down the hill it all came back to me. I will say that I am horribly out of shape and I’ll be working my way up to hill climbing bit by bit… whew.

Oh, I also got a set of rubber block pedals which work great, they’re easy to get on and off and they don’t have spikes on them that will claw you in the leg if you accidentally let a pedal whip around.

The drive train needs a good cleaning and lubrication, I have a cleaning and lube kit coming Tuesday.

Here’s a picture of the current state:

Afternoon update 2:23 PM:

I got the rack installed on the back of the bike this afternoon. As with many things labeled as “universal” there was some bodging to make it fit and I had to substitute some alternate fasteners, but in the end it is on and secure. Picture below:

Bike rehab project...

I’d like to get back to riding my bike as a form of local transportation for errands and also just for fun and exercise. I’ve got a couple of bikes from years ago when I was riding a lot so, I’m going to try getting them working before I get into buying a new bike. I’m starting with my Trek 930 from 1994, amazingly all the cables, brakes, and derailleurs seem to work fine ( delta some cleaning and lubrication ). It needs new tires and tubes, they’re pretty dried out and I wouldn’t trust them. The rear wheel seems a bit out of true, I’m going to try truing it up since the drive train on this bike is long discontinued and if I have to upgrade it I might as well buy a new bike. The bike needs a bath as well, I’ll probably do that later today. So here’s the before picture:

I also ordered some fenders, a chain guard, and a rear rack to make it more of a utility bike than a mountain bike. Once it is in riding condition I’ll see if I want to replace the handlebars.

Basement progress... Phase 2 complete...

Phase 2 went quickly since I had all the materials. Most interesting problem was trying to drive screws into the old timber in the house. Apparently after 91 years the wood gets pretty tough. In any case, there was a bit of work moving things out of the way and then realizing that the pallet that we use to keep stuff off the damp floor was going to need to be cut down to allow for the new shelves. Here is a picture of it partially framed and then finished with stuff put away. We reclaimed many square feet of floor space…

Of course this has generated a bunch of debris that now needs to be packaged up to be acceptable to the trash folks… But, now there is clean space to work on other projects. Next up, getting some bicycles in operating condition.

Basement progress... Phase 1 complete...

Whew, I put the plywood shelf skins on today. The most challenging part of that process was digging the sheets of plywood out of my pile of scrap wood in the garage. Remind me to be more thoughtful about what order I stack it in when I’m done with all of this… So here is the finished product for phase 1:

It has allowed me to get rid of an old wheeled storage cart that was on the left freeing up floor space. Now I’ve got room to spare and everything is well separated from the sometimes damp floor. On to phase two tomorrow, on the other side of the room. Have to clear a bunch of stuff out of the way to get there.

Why not start in the basement...

When we first moved into this house twenty four years ago, we went through the usual first time homeowner spasm of house projects. There was painting, wall papering, gardening, and building storage and work space in the basement. As with many folks, our ardor for house projects waned as other priorities took over. So here I am twenty four years down the road and I’ve decided to work on building some more storage in the basement for tools and materials etc… One part of it will be next to the workbench that I built back then (and I really don’t recall why I didn’t tear out the decrepit table they had in the corner at the time) and the other will replace some rickety metal shelving I have over by the chimney base.

Here’s what the old table (which could be from 1929 based on the lumber) looked like:

You’ll have to imagine all the crap I had to clear off of it and from below it to get to this state. Here is the result of today’s efforts in breaking it down and removing it.

Next thing is to buy and somehow transfer the mess of 2x4’s that I need back to the house in the Audi… Maybe today, maybe tomorrow…