Software Project: slides-sound v1.0.0 shipped...

So, the final project that I’m going to convert to Python 3 has shipped. I restructured it to a standard python project, converted it to Python 3, fixed a bunch of bugs and things that just weren’t good… added packaging and did some manual testing. It is here on github (https://github.com/jpfxgood/slides-sound) and from pypi by doing python3 -m pip install slides-sound.

This is really an experiment if anything and I intend to keep iterating on it, so possibly massive changes will come. I’m not doing formal tests for this one because honestly it doesn’t warrant it yet. If it ever stabilizes then I’ll dig in and do that.

There were more changes than I expected because some of the packages I used had changed in unexpected ways, some of the way I was doing things was completely inefficient, and stuff was broken because it was just hacked together ;-). So now it is a lot cleaner and less hacked…

You can check out the readme and the doc for further information…

This is the end of the Python 2 to Python 3 saga for now, there are a few other things that I haven’t published that I may convert in the future, but these are the ones I use most often and they’re published.

Here’s a parting video generated using the slides and music scripts:

Pictures from my several years commuting to Berlin.

Software Project: bkp-sync v1.0.0 shipped!

The suite of backup, restore and sync tools that I use to back up and sync all of our home systems is now officially a v1.0.0. The repository is here https://github.com/jpfxgood/bkp and the package is in pypy as bkp-sync so you can just do python3 -m pip install bkp-sync. The scripts will get installed on your ~/.local/bin…

The transition from the original code base to this one was a massive change, I converted to Python 3, restructured the project to be a standard Python project format, wrote pytest automation for all of the modules and the command line scripts.

I just converted my main system over to using the packaged version in “production.”

In the process of doing this I cleaned up the code a lot, the original had lots of globals and module level state which made the modules sorta useless as an API. I fixed all that, encapsulating everything into classes so now the modules form a very powerful API for moving files between file systems, sftp sites, and Amazon s3. In the process the command line tools gained new capabilities that they didn’t have before, “sync” for example was never intended to work with s3 and now it does, so you could set up a sync between a local folder or folders and an s3 bucket and changes would just get automatically pushed and pulled.

The backup api creates versioned sets of changed files which is very cool as a concept to build other things besides backup software. The controls over what gets backed up are very flexible and powerful.

The whole process went faster start to finish than the editor project partly because it was a smaller codebase with less “features” but also because I had already climbed the learning curve of the tools and such.

So, I think I’ll get https://github.com/jpfxgood/slides-sound converted to Python 3 but without the rigorous testing since I really don’t use those tools for anything but experimenting. The https://github.com/jpfxgood/my_radio project is probably dead, it wouldn’t be worth converting it to Python 3 so I’ll just leave that there as an archive.

Software Project Python 2to3 continued... 66% coverage, >75% in the core editor...

This has been a very healthy process for the editor code, as I’ve been writing tests I’ve also rewritten code to make it much better than when I originally hacked it up. That is the thing when you’re in a hurry, often you’ll stay on a bad implementation track because you’re “close” to getting it to work. It is nice to go back and have the time and focus to reconsider a bunch of things. The tests themselves are interesting, for example I had to figure out how to resize the terminal window to test the code that resizes all the editor panes in response. That code always annoyed me and had a lot of bugs, now it is rewritten and it is very stable.

I’ve been using the python coverage tool to measure the coverage and it has also helped point out a few modules that aren’t used anymore so I’ve been improving coverage through deletion as well. I’m going to try and get the overall coverage over 70% and I need to make another pass over the minimal redraw logic as well… It still has some odd behaviors…

Then on to the backup and sync scripts that I wrote, it’s going to be interesting to create valid tests for those…

Software Project Python 2to3 continued... 47% coverage, 70% on the core editor...

Whoo… a lot of bug fixing, a lot of test cases, and getting all the test cases to work in both non-wrapping and line-wrapped mode… The cool part is that now when I’ve made some minor improvements, the tests are showing me areas that are impacted by the changes and catching side effects. I should have done this a long time ago. I’m done with the core editor module and I’m going to do the module that manages windows and switches between files and such at a high level next. I’m hoping to get the overall coverage over 60% with this one… we’ll see…

Here’s the latest video capture with the coverage report at the end:

This gets me up to about 47% coverage but 70% coverage of the core editor.

Software Project Python 2to3 continued... Wow, such bugs...

The process of converting the editor ( https://github.com/jpfxgood/ped ) to Python3 and writing tests is going well. I am fixing a LOT of minor bugs and edge conditions that don’t really come up a lot in my normal use of the editor but are very evident doing detailed test cases. I came up with some cool ways to test features that are only available interactively by using the keyboard macro feature of the editor to feed the required interaction to the editor main processing loop. It reminds me of testing 1-2-3 for DOS where virtually all of the testing was done with keyboard macros and screen comparisons. It also tests the macro feature, and it already helped me fix a hidden bug in the macro feature where it was accumulating null characters when no key was being pressed. If one left macro record on it would have eventually sucked up all of the heap memory, though I guess it would have taken a while on modern machines. I may have to refactor the main test function since it is getting pretty long, I’ll see where it’s at when I finish.

Oh and running pytest inside VSCode is wonderful for debugging both the code under test and the test automation at the same time.

Here’s the simple launch config that I use:

{
    // Use IntelliSense to learn about possible attributes.    
    // Hover to view descriptions of existing attributes.    
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "externalTerminal"
        },
        {
            "name": "Python: Pytest",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "console": "externalTerminal"
        }
    ]
}

If you want to watch the tests run I have this handy screen recording: