Inconsistent Records Blog

The claws are open, until they close around you...

2026-02-18T00:00:00Z

...and they may well not be under your control

Pretext

I recently changed my job. Why? It made sense with a young family to try and cut the amount of travelling I was doing. I don't fly, but being away from them and burning a lot of diesel in the process made it a bit silly. Especially when you consider that I'm predominantly an experienced software engineer with a penchant for enabling others to compute, and compute hard.

Openclaw is the flavour of the moment and I first came across it through Moltbook. The number of name changes to OpenClaw is indicative of it's ridiculous development velocity (I know of three, yes?) Given that it can't be more than a year old, the impact of this project is astounding.

Let's be clear, with what I now say, the ingenuity of the invention is in no doubt. This is machine learning (don't get me started on how it doesn't represent all AI) finally working hard.

Agentic AI unleashed in this way is a terrible idea.

Why bring this up?

With the sheer level of integration possibilities into end user systems, we're allowing unproven and poorly engineered bullshit generators to integrate with our lives. Given the spread of misinformation globally, this poses an information nightmare for the future generations, whom we owe a responsibility to. If we're employing technology, there needs to be an end goal in sight that discernibly makes things better than we found them.

Environmental concerns

Everything takes power to run and these agents are no less power hungry than, well, something that could be far more useful. Rather than letting them loose as an exercise in seeing what they can do, perhaps isolate and experiment with them responsibly. I'm a hypocrite at this point, because I've acknowledged the scale of the problem with others. However, we stated that the first thing is to be aware of the cost of what you're doing. That's why I'm working with what I do today.

Ethical concerns

Therein lies the ethical rub. Why are we facilitating integration without considering impacts such as time-wasting for individuals in the real world? We are wasting the worlds resources with these highly experimental bullshit generators before extracting the usefulness from the technology we already have to hand.

And there are great uses for deep learning based technologies, among others. I've been very privileged to be involved with many projects that exemplify the potential.

But it's still only potential, and we are only just finding our feet. We must use technology responsibly and cannot absolve ourselves of the responsibility of judging each and every action we take as technologists, because each joule of energy and millilitre of water taken up by these technologies is taken away from someone else.

I have intentionally tried to tone down the criticism of the AI hype bubble which quite frankly is a load of 40-somethings deluded in their faith in themselves. This constant pursuit of agentic AI being a gamechanger without goals was really nicely highlighted as a concern in a recent interview with Meredith Whittaker of Signal. Kudos to her.

What the fook is a pipeline?

2025-05-24T00:00:00Z

Introduction

One thing I've grown to love about managing a team of software engineers is that they call out the hand-wavey terms you use without thinking. This is not to say the terms are not well founded or even, arguably, used with a decent underpinning ethos in mind. It's more to say, be prepared to define something when you talk about it. Pipeline is a term that definitely fits this notion.

Some etymology

My favourite thing to do is to look at the Wikipedia disambiguation page and there are some really nice links to definitions in there.

But this isn't etymology, though I can imagine the first of those goes some way towards defining the word. Pipeline from a typical etymology defining website is interesting, offering the definition "channel of communication", derived from pipe + line. That's it: it's literally a flow of information in one direction, but with no other defining characteristic.

This is funny. Think about it. You draw complex pictures from many lines. You (depending on style) might draw many lines to draw a flowing directional line.

The definition of a pipeline is, I feel, much more than the sum of its parts.

So what do I think a pipeline actually is!?!

A unix pipe. Put programs either side of it. Repeat.

A pipe is a flow of information. Put some things either side and transfer information using that flow. Repeat.

Is that useful?

Well yes, but not so much as a definition that tells you how to design a pipeline. It's a useful way to think about how things interoperate. I like pipeline definition #1 simply because it's easily demonstrated in any *nix shell with doing command chaining things like this:

$ downloadmydata.sh | prepmydata.sh | savemydata.sh | runmyprocesing.sh

That works to demonstrate the example. If the programs are doing more interesting things and processing each others data, literally feeding one another resiliently, it becomes really powerful as tool for breaking apart steps in an algorithm. As usual for me, selective parts of the unix philosophy offer a nice engineering approach to think about things: write programs that do one thing well and that use each others outputs as inputs.

I think pipelines as a concept are even more important though, especially if you think about them beyond the shell and *nix environment. For example, thinking about a machine learning or remote sensor network, having components via a sequential flow that is well understood is pivotal when deploying applications that started out as research or exploratory engineering efforts.

Processing and data flows

A useful intuition is to think about data transfer between processing engines. This gives a reason for pipelines more generally and then being able to step back when you talk about them. In my day job, there are numerous examples where pipelines exist:

Remote renewable sensor networks: we wait for things to wake up; the things send data back to us; data processing chains that lead to insight or outcomes are executed.
Data management: acquisition processes deposit data in stores; processing starts generating datasets; these are annotated and published or processed downstream.
Machine learning: training datasets are composed from various sources; models are trained and evaluated; incremental training or inference datasets are generated as new data arrives for pretrained models; predictions are pushed out from the arrival of new data.
Procurement: a project starts; equipment is bought based on designs; it arrives and development and deployment builds are arranged; things are built; data processing pipelines are configured; deployments take place; data flows back.

The list goes on, and on, and on, and....

Though there are so many examples, there aren't necessarily well engineered examples everywhere. I think this is what I've tried to exemplify, particularly through my work on developing ensembling workflows for environmental modelling and machine learning for operations in the polar environment. As such, I'd like to highlight two key characteristics that help to communicate the benefits of thinking of these flows as data pipelines: configuration as data and the separation of concerns.

Configuration as data

For me the critical notion to a pipeline is that the data, processing and configuration controlling the processing of those data are kept separate, but all influence each other. This is a design imperative. If you have processing to deal with certain types of data, aside from being well engineered to handle anomolies, that processing should be controlled by independently managed configuration. The configuration needs to be the controller of the processing, so that it can be managed as its own data source that scales, is well managed, describes the operation of the processing and data used...

Separation of concerns

I like thinking about pipelines in a way that doesn't describe the execution infrastructure. To be fair I have a really bad habit of leaning on simple implementations to describe pipelines, with software tooling and configuration bound through something nice and portable like shell scripts.

This is not to say that I don't favour things like Airflow, Cylc, BPM or other workflow management systems. My intention is simply that when I implement pipelines in bash, connecting together tooling, feeding it via configurations, to process data, it's all done in a way that means you can hop the set of actions between these WMS systems. That way you only have integration of known workflows in sets of portable steps into a new graph based execution system to worry about, hopefully without implementation changes!

Conclusions

The flow in one direction might seem like a limitation, but ultimately any fully integrated system still has to work around the principle of unidirectional information flow. If there is a real world object and a decision making entity independent of it, information must flow one way before the information provided by the entity flows back the other, still considered a unidirectional flow.

I'd be interested to understand from information theorists whether this flies in the face of cybernetics and feedback systems, but I don't think it does. What I describe is merely a way of handling that complexity, by splitting it into individual information exchanges. This then helps me to build more complex systems with well engineered wares!

Comprehending the simplicity of data flows is important, especially if you build them.

As I am hopeful to describe what I do more to others in the future, it will be even more important in being able to describe them. "Pipeline" is an unhelpful term, if you can't really work out how you apply it in the digital domain.

The next rambling piece will be "what the flip is a..." something or other.

Please give me your thoughts below, I'd like to talk pipelines some more!

Not talking about work - but why!?

2025-03-13T00:00:00Z

Introduction

I've decided to write this article because I hope it addresses some thoughts that others might well feel within themselves. Can I solve them? Probably not. I can elaborate as an elder millennial a relaxation of my opinions as time goes by.

Being restrained

In the last 20+ years of being a worker (not always in computing, but mostly) I've kept personal and professional fairly distinct. My family didn't really know what I do, nor did I tell them or my friends what I did. This worked particularly well for the first half of that time I wasn't interested in explaining it and there was no real interest in understanding it. Technology was and has been pervasive, but not obvious, which is how we've ended up in a poor situation societally.

Being "proud" or "happy"

Things changed when I left industry and was lucky enough to spend time being a sysadmin / data manager (a term new to me) in Antarctica (link is to the photo posts). I had been a keen follower of science generally, but not knowledgeably, for most of my life. Then I started to understand and contribute to the environmental sciences. I became passionate about continuing.

Since then I've continued with this role. [It's grown]. I manage a team and have a life I love living that is mixed with technology. Why? Because I am passionate about the work that I do but open to other experiences around me. That's extremely lucky. It's also a benefit of a change from working for others and now working for my own motivations.

Not taking it seriously

The important thing now is that I want to talk and write about what I do, but I really don't want to be serious about it. I really dislike being serious. When I'm forced to do it, it's worse for everyone involved.

But I do want to communicate.

There are serious issues around for sure. If I'm to blog about what I do, then for sure it's important to do it in a way enjoyable to read. It's important to think about communicating with others for the joy of creatively communicating, if nothing else. I haven't worked that out yet, but the point of writing this is that it might persuade others in a similar position, finding a job and life they are content with, to change a culture of silence within themselves.

Having some (illusion of) control

One reason I feel really happy about doing this, is that I've gone to the trouble of provisioning the site and associated services on my own server. This is a great learning experience and I at least know that the motivation is just to leave a record of a discourse worth having. I know that I don't have to rely on or care about SEO or platform competitiveness or any of that, this is just about content and (admittedly) self-satisfaction.

Let's see if I can find something more interesting to say about my role next time. Because it's exciting, especially given we're set to build environmental digital research infrastructure that is innovative and, hopefully, forward thinking.

New Year, New Laptop, New Observations

2025-01-01T00:00:00Z

Welcome to 2025

So I never publish blogs when the publication date actually says in Lektor, they appear afterwards. This time, however, I start on the 1st of January and aim to have this done by the time I'm off of my paternity leave. Software engineering mixed with child rearing, this is the hippest place to be in 2025!

That said, it offers an outlet for my ramblings that isn't my ever patient partner. Which is good, because the content of this and many other articles are just not their bag. However, it is in my nature to want to write about things, so I'm going to try and chart what I'm doing on the day to day, week to week, month to month a bit better so that my technical frustrations and explorations have an outlet.

New Laptop

Lenovo ThinkPad T15g 4K Xeon W-10885M 64GB 1TB SSD NVIDIA GeForce 2080 Laptop (image courtesy of Newandusedlaptops4u)

I've just gotten hold of a dedicated coding machine. I've again gotten it from a refurb seller on ebay who I've used previously for many Thinkpad: two T410's, a T430, a T430s, that have tided me and members of my family over brilliantly. My T430s is still going, serving as a general purpose laptop, having spent eight years as my primary "home" machine. I don't know why people don't use refurbed machines more often. The only exceptions I've made are for servers, which I go for as tower units.

That said, this is the "newest" machine I've ever bought and by far the most powerful. It's going to be a dedicated experimentation and code machine (plus maybe the odd game with some friends given its power). This was justified simply as everything else I own is now in use elsewhere and isn't powerful enough for local model development and, ahem, Age of Empires. More genuinely, I find that not having a dedicated code machine has resulted in a dangerous conflation of purpose with my work machine and that causes me concentration problems.

Fedora

I've been using CentOS for server and desktop, and Fedora for desktop, for easily over a decade now. CentOS disappearing has left me saddened and given work have shifted to Rocky 9, that's what I'm replacing trusted CentOS with for the moment.

I really am keen to stick with Fedora. But the new laptop is presenting two challenges: the WIFI is unstable and XFCE dislikes any 16:9 resolutions that aren't 4k!

WIFI problems, a blast from the past

The T15g is not new enough that it's bleeding edge and I've noticed circumstantially the following:

Fedora 39 (out of support) - it seems to work fine on power up, not on reboot
Fedora 40 (about to go out of support) - it seems to work fine on power up, not on reboot
Fedora 41 (current version) - not working

This thread on Fedora discussion appears to have similar issues as the error code probe of 0000:00:14.3 failed with error -110 is exactly what I'm experiencing. But there are key differences and that thread doesn't conclude with specific solutions to the problem. Other than that I've found little else to help, so I'm going to undertake some diagnostics based on that to bring me up to speed before progressing to raising a discussion item!

I suspect the issue comes from changing firmware versions loaded by iwlwifi between kernels in the respective OS's, but given we're fighting the sleeping patterns of a newborn I want to take the opportunity to investigate and get this new laptop working so I can code near the sleeping baba!

My steps to investigate are:

Find out exactly what equipment is here (which is good to know) and investigate how it's failing. I don't think is the same device code as that mentioned above and firmware support suggest it should work, so best to make sure it isn't.
Double check there's no existing issue that explains it (such as this one, I appear to be very non-linear writing this, because I did a quick check before anything else).
Raise it in the appropriate place such as the Linux kernel Bugzilla.

This is very unusual though, as I've not had any issues like this for a good long while with Fedora or similar, so it is interesting that I've ended up in a situation of diagnosing hardware issues like it's two decades ago. Eeek! I so wanted to be moving quickly into developing some toys as well!!!

So these steps are useful to go through, it's been ages since I did any real investigation of Linux systems: I've not worked in sysadmin or devops for over four years and not seriously for eight or so. Therefore, these might be interesting steps to take for others, especially if you're looking at iwlwifi...

1. Work out what's happening

hw-probe is new to me, but running it on my new laptop is reasonable so I have done that to add to the collective consciousness of the internet. What a cool project, I hadn't come across it before. It was easily found in the Fedora repositories and uploaded. I'd be a lot more concerned about privacy and looking in depth if it wasn't a completely fresh laptop and install. ;)

$ sudo -E hw-probe -all -upload
Probe for hardware ... Ok
Reading logs ... Ok
Uploaded to DB, Thank you!

Probe URL: https://linux-hardware.org/?probe=c5dc058f75

The probe includes the error I have in dmesg, with nmcli and rfkill from the same probe both showing that there is no wifi network apparently present from the Comet Lake PCH CNVi WiFi listed. Interestingly, the 8086:06f0 device code doesn't match the Fedora thread I saw previously so that quickly tells me to largely ignore that (it seemed to get rambly after a while anyway) and "The device is supported by kernel versions 5.7 and newer according to the LKDDb".

Confirming some details, there is definitely a device and firmware available, but I would perhaps expect there to be

# inxi -Nxx
Network:
  Device-1: Intel Comet Lake PCH CNVi WiFi driver: N/A bus-ID: 00:14.3 chip-ID: 8086:06f0
  Device-2: Intel Ethernet I219-LM vendor: Lenovo driver: e1000e v: kernel port: N/A
    bus-ID: 00:1f.6 chip-ID: 8086:0d4c

# lsmod | grep wifi
iwlwifi               557056  0
cfg80211             1404928  1 iwlwifi

# modinfo iwlwifi
... lots of output, including plenty of available firmware ...

So why with dmesg do we see [ 17.780760] iwlwifi 0000:00:14.3: probe with driver iwlwifi failed with error -110?

Stab in the dark turning off UEFI network stack: it's always good fun idea to try something random, for a bit of excitement. New to the software engineering with parenting game, I'm trying to keep myself excited somehow. It turns out UEFI secure boot was already off, so unchecking the stack options (a) probably had no effect and (b) didn't fix anything even if they did. It was nice to try and live dangerously for a moment!!! :P

2. Investigate

So next, assuming both the intel hardware support and device information pages are correct, the card should be working. Before I start debugging iwlwifi, perhaps looking at the kernel bugzilla is a good shout, rather than the Fedora forums, as it obviously isn't a Fedora issue if it's in the kernel logs.

The best candidate I came up with from a specific search for the error code was this issue, which matches the error code to a different type of adapter, though it is from intel. Interestingly, there's mention of "Disabling Windows Fast Startup fixed the issue", as well as this being linked to the hibernate / shutdown of the adapter in Windows. This I confirm is mentioned in the iwlwifi documentation and so seems like a good candidate.

I still have the active Windows partition issued with the unit, so will see again if it works (it has every time so far!)

No luck. Sad face... :(

Raising this as an issue seems like the only option, so have done so in the kernel bugzilla.

Graphic problems, I'm probably stuck in the past

Why on earth is a resolution listed that doesn't seem to do anything!? If I select 16:10 resolutions or 4:3 resolutions, it all works fine. If I go for any 16:9 that isn't 3840x2160 it doesn't work.

Until I change the refresh rate from 60.01 to <60Hz!

Then it works fine.

That's something at least.

Takeaways

Sometimes there seems like too much information, especially with all the distros cross-fertilising in advice terms. The truth is, think logically about where the problem is and it gets a lot easier.
It would be nice if there was a simple algorithm painted for diagnosing hardware failures and understanding as a semi-pro linux user with no time (thanks parenthood!) Perhaps that could be a subject for a future post.
A nice map of all the most poignant sites to uncover information about the interconnected world of linux and software would be nice. A bit like this fancy open infrastructure map I just found as I checked it didn't already exist!

Diving into Julia: some initial observations

2024-11-11T00:00:00Z

Julia: my introduction

Julia is a language that I've been after getting better acquainted with for a while. I won't lie, I've actually tinkered with it several times, but now I have good reason at work to go in more deeply, so I am.

I think I know enough to appreciate that Julia is an elegant and very powerful language, but I've not found the transition to a JIT compiled language easy to deal with. Primarily, this is because the best developer worklow that works is more difficult to determine. With compiled languages, you know you have to go on and get on with compiling on amendment. With scripting languages, it's more immediate to get results and understand flow of execution, almost making it too easy to throw things together and debug-as-you-go (yes, I've gotten spoilt in recent years!)

Why transition to Julia?

I am a pretty experienced programmer, having spent more time with Python, C, Java and Perl in commercial worlds than research. MATLAB I never enjoyed being proprietary and Octave, though I've used it, was only used for explicitly research experimentation and never for anything that would be automated or configuration driven.

Julia makes serious effort to be both scientific-computation adept and able to perform as a systems programming language. This is in addition to being extremely well-thought out from a language design perspective. Primarily, as I move into research computing for natural environment modelling, I'm explicitly drawn towards something that is designed, well, as a language for distributed computing on these workloads. I know enough to know that Julia is worthwhile on this front.

Challenges

For someone who is well versed at having effective workflows when programming, I hope it's helpful to highlight some things that I've struggled with.

Versions

Previously, I was using Julia 1.5+ through a manual tarball extract and/or RPM install, either symlinked or at the behest of the RPM channels.

Then came juliaup!!!

I haven't spent time using this but was jumping from 1.9 to 1.11 and it has suddenly appeared in the intervening months I wasn't paying attention. Go get JuliaUp!!!. It's basically nvm (the thing that makes NodeJS and NPM usable) for Julia and is a welcome addition.

Managing Julia versions manually is annoying, so get rid of this as a first step into the language. I'm glad it's appeared and is now taking that pain away.

Development Tooling

Pluto.jl

First and foremost, I can't get on with VS Code + Julia extensions. I find the interface wholly unappealing as someone who alt-tabs a lot. I don't like learning lots of new shortcuts to use an IDE effectively (VIM was the last time). In addition, the plotting window is horrible to look at. I'm likely missing something, but both Jupyter via IJulia and Pluto.jl seem like better options.

Starting the Pluto Notebook Server.

The deciding factor having tried Jupyter is that I didn't want the burden of IJulia managing miniconda to produce a Jupyter environment to instantiate IJulia so that I can run Julia. I use Jupyter notebooks elsewhere, and this seems a bit around the houses.

So, Pluto is where I'm at!

Things I immediately like:

Running the Pluto Notebook.

The fact that it is taking advantage of the Julia ecosystem: this is a big win!

The reactivity of the notebooks: I'd previously tried using Observables with GLMakie (which is awesome) in VS code and this is what left me wanting!
The ability to embed pretty much anything into them
The speed: debugging might be an issue and these are notebooks, but using this doesn't stop me using a debugger seperately
The export functionality that allows Julia code to be linearly run straight from the notebook.
The level of visibility for diagnosing the startup and operation of the notebook.

Finding libraries

Initially, I was playing simply with plotting and the example notebooks. The system is managed via the Pkg.jl and relays to the packages ecosystem. The wealth of libraries in Julia is overwhelming, but that's easily overcome if you have some intended purpose behind you!

Suddenly things changed

I got distracted (life is distracted with a newborn) by lots of other things in writing this post. So over the month or so since I started it, I got frustrated with Pluto due to the onerousness of JIT happening in temporary environments constantly, as well as finding that things aren't entirely translatable directly to Julia. All in all, it wasn't a transparent development process.

Back to VSS

So I'm back to VSS, from Flatpak: the open-source version without the extra analytics. That and vim, though that's a slower game for getting things done. We'll see how things go as I employ tactics such as Revise and get into more complex projects. I am struggling with VS language server issues in the OSS version, but will persist for a while...

Getting back to tech: a software engineer is easily distracted

2024-11-02T00:00:00Z

A software engineer is easily distracted

Crapoly, it's been a long time since I wrote anything. Ho hum, it's mostly for me anyway (if not, and you like this, leave a comment!)

If you get asked to lead a team, undertake challenging projects that aren't "the flavour of the month", or change your life circumstances, it could be the end of you doing serious "coding". Most people tell me all these things will make that so. Hopefully as this post demonstrates, it doesn't need to be the end of you doing engineering. It could even be the start...

For almost twenty years I skirted aside from the dangers of line management, undesirable projects and splitting my attention to other personal life goals. No longer. Since the time of the last blog (that was over 18 months ago), a lot has changed. Nearly all for the better: work, tech and life have all seen massive upheavals and though challenging, it's rewarding. Some I'll share in detail, some I won't.

The basic reason for this post is to describe some lessons learned and useful advice I've had along the way, if applicable. Or at least prime the reader for future posts if not.

Leading a team

Off the back of years advocating for sustainable software engineering, mostly off the back of a fellowship, I "accidentally" got to start a team of engineers. There will be more about this another time, but some key lessons I've learnt are:

Line management and leadership are not the same: this is perhaps obvious, but I didn't know it. Line management means you care about the wellbeing of those you (try to) direct in mutually agreeable directions. Leadership means you care about the organisations interests when (trying to) direct[ing] people.
Line management and leadership are both required: don't try and do one without the other (see above). The role is to ensure that you work on keeping the balance of the two, pretty much nothing else.
You'll learn more about your own character than you might have before: I've managed plenty of people and projects, but being in a position where you're caring and leading is really complicated. You'll understand more about yourself in the process. Sometimes that's good, sometimes it's not. Always it's helping you grow so lean into it and learn from it.

This was not a career aspiration for me. In fact, I kind of feel like I "fell" into it through pure willingness to try and make things better for an organisation I care about working for. The idea that I could perhaps influence some positive change was a driver, but this comes at a cost and potentially might not be realised. Therefore, it's worth seeing the other benefits, which I now (after a year) see the above as being.

Pivoting to environmental science focused software engineering

I was a big data, middleware type of infrastructure / devops engineer type person, dealing with keeping systems up for profit-making companies and service providers. There is nothing wrong with this, it leads to interesting infrastructures to work on. When I went to Antarctica, the pivot was from highly available, fault-tolerant, scaleable and secure infrastructure to resilient, deeply redundant and continuous infrastructure that scales. The motivators were different.

Now I work specifically on software that can, in theory, be responsible in allowing people to scale compute and serve data products for environmental science. Working on projects that seek to integrate environmental data; modelling and AI data pipelines; supercomputing platforms; and data publishing platforms is amazing fun.

The challenge is (a) the breadth of focus and (b) convincing myself that we're going in the right direction. AI in particular has a lot of questions to answer and is really very high in the peak of inflated expectations, but the same can be said for many other "cutting-edge" technologies and groundbreaking ideas_.

The method I try to employ is to (a) always have one tangible project to deliver to keep me grounded and interested in real-world delivery; and (b) to always remain skeptical!

Life taking priority

The biggest thing that has had me in recent times is life itself. Family, home life and other businesses now pull at my attention more than ever before. As an "early millennial" (to gauge that, I recall watching Innerspace the first time it reached TV), the reality has become that the most important thing I can do is not fuck things up for the upcoming generations...

...and this gets to why I've been concentrating on the previous two items alongside the third. They are the thing I can do to try and have a positive impact. But they are not the most important thing. I can demonstrate what I've learned, but others now can learn and adapt way faster than me.

And in this vein, I've been learning more than ever so that there is something to express (writing, music, math, science, technology) from my knowledge base. I've been enjoying learning again, but it is the application of that knowledge and communication which will become more important.

That's why I'm back here, blogging pointlessly (perhaps) into the digital void: to illustrate what I'm doing for posterity and to guide myself as things distract me. In no particular order, I will hopefully describe:

How to rebuild a long forgotten electronic music setup and coupled it with guitars.
How to dust off maths knowledge and rekindle long forgotten passion in doing so.
How to build environmental science tooling for field, compute and data platforms.
How to take the pain out of provisioning systems be they small board computers or multi-host, multi-site infrastructures.
How to annoyingly conceptualise everything in technology, with a view to communicating about sustainable technology.
How to get a hold of the modern day internet for research software engineering, without getting smothered by its scale.

Achieving any one of these will be fun. This has been rambling, but hopefully it won't be 18+ months before the next time!

What's open on my phone: start of 2023

2023-02-19T00:00:00Z

Introduction

I'm determined to continue writing publicly, but haven't been able to find much to focus on and understand in relation to my path. Perhaps, this is thanks to the uncertainty of that path, especially in my professional life. That needs sorting, but I love writing so decided to save ten tabs that I found interesting or at least slightly worthwhile from my phone. I don't like reading on my phone, but when I do it allows me to be a very networked affair, inspired by what I'm doing elsewhere, which means there's both a little connectivity and diversity to my general activities without being too focused and specific on a narrative.

Hopefully people find the pieces herein interesting.

UK: Online Safety Bill is a serious threat to human rights online

This was a really important read for me. I've known that some of the elements of this bill have been problematic, but never quite got my head round exactly what the key issues are. I appreciate these will be subjective, but over time the rhetoric of UK news has felt biased towards the idea that the "Online Safety Bill", intrinsically given a name to that belies it's coverage (no change there), is benevolent at it's heart and not at all leveraging public fear and misunderstanding with respect to technology to control information freedoms.

I really advise reading this, it's a bit old now but it's a great introduction and I will be reading far more from Article 19 in the future.

The big idea: has organic food passed its sell-by date?

This was an interesting read, especially as (to me) it seemed like a very pointless and damaging journalistic piece, using lazily cherry-picked statistics (from an unknown source) and divisive assertions to express the opinion of the author. The ideas stated in the article that organic farming is trying to overtake the market, claiming health benefits (via the consumers) or working at the expense of other novel approaches to food production is assumptive, especially when the consumers are cited as the proponents of these messages. It seems to me to be an example of poor journalism, but on an interesting topic to consider: people are becoming ever more zealous in their discussion around food and this article highlights that from the way it's written.

I really like reading and writing, as well as food (organic or otherwise), but I lost respect for the arguments of the author with the tone of the language. That's a real shame, as the last paragraph was the best at bringing across a sensible message that we need sustainable and varied solutions for food production into the future. Organic farming offers insight into how things can be done better, in an accredited, responsible and controlled fashion: that's something that can be learnt from positively. I think it foolish to talk negatively about any industry that is trying to diversify and experiment for the benefit of us all.

Ownership of AI-Generated Code Hotly Disputed

This, especially now (since the subsequent release of ChatGPT) is a very poignant article that really does get to the heart of the issue with using Large Language Models (LLM), in this specific case for the generation of code. The fact that Microsoft developed the system based on its unprecedented access to open source code for the purposes of generating revenue flies in the face of their "commitment to developer freedom, openness and innovation."

This is worth thinking about, as it's a blatant exploitation of the openness of the internet. Not one that is unprecedented so much as one that highlights how things aren't changing in this space as time presses on, despite our collective knowledge that a few big players are taking the piss.

An Urgent Year for Interoperability: 2022 in Review

This is an interesting piece to read. What's more interesting is how interoperability and federation are used as terms, they're extremely vague and generalisable. I recently stated in a meeting "federation has no discernable meaning alone, you need to state what you're federating." People love using these without clear context because it sounds really impressive if something is interoperable and federated. In the dire rhetoric-fest of Digital Twinning, these come up a lot.

This article is quite good at highlighting that interoperability is an important subject. Federation is mentioned too, hence my mentioning it. It describes how people are thinking at the legislative level how to ensure people are not trapped and beholden to individual providers. More importantly though, it shows how woolly and vague these terms can be, lacking in manifestation for readers. I like this article on many levels, and dislike it on many others. That's probably a good thing!

Earthwatch: Citizen science

"Earthwatch has been using ‘citizen science’ – actively involving non-scientists in scientific research - long before that term was invented."

I like the concept of citizen science, but mainly saved this page as I thought that the quote above was hilarious. Does anybody else get the impression that this is the culmination of two children one upping each other in a conversation until the one-ups get so ridiculous, that it just ends in "whatever..."?

Velociraptor Aerospace Dynamics Internet Map Posters

I really like maps and posters, and I like that someone has gone to the pain of making these. Also really nice to see both IPv4 and IPv6 (partially) represented!

Edgar Ansel Mowrer / Dorothy Thompson

Whilst watching a particularly involved documentary called "The US and the Holocaust", the experiences of both Edgar Ansel Mowrer and Dorothy Thompson came to my attention in one episode. The film is really insightful and extremely engaging, though understandably horrific. I implore people to watch it, but I think with my desire and fascination around journalism, I'll be following up on my studies regarding these two in particular.

Dempster–Shafer theory

I've saved this tab as I thought it was interesting. However, it highlights that no matter how much interest I might have in reading something, without a reason to read it I just don't find the time if it's on a phone. I've kept it, because it was the end result of a wikipedia bimble whilst watching the film above, and sounded like a very interesting theory to read up on.

Smart Meter Data

I'm very interested in being able to monitor and model the house I live in, and in the UK getting data from your own electricity supply, which runs a Home Area Network HAN in your home without any transparency to the user, seems somewhat cheeky. Following some conversations I narrowed in on a potential source of data, which I will investigate in due course.

Netblocks: Mapping Internet Freedom in Real Time

One final tab, that is a really useful tool for seeing where internet oppressions might be taking place. This is a real eye-opener and a tool I've no doubt useful for me and others. Hopefully, we in the UK won't find ourselves on this at any point, but I'm an outward looking person and it's a fantastically useful site to help highlight the oppression of information freedoms elsewhere. Definitely take a look.

What's my motivation?

2022-08-20T00:00:00Z

Preamble

I write from a position of privilege. This is an important premise to this piece to acknowledge, because it underpins the need to build my motivation. Reflecting on the topic herein has been important in addressing that not saying anything is far worse than the alternative: to have stood impassively at the watching, and feeding off of, technological development having without trying to influence it for the better.

And this is where I am. I've been lucky enough to find a comfortable home, a great job that allowed me to travel the world for a few years and I'm now passionate about who I work for. However, this niggling feeling that I'm taking more than I'm giving remains, and that's because I'm in a position of relative privilege and not doing my best to at least try and influence technology for the better. Part of this is disenfranchisement, it's easy to become that way when you look at the reality of what you can really influence. Mostly though, and this is what I've reflected on, it's just outright laziness.

A sense of responsibility

During my times in Antarctica and even during the pandemic when I was treading water, enveloped in what a wise person I know calls "life procrastination." I noticed that I could understand my civic responsibility in these scenarios and tried to work towards it. Once the freedom to move around returned, I figured that I could build an ideal base for exercising and developing this sense of responsibility.

What responsibility?

In Antarctic seasons, you have to offer your personal energy for the benefit of a relatively closed community. In the pandemic it was (and remains) hugely important to think of others in the wake of an extreme immunological event. For the former I offered as much energy as I had, and in the latter I became a vaccinator because I trust science to provide (in that case) solutions, which it did. Since I moved after the lockdowns finally lifted, I've known that I wanted to develop the role of a British Antarctic Survey (BAS) Research Software Engineer (RSE) for the benefit of polar science, which I think I'm roughly getting there with. For some reason that hasn't been enough to satisfy my desire to make a positive difference.

Developing a motivation

I took a week off, as the need to develop a clear motivation towards what I do has been influenced by reading The Age of Surveillance Capitalism by Shoshana Zuboff, which I'll refer to as T.A.O.S.C. I started it in February (I think) thinking it would be another "dump social media" type parable that tries to sell logic through fear. I finished it last night, having had every paragraph sink into my mind deeply, instilling a sense of awareness of my own oversight and ignorance in the face of our changing world. I believe this is one of the most important books anyone involved in data science can read, on a par with Rebel Code by Glyn Moody, which I think every software engineer should read, especially if they are born after the 80's.

How capitalism undermined (in particular) Gen Z

I've been involved with technology, from the programming sense, since my age was in single figures. Though this is common these days, at the time it was rare for people to start programming so young. Though considered a Millennial I hold clear memory of a time without the internet, mobile technology and email. My passion for computing was not limited to gaming as it was for many at the time, but more in trying to understand how computers worked, so programming became a natural practice for a young person with LOTS of time to spare. It turns out this was a really great time for people to build an interest in technology, communications and the internet, and it's these people who produced the dot-com boom and the big data revolution(s) that followed it.

For those that grew up after those who saw this transition, the utilisation of computers seems to have shifted to be geared towards interaction over investigation. I distinctly remember being fearful in the early 2000's that the young were going to be culturally and socially impacted in ways too complex to understand and, to be honest, did nothing but watch. T.A.O.S.C really drills into this point in it's later chapters (I'm going to be writing based on this book for a while.) Social media has distorted the very fabric of all generations that followed mine, just as aspirational media sort of did for my generation and the one before it. However, the net effect is far more pronounced for Gen Z and those after them and there is little to no ability for this to be unwoven from the makeup.

This all makes me realise how lazy I've been in looking at these issues, in spite of working full time in technology for a long while and having lived the transitions first-hand. I turned my attention away from profiteering through technology. I realise now that in doing so I turned my gaze away from a desperate situation that needs to be paid attention to by everyone who utilises technology. There's a very clear message that you can switch off and be free which is not misplaced (and I recommend the book linked to highly), but it's often not practical, and potentially even anxiety-inducing, for those who have never existed without this level of communication. Another approach is required that adjusts the system, even exploits it equally, rather than creating a bipolar opposition to it. I feel T.A.O.S.C offers a good insight to leveraging existing frameworks to right the wrongs caused by the extremes of early adoption.

Environmental Data Science and Ubiquitous Computing

A pervasive topic within T.A.O.S.C is the reliance on adjusting society for the benefits of surveillance capitalism, namely to maximise benefits via the increase in certainty of those populations influenced by "Big Other." The primary benefit of utilising ubiquitous computing towards the ends of environmental data science is to incorporate uncertainty quantification into the picture. There is no realistic (as far as I know, but I'm limited in understanding, so please get in touch if this needs revision) proposition for ensuring certainty within one or more environmental domains.

This offers us an interesting proposition for a shift through utilisation of technology (which is all but inevitable, but also rather sensible) to help us understand the environment and address climate issues, which could also feed back for the point of attenuating the need for certainty within the surveillance capitalist domain. One could argue that we should've been developing technology for this purpose initially, but that's not how it worked out, so hey! What I can ascertain is:

We (particularly technologists) have a responsibility as we move towards developing environmental Digital Twins, from localised to broad that we don't act blindly or unethically, as surveillance capitalism has, in the desires of achieving certainty and increasing capabilities to render real-world behaviour virtually.
That we use this opportunity to stand on the shoulders of giants, and feed back what we learn for the betterment of humankind and the environment.

Conclusion

In writing that last part of this piece, the link between what I do day-to-day and my personal feelings has been established. It's obvious that I have a motivation but without saying anything about it, it's difficult to really get behind it, so writing this is the first step towards writing a personal manifesto of sorts! Being openly vocal about things like this shouldn't be something that I fear, but for some reason I do. Possibly I don't feel educated enough to talk about it, possibly I worry that by writing about this I have the potential to upset people. If there is a risk there, then it's worth taking, because by not voicing an opinion about an industry I've worked in for years, I've been implicitly exploiting my privileged position.

This is the first step towards building a motivation: doing something I believe in and ensuring it's not ignorant to the possibilities for harm. Tracking the progress in relation to industry development I was aware of and, to a very small degree, part of, is something of potential value to be contributed to the discussion. The above two points couple this agenda to technological and scientific areas I love working in. The areas I work in have the potential to help us gain much needed understanding about our world, what we're doing to it and how to manage our future. If this rambling piece does anything, even points someone at a book or paper worth reading, I'll be bloody glad I wrote something.

Why I've updated my about page

2022-01-11T00:00:00Z

As usual, all opinions are my own

A prelude

So this post is about why I've updated my about page, which probably seems pointless but actually is quite an achievement for me as I'm not predisposed to public writing unless it leaves out all semblance of personal information (which hasn't, if I'm honest, done me any favours in the long run!) There have been a few things that have tipped the balance for me recently though, including being fortunate enough to be able to work as an RSE for the polar science community in the British Antarctic Survey (BAS) AI Lab, probably because my incomprehensible sense of idealism when it comes to technology fits the values associated with the role.

Now I don't like job titles, but the fact is academia has long needed people working as software engineers to be recognised for the role they play in research, because otherwise it'll forever be an ad-hoc endeavour, which in turn could put people off (be they scientists or no) taking up the role. I do find it interesting how there seems to be a prevailing assumption that scientific researchers become software engineers, rather than vice-versa, but it's a small bugbear. I think this speaks more to an issue with there being enough software engineers moving towards research rather than industry. Therefore, I'm very happy to be called a Research Software Engineer, because I've had some completely deranged job titles before now, in the IT industry as well as others...

I'm very privileged to work in my role and to have had the honour of spending several years supporting Antarctic research projects (retrospective Antarctic blog started here if you're interested) in addition to my previous experience as well as my current opportunity...

The actual article

...which leads me onto why I've finally decided to write a better about page that explains who I am.

One of the biggest differences I notice in the academic world is that people actually put their name to their work. Now that I'm working with people I want to do the same, because I believe that the work we're all involved with at BAS is something worth communicating. This is the first time I've really thought "hey, this could be worth communicating about..."

Almost certainly © 20th Century Fox

This might sound slightly bonkers, especially if you're younger and grew up with the internet, but I'm in my late 30's and I started programming very young. When the internet came along I didn't want to be an entrepreneur, nor did I harbour any illusions of making millions. By the time I got a "proper IT job" I was just trying to make an honest living and live a life without adopting these new mass data collectors into my life ubiquitously (I will suggest a few reading materials on this subject at some point, but this book is a great read to consider the implications.) I was inherently distrustful of crawlers and social media from the outset (probably not a bad thing), but now I figure they're less the problem than those who seek to abuse the technology. I put some of this luddite-like approach down to spending a lot of time rural areas when I grew up and having a Fox Mulder "trust no one" poster on my wall, though I suspect I'm over-simplifying the true reasons. That's probably for another article...

Also in the "noughties" and beyond I worked for my fair share of companies who didn't want their technical staff to speak about what they were doing (and sometimes with good reason...) It meant I didn't get swept up in the melee of it all, but it and some significant life events left me wondering along (in a reasonably capable technical career, I won't lie) trying to fathom whether there was actually a route to using my skills for a more beneficial purpose.

Eventually, following some misguided attempts at charity and socially responsible computing (from whence came the moniker Circadian) alongside my permanent roles, I ended up at BAS. I say ended up, I sought it out and waited for the opportunity, for a long time. When I did it, it wasn't exactly like I thought it would be (that would've been very spooky had it been), but the scope for making something out of it was much greater than I could imagine...

It's been an interesting journey and I see often questions on places like Hacker News and occasionally talk to, or get passed questions from, people asking me about how they can work for the benefit of science and research or look at a more morally satisfying switch out of interest. There is no answer or trick, it's just about going, persuading people you're who they want and then working for an organisation who you agree with. You'll have to prove that you can do it though. Why? Because in research there is f**k all money and often scarce expectation that people will leave industry which means you have to exemplify your credentials for roles that might be looking for academic qualifications. The side effect for academia is that valuable and far less numerous researchers are forced to split their attention because there isn't a massive pool of people leaving industry (I can't find a link, but usually it flows the other way.) Having researchers do this is no bad thing, but it needs to be balanced, so that experience can flow out of industry to nourish the academic and research domains.

And this is why I have update my about page, because if you're willing to leave industry there are dozens, hundreds, hell, possibly thousands of research organisations who would gladly pay for experienced IT professionals to come and work for them. You might not get a silicon valley or London City lifestyle, but if you want to contribute to the very broad field of academic research at a time when it's most needed, the role of RSE not exclusively offers a path to do so. From my own experience every day is a heck of a lot more fun and interesting than I ever had in industry, which is why I still do it even though I don't go to Antarctica any more...

When I mention that RSE offers a non-exclusive path into the research domain, I really mean it. Consider that before being a RSE I've been a "Data Manager", "UNIX Engineer", "Principal Engineer", "Senior Engineer", "Senior Software Developer", "Technical Manager"... the list goes on. For me, RSE is the first job title that actually says what I do properly, with a definition to go with it...! Ignore the job title, find the organisation, then find a way to build the role into what is well defined as a RSE, it'll be rewarding if you do!

I hope my writing this and about myself more honestly helps me to blog better, because trying to blog whilst being a "Fox Mulder" and not giving anything away wasn't working. I've been pretty honest here about how I feel the switch from industry to a publicly funded research organisation (BAS is funded by the Natural Environment Research Council in the UK, if you wondered) has gone. Hopefully this one day crosses the path of someone wondering what to do next, as I was a decade ago when I decided to try and get a job in Antarctica, and persuades them to give it a go...

Solutions and Problems. Problems and Solutions

2021-07-05T00:00:00Z

Introduction

Holiday and life has given a little break between the blog posts recently, but back to work today and I'm over the moon to be starting a new project in my role as RSE (rather than those I'm doing which are carried over.) There will be more on this technically without doubt, but I want to start with something more abstract and conceptual to write about, which arose during my first day back without having thought about software for a while.

This is purely an opinion piece, derived from personal experience and a trigger to a thought process I have regularly. Nothing so much happened to create this ramble today, it just occurred to me it's a line of thinking I often think of. There are always better practitioners out there and I'd love to hear from you in the comments if you would like to feed back. I will respond to everyone who does!

Problems to Solutions

Once, many moons ago, somebody on a conference call suggested to me that we switch the entire infrastructure from on-premise VM infrastructure (effectively a LAMP stack on steroids) to an on-premise containerised infrastructure. This was in the mid-2010s and my reaction, following the call, was simple: "why on Earth would we consider that a viable proposal when their codebase is fucked?"

This company was a billion£+ turnvover company and the software infrastructure had numerous problems, not least that it wasn't easily able to scale horizontally because of the way the application stack was built. That's not necessarily important, but what is, is that this was a prime example of the "shiny solution" arriving before identification or addressing of the problems. So instead of solving the problems, they thought that a much more abstracted technology would automatically solve it (and without any significant cost, in spite of us being a contractor!)

What I mean by this is that with anything developed externally, adopted via legacy or technical debt, or inherited from a previous incarnation we have to identify solutions to problems before throwing it into a new infrastructure or adopting solutions that don't meet the problem statement. The sad truth of the matter is that the path of least resistance often doesn't lead us there and thus we inherit the debt, design issues and flaws of previous iterations when we use a solution to wrap up problems. I think this is dangerous, as well as being lazy and more importantly it doesn't help us to understand or fix old code which is important for new developers. Addressing technical debt allows us to redesign, rearchitect and understand the development of technologies over time, which is not something to be sniffed at.

Solutions to Problems

Applying solutions to problems is purely a case of misappropriation, so we should always look to solve problems on a case by case basis. This very much follows on from the previous section, is simple to understand when you read the words, but is not the way industry and science always behave. The fact is that computing in the last 30 years (about the time I've been programming) has gone from "get all you can" to "have all you want"; that is, we are so abundantly provided with computational power we can make any shit stick to the problem based on "preferred" solutions.

Why is this a problem? Well, think about the cost. Firstly, computing power costs energy, something I'm always keen not to waste. If you spend time recycling at home, why are you willing to be lazy at work, especially with something as valuable as computing resource (think about electricity, cooling, precious metals and sheer manufacturing cost and you start to get a picture of the cause-and-effect relationship to mistreating computing resource.)

Secondly, it costs money; if you (like I do) work for a public sector organisation, that's easy to understand. In the private sector, you're only ever cutting into profit margins, reducing your own pay, or raising costs for the consumer and thus contributing to the ever present dangers of inflation. This all feeds back to the supply chain as well, much like the first example did.

Finally, think about the technological development and the support chains that are created. If we homogenise technology by applying solutions uniformly to problems irrespective of their suitability, we damage innovation and promotion of skills. A world that is composed of few technologies is less likely to adapt, so it is important that we make every effort to identify where the most appropriate or optimal solution doesn't exist for our use case and work out how to arrive at it, because that innovation will likely serve others.

Is there a conclusion

Not really, it's mostly a ramble. However, it does lead me to think that this hints at one of many interesting continuums in IT. Maybe that's something I should really be writing about: the software continuums.

Every day I learn a little bit more now the context for my work has changed. Today was no different. The discussion topic for this post was something I think about often, so it interested me; whether we approach solutions from the most appropriate or sustainable fashion throughout IT as a whole. Today I also learned other things that I and others have realised, such as the fact that meetings and administration suck concentration away from a role (which I'm seeing could be more problematic in a research-oriented role.) I learned (again) that small actions can build bridges in the information world, such as the importance of simply forwarding information on.

These latter items I didn't focus on because every day, you realise that something you think "a-ha" to is probably something you learned at least once before and with me that happens a lot throughout the day. So today I had to pick a topic to try and write about.

Welcome to the pain in the arse that is the human mind: humans like building habitual behaviours, and that's why I wrote about a habitual thought that keeps cropping up. Ultimately that's the benefit of having a little experience, when someone arrives at a behaviour you have identified as sub-optimal, your habits tell you to ensure you don't follow blindly.

First week in a life of a new RSE

2021-06-19T00:00:00Z

Introduction

My home office. Very lucky to have the dedicated space to keep work away from relaxation!

So this week was my first as a Research Software Engineer for the British Antarctic Survey, whom I've worked for over the last five years in various roles. I'm keen to blog regularly about this, as I feel it's an important culmination of 20 years working in (and around) the tech industry, with longer spent programming, but especially because I'm not arriving as an academic.

If anything I can write is inspiring or informative about the research software engineer role, the British Antarctic Survey or the science we do for anyone, then it'll be worth blogging!

The next section was written as I started, then I got on with some work, then I decided to actually write after the first week. This intro was written at the end of the week, so I best reread what I wrote on my first day! ;-)

Transition first, is it!?! (14/6/21)

So today is my first day as a Research Software Engineer. Nothing feels different, but that's perhaps not so surprising given my line manager is the same, the organisation is the same and I still have a pressing need to deal with the same projects I did on Friday.

All that said, I am very excited!

So what has changed? Well firstly, I don't have to deal with ongoing support request (as much) which is a massive time saver (I was in a role that is best described as third line support up until this point.) Secondly, I know that once I get through the initiatives I'd started in my previous, I've got a modus operandi to really take on board what we're doing with software in the British Antarctic Survey and make it more accessible and more beneficial for everyone in the organisation. This is a loose remit, but a very important one.

That's my message to myself as I transition into this role: keep your eyes on why you wanted to do this in the first place!

First week down!

As anyone who has even remotely followed this blog knows, I'm not a natural blogger. It's definitely something different to writing fiction, which I've always been more at home with. I'm keen to work on a way of distilling an experiential and technical narrative that is clear in its message, as time goes by, but in truth I'd like for my internal narrative to flow freely as well!

Aspirations for BAS and its collaborators

Software development, if anything, has been my lifes work so far. I've been fortunate and privileged enough to work in a variety of industries, with a lot of different technologies, as well as gain insight into non-technical areas that are strongly associated, for ill or for gain, with its practice. I've been through a simply ridiculous number of job titles, most of which have seemed rather arbitrary, since I've nearly always just ended up being "that guy who is keen on the software stuff..." The Research Software Engineer, a role described only really relatively recently, seems to have a clearer goal in sight than any I've ever had before.

Technology as a whole is, to me, a wonderful toolbox for discovery and exploration in both our physical and intellectual pursuits. To me I've really struck lucky in doing what I love with the express aim of finding new ways to improving the way technologists, scientists and any other interested parties can collaborate and discover things that help them use technology! That's really the goal for me, to help improve the tools and culture both within and outside of the organisations I'm involved with.

Building a profile

My living space. This is where I go to relax, separate from my office, or generally to think.

For all my over-exuberant and bold claims above, the truth is you have to take small steps to move forward.

The conclusion I've come to recently, before taking this job (and with growing older), is that if you're really quite passionate about something then it's worth building a profile, personal or not, to communicate through. Working in very, very commercial settings, this almost didn't feel possible. In fact, for me historically, the IT industry often actively inhibits openness. This thankfully has been changing significantly in the last decade or more, as newer companies have facilitated the ability for staff to communicate on their own terms (though as we've seen with recent stories this isn't always what it's cracked up to be.) Thankfully, the alternative for me in working in scientific and research oriented domains, has been another manner by which to start thinking about what and how I'd like to communicate with the world!

This is going to be an ongoing task and somewhat of a learning experience to do virtually. I've been lucky to be relatively good with communicating with people and organisations face to face, but the world has obviously been shifting and taken a proper swing (thanks to the pandemic) towards the importance of virtual engagement (we'll ignore the societal and psychological dangers of technology here. There are many positives to the judicious use of tools for social activity and community engagement, so that's not a discussion I'm getting into here!) Alongside the exciting work I have lined up (which I will write about in subsequent posts), I'm going to make more effort to use the blog to collect together resources for communication, such as my relentless love for music and cooking (well, eating and cooking). This will hopefully get me more active in communicating how fun it is to be a techie, working for science and looking at the future of research and the environment with a hopeful, yet realistic mindset!

The next few posts I hope to write about interesting projects I'm allocated to work on at BAS, those that I've been working on which still need concluding (but are interesting from my last role) and the efforts I'll put in to make this blog and site a little more interesting to read.

I hope maybe to give some insight about the perhaps slightly unconventional approach I've taken to get here as well, as I think it could be reassuring for those who, like me, are coming from non-standard, non-academic backgrounds wanting to contribute to science and research. Diversity and acceptance is more important now than at any other time in history, so contributing to the greater narrative can't really do any harm even if it's from a relatively humdrum individual!

Welcome to the life of a Research Software Engineer...

Science at Halley and Out of Antarctica 2018-2019

2021-06-13T00:00:00Z

So this is the last of the galleries from my time South. Though I did go again for a Halley summer in 2019-2020, I wasn't so active with the camera!

These were fantastic seasons to be involved with. The shutdown of Halley in 2017 left a big gap in winter science, especially in the long running Dobson dataset, amongst others. The British Antarctic Survey, presented with unprecedented and unforeseeable circumstances, automated the science infrastructure in less than two years giving us now, in 2021, a completely different landscape for Antarctic science into the future.

It's for this reason I love working for this organisation. It's a genuine bastion of scientific endeavour, one of many, and that gives me hope for the future because it shows that human civilisation has not lost it's desire for a sense of exploration!

The reason I've pumped these galleries out sharpish is to ensure they're done, that the chapter of my previous roles at BAS (Data Manager and Linux Engineer) reach a nice conclusion in my own mind. Come tomorrow I start a new role, that of Research Software Engineer, which plays to my previous experience as well as gives me hope that I can do something beneficial towards the scientific efforts of planet Earth.

I hope you enjoy the last of these galleries and into the future I hope that my writings about software, in and for science, among other subjects prove interesting to people, no matter what they do.

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Flying for Science at the start of Summer 2018

2021-06-12T00:00:00Z

With the start of the summer season well under way, I was privileged to get out as co-pilot on a couple of key flights. Firstly, to install the Fossil Bluff backup landing site, which was just a mind-blowingly good day out. Secondly, a trip to Halley as part of the season startup process (a team had already started work to start the season there) to start up some of the science devices we look after.

This was all towards the end of my time at Rothera, so the next gallery is the last, detailing the transition out of Rothera and the summer at Halley.

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

Working in Southern Springs 2018

2021-06-11T00:00:00Z

As the sun returns, so too does the encroachment of the world into Antarctica, with the first planes arriving and the preparations for the upcoming summer season. That said, some of the fruitiest weather is into spring as the sun and the global weather systems push back slowly into the continent.

The wonder of the station isn't lost in these times, as you reflect and enjoy life on base as much as you can, but mainly prepare for the switch in pace back to the austral summer season. A couple of us, originally Halley, returned there for the summer, as Halley itself remains uninhabited during the winter.

Some people do ask which I prefer, and there isn't an answer. I know for sure I love Rothera and I know I wish I could winter at Halley!

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

Winter Tripping 2018

2021-06-10T00:00:00Z

I've been a bit offline whilst some life stuff has been happening, but here is the first of four (to be released daily) of the last of my Antarctic galleries.

This is the record of my second winter trip. With the legendary field guide Dr Snow as my collaborator for the occasion, we decided to maximise recreational endeavours by staying in the hut in the recreation area and make daily trips out. This was opposed to gunning for a much more elaborate trip to the other side of the island.

I couldn't have been happier with the decision as we got several day trips in, with spectacular weather to boot. There's absolutely no way to describe how grateful I am for the experience.

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

Winter Life in Rothera 2018

2021-05-14T00:00:00Z

The irony of this being called Winter Living as a post is that the first photo is already past the Winter Equinox. However due to the ship only having left in late May and with the lead up to the midwinters week off being so busy with gift making, this album really chronicles the first bedding in of the winter team in isolation.

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

The beauty of Rothera in winter 2018

2021-05-08T00:00:00Z

Having finished the winter trip a couple of months before, this gallery shows the lead up to and fun times of midwinter week in June, a celebration akin to Christmas for people in the Antarctic. Full of fun and celebration, everybody makes gifts for one another and enjoys a bit of time to relax, whilst the sun doesn't rise.

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

It's a wonderful autumn in Rothera, 2018

2021-05-03T00:00:00Z

This gallery shows the more recreational side of the autumn at Rothera. The first of my winter trips as well as a shindig we put on in the Rothera garage. As we work hard, it's important to cut loose and have a break. The winter trips also provide the only respite from life on base for many, giving us an opportunity to go out and see the wider environment. It's a massive privilege to have been able to go out, especially with such experienced and lovely people.

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

Nature and Environment at Rothera 2018

2021-05-02T00:00:00Z

Yes, I know, it's amazing that there's a second gallery about the nature and environment of Rothera! This gallery accounts further amazing explorations of the local area whilst doing training and getting to know the team. Truly fantastic.

I hope you enjoy the pictures as much as I enjoyed the opportunity that afforded my taking them!

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

An introduction to life at Rothera, 2018

2021-05-01T00:00:00Z

This gallery really just records my introduction into the life and the beauty of Rothera, having arrived in 2018 and getting myself settled as a new member of the team. The majority of the winterers spend the summers either side on the base you winter at, but since I and some others were doing our summer (which are generally intense at either base) at Halley, we had to adjust to the new base and team.

As you can see it wasn't difficult. Being a lover of training to be useful, the operations and training to support them were a joy to behold!

I kind of realise a little now why I rarely look at these photos, it make me instantly want to go back!

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

Bye Halley, Hello Rothera 2018

2021-04-30T00:00:00Z

This gallery shows the last photos I took at Halley while we prepared for the shutdown of the base for the second time. The winter season I was moving over to Rothera, a base I'd not yet visited but the other of the two wintering bases on the continent (and now the only inhabited BAS base on the continent during Winter.)

Halley being on an ice shelf and Rothera being on the peninsula of the Antarctic, they're very different bases, but both ultimately set in quite beautiful environments!

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

Some Seasonal Activities at Halley - Summer 2017/2018

2021-04-28T00:00:00Z

During the summer season 2017-2018 (remember, this is the Southern hemisphere) I was privileged enough to get out for some brief fieldwork. Also we have to ensure our training is kept up to date, which gave some opportunity for on-base shenanigans to simulate breathing apparatus based search and rescue.

If you've missed the last post you can find it here, or if you've been ever so unfortunate to have missed all of them you can start here!

Next post in the series now available here

Return to Halley: start of the 2017/2018 summer season

2021-02-13T00:00:00Z

Following our being unable to winter Halley in the 2016-2017 season our team had to return to the UK, which was a massive shame. The prospect of spending winter at Halley in 2017 was most exciting. However, we were there to do a job and the right decision was made at the time, and it's afforded me many other opportunities!

Returning to Halley roughly 9 months later was a great feeling, hopefully illustrated in these photos, but not without its challenges (even to get there!) Returning as both science coordinator and "data manager" was unexpected, as we still had to move the science infrastructure 24km from Halley 6 to Halley 6A. It's amazing what was achieved, and it's definitely a life changing experience to have been involved.

Thankfully, this is just the first gallery of many more as I stayed in Antarctic over a year!

If you missed the first post covering my first season, which is way overloaded with photos, you'll find it here! Don't worry, from now on I've broken it down... ;-)

Next post in the series is available here

Blogging from the Britainosphere

2020-12-23T00:00:00Z

I've not yet been able to adequately post on here, primarily because I've not gotten comfortable with publishing my internal narrative. The brain is a strange place when confronted with isolation, which I know from previous, non-pandemic related experience. That said, I feel it worth having a bit of a ramble this morning, mainly to try and see whether I can write something that might be useful, interesting or at least publishable!

The subject of this is mainly just obvservational, it doesn't have a specific purpose!!! :-)

TL;DR

What's going on?

In the UK we've been greeted by Christmas restrictions, as have many other countries in the world. One wonders whether things have been more complicated than they need to be. There's an odd Churchillian rhetoric from the government whilst the advice is clear, we should probably be locking down again. There's no great answer, and I sympathise enormously with everyone's situation in the UK and around the world, because I don't think there's anyone who isn't having to adapt.

With that in mind, I look at the news as I sit here with a coffee and realise that my circumstances are very privileged, for which I'm grateful. I have a hot drink, a kettle and somewhere to sleep. I don't get to see family on Christmas, but given what the situation is and what we need to do, I don't really want to. The risk to both them and others to have someone travelling across the country is too great.

Looking at the media

I like watching the news, in many different forms, even though I find much of it inane and well deserving of firm satirical review. For example, when there's important news from around the world why has the UK media fixated on Nicola Sturgeon forgetting her mask for seconds, a HUMAN error to have made (which is stupid, but not exactly a scandal of Berlusconi proportions!) Why are they even listening to the opinions of Tony Blair, someone who I personally feel should still be tried for war crimes: hey, he started a war under false pretences, thus he's a criminal! Also he is not a scientist, he hasn't modelled anything, he doesn't show evidence relating to the efficacy of the vaccine between one or two doses and more to the point he's not taking into account the regulatory restrictions that have been approved based on a second dose!

Looking at the world, we have disasters unfolding that people should be educated about. I do wish the media would stop finding ways to make inane items of news dramatic, and just concentrate on the dramas that already exist.

The reason why I say this is, is that people are starving, suffering and being misled and we require honest investigative journalism to highlight the realities of the situations. In the UK for example, today millions are in trouble financially, the borders are closed stranding thousands of lorry drivers and many, many people are going above and beyond to support public services and healthcare.

There's plenty of things to report on without publicising the opinions of irrelevant has-beens and slight and irrelevant human errors by those (who admittedly should know better) who govern. Focus on the errors and solutions that matter. Show me how people help and how I too can help...

Staying positive in the face of news

Watching/reading the new agencies output makes me feel connected. However, you've got to be careful in these times. I select multiple sources for news (I primarily watch or read, in order, Al Jazeera, Reuters, Channel 4,BBC and then random bits and bobs on Youtube for US news) and multiple sources for satire (whatever I find). I think that's really important.

To remain positive this variety helps. The other thing is to ensure I don't watch the news when I'm feeling frustrated or angry about anything at all. Staying positive is important and getting irate or frustrated at the media doesn't make you feel connected, just more isolated. That's not great in these COVID times. That's when I turn to satirical programmes for a laugh.

I do feel connected when watching the news and manifesting compassion for those I see. Not only can the realisation of how lucky I am to be warm and safe be comforting, but it makes me realise the importance of what little civil responsibility I can exude by remaining at home this Christmas.

Concluding

I guess I'm kind of just waking up and realising this morning that my activities during these times are the correct one, and taking an interest without getting angry or frustrated is important to avoid becoming overly cynical.

There, see, it's not so interesting, but at least I've finally blogged about something other than technology. Perhaps I should just stick to that...

Deploying Analytics

2020-11-14T00:00:00Z

Overview

I've given in. It would be nice to have some analytics on the site and in looking for alternatives to Google Analytics I came across this very succint but informative comment on HN. Looking at Matomo it seemed like a good idea, I liked the "guarantee" that the self hosted option will be available for free, forever, without data leakage (hope this is true). However, THIS post about not burdening users is more my cup of tea, the way I was doing my own basic analysis was via the logs, so as usual why not use the data already present with a OSS solution!

I'm a big believer in seperation of concerns too, it's nice not to alter the site for a server side analysis!

I'm not interested in any more than looking at whether I get traffic. I'm not really interested in who comes to the user or where it is they arrived from. It'd be interesting for me to see whether, as that commenter states, any of my ramblings have been worth reading!

Deploying GoAccess

So at time of writing this is still a bit incomplete. For some reason the real-time data update isn't working (which I could probably fix easily, but I want to write this article before my long weekend off!) Also, this is only designed as a pattern for internal use only (via port forwarding, which if you don't know about, and you want to be a sysadmin, you definitely should. I'm sorry I don't know how good that tutorial is, but it looked alrigh at first glancet!!!)

Why don't I fix these up? It's because this is a quick and dirty deployment just to put in a tiny bit of monitoring based on the available data...

jimcircadian-role-goaccess and ansible role publishing

I'm planning on getting the docs done this weekend but here is the role I used. Having searched t'internet (west country slang there) there were only two contenders for roles relating to goaccess, this one byy barm0leykin and this one by Anon511. I thought about working off these, but I didn't, because by inference they're pinned for a static version of goaccess on Ubuntu, and weren't configurable. It was easier to start from scratch being a CentOS user...

Though I appreciate the usage of such projects, this is one of the downsides of configuration management systems: when publishing there's no impetus to do anything in a role past the barebones installation of packages. This is not best practice: give people the ability to build on a solid foundation! :-)

Tying the role into my ansible provisioning

If you've read my post on the provisioning setup I use which is hosted and freely available here then the addition of a new piece of functionality was trivial to achieve. In the web.yml include for the vps.yml playbook which gets run, we simply introduce the role:

- import_role:  name=jimcircadian-goaccess
  tags:         goaccess

Since I use include_tasks to group the includes together, the tag is also added to vps.yml. After that, using this provisioning system (which leverages invoke) I can simply run:

invoke ansible -e staging -t goaccess vps

To deploy to my staging vps server (which is actually sat behind me at the moment, quietly, since I added 6TB of storage it got a bit noisier)... There is also an annoying symlink in ansible/roles to the checkout of the role in my filesystem, which is a grimness I really need to fix: please bear in mind you will need to amend these as I've not submoduled anything at the moment!!!

For any experienced ansible dev, this will be super-trivial. However, in order to produce the necessary configuration I wanted for a trial run, I've obviously set some variables for the roles behaviour. This is the pattern I use for all behaviour exhibited as a result of the provisioning repository: the data dictates the behaviour the provisioning and role code exhibits... ;-)

I don't publish the variables controlling my provisioning code, for obvious reasons! However, for this implementation of goaccess I have sanitised and described it here so you can hopefully replicate it. This is not a final solution, just good enough for a trial...

XFF Log Format

The first thing that needed changing was my log formats for httpd. I run Apache behind haproxy and previously the proxied traffic could come in using the form n.n.n.n or n.n.n.n, n.n.n.n which was no good, as there was no clear identifier for goaccess to delimiter the host from in spite of the ~h{," } field described here. Therefore the host I believe is far better quoted to enable this extraction, and kudos to the this github issue and this one that resulted in changes to support XFF formats.

Therefore the first bit for me was to change:

      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
      LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

      LogFormat "\"%h\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
      LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

A virtual host

Since I use geerlingguy-apache for deploying httpd, the configuration for the virtual host was dead simple. This runs specifically on a non-standard port...

  - servername:   "metrics.inconsistentrecords.co.uk"
    documentroot: "/var/www/goaccess/html"
    extra_parameters: |
      DirectoryIndex report.html

This is thus accessible using a port forward to the non-standard port for access, with /etc/hosts doing the job of naming the destination to localhost on my local machine meaning the host field picks up the correct vhost in apache:

ssh -L <port>:127.0.0.1:<port> staging-vps

I think this is a nice little trick for things you just want to bind to the localhost on the vps, either for a trial or because you'll never need external access...

Goaccess configuration

With the role I've made there's minimal configuration to pick up the logs and produce some kind of data. The main issue is that I should need daemonize on to continuously collect data, but since I only go on there every now and then to look at what's happened over the last few days, I've not yet bothered to get it working properly with the systemd service.

goaccess_addr:        127.0.0.1
goaccess_daemonize:   false
goaccess_log_file:    "{{ goaccess_apache_logs_dir }}/access_log"
goaccess_log_format:  '~h{", } %^ %e [%d:%t %^] "%^" %s %^ "%U" "%u"'
goaccess_real_time_html:  true
goaccess_output:      "{{ goaccess_html_dir }}/report.html"

Conclusions

This is far from a complete project, but a good starting point and very much good for internal system use. Goaccess is a really lush project offering use of data collected unintrusively, which is exactly what I wanted. My main hope is that people find the role usable for CentOS and extendible for other uses...

Things to do:

Get daemonize working so that data is continuously updated
Add non-server specific auth to the role
Do the docs and show to people (this weekend)
Maintain as best as I can, in line with newer releases of goaccess and centos
Incorporate other OS implementations, which should be reasonably easy

Provisioning This Site

2020-11-13T00:00:00Z

Introduction

This really culminates the articles "Making this site" (part 1, part 2 and part 3), surrounding the development of this blog. I've been terrible at getting round to doing this for a whole host of reasons, not least due to the onset of lockdown part two!

Ansible is a tool I've been using for years, almost as long as I've been doing devops style stuff over "plain old" software development. Alas, I've never been as involved as some in the development of it as it's moved from strength to strength. That said, one thing I've never seen a lot of is people exposing their ansible infrastructure or the methodologies behind it. I can, for sure, say that some of the influences for mine have come from implementations I've worked on for massive infrastructures, but I feel compelled to open mine up as an example people can use.

This too is true for this site, though the Lektor structure I have put together is very rudimentary from what I've seen around. There are much better examples, but it's a work in progress. Therefore this article focuses on the ansible setup I use to deploy the site, and from here on out it's just going to be an incremental process as the passage of time flows on!

The provisioning repository

The point of having a provisioning repository was to make it shareable. The repository is intended as a top level container that links together the full orchestration process via use of the Invoke tool. However, I've never quite finished automating this too well, but it does contain everything nicely.

Please do forgive how loosely everything is put together. If you follow instructions or inspect it, you should find some method to the madness, but this is very much just what was required for recreational usage...

Structure

We have the following:

ansible: the ansible code required to build everything with some generic-ish inventory, playbooks and role setups
ansible/environments: this should be a seperate repository designed to provide the var_files looked up from the playbooks (in my case this is very much a private repo!!!)
ansible/playbooks: you should know what these are if you're interested in ansible, very much the starting point of the documentation!!!
ansible/roles: this generally contains only my personally developed repositories, plus requirements from galaxy.
kickstarts: contains CentOS7 kickstarts for various scenarios.
tasks: defines all the actions Invoke can perform from the CLI
Some requirements and configuration files...

The idea is that you always invoke tasks from the this base directory. The only time you need to go anywhere else is to copy kickstarts to an http server (you only do that to begin with) and to edit stuff.

Each of the following sections goes through these areas of functionality in order...

Kickstarts

There were two originally drafted methods of deploying my online presence.

The first was originally to set up a libvirt host using an ansible role with a set of VMs. This used to make sense when I used a physical server in a data center, but it didn't work so well with a VPS since they didn't provide virtualisation extensions when I moved host (lesson learnt, but it's way cheaper.) In the end, I redeveloped the ansible to make both options available, though you should consider the vps approach the only mature one.

Invoke

This is not very mature, but I use it as an easy proxy for running ansible-playbook. Basically, I can run any playbook with some structure via the following:

invoke ansible -e <env> [-t <tag(s)>] playbook

The point here is that it automates vault inclusion and saves me the hassle of doing too much manipulating the commands I want to run. "Is it worth it" I hear you ask? Well, it certainly is if you use this same approach on massive infrastructures, as you can automate injection of parameters via invoke to drive the ansible inventories and var lookups, but this isn't a great example of that...

I also use invoke for vault editing, there's just some more efficient command line-fu you can get too.

With the release of Fabric v2 I thought it would be easy to migrate, but I vaguely remember it wasn't as trivial as I hoped. Therefore, I just install invoke on its own. Though I love Fabric, Invoke and Paramiko, the developments and integrations within the ecosystem over the recent years have been a little hard to keep track of!

Ansible

It really is just ansible-playbook initiating playbooks. However, it's worth noting that recently the focus under RedHat has seen the adoption of collections, which I'm yet to understand, as well as more complexity in the underlying system.

You'll notice at the end of playbooks I have the following snippet:

  vars_files:
    - ../environments/all.yml
    - ../environments/data.yml
    - "../environments/{{ env }}/all.yml"
    - "../environments/{{ env }}/data.yml"
    - "../environments/{{ env }}/vault.yml"

The reason I do this is to make very explicit the variable hierarchy that I'm working to. env is provided by the invoke command line invocation (eg. staging or production at the moment, I've never bothered getting round to making docker work for my personal development process, as I've no requirement for it just yet, but there is something in the pipeline that would justify the effort...)

This is probably going to annoy ansible pro's, but I find it very important. A lot of the time with things like cloud-init, there's no reason to have host_vars. I consider it bad practice myself, for example web servers should be set up when scaling out horizontally and the orchestration process should provide host level variables, not the ansible repo.

I have in the past investigated the variable precedence in ansible and find it irritating to recall, especially as I seem to remember it changing or extending, or otherwise not suiting my group-oriented approach. By using var_files you can easily overcome the need to research this and limit the hierarchy to a set number of levels.

This is one thing I liked from Puppets Hiera (I don't like Puppet because of the DSL and it's huge amount of changes over the years), which asked you to DEFINE the hierarchy. When a data hierarchy is driving the configuration managements effect on deployment, it needs to be easily understood and obviously specified. Also, in my opinion, working with people of varying amounts of experience with such systems, a limit on the amount of levels is no bad thing.

Hopefully that explains the design a little. The data drives the code, which largely (there is an exception for an LetsEncrypt ACME deployment I've not yet moved into a role) is functionally defined by roles, included from playbooks...

Simples! ;-)

Environments

This is a private repository for me, but lives in the location mentioned above: ansible/environments and is structured to meet the requirements of var_files in the playbooks. Though I won't reveal the setup I have, it's up to the user of this repo to define what's necessary based on the knowledge of technologies contained therein to define the data in a way that deploys they way they want.

That would be a very large article, so for the moment I'll just show you the structure of my repo so you know how to start, then you can define based on identifying key variables from the underlying roles.

./data.yml
./development
./development/all.yml
./host.yml
./production
./production/all.yml
./production/data.yml
./production/proxy.yml
./production/vault.yml
./production/web.yml
./staging
./staging/data.yml
./staging/web.yml
./staging/host.yml
./staging/proxy.yml
./staging/vault.yml
./staging/all.yml
./web.yml
./proxy.yml
./all.yml

Ignore the development environment in my case, I never did anything with it, as I said before! ;-)

Improvements I've never made

Here's a load of things I'd like to do, but never did:

I never created a kickstart hosting invoke task to save on having kickstarts deployed - never needed it yet, but easy to do
I intend to use docker for playing about with the web, data, proxy structure: but that was pre-VPS
I've never quite settled on whether include_role, include_tasks, import_role or import_tasks is the way to go. It's largely because I find tags EXTREMELY useful for rerunning tasks, such as renewing certificates, and imports don't work properly. If I'm honest I've just never solidly bothered to read the docs since they added all these in. Though I think things are working, who knows for some of it...
I really should get round to publishing the ISSO role I wrote, which seems to work well for me!!! I love ISSO, it offers comments without selling people out for leaving them...
I still haven't worked out how I feel about cookies, especially in relation to analytics. So at the moment this should be largely cookie free.
The blog section is the only part really completed so far: the projects need deriving from github, the music section doesn't exist yet and I want to offer a better way of accessing my Antarctic galleries without having to trawl through posts.
I do a lot in my job that I want to blog about, far removed from what this site is so far about. Give it time...
Probably should have an RSS feed or whatever the vibe is these days! ;-)
There are loads of things to consider, especially as I have other web applications to bring under here. The staging environment becomes far more complex as a result (it already is). By inference I should split the role requirements based on the ansible environment being deployed....

The ansible I write is completely idempotent (as far as I've ever realised), so you should be safe to run over and over and over and over again... provided you configure environments properly...

Conclusion

Since we're in the middle of lockdown number two, I'm happy I've concluded this piece on building the VPS out with Lektor. I really hope people find things useful, but also welcome comments about what information I've left out.

There are plenty more articles on the way, but hopefully this has got me caught up now things are working properly!!!

Making This Site (Part III)

2020-10-18T00:00:00Z

It's been a bit busy recently but, now, I return to the blogosphere mainly as an escape from the insanity of modern life!

There are a few things that I'm keen to solve, which I must admit I've not found intuitive to get my head round in Lektor just yet. I've not tried hard, but definitely if it's not immediately intuitive (or is outstanding from part 1 or part 2) then I fancy having a crack at wrapping it up in this post. At least from the website point of view.

The final post will focus on the underlying Ansible provisoning I've put in place which, to be honest, I'm hoping others might benefit from. Anyway, the remainder of this post will focus on what I consider to be the final bits required to make this a usable site!

Querying the project settings from Lektor in templates

Data bags are in fact the manner by which to approach this, but you can query the core site settings too. This was an outstanding thing I wanted to find the solution too, it's just about RTFM :-)

Site settings

This really did baffle me a little bit, though in the end it was quite trivial:

{{ config.PROJECT.url }}

Will pull the url out of the project group in the lektorproject file for the site. This was clearly spelled out on this documentation page so no criticism there! What I don't really understand is the uppercasing of configuration groups, and of course that wasn't documented. One day I'll figure out why that is.

Data bags, and considering "dev" alternate data

Data bags turn out to be the other very useful element of Lektor. I configure the comments settings (see the next section) using these INI based configuration settings. That, coupled with the above finishing of looking up project settings, is awesome.

One thing that has made me think is that there is no accounting for alternate settings when, for example, you're using lektor serve to view/edit the site. Therefore when using comments it'd be great to be able to alter the endpoint, which of course is contained in the static HTML as you might expect.

What I'd like to be able to do is use lektor deploy to dynamically either:

index different databags on deployment for the output files from a command line setting
allow execution of arbitrary build actions with the static file generation from the command line

This might seem odd, but I noticed with Isso that I really wanted the dev, staging and production instances to point at their respective Isso backends. Unfortunately that does not seem possible because the data is not obviously easy to reference based on the deployment/serving environment.

Comments with Isso

This I thought would be much more of a mission than it's ended up being, but actually I really rate the Isso project. It's simple and unintrusive, and it's nice to know that I'm not feeding peoples data to an external party.

I think there's an entire article able to be devoted to this, an Isso role for ansible has been created as part of the provisioning setup I use to build out the VPS. So the provisioning article will start me off on that one! :-P

All of this needs a bit more documentation, so please forgive how messy those links look! I also don't think the mail subsystem for isso is playing nicely with Postfix yet. For the moment, here's the macro, and that's really all you need for the frontend!

    <script data-isso="{{ bag('comments.url') }}"
            data-isso-vote-levels="true"
            src="{{ '/static/js/embed.min.js'|url }}"></script>
    <section id="isso-thread"></section>

Image Gallery

This was fun. Again maybe a little article on its own. Basically I wanted to start the process of sharing the amazing Antarctic experiences I am privileged to have, and hopefully continue having, as a result of my work.

I have a lot of love for the lightSlider and lightGallery projects. :-)

As with Isso, the galleries are all about taking advantage of Lektors captioning (via attachments) and thumbnailing capabilities with a nice and simple macro:

{% macro render_gallery(item, id="gallery", title="Photos") %}
  <ul id="{{ id }}">
    {% for image in item.attachments.images|list|sort(attribute='path') %}
    <li data-thumb="{{ image.thumbnail(300) }}"
        data-src="{{ image|url }}"
        data-sub-html="{{ image.caption }}">
        <img src="{{ image.thumbnail(300) }}" alt="{{ image.caption }}">
    </li>
    {% endfor %}
  </ul>
  <script language="javascript">
    $(document).ready(function() {
      $('#gallery').lightSlider({
          mode: "slide",
          thumbItem: 6,
          currentPagerPosition:'left',
          controls: true,
          pager: true,
          loop: true,
          auto: true,
          speed: 800,
          enableTouch: false,
          enableDrag: false,
          onSliderLoad: function(el) {
            el.lightGallery({
              selector: '#{{ id }} .lslide'
            });
          }
      });
    });
  </script>
{% endmacro %}

Issue with Relative Links

So this does not work:

[Next post in this series...](/blog/making-this-site-ii)

But this does:

[part 1][1]
...
[1]: /blog/making-this-site/

The former is the case for the original post in this series (which at the moment has the FQ URI associated), the latter is used in this article. In the first case I use .../blog/blog/... which is a bit odd. Removing blog doesn't work, I end up with no blogs at all!

I need to look into this more, but if you suffer the same problem, use the second option... :-)

Issue with cache not responding to macro changes

Another thing I've noticed is that updating HTML macro files in the templates/macros directory does not trigger caching. Another thing to have a look into on my part, but do let me know if you've noticed something similar!

Conclusion

Most of the items in this article were the items I thought needed sorting out prior to offering the site out for people to view. Obviously it's indexable by search engines and wotnot, but I might push this out somewhere as I hope some of my experiences and write ups will help someone at some point! I didn't want to do that until people could look at some pictures and/or comment on things.

You may ask about the backend. Well, the provisioning is going to be a future write up. Analytics are not my bag. I don't want people to visit the site and worry about whether it's another vector for their browser information to be ingested.

I'm certain there's nothing on here or any analysis being done that compromises peoples privacy. I intend it to stay that way.

Ultimately, Lektor has more than proven it's worth. The fact it's so easy to write this with a beer is the only reason I ever do. Hopefully, this is the start of much more in spite of lifes distractions. I'm looking forward to writing about the provisioning system I use, one of many I've developed/worked on.

I'm also keen to write a bit more about what I do during 95% of the time, when I'm not drinking beer (which is when I work on this site.) :-P

Antarctic Part I, Summer 2016-2017

2020-10-07T00:00:00Z

These are some photos of my first season in Antarctica. This is partly a desire to develop the site more, but motivated mainly by a desire to share the wonderful experience I had.

Hopefully these posts will serve inspiration or interest to budding technologists. It's important highlight that the possibilities are endless in life if we step out of our comfort zone once in a while. It's also important to realise that your skills, no matter what they are, are required all over the world in some form: I've never found any other experience that allows you to work with such a broad variety of people!!!!

This was my first summer, which was meant to be the start of a 16 month winter at Halley Research Station. Due to the glaciological situation of the site on which it resides, we weren't able to winter in the end, but it still started a path that I continue to walk to this day, for which I'm very grateful!

Use the dots or controls to scroll, at the moment if you click on an image it'll open the full gallery...

I realise this gallery is rather large, subsequent ones will be split up! ;-)

Next post in the series available here!

Life in the age of Lockdown

2020-09-17T00:00:00Z

I've no interest in writing about the positives or negatives of life lived under lockdown. Being in a pandemic, the world mostly knows what's happening as a result of recent events. I only hope that it afflicts fewer as time progresses, the world has enough to deal with.

What I do want to write a little about, since this is the first time I've really thought about publicising my thought stream (not that I, for a single moment, think anyone will actually read or take notice of it), is the effect it's had on me as an individual. There are numerous people like me around the world, and with the adoption of technology ever more, who find themselves interacting socially, learning and operating generally over the medium of the internet.

I, apparently and with much chagrin, am defined as a Millenial. It's kind of bollocks really, since neither the internet (in the form of the World Wide Web) nor social media existed when I grew up. Compuserve did. Telex was around. Those in the UK know about Ceefax and Teletext. When I started using computers, at the hearty age of five years old, nobody told me that I could use a modem. This was pre-1990s. So I became a programmer.

Compare this with my niece, who is also defined as a Millenial, who to be fair did not adopt programming as a hobby prior to turning to double digits in temporal cellular growth, and you'll see why I find such terminology dumb. As fuck. Somebody born in the 1990s had to deal with what could only be described as a genuine onslaught of information thrust uncompromisingly into their face, with zero guidance about how to apply filters to that onslaught. Those born in the 90s had to grow up, for the first time in human history, with too much information to process and too many devices available with which to access it. The adults who were meant to teach them were clueless and incapable of providing guidance.

This brings me back to the point of the pandemic and my reaction to it. The very fact I'm writing this blog, have learned to learn using Coursera, use messaging applications and make video calls is intrinsically linked to the fact that I won't realisitically (sensibly) see people for socially for a considerable amount of time. Whether you agree with isolation for the protection of others is irrelevant, we've all adapted, young and old, to use technology for the most positive purposes we can derive in such strange social situations.

To that end, there is a silver lining: the value of technology is being realised. Don't get me wrong, there is still doubt about our information rights, the morality of profiteering and the powerlessness of individual agency in the current political landscape; but there is also some realisation from people across all demographics that these evil little money-grabbing, attention-seeking devices the children of the 1990s became addicted to might genuinely serve some beneficial purpose!

For a child of the 80s, cast with the sullied reputation of post-Berners-Lee ignorance to the value of the Internet (based on age rather than knowledge), who took Fox Mulders "Trust No One" far too literally, this fills me with hope. People are hopefully beginning to realise that technology AND science holds some answers to the challeges of recent and future generations.

This rambling post was written with a beer, listening to this quite beautiful concert by Olafur Arnaulds...

Making This Site (Part II)

2020-09-01T00:00:00Z

So, on I continue with making this site a bit more satisfying. It's one of those nice to get some dub on and just settle into something easy to achieve with minimal brainpower.

Lektor Plugins

lektor-tags

So the first thing I've done is to try and use the lektor-tags plugin, which seems to work pretty well: it really is about just following the README!

A couple of things I've noticed during this is that lektor using server locally does not tell you which plugins actually load or don't load. Even with -vvvv there is no output to tell you that the plugin has loaded or not and because I made a small spelling mistake this became a bit annoying as there was no logging to tell me the plugin in the project didn't load.

lektor-rst

So there's no README for the lektor-rst project which is a shame and once again there is no conclusive information about the success of the load, though it is clear from the virtualenv installation that it installed into the build directory:

$ lektor plugin add lektor-rst
$ lektor server -vvvv
Updating packages in ~/.cache/lektor/packages/3fcdd194880fbf2be796613aee89c221 for project
Collecting lektor-rst==0.2.0
...
Successfully installed ...lektor-rst-0.2.0...

I'm guessing that I can now select rst as a document type for my models which is nice. It's a slight shame that it's mutually exclusive (since all I want to do is have the option), so I wonder if I can implement a checkbox mechanism to experiment with this only in certain posts, or just suck it up and have two fields.

lektor-markdown-header-anchors

So this was interesting, in adding this project we avoid the virtualenv issue with plugins (see below). The full process for this is a lot more lightweight due to utilising this bdist_wheel --universal feature

$ lektor plugin add lektor-markdown-header-anchors
Package lektor-markdown-header-anchors (0.3.1) was added to the project
$ lektor server -vvvv
Updating packages in ~/.cache/lektor/packages/3fcdd194880fbf2be796613aee89c221 for project
Collecting lektor-markdown-header-anchors==0.3.1
  Downloading lektor_markdown_header_anchors-0.3.1-py2.py3-none-any.whl (4.1 kB)
Installing collected packages: lektor-markdown-header-anchors
Successfully installed lektor-markdown-header-anchors-0.3.1
WARNING: Target directory ~/.cache/lektor/packages/3fcdd194880fbf2be796613aee89c221/__pycache__ already exists. Specify --upgrade to force replacement.
 * Project path: ~/.../inconsistentrecords.lektorproject
 * Output path: ~/.cache/lektor/builds/0fdf35440669b574822909b8822e9aa1
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Lektor and virtualenv

I'm very intrigued by the reluctance that Lektor has to optimise for virtualenv installation. Since I generally use virtualenvs or conda for everything I do on my system, having a system managing itself via these mechanisms without letting me manually go about the installation is a negative experience for me.

Another problem with this is after virtualenv installation of packages the overhead in the lektor/packages directory is pretty high.

[ ... ~/.cache/lektor/packages ]$ du -hs *
84M 3fcdd194880fbf2be796613aee89c221
74M d00edb3d4a285124a2ee822f57b33698

For two relatively simple packages it is quite a high price to pay to use 150MB for the convenience of having dependency management taken out of my hands. What I will do is elaborate on whether it's possible to circumvent this easily, though for the moment I took the bootstrapped environment for the sake of simplicity. I can see the logic for usability, but am interested in the ease of development/contributing, as I am keen to switch this mechanism off so I can easily install these plugins locally and contribute back.

Conclusion

I still really like Lektor, as with any new project it's great to see the innovations made and get the bit between my teeth to delve into something new. This was always about having some really quick blogging software as well, and I can totally see this fulfils that goal as I haven't scratched the surface of Lektor and it's plugin ecosystem yet. That said, I'm skipping past doing this a little to get things published and done!!!

At least there's a record of what I need to do now. ;-)

PS. Also noticed some weirdness in trying to link between blogposts using relative URLs in the markdown body. Best investigate that too in case I'm being dense!

Making This Site

2020-08-29T00:00:00Z

I had a good play with raw bootstrap based static sites, using Jinja. I played around with Jekyll in the first iteration of the site which I liked: I thought might be nice to learn Ruby off the back of that. But in reality these "nice" ideas always end up being more work than they're worth. To be honest I really like using Python, I don't need any more barriers to putting something. I don't like blogging, or self promotion or anything like that, so barriers are easily constructed if you're not careful.

As a result, I'm using Lektor, it's a nice, Python based file based CMS which is totally extensible. That's easy, flexible and functional enough not to produce any barriers.

I've munged in a bit of bootstrap and jquery around the sides, as I don't do aesthetics. I have looked at NPM and the like for resource management but I am not doing anything fancy enough to bother at present (this may well change!)

Again, too many barriers. I'm keeping it simple. Thankfully we live in a world where one doesn't have to pour hours into determining if something will work across browsers. In fact having to do that 10, 20, 25 years ago is probably what put me off web development and design.

It was mind-numbingly boring in the old days. As it is I might never end up updating the site, but we'll see... In this age of lockdown, putting yourself out there on the internet is not a bad thing, especially considering I was very much a tech-for-engineering, face-to-face socialising type of animal before all this.

I salute the W3C and framework builders around the world, you've finally made putting together a website easy enough that the most stubborn backend/infrastructure programmer will put one together.

Anyway, this is a working base, but I'll make some new parts to this post to cover what I do with lektor to:

Tag blog posts
Integrate with my github so I can blog about stuff I genuinely like doing
Really get under the hood of Lektor to add richer content and automation (e.g. Flows, Data Bags and Image Operations are all interesting and deserve some blogging about I think)
Blog about some of the things I actually do day to day
Make the styling a bit less gross!

Next post in this series...

Welcome

2020-08-28T00:00:00Z

Welcome to Jims Blog. As I’m sure is so often the way for a life-long techie, I’ve been intending on doing this for years and just never really gotten round to it.

I never really thought about what I might like to write!

My experience is rooted in software development and for years and years I worked in organisations that didn't allow or need me to create a public presence. As it is I am still employed in that manner, but thanks to lockdown and a desire to communicate anything that might be useful to someone, somewhere, I found the time to get round and do this.

I hope you enjoy this blog and find something useful along the way.

I hope I keep adding to it once it’s online! :-)

As of publishing this and the making this site post, I'm really just getting something in place to build on, so excuse the rough edges...