This is another one of those “I kept googling the same thing over and over again” things that needed a post, except this time I made an issue to make a post and then started to repeatedly refer to that. TL;DR When you want to spin up an experimental environment and get it tied in to your Jupyter environment of choice (I actually quite like JupyterLab Desktop these days…), you need two steps. ...
Messing around with Elections NI data Sources: Live Data (for 2023) 2022 Assembly Elections Creating your own Google Sheet and referencing the crowdsourced data The above linked spreadsheets are naturally not editable by everyone; this is great for reliable data but isn’t so great when you want to make pretty graphs. Google Sheets supports the live referencing of external sheets in your own sheets, so you can ‘import’ the data from the read-only sheets as they evolve over the count, and then reference those data in your own visualisations. ...
This was going to be a really quick post yesterday, but I’ve spent the guts of a day (between actual work) just getting the simplest bit of this working. The intent I was silly enough to say this to someone recently in work: It’s a good day when I end up with fewer firefox tabs open than when I started And a plan was hatched; Find API to query Firefox for current number of open Tabs across all window instances Send that value to my internal MQTT node (that has telegraf, influx, home assistant and grafana integrations too) Repeat Spoiler alert, number 3 was the bastard ...
The below was presented as part of the Belfast Linux Users Groups’ May Technical Meetup. I’m also maintaining this ‘short’ as part of my gradually accumulating ‘python_data_workshop’, so that may be the best place to see updates if you’re reading this any later than 2021. Conditions Theme: ‘Universal Screwdrivers’ Event: BLUG Time: ‘5-10mins’ (take the 10) Slides Transcript Evening folks, and I’m happy to be appearing in what we hope is the last virtual BLUG! We’ve all spent a long time toiling in our own little worlds in relative isolation over the course of this sequence of lockdowns, and its appropriate that now as we begin coming back together, that we’re talking about projects that operate and complement across ecosystems. ...
I was asked to help out a friend who had an installation in an art gallery that stopped booting properly, and was reminded that I keep forgetting to actually write this post. Running a python script as a reliable, retryable service on a Raspberry Pi that waits for an ‘up’ network connection, because I’m an idiot who keeps changing his mind how to do it. Fail gracefully and with informative error messages: try: ... except:... is your friend ...
I am a heavy user of Jupyter Notebook, both personally for wrapping my head around Open Data, professionally for analysis and reporting, and for education/presentations. So am very comfortable with just spinning up new notebooks all over the show. However, this ends up looking like this… Less than informative and impossible to work out WTF I was doing. Helpfully, there’s a way to change it. In your jupyter_notebook_config.py file (Normally in ~/.jupyter), add the following somewhere sensible ...
After last weeks return to posting, I thought it was time to do something vaguely useful (and actually talk about it) so I’m tidying up a few meetup sessions I’ve presented at into a series of Basic Data Science (with Python) posts. This is the first one and covers my Python environment, the Jupyter notebook environments I use for analysis, and some on the Plot.ly graphs and RISE / Reveal.js methods I use to turn those notebooks into presentations. ...
This post is a little different from my usual fare; Basically, there was a tweet from MATRIX NI that caught my eye; the latest Office of National Statistics report on Internet Use in the UK. @MATRIX_NI @ONS And the tables show Northern Ireland being around 7% behind the Avg and 2% behind the next-worst-region... — Andrew Bolster (@Bolster) May 22, 2015 Basically, NI “lost”. So I thought it was a good opportunity to play around with the data a little bit instead of my usual stuff. ...
Quick and dirty one that tripped me up. Recently I’ve been doing lots of multiprocessing and joblib-based parallel processing, with loooong simulation times. In an effort to make sure that my machine was still useable during these runs, I changed the ’niceness’ value of the spawned processes… or so I thought. import os ... def thread_mask(args): # Properly Parallel RNG #http://stackoverflow.com/questions/444591/convert-a-string-of-bytes-into-an-int-python myid=current_process()._identity[0] np.random.seed(myid^struct.unpack("<L",os.urandom(4))[0]) os.nice(5) return long_simulation(args) First part is a handy way to make sure that your subprocess simulations actually use different random numbers…. which for Monte Carlo style simulation is pretty damned important… ...
Quick one more as a reminder to me than anything else. As part of my PhD work I’m building different behaviours for virtual submarines. I’ll be explaining some parts of my work in a separate post, but basically, I needed to random walk. Random walk in 2 dimensions is easy; pick two random numbers, go that way. Unfortunately doesn’t work that way on a spherical surface So to make things easier, I stole this StackOverflow answer from dmckee and tidied it up a bit for my purposes. (Assuming everyone else is like me and does import numpy as np) ...